Turn Your Website Into an AI Knowledge Base
Crawl your entire website and convert it into a searchable AI knowledge base with automatic vector embeddings and incremental refresh.
问题
Your website contains the most up-to-date information about your products, pricing, documentation, and policies — but your AI workflows can't access it. Teams manually copy-paste web content into documents, which go stale immediately. Support agents answer questions from outdated knowledge bases. Sales reps reference pricing pages that changed last week.
解决方案
JieGou's website crawl pipeline automatically discovers, crawls, and indexes your entire website into a searchable knowledge base. Point it at your sitemap, configure crawl rules, and the pipeline handles the rest — extracting content, chunking for optimal retrieval, generating vector embeddings, and storing everything in Firestore with sub-second search. Incremental refresh keeps your knowledge base current without re-crawling unchanged pages.
工作流程步骤
Sitemap Discovery
配方步骤Fetches your sitemap.xml and discovers all indexable pages. Supports sitemap index files, nested sitemaps, and URL-based discovery fallback.
Smart Filtering
条件Applies exclusion patterns (e.g., /admin/*, /staging/*), URL canonicalization, and depth limits. Pre-crawl estimation shows page count and estimated processing time.
Crawl & Extract
平行处理Crawls pages in parallel with configurable concurrency. Opt-in headless Chromium for JavaScript-rendered SPAs. Extracts clean text content, stripping navigation, footers, and boilerplate.
Chunk & Embed
配方步骤Splits content into optimal chunks using heading-based splitting with paragraph fallback. Generates vector embeddings via OpenAI text-embedding-3-small and stores in Firestore.
Incremental Refresh
回圈Scheduled re-crawl checks for changed pages using content hashes. Only re-processes pages that have actually changed — saving compute and embedding costs.
Vector Search Ready
配方步骤Knowledge base is immediately available for all recipes and workflows. Firestore-native vector search with Redis caching delivers sub-second retrieval.
预期成果
- Your entire website becomes a searchable AI knowledge base in minutes
- Support workflows reference the latest product docs automatically
- Incremental refresh keeps knowledge current without manual intervention
- Sub-second vector search retrieves relevant content for every AI interaction
- No external vector database required — Firestore handles everything
学习回圈实战
Website is fully indexed. Recipes and workflows start retrieving web content via RAG. Retrieval relevance is good for well-structured pages.
Incremental refresh has run multiple cycles — knowledge base tracks website changes automatically. Teams stop manually updating FAQ documents.
Knowledge base covers 100% of website content. Redis caching delivers sub-second retrieval for repeat queries. Support accuracy improves measurably from always-current web content.