Engineering

Turn Your Website Into an AI Knowledge Base

Crawl your entire website and convert it into a searchable AI knowledge base with automatic vector embeddings and incremental refresh.

Le problème

Your website contains the most up-to-date information about your products, pricing, documentation, and policies — but your AI workflows can't access it. Teams manually copy-paste web content into documents, which go stale immediately. Support agents answer questions from outdated knowledge bases. Sales reps reference pricing pages that changed last week.

La solution

JieGou's website crawl pipeline automatically discovers, crawls, and indexes your entire website into a searchable knowledge base. Point it at your sitemap, configure crawl rules, and the pipeline handles the rest — extracting content, chunking for optimal retrieval, generating vector embeddings, and storing everything in Firestore with sub-second search. Incremental refresh keeps your knowledge base current without re-crawling unchanged pages.

Étapes du workflow

Sitemap Discovery

Étape recette

Fetches your sitemap.xml and discovers all indexable pages. Supports sitemap index files, nested sitemaps, and URL-based discovery fallback.

Smart Filtering

Condition

Applies exclusion patterns (e.g., /admin/*, /staging/*), URL canonicalization, and depth limits. Pre-crawl estimation shows page count and estimated processing time.

Crawl & Extract

Traitement parallèle

Crawls pages in parallel with configurable concurrency. Opt-in headless Chromium for JavaScript-rendered SPAs. Extracts clean text content, stripping navigation, footers, and boilerplate.

Chunk & Embed

Étape recette

Splits content into optimal chunks using heading-based splitting with paragraph fallback. Generates vector embeddings via OpenAI text-embedding-3-small and stores in Firestore.

Incremental Refresh

Boucle

Scheduled re-crawl checks for changed pages using content hashes. Only re-processes pages that have actually changed — saving compute and embedding costs.

Vector Search Ready

Étape recette

Knowledge base is immediately available for all recipes and workflows. Firestore-native vector search with Redis caching delivers sub-second retrieval.

Voir le workflow Engineering en action

Résultats attendus

Your entire website becomes a searchable AI knowledge base in minutes
Support workflows reference the latest product docs automatically
Incremental refresh keeps knowledge current without manual intervention
Sub-second vector search retrieves relevant content for every AI interaction
No external vector database required — Firestore handles everything

La boucle d'apprentissage en action

Semaine 1

Website is fully indexed. Recipes and workflows start retrieving web content via RAG. Retrieval relevance is good for well-structured pages.

Semaine 4

Incremental refresh has run multiple cycles — knowledge base tracks website changes automatically. Teams stop manually updating FAQ documents.

Semaine 8

Knowledge base covers 100% of website content. Redis caching delivers sub-second retrieval for repeat queries. Support accuracy improves measurably from always-current web content.