1
Start the crawl
POST /v1/knowledge with one or more site URLs. It returns immediately at status: discovering — the crawl runs in the background.http:// or https://, and at least one is required.What the crawl does. For each seed it walks the site’s own link graph breadth-first, staying on the same root domain, and unions that with whatever the sitemap lists — robots.txt is read for Sitemap: directives and the usual /sitemap.xml paths are tried, with sitemap indexes followed. Links are fetched as plain HTTP, which is fast and is enough for almost every site. If that turns up nothing at all — a site rendered entirely by JavaScript with no server-side links — it falls back to rendering pages in a headless browser to collect URLs.PDFs linked from the site are collected as pages in their own right rather than crawled into.The crawl also records which page linked to which, which is what the review step groups by and what the transcription stage uses as context.max_pages defaults to 1000 and is clamped to 5000; anything higher is silently reduced.Poll until status is review.2
Choose the pages worth indexing
At This choice decides the quality of the whole knowledge base. Prefer pages that answer questions a caller would actually ask over everything the crawl happened to reach. Every extra page is another entry competing for the model’s attention in the map, another summary that can be confused for the one it wanted, and another page of model spend at build time. A knowledge base of 60 good pages beats one of 600 that includes the careers section.The selection is truncated to the first 5000 URLs. An empty selection is a 400.In the dashboard this step is a pruning tree rather than a picking list: URLs are shown nested under the page that linked to them, siblings sharing a path are folded into one collapsible folder, and you delete branches you do not want. Whatever remains is what gets indexed. Clicking a URL previews the page in place, which is how you tell
review, GET /v1/knowledge/{kb_id} returns discovered — every URL the crawl found — and parents, mapping each URL to the page it was discovered on. Pass the subset worth indexing to POST /v1/knowledge/{kb_id}/build./kb/a/2019-11-14 apart from /kb/a/2019-11-15 without opening twenty tabs.3
Wait for it to finish
The knowledge base moves to What transcription does. Each selected URL is rendered in a headless browser, and a model rewrites the extracted text into clean Markdown: navigation, cookie banners, footers and repeated boilerplate are dropped, headings, lists and tables are kept, and every fact — names, numbers, dates, values — is reproduced verbatim rather than summarized. The model also writes the page’s title and a two-to-three sentence summary, and it is those two that end up in the agent’s map.Up to 3 content images per page are transcribed by a vision model and appended to the page text. A page that yields fewer than 40 characters after all of that is recorded as
transcribing, then generating, then ready. Keep polling GET /v1/knowledge/{kb_id}: progress says which stage it is in and how far through, and error says if it stopped.failed — a page with nothing on it and a page that could not be fetched land the same way.PDFs. Every page of a PDF is rendered and vision-transcribed to Markdown, which handles scanned and heavily designed documents the same way as text ones. A PDF of 6 pages or more is split into chapters — from its embedded bookmarks if it has them, otherwise from a printed table of contents, otherwise from headings, otherwise by asking a model where the breaks are. Each chapter becomes its own page at <pdf url>#1, #2, and so on, under a document stub whose content is the chapter listing. That is deliberate: a 100-page manual returned whole would be cut off by the fetch limit, whereas a chapter fits.Pages are transcribed parents-before-children so a child page gets its parents’ titles and summaries as context — which is how a brochure PDF linked from a product page gets a title that names the product rather than “Brochure”.Each page is saved the moment it finishes, so a build that is retried internally picks up where it left off rather than starting over.Then one model pass reads every page’s {url, title, summary} — laid out as the site’s own link tree — and writes the table of contents plus the overview paragraph. That is the generating stage, and reaching ready is what makes the knowledge base attachable.Statuses and progress
status is the state machine. progress.phase is where inside a stage it is.
progress also carries a human message — the string the dashboard shows — and on the transcription stage total, done and failed.
Watching a build live
GET /v1/knowledge/{kb_id}/events is a text/event-stream that stays open for as long as you hold it. It sends a snapshot frame first (and again on every reconnect), then progress frames as pages complete, then a status frame when a stage finishes.
Each frame is
event: <name> followed by data: <json>, and the JSON repeats the name in its own event field. A line beginning : is a keep-alive, sent every 15 seconds. Read it with an SSE client — the body is never a single JSON document.
GET /v1/knowledge/{kb_id} reads, so an event dropped while nothing was listening self-heals on the next read. Polling get_kb is a complete substitute; the stream only saves you the polling.
What can go wrong
These are the operational facts worth knowing before you build one in anger. Builds take minutes. Every page is a browser render plus at least one model call, and a large PDF is one model call per page on top of that. Do not build one inside a request a person is waiting on. Only one build may run per knowledge base. Starting a second whilediscovering, transcribing or generating is a 409 naming the current status: knowledge base is transcribing — wait for it to finish. The same applies to regenerate_toc.
A failing stage never retries. It lands at error with the whole incident in the error field, and stays there. There is no backoff and no second attempt.
A page that fails does not fail the build. It is marked failed in list_pages and the rest carries on. The build only fails outright when nothing at all could be extracted.
Some sites cannot be crawled. A site that blocks automated browsers, sits entirely behind a login, or serves nothing on a bare fetch will produce few URLs or none. no URLs found (site unreachable or nothing crawlable) is the error for the last case. Talqing will not follow a redirect to a private or internal address, at discovery or at render time.
What is fixed at creation
name, urls, max_pages and notes are set when you create the knowledge base and there is no endpoint to change them. Building again re-uses the URL inventory the original crawl produced; to crawl a different site, or to crawl the same one wider, create a new knowledge base.
notes is worth spending a minute on. It is plain-English guidance handed to both the transcription model and the table-of-contents model — “skip anything about investors”, “group products by series”, “keep model numbers exactly as written”. It cannot filter which URLs are crawled; that is what the review step is for.
What it costs
Crawling, transcription and the table-of-contents pass run on Talqing’s own models. They do not use your provider keys, and they do not draw on your credit balance. What a knowledge base costs you at runtime is what its map adds to the agent’s system prompt on every turn, and the tokens of whatever the agent fetches — both on your own key, like every other model call an agent makes. See using knowledge in agents.Next
Fixing one that came out wrong
Read the map the agent gets, find the pages that never made it in, and repair entries by hand.
Attaching it to an agent
kb_ids, knowledge_fetch, and prompting an agent so it consults its knowledge on a live call.