ketch

$ brew install ketch

$ npm install -g ketch-cli
$ npx -y ketch-cli <command>

$ go install github.com/1broseidon/ketch@latest

ketch is one of the chain.sh tools. One command installs the set:

$ curl -fsSL https://chain.sh/bootstrap.sh | sh

$ ketch search "query" --limit 10
$ ketch search "query" --scrape          # fetch full content per result
$ ketch search "query" -b brave
$ ketch search "query" --multi           # federate, rank-fuse
$ ketch search "query" --multi=brave,exa # a specific set

--multi fuses rankings with Reciprocal Rank Fusion — a page several engines rank highly floats to the top — deduplicating by URL and tagging each result with the engines that returned it.

$ ketch code "http.NewRequestWithContext" --lang go --limit 2
---
query: http.NewRequestWithContext
lang: go
backend: grepapp
result_count: 2
---
harness/harness  registry/app/remote/clients/registry/client.go  (line 207)
  req, err := http.NewRequestWithContext(ctx, http.MethodGet, buildPingURL(c.url), nil)
  https://github.com/harness/harness/blob/main/...

Regex support is per-backend: grepapp and sourcegraph accept it, github rejects it with a pointer to the other two.

--repo means the same repository on every backend. grep.app and Sourcegraph filter more loosely (name substrings, an unanchored pattern), so ketch keeps only exact matches: --repo golang/go never returns golang/gofrontend. grepapp searches the query as literal code, so a repo: or lang: typed into the query is matched as text; ketch warns and points at the flags. A missing repository is never a silent empty result: sourcegraph and github exit 3 and name the other backends, and grepapp, which indexes a subset of public repositories, warns when a repository search comes back empty.

$ ketch docs "routing" --library /vercel/next.js
$ ketch docs --resolve "next.js"     # name → Context7 IDs

docs is a two-step: resolve the name, vet the matches, then fetch by ID. Resolve never returns empty. A bad name still returns confident fuzzy matches, so check the name, not just the trust score.

$ ketch scrape <url>
$ ketch scrape <url1> <url2> <url3>      # concurrent batch
$ ketch scrape urls.txt                 # one URL per line
$ ketch scrape '["url1","url2"]'        # JSON array
$ echo "url1\nurl2" | ketch scrape      # stdin

After configured URL rewrites, scrape and crawl fetch only absolute http:// and https:// URLs. Other final schemes, including file://, are rejected. Private or internal HTTP(S) addresses are still fetched when reachable from the machine running ketch.

Content selection reads the page's own structure: the landmark it declares (main, article), a document assembled from uniform sections, or the smallest element holding its prose. Site furniture goes by what it is — navigation, hidden and collapsed controls, link rails, tables of contents — and readability is the fallback for a page that declares no structure. The extract_mode config key sets what pruning may drop: clean (the default) also drops blocks by name and phrase — related-post rails, comment threads, share bars, "was this helpful?" boxes — for the leanest markdown; complete keeps everything the structure does not condemn, trading a little precision for the last fraction of recall.

$ curl -L https://example.com | ketch extract
$ cat page.html | ketch extract --select article --max-chars 4000

No fetch, no cache, no browser — just content selection and the markdown pipeline, in the configured extract_mode. Useful when you already have the bytes.

$ ketch crawl https://docs.example.com --depth 2
$ ketch crawl https://docs.example.com --sitemap
$ ketch crawl https://docs.example.com --background
$ ketch crawl status [id]
$ ketch crawl stop <id>

A foreground crawl interrupted by SIGINT exits 0 with partial results, by design.

$ ketch search "guacamole ldap authentication" --scrape --tag remote-access
$ ketch tag add remote-access https://example.com/read-later   # no network
$ ketch tag show remote-access               # newest 50, with totals
$ ketch tag show remote-access --limit 0     # every entry
$ ketch tag list                             # every tag, with entry counts
$ ketch tag remove remote-access <url>...    # no URLs: drop the whole tag

A bookmark keeps the source URL, a bounded title and description, and the time it was tagged; full bodies stay in the page cache. With --json, show reports entries and cached totals for the whole tag, shown for the returned pages, and cache_status — unavailable when the page cache cannot be read, in which case cached: false is unverified rather than wrong. tag add takes absolute http(s) URLs and fetches nothing; anything else is bad input, exit 2. tag remove with URLs names the ones that were not under the tag as missing; without URLs it drops the tag and says how many entries went with it. Bookmarks survive cache expiry and cache clear, and nothing expires them.

The index is its own tags.db beside the config file — KETCH_TAGS_PATH overrides the filename — opened only for short operations, so an idle MCP server or a background crawl never blocks it.

$ ketch browser status
$ ketch browser install    # download Chromium to ketch's cache dir

Scraping is fast path first: plain HTTP by default, with the browser used only when a JS shell is detected. --force-browser overrides the detection.

$ ketch config                  # effective config + active backends, as JSON
$ ketch config init
$ ketch config set backend searxng
$ ketch config path

Plain ketch config is the discovery call: one invocation returns everything an agent needs to know about what's active, including *_key_set presence booleans that never reveal the key itself.

$ ketch cache
$ ketch cache clear

bbolt-backed, 72-hour default TTL. Repeat scrapes and crawls read from cache when their required data is present. Crawl entries also store every resolved outgoing HTTP(S) link, including external URLs and query strings, beside the page for the same TTL. This link metadata does not require or store full HTML, and is not sent elsewhere. A deeper crawl fetches a still-fresh page when its link metadata is unknown, and a later ordinary scrape refresh can make the metadata unknown again. If that backfill fetch fails, crawl returns the cached page and reports that link discovery is incomplete: on the CLI the diagnostic goes to stderr while JSON page output stays on stdout; MCP returns the page in pages and the diagnostic in errors; background status counts the returned page and error, without retaining the individual diagnostic text. The file is opened only for each read or write, so the CLI, background crawls and any number of MCP servers share it. Expired pages are swept automatically. ketch cache also reports the tag index — path, tag and entry counts. cache clear deletes the page cache, returning its space, and leaves bookmarks alone.

$ ketch doctor

Concurrent read-only probes against every backend, plus the browser, the cache, and an informational tags row for the bookmark index that never affects the exit code. Each comes back ok, no_key, unreachable, misconfigured, or skipped. Exits 0 when healthy and 5 when a configured surface is broken — so it works in CI.

$ ketch mcp serve

The five surfaces plus tag as MCP tools, on the same config and backends as the CLI. Every research tool accepts a tag option, and tag itself is the one tool that writes locally and never touches the network. The mcp_tools config key narrows the published set. config, cache, and doctor are deliberately not tools — they're operator actions, not research surfaces.

$ ketch version
ketch v0.17.0
  commit: 8f41359
  built:  2026-09-16T22:51:07Z
  go:     go1.25.7 linux/amd64

parallel, exa, keenable, youcom, firecrawl, and ddg answer with no setup. Keys for exa, keenable, youcom, and firecrawl are optional and lift the hosted caps. ddg rate-limits readily under fan-out.

$ ketch config set brave_api_key <key>
$ ketch config set tavily_api_key <key>
$ ketch config set tinyfish_api_key <key>
$ ketch config set serpbase_api_key <key>
$ ketch config set serply_api_key <key>

Providers accept multiple keys for rotation — brave_api_keys takes a list, and one is picked at random per request to spread rate limits.

$ ketch config set searxng_url http://my-searxng:8080
$ ketch config set degoog_url http://my-degoog:8090

An instance behind an auth proxy, such as Cloudflare Access, takes headers keyed by its origin. Every backend with a configurable instance URL — SearXNG, Firecrawl (firecrawl_url), Degoog, and Sourcegraph (sourcegraph_url) — sends them, on searches and on ketch doctor probes:

$ ketch config set http_headers '{"https://searx.example.com":{"CF-Access-Client-Id":"<id>","CF-Access-Client-Secret":"<secret>"}}'

Headers go only to requests whose scheme, host, and port match the key, so a redirect to another origin, or a --searxng-url pointing elsewhere, never carries them. A header the provider sets itself, such as Firecrawl's Authorization, wins. ketch config lists the header names, never the values, and neither ketch doctor nor an error message ever prints one — an error that quotes an instance's response redacts them. Setting http_headers replaces the whole map; '{}' clears it.

Grep and Sourcegraph need nothing. GitHub uses gh auth login, $GITHUB_TOKEN, or ketch config set github_token <tok>. Context7 takes a free key via ketch config set context7_api_key <key>.

• Per-provider key vars accept a comma-separated list and replace the provider's whole key pool — there are no plural *_API_KEYS vars.
• KETCH_HTTP_HEADERS takes the same JSON object as http_headers and replaces it whole.
• KETCH_GITHUB_TOKEN beats the config file, which beats an ambient $GITHUB_TOKEN.
• url_rewrites and spa_markers are file-only — their JSON and regex values don't survive env quoting.
• ketch config reports an env_overrides section, so you can always see which values came from the environment.
• Invalid values fail loudly, naming the offending variable. Secret KETCH_* vars are stripped from spawned subprocesses.

$ ketch cache          # stats
$ ketch cache clear
$ ketch scrape <url> --no-cache

The page cache opens bbolt only for individual operations, so an idle MCP server holds no cache lock and can share the cache with CLI and background crawls. Bookmarks are separate: tags.db is also opened only for short index operations.

$ ketch config set browser chrome       # from PATH
$ ketch config set browser /usr/bin/google-chrome-stable
$ ketch browser install                 # download Chromium
$ ketch browser status

• url_rewrites — regex rewrites applied before fetch
• spa_markers — extra tokens for JS-shell detection
• cache_ttl — cache lifetime
• user_agent — User-Agent override for HTTP and browser fetches
• extract_mode — clean (default) or complete; a non-default mode scopes cached pages, so a page cached under one mode is never reused under the other
• mcp_tools — allowlist of MCP tools to publish; unset publishes all six
• external_pdf_to_md_converter_command — external PDF-to-Markdown converter; must contain exactly one {input} placeholder. Once set it is authoritative, with no silent fallback
• http_headers — headers for self-hosted SearXNG, Firecrawl, Degoog, and Sourcegraph instances, keyed by origin (see Self-hosted under Backends); values are secrets