Go 1.26
context, goroutines, bounded channels, singleflight, typed domain errors, and strict dependency injection for deterministic tests.
Project 02 / Search & Read Infrastructure
SearchX.
A provider-aware WebSearch service and a policy-guarded WebFetch pipeline, built as two independently deployable Go applications with explicit failure, cache, and fallback behavior.
Why it exists.
Search reliability is more than sending an HTTP request and accepting a 200 response.
Providers differ in supported query syntax, continuation state, browser requirements, capacity, CAPTCHA behavior, and result shape. Content reading adds a separate risk surface: SSRF, redirects, JavaScript shells, extraction quality, and output conversion.
SearchX keeps those concerns behind two atomic services. WebSearch returns ranked links and opaque pagination state; WebFetch applies explicit URL policy and turns one public URL into a canonical document. They can evolve and deploy independently instead of becoming one coupled scraping process.
Implementation architecture.
The external gateway routes one contract to the appropriate service. Each service owns its policy, orchestration, adapters, and degradation path.
WebSearch flow.
A request is normalized once, then executed only by providers that can honor its declared semantics.
- 01Contract
Gin validates the request, timeout, region, routing order, filters, advanced query options, and optional opaque cursor.
- 02Search plan
A capability-aware compiler maps supported operators to Brave or DuckDuckGo and removes incompatible providers before execution.
- 03Cache guard
Fresh memory cache and
singleflightcollapse identical work; stale cache is an explicit degraded fallback when live search fails. - 04Ordered provider chain
Public requests execute provider adapters sequentially; each adapter leases one profile. The first valid result wins, while a retryable failure advances to the next provider.
- 05Provider adapters
Baidu, Bing, Brave, and DuckDuckGo adapters own transport, browser state, parser behavior, and typed failure classification.
- 06Stable output
URLs are canonicalized, domain rules are enforced, duplicates are removed, and AES-GCM protects provider continuation inside a request-bound cursor.
WebFetch flow.
Content reading is an explicit pipeline with replaceable seams and quality decisions at every stage.
- 01Safe target
The URL policy accepts only HTTP(S), rejects credentials and unsafe ports, resolves every address, blocks private and reserved ranges, and pins approved DNS answers.
- 02Site strategy
A longest-domain and path registry selects optional site-specific preparation while the generic strategy preserves default behavior.
- 03HTTP first
A bounded HTTP reader handles redirects and body limits before any browser is allocated.
- 04Extract & evaluate
MIME detection chooses HTML or plain-text extraction. A quality evaluator classifies usable content, JavaScript shells, short pages, login walls, and CAPTCHA pages.
- 05Browser fallback
Eligible failures and render-required pages use a slot-limited Chromium adapter, then pass through the same extraction and quality path. This branch rechecks the final URL but does not pin DNS or intercept subresource requests.
- 06Canonical output
Readability-backed content is cached independently of presentation, then converted to Markdown or text and truncated by Unicode code point.
Technical stack.
The stack is deliberately small: standard Go concurrency around focused adapters for browser automation, extraction, and delivery.
Gin + OpenAPI
Gin 1.12 HTTP adapters, strict YAML configuration, CORS, request-scoped timeouts, health/readiness routes, and versioned API schemas.
chromedp + goquery
Chrome DevTools Protocol automation, provider parsers, Readeck Readability, HTML-to-Markdown, MIME detection, and plain-text extraction.
SSRF policy + AES-GCM
DNS/IP rules and target pinning on the HTTP reader path; encrypted, expiring, request-bound cursor state for search pagination.
Docker + Kubernetes
Separate images and manifests for WebSearch and WebFetch, shared Runtime helpers, environment/YAML overlays, and API Gateway integration.
Test · race · vet · build
Unit and integration fixtures, provider parser suites, cursor and SSRF tests, Go benchmarks, race detection, vet, and reproducible curl, Python, and Go examples.
What is implemented.
- Ordered Baidu, Bing, Brave, and DuckDuckGo execution with typed failure handling, one-profile leases, capacity state, quarantine, and sequential provider fallback.
- Capability-aware advanced search, include/exclude domain enforcement, URL canonicalization, deduplication, and encrypted continuation cursors.
- Fresh/stale in-memory caching, duplicate-request collapse, bounded live concurrency, queue limits, and request-scoped diagnostics.
- Single-URL fetch with SSRF defenses on the HTTP path, bounded redirects and bodies, HTML/plain-text extraction, quality gates, and Chromium fallback.
- Markdown/text conversion, OpenAPI contracts, API Gateway examples, local demo, Docker images, Kubernetes manifests, and shared logging/timeout helpers.
Honest boundaries.
SearchX is retrieval infrastructure, not an answer engine.
- WebSearch returns provider-ranked links; it does not fetch result bodies, run an LLM, synthesize answers, or perform a semantic reranking pass.
- The capacity-aware auto router exists for internal calls, but public HTTP requests materialize a provider list and use strict sequential fallback; providers are not queried in parallel or aggregated.
- WebFetch reads one public URL. PDF, Office files, images, OCR, batches, authenticated pages, paywalls, and CAPTCHA solving are outside the current contract.
- The HTTP reader pins validated DNS targets and checks redirects. Chromium fallback currently rechecks only the final URL; it does not pin DNS or intercept subresource requests, so the two paths do not provide identical SSRF guarantees.
- Cache and browser profiles are Pod-local by design; WebSearch cursors require a shared secret across replicas, but the services do not depend on a shared database.