14 KiB
WebAccess.MCP Architecture
Purpose
WebAccess.MCP is a .NET 10 web-access service for LM Studio and compatible Model Context Protocol (MCP) clients. The target production host is an ASP.NET Core application exposing MCP through Streamable HTTP. Browser interaction remains Playwright-backed, while web search is delegated over HTTP to a dedicated SearXNG service.
The repository contains three sibling projects:
WebAccess.MCP, the production server.WebAccess.MCP.TestRig, an independent integration client with no project reference to production.WebAccess.MCP.Tests, focused unit tests for production infrastructure.
Migration State
The source is being migrated in small, independently validated steps. Production now uses ASP.NET Core Streamable HTTP MCP, the TestRig connects to a separately started HTTP server, and the application-owned SearXNG lifecycle subsystem has been removed. Container, Compose, deployment, and LM Studio migration remain future work.
Future changes must move toward the target described below without combining unrelated transport, testing, lifecycle, container, deployment, or LM Studio changes into one step.
Target Runtime Topology
The deployed topology is:
LM Studio
|
| Streamable HTTP MCP
v
WebAccess.MCP container
|
+--> Playwright / headless Chromium
|
+--> HTTP --> dedicated SearXNG container
| |
| +--> Google
| +--> Brave
| +--> DuckDuckGo
|
+--> PostgreSQL --> durable page-cache persistence
Podman Compose owns the WebAccess.MCP, dedicated SearXNG, and PostgreSQL containers, private network, dependency and readiness orchestration, host port mappings, and service startup and shutdown. PostgreSQL belongs to the webaccess-mcp project and is unrelated to the separate postgres-dev stack.
WebAccess.MCP does not invoke Podman, Docker, or Compose; does not know where a container runtime is installed; and does not start, stop, or otherwise own SearXNG. From the application's perspective, SearXNG is only an HTTP service identified by configuration.
The existing ai-searxng deployment is unrelated and off limits. Its container, configuration, data, identity, and host port 18080 must not be reused or treated as a template or dependency.
MCP Transport and Endpoints
The target production host uses:
- ASP.NET Core on .NET 10.
ModelContextProtocol.AspNetCore2.2.0.- Streamable HTTP MCP.
- A stateless MCP HTTP transport initially.
- The MCP endpoint
/mcp.
The container listens on port 8080. Compose publishes that endpoint only on host loopback:
http://127.0.0.1:28081/mcp
LM Studio will ultimately connect to this URL instead of launching a native WebAccess.MCP command.
Stateless MCP transport does not make browser navigation stateless. Browser state is owned by application singletons independently of MCP transport sessions. A stateful transport should be introduced only if a later MCP requirement needs transport-level session state.
Neither the WebAccess.MCP endpoint nor SearXNG should bind to LAN interfaces by default.
Browser Runtime and Tools
Browser tools use Microsoft.Playwright 1.62.0 and headless Chromium. The existing browser design is preserved during the transport and container migration:
- One long-lived Playwright runtime.
- One long-lived Chromium browser.
- One persistent browser context and page, created lazily on the first browser operation.
- Serialized access to the persistent page so navigation and interaction state survive safely across tool calls.
Chromium itself currently starts eagerly as part of hosted-service startup. Converting Chromium startup to lazy initialization is explicitly deferred until the HTTP and container migration is stable.
The expected MCP tool surface is 16 tools:
health_checkopen_urlget_page_titleget_page_urlget_page_infoget_page_textfind_page_textget_linksget_interactive_elementsclicktypepress_keygo_backgo_forwardreloadweb_search
Tool names and contracts should remain stable throughout the transport migration.
Search Architecture
web_search does not use Playwright and does not directly scrape Google, Brave, DuckDuckGo, Bing, or SearXNG pages. Its application flow is:
SearchTools
-> IWebSearchService
-> SearXngWebSearchService
-> HTTP GET /search?format=json
-> dedicated SearXNG
-> normalized WebSearchResponse
The dedicated SearXNG instance performs searches using the intended selectors:
googlebraveduckduckgo
Bing is not required. The standard selected engines require no downstream search-engine API keys, and the separate braveapi engine is not used.
The search service retains query and result-limit validation, configured default engines, per-call selector overrides, selector normalization, JSON response handling, HTTP/HTTPS URL validation, result normalization, deduplication, timeout handling, and safe MCP error translation.
Each successful web_search response includes one short deterministic nextStepHint encouraging targeted research through find_page_text(url,query) before get_page_text. This is guidance only: web_search does not call either tool, and search providers, ranking, filtering, ordering, and result records are unaffected. The hint is intentionally small and isolated so the experiment is easy to revert. Adjacent-page behavior and loop protection remain separate concerns.
Dedicated SearXNG Service
The Compose service name is searxng, and WebAccess.MCP reaches it through the private Compose network at:
http://searxng:8080/
SearXNG listens on container port 8080. Compose publishes diagnostic and host access only on loopback:
http://127.0.0.1:28080/
The dedicated instance must enable JSON output and use independent configuration and data beneath the WebAccess.MCP deployment boundary. Its lifecycle and readiness are Compose responsibilities.
Application Configuration
Application configuration uses standard .NET configuration providers. Application-level SearXNG configuration contains only HTTP client concerns:
Search:SearXNG:BaseUrlSearch:SearXNG:TimeoutSearch:SearXNG:DefaultEngines
Compose supplies the production base URL as http://searxng:8080/.
Page-cache configuration uses:
ConnectionStrings:WebCachefor the internal PostgreSQL connection.WebCache:DefaultExpiration, which defaults to one hour and must be greater than zero.WebCache:ChunkSize, which defaults to 4000 characters and supports values from 1 through 50000.
PostgreSQL is the durable page-cache persistence layer. WebCacheService owns requested-URL validation, safe fragment-free URL identity, expiry decisions, and non-overlapping text chunk creation. Cache keys are lowercase SHA-256 hashes of normalized requested URLs, and one current snapshot is retained per unique cache key. PostgresWebCacheRepository owns factory-scoped EF Core persistence and transactional replacement of page metadata and ordered chunks.
Cached page snapshots use contiguous, 1-based chunks whose configurable logical size defaults to 4000 characters; optional raw HTML remains on the parent snapshot. URL-based page reads request individual chunks instead of repeatedly returning the beginning of a page. This persistence layer will also support later repeated-work suppression and search enrichment without duplicating normalized text on the parent row. Redis is not part of this design at this time.
PostgreSQL also provides deterministic lexical relevance search over cached chunk text. Each chunk has an automatically maintained tsvector generated with PostgreSQL's simple text-search configuration and indexed with GIN. Full-text search identifies candidate locations within one selected cached page; deterministic in-process scoring then weights the exact query more strongly than caller-supplied related terms and rewards multiple nearby matches. It does not use an LLM, embeddings, pgvector, or additional PostgreSQL extensions.
find_page_text(url,query,relatedTerms?) exposes that retrieval through MCP. url and query are required; optional related terms are trimmed, bounded, deduplicated hints used only for deterministic matching inside the already selected cached document. They never alter web_search, select another URL, invoke an LLM, or become cached page content. A fresh snapshot is searched entirely through PostgreSQL-backed data; a missing or expired snapshot is populated with one serialized browser navigation/extraction and then searched.
Storage chunks are not treated as semantic boundaries. After candidate discovery, the service reconstructs the complete bounded normalized snapshot from persisted StartOffset, Length, and Text, then returns one hard-bounded context window of approximately the configured 4000-character page size around the strongest match. That window may span multiple storage chunks without gaps, overlap, or duplicated boundary text and reports the corresponding 1-based get_page_text page or page range. If targeted retrieval is weak or finds nothing, the response does not claim the information is absent; it directs the caller to sequential get_page_text reading as the correctness fallback. Repeat/loop protection remains separate.
The preferred targeted-research flow is web_search -> find_page_text -> get_page_text only for selected returned pages when snippets are insufficient. Neither open_url nor get_page_text page 1 is required to seed the cache. Sequential page reading remains valid for broad summaries, comprehensive document reading, and explicit continuation when a selected page is incomplete. Automatic adjacent-page expansion is not implemented and remains a separate concern.
get_page_text returns one configured-size text page per call. Its public page parameter is 1-based and defaults to 1; the former public maxChars parameter is removed. URL-based reads return a fresh PostgreSQL snapshot when available. A miss or expired snapshot causes one serialized browser navigation and extraction, followed by transactional snapshot replacement, while the requested page is returned directly from the normalized text already in memory. Reads of the current interactive browser page bypass durable URL caching because click, type, and key operations may have changed its state.
Later pages contain only their new, non-overlapping text. PostgreSQL failures fail open to the existing browser path, and a failed cache write does not suppress an otherwise successful browser result. The existing 50000-character normalized extraction ceiling remains an internal safety limit; a final available page indicates when the stored snapshot was truncated. Repeat/loop protection remains a separate future step.
The removed application-owned lifecycle subsystem and its fields must not be reintroduced:
AutoStartStopOnShutdownStartupTimeoutProbeIntervalStartExecutableStartArgumentsStopExecutableStopArgumentsWorkingDirectory
No application configuration should contain a Podman executable path, Windows container-runtime path, or generic SearXNG launcher command.
Use Microsoft.Extensions.Logging for application diagnostics. Normal structured console and container logging is allowed for the HTTP host. Prefer ILogger over ad hoc Console.WriteLine diagnostics.
TestRig Direction
WebAccess.MCP.TestRig remains a separate sibling .NET 10 console project with no direct production project reference.
WebAccess.MCP.TestRig connects to a separately started production server through the real Streamable HTTP MCP endpoint. It does not own production as a child process. Deterministic browser loopback fixtures and fake-SearXNG validation remain useful; obsolete application lifecycle tests have been removed.
Integration validation should continue to verify the 15 expected tools, persistent browser navigation state, deterministic browser interactions, normalized search behavior, error handling, and an explicitly selected real SearXNG environment.
Container Strategy
The initial WebAccess.MCP container prioritizes compatibility over image-size optimization:
- Build and publish with .NET 10 on Ubuntu Noble.
- Use
mcr.microsoft.com/playwright/dotnet:v1.62.0-nobleas the compatible final-image baseline. - Keep the application package at Microsoft.Playwright 1.62.0.
- Use the matching image's Chromium binaries and Linux browser dependencies.
Image-size optimization and a thinner runtime image are later work. They must not be combined with the architecture reset.
Deployment Boundary
The complete deployed stack will eventually live beneath:
D:\ai\mcp\WebAccess.MCP
The conceptual runtime layout is:
D:\ai\mcp\WebAccess.MCP\
compose.yml
.env
webaccess\
searxng\
settings.yml
data\
searxng\
logs\
The existing native publish in that directory remains untouched until the container stack has been deployed and validated and its removal is explicitly approved.
Security and External-Service Behavior
Host port mappings use loopback by default:
- WebAccess.MCP:
127.0.0.1:28081to container port 8080. - SearXNG:
127.0.0.1:28080to container port 8080.
LAN exposure requires a separate explicit decision. Browser and search operations must not attempt to bypass CAPTCHAs, anti-bot protections, access controls, or provider restrictions. External-engine failures should be surfaced clearly without introducing application-owned process or container recovery behavior.