Running a Team of AI Agents on One Self-Hosted Server: 25 Design Decisions

Hive is paused; this post describes how it is built.

Hive is a multi-agent platform I built to run on a single self-hosted node, handling my own research, email triage and scheduled workflows, with Docker sandboxing, zero-trust networking, encrypted storage, and a cost-governance layer that holds spend under a hard monthly cap.

This post covers the architectural decisions behind running it as a long-lived system rather than a demo, and why I wrote 25 formal Architecture Decision Records to keep it that way.

The Node

The platform is designed to run around the clock on a single commodity self-hosted node on a headless Linux server. The OS is provisioned entirely through cloud-init, so the setup is version-controlled and reproducible from a cold drive, and a UPS with monitoring handles graceful shutdown on battery low.

Agent Orchestration: Depth-2 Nesting

The system uses a modular domain team architecture with three layers:

Layer 0: Orchestrator (main). Routes all Discord messages, spawns domain teams, and handles cross-domain synthesis. Runs Gemini 3 Flash for this dispatch and routing work; the reasoning-heavy load runs on Pro-tier models in the domain leads (see ADR-024 for the per-agent model tiering). This is the only agent that can run commands on the host (gated behind approval).

Layer 1: Domain leads. Specialized agents, starting with the research lead and added incrementally. Each lead owns a Discord channel and can spawn workers. The research-lead was the first pilot, exercising most subsystems: Gemini API, Google Search grounding, worker spawning, and file-based artifact patterns.

Layer 2: Workers. Ephemeral sub-agents spawned on-demand by team leads. They run the cheapest model (Gemini 3 Flash), have minimal tool access (no browser, no cron, no session management), and are archived after task completion.

This depth-2 pattern (orchestrator → leads → workers) allows new domains to be added without architectural changes. The expansion protocol is five steps: add agent, set permissions, write bootstrap docs, update orchestrator's allowed agent list, bind a Discord channel.

flowchart TD
  accTitle: Hive depth-2 agent orchestration on a self-hosted node
  accDescr: Discord messages reach the orchestrator (main, Gemini 3 Flash), which spawns domain leads for research and other domains, each Docker-sandboxed with no network access. Leads run Pro-tier models and spawn ephemeral workers on Gemini 3 Flash. The orchestrator uses a local QMD hybrid memory index and routes all model calls through a LiteLLM proxy that handles cross-provider failover and a hard monthly budget cap before any HTTPS call leaves the host. The node has zero public ports and is reachable only over a Tailscale mesh.
  discord["Discord (per-agent channels)"] --> main["Orchestrator: main (Gemini 3 Flash)"]
  subgraph host["Self-hosted node: zero public ports, Tailscale mesh only"]
    main --> leads["Domain leads: research and others (Docker sandbox, network none)"]
    leads --> workers["Ephemeral workers (Gemini 3 Flash)"]
    main --> qmd[("QMD memory: BM25 + vector + MMR, local embeddings")]
    main --> litellm["LiteLLM proxy: failover + hard monthly cap"]
    leads --> litellm
    workers --> litellm
  end
  litellm -->|"HTTPS outbound from host"| providers["Gemini / OpenAI / Anthropic"]

Layered Security

Agents that can execute code and browse the web are a standing attack surface, so the model is defense in depth, layered from the network to the supply chain, plus prompt-injection awareness:

Network Invisibility

The gateway binds to loopback only. The iptables INPUT policy is DROP on everything except loopback and the tailscale0 interface, so external nmap scans show zero visible ports. All access flows through a Tailscale mesh: encrypted WireGuard tunnels with MagicDNS hostname routing. Gmail is polled by a host cron job (ADR-021), which retired the system's last public endpoint, a Tailscale Funnel webhook.

Secrets Management

No plaintext API keys on persistent disk. All credentials live in a 1Password vault and are resolved at startup into a RAM-only tmpfs env file (mode 600) that the systemd unit loads; the tmpfs is gone on reboot. Credential fields use structured SecretRef objects rather than inline strings, and the gateway also supports an exec provider that runs 1Password CLI commands to resolve individual secrets at startup. That resolution is eager: if the 1Password session is unavailable, the gateway fails fast rather than degrading silently.

Access Control

Every channel uses DM pairing (a one-time code the owner approves), sessions are isolated per person per channel (session.dmScope: "per-channel-peer"), and group channels are mention-gated. Tool access runs through a layered, deny-wins policy pipeline: a deny at any level cannot be overridden below it, so each agent only ever holds the tools explicitly left to it. The effective policy for any agent is inspectable with openclaw sandbox explain.

Prompt Injection Awareness

Untrusted content (email bodies, web pages, tool and script output) is treated as data, not instructions. It is processed by sandboxed agents whose tool policies deny the cross-agent messaging tools (sessions_send, sessions_spawn), so a compromised agent cannot pivot to others. The main orchestrator carries no web tools at all; it delegates web work to a sandboxed research agent (ADR-019). The design is explicit that prompt-injection risk is reduced, not eliminated.

Execution Isolation

Every non-main agent runs in a Docker container with --cap-drop=ALL, --security-opt=no-new-privileges, and --network=none. There is no outbound network from inside the sandbox: web tools (browser, search) execute on the gateway host, not in the container, so a tool appearing in an agent's allowed list never means its container has network access. Workers inherit their lead's container and its isolation.

Infrastructure

LUKS full-disk encryption, with TPM2 auto-unseal via Clevis bound to PCR 7 for zero-touch boots, keeps the data unreadable if the NVMe is physically removed (a 1Password-stored passphrase is the permanent fallback). A dedicated, disabled-password hive service user runs the workload; the admin account carries minimal group membership; config files are mode 600 and the OpenClaw directory 700; unattended security upgrades and sensitive-value log redaction are on by default.

Supply Chain

Only explicitly trusted plugins load (plugins.allow allowlist). ClawHub skills are version-pinned on install, never @latest, and the gateway itself is pinned to a specific version at least a few days old, with the built-in auto-updater disabled in favor of a Hive-managed daily update that carries a rollback safety net. openclaw security audit --deep runs on a regular cadence.

Cost Governance with LiteLLM

With multiple agents running around the clock, token spend compounds fast. LiteLLM sits between the gateway and the model providers, splitting responsibility two ways:

• OpenClaw (gateway) handles agent-level model selection and per-model timeouts
• LiteLLM (proxy) handles cross-provider failover chains, semantic caching, and hard budget caps

The failover chain keeps the system answering through a provider outage. If Gemini returns a 429 rate-limit error, LiteLLM cascades to the next provider:

gemini-3.1-pro → gemini-2.5-pro → claude-sonnet-4-6
gemini-3-flash → gemini-2.5-flash → claude-sonnet-4-6

These chains reflect the original deployment. ADR-024 later put an OpenAI step (GPT-4.1 or GPT-4.1-mini) between the Google and Anthropic fallbacks, and moved the terminal Anthropic fallback to Claude Haiku 4.5 for every agent except the research lead, which keeps Sonnet 4.6 as its Anthropic fallback; Sonnet itself fails over to Haiku.

Budget enforcement uses LiteLLM virtual keys with monthly caps. At 80% spend, the orchestrator's observability cron job fires an alert. At 100%, the proxy refuses further calls, so a runaway loop stops at the ceiling. Per-agent cost tracking is visible in the LiteLLM dashboard.

Once the free Google Cloud credits that covered model usage ran out, per-agent model tiering took over holding spend under the hard monthly ceiling (see ADR-024 below).

Hybrid Memory: BM25 + Vector Search

Each agent maintains its own memory index using QMD, a hybrid search backend that combines BM25 text matching (weight 0.3) with vector semantic search (weight 0.7). Results are re-ranked using Maximal Marginal Relevance to eliminate near-duplicates, and a 30-day half-life weights recent context more heavily.

The embedding model runs locally as a ~600MB GGUF file via node-llama-cpp. Zero API cost for memory operations. The index updates every 5 minutes with debouncing.

Each agent's memory has a clear hierarchy:

• MEMORY.md: durable lessons learned (never temporally decayed)
• memory/YYYY-MM-DD.md: daily pre-compaction flush (ephemeral context)
• memory/reviews/YYYY-WXX.md: weekly self-assessment summaries
• memory/worker-results/: file-based worker output artifacts

Early testing showed Gemini 3.1 Pro dropping earlier worker context when later workers finished, so workers write results to files and the team lead reads all of them before synthesizing.

Why 25 ADRs?

Architecture Decision Records are what make this system maintainable rather than a pile of undocumented choices. Each ADR records the decision, the alternatives considered, the trade-offs accepted, and the verification criteria that prove it works.

Some highlights:

ADR-002 (QMD over Qdrant): Self-hosted hybrid search with zero API cost vs. a managed vector database. QMD's BM25+vector combination with MMR re-ranking and temporal decay met all requirements without adding an external dependency.

ADR-012 (Standard Docker, not rootless): For a single-user, single-purpose machine, I accepted standard Docker over rootless to keep setup simple, and made the container itself the boundary: --cap-drop=ALL, --security-opt=no-new-privileges and --network=none, with each agent's workspace on its own bind mount.

ADR-015 (API Keys over Subscription): Multi-provider pay-per-token (Gemini + Anthropic fallback) is cheaper than a flat-rate Claude Pro subscription for this usage pattern. LiteLLM makes provider switching transparent.

ADR-016 (Adaptive Self-Improvement): Five integrated patterns, all using native primitives, no custom code. Reflection prompting (agents write lessons when corrected), suggestion surfacing (orchestrator scans for tool gaps), weekly self-assessment (cron-triggered reviews), config self-modification with trust tiers (autonomous for low-impact changes, ask-first for structural ones), and cross-domain knowledge sharing via sessions_send.

Automation

Four recurring cron jobs kept it maintained while it ran:

The health check flow is intentionally decoupled: a host-level shell script (hive-monitor.sh) runs every 30 minutes via system cron, writing structured metrics to a file. The AI agent reads that file every 6 hours via the gateway's read tool. The agent never gets host-level access for monitoring; it reads a pre-generated report.

Running It Day to Day

The unglamorous parts were most of the work: restart checks, secret handling, cost alerts, log retention and runbooks. Hive had:

• Automated startup verification via a Makefile target that checks Tailscale, LUKS, firewall, disk, UPS, Docker, LiteLLM, gateway, agents, and secrets in sequence
• IaC provisioning via cloud-init; the entire OS configuration is version-controlled and reproducible
• Boot runbook in docs/BOOT.md covering the 7-step post-reboot verification sequence
• 130+ completed tasks across phases 0 through 7 with explicit verification gates, with later phases built the same way

What's Changed Since (May 2026 Update)

Two of the ADRs added since this post first went up:

ADR-024: Cost Optimization Post-GCP Credits

The free Google Cloud credits that covered Gemini usage ran out in early April. Every Gemini token then billed directly, and the unoptimized projection without credits ran well over the cap. Part of the problem was the gauge: one family of models was still priced at zero from the credit period, so budget tracking had been undercounting spend. I fixed the tracking first, then matched models to the job instead of cutting across the board. Goal: best value for spend, not cheapest possible. The LiteLLM hard monthly cap stayed, and per-agent model tiering brought steady-state spend back under it.

Per-agent model assignments:

The orchestrator does dispatch and tool calling, which is classification work, so Flash carries no quality risk there. Leads whose output quality depends on the model keep Pro. The Anthropic fallback shifted from Sonnet to Haiku 4.5 everywhere except the research lead, which kept Sonnet as its Anthropic fallback (Sonnet itself fails over to Haiku), because factual accuracy on research documents still matters when infrastructure is degraded. Two infrastructure fixes came along for the ride: the cost metadata that had been undercounting Gemini 2.5 spend was corrected, and Vertex AI routing was reordered to prefer the Gemini Dev API (which has a free tier) over Vertex (which doesn't).

ADR-025: Active Memory Plugin Adoption

OpenClaw 2026.4.12 added an Active Memory plugin: an opt-in memory sub-agent that runs before the main reply and pulls relevant memory into context automatically. By then the work repository's memory held roughly 20 files and was projected to reach 500 to 1,000 within twelve months. The summary-based context-loading pattern degraded past about 50 files and would become untenable past a few hundred. Active Memory ran retrieval over the existing QMD memory before each reply. It was enabled for main and the domain leads, and skipped on ops because email triage and health checks are mostly stateless.

The trade-off is DM latency from running a sub-agent call before every primary reply. The estimate was 1 to 2 seconds added to each DM reply, with a 15-second timeout as the ceiling, so latency was watched and the setting would be revisited if replies felt slow.

Tech Stack

• Hardware: single commodity self-hosted node, cloud-init provisioned
• OS: headless Linux server, cloud-init provisioned
• Agent Framework: OpenClaw (gateway + agents + cron + memory + sandboxing)
• Models (post-ADR-024 tiering): gemini-3-flash-preview for orchestrator and ops, gemini-3.1-pro-preview-customtools for the domain leads, Haiku 4.5 as Anthropic fallback (Sonnet 4.6 kept only as the research lead's Anthropic fallback, itself failing over to Haiku)
• Memory plugin: OpenClaw Active Memory (retrieval over QMD memory before each reply)
• Proxy: LiteLLM + Redis + PostgreSQL (Docker Compose)
• Networking: Tailscale (WireGuard mesh, MagicDNS, zero public ports)
• Encryption: LUKS + Clevis-TPM2 (PCR 7 binding)
• Secrets: 1Password CLI → tmpfs injection
• Memory: QMD (BM25 + vector, local GGUF embeddings, MMR re-ranking)
• Monitoring: UPS monitoring and a custom host health script
• Interface: Discord (channel bindings per agent)