Technology · Protocol

TLS and mTLS

TLS with hot certificate rotation on every public surface; mutual TLS between my hosts for LAN GPU provisioning.

Inference Fabric · FAFOHive · Agent SwarmMutual TLS host controlCertificates rotated under traffic
A LAN workstation joins the fleet
SSH bootstrapmutual TLS from then on
GPU capacity, controllable
  • 1.2/1.3the only TLS versions the public fronts accept
  • 0certificate files on disk for the embedding-service client
  • onceSSH login per host, then never again

SSH once, then mutual TLS for a GPU fleet

Inference Fabric and FAFO Core and Function Forge needed to drive Docker on spare LAN workstations, and I did not want a shell login to be the control channel for a GPU fleet.

One call bootstraps the machine over SSH, issues it a certificate from my own certificate authority, reconfigures its Docker daemon to accept only mutually authenticated TLS, checks the handshake and rolls back if the check fails. From then on every container start, log read and teardown on that host goes over mutual TLS. A LAN machine becomes controllable capacity without Kubernetes and without a shared password.

SSH bootstrapcertificate from my CAdaemon to mutual TLShandshake checkevery start, log read and teardown over mutual TLS
If the handshake check fails, the host rolls back.

Rotating certificates without disconnecting anyone

FAFOHive is the endpoint every agent seat connects to, so restarting it to change a certificate would disconnect all of them. I made rotation an operation under traffic: the listener holds its certificate behind an atomic swap and replaces it on a reload signal. Its calls to the shared embedding service use client certificates held in memory.

Certificate
oldnew
Seat
Seat
Seat
▲ reload signal, atomic swap

One trust anchor per transport

Agent Swarm runs its message bus and database connections over TLS only. I made the Redis client accept my internal CA as its sole trust anchor and reject a plaintext URL when configuration loads, so a mistake fails at start-up. Database access uses certificate-based mutual TLS through a connector that accepts the connection strings the standard PostgreSQL tools use.

Helix uses mutual TLS for its control-center API and for the adapter that reads grounding documents, each with its own CA, so a service's identity comes from its certificate and the copied-token problem goes away. I kept every anchor separate so a certificate change on one transport cannot break another.

Anchors above, transports below
Message bus
internal CA
Database
its certificates
Control-center API
its own CA
Grounding adapter
its own CA

Hardened public fronts

Auth and access plane fronts hosted services with NGINX, HSTS, security headers and per-endpoint rate limits, with an ACME responder for certificates. The voice endpoint sits behind an access policy that answers authenticated callers and refuses anonymous ones, and the Capabilities model discovery service serves its HTTP face over rustls against the estate CA with an origin lock.

The TLS libraries are standard; my work is the certificate authorities, the trust layout and the rotation.

I treat transport security as something the systems operate themselves: they issue, rotate and check certificates without a person in the loop.

Client
modern TLS only
HSTS and headers
Per-endpoint rate limit
Service
An older protocol
Client
✕Refused