oxid/ docs

// multi-node

One control plane, N Docker endpoints

When one server runs out of room, Oxid adds another — and adds nothing else. The daemon keeps the database, the git cache, the secrets and the audit trail; a node is a Docker API plus an address. There is no agent to install, no cluster to join, and no new dependency: the same bollard client that talks to the local socket talks to a remote one. 

                       oxid CLI · webhooks · dashboard
                                    │
   ┌────────────────── control plane node ──────────────────┐
   │  Traefik ──(http provider)──> oxidd /api/v1/traefik/config
   │     │                            │                     │
   │     └──> per-branch proxy (stable public_port)          │
   │              audit.sqlite · git-cache · secret.key      │
   └───────────────────────┬─────────────────────────────────┘
                           │  TCP to node address:host_port
        ┌──────────────────┴───────┐   ┌────────────────────┐
        │ node "eu-1"              │   │ node "eu-2"        │
        │ dockerd tcp/2376 (mTLS)  │   │ dockerd            │
        └──────────────────────────┘   └────────────────────┘

Builds run on the node — the build context is streamed to whichever endpoint the deploy is aimed at — while the git checkout never leaves the control plane. That is the main reason a remote Docker endpoint beats writing an agent.

Registering a node

Expose Docker over mTLS on the node (Docker's own protect-access guide generates the material), copy ca.pem, cert.pem and key.pem to the control plane's disk, then: 

oxid node add eu-1 tcp://10.0.0.4:2376 \
  --address 10.0.0.4 \
  --tls-ca   /etc/oxid/nodes/eu-1/ca.pem \
  --tls-cert /etc/oxid/nodes/eu-1/cert.pem \
  --tls-key  /etc/oxid/nodes/eu-1/key.pem

oxid node ls
ID    NAME     ENDPOINT                   ADDRESS      STATE    MEMORY    ENVS
1     local    local                      -            active   31.2 GiB  7
2     eu-1     tcp://10.0.0.4:2376        10.0.0.4     active   15.6 GiB  0

The connection is made and probed before anything is written, so a bad endpoint or a missing certificate fails at registration rather than hours later on somebody else's push.

--address is not the endpoint

The endpoint is where the Docker API lives. The address is where the control plane's proxy dials the ports that node publishes. They are frequently different interfaces, and omitting the address makes Oxid dial loopback — its own machine — so the branch deploys successfully and is unreachable. The post-deploy readiness probe uses the address, which turns a mistyped one into an honest deploy failure instead of a green report.

A remote endpoint with no TLS material is refused. A Docker socket over plain TCP is root on that machine for anyone who can route to it, and mTLS is the only thing bounding who that is. OXID_ALLOW_INSECURE_NODES=1 overrides it, following the precedent OXID_ALLOW_OPEN_API set — for a network you control end to end, and nowhere else.

Routing across the fleet

Traefik's Docker provider only ever sees the socket it is reading, so it is structurally incapable of learning about a container on another machine. Point it at the daemon as well: 

# docker-compose.yml, traefik service — add, don't swap
command:
  - --providers.http.endpoint=http://oxid-daemon:8080/api/v1/traefik/config
  - --providers.http.headers.Authorization=Bearer ${OXID_API_TOKEN}
  - --providers.http.pollInterval=5s

Both providers run together. The Docker one keeps routing everything it routes today; the HTTP one supplies the two classes of environment labels cannot describe — those on another node, and those whose container is stopped.

That second one is a bonus worth naming. A sleeping branch used to have no router at all, which is why the fragile lowest-priority oxid-wake-catchall on the daemon's own container had to exist. A router built from a database row exists whether the container runs or not, so the request reaches the branch's own router, the proxy has no target, Traefik answers 502, and wake-on-request fires. Keep the catch-all anyway — it is still the wake path in direct-publish mode — but oxid infra status stops flagging its absence once the HTTP provider is live.

The endpoint is authenticated and ETagged: the document names every branch on this daemon and how to reach it, and Traefik polls it forever.

Placement, draining and failure

BehaviourWhat happens
Placement Affinity first, then most free memory. A redeploy stays on the node it is already on: images are not distributed — each node builds its own — so a branch that moves rebuilds from scratch.
Admission Per node. Memory promised on eu-2 says nothing about whether a deploy fits on eu-1. A request no node could ever satisfy is refused immediately; one that fits nowhere right now is queued.
oxid node drain eu-1 Stops new environments landing there. Touches nothing already running.
oxid node drain eu-1 --evacuate Also moves every live branch off, one redeploy each through the ordinary zero-downtime path (build, wait for ready, cut over, then remove the old container). Each branch is rebuilt at the commit it is running, never at its current head — draining is an infrastructure operation, not a licence to ship whatever was pushed since. A branch that will not build stays where it is and is named.
A node stops answering Marked down by the health probe; receives no new work. Its environments are left exactly as they are — nothing moved, marked destroyed or rebuilt. It rejoins on its own when it answers again, with no restart.
oxid node rm eu-1 Refused while any environment still points at it, destroyed ones included: the audit trail hangs off environment rows, so freeing the node would delete that history as a side effect.

A network partition is indistinguishable from a dead machine, seen from the control plane. That is why nothing is ever evicted on one: acting on a partition is how two live copies of a branch end up fighting over a single URL.

The two costs, stated plainly

The control plane is in the data path

Traffic for a branch on a remote node goes Traefik → the daemon's per-branch proxy → the node. Restarting oxidd therefore cuts in-flight connections to remote environments and stalls new ones until the accept loops rebind at startup. On a single node nothing changes: Traefik reaches the container directly.

If control-plane bandwidth or its restart window becomes the limit, that is the point at which a per-node agent would earn its complexity. The design leaves room for one — an agent implementing the same container port slots in beside the Docker adapter with no change to the control plane — and deliberately does not ship it before it is needed.

The control plane is a single point of failure

Running environments keep serving: containers carry unless-stopped, and Traefik is its own container. What stops is deploying, waking, the GC, the API and — per the point above — cross-node traffic. Recovery is restore-and-restart.

Backups snapshot the SQLite file, which brings back the node rows and not the certificate files they name. Those stay your responsibility, alongside secret.key.

Explicitly out of scope

  • Image distribution. Each node builds its own copy. A branch that changes node rebuilds from scratch: slower, but correct, and with no registry to operate.
  • A highly-available control plane. SQLite here is a product decision with measurements behind it, not an implementation detail. Swapping it for an external database trades that away for a dependency you would have to make highly available anyway.
  • Live migration of a running environment. Moving a branch is a redeploy and a cutover, which the zero-downtime path already does. Never container checkpointing.
  • Autoscaling or provisioning nodes. An operator registers them. Oxid does not create machines.
  • Per-node secret keys. Secrets stay encrypted on the control plane and are injected over the mTLS connection. A node never stores secret.key — though environment variables remain readable with docker inspect on the node, exactly as they are today on one machine.