Robert Davis — CEO, CTO, and Principal Architect
This platform runs a real business — customers, payments, field crews, vehicles. I designed it, built every service, and operate it alone: CI/CD, observability, SLO tracking, backups, and on-call included. The application code is proprietary; the engineering is documented here.
Grouped the way the system actually fails: edge, applications, async workers, platform, and hardware in the field.
Green is the request path, blue is data and telemetry, dashed gray is observability, and the amber boundary is one tenant stack — everything inside it is replicated per tenant.
Every service ships the same way. Pull requests build without publishing; version tags cut releases.
The deploy step is one idempotent tool: it pulls service repos, pre-builds the Next.js apps, provisions per-environment infrastructure, regenerates the edge routes, and brings the stack up under Docker or Podman — rebuilding only the services whose commit SHA actually moved. Backend images are built once per SHA and shared across every stack; the Next.js frontends bake per-environment branding at build time, so those build per stack. The genericized form is published as compose-multienv-deploy.
One host, N fully isolated stacks. Each gets its own network, its own MySQL server, its own object storage, its own secrets, and its own domain routes — nothing in the data path is shared between environments. The single shared component is one Caddy edge that terminates TLS and routes by hostname; every stack attaches its own routes at deploy time.
The live platform.
Production-shaped — same tooling, same generated edge config. Configuration drift surfaces here first, not in front of customers.
A dedicated customer stack: own database, object storage, domain routing, and secrets. Standing one up is one command, not a re-architecture.
Provisions an environment end to end: network, database, object storage, generated secrets, routes. The schema is seeded from the API repo's checked-in SQL and tracked by a marker table, so re-running provisioning is safe.
Rebuilds only the services whose repo SHA moved; already-built SHAs deploy from cache. Services resolve each other by container DNS, so deploy order is irrelevant.
Re-runs the previously recorded image for one service, in one environment only.
Gzipped dump of both of the stack's databases with drop-and-create semantics. Newest 72 retained — roughly three days of hourly backups.
A customer claims a name and gets a full isolated stack at name.base-domain, on wildcard DNS, with certificates issued automatically on first deploy.
node · express · react spaEvery stack as a grid, one cell per service, with live status, the tenant list, and a job queue that streams deploy logs as they run.
jobs run one at a timeOwner and member roles, expiring invitations, and a per-tenant audit log.
rbac · audit trailSubscription plans with entitlements from Community to Enterprise, annual terms and add-ons, plus Ed25519-signed licenses enforced at the stack level for self-hosted installs.
ed25519 · signed licensesThe control plane never reimplements deploy logic. Every state change shells out to the same stackctl I run by hand, and status is read back from its registry plus one container-engine query. A second implementation of "what does deployed mean" is a guarantee that the dashboard and the CLI will eventually disagree — usually mid-incident, when the dashboard is the thing you're trusting.
Shared-nothing costs RAM. A full MySQL and object store per environment buys a blast radius that stops at one stack, and pays for it in memory and disk. That's the right trade while stack count is small; it is explicitly the thing to revisit before it isn't. For the same reason self-service provisioning is gated by default — an anonymous form that instantly consumes a stack's worth of resources is a denial-of-wallet vector, so signups queue for approval unless they're gated upstream.
The tooling is built in-house, sized to the platform, and runs as part of it.
A dedicated service discovers containers through the Docker/Podman API and probes each on a background loop. No manual registration, no stale check configs.
probes: http · mysql · tcpPurpose-built dashboard: container metrics, log aggregation, alerting, and SLO tracking against explicit targets, behind database-backed RBAC.
slo targets · error budgetsThe control-plane process drives hourly per-tenant database backups — one gzipped dump of a stack's business and monitoring schemas, written drop-and-create so a restore rebuilds them whole, and pruned to the newest 72. Restore plus redeploy is the entire disaster-recovery story, by design.
hourly · per-tenant · 72 keptCore datastores are verified before any service swap — started if stopped, deploy aborted if they won't come up. Deploying against a down datastore would bake dead endpoints into every rebuilt container.
mysql · object storage · abort-on-failVehicle firmware assumes the network is lying: every reading lands on SD first, then uploads as gzip batches with retry and backoff. Late data is fine; lost data is not.
store-and-forward · ndjson · lteA production case study in operating an LLM feature end-to-end: generate, constrain, publish, measure, feed back.
Vision captioning runs against a JSON schema (title / caption / hashtags), and the caption contract — contact placement, hashtag count and casing — is enforced in code, not trusted to the model.
structured outputs · schema-strictCaptions rotate through engagement “angles” and every caption records which angle produced it. Publishing stores platform post ids; an insights endpoint batch-fetches engagement and rolls it up per angle.
per-angle scoreboard · graph apiPrompts, style lists, and posting parameters live in a settings table — code-seeded defaults with owner overrides beside them. Tuning the voice never requires a deploy.
db-backed · no-deploy tuningThird-party feeds layer browser persistence over a server TTL cache; an explicit refresh is the only real upstream fetch, floored so even button-mashing can’t burn the quota. Token expiry flips accounts to needs-reconnect before posts start failing.
layered cache · refresh floor · expiry lookaheadReliability claims are cheap. Here are real incidents from this platform's history — reconstructed from the commit log — hashes included.
The monitoring dashboard gained a live topology view — container stats, sparklines, request traces — on top of its real-user-monitoring ingest. The monitoring database lives on the same MySQL server as production. Within hours the topology view was timing out, and every one of those timeouts represented load pressure on the database that also serves the business.
The first fix made it worse. Parallelizing all DB queries and trace fetches is the intuitive move when something is slow — and it multiplied the concurrent load on an already-stressed database. Reverted the same day. The durable fix went the opposite direction: batch the RUM inserts, reduce query limits, poll slower, fetch fewer traces, and put explicit timeouts and error containment on every query so the dashboard degrades instead of hammering.
The lesson: monitoring is production. The observer carries the same load budget as the observed — and parallelism is not a fix for overload, it is a multiplier on it.
An image-processing dependency ships its native binaries as optional packages carrying a node >= 20.9 engines constraint; the runtime image ran Node 18. npm skips engine-mismatched optional dependencies silently — the image built green, the container came up healthy, and the first require() at request time threw, sending every image the dashboard serves through the framework’s generic 500 page.
Diagnosis came from the outside in: sibling routes sharing every import except the image library answered clean JSON errors, isolating the failing module without a shell on the box. The fix was one line. The finding was not: a green build proves the dependency graph resolved, not that it can load. Engines mismatches on optional dependencies are a silent runtime landmine — the class of failure that only surfaces in production, on the first request.
Sized to the platform, enforced by structure rather than policy documents.
Every tenant stack gets its own network, database, object storage, and secrets. There is no shared state to leak across — isolation is the architecture, not a setting.
per-tenant network · db · objectsCredentials live in per-environment env files injected at deploy time — never in images, never in git. Each environment holds only its own.
per-env injection · nothing in imagesEvery stack fronts through its own edge proxy with TLS termination; certificates are provisioned and renewed automatically.
auto-provisioned certsDatabase-backed RBAC with role-to-permission mapping, session management, and a token blacklist for immediate revocation.
rbac · sessions · token revocationThe customer portal has no passwords to steal: sign-in is a 15-minute magic link over SMS. The request endpoint answers identically for any input and is rate-limited, so it can't be used to probe who is a customer. Sessions are stateless JWTs — rotating one secret revokes everything.
magic links · enumeration-proof · rate-limitedThe choices interviewers ask about, answered up front.
I operate multi-tenant Kubernetes at day-job scale — which is exactly why this platform doesn't use it. On a single host, Kubernetes buys autoscaling and rolling deploys this system doesn't need, at the price of a control plane to patch and a much larger failure surface to debug alone. The revisit trigger is explicit: a second host, or a genuine need for zero-downtime rollouts.
Hosted observability is priced per-container and per-GB of ingest — for a single-host platform that bill would rival the entire infrastructure budget. The requirements were narrow and Docker-native, and building purpose-fit kept everything on one pane, data on-host, and SLO tracking implemented the way I practice it professionally.
Field vehicles are the harshest environment in the system: LTE dead zones, uploads dying mid-flight, power cut at ignition-off. The SD card is the source of truth; an uploader drains it whenever connectivity allows. For telemetry, durability beats latency.
The AI publishing prompts and style rotations live in the database — code-seeded defaults, owner overrides beside them. Prompt tuning is the highest-frequency change in the subsystem; making it a deploy would freeze iteration. The angle-level engagement scoreboard makes tuning same-day actionable.
Staging shaped exactly like production catches configuration drift before customers do, and the same isolation model makes a dedicated customer instance a one-command operation instead of a re-architecture. Isolation is the product's oldest feature, not a bolt-on.