Robert Davis — Staff Site Reliability Engineer
This platform runs a real business — customers, payments, field crews, vehicles. I designed it, built every service, and operate it alone: CI/CD, observability, SLO tracking, backups, and on-call included. The application code is proprietary; the engineering is documented here.
Grouped the way the system actually fails: edge, applications, async workers, platform, and hardware in the field.
Green is the request path, blue is data and telemetry, dashed gray is observability, and the amber boundary is one tenant stack — everything inside it is replicated per tenant.
Every service ships the same way. Pull requests build without publishing; version tags cut releases.
The deploy step is one idempotent script: it pulls service repos, pre-builds the Next.js apps, generates the nginx config, and brings the stack up under Docker or Podman. Published (genericized) as compose-multienv-deploy.
One host, multiple fully isolated stacks — each environment gets its own env file, network, containers, and volumes. Nothing is shared.
The live platform.
Production-shaped: same compose file, same generated nginx config — drift surfaces here first.
A dedicated tenant stack: own database, object storage, domain routing, and secrets.
Multi-tenancy as a product. The per-customer isolation model is growing a control plane — tenant signup, an admin dashboard, and customer self-service — turning a single-business platform into multi-tenant SaaS, one isolated stack per tenant with hourly backups.
The tooling is built in-house, sized to the platform, and runs as part of it.
A dedicated service discovers containers through the Docker/Podman API and probes each on a background loop. No manual registration, no stale check configs.
probes: http · mysql · tcpPurpose-built dashboard: container metrics, log aggregation, alerting, and SLO tracking against explicit targets, behind database-backed RBAC.
slo targets · error budgetsEvery tenant stack takes hourly backups of its database and object storage. Restore plus redeploy is the whole disaster-recovery story — by design.
hourly · per-tenantVehicle firmware assumes the network is lying: every reading lands on SD first, then uploads as gzip batches with retry and backoff. Late data is fine; lost data is not.
store-and-forward · ndjson · lteReliability claims are cheap. Here is a real incident from this platform's history — reconstructed from the commit log — hashes included.
The monitoring dashboard gained a live topology view — container stats, sparklines, request traces — on top of its real-user-monitoring ingest. The monitoring database lives on the same MySQL server as production. Within hours the topology view was timing out, and every one of those timeouts represented load pressure on the database that also serves the business.
The first fix made it worse. Parallelizing all DB queries and trace fetches is the intuitive move when something is slow — and it multiplied the concurrent load on an already-stressed database. Reverted the same day. The durable fix went the opposite direction: batch the RUM inserts, reduce query limits, poll slower, fetch fewer traces, and put explicit timeouts and error containment on every query so the dashboard degrades instead of hammering.
The lesson: monitoring is production. The observer carries the same load budget as the observed — and parallelism is not a fix for overload, it is a multiplier on it.
Sized to the platform, enforced by structure rather than policy documents.
Every tenant stack gets its own network, database, object storage, and secrets. There is no shared state to leak across — isolation is the architecture, not a setting.
per-tenant network · db · objectsCredentials live in per-environment env files injected at deploy time — never in images, never in git. Each environment holds only its own.
per-env injection · nothing in imagesEvery stack fronts through its own edge proxy with TLS termination; certificates are provisioned and renewed automatically.
auto-provisioned certsDatabase-backed RBAC with role-to-permission mapping, session management, and a token blacklist for immediate revocation.
rbac · sessions · token revocationThe customer portal has no passwords to steal: sign-in is a 15-minute magic link over SMS. The request endpoint answers identically for any input and is rate-limited, so it can't be used to probe who is a customer. Sessions are stateless JWTs — rotating one secret revokes everything.
magic links · enumeration-proof · rate-limitedThe choices interviewers ask about, answered up front.
I operate multi-tenant Kubernetes at day-job scale — which is exactly why this platform doesn't use it. On a single host, Kubernetes buys autoscaling and rolling deploys this system doesn't need, at the price of a control plane to patch and a much larger failure surface to debug alone. The revisit trigger is explicit: a second host, or a genuine need for zero-downtime rollouts.
Hosted observability is priced per-container and per-GB of ingest — for a single-host platform that bill would rival the entire infrastructure budget. The requirements were narrow and Docker-native, and building purpose-fit kept everything on one pane, data on-host, and SLO tracking implemented the way I practice it professionally.
Field vehicles are the harshest environment in the system: LTE dead zones, uploads dying mid-flight, power cut at ignition-off. The SD card is the source of truth; an uploader drains it whenever connectivity allows. For telemetry, durability beats latency.
Staging shaped exactly like production catches configuration drift before customers do, and the same isolation model makes a dedicated customer instance a one-command operation instead of a re-architecture. Isolation is the product's oldest feature, not a bolt-on.