Anatomy of a Solo-Operated Production Microservice Platform

Introduction

Most architecture writing describes systems built by teams for scale. This paper describes the opposite corner of the design space: a system built to be operable by one person while running a revenue-generating business with field crews, customer billing, and vehicles on the road. Every choice below is downstream of that constraint. Complexity is spent only where it buys legibility, recoverability, or isolation; anything that would demand a second operator to babysit is rejected regardless of how standard it is elsewhere.

The platform has been in continuous production since 2023. Table 2 gives current production volume. The remainder of this paper proceeds bottom-up: topology (§2), networking (§3), data (§4), deployment (§5), observability (§6), an AI subsystem case study (§7), field hardware (§8), security (§9), failures (§10), and design rationale (§11).

System Overview

The platform comprises fifteen services grouped the way the system actually fails: an edge tier, application services, asynchronous workers, platform services, and hardware in the field. Figure 1 shows the topology; Table 1 the inventory.

Networking and the Edge

All public hostnames resolve through DNS to a CDN/proxy layer, which forwards to the origin host's nginx edge. nginx terminates TLS and routes by hostname into per-stack container networks. Figure 2 traces one request.

Service discovery: names, not addresses

Inside a stack, services address each other by container name (http://acme-api:5002), resolved by the container engine's DNS on every connection. This is a lesson written in downtime: the previous generation substituted live container IP addresses into the proxy configuration and each service's environment at deploy time. The container engine assigns a new IP whenever a container is recreated, so every consumer kept pointing at the old address until it was redeployed too. In one recorded incident, a database restart moved MySQL to a new address and application containers logged EHOSTUNREACH against the stale one for two hours — while a second consumer failed soft and silently stopped recording analytics. Databases are therefore addressed by name everywhere; deploy order no longer matters.

Certificates and multi-domain routing

The multi-environment layer fronts through a single Caddy proxy that terminates TLS for every routed domain, provisioning and renewing certificates automatically — including each new tenant subdomain on first deploy, on wildcard DNS. The legacy single-business edge uses nginx with mounted certificates and a generated configuration; both attach per-stack routes at deploy time.

Ingestion from the field

Vehicle hardware uploads over LTE as gzip-compressed NDJSON batches with retry and backoff (§8); the edge treats these as ordinary authenticated HTTP posts. Kiosk timeclock punches carry GPS coordinates and are validated server-side against site geofences in both directions — clock-in and clock-out — because a location check applied only at clock-in invites clocking out from anywhere.

Data Layer

Relational core

Each environment runs its own MySQL 8 server — no shared database between tenants. Timestamps are stored UTC and converted to the business timezone only at presentation and at business-rule boundaries: calendar-day constraints (for example, "one social post per day") are computed on the business timezone, because a UTC day boundary and an Eastern business day disagree for four to five hours out of every twenty-four. Getting this wrong is the platform's most reliable class of bug, so the conversion points are deliberately few and explicit.

Object storage and the media pipeline

MinIO provides S3-compatible storage, partitioned into domain-scoped buckets (receipts, social media, website media, message attachments). Media is normalized at upload: phone camera formats (HEIC/HEIF, AVIF) are decoded — via a WASM decoder for the patent-encumbered codec, since prebuilt native image libraries do not ship HEIC support — and re-encoded to platform-safe JPEG before they ever reach a bucket. Serving goes through on-the-fly WebP thumbnail derivation with immutable cache headers keyed on immutable object names, so a 10 MB original is fetched once and a 26 KB derivative serves every subsequent thumbnail request.

Backups

The control plane drives hourly per-tenant backups: one gzipped dump covering a stack's business and monitoring schemas, written with drop-and-create semantics so a restore fully reconstructs both, stored mode-600 (the file is the dataset), pruned to the newest 72 (≈3 days). Restores are destructive by definition and require typed confirmation. Restore-plus-redeploy is the entire disaster-recovery story, by design.

Deployment and Environments

Every service builds in GitHub Actions to multi-architecture images (amd64/arm64 via Buildx + QEMU) published to GHCR; semantic version tags cut releases, and pull requests build without publishing. Deployment is one idempotent tool, stackctl, and a control plane that turns environments into a product (Figure 3).

Key properties, each earned rather than aspirational:

• Naming is the isolation boundary. Everything is prefixed by stack name — network acme-net, containers acme-api, database acme, storage acme-minio. Isolation is spelled, not configured.
• Change detection by SHA. A service redeploys only when its repository SHA differs from the stack's recorded deployed SHA; already-built SHAs deploy from cache. Backend images are shared across stacks per SHA; frontends bake per-environment branding and build per stack.
• Preflight before swap. The deploy path verifies the core datastores are running — starting them if stopped, aborting if they will not come up — because a deploy that proceeds against a down datastore bakes dead endpoints into every rebuilt container's environment.
• Feature flags per environment (ENABLE_SMS, ENABLE_KIOSK, …): a disabled service is neither deployed nor routed.
• Serialized jobs. Deploys are heavy and not concurrency-safe per stack; the control plane's job runner executes exactly one at a time, with a streaming log drawer.
• Idempotent schema bootstrap. A fresh database seeds from the API repository's checked-in schema, tracked by a marker table; re-running provisioning is safe.

The control plane adds the tenant layer: signup-to-workspace provisioning (gated by default — an anonymous form that consumes a stack's worth of RAM is a denial-of-wallet vector), owner/member roles with expiring invitations and per-tenant audit logs, and subscription entitlements with Ed25519-signed licenses enforced at stack level for self-hosted installs. Every state change shells out to the same stackctl used by hand, and status is read back from its registry — so the CLI and the UI cannot drift (§11).

Multi-tenant productization

As of August 2026 the platform is offered multi-tenant under the product name Field Platform (fieldplatform.io). The control plane — signup and admin UI over stackctl — itself runs containerized on the same host and drives the host container engine over its socket (Docker-outside-of-Docker); the repository is bind-mounted into the container at an identical absolute path, so every host-path the tooling constructs remains valid on both sides of the boundary. The engine is addressed through the Docker-compatibility API rather than the native libpod endpoint, because the host's podman 3.4 rejects newer remote clients.

Each customer stack is isolated exactly as production and staging are: its own bridge network, its own MySQL and MinIO, per-stack secrets, and images tagged by git SHA — backends shared across stacks, frontends built per stack because NEXT_PUBLIC branding is baked at build time. Deploys remain change-detected by SHA, and an optional registry pull-through (IMAGE_REGISTRY) substitutes prebuilt backend images for local builds. Provisioning is fully dynamic: stack creation writes the stack's DNS record set through the Cloudflare API — CNAMEs to a single unproxied origin record, each tagged with a stackctl:<name> comment so that destroy removes exactly what create wrote — and stretches one Let's Encrypt certificate over every stack's <domain> and *.<domain> via user-owned certbot DNS-01. Renewed pem files are hot-swapped by content overwrite, preserving the inode that nginx's bind mounts track, followed by a graceful reload.

Routing is two-tier behind the legacy nginx during the pre-cutover phase. nginx terminates TLS with one wildcard-per-stack certificate and forwards a single *.fieldplatform.io catch-all to a shared Caddy edge on an alternate port (:8880, with plain-HTTP site addresses to avoid auto-HTTPS redirect loops). The edge routes by Host header to each stack's own Caddy proxy at a published host port — container-DNS attachment is impossible for a running rootless podman 3.x container, so a host port is the join point — and the stack proxy in turn routes to the stack's services by container DNS on the stack network. Adding a customer therefore touches no nginx configuration. A demo stack (demo.fieldplatform.io) exercises the entire path.

The first tenant deploy surfaced and fixed three latent defects invisible to a long-lived single-tenant install: a stale seed schema (fresh databases missing tables the API imports at boot), frontend prebuilds resolving an ancient node in non-interactive shells, and next build dirtying checkouts by auto-installing @types/node.

Observability and Reliability

Health aggregation. A dedicated service auto-discovers containers via the container-engine API and probes each over HTTP, MySQL, or TCP on a background loop — no manual registration, no stale check configuration.

Monitoring and SLOs. A purpose-built dashboard tracks container metrics, aggregates logs, fires alerts, and tracks SLOs against explicit targets, behind database-backed RBAC. Building rather than buying kept ingest on-host and the bill at zero (§11).

Notification stream. Operational events a solo operator must not miss — failed publishes, OAuth tokens near expiry, unanswered customer comments — flow through an event table into the operations app's notification bell, piggybacked on existing scheduler cadence rather than adding new pollers.

Case Study: AI-Assisted Publishing as a Closed Loop

The social publishing subsystem is a compact study in operating an LLM feature in production, end-to-end (Figure 4). The design goal was not "generate captions"; it was a measurable loop in which generation quality is judged by platform engagement and tuned without deployments.

• Constrained generation. Vision calls return against a strict JSON schema (title / caption / hashtags); the caption contract — contact-line placement, hashtag count and casing — is enforced in code, never trusted to the model. Off-contract output is corrected or the offending sentence dropped, not shipped.
• Styles judged by results. Captions rotate through engagement "angles" (challenge, question, micro-story, …); each generation records its angle, and an insights endpoint batch-fetches platform engagement against the stored ids, rolled up per angle — a scoreboard of which prompting styles actually earn comments.
• Prompts as data. Prompts, angle lists, posting times, and formatting rules live in a settings table: system defaults seeded and refreshed from code, owner overrides beside them. Voice tuning never requires a deploy (§11).
• Server-enforced calendar constraints. One post per business day, enforced at create/update and inside the autopilot allocator; conflicts slide forward and the API response says why the date moved.
• Rate-limit-aware caching. Third-party feeds layer three defenses: browser persistence paints instantly across page loads; a server TTL cache absorbs passive traffic; an explicit refresh is the only real upstream fetch, floored so even a mashed refresh button cannot burn API quota.
• Failure surfacing. Publish failures alert; OAuth tokens are flipped to needs-reconnect before expiry rather than after the first failed post; new-comment deltas nudge the operator to reply — all via the notification stream of §6.

Field Hardware: Store-and-Forward Telemetry

Field vehicles are the harshest environment in the system: LTE dead zones, uploads dying mid-flight, power cut at ignition-off. The firmware treats the SD card as the source of truth — every OBD-II/GPS reading lands on disk as NDJSON before anything else, files rotate by size and time and survive reboots, and an uploader drains them as gzip-compressed batches with retry and backoff whenever connectivity allows. For telemetry, durability beats latency: a reading that arrives ten minutes late is fine; one that never arrives is not. The fleet has delivered 100,000+ readings to date under this discipline.

Security Model

• Isolation by default. Every tenant stack has its own network, database, object storage, and secrets; there is no shared state to leak across. Isolation is the architecture, not a setting.
• Secrets out of band. Credentials live in per-environment env files injected at deploy time — never in images, never in git.
• TLS at the edge, automatically. Certificates are provisioned and renewed per routed domain, including new tenant subdomains on first deploy.
• Passwordless customers. Portal sign-in is a 15-minute magic link over SMS; the request endpoint answers identically for any input and is rate-limited (enumeration-proof), and sessions are stateless JWTs revocable by one secret rotation.
• AuthN/AuthZ. Database-backed RBAC with role-to-permission mapping, session management, and a token blacklist for immediate revocation.
• Physical-world checks. The timeclock validates GPS proximity to a configured site radius on both clock-in and clock-out; punch and timecard records are referentially linked so removing one cannot strand the other.

Failure Case Studies

Reliability claims are cheap; commit logs are not. Two incidents, reconstructed with hashes.

June 2026 — the observer effect

The monitoring dashboard gained a live topology view — container stats, sparklines, request traces — on top of its real-user-monitoring ingest. The monitoring database lives on the same MySQL server as production, and within hours the topology view was timing out, each timeout representing load pressure on the database that also serves the business. The first fix made it worse: parallelizing all queries and trace fetches multiplied the concurrent load on an already-stressed database, and was reverted the same day. The durable fix went the opposite direction — batch the ingest, reduce limits, poll slower, fetch less, and put explicit timeouts and error containment on every query so the dashboard degrades instead of hammering.

Monitoring is production. The observer carries the same load budget as the observed, and parallelism is not a fix for overload — it is a multiplier on it.

jun 08  ff77443  Fix topology timeout: batch RUM inserts into chunks of 25
jun 08  730466e  Fix topology timeout: parallelize all DB queries and trace fetches
jun 08  d4ce6c3  Revert "Fix topology timeout: parallelize all DB queries ..."
jun 08  9036fc7  Fix topology timeout: reduce query limits safely
jun 08  b743295  Fix topology timeouts: slower polling, fewer trace fetches
jun 09  4aa70e5  Fix topology loading: reduce limits, add timeouts, catch errors

August 2026 — the build that lied

An image-processing dependency ships its native binaries as optional packages carrying a node ≥ 20.9 engines constraint; the runtime image ran Node 18. npm skips engine-mismatched optional dependencies silently — so the image built green and the container came up healthy, but the first require() at request time threw, and every image the application serves returned a 500 through the framework's generic error page. Diagnosis came from the outside in: sibling routes sharing every import except the image library answered clean JSON errors, isolating the failing module without a shell on the box. The fix was one line; the finding was not.

A green build proves the dependency graph resolved, not that it can load. Engines mismatches on optional dependencies are a silent runtime landmine — the class of failure that surfaces only in production, on the first request.

aug 25  8700923  Bump runtime image to node:20-alpine for sharp 0.35

Design Rationale

Compose over Kubernetes. The author operates multi-tenant Kubernetes at day-job scale — which is exactly why this platform does not use it. On a single host, Kubernetes buys autoscaling, bin-packing, and rolling deploys this system does not need, at the price of a control plane to patch, upgrades to sequence, and a far larger failure surface to debug alone. Compose gives a one-file topology, deterministic deploys, and disaster recovery that amounts to "restore volumes, re-run the deploy." The revisit trigger is explicit: a second host, or a genuine need for zero-downtime rollouts.

Build over buy for monitoring. Hosted observability is priced per container and per GB of ingest; for a single-host platform that bill would rival the entire infrastructure budget. The requirements were narrow and container-native — stats, logs, alerts, SLOs — and building purpose-fit kept data on-host and SLO tracking implemented as practiced professionally: explicit targets reviewed against reality.

Prompts as data, not code. Prompt tuning is the highest-frequency change in the AI subsystem; making it a deploy would either freeze iteration or turn every voice tweak into a release. The cost is a settings table and a seeding pass at startup; the payoff is a feedback loop — the per-angle engagement scoreboard — that can be acted on the same day it reports.

Per-environment isolation on one host. Shared-nothing per environment means a full MySQL and object store per stack, trading RAM and disk for a blast radius that stops at one environment. Staging shaped exactly like production catches configuration drift before customers do, and the same mechanism makes a dedicated customer instance a one-command operation. The right trade while stack count is small — and the thing to revisit before it is not.

One source of truth for deploys. The control plane could talk to the container engine directly and be faster to build. Instead, every state change shells out to the same stackctl used by hand, and status reads back from its registry. A second implementation of "what does deployed mean" guarantees the UI and the CLI eventually disagree — usually during an incident, when the dashboard is the thing being trusted. Shelling out costs a process spawn; disagreeing costs the outage.

Conclusion

A production platform sized honestly to its operating constraint — one operator — looks different from one sized to a résumé: fewer moving parts, names over addresses, isolation over sharing, purpose-built observability over hosted bills, and prompts, settings, and deploy state kept where a single person can reason about them at 2 a.m. The system has run a real business continuously since 2023, and its two best-documented failures each produced a rule that the architecture now enforces structurally. The author is happy to walk through any component in depth — including code — in an interview setting.

References

Revised August 26, 2026 · typeset in Latin Modern · interactive overview · LinkedIn · GitHub