Fetch
A polyglot microservices food delivery platform — Python/FastAPI and Node.js/TypeScript services side by side — built to solve distributed-systems problems properly: distributed transactions, exactly-once effects over at-least-once delivery, and failure isolation that does not cascade.
6
services, 2 languages
11
event types over one exchange
5
provisioned dashboards
// stack
// architecture
6 services across Python/FastAPI and Node.js/TypeScript · database-per-service PostgreSQL · RabbitMQ saga with outbox, inbox, retry queues and DLQs · hand-written OpenTelemetry tracing to Jaeger in both languages · Prometheus + 5 provisioned Grafana dashboards · Kubernetes with HPA/VPA · Terraform for kind and EKS
// overview
Six independently deployable services spanning Python/FastAPI and Node.js/TypeScript, behind an API gateway, each owning its own PostgreSQL database. Order placement runs as a choreographed saga over RabbitMQ with a transactional outbox on the way out and an inbox claim on the way in, so a broker outage cannot lose an order and a redelivery cannot double-charge a customer. Every service is traced end to end with hand-written OpenTelemetry instrumentation, exports metrics to Prometheus, and runs on Kubernetes with resource limits, horizontal autoscaling and Terraform environments for both a local kind cluster and AWS EKS. Almost every claim in the README was verified live against the running system — real orders placed, RabbitMQ and PostgreSQL deliberately stopped mid-flow, synthetic poison messages published — rather than inferred from the code.
// what was built
- ·Six independently deployable services behind an API gateway — auth, restaurants, orders, payments, driver matching, notifications — deliberately polyglot across Python/FastAPI and Node.js/TypeScript so the boundaries have to be real contracts, not shared code.
- ·The payments and driver-matching services are Python and FastAPI end to end: async handlers, Pydantic models on every boundary, aio-pika consumers, SQLAlchemy over their own PostgreSQL databases, and pytest suites — full participants in the saga, not sidecars bolted onto a Node system.
- ·Four of those services run as two processes from one codebase: an API that only serves HTTP and a worker that owns every piece of async work, so a slow consumer can never starve request handling and each half scales on its own.
- ·RS256-signed JWTs verified independently at each service boundary against auth-service’s public key, so no service has to call auth to authenticate a request.
- ·Order placement is a choreographed saga over a single RabbitMQ topic exchange — no orchestrator. The failure path is the interesting half: an unassignable order triggers a compensating refund and unwinds to CANCELLED instead of stranding a charge.
- ·Transactional outbox: an order and its outbox row commit in the same Postgres transaction, and a separate publisher process drains it. Verified by stopping RabbitMQ, placing an order (still 201), restarting, and watching the saga complete from the queued row.
- ·Transactional inbox keyed by publisher-assigned event id, with a RECEIVED → PROCESSING → PROCESSED/FAILED lifecycle, so a crash mid-handler leaves a queryable record rather than a vanished message.
- ·Delayed retry queues for transient failures and a diagnostics-carrying dead letter queue for poison messages — never silently dropped, never retried forever.
- ·Event schema versioning with version-aware parsers on both consumers, independently implemented in Python and TypeScript, plus JSON Schema contracts in a shared directory validated fail-fast on the producer and defensively on the consumer.
- ·Redis-backed sliding-window rate limiting at the gateway, implemented as a single atomic Lua script and failing open on a Redis outage — with a separate counter for fail-opens, because "clients are being throttled" and "throttling is not happening at all" are different operational facts.
- ·Circuit breaker plus hand-rolled retry with exponential backoff on the one synchronous inter-service hop, so a dead restaurant-service produces fast 503s instead of a self-inflicted retry storm.
- ·End-to-end OpenTelemetry tracing, deliberately without auto-instrumentation and hand-written twice — once against the Python SDK and once against the Node one — exporting to Jaeger; metrics to Prometheus; five Grafana dashboards and three alert rules, all provisioned as code.
- ·Kubernetes-native: Deployments, StatefulSets, resource requests and limits, Horizontal and Vertical Pod Autoscalers, host-based Ingress — plus a Terraform foundation covering both a local kind cluster and AWS EKS.
System design
Drawn from the actual source: services, data ownership, message flow and failure paths. Drag to pan, scroll to zoom, or open any diagram fullscreen.
Platform architecture on Kubernetes
Six independently deployable services across two languages, one database per service, a single async backbone, and observability wired through everything. Four of these services run as two processes each: an API and a worker, scaled independently.
// engineering notes
The decisions worth talking through
The parts of this project where the interesting work was choosing between options, not writing the code.
Reconstructing a trace across a process boundary and a time gap
order.created never publishes from a live span — it is written to an outbox table and picked up seconds later by a different OS process, with no span left to inject from. The fix was to capture the current trace context into its own column at construction time (deliberately not inside the payload, which is published byte-for-byte and schema-validated with additionalProperties: false), then read it back at publish time and re-parent the outgoing span. It is the one part of the observability work that could not be solved by wiring up a library.
Histogram buckets are a design decision, not a default
The OpenTelemetry SDK ships default histogram boundaries tuned for millisecond-scale values. Recording sub-second _seconds durations against them collapsed every observation into the first bucket or two, producing a technically-populated but useless histogram. Every _seconds histogram now gets explicit boundaries matched to its actual scale — sub-10s latency buckets for HTTP, a much wider 0.1s–60s range for saga duration, since a saga is a whole business transaction rather than one request.
Metric cardinality, caught by measuring rather than guessing
The first pass labelled HTTP metrics with the resolved request path, which turned every order id into its own label value and blew up cardinality. Every service now normalises to a route template — a regex placeholder swap on the Node side, and Starlette’s own matched route path on the FastAPI side, which is more precise because it is the actual route definition rather than a reconstruction.
Two real bugs found by verification, not review
Running the security hardening against the live services surfaced two genuine defects in shared error-handling middleware: CORS running before request-id meant a rejected cross-origin request crashed the error handler and leaked a stack trace to the client, and the handler discarded correct HTTP statuses from library errors by collapsing anything that was not a local AppError into a 500. Both were fixed; both were invisible on a code read.