Quality

Load tests & reliability

We pushed the backend until it broke, on purpose: every endpoint under load, races on a single slot, hostile input, killed workers, and a database and Redis taken away mid-test. This page shows what held, what broke, and what we changed.

69,424

requests across all 89 routes

621

complete user journeys, up to 90 concurrent users

wrong answers and 0 server errors under load

0 / 1,957

requests failed while API workers were being killed

How we tested#

k6 scripts drive the real API: 500 drivers, 50 hosts and 1,000 listings in an isolated database.
Profiles: full user journeys, breakpoint ramps per endpoint, a slot race, and chaos (killing workers, stopping Postgres and Redis).
A Python probe sends hostile input to every route: malformed and deep JSON, NaN and 1e308, year 0001, NUL bytes, tampered and alg=none tokens, path traversal.
Local machine (16 cores, 4 API workers, k6 on the same box), so absolute numbers are pessimistic; where and how things break carries over.

loadtest/text

./loadtest/k6.sh everything -e PROFILE=stress -e VUS=15      # every endpoint, up to 90 users
./loadtest/k6.sh breakpoint -e TARGET=search -e MAX_RPS=3000  # ramp one endpoint until it breaks
./loadtest/k6.sh race                                          # many drivers, one slot
uv run python ../loadtest/probe_edges.py                       # hostile input
./loadtest/chaos.sh                                            # kill workers during load

Capacity per endpoint#

GET /v1/config

1,634 req/s

GET /v1/me

1,009 req/s

GET /v1/listings (search)

865 req/s

GET /listings/{id}/reviews

663 req/s

POST /v1/bookings/quote

562 req/s

GET /v1/listings/{id}

441 req/s

GET /v1/bookings

266 req/s

POST /v1/bookings

117 req/s

POST /v1/auth/login

23 req/s

Peak sustained requests per second, 4 workers. Reads are fast; login is deliberately slow because of argon2 password hashing (measured before hashing moved off the event loop).

Correctness under pressure#

1 booked99 clean 409 slot taken

100 drivers booking one slot for the same window at the same instant.

Scenario	Result
20 parallel retries with one Idempotency-Key	1 booking
20 parallel sign-ups with one email	1 account, 19 × 409
Hostile input on every route	all clean 4xx, no 500s
Brute-force login	429 with Retry-After after the limit
API worker killed under load	respawned in seconds, 0 of 1,957 requests failed
Database during the whole run	0 deadlocks, never more than 6 queries active at once

What broke, and what we did about it#

Finding these was the point of the exercise. The fixed ones each have a regression test, so they cannot quietly come back.

#	Failure under load	Fix	Status
1	Search at ~860 req/s ran out of DB connections: requests waited 30 s, then 500	Pool wait capped at 3 s, answered as 503 + Retry-After; search cache	Fixed
2	Password hashing ran on the event loop: login capped at ~23/s and every other request stalled	Hashing in a bounded thread pool; DB connection released before hashing	Fixed
3	Postgres down: every request hung 15 s, then 500	Short connect and pool timeouts, 503 instead of 500	Fixed
4	A locked table made search and quotes 500	Lock waits classified and answered as 503	Fixed
5	A NUL character in any text crashed the request	Rejected everywhere with 400 before touching the database	Fixed
6	Redis down: readiness failed, so a load balancer would pull every instance	Readiness stays up as “degraded”; Redis-backed features degrade instead of failing	Fixed
7	No body size limit: 20 MB bodies were parsed	1 MiB cap, refused before reading	Fixed
8	Redis pool too small at peak, pushing load onto the database	Pool size is configurable; to be sized for production traffic	Partly
9	Worker down meant notifications silently piled up	Outbox backlog and dead letters exposed in /metrics; alerts still to be wired	Partly
10	If the server's master process died, workers ran unsupervised	Container restarts the pair together on Render	Partly
11	Listing detail slower than search	Cache it and use one session	Open

Observability#

The API exposes Prometheus metrics at /metrics (behind a bearer token): request rate and latency per route template, outbox backlog and dead letters, password-hashing latency, and login-protection outcomes. Every request is logged as one JSON line with a request id, which is also returned to the client in X-Request-ID for support.

/metrics (format, illustrative values)text

parkezy_http_requests_total{method="GET",route="/v1/listings",status="200"} 51234
parkezy_http_request_duration_seconds_bucket{method="POST",route="/v1/bookings",le="0.25"} 9112
parkezy_outbox_pending 0
parkezy_outbox_dead_letters 0
parkezy_login_attempts_total{outcome="throttled"} 0