Saturday, 12:07. The system is doing sixteen orders a minute and I'm standing behind a counter watching a Grafana dashboard on my phone instead of panicking.

Last year I wrote about building a checkout system at 16 for a recurring event. That post ended with the system working and me relieved. What I didn't expect: the system getting asked back.

This summer it ran a five-day festival. Food, coffee, a merch shop with shirts and books. 4,051 orders, 6,382 items. This post is about everything that had to change in between.

You can't just point it at a new event

When the request came in, my first instinct was "sure, it already exists." That was maybe 20% true.

The original system was built for exactly one organizer. Their name was hardcoded in the header, the receipts, the emails. The database schema said "menu" everywhere because everything was food. There was no concept of a second customer existing at all.

I had two options. Fork the repo per event and diverge forever, or rebuild it as a proper multi-tenant system where each organizer gets their own shop, branding, domain and data. I picked the rebuild, partly because forking felt like a trap and partly because I wanted to learn how real multi-tenancy is done.

The isolation lives in the database, not in the application code. Every table has a tenant id, and Postgres Row Level Security enforces it: each request opens a transaction, sets the tenant for that transaction, and from then on the database physically refuses to return rows from anyone else. There are two database roles, a privileged one for migrations and a restricted one for request traffic. The nice property is that an application bug can't leak data across tenants, because the query just comes back empty. I wrote cross-tenant tests anyway.

Tenants resolve by hostname, and organizers can bring their own domain. Certificates get issued automatically when they point their DNS at us. The old repo ended at 928 commits. The rebuild added about 200 more on top, including a live schema migration that renamed "menu" to "bundle" because not everything is a menu when half your catalog is t-shirts.

Less food-shaped

A festival with a merch stand breaks a lot of quiet assumptions a food system makes.

Shirts come in sizes and colors, so products needed variants with their own stock counts, all the way down to the live availability map the shop shows. A shop needs sub-categories, because "Merch" with forty items in one list is unusable. The organizers wanted a "3 for 2" shirt deal, which meant bundles had to work with the new variant system (more on that later, it bit me). Reusable cups carry a two-franc deposit, so the system needed deposit lines on orders and a refund flow that pays people back through the original TWINT payment instead of a cash drawer. And artists needed comp vouchers: printed QR codes they can trade for food, built as a plugin so the feature only exists for tenants who turn it on.

None of these features is remarkable on its own. The interesting part was that every one of them started as a conversation with the people running the stands, and then had to survive a queue of hungry customers five days later.

Design for dead spots

Orders are picked up by scanning a QR code at a station. Festival Wi-Fi is a rumor, and I didn't want a fulfillment queue to die because a router did.

So the pickup QR doesn't reference the order, it is the order: a small binary payload with the products and quantities, signed with an Ed25519 key. A station can verify the signature and read the contents with zero connectivity. Redemptions get written to a local ledger on the device and sync back when the network returns. The same mechanism covers the artist vouchers, plus a block list the stations sync so a revoked voucher stays dead even while a station is offline.

Station setup got the same treatment. Whoever has the shift opens a link we hand them as a QR code, taps confirm, and their own phone is now a logged-in station that only sees the products assigned to that stand. We had four cheap Xiaomi phones as backups, but most shifts just used whatever was in someone's pocket.

The hardware corner

The part of the last post people asked about most was the hardware, so: two Samsung Galaxy Tab A9 tablets as POS, two SumUp card readers, two thermal printers, and the four backup phones. The budget philosophy from the first event survived: consumer hardware, nothing that couldn't be replaced the same day.

The printers deserve a closer look, because they're where the most money got saved. Real receipt printers, the kind bolted to restaurant counters, cost a few hundred francs each, and the usual way to talk to them is through a vendor SDK. Mine are Phomemo T02s: pocket thermal sticker printers, around thirty francs each, meant for printing photos and to-do lists, not receipts. What makes them usable is that underneath they're just a Bluetooth serial port. The app opens a raw RFCOMM socket and writes bytes at it: two standard init commands, one vendor-specific "start raster" sequence, and then the receipt itself as a raw 1-bit image, 384 dots per row, eight pixels packed into each byte, streamed in blocks of up to 255 lines. The printer has no idea it's printing a receipt. It thinks it's printing a sticker.

The receipt is drawn as a bitmap on the tablet before any of that happens, with anti-aliasing turned off, because a thermal head has no concept of gray: every soft edge gets thresholded into speckle. That's also why I don't use the printers' built-in fonts even on the normal ESC/POS path, they ink unevenly and look worse than a clean bitmap. The whole hack carried over from the first event and ran unchanged for five more days.

The Android app around all of this is deliberately thin. It's a WebView of the web POS with JS bridges to the SumUp SDK and to that printer socket. I never wrote a second UI. When something needed fixing during the festival, I fixed the web app, deployed, and the tablets had it on the next reload. No app store, no update prompts.

Making it boring

Before the event I load-tested the whole order path with k6. The first run fell over in a way I didn't expect: Postgres ran out of connections while the CPU was idling. The fix was PgBouncer in front of the database and properly sized pools, and the retest held around 200 orders per second.

The festival's actual peak was 0.27 orders per second. A 750x margin sounds absurd, and it is, but that margin is why I spent Saturday watching a dashboard out of curiosity instead of fear.

Across the five days: 415,000 requests, 22 of them ended in a server error, p95 latency around half a second in the busiest hour, one small node that never left single-digit CPU. The database pool peaked at 18 connections. Inventory held too. The checkout takes a row lock on stock, so even when 283 payments landed inside the peak lunch hour, nothing oversold. Over the five days the system turned away exactly 50 order attempts for items that had just run out. A rejected order is annoying. Handing out refunds for burgers that don't exist is worse.

What broke anyway

I was on site all five days with a laptop in my backpack, and it did not stay closed.

The first bug was self-inflicted history. Bundles were built before variants existed, so no bundle had ever contained a product with variants. The "3 for 2" shirt deal was the first, and configuring it crashed the API. Nobody could have hit this before the festival, because the combination wasn't possible until three weeks before it. I fixed it on site and shipped it mid-event, one of two deploys that went out while the festival was running.

The second one bugged me more. A handful of orders came back from the TWINT redirect without their pickup QR. The payment went through, the order existed, but the phone showed nothing to scan. I never managed to reproduce it, and chasing a redirect race on other people's phones during a festival is a losing game.

So instead of fixing the bug I removed its habitat. The order is now created and stored before the customer ever leaves for the payment page, the phone keeps the order id, and the order shows honestly as unpaid until the backend confirms the money. Confirmation stopped relying on the payment provider's webhook alone; the backend also pulls the payment status itself. The end result is that you can start a payment, kill the app completely, finish paying, reopen the app, and your paid QR is just there. The original bug still exists somewhere, probably. It no longer has anything to break.

Five days later

The data the system collected is my favorite part. The inventory ledger recorded every sale, restock and refund with a timestamp, so the organizers now know things like: both burgers sold out mid-peak on Saturday, the ice cream sold four and a half times its opening stock thanks to live restocking, and 69% of the 1,066 deposit cups came back. Next year's stocking plan is a spreadsheet instead of a feeling. About 95% of payments went through TWINT. The card readers were almost decorative.

Last time the lesson was that the knowledge comes from doing it. This time it was subtler: building a system and keeping one alive are different jobs. The first version proved I could make it work. This one had to work for people who don't know me, on their own domain, with their own products, while I stood in the crowd hoping the dashboard stayed green.

It mostly did. And where it didn't, I was there.

Next goal: an event where I don't have to be.