Reverse engineering Rausgegangen

Rausgegangen is an event platform for German cities. Concerts, club nights, exhibitions, readings, flea markets, that kind of thing. If you live in Leipzig and want to know what’s on this weekend, it’s usually the first place you look.

I’d been using it for years without thinking much about it. Then I wanted to know what a city actually programmes over a month, and how that differs from the next city over. The app can’t tell you that. It shows you a list.

So I went and got the data. Same rabbit hole as Nextbike last year, entered the same way.

Reverse engineering the app

Same setup as last time: Android Studio’s emulator with HTTP Toolkit attached to it. Install the app, open HTTP Toolkit, scroll around, watch the requests come in. It still just works, which is nice, because I remembered nothing from last time.

Everything interesting goes to one host:

https://api.rausgegangen.de/rausgegangen/api/v2

Nothing authenticates it. The only cookie that comes back is HCLBSTICKY, which sounds like identity and is just load balancer stickiness. The hash names a backend node and is identical across unrelated clients. I don’t send it back, because replaying a stale one pins every request to a single node, which is rude.

Standard so far. The interesting part came when I opened one of those URLs in a normal browser instead of the emulator.

The API documents itself

It’s a Django REST Framework app, and the browsable HTML renderer is still enabled in production.

That’s not DEBUG=True. There are no stack traces and no settings, nothing actually leaked. What it does render is the view’s docstring, at the top of the page, written by whoever wrote the endpoint.

Which is how I found this, on the dispatch view:

Accept-header router: v2.5+ clients get the price/daytime filters and the gated location results, older ones keep the pre-2.5 result set.

I have never had an undocumented API explain its own versioning to me before. This is the best documentation that exists for this thing, and it exists by accident.

It also saved me from a trap I would have walked straight into.

The traps

The accept version is load-bearing. The app sends application/json; version=2.3, and I copied the app’s headers, like you do. But the price and hour filters are gated behind 2.5, and below that they are silently ignored rather than rejected. So a client pinned to the app’s own version passes price_max=10, gets back everything, and believes it filtered. 2.5 is the highest the server accepts. 2.6 returns a 406.

Copying the app exactly was the wrong move here, which is not a sentence I expected to write.

accept-language changes the content rather than the labels. With en you get machine translations ("Free admission", "24,99 to 25,00 €"). With de you get what the organiser actually typed. The payloads even carry an is_machine_translated flag confirming German is the source. So I collect in German and can translate later if I ever want to. You can’t go the other way.

numFound is a lie. Not a small one either:

City	numFound	Actually reachable
Leipzig	41,460	12,575
Berlin	51,879	38,917

I burned an evening on this. My crawl kept “finishing” at about a third of what it said it should find, so I assumed I was being rate limited or silently truncated. The results stream is collapsed somewhere in the pipeline while numFound counts documents from before that. The reachable numbers match each city’s venue census almost exactly, so nothing is missing. The counter is measuring a different thing.

Page until a response comes back empty. Never use numFound as a loop bound.

page and offset do nothing. They’re accepted, they look like pagination, they are ignored. Only start works. Unrecognised parameters are dropped silently across the whole API, so a filter that seems to have no effect might just be a typo.

There is no past. Any date before today returns zero results. The archive cannot be backfilled through search. Historical data only exists if somebody collected it while it was happening, which is most of why this project exists at all.

The entity model

This took me longer to work out than any of the header stuff, and it’s the thing that would actually break your data if you got it wrong.

There are four types:

• event is the concept. “Familienzeit: Schatzsuche”. It carries the description, the category and the pricing.
• subevent is one dated occurrence of it. This is the row you actually want. One event can have hundreds.
• page is a venue and an organiser and an artist, all one type split by a type field. There is no /locations/{id}, it 404s.
• city is a slug and a centre coordinate.

The part that gets you: inside an event payload, organizers and future_subevents are lists of bare id strings, and the actual objects are delivered separately under prefetchedData. Read those id lists as objects and everything downstream is quietly wrong.

Once you understand this, the crawl strategy falls out of it. /events/{id} returns every occurrence of the event in prefetchedData, 203 of them in one case I hit, so you crawl per event rather than per subevent. That alone cut my request volume by about 3x.

Collecting it

Day by day, city by city.

Date filtering matches on start_date rather than overlap, which sounds like a limitation and is actually a gift: every subevent falls into exactly one day’s window, so a day-by-day sweep partitions cleanly and never double counts. The cost is that a two-week festival only shows up on the day it starts.

The crawler is Python, runs once a day, and writes into Postgres. It uses 4 concurrent requests and refetches event detail only when the stored copy is more than 7 days old. I tested higher concurrency and hit no throttling at all, which is a reason to be careful rather than a licence. There’s no rate limiting here and no auth, so the only thing keeping this polite is me deciding to be.

I also swept one and two letter prefixes through /cities?search= out of curiosity. The bare /cities call returns 83 promoted cities. The prefix search reaches 12,881. It matches on prefix rather than substring, so search=ulm finds ulm, ulmet and ulmen but not neu-ulm. Almost none of those 12,881 have real event coverage, so I crawl the promoted ones, but it was a fun 21 seconds.

What it looks like

The charts below are live. They’re not screenshots and they’re not baked into this post.

They’re ES modules exported from my Observable Framework data site, which rebuilds hourly off the same Postgres database, and this page imports them at runtime. So they’ll drift as the crawler keeps running, and this post will quietly go out of date around them. That’s sort of the point.

Here’s the volume, by city, over the current window:

The chart could not be loaded. It lives on the data site.

Berlin being enormous is not a surprise. The interesting part is that the order here is not population order.

Programming is more telling than volume. This is the share of each city’s own listings by category, so cities are comparable despite wildly different sizes:

The chart could not be loaded. It lives on the data site.

And the shape of a month, which has a weekly rhythm you can see without squinting:

The chart could not be loaded. It lives on the data site.

What’s still wrong with it

Plenty.

Coordinates ride along on search results, and a chunk of listings carry none, so any map I draw is missing rows. Multi-day festivals sit entirely on their start date, which distorts the daily chart. Prices are messy free text more often than they’re numbers. And the whole thing only goes forward, so the archive is exactly as old as the day I started running it, and no older.

Some things I want to do with it eventually:

• Actual time series, once there’s more than a few months of it
• Venue churn: which places stop programming, and when
• Category drift across the year
• Whether cities copy each other’s programming, and with what lag
• Some measure of how concentrated a city’s nightlife is across venues

The crawler and the data site are both in my playground repo. There’s an API.md in there with everything I worked out about the endpoints, which is more complete than this post.