Pages that have since changed or gone offline are still readable exactly as they were published. Everything built on top of them can be thrown away and rebuilt without touching the originals.
Written one way only. This is my own system with no client behind it, so there is no second audience to translate it for. If you are not technical and want a sense of how I work, read the Mister & Miss entry instead. There is no link to the code. It is not open, and I would rather say so than leave you wondering.
Context
Curaçao's public record lives in a lot of places and holds still in none of them. News sites, government pages, official documents and rulings, gazettes, listings, port movements, oversight reports. Pages get rewritten, moved or taken down, and when they do the earlier version is simply gone. This is my own system. My IP, no client, and nothing in it that was not already published.
The thesis
Hoard the raw material like a dragon. Model it like a minimalist. The archive of original files is the only part that cannot be replaced. Everything after it is an opinion I will change my mind about sooner or later. So collecting is permanent and interpreting is disposable.
Why that boundary is the whole design
You will be wrong about how you organise the data. Repeatedly, as you learn what is actually in it. If the interpreting is tangled up with the collecting, every mistake costs you material you cannot go back for, because the source has moved on. Keeping them apart turns a permanent loss into a compute bill. That is the trade the whole design exists to make, and it is why the system survived several rounds of me changing my mind.
Acquire, and nothing more
The collecting layer fetches files and stores them exactly as they came, and deliberately understands nothing about them. Nothing is ever overwritten. Behind it sit source plugins, a scheduler, a job queue and a pool of workers, and it keeps its place well enough that an interrupted run picks up rather than starting over. It does not read meaning, because anything that both fetches and interprets will eventually throw away something it failed to understand.
Refine: no network, no model, no randomness
The next stage turns those raw files into one common shape, and it is boring on purpose. Plain parsers only. No network, no model, nothing that could give a different answer twice. One raw file makes one clean document, and improving a parser adds a new version rather than overwriting the old one. The original text is always kept. Half-finished results are fine and are marked as such. A document the parser only half understood is more useful than one it pretended to understand.
Where models are allowed, and where they aren't
Models get used in exactly one place: turning written prose into structured records, where ordinary parsing genuinely runs out. The shape of the answer is forced rather than politely requested, and the provider can be swapped so the system is not stuck with one company's pricing. The rule that saves the most money is the simplest one. Never send a model data that is already structured. If it arrived as a table, read the table.
Spending money on inference on purpose
Model bills are the failure nobody designs for until it has happened to them once. So the paid queue starts paused, and nothing begins spending just because a process restarted. Paid work gets estimated on a dry run before anyone approves it. There is a daily ceiling. And results are cached against the provider, model, prompt, shape and input, so running the same work over material it has already seen costs nothing. That is what makes rebuilding affordable instead of theoretical.
Deciding who is who
Working out that two mentions are the same person or company, across thousands of documents, is where systems like this usually start quietly lying. So there is an order to it. Plain rules and known name lists first. Similarity only proposes candidates. A model only judges when an operator asks it to. A person decides anything genuinely unclear. The rule underneath: models suggest, rules or people decide. Human decisions survive a rebuild. The map is disposable. The judgement is not.
What went wrong
Something broke in production, on my own machines, with nobody to blame. What made it useful is what came out of it. It was written up and turned into standing rules rather than a patch. One supervisor process. A limit on restarts, so a crash loop cannot look like uptime. Operator screens kept off the public internet. And enough saved state that a job killed mid-run picks up where it stopped. The write-up is still in the repository. I would rather run a system with one documented failure and a written response than one that claims none.
On "public data", precisely
Everything here was published in public by whoever published it. That is worth saying carefully rather than waving at. Public means the material was openly available. It does not mean every later use of it is automatically fair. Holding and organising material so it can be searched is one question. Republishing it is another. The second one gets decided source by source, not assumed for all of them at once. A system built on knowing where facts came from should be exact about where its own came from.
What this demonstrates
Working out which part cannot be undone, protecting that, and making everything else cheap to redo. Model use that is capped, budgeted and cached rather than sprinkled around. And the habit that separates a working system from a demo: deciding which stage is allowed to be clever, and not letting the others try.
TypeScript · Bun · SQLite · Deterministic parsers · Schema-forced LLM extraction · Embeddings · Entity graph