Known gaps
This page lists what Keys does not handle, and what its numbers can and cannot show. Read it before you trust a figure in the README.
The data is synthetic
Every building, home, lease, payment, ticket and person is generated from a seed (python -m keys.world.generate --seed 23 --out data/keys.duckdb). Nothing comes from a real portfolio.
What that gives:
- The generator records every fault it plants, so precision and recall can be checked against an answer key. Production data has no such key.
- A run is repeatable. The same seed gives the same world.
- The controller, the budgets and the approval queue run at a realistic size (8,000 homes, 90 days of rent, tickets, invoices and alarms, half a million badge events) without touching anyone's systems.
What it cannot show:
- Real behaviour. Payment habits, fault rates, vendor reliability and timing follow assumptions about running residential buildings. If an assumption is wrong, the numbers are wrong with it.
- Faults the world does not contain. The detectors were written against the ten kinds the generator plants. Recall says how many planted faults were found. It says nothing about kinds nobody planted.
- Right thresholds. Ten days for arrears, 60 days for a renewal offer, 45 days for a vacancy, half the SLA for a dispatch: these are settings to agree with the people who own them, not findings.
- Real false-alarm rates. The generated world is cleaner than a live one: one fault per thing at a time, honest timestamps, no data-entry backlog. False alarms are probably undercounted, and the near-perfect recall says more about the generator than about a real portfolio.
- A hold-out in the strict sense. The detectors and parts of the generator were tuned on seeds 7 and 11 (listed in
keys/dictionary/TUNING.md). The README numbers come from seed 23, which was never used for tuning. A new seed is a new draw from the same generator: it guards against fitting one draw, not against a generator that is wrong about the world.
The weakest detector: duplicate ticket
On the seed 23 week, duplicate ticket has the lowest precision of the ten detectors: 88.5% on 26 detections, and 4 of 6 on the busy day. Two separate faults of the same kind in the same building, opened close together, look like one report made twice: two leaks on different floors, two hallway lights. The common-area window was narrowed from 48 to 8 hours on the tuning seeds (keys/dictionary/TUNING.md), and tickets raised by two different alarms are never paired, but the definition still has no floor or location inside the building to tell two hallway faults apart. The scripted maintenance lead reverted three of these merges in the replay ("two separate faults on different floors, not one"). A real help desk with a location field would close most of the gap; merging stays tier 0 because a revert is cheap.
The world answers only a little
The operator's tools change the world, and the world now replies to five of them: a vendor the operator dispatches attends (inside what is left of the SLA, or within two hours) and the job is done and the ticket closed a few hours later; the site team's later ticket for an alarm the operator already ticketed is merged; a tenant who got an arrears reminder pays in full 1 to 3 hours later (60% of them, fixed per lease); a home listed at the band price is let 3 to 9 hours later (30%), a home with a concession 1 to 5 hours later (70%), with a new lease written. Those delays are compressed to hours so a twelve-hour replay can show them; real tenants take days. A re-issued renewal offer is still not answered. The answer key does not move with the operator's actions, and the replay cannot show whether an action helped. It shows whether the operator saw the fault and did what the policy says.
Who decides: rules by default, the model only for a real choice
Most incidents are decided by rules: when the runbook leaves one action at tier 0 or 1, a fixed rule fills it in and no model is called. The model is asked where there is a real choice (dispatch, re-dispatch or replace a vendor; remind or go to legal; list or offer a concession). The README run used rules only; no model was configured, so the model's quality on these choices is not measured.
What the numbers mean
- Precision counts detections, recall counts faults. Rent arrears give one detection per reminder stage, so there are more detections than faults.
- Recall is not fixing. A detected fault counts even if the action did not end it (see "The world answers only a little").
- Time to resolve is short by construction. It runs from the operator's earliest detection to the action that resolved the incident; a tier 0 action happens in the same step, so it is zero for those kinds. How old a fault already was when the operator saw it is reported separately (age at detection), and for the arrears and renewals that existed before the replay began it is days.
- The people are scripted. Approvals, rejections and reverts come from a scripted on-call shift with written rules and delays (
keys/agent/humans.yaml), labelled synthetic on every row. Real people disagree more.
The main screen
- Play replays the recorded run. The Buildings screen plays back the ledger the replay wrote, at the times the operator wrote it. It does not run the operator again in the visitor's browser, so pressing Play twice shows the same day. A fresh run is
scripts/replay_day.pyon a new copy of the world. - Ten-minute frames. Home colours are the definitions evaluated every ten minutes; a ticket opened and closed inside one frame does not colour its home. Ticket cards and receipts move at the exact minute.
- One meaning each. Cells use the lettings meaning of vacant (renovation shown separately as not let) and the credit control meaning of arrears (a rent past its due date). The finance meanings are on the Definitions and Ask screens.
- One colour per home. A home in arrears with an open ticket shows the ticket; the tooltip names both.
- Arrears hardly move during the day. Most of the day's rent payments post before 08:00, so the arrears count is almost flat across the busy window; the arrears work (reminders, legal referrals) happened in the replayed days before.
- The queue is empty at the end of the replay. The scripted on-call shift answers every tier 2 item inside the window, so the Queue screen shows the people's decisions rather than waiting items. A replay run with
--no-humansleaves its tier 2 items there for the visitor to decide.
Not built
- No adapter calls a real system. Every endpoint on a ledger row is a label for the call an adapter would make.
- No reader for a real property management system, rent ledger or help desk.
- No messages are sent: tenant letters are rows in
tenant_notices, notes to staff and vendors are drafts in the outbox for a person to send. - No per-building or per-contract rules: one arrears schedule, one set of SLAs and one band premium for everyone.
- Bands are per building and bedroom count only; size, outlook and condition do not move the price.