Incident report · canonical source docs/incidents/2026-07-2026-08-host-incidents.md

Host incidents, July–August 2026: two host-suspend windows and a boot-environment database fork

Report date 2026-09-08 · status: final · audit rows fl_downtime 984–991 · replaces the interim operational notice published 2026-09-03 (commit 8320951).

Revision 2026-09-10: commit identifiers updated after the pre-publication history filter; see docs/09-publication-attestation.md.

Summary

The record's correctness was never affected. Every published outcome is the category that on-time scoring produces — provably so for the cross-domain outcomes, whose date-based windows read a referee store verified unchanged, and reproduced for the single-domain outcomes that were scored twice; the referee data saw zero revisions across both windows. The record's completeness and timeliness were affected, and both have been repaired and disclosed: late classifications carry their true classification timestamps against their deadlines in the CSV, and the rows lost to the database fork have been restored under new identifiers with their provenance recorded.

Incident 1 — host suspend, 2026-07-23 → 2026-08-10 and 2026-08-14 → 2026-08-16. The workstation hosting the entire engine was suspended, in six offline segments over eighteen days and once more for two days. While suspended nothing ran: 81 cross-domain classification deadlines passed without the scoring run executing, seventeen daily referee fetches and two weekly aggregates were missed. On the first scheduled run after the host returned (2026-08-11 09:00) the unmodified job scored the backlog — 84 cross-domain and 8 single-domain classifications, 81 of them 1–13 days late — against a referee store that had been verified byte-identical to its pre-incident state. The operational-day clock paused for 18 days in total.

Incident 2 — boot-environment database fork, 2026-08-21 17:36 → 2026-08-30 19:22. An operating-system patch created a new boot environment; the next boot mounted it, and because the PostgreSQL data directory lived inside the boot-environment dataset, nine days of database writes landed in a forked copy that the reboot of 2026-08-30 silently reverted. The engine ran correctly the whole time against the database it could see. Lost from the live database: 109 published firings, 10 scored outcomes, 9 referee fetch runs, 470 candidate rows and the input-history rows of the window. The phantom state was preserved in three held copies, its rows were extracted forensically, and on 2026-09-06 the 109 firings and the 10 outcomes were restored in one governed transaction — new identifiers, original publication timestamps, provenance columns, canonical live outcomes untouched. The structural cause was eliminated on 2026-09-02. The clock did not pause: the engine operated throughout; its persistence layer failed.

Incident 1 — host suspend

Timeline by segment

One fl_downtime row per continuous offline period. Boundaries are last-observed / first-observed cron-log evidence at 15-minute tick granularity; the final end of the July window is the host boot record. Inside the wake windows the engine ran normally (firings were captured; the 2026-07-27 09:00 scoring run executed), so a single spanning row would have contradicted the record.

The July window was offline for about 17 of its 18.3 days. Segment 985 spanned 2026-07-29, the first classification deadline under Amendment 5 (firings 5966 and 5967).

Cause

The host — a personally-operated laptop that carried capture, scoring, rendering, alerting and backups — was suspended. The engine itself, its configuration and its data were unchanged.

Impact

• Classification timeliness. 81 post-anchor cross-domain classification deadlines dated 2026-07-29 → 2026-08-10 passed without the scoring run executing. No scoring run ever executed against a stale referee store: the freshness gate was never reached, and zero outcomes were manufactured during the incident. The deadline scanner alerted the breaches as soon as the host was awake — at the 2026-08-03 00:30 wake for the first ones (including 5966 and 5967) and at 17:30 on 2026-08-10 for the remaining overdue queue (74 cross-domain and 7 single-domain), with de-duplication holding across the two.
• Referee fetches. 17 scheduled 07:30 OFR fetches were missed in the July window (run 108 on 07-23 was the last before; run 109 executed during the 07-27 wake), and three more on 08-14/15/16 (run 113 → 114).
• Scheduled jobs. The 09:00 scoring heartbeats of the offline days; the 06:00 weekly aggregates of 08-03 and 08-10; nightly dumps of the offline nights (08-15 and 08-16 in the second window); the Friday 16:33 collections of 08-14.
• Detection. Fusion was not running while offline, so firings that would have been detected in those hours are unobservable by construction. The forward log records captures only; nothing is counted or estimated for the offline hours, and the downtime rows carry firings_lost = 0 for that reason.
• Second window (row 990). Zero classification deadlines breached: the only deadlines dated inside it (2026-08-15) belong to single-domain WATCH firings 6938 and 6939, which have no terminal-status pathway; the next scheduled scoring run (2026-08-17 09:00) executed on time.

Recovery and the late scoring

Same-day recovery on 2026-08-10, in order: the 17:30 scanner breach-alerted the overdue queue; a manual OFR fetch at 19:43 (run 110) brought the store current — 6,731 rows, 11 new, 6,720 verified, 0 revisions; batch scoring was deliberately left to the unmodified scheduled run of 2026-08-11 09:00, because the store was fresh, the selector picks up its backlog on its own, and the scoring windows are date-based, so late execution changes neither row selection nor window contents.

That run produced 92 classifications, all successful: 84 cross-domain (81 breached — 6 by one day, 8 by two, 8 by three, 31 by six, 7 by seven, 5 by eight, 6 by nine, 8 by ten, 2 by thirteen — plus 3 due that day), every one STRESS_NONE; and 8 single-domain (5 NORMALIZED, 3 SUSTAINED_REGIME). Six single-domain classifications ran on schedule inside the window's awake moments: three at the 09:00 run of 2026-07-23, immediately before the first suspend (firings 5348, 5350, 5351), and three at the 09:00 run of 2026-07-27 inside the wake window (5371, 5616, 5631). Each late row's classification timestamp stands against its deadline in the public CSV.

Why the late cross-domain outcomes are identical to on-time outcomes

The cross-domain scorer reads the OFR Financial Stress Index for a window fixed by dates (firing date to deadline). Late execution reads the same dates. Identity with on-time scoring therefore reduces to one question: did the referee values change between the day they would have been read and the day they were read?

• Values published before the incident: provably unchanged. The 6,720 rows stored before 2026-07-23 were re-verified byte-identical against the current OFR vintage on 2026-08-10 (run 110, 0 mismatches) and on every daily fetch since.
• The 11 dates first published during the offline window (observation dates 2026-07-23 → 2026-08-06): entered the store at the 2026-08-10 vintage. A revision to one of them between its publication and 2026-08-10 is unobservable by construction — the host was off. The claim for these dates rests on OFR's measured revision behaviour: in the life of the store, the revision sentinel has recorded exactly one event (23 values for 2026-05-15/18/21/22, detected 2026-07-13 between the 07-09 and 07-10 vintages) and zero drift in every fetch before and since, including runs 111–127 through 2026-09-08.

The 8 single-domain outcomes use closed price windows that are equally date-fixed, but one evidence field — the cumulative-return percentile, ranked against a trailing three-year history that advances with the run date — is run-date-dependent. The same effect appears in Incident 2, where firing 6943 was scored twice ten days apart: the percentile and volatility ratio differed, the category did not. No on-time reference scoring exists for the July single-domain rows, so their identity is stated as a reproduction of category, not of every evidence value.

Operational clock

Under FR-016, engine-offline periods of 24 hours or more pause the operational-day clock (UTC-date clipping). Segments 984, 985, 987, 988, 989 and 990 pause; segment 986 (12 h 15 m) does not. The July window contributed 16 pause-days (calendar day 97 → operational day 81 as of 2026-08-10), the August window 2 (operational day 105 → 103 as of 2026-09-03). As of this report: calendar day 126, operational day 108, 18 pause-days in total.

Incident 2 — boot-environment database fork

Mechanics

On 2026-08-21 an operating-system patch session (pkg upgrades including the PostgreSQL server 17.10 → 17.11, plus freebsd-update) created boot environment 15.0-RELEASE-p12_2026-08-21_150946. The 17:34 boot mounted that boot environment instead of default. The PostgreSQL data directory lived at /var/db/postgres, inside the boot-environment dataset, so from the moment the database server started (~17:36) every write landed in a copy invisible to default. The host was rebooted on 2026-08-30 at 19:24 into default, which silently reverted all persistence of the window to its 2026-08-21 state. Everything else — cron, logs on /home, backups on /home — was shared between the two boot environments and continued uninterrupted.

Discovery

2026-08-31, morning: the public CSV stood at 1,372 rows where the last render before the reboot (2026-08-30 19:15) had published 1,471. The reversion was traced the same morning to the boot records — 17:34 on 2026-08-21 into the patch-created boot environment, 19:24 on 2026-08-30 back into default — and to the data directory's location inside the boot-environment dataset. The nightly dumps of 2026-08-21 → 2026-08-30, taken on /home and therefore outside the fork, were secured first; the phantom boot environment was snapshotted, held and streamed to the evidence vault that evening.

Impact

Lost from the live database at the reboot (counts from the forensic extraction of the phantom's final state, 2026-08-30 19:22):

Public surface: the published CSV regressed from 1,471 to 1,363 rows at the 20:15 render of 2026-08-30 and stayed there until the 09:15 render of 2026-08-31 (about 13 hours). The primary-key sequences reverted with the data, so the live engine reissued identifiers from 7039 onwards to later, different events: of the 109 phantom numbers, 51 (7039–7075 and 7134–7147) are now live rows of other events captured 2026-08-30 → 2026-09-04, and 58 (7076–7133) were issued without ever becoming rows and are unused gaps below the sequence.

Engine conduct

The engine ran correctly for the whole window against the database it could see: detection, capture, scoring, rendering, alerting and backups all executed on schedule (cron log unbroken; first phantom tick 17:45:02 on 08-21, last 19:15:13 on 08-30). Ten firings whose deadlines fell inside the window were scored in the phantom and, after the reversion, scored again by the live engine on 2026-08-31: seven cross-domain and two geopolitics outcomes reproduced byte-identical evidence and category; firing 6943 (financial, NORMALIZED) reproduced its category with a recomputed cumulative-return percentile (20.17 → 16.27) and realised-volatility ratio (0.4966 → 0.5597), the run-date effect described above. The live scorings of 2026-08-31 are the canonical outcomes.

Evidence chain

Three held copies of the phantom's final state, all hashed:

1. the phantom boot environment itself, snapshot zroot/ROOT/15.0-RELEASE-p12_2026-08-21_150946@incident-2026-08-21-fork-evidence, made undeletable with zfs hold incident-evidence;
2. a full zfs send stream of that snapshot, 43,481,077,736 bytes (40.5 GiB), sha256 311b7d6d0ceb7601663eb7b72e91ddc6e9235b8d9adfb33506887f5ae7eea945, with a dry-run receive verified;
3. the USB backup replica taken on 2026-08-31 (backup_pool, snapshot @full-20260831_182631).

From the held snapshot a forensic dump of the phantom database at 2026-08-30 19:22 was extracted on 2026-09-05 on a writable clone with a temporary, socket-only database server (the snapshot itself was never mounted read-write): systemate_phantom_2026-08-30_1922.dump, sha256 7b4fea11dcca09cadd6f720ea2d0c91a8203565fd36ed9f4de2bb8370a14155e. The ten nightly dumps of 2026-08-21 → 2026-08-30 are held alongside it, all in a sha256 manifest.

Remediation

On 2026-09-02 the PostgreSQL data directory was moved to its own dataset, zroot/data/postgres, outside zroot/ROOT, in a paused window with the copy verified file-by-file (1,753 files sha256-identical, control data identical, row counts identical before and after). Boot-environment switches can no longer fork the database: whichever boot environment is active, the same dataset is mounted over the data directory. The fork class is eliminated.

Operational clock

This was not engine downtime. The engine operated continuously and correctly; its persistence layer failed. FR-016 pauses the clock for engine-offline time, and there was none, so row 991 carries pauses_operational_clock = FALSE and zero pause-days (a spanning pause would have removed nine). The precedent is row 674, a subsystem-degraded event logged the same way. Row 991 anchors the restore as its FR-022 downtime window.

The governed restore (2026-09-06)

Design rulings

• New identifiers. The phantom identifiers 7039–7147 had already been reissued to different live events, so the restored rows take fresh identifiers: a contiguous block after the current maximum, in phantom order — 7168–7276.
• Flag and provenance (migration 007). is_recovered_from_downtime = TRUE marks every restored row (its FR-022 meaning — brought back after a window without live coverage — fits exactly), and two new columns carry what the flag cannot: restored_from_phantom_firing_id, the identifier the row carried in the forked database, and restored_at. Both are set together or not at all, each phantom identifier can be restored at most once, and both are immutable under the append-only guard.
• Publication timestamps preserved verbatim. The phantom rows were real first publications: they were rendered to the public page and CSV at the time. The restored rows carry those timestamps unchanged; the renderer stamps only rows that have none.
• source_fusion_alert_id and source_fusion_score_id set to NULL. The phantom's fusion alerts and scores were reverted with the fork, and the live engine has since reissued those identifiers to unrelated events — a verbatim copy would have made 84 of the 109 rows point at a different event. The phantom values are retained in the published mapping.
• Twice-scored outcomes as non-current history. The 10 phantom outcomes were inserted with is_current = FALSE, each linked by an fl_corrections row (790–799) to the canonical live re-score of 2026-08-31, with the evidence comparison recorded in the note. The aggregates read current outcomes only, so nothing was double-counted and no published statistic moved.
• Document-only for the fetch-log runs (the live re-fetch of 2026-08-31 re-observed the same referee dates with zero revisions), the candidate rows and fusion alerts (audition inputs and reverted alerts), and the input-history rows. The input history is the detectors' baseline — the trailing series the thresholds are computed against — and is never altered mid-epoch; the window remains a gap in that history, recorded here rather than back-filled.

Execution

One transaction, executed on 2026-09-06 at 12:18 in a paused publication window after a dry run against the live database at 12:17 whose every assertion passed and which left no trace (Appendix D). The transaction locks the three tables, computes the identifier block after the lock (so a concurrent capture either precedes it and is inside the maximum, or waits), inserts the 109 firings and the 10 non-current outcomes with their 10 correction links, verifies its own result — row counts, a column-by-column round trip against the phantom rows, md5 digests of every pre-existing row, the ticker partition, the aggregate row set — and only then syncs the sequences and commits. An insurance dump was taken immediately before. The publication of the day comprised three write events in total: the migration, the restore transaction, and the sequence sync inside it.

Result on the public page: firings 1,433 → 1,542; informational +41, pending +68; classified, voided, the cross-domain scored count and both aggregate blocks unchanged; 109 rows marked RESTORED.

109 versus 108

Row 991 records firings_lost = 108. It was written on 2026-09-03 from the last nightly dump of the phantom (2026-08-30 02:10), with the row's own text declaring the figure a dump-counted floor pending the forensic extraction of the final 17 hours. The extraction found 109 — three more firings captured on 2026-08-30 between 02:10 and 18:15. The row is append-only and keeps its floor; this report and the mapping carry the count.

For holders of CSVs downloaded between 2026-08-21 and 2026-08-30

Identifiers 7039–7147 in a CSV from that period refer to the phantom rows. In the current export, 51 of those numbers (7039–7075 and 7134–7147) belong to different, later events captured 2026-08-30 → 2026-09-04, and 58 (7076–7133) do not appear at all; none of them is a restored row. The restored rows carry identifiers 7168–7276. The restored rows are the ones with restored_from_phantom_firing_id set; the phantom-to-current mapping is published alongside the current export as phantom-id-map.csv (sha256 c87b2f8beef4264179a309e6f3fadc73cb76ed00a8ac5fd9a5fbd86564522a2b), listing for each restored firing its phantom identifier, current identifier, domain, level, status, original publication timestamp, deadline and the phantom's alert and score identifiers, and for each restored outcome its phantom, restored and canonical identifiers. Always key on restored_from_phantom_firing_id; never on the old numbers.

Lessons

Both incidents were events of the host substrate — a lid and a patch — on a personally-operated laptop; neither was an event of the engine or of the methodology, and neither changed a scoring rule, a threshold or an outcome. The fork class is eliminated structurally: the database no longer lives where a boot-environment switch can reach it. The suspend class is a hosting question, under active review; no decision is claimed here.

Appendix A — downtime rows 984–991

Appendix B — commits

Identifiers shown here are the public values. The commits were made under the local identifiers recorded in the pre-publication map (docs/prepublication-hash-map.txt); content, timestamps and messages are unchanged.

Appendix C — evidence hashes

Appendix D — assertions of the live dry run (2026-09-06 12:17)

Each assertion is executed inside the restore transaction and aborts it on failure; the dry run rolled back, the execute run of 12:18 committed with identical results.