ODA

Operations

Incidents

Blameless postmortems from a personal production stack. What broke, how it was found, what changed.

When something breaks in Oda, it is written down. Each one follows the same shape: what happened, how it was found, the root cause, and what changed so it does not happen again.

Fabricated quotes in a published Yūkan edition

When: 2026-09-23.

The evening edition shipped with quotes from two people who do not appear to exist: a union spokesperson and an industry analyst, both invented by the writing model. The paper checked links and images, but nothing checked quotes. I found it reviewing our own published output.

Fix: a quote gate now runs at the editor desk before assembly. Every quote needs a named speaker and a URL fetched that run, or the quote is stripped or paraphrased. The corrected edition replaced the original on the front page and in the archive on Sept. 24. The gate runs on every edition from then on.

Yūkan desk and publish interrupts

When: 2026-09-14 through 2026-09-22 (seven interrupt alerts).

Editorial gates rejected drafts mid-run, and the pipeline stopped instead of recovering. Each stop raised an alert, and recovery playbooks resumed the run and waited for a human before publishing. The pipeline was hardened to rewrite and recover instead of quitting when a gate refuses. Interrupt count is published next to edition count as monitoring evidence.

Arc B580 hang and RAM collapse

When: 2026-09-22 / 23.

Arc memory-manager workers hung; free RAM on a 30 GiB box fell to about 1.6 GiB while process RSS explained only about 5 GiB. Mixed 100 Hz / 60 Hz refresh contributed. Fix: stop Desktop, lock both monitors to 60 Hz, retire the extra 8 GiB swapfile, return to a single 2 GiB swap, swappiness 10.

Grok Bot silent software-rendering fallback

When: found during the same stability pass.

The desktop client turned acceleration off when hardwareAccelerationEnabled was missing from settings — before Chromium flags applied. Fixed with a launcher that pins the key and runs on the Arc with ANGLE.

What changed

These incidents became standing rules: the main model is never restarted casually; any recovery that ends in publishing waits for a human; only the main model runs on the RTX 5080; swap stays small; and the Arc-driven displays run at 60 Hz.

See the operating policy