History links: a test that names missing entries, and a dev server that survives them

@@ -14,4 +14,5 @@ guides for every agent working on this repository; this skill only points to the
    `docs/research/expansion-<date>.md`.
 4. If you use subagents, follow section 4 of `docs/EXPAND-COUNTRIES.md`: each worker owns different
-   countries and returns its history entries as a `history-batch.cjs` file; only you edit
-   `changes.yaml`, build and commit.
+   countries and returns its history entries as a `history-batch.cjs` file, with its articles
+   drafted in `.cache/drafts/articles/`, not in `articles/`; only you edit `changes.yaml`, move the
+   articles in after their entries, run `npm test`, build and commit.

@@ -14,2 +14,5 @@ repository; this skill only points to it.
    context sections, culture and the country's words for cannabis, sources, then build and commit
    only your own files.
+   Add a history entry to `changes.yaml` before you write its article (`articles/<entry id>.md`) or
+   link to it; an article or link without its entry stops the build for everyone. Run `npm test`
+   before you finish: it names any that are missing.

@@ -22,4 +22,7 @@ cannabis law (`changes.yaml`, `articles/`) and a cultural atlas (`culture/`), bu
   `npm run build:quick` builds all pages without social cards. Builds lock automatically;
   if another build is running, wait and retry. Never delete another process's lock.
+- A history entry goes into `changes.yaml` before its article (`articles/<entry id>.md`) or any
+  link to `/history/<id>/`; otherwise the build stops for everyone. `npm test` names any that are
+  missing (the dev server only warns and leaves them out).
 - Both builds run `npm test` first (answers, profile claims, history, charts, research, survey; a
   few seconds) and stop if a test fails. Run `npm test` alone for a quick check. A new test file

@@ -69,6 +69,10 @@ Parallel work is fine if each agent owns its countries and one agent coordinates
   countries) and returns its new or corrected history entries as a `history-batch.cjs` JSON file
   instead of editing `changes.yaml`, plus a short list of what it found and could not verify.
-- The coordinator applies each worker's history file, builds once, checks the pages, and commits
-  each country separately.
+  Its articles wait with that file, outside `articles/` (in `.cache/drafts/articles/<entry id>.md`),
+  because an article whose entry is not yet in `changes.yaml` stops the build for everyone; the
+  same goes for profile text that links to a `/history/<id>/` entry not yet added.
+- The coordinator applies each worker's history file first, then moves its drafted articles into
+  `articles/` and runs `npm test`, builds once, checks the pages, and commits each country
+  separately, with the entries and their articles in the same commit.
 - Never let two workers have the same country, and never let a worker revert or reformat a file it
   does not own.

@@ -63,4 +63,10 @@ today:
 ## 4. Articles
 
+- **The history entry comes first.** An article is the long read of one entry: its file is
+  `articles/<entry id>.md`, and the entry must already be in `changes.yaml` when you save it.
+  The same holds for every `/history/<id>/` link and every atlas entry's `history:` list. Save an
+  article or a link before its entry (or rename an entry without renaming its article) and the
+  build stops for everyone working here. If you cannot add the entry yourself (a worker in a
+  batch, see EXPAND-COUNTRIES.md section 4), keep the draft out of `articles/` until it is in.
 - Pick from `article-backlog.cjs --country <id>`: score-moving entries and the current law first.
 - 350–500 words, three to five short sections, at most one small table (WRITING.md, "Articles").
@@ -132,4 +138,6 @@ node scripts/country-gaps.cjs ke   # the list should be shorter
 ```
 
+- Run `npm test` before you finish or hand over: it takes seconds, and
+  `scripts/history-links.test.cjs` names any article or link whose history entry is missing.
 - Include changed atlas entries' `culture/<section>/<slug>` paths in the review build too.
 - Look at `.cache/page-review/country/<id>/index.html` and the changed entries for the sections,

@@ -6,5 +6,5 @@
   "scripts": {
     "dev": "astro dev",
-    "test": "node --test scripts/answers.test.mjs scripts/profile-claims.test.mjs scripts/use-freedom-ratio.test.mjs scripts/history-scope.test.cjs scripts/context-coverage.test.mjs scripts/chart-semantics.test.cjs scripts/survey.test.mjs scripts/books-status.test.cjs",
+    "test": "node --test scripts/history-links.test.cjs scripts/answers.test.mjs scripts/profile-claims.test.mjs scripts/use-freedom-ratio.test.mjs scripts/history-scope.test.cjs scripts/context-coverage.test.mjs scripts/chart-semantics.test.cjs scripts/survey.test.mjs scripts/books-status.test.cjs",
     "prebuild": "npm test",
     "build": "astro build",

@@ -0,0 +1,38 @@
+// Every reference to a history entry must find it in changes.yaml: an article (articles/<id>.md is
+// the long read of entry <id>), a culture entry's `history` and `history_cards`, and every link to
+// /history/<id>/ in a profile, article, culture entry or page introduction. A build stops on a
+// broken one; this test names them all at once, in a second, before the build starts.
+// The usual cause is order: write the history entry in changes.yaml first, then the article or
+// the link (docs/EXPAND-COUNTRY.md, "Articles").
+const { test } = require("node:test");
+const assert = require("node:assert/strict");
+const fs = require("node:fs");
+const path = require("node:path");
+const y = require("js-yaml");
+
+const ids = new Set([...fs.readFileSync("changes.yaml", "utf8").matchAll(/^- id: ([^\s#]+)/gm)].map((m) => m[1]));
+// the history pages that are not entries
+const PAGES = new Set(["period", "kind", "tag"]);
+const walk = (dir, ext) => fs.existsSync(dir) ? fs.readdirSync(dir, { withFileTypes: true }).flatMap((e) => e.isDirectory() ? walk(path.join(dir, e.name), ext) : e.name.endsWith(ext) ? [path.join(dir, e.name)] : []) : [];
+const frontMatter = (file) => { const m = /^---\n([\s\S]*?)\n---/.exec(fs.readFileSync(file, "utf8")); return m ? y.load(m[1]) ?? {} : {}; };
+const FIX = "Write the history entry in changes.yaml first, or correct the id; an article's file name is its entry's id.";
+
+test("every article belongs to a history entry", () => {
+  const orphans = walk("articles", ".md").map((f) => path.basename(f, ".md")).filter((id) => !ids.has(id));
+  assert.deepEqual(orphans.map((id) => `articles/${id}.md`), [], `No history entry for these articles. ${FIX}`);
+});
+
+test("culture entries list only history entries that exist", () => {
+  const missing = walk("culture", ".md").flatMap((f) => {
+    const d = frontMatter(f);
+    return [...(d.history ?? []), ...(d.history_cards ?? [])].filter((id) => !ids.has(id)).map((id) => `${f}: ${id}`);
+  });
+  assert.deepEqual([...new Set(missing)], [], `Culture entries name history entries that do not exist. ${FIX}`);
+});
+
+test("every link to /history/<id>/ leads to an entry", () => {
+  const files = [...walk("countries", ".yaml"), ...walk("articles", ".md"), ...walk("culture", ".md"), ...walk("context", ".md"), ...walk("datasets/intros", ".md")];
+  const missing = files.flatMap((f) => [...fs.readFileSync(f, "utf8").matchAll(/(?<![\w.])\/history\/([a-z0-9][a-z0-9-]*)\/?(?=[)#"'\s]|$)/g)]
+    .map((m) => m[1]).filter((id) => !PAGES.has(id) && !ids.has(id)).map((id) => `${f}: /history/${id}/`));
+  assert.deepEqual([...new Set(missing)], [], `Links to history entries that do not exist. ${FIX}`);
+});

@@ -2,4 +2,5 @@
 // pages need: index scores, ranks, averages. Runs once at build time.
 import { getCollection } from "astro:content";
+import { broken } from "./integrity.js";
 import { existsSync } from "node:fs";
 import { CHANGE_CATEGORIES, CHANGE_TAGS, SEGMENT_IDS, STATUS_SECTIONS } from "./segment-ids.js";
@@ -318,5 +319,5 @@ export const changes = changeEntries
 // Long reads: articles/<change id>.md. Every article must belong to a history entry.
 export const articles = new Map((await getCollection("articles")).map((a) => [a.id, a]));
-for (const id of articles.keys()) if (!changes.some((ch) => ch.id === id)) throw new Error(`articles/${id}.md: there is no history entry "${id}" in changes.yaml`);
+for (const id of articles.keys()) if (!changes.some((ch) => ch.id === id)) { broken(`articles/${id}.md: there is no history entry "${id}" in changes.yaml`); articles.delete(id); }
 
 // A country's population at a moment in time, on a straight line between its snapshots.

@@ -3,4 +3,5 @@
 // entries it touches; both must exist, so a typing mistake stops the build.
 import { getCollection } from "astro:content";
+import { broken } from "./integrity.js";
 import { places, changes } from "./countries.js";
 import { sourceTuple } from "./sources.js";
@@ -31,23 +32,23 @@ export const cultureEntries = (await getCollection("culture"))
     const history = d.history.map((id) => {
       const ch = changes.find((x) => x.id === id);
-      if (!ch) throw new Error(`${file}: there is no history entry "${id}"`);
+      if (!ch) broken(`${file}: there is no history entry "${id}"`);
       return ch;
-    }).sort((a, b) => a.date.localeCompare(b.date));
+    }).filter(Boolean).sort((a, b) => a.date.localeCompare(b.date));
     const historyCards = d.history_cards.map((id) => {
       const ch = changes.find((x) => x.id === id);
-      if (!ch) throw new Error(`${file}: history_cards names an unknown history entry "${id}"`);
+      if (!ch) { broken(`${file}: history_cards names an unknown history entry "${id}"`); return null; }
       if (!d.history.includes(id)) throw new Error(`${file}: history_cards entry "${id}" must also be in history`);
       return ch;
-    });
+    }).filter(Boolean);
     // every history entry the text relies on is linked in the text, and every history entry the
     // text links to is listed in `history`, so the entry, its history list and the history pages
     // stay interlinked (docs/CULTURE.md, Countries and links)
     const linked = new Set([...(entry.body ?? "").matchAll(/\]\(\/history\/([^/)#]+)/g)].map((m) => m[1]));
-    for (const id of d.history) if (!linked.has(id)) throw new Error(`${file}: "${id}" is in history but the text never links to /history/${id}/; mention and link it in the text`);
-    for (const id of linked) if (!d.history.includes(id)) throw new Error(`${file}: the text links to /history/${id}/ but "${id}" is not in history`);
+    for (const id of d.history) if (!linked.has(id)) broken(`${file}: "${id}" is in history but the text never links to /history/${id}/; mention and link it in the text`);
+    for (const id of linked) if (!d.history.includes(id)) broken(`${file}: the text links to /history/${id}/ but "${id}" is not in history`);
     // links in the text to pages of this site must lead somewhere
     for (const [, kind, id] of (entry.body ?? "").matchAll(/\]\(\/(history|country)\/([^/)#]+)\/?[^)]*\)/g)) {
       const found = kind === "history" ? changes.some((x) => x.id === id) : places.some((x) => x.slug === id);
-      if (!found) throw new Error(`${file}: the text links to /${kind}/${id}/, which does not exist`);
+      if (!found) broken(`${file}: the text links to /${kind}/${id}/, which does not exist`);
     }
     return { entry, section, slug, url: `culture/${section}/${slug}/`, ...d, countries, history, historyCards, sources: d.sources.map(sourceTuple) };
@@ -57,5 +58,5 @@ export const cultureEntries = (await getCollection("culture"))
 // and so must links from one entry to another
 for (const e of cultureEntries) for (const [, path] of (e.entry.body ?? "").matchAll(/\]\(\/(culture\/(?:language|traditions|people)\/[^/)#]+)\/?[^)]*\)/g)) {
-  if (!cultureEntries.some((x) => x.url === path + "/")) throw new Error(`culture/${e.entry.id}.md: the text links to /${path}/, which does not exist`);
+  if (!cultureEntries.some((x) => x.url === path + "/")) broken(`culture/${e.entry.id}.md: the text links to /${path}/, which does not exist`);
 }
 // Links in a country profile's segment and context texts to a history entry or a culture entry
@@ -66,6 +67,6 @@ for (const c of places) {
   for (const b of blocks) for (const [, , href] of (b?.cite?.text ?? "").matchAll(LINK_MARK)) {
     const history = /^\/history\/([^/#]+)\/?$/.exec(href), culture = /^\/(culture\/(?:language|traditions|people)\/[^/#]+)\/?$/.exec(href);
-    if (history && !historyIds.has(history[1])) throw new Error(`countries/${c.slug}.yaml: a text links to ${href}, but there is no history entry "${history[1]}"`);
-    if (culture && !cultureEntries.some((x) => x.url === culture[1] + "/")) throw new Error(`countries/${c.slug}.yaml: a text links to ${href}, which does not exist`);
+    if (history && !historyIds.has(history[1])) broken(`countries/${c.slug}.yaml: a text links to ${href}, but there is no history entry "${history[1]}"`);
+    if (culture && !cultureEntries.some((x) => x.url === culture[1] + "/")) broken(`countries/${c.slug}.yaml: a text links to ${href}, which does not exist`);
   }
 }

@@ -0,0 +1,9 @@
+// A broken reference between the data files (an article without its history entry, a link to an
+// entry that is not there). A build stops on it, and `npm test` names it first
+// (scripts/history-links.test.cjs). The dev server only warns and leaves the reference out, so
+// that one agent's half-saved work does not take the site down for everyone else working here.
+const warned = new Set();
+export function broken(message) {
+  if (!import.meta.env.DEV) throw new Error(message);
+  if (!warned.has(message)) { warned.add(message); console.warn(`[data] ${message} (left out in dev; a build stops here)`); }
+}