Guide for expanding several countries, with a ranking of what most needs work

@@ -0,0 +1,17 @@
+---
+name: expand-countries
+description: Expand several countries of the Cannabis Freedom Index in one batch - pick the countries that most need work (or a given region or list) and take each through the expand-country workflow, committing country by country. Use when asked to expand, deepen or research multiple countries, a region, or "the next batch" (e.g. "expand five African countries", "/expand-countries Africa 5").
+---
+
+Follow `docs/EXPAND-COUNTRIES.md`, and for each country `docs/EXPAND-COUNTRY.md`. They are the shared
+guides for every agent working on this repository; this skill only points to them.
+
+1. Read both guides.
+2. Choose the batch: `node scripts/country-gaps.cjs --rank 25` (add `--region <Region>` when a region
+   is given), or use the countries named in the arguments. Skip any marked BUSY.
+3. Take the countries one at a time through `docs/EXPAND-COUNTRY.md`, building and committing each
+   country (only your own files) before the next, and keep the batch's progress note in
+   `docs/research/expansion-<date>.md`.
+4. If you use subagents, follow section 4 of `docs/EXPAND-COUNTRIES.md`: each worker owns different
+   countries and returns its history entries as a `history-batch.cjs` file; only you edit
+   `changes.yaml`, build and commit.

@@ -6,4 +6,6 @@ cannabis law (`changes.yaml`, `articles/`) and a cultural atlas (`culture/`), bu
 - **To expand or research a country**, follow `docs/EXPAND-COUNTRY.md`. Start with
   `node scripts/country-gaps.cjs <id>`.
+- **To work through several countries**, follow `docs/EXPAND-COUNTRIES.md`. Start with
+  `node scripts/country-gaps.cjs --rank 25` (optionally `--region Africa`).
 - Data rules: `docs/DATA-MAINTENANCE.md`. How texts read: `docs/WRITING.md`. Atlas entries:
   `docs/CULTURE.md`.

@@ -936,5 +936,7 @@ node scripts/data-status.cjs chain de     # one country's full timeline
 
 To expand one country end to end (history, articles, context sections, penalty tables, links),
-follow `docs/EXPAND-COUNTRY.md`; `node scripts/country-gaps.cjs <id>` prints its work list.
+follow `docs/EXPAND-COUNTRY.md`; `node scripts/country-gaps.cjs <id>` prints its work list. For a
+batch of countries, follow `docs/EXPAND-COUNTRIES.md`; `node scripts/country-gaps.cjs --rank` ranks
+them by how much is missing.
 
 1. **Pick countries.** `node scripts/data-status.cjs thin` lists places with two or fewer

@@ -0,0 +1,104 @@
+# Expanding several countries
+
+How to work through a batch of countries, one after another or with several agents at once. Each
+country is done exactly as `docs/EXPAND-COUNTRY.md` describes; this file covers choosing the batch,
+the order, sharing the work and keeping it safe while others work in the same repository.
+
+## 1. Choose the batch
+
+```bash
+node scripts/country-gaps.cjs --rank 25                 # the countries that most need work
+node scripts/country-gaps.cjs --rank 25 --region Africa # or within one region
+```
+
+The ranking weighs each country's gaps (missing sections, penalty tables, empty history decades,
+Wikipedia-only texts, key entries without an article) by population. Take the batch from it, from
+a region, or from a list you were given.
+
+- **Five to ten countries per batch.** A batch is finished country by country; a half-done batch
+  of thirty helps nobody.
+- **Skip countries marked BUSY**: someone has uncommitted changes in their files right now. Run
+  the ranking again before each new country, since others move on too.
+- Neighbours and countries with a shared history (a colonial power and its colonies, one region's
+  treaties) go well together: sources found for one often cover the next.
+
+## 2. Work country by country
+
+For each country in the batch:
+
+1. `git status` and `git log --oneline -5`: is anyone working on this country now? If so, skip it.
+2. `node scripts/country-gaps.cjs <id>`: the country's work list.
+3. Work through `docs/EXPAND-COUNTRY.md` in its order: current law and penalty tables, history,
+   articles, context sections, culture, links.
+4. Build (`npm run build`; wait if another build is running), then read the built page in
+   `dist/country/<id>/`.
+5. **Commit that country before starting the next**, with only your own files, and a message that
+   names the country and what changed ("Morocco: penalty tables, hemp and industry sections, 4
+   history entries, 1980s Rif eradication article").
+6. Add a line to the batch's progress note (below).
+
+Finishing and committing each country means the work survives if the batch is interrupted, and
+others see at once which countries are done.
+
+## 3. Shared files
+
+Most work touches only the country's own files (`countries/<id>.yaml`, its articles, its atlas
+entries). A few files are shared by every country, and two agents editing them at once is where
+work gets lost:
+
+| Shared file | How to treat it |
+|---|---|
+| `changes.yaml` | Add entries with `node scripts/history-batch.cjs batch.json` rather than editing by hand, and commit soon after. Never stage someone else's entries. |
+| `glossary.yaml` | Add words at the end of the right group; commit with the country. |
+| `culture/*` atlas entries for several countries | Edit only the lines you need; check `git status` first. |
+| `docs/DATA-MAINTENANCE.md`, `docs/WRITING.md` | Only when a rule really changes; not for progress notes. |
+
+If a shared file already holds someone else's uncommitted changes, stage only your own lines
+(`git add -p`, or write the file with only your changes into the index), and say so in your report.
+
+## 4. Several agents at once
+
+Parallel work is fine if each agent owns its countries and one agent coordinates:
+
+- **The coordinator** picks the batch, gives each worker a different set of countries, and is the
+  only one who edits `changes.yaml` and runs builds.
+- **Each worker** edits only its own countries' files (profiles, articles, atlas entries for those
+  countries) and returns its new or corrected history entries as a `history-batch.cjs` JSON file
+  instead of editing `changes.yaml`, plus a short list of what it found and could not verify.
+- The coordinator applies each worker's history file, builds once, checks the pages, and commits
+  each country separately.
+- Never let two workers have the same country, and never let a worker revert or reformat a file it
+  does not own.
+
+## 5. Progress note
+
+Keep one note per batch in `docs/research/expansion-<date>.md`:
+
+```markdown
+# Expansion batch, 28 October 2026
+
+Chosen from `country-gaps.cjs --rank 25 --region Africa`.
+
+| Country | Done | Left | Commit |
+|---|---|---|---|
+| Morocco | penalty tables, hemp, industry, 4 history entries, 1 article | 1950s still thin; no source for 2019 arrest figures | abc1234 |
+| Senegal | skipped: busy (another agent) | | |
+```
+
+Update it after each country and commit it with that country. It tells the next agent where to pick up.
+
+## 6. When to stop or skip
+
+- **The build fails because of someone else's unfinished work**: don't fix or revert their files.
+  Finish your country's files, check them in the built page if the build gets far enough, note the
+  blocker, and move on. If a committed file is broken and you were asked to keep the build working,
+  make the smallest fix and say what you changed.
+- **A country has almost no reachable sources**: write what the sources support, note the gaps in
+  the progress note, and move on; never fill gaps with guesses or generic text.
+- **Quality before count.** Three countries done properly beat ten with thin sections.
+
+## 7. Report
+
+At the end, report per country: what was added (sections, tables, entries, articles, atlas
+entries), what was corrected (scores, statuses, facts), what could not be verified, and what is
+left; plus the commits and anything that blocked you.

@@ -5,4 +5,6 @@ penalty tables and links. Written for any agent or person working on the data. T
 live in the other guides; this file is the order of work and the rules that are easy to miss.
 
+For a batch of several countries, or several agents at once, see `docs/EXPAND-COUNTRIES.md`.
+
 Read first, once: `docs/DATA-MAINTENANCE.md` (data, sources, history entries, context sections,
 penalty steps), `docs/WRITING.md` (how every text reads), and `docs/CULTURE.md` if you write an

@@ -1,89 +1,172 @@
-// What is still missing for one country, as a work list for expanding it (.claude/skills/expand-country).
+// What is still missing for a country, as a work list for expanding it (docs/EXPAND-COUNTRY.md),
+// or a ranking of all countries by how much is missing (docs/EXPAND-COUNTRIES.md).
 //
-//   node scripts/country-gaps.cjs ke          Kenya
-//   node scripts/country-gaps.cjs us/ca       a state or province file
+//   node scripts/country-gaps.cjs ke                      one country's work list
+//   node scripts/country-gaps.cjs us/ca                   a state or province file
+//   node scripts/country-gaps.cjs --rank                  the 25 countries that most need work
+//   node scripts/country-gaps.cjs --rank 50 --region Africa
 //
-// Reads the files directly; run from the project root. Lists the profile's missing or thin context
-// sections and statuses, penalty tables, texts without inline citations, sources that are only
-// Wikipedia, the history by decade with its gaps, entries without a check or an article, and the
-// atlas entries that belong to the country.
-const fs = require("fs"), path = require("path"), y = require("js-yaml");
+// Reads the files directly; run from the project root. The work list names the profile's missing or
+// thin context sections and statuses, penalty tables, texts without inline citations, texts resting
+// only on Wikipedia, the history by decade with its gaps, entries without a check or an article, and
+// the atlas entries that belong to the country. The ranking weighs the same gaps by population and
+// marks countries whose files someone has uncommitted changes in ("busy"), which should be skipped.
+const fs = require("fs"), path = require("path"), y = require("js-yaml"), { execFileSync } = require("child_process");
 
-const id = process.argv[2];
-if (!id) { console.error("usage: node scripts/country-gaps.cjs <country file name, e.g. ke or us/ca>"); process.exit(1); }
-const file = fs.existsSync(`countries/${id}.yaml`) ? `countries/${id}.yaml` : `countries/${id}/index.yaml`;
-if (!fs.existsSync(file)) { console.error(`no profile at countries/${id}.yaml or countries/${id}/index.yaml`); process.exit(1); }
-const p = y.load(fs.readFileSync(file, "utf8"));
-const slug = id.split("/")[0];
-const ctx = p.context ?? {};
-const out = (s = "") => console.log(s);
-const list = (label, items) => out(`  ${label}: ${items.length ? items.join(", ") : "none"}`);
-
-out(`${p.name} (${file}), reviewed ${String(p.updated ?? "?").slice(0, 10)}, recreational ${p.recreational}, medical ${p.medical}`);
-out();
-
-// ── the profile ──
-out("PROFILE");
-// a state or province file holds its scores and segment texts only; the context is the country's
-const unit = id.includes("/") && !file.endsWith("index.yaml");
-if (unit) out(`  a state or province file: context sections live in countries/${slug}/index.yaml`);
 const REQUIRED = ["recreational", "medical", "hemp", "production", "consequences"];
 const RESEARCHED = ["clubs", "planting", "visitors", "driving", "black_market", "arrests"];
 const OPTIONAL = ["culture", "research", "spotlights", "genetics"];
-if (!unit) list("missing standard sections", REQUIRED.filter((k) => !ctx[k]));
-if (!unit) list("missing researched sections", RESEARCHED.filter((k) => !ctx[k]));
-if (!unit) list("optional sections not written", OPTIONAL.filter((k) => !ctx[k]));
-if (p.recreational === "death" && !ctx.death_penalty) out("  ! recreational is \"death\" but there is no death_penalty section");
-if (!unit && ctx.products) out("  products still present: fold it into recreational, medical and hemp, then delete it");
-list("statuses not set", ["hemp", "production", "research"].filter((k) => ctx[k] && !ctx[k].status).map((k) => `${k}.status`));
-if (!unit) list("penalty tables missing", ["possession", "cultivation"].filter((k) => p.segments?.[k] && !p.segments[k].penalties));
-const blocks = [...Object.entries(p.segments ?? {}).map(([k, b]) => [`segments.${k}`, b]),
-  ...Object.entries(ctx).flatMap(([k, b]) => (Array.isArray(b) ? b.map((x, i) => [`context.${k}[${i}]`, x]) : [[`context.${k}`, b]]))];
-list("texts without inline citations", blocks.filter(([, b]) => b?.text && !/\[@/.test(b.text)).map(([k]) => k));
-list("texts with no source at all", blocks.filter(([, b]) => b?.text && !(b.sources ?? []).length).map(([k]) => k));
-const wiki = (key) => /wikipedia/i.test(p.sources?.[key]?.title ?? "") || /wikipedia\.org/.test(p.sources?.[key]?.url ?? "");
-list("texts resting only on Wikipedia", blocks.filter(([, b]) => (b?.sources ?? []).length && b.sources.every(wiki)).map(([k]) => k));
-const linked = blocks.filter(([, b]) => /\]\(\//.test(b?.text ?? "")).length;
-out(`  internal links in texts: ${linked} of ${blocks.length} texts link to history or culture`);
-out();
+const fileOf = (id) => (fs.existsSync(`countries/${id}.yaml`) ? `countries/${id}.yaml` : `countries/${id}/index.yaml`);
 
-// ── the history ──
-out("HISTORY");
-const changes = (y.load(fs.readFileSync("changes.yaml", "utf8")) ?? []).filter((e) => e.country === slug);
-const articles = new Set(fs.readdirSync("articles").map((f) => f.replace(/\.md$/, "")));
+// loaded once: the history, the articles, the atlas entries by country, the populations
+const CHANGES = y.load(fs.readFileSync("changes.yaml", "utf8")) ?? [];
+const ARTICLES = new Set(fs.readdirSync("articles").map((f) => f.replace(/\.md$/, "")));
+const ATLAS = {};
+for (const section of ["language", "traditions", "people"]) {
+  for (const f of fs.readdirSync(path.join("culture", section)).filter((x) => x.endsWith(".md"))) {
+    const fm = y.load(fs.readFileSync(path.join("culture", section, f), "utf8").match(/^---\n([\s\S]*?)\n---/)?.[1] ?? "") ?? {};
+    for (const c of fm.countries ?? []) (ATLAS[c] ??= []).push(`${section}/${f.slice(0, -3)}`);
+  }
+}
+const POP = {};
+for (const line of fs.readFileSync("datasets/population.csv", "utf8").trim().split("\n").slice(1)) {
+  const [c, yr, n] = line.split(",");
+  if (!POP[c] || +yr > POP[c][0]) POP[c] = [+yr, +n];
+}
 const year = (e) => parseInt(String(e.date), 10);
-const dated = changes.filter((e) => year(e)).sort((a, b) => year(a) - year(b));
-out(`  ${changes.length} entries, ${changes.filter((e) => e.before !== undefined && e.before !== null).length} that move the score, ${changes.filter((e) => articles.has(e.id)).length} with an article`);
-if (dated.length) {
-  const first = year(dated[0]), last = year(dated.at(-1));
-  out(`  from ${first} to ${last}`);
+const THIS_YEAR = new Date().getFullYear();
+
+// Everything missing for one profile file
+function gapsOf(id) {
+  const file = fileOf(id);
+  const p = y.load(fs.readFileSync(file, "utf8"));
+  const slug = id.split("/")[0], ctx = p.context ?? {};
+  // a state or province file holds its scores and segment texts only; the context is the country's
+  const unit = id.includes("/") && !file.endsWith("index.yaml");
+  const blocks = [...Object.entries(p.segments ?? {}).map(([k, b]) => [`segments.${k}`, b]),
+    ...Object.entries(ctx).flatMap(([k, b]) => (Array.isArray(b) ? b.map((x, i) => [`context.${k}[${i}]`, x]) : [[`context.${k}`, b]]))];
+  const wiki = (key) => /wikipedia/i.test(p.sources?.[key]?.title ?? "") || /wikipedia\.org/.test(p.sources?.[key]?.url ?? "");
+  const changes = CHANGES.filter((e) => e.country === slug && (!unit || e.subunit === id.split("/")[1] || !e.subunit));
+  const dated = changes.filter((e) => year(e)).sort((a, b) => year(a) - year(b));
   const decades = {};
   for (const e of dated) decades[Math.floor(year(e) / 10) * 10] = (decades[Math.floor(year(e) / 10) * 10] ?? 0) + 1;
-  const row = [];
-  for (let d = Math.min(1900, Math.floor(first / 10) * 10); d <= 2020; d += 10) row.push(`${d}s ${decades[d] ?? "—"}`);
-  out(`  by decade: ${row.join(" · ")}`);
+  const first = dated.length ? year(dated[0]) : null, last = dated.length ? year(dated.at(-1)) : null;
   const empty = [];
-  for (let d = Math.max(1920, Math.floor(first / 10) * 10); d <= 2020; d += 10) if (!decades[d]) empty.push(`${d}s`);
-  list("decades with nothing (look for laws, cases, raids, reports)", empty);
-  const thisYear = new Date().getFullYear();
-  if (last < thisYear - 1) out(`  ! nothing since ${last}: check recent news and law changes`);
+  if (first) for (let d = Math.max(1920, Math.floor(first / 10) * 10); d <= 2020; d += 10) if (!decades[d]) empty.push(`${d}s`);
+  const atlas = ATLAS[slug] ?? [];
+  const scored = (e) => e.before !== undefined && e.before !== null;
+  return {
+    id, file, unit, name: p.name, region: p.region, updated: String(p.updated ?? "?").slice(0, 10), rec: p.recreational, med: p.medical,
+    missingRequired: unit ? [] : REQUIRED.filter((k) => !ctx[k]),
+    missingResearched: unit ? [] : RESEARCHED.filter((k) => !ctx[k]),
+    missingOptional: unit ? [] : OPTIONAL.filter((k) => !ctx[k]),
+    deathWithoutSection: p.recreational === "death" && !ctx.death_penalty,
+    products: !unit && Boolean(ctx.products),
+    statuses: ["hemp", "production", "research"].filter((k) => ctx[k] && !ctx[k].status).map((k) => `${k}.status`),
+    penalties: unit ? [] : ["possession", "cultivation"].filter((k) => p.segments?.[k] && !p.segments[k].penalties),
+    uncited: blocks.filter(([, b]) => b?.text && !/\[@/.test(b.text)).map(([k]) => k),
+    unsourced: blocks.filter(([, b]) => b?.text && !(b.sources ?? []).length).map(([k]) => k),
+    wikiOnly: blocks.filter(([, b]) => (b?.sources ?? []).length && b.sources.every(wiki)).map(([k]) => k),
+    linked: blocks.filter(([, b]) => /\]\(\//.test(b?.text ?? "")).length, texts: blocks.length,
+    entries: changes.length, scoredEntries: changes.filter(scored).length, withArticle: changes.filter((e) => ARTICLES.has(e.id)).length,
+    first, last, decades, empty, stale: last !== null && last < THIS_YEAR - 1,
+    unchecked: changes.filter((e) => !e.checked && !e.estimate).map((e) => e.id),
+    estimates: changes.filter((e) => e.estimate).map((e) => e.id),
+    noArticle: changes.filter((e) => !ARTICLES.has(e.id) && scored(e)).map((e) => e.id),
+    atlas, unlinkedAtlas: ctx.culture ? atlas.filter((c) => !ctx.culture.text.includes(`/culture/${c}/`)) : [],
+    atlasWithoutCulture: !ctx.culture && atlas.length > 0,
+  };
 }
-list("not checked against a source", changes.filter((e) => !e.checked && !e.estimate).map((e) => e.id));
-list("estimates (no source yet)", changes.filter((e) => e.estimate).map((e) => e.id));
-const noArticle = changes.filter((e) => !articles.has(e.id) && e.before !== undefined && e.before !== null).map((e) => e.id);
-list("score-moving entries without an article", noArticle);
-out("  more article candidates, ranked: node scripts/article-backlog.cjs --country " + slug);
-out();
 
-// ── culture ──
-out("CULTURE");
-const culture = [];
-for (const section of ["language", "traditions", "people"]) {
-  for (const f of fs.readdirSync(path.join("culture", section)).filter((x) => x.endsWith(".md"))) {
-    const fm = y.load(fs.readFileSync(path.join("culture", section, f), "utf8").match(/^---\n([\s\S]*?)\n---/)?.[1] ?? "") ?? {};
-    if ((fm.countries ?? []).includes(slug)) culture.push(`${section}/${f.slice(0, -3)}`);
+// How much work a country still needs, before weighing by population
+const workOf = (g) => g.missingRequired.length * 4 + g.missingResearched.length * 1.5 + (g.products ? 2 : 0) + g.statuses.length
+  + g.penalties.length * 2 + g.uncited.length * 0.3 + g.unsourced.length * 2 + g.wikiOnly.length + (g.deathWithoutSection ? 4 : 0)
+  + g.empty.length * 1.5 + (g.stale ? 3 : 0) + (g.entries < 5 ? 5 - g.entries : 0) + g.unchecked.length + g.estimates.length * 2
+  + g.noArticle.length * 1.5 + (g.atlasWithoutCulture ? 2 : 0) + g.unlinkedAtlas.length * 0.5;
+
+// files someone has uncommitted changes in: those countries are being worked on now
+function busy() {
… 88 more lines