#186 fix GDELT global news denominator labels

@@ -1638,19 +1638,23 @@
   publisher: The GDELT Project
   credit: GDELT
-  title: The GDELT Project — DOC 2.0 API, share of online news articles mentioning cannabis or marijuana, by the country of the source
+  title: The GDELT Project — DOC 2.0 API, cannabis articles from each source country as a share of global coverage
   url: https://www.gdeltproject.org/
   data_url: https://api.gdeltproject.org/api/v2/doc/doc?query=(cannabis%20OR%20marijuana)%20sourcecountry:germany&mode=timelinevol&format=json&startdatetime=20170101000000
   data_label: DOC 2.0 API, Germany as an example
-  when: 2017 to 2026
+  when: 2017 to 2025
   about: >-
-    Of all the online news articles GDELT monitors from a country's own news sites, the share
-    that mention cannabis or marijuana, averaged over each year; GDELT translates 65 languages
-    to find the words. A measure of attention: how much a country's press talks about cannabis,
-    whatever it says. Countries with few monitored sites give noisier figures.
+    Cannabis or marijuana articles from each source country as a percentage of all global news
+    monitored by GDELT, averaged equally over returned daily percentages. The denominator stays
+    global when a source-country filter is applied. This measures contribution to global indexed
+    coverage, not the share of that country's news about cannabis. Uneven outlet coverage,
+    national news output and the global denominator affect comparisons.
   origin: >-
     Downloaded on 22 October 2026 by scripts/fetch-open-data.py (news), one request per country
-    by name; `days` is how many days of a year GDELT returned.
+    by name using TimelineVol; `days` is how many daily observations GDELT returned. Values
+    are arithmetic means of daily global-coverage percentages, not ratios of annual article
+    totals. Denominator corrected in the descriptions on 27 October 2026; stored values unchanged.
+    Method: https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/
   profile:
-    headline: "{share} of news articles mention cannabis"
+    headline: "{share} of global news: cannabis articles from here"
     details: "{year} average, {days} days"
     each: "{share} in {year}"
@@ -1659,5 +1663,5 @@
   year: { column: year }
   fields:
-    share: { column: share, format: percent, about: "Share of the country's news articles that mention cannabis: map, Charts page and profile" }
+    share: { column: share, format: percent, about: "Daily global-coverage share of cannabis articles from this source country, averaged by year" }
     days: { column: days, about: "Days of the year with data" }
   map:
@@ -1670,13 +1674,13 @@
     missing: Too little news monitored by GDELT
     steps: [[0, "Under 0.001%"], [0.001, "0.001–0.003%"], [0.003, "0.003–0.01%"], [0.01, "0.01–0.03%"], [0.03, "0.03% or more"]]
-    title: How much each country's news talks about cannabis
+    title: Cannabis news from each country as a share of global coverage
     lede: >-
-      The share of a country's online news articles that mention cannabis or marijuana, the
-      average of the latest full year, from the GDELT Project. The stronger the blue, the more the
-      press talks about it.
+      Cannabis or marijuana articles from each country's news sites as a share of all global
+      news indexed by GDELT, averaged over the latest full year. This is each country's
+      contribution to global coverage, not cannabis attention within its own press.
     sub: >-
-      The stronger the blue, the larger the share of news articles that mention cannabis.
+      The stronger the blue, the larger the contribution to global indexed cannabis coverage.
     note: >-
-      The GDELT Project, DOC 2.0 API. Attention, not opinion; not part of the score.
+      The GDELT Project, DOC 2.0 API TimelineVol. Global denominator; uneven outlet coverage. Not part of the score.
   pages:
     - { label: its map, path: map/in-the-news/ }

@@ -0,0 +1,19 @@
+# Issue corrections — 27 October 2026
+
+Read DATA-MAINTENANCE, WRITING and the repository overview. Each section records
+its source checks, scope and score reasoning.
+
+## #186 — GDELT global coverage denominator
+
+Read the official DOC 2.0 specification (TimelineVol) at
+https://blog.gdeltproject.org/gdelt-doc-2-0-api-debuts/ and compared the committed
+collector. Source-country filtering selects the numerator; the denominator remains
+all global indexed news. Retained the observations as that metric, rather than
+fabricating a country denominator. Dataset metadata, profile headline, map title,
+legend explanation, chart heading, tooltip and collector documentation now agree.
+Removed the chart's misleading within-press attention comparison/correlation.
+The yearly statistic is an equal-weight mean of returned daily percentages, not
+an article-weighted annual ratio. The CSV is unchanged. A future country-normalised
+series requires matched local numerator/denominator counts over identical complete
+intervals, validated historical API availability and rate limits; it must use a
+separate definition/file. Do not rescale these percentages to pretend to recover it.

@@ -14,5 +14,5 @@
   python3 scripts/fetch-open-data.py trials      # clinical trials of cannabis medicines, by country and year
   python3 scripts/fetch-open-data.py parties     # political parties founded to legalise cannabis (Wikidata)
-  python3 scripts/fetch-open-data.py news        # the share of each country's news that mentions cannabis (GDELT)
+  python3 scripts/fetch-open-data.py news        # cannabis articles by source country / global GDELT coverage
 
 gbif      GBIF occurrence search, genus Cannabis (all its species and synonyms), faceted by the
@@ -66,7 +66,10 @@ parties   Wikidata: political parties whose name or ideology mentions cannabis,
           and the year they were founded (and dissolved, if they were). Writes
           datasets/wikidata-cannabis-parties.csv, one row per party.
-news      The GDELT Project's DOC 2.0 API: for the online news published in each country, the
-          share of articles that mention cannabis or marijuana (GDELT matches the words in 65
-          languages through its own translation), day by day from 2017, averaged here by year (`days` is how many days a year has).
+news      The GDELT DOC 2.0 TimelineVol API: cannabis articles from each source country
+          divided by ALL GLOBAL indexed articles, not that country's output. Source-country
+          filtering does not change the denominator. Annual values are equal-weight means
+          of returned daily percentages, not annual article-count ratios (`days` counts points).
+          Historical availability must be checked on each run; an API response is not proof
+          that every requested historical day was returned.
           Countries are asked for by name, one every ten seconds and after a minute's wait when
           GDELT refuses, so a run takes about forty minutes. Writes datasets/gdelt-cannabis-news.csv.

@@ -343,2 +343,13 @@ test('San Marino distinguishes authorised flower from verified dispensing (#187)
   assert.deepEqual(sm.scores, { possession: 1, cultivation: 1, enforcement: 2, sharing: 1, consumption: 1, access: 1 });
 });
+
+// Issue #186
+test('GDELT observations are labelled as daily global shares, never local attention (#186)', () => {
+  const news = yaml.load(fs.readFileSync(path.join(root, 'datasets/datasets.yaml'), 'utf8')).find(d => d.id === 'news');
+  assert.match(news.about, /denominator stays global/);
+  assert.match(news.profile.headline, /global news/);
+  assert.match(news.origin, /arithmetic means of daily global-coverage percentages/);
+  assert.match(news.map.lede, /not cannabis attention within its own press/);
+  const chart = fs.readFileSync(path.join(root, 'src/pages/charts.astro'), 'utf8');
+  assert.doesNotMatch(chart, /press talks about cannabis most|attention and the score go together/);
+});

@@ -734,5 +734,5 @@ const punishTreat = countries.map((c) => {
 }).filter(Boolean).sort((a, b) => a.c.name.localeCompare(b.c.name));
 
-// ── In the news: the share of each country's news articles that mention cannabis ──
+// ── Cannabis articles by source country, as a share of global GDELT coverage ──
 const NEWS = DATASETS.news; // absent until datasets/gdelt-cannabis-news.csv has been fetched
 const NEWS_YEAR = NOW - 1; // the latest full year
@@ -743,9 +743,4 @@ const newsRows = NEWS ? countries.map((c) => {
 }).filter(Boolean).sort((a, b) => b.share - a.share) : [];
 const newsTop = newsRows.slice(0, 20);
-// on a log scale, so only the countries with any measurable share
-const newsSome = newsRows.filter((n) => n.share > 0);
-const newsFit = newsSome.length > 20 ? corr(newsSome.map((n) => ({ x: Math.log10(n.share), y: n.c.raw }))) : { r: 0 };
-// "The United States' press", "Germany's press"
-const pressOf = (name) => `${/^United |^Netherlands|^Philippines/.test(name) ? "The " : ""}${name}${name.endsWith("s") ? "'" : "'s"}`;
 const sparkOf = (rows) => { const top = Math.max(...rows.map((r) => r.share)); return rows.map((r, i) => `${i ? "L" : "M"}${pct(r.year, 2017, NEWS_YEAR).toFixed(1)},${(100 - (r.share / top) * 90).toFixed(1)}`).join(""); };
 
@@ -1877,6 +1872,6 @@ const kick = (id) => { const i = sections.findIndex(([x]) => x === id); return `
     <section class="ch" id="in-the-news">
       <p class="ch-kicker"><a href="#in-the-news">{kick("in-the-news")}</a></p>
-      <h2>{pressOf(newsTop[0].c.name)} press talks about cannabis most: {(newsTop[0].share / newsTop[1].share).toFixed(0)} times the share in {pressOf(newsTop[1].c.name).replace(/^The /, "the ")} news in {NEWS_YEAR}</h2>
-      <p class="ch-dek">Of all the online news GDELT monitors from each country's own sites, the share of articles that mention cannabis or marijuana in {NEWS_YEAR}, the {newsTop.length} highest of {newsRows.length} countries. Beside each, its share year by year since 2017. Attention, not opinion: a crackdown makes news as much as a reform. Across the {newsSome.length} countries with any measurable share, attention and the score go together with r = {newsFit.r.toFixed(2)}.</p>
+      <h2>{newsTop[0].c.name} contributes the largest global share of indexed cannabis news in {NEWS_YEAR}</h2>
+      <p class="ch-dek">Cannabis or marijuana articles from each source country, divided by all global news monitored by GDELT: the {newsTop.length} highest daily-average shares of {newsRows.length} countries in {NEWS_YEAR}. Beside each, its series since 2017. These figures reflect national news output and uneven outlet coverage as well as cannabis reporting. They do not measure the share of cannabis stories within each country's press, or public opinion.</p>
       <ol class="ch-bars ch-news" data-no-preview>
         {newsTop.map((n) => (
@@ -1892,5 +1887,5 @@ const kick = (id) => { const i = sections.findIndex(([x]) => x === id); return `
       </ol>
       <p class="ch-source">{NEWS.title}; years with fewer than 200 days of data left out. <a href={`${base}sources/#dataset-news`}>About the dataset</a>.</p>
-      <p class="ch-more"><a href={`${base}map/in-the-news/`}>News attention on the map →</a> <a href="#looked-it-up">Wikipedia page views →</a></p>
+      <p class="ch-more"><a href={`${base}map/in-the-news/`}>Global news contributions on the map →</a> <a href="#looked-it-up">Wikipedia page views →</a></p>
     </section>
     )}
@@ -2359,5 +2354,5 @@ const kick = (id) => { const i = sections.findIndex(([x]) => x === id); return `
     document.querySelectorAll("[data-news]").forEach((a) => card(a, () => {
       const [name, share, rows, score, tier] = D.news[a.dataset.news];
-      return `<b>${esc(name)}</b><span class="ch-tip-sub">share of the country's news articles mentioning cannabis</span>` + rows.map(([y, v]) => `<span class="ch-tip-row">${y}<em>${v.toFixed(3)}%</em></span>`).join("")
+      return `<b>${esc(name)}</b><span class="ch-tip-sub">cannabis articles from this country as a share of global indexed news</span>` + rows.map(([y, v]) => `<span class="ch-tip-row">${y}<em>${v.toFixed(3)}%</em></span>`).join("")
         + `<span class="ch-tip-row">Score<em>${score} · ${esc(tier)}</em></span><span class="ch-tip-go">Open its datasets →</span>`;
     }));