Code, design and documents 12 files
datasets/datasets.yaml +27 −21
@@ -378,5 +378,6 @@
reports several studies in a year, their means are averaged. The EUDA notes that Dutch
prices are for cannabis bought anonymously in coffeeshops, not on the street, and that
- Türkiye reports one price for herb and resin together. Prices are not adjusted for
+ Türkiye reports one price for herb and resin together and is excluded from these separate
+ product series. Prices are not adjusted for
inflation.
origin: >-
@@ -384,5 +385,5 @@
October 2026 from its data service, sent compressed and saved unchanged; each file's SHA-256
matches the browser's copy. scripts/extract-euda.py (prices) turns them into one row per
- country and year.
+ country and year, excluding Türkiye’s inseparable herb/resin figures.
profile:
headline: "€{herb_eur} a gram of herb"
@@ -1390,5 +1391,5 @@
- id: wikipedia_views
file: wikipedia-cannabis-views.csv
- name: Readers of the cannabis article on Wikipedia
+ name: Views of the cannabis article on Wikipedia
publisher: Wikimedia Foundation
credit: Wikimedia
@@ -1399,8 +1400,8 @@
when: July 2015 to September 2026
about: >-
- How many times a month people (not bots) opened the Wikipedia article on cannabis as a drug,
- in 26 languages spoken mainly in one country: German for Germany, Thai for Thailand and so
- on. A measure of curiosity: it jumps when cannabis is in the news. German is also read in
- Austria and Switzerland, Dutch in Belgium.
+ Monthly page-load events for the cannabis-as-a-drug article in 26 Wikipedia language editions,
+ using traffic classified as user rather than automated. These are page-load events, not unique readers;
+ repeat visits count again. Language editions identify worldwide traffic, not the location of
+ people viewing them. German Wikipedia is also read in Austria and Switzerland, Dutch in Belgium.
origin: >-
Downloaded on 22 October 2026 by scripts/fetch-open-data.py (wikipedia); the article in each
@@ -1410,5 +1411,5 @@
nl: Dutch Wikipedia, also read in Belgium.
profile:
- headline: "{views} readers in {month}"
+ headline: "{views} page views in {month}"
details: "{article} · {language} Wikipedia"
country: { column: country, by: id }
@@ -1504,16 +1505,20 @@
data_url: https://clinicaltrials.gov/api/v2/studies?query.intr=cannabis%20OR%20cannabidiol%20OR%20THC%20OR%20dronabinol%20OR%20nabilone%20OR%20nabiximols%20OR%20cannabinoid%20OR%20marijuana
data_label: API v2 query
- when: by the year each study started
+ when: actual starts recorded as of 26 October 2026
about: >-
- Registered clinical studies whose intervention is cannabis, cannabidiol (CBD), THC or a
- cannabinoid medicine, counted once for each country where the study has a site, by the year
- it started, with the running total; studies not yet started are left out. ClinicalTrials.gov is American and holds most, not all,
- of the world's registered trials; a study run in several countries counts in each.
+ Registered studies with a cannabis or cannabinoid intervention and a recorded actual start,
+ counted once for each country with a site. Only starts explicitly marked ACTUAL on or before
+ 26 October 2026 are included. Estimated or missing start types, not-yet-recruiting studies and
+ withdrawn studies are excluded. ClinicalTrials.gov does not cover every trial worldwide;
+ a study with sites in several countries counts in each. Current-year counts are incomplete.
origin: >-
- Downloaded on 22 October 2026 by scripts/fetch-open-data.py (trials), which matches the
- registry's country names to the country files.
+ Rebuilt on 26 October 2026 by scripts/fetch-open-data.py (trials), matching registry country
+ names to country files. The query explicitly requests StartDateType, StartDate and OverallStatus.
+ Complete dates are compared with the extraction date; month-only and year-only actual dates use
+ their last possible day, conservatively. Duplicate study/country pairs count once. Zero-start
+ years preserve cumulative totals. The as_of column records the snapshot date.
profile:
headline: "{to_date} studies so far"
- details: "{trials} started in {year}"
+ details: "{trials} actual starts in {year} · as of {as_of}"
each: "{to_date} by {year}"
years: [2010, 2020]
@@ -1521,4 +1526,5 @@
year: { column: year }
fields:
+ as_of: { column: as_of, type: text, about: "Registry snapshot date; current-year counts are incomplete" }
to_date: { column: to_date, about: "Studies so far: map, Charts page and profile" }
trials: { column: trials, about: "Studies that started that year: Charts page and profile" }
@@ -1530,17 +1536,17 @@
slug: clinical-trials
label: Clinical trials per million people
- missing: No registered cannabis trial
+ missing: No eligible recorded actual start
steps: [[0, "Under 0.2"], [0.2, "0.2–1"], [1, "1–3"], [3, "3–10"], [10, "10 or more"]]
title: Where cannabis medicines are tested
lede: >-
Registered clinical studies of cannabis, CBD, THC or cannabinoid medicines with a site in
- each country, per million people, from ClinicalTrials.gov. The stronger the blue, the more
- studies.
+ each country, per million people, from ClinicalTrials.gov. Only actual starts through
+ 26 October 2026 are included; current-year counts are incomplete. Stronger blue means more studies.
sub: >-
The stronger the blue, the more cannabis studies per million people. Countries with none
are hatched.
note: >-
- ClinicalTrials.gov, all years; a study with sites in several countries counts in each. Not
- part of the score.
+ ClinicalTrials.gov, actual starts as of 26 October 2026; current-year counts are incomplete.
+ A study with sites in several countries counts in each. Not part of the score.
pages:
- { label: its map, path: map/clinical-trials/ }
docs/DATA-MAINTENANCE.md +24 −0
@@ -1847,2 +1847,26 @@ history supplies the appeal account. The largest recorded gap falls from 93 to
48 years. The first-restriction estimate and the later custody-to-fines transition
remain open; no existing scores or historical score vectors change.
+
+### Product scope, recorded starts and pageview units (26 October 2026)
+
+- A source that combines herb and resin, or does not distinguish them, cannot supply a
+ flower-only price. Preserve rejected figures in research notes; do not infer a flower
+ value from the combined average. EUDA's Türkiye footnote applies to the extracted
+ product-price series too, so `extract-euda.py prices` excludes those inseparable rows.
+ Retained regional price observations use `comparison: false`, including Brazil's
+ three-state police sample. Check maps, country headlines, compare/API data and exports.
+- `fetch-open-data.py trials` counts recorded **ACTUAL** starts on/before `TRIALS_AS_OF`
+ (today by default), excluding unknown/estimated types, not-yet-recruiting and withdrawn
+ studies. Request **StartDateType** explicitly: the API's StartDate field alone omits it.
+ Month/year-only dates use their latest possible day. Regenerate the CSV and update its
+ dated dataset metadata; `as_of` is published with the data and profile details. Keep zero
+ years so cumulative figures do not disappear. `TRIALS_SNAPSHOT` can save the projected
+ registry records locally for audit. Run `python3 -B scripts/fetch-open-data.test.py`.
+- Wikimedia per-article pageviews are content-request events, not unique readers or devices.
+ The `user` traffic filter does not deduplicate people, and language editions are not
+ geographical audience samples. Preserve the figures and label the unit correctly.
+- Check culture prose as well as profiles/articles when correcting repeated historical
+ claims. Bengal's 1798 Board letter recommended taxes; it does not establish enactment.
+- An authorised medical product category is separate from verified pharmacy availability.
+ San Marino's framework includes inflorescences; it does not establish that flower is
+ actually dispensed. Preserve the access score until practical supply evidence warrants it.
docs/research/issues-182-185-187-190-2026-10-26.md +118 −0
@@ -0,0 +1,118 @@
+# Follow-ups 182, 185, 187–190 — 26 October 2026
+
+Read DATA-MAINTENANCE and WRITING. Country/history scores remain unchanged. Existing
+source-archive staging and concurrent history work are outside this commit.
+
+## #182: Bengal's recommendation
+
+Read IHDC volume I, chapter XIV, paragraph 556, printed p.267 (PDF6), including
+the reproduced Board letter No.22 of 29 May 1798. Visually checked the date and
+its recommendation to restrain use and improve revenue through taxes. It does
+not prove implementation on that date. Bhang and Charas now say recommended;
+Bhang's later “century after the first tax” wording is removed. Later territorial
+excise examples remain separately dated. Both pages list the inspected source.
+The corpus guard now includes culture files and both reported false phrases.
+https://newrevenue.org/wp-content/uploads/2017/08/ihd-vol-1-pages-263-282-or-so74464868_53_72.pdf
+
+## #185: product-mixed prices
+
+Read the retained official EUDA XML's Türkiye footnote 5: prices are submitted
+regardless of herb/resin and represent both. The 2019 TR-2 change does not supply
+separable historical product figures. Removed tr-street-2024 (€8.50; US$9.54)
+from the flower observations and exclude Türkiye's 11 rows from both separate
+EUDA product-price series. Original XML is preserved; the extractor enforces the
+exclusion on regeneration. No invented adjusted price is substituted. Chart
+cohort fixtures now have 8 endpoint observations in 2002 and 15 in 2024; the
+balanced THC-per-euro cohort is Spain, Netherlands and Portugal.
+https://www.euda.europa.eu/data/stats2026/ppp_en#displayTable:PPP-02-1-1-2-3
+
+Audit of other explicit product caveats found xk-street-2022 (€5.50; US$6.17).
+Read the GI-TOC source: its mean is labelled simply Cannabis, with no herb/resin
+separation. Its source entries span different quantities/places. Removed this
+unverified flower observation too. These rejected numeric values are retained
+here rather than in flower-price exports.
+https://see.globalinitiative.net/hotspots/drugs/country/kosovo
+
+## #187: San Marino authorisation and availability
+
+Read the official English Law 113/2021, especially articles 1–3 and 9–13, the
+health-authority cannabis page and FAQ Q3. The law names flowers/inflorescences;
+state pharmacies retail prescribed medicines and pharmaceutical-grade raw
+material. Read the relevant implementing provisions in Decree 28/2023
+(articles 2–3, 5–6) and Decree 53/2023 (authorisation/application and production
+requirements). Neither establishes a named flower product currently dispensed.
+No state-pharmacy flower stock/register was verified. Replace absolute no-flower
+claims while retaining medical: limited, all scores and the home-growing ban.
+The 2016 initiative is not represented as the 2021 licensing law.
+https://www.consigliograndeegenerale.sm/on-line/documento17136319.html
+https://www.gov.sm/pub2/GovSM/Authority-Sanitaria/Medicinali/Medicinali-stupefacenti-e-psicotropi/Cannabis-terapeutica.html
+https://www.gov.sm/pub2/GovSM/Authority-Sanitaria/Medicinali/Medicinali-stupefacenti-e-psicotropi/Cannabis-terapeutica/FAQ---frequently-asked-questions.html
+https://www.gov.sm/pub2/GovSM/dam/jcr:723b46d7-2494-4bde-8a41-3138bd74859f/DD028-2023%20-%20modifiche%20cannabis.pdf
+https://www.gov.sm/pub2/GovSM/dam/jcr:79eb5765-535d-4b5a-ac43-675a2915497e/DD053-2023.pdf
+
+## #188: actual study starts
+
+Read the registry's definition of a study start (actual first enrolment versus
+estimated recruitment opening) and queried API v2 directly. A projection with
+StartDate alone returns the date **without type**. Explicitly requesting
+StartDateType restores ACTUAL/ESTIMATED. This detail is covered by an API-query
+regression as well as date/status tests.
+https://clinicaltrials.gov/policy/protocol-definitions
+
+Re-fetched all 1,795 matching records in two pages on 26 October 2026; saved the
+projected registry objects in ignored local research storage. Types: 1,182 ACTUAL,
+219 ESTIMATED, 394 missing. The policy deliberately excludes unconfirmed starts
+even if an estimated date has passed; these are confirmed recorded starts, not
+an estimate of every study actually conducted. NCT07694466 is present with
+NOT_YET_RECRUITING and ESTIMATED 2026-11-01 and contributes no count.
+
+Full dates must be on/before 2026-10-26. Partial month/year dates use their last
+possible day conservatively; malformed dates are excluded. Each NCT/country
+pair counts once. Zero-start years carry cumulative totals through the snapshot
+instead of making an inactive country's earlier studies disappear from the map.
+Regenerated 628 rows for 52 countries, with explicit as_of dates. The latest
+rows are US 37 starts / 682 cumulative, Canada 5 / 126, UK 3 / 48. Differences
+from the old series reflect the stricter evidence rule and a fresh registry
+snapshot, not solely deletion of November/December 2026 plans.
+
+## #189: Brazil's regional police sample
+
+Downloaded the original joint report; read Table 16, printed/PDF p.94, and the
+surrounding market discussion. Visually checked retail R$3.69, n=52, SD2.25,
+for SP, PR and PE. Read the UN launch announcement: the broader study involved
+four states, but that cannabis retail table covers three. Keep the numeric
+observation and its qualification with comparison: false. The shared selector
+already excludes it from country price headlines, maps, rankings and compare
+API values; history/exports retain the flagged regional observation.
+https://cdesc.org.br/wp-content/uploads/2023/01/Livro-DINAMICAS-DROGAS-ILICITAS-CdE.pdf
+https://brasil.un.org/pt-br/212764-unodc-e-mjsp-lan%C3%A7am-estudo-sobre-mercado-de-drogas-il%C3%ADcitas-no-brasil
+
+## #190: views are events
+
+Read Wikimedia's pageview definition and per-article API documentation. A view
+is a content request with the stated response conditions; user-filtered traffic
+does not identify unique people. Rename the dataset/card headline and remaining
+chart navigation to page views; preserve the raw CSV byte for byte. The metadata
+also keeps the worldwide language-edition/geolocation distinction.
+https://doc.wikimedia.org/generated-data-platform/aqs/analytics-api/concepts/page-views.html
+https://doc.wikimedia.org/generated-data-platform/aqs/analytics-api/reference/page-views.html
+
+## Validation
+
+All 98 tests pass: 4 Python extraction tests, 14 chart/data tests, 40 history
+tests and 40 answer/profile tests. The full production build with prices enabled
+passes (4,329 pages). Rendered checks confirm both culture corrections, San Marino
+product wording and prescription quick answer, BR/TR/XK comparison exclusions,
+retention of Brazil’s flagged source observation, updated pageview card/API units,
+trial as_of dates and the US/Canada/UK totals. All 52 country series have continuous
+zero-year/cumulative totals through 2026. Browser checks confirm San Marino’s
+context, the US actual-start panel and snapshot date, and the German pageview card.
+San Marino’s scores/status are unchanged and its chain has no flagged steps; two
+pre-existing unscored licensing events remain open in triage. The raw Wikipedia
+CSV is byte-for-byte unchanged. The registry audit finds 1,158 eligible recorded
+actual starts and 1,335 country-study memberships; NCT07694466 is excluded.
+
+Regression coverage includes
+culture-wide 1798 wording, San Marino scope versus supply, mixed-product prices,
+regional comparison eligibility, pageview units, actual/estimated/future starts,
+partial/invalid dates, duplicate countries/studies, zero-year totals and pagination.
prices.yaml +1 −24
@@ -396,16 +396,4 @@
url: https://www.euda.europa.eu/data/stats2026/ppp_en#displayTable:PPP-02-1-1-2-3
-- id: tr-street-2024
- country: tr
- date: "2024"
- market: street
- currency: EUR
- price: 8.5
- usd: 9.54
- note: Mean retail price for one gram of herbal cannabis as reported to EUDA by Türkiye; the figure covers both herb and resin, as reported by the country. Latest year available in the table.
- source:
- title: EUDA — Statistical Bulletin 2026, price, purity and potency (retail herbal cannabis, mean)
- url: https://www.euda.europa.eu/data/stats2026/ppp_en#displayTable:PPP-02-1-1-2-3
-
- id: hu-street-2023
country: hu
@@ -1549,4 +1537,5 @@
- id: br-street-2022
+ comparison: false
country: br
date: "2022"
@@ -1908,16 +1897,4 @@
url: https://wdr.unodc.org/wdr2020/field/Annex/8.1._Prices_and_Purities_of_drugs.pdf
-- id: xk-street-2022
- country: xk
- date: "2022"
- market: street
- currency: EUR
- price: 5.5
- usd: 6.17
- note: Mean retail price per gram of cannabis for 2021/2022 from a civil-society observatory's drug-market monitoring; the page does not separate herb from resin, and its underlying open-source entries for one gram run from 2.80 to 10.
- source:
- title: Global Initiative Against Transnational Organized Crime — South-Eastern Europe Observatory, drug markets in Kosovo
- url: https://see.globalinitiative.net/hotspots/drugs/country/kosovo
-
- id: pa-street-2015
country: pa
scripts/chart-semantics.test.cjs +35 −3
@@ -74,8 +74,8 @@ test('THC trend holds the endpoint cohort constant and omits incomplete years (#
return t ? [{ c: { id: p.country }, mg: t.herb_thc * 10 / p.herb_eur }] : [];
}) }));
- assert.equal(years[0].rows.length, 9);
- assert.equal(years.at(-1).rows.length, 16);
+ assert.equal(years[0].rows.length, 8);
+ assert.equal(years.at(-1).rows.length, 15);
const panel = matchedThcPanel(years);
- assert.deepEqual([...panel.cohort].sort(), ['es', 'nl', 'pt', 'tr']);
+ assert.deepEqual([...panel.cohort].sort(), ['es', 'nl', 'pt']);
assert.ok(panel.years.length > 2);
for (const y of panel.years) assert.deepEqual(y.rows.map((r) => r.c.id).sort(), [...panel.cohort].sort());
@@ -161,2 +161,34 @@ test('raw survey rates do not acquire country percentiles or comparison deltas (
assert.match(compare, /META.useComparisonNote/);
});
+
+test('mixed or unspecified herb/resin prices stay outside flower metrics (#185)', async () => {
+ const { isMixedCannabisPrice } = await import('../src/lib/price-selection.js');
+ for (const note of ['covers both herb and resin', 'herb + resin', 'does not separate herb from resin', 'combined herb and resin']) assert.equal(isMixedCannabisPrice(note), true);
+ assert.equal(isMixedCannabisPrice('Mean retail price per gram of herbal cannabis'), false);
+ const prices = yaml.load(fs.readFileSync('prices.yaml', 'utf8'));
+ assert.ok(!prices.some((p) => ['tr-street-2024', 'xk-street-2022'].includes(p.id)));
+ for (const p of prices) assert.equal(isMixedCannabisPrice(p.note), false, p.id);
+ const europe = csv('euda-cannabis-prices.csv');
+ assert.ok(!europe.some((p) => p.country === 'tr'));
+ assert.match(fs.readFileSync('scripts/extract-euda.py', 'utf8'), /excluded_countries=\{"tr"\}/);
+ assert.match(fs.readFileSync('src/content.config.ts', 'utf8'), /isMixedCannabisPrice\(note\)/);
+});
+
+test('acknowledged regional or sentinel prices cannot enter country comparisons (#189)', async () => {
+ const { latestComparablePrices } = await import('../src/lib/price-selection.js');
+ const prices = yaml.load(fs.readFileSync('prices.yaml', 'utf8'));
+ for (const p of prices.filter((p) => /not a national figure|not a representative national street price/i.test(p.note || ''))) assert.equal(p.comparison, false, p.id);
+ const br = prices.find((p) => p.id === 'br-street-2022');
+ assert.equal(br.price, 3.69);
+ assert.equal(br.usd, 0.71);
+ assert.match(br.note, /52 police records.*three states only/);
+ assert.equal(latestComparablePrices([br]).street, undefined);
+});
+
+test('Wikimedia event counts remain page views rather than readers (#190)', () => {
+ const wiki = yaml.load(fs.readFileSync('datasets/datasets.yaml', 'utf8')).find((d) => d.id === 'wikipedia_views');
+ assert.equal(wiki.name, 'Views of the cannabis article on Wikipedia');
+ assert.equal(wiki.profile.headline, '{views} page views in {month}');
+ assert.match(wiki.about, /page-load events, not unique readers/);
+ assert.match(wiki.about, /worldwide traffic, not the location/);
+});
scripts/extract-euda.py +4 −3
@@ -88,8 +88,8 @@ def by_country_year(path, names, missed):
-def kinds_table(task, sources, out):
+def kinds_table(task, sources, out, excluded_countries=()):
names, missed = our_names(), []
tables = {kind: by_country_year(path, names, missed) for kind, path in sources.items()}
- keys = sorted(set().union(*tables.values()))
+ keys = sorted(k for k in set().union(*tables.values()) if k[0] not in excluded_countries)
rows = []
for cid, year in keys:
@@ -109,6 +109,7 @@ def kinds_table(task, sources, out):
def prices():
+ # EUDA footnote 5: Türkiye reports herb/resin together, so neither product price is separable.
kinds_table("prices", {"herb_eur": "datasets/euda-sb2026-ppp-02-1-1-2-3-herb-price.xml",
- "resin_eur": "datasets/euda-sb2026-ppp-02-1-1-1-3-resin-price.xml"}, "datasets/euda-cannabis-prices.csv")
+ "resin_eur": "datasets/euda-sb2026-ppp-02-1-1-1-3-resin-price.xml"}, "datasets/euda-cannabis-prices.csv", excluded_countries={"tr"})
scripts/fetch-open-data.py +69 −26
@@ -9,5 +9,5 @@
python3 scripts/fetch-open-data.py religion # the most common religion (Pew, via OWID)
python3 scripts/fetch-open-data.py gdp # GDP per person at purchasing power parity (World Bank)
- python3 scripts/fetch-open-data.py wikipedia # monthly readers of the cannabis article, by language
+ python3 scripts/fetch-open-data.py wikipedia # monthly page views of the cannabis article, by language
python3 scripts/fetch-open-data.py development # internet, towns, schooling, age and health (World Bank)
python3 scripts/fetch-open-data.py books # how often books name cannabis, by language (Google Books Ngram)
@@ -44,5 +44,5 @@ gdp The World Bank's GDP per person at purchasing power parity, in constan
international dollars (NY.GDP.PCAP.PP.KD), every year from 1990. Writes
datasets/wb-gdp-per-person.csv.
-wikipedia Monthly page views by people (not bots) of the Wikipedia article on cannabis as a drug,
+wikipedia Monthly page views classified as user traffic (not unique readers) of the Wikipedia article on cannabis as a drug,
in the languages spoken mainly in one country, from July 2015, from the Wikimedia
pageviews API; the article in each language is the one the English "Cannabis (drug)"
@@ -60,6 +60,6 @@ books Google Books Ngram Viewer, the share of all words in each year's print
trials ClinicalTrials.gov (API v2): every registered study whose intervention mentions cannabis,
cannabidiol, THC, dronabinol, nabilone, nabiximols or a cannabinoid, counted once for
- each country where it has a site, by the year it started, with the running total; studies that have not started yet
- are left out. Writes
+ each country where it has a site, by its recorded ACTUAL start, on or before the snapshot
+ date. Estimated/unknown starts and not-yet-recruiting/withdrawn studies are left out. Writes
datasets/clinicaltrials-cannabis.csv.
parties Wikidata: political parties whose name or ideology mentions cannabis, with their country
@@ -76,5 +76,5 @@ map's country shapes.
Python's standard library only. Run from the project root.
"""
-import csv, datetime, glob, json, os, re, sys, tempfile, time, urllib.parse, urllib.request
+import calendar, csv, datetime, glob, json, os, re, sys, tempfile, time, urllib.parse, urllib.request
AGENT = "cannabis-freedom-index dataset script (https://github.com/burningtree/cannabis-freedom-index)"
@@ -277,5 +277,54 @@ TRIAL_NAMES = {"Korea, Republic of": "kr", "Turkey (Türkiye)": "tr", "Russian F
-def trials():
+def actual_start_year(status, as_of):
+ """A recorded actual start, wholly on/before the snapshot date; never infer a start."""
+ start = status.get("startDateStruct", {})
+ if start.get("type") != "ACTUAL" or status.get("overallStatus") in {"NOT_YET_RECRUITING", "WITHDRAWN"}:
+ return None
+ date = start.get("date", "")
+ try:
+ if re.fullmatch(r"\d{4}-\d{2}-\d{2}", date):
+ latest = datetime.date.fromisoformat(date)
+ elif re.fullmatch(r"\d{4}-\d{2}", date):
+ year, month = map(int, date.split("-"))
+ latest = datetime.date(year, month, calendar.monthrange(year, month)[1])
+ elif re.fullmatch(r"\d{4}", date):
+ latest = datetime.date(int(date), 12, 31)
+ else:
+ return None
+ except ValueError:
+ return None
+ return latest.year if latest <= as_of else None
+
+
+def trial_rows(studies, names, as_of):
+ """Count one actual-start study per country; keep zero years for cumulative series."""
+ counts, seen, missed = {}, set(), set()
+ for study in studies:
+ ps = study.get("protocolSection", {})
+ nct = ps.get("identificationModule", {}).get("nctId")
+ year = actual_start_year(ps.get("statusModule", {}), as_of)
+ if not nct or year is None:
+ continue
+ for country in {loc.get("country") for loc in ps.get("contactsLocationsModule", {}).get("locations", []) if loc.get("country")}:
+ cid = names.get(country)
+ if not cid:
+ missed.add(country)
+ continue
+ if (nct, cid) not in seen:
+ seen.add((nct, cid))
+ counts[(cid, year)] = counts.get((cid, year), 0) + 1
+ rows = []
+ for cid in sorted({c for c, _ in counts}):
+ total = 0
+ for year in range(min(y for c, y in counts if c == cid), as_of.year + 1):
+ n = counts.get((cid, year), 0)
+ total += n
+ rows.append([cid, year, n, total, as_of.isoformat()])
+ return rows, missed
+
+
+def trials(as_of=None):
+ as_of = as_of or datetime.date.fromisoformat(os.environ.get("TRIALS_AS_OF", datetime.date.today().isoformat()))
names = {}
for path in glob.glob("countries/*.yaml") + glob.glob("countries/*/index.yaml"):
@@ -284,30 +333,23 @@ def trials():
names |= TRIAL_NAMES
terms = "cannabis OR cannabidiol OR THC OR dronabinol OR nabilone OR nabiximols OR cannabinoid OR marijuana"
- counts, token, missed = {}, None, set()
+ studies, token = [], None
while True:
- q = {"query.intr": terms, "fields": "NCTId,LocationCountry,StartDate", "pageSize": 1000}
+ # StartDate alone projects only the date: StartDateType is required to retain ACTUAL/ESTIMATED.
+ q = {"query.intr": terms, "fields": "NCTId,LocationCountry,StartDate,StartDateType,OverallStatus", "pageSize": 1000}
if token:
q["pageToken"] = token
page = get("https://clinicaltrials.gov/api/v2/studies?" + urllib.parse.urlencode(q))
- for st in page["studies"]:
- ps = st["protocolSection"]
- start = ps.get("statusModule", {}).get("startDateStruct", {}).get("date", "")[:4]
- for country in {loc.get("country") for loc in ps.get("contactsLocationsModule", {}).get("locations", []) if loc.get("country")}:
- cid = names.get(country)
- if not cid:
- missed.add(country); continue
- # a study planned for a later year has not started: it is left out
- if start and int(start) <= datetime.date.today().year:
- counts[(cid, int(start))] = counts.get((cid, int(start)), 0) + 1
+ studies.extend(page["studies"])
+ print(f" retrieved {len(studies)} registry records", flush=True)
token = page.get("nextPageToken")
if not token:
break
+ if os.environ.get("TRIALS_SNAPSHOT"):
+ with open(os.environ["TRIALS_SNAPSHOT"], "w", encoding="utf-8") as f:
+ json.dump({"as_of": as_of.isoformat(), "query": terms, "studies": studies}, f)
+ rows, missed = trial_rows(studies, names, as_of)
if missed:
print("no country file for:", "; ".join(sorted(missed)), file=sys.stderr)
- rows, total = [], {}
- for (c, y), n in sorted(counts.items()):
- total[c] = total.get(c, 0) + n
- rows.append([c, y, n, total[c]])
- write("datasets/clinicaltrials-cannabis.csv", ["country", "year", "trials", "to_date"], rows)
+ write("datasets/clinicaltrials-cannabis.csv", ["country", "year", "trials", "to_date", "as_of"], rows)
@@ -405,5 +447,6 @@ TASKS = {"gbif": gbif, "openalex": openalex, "alcohol": alcohol, "governance": g
"corruption": corruption, "religion": religion, "gdp": gdp, "wikipedia": wikipedia, "development": development,
"books": books, "trials": trials, "parties": parties, "news": news}
-if len(sys.argv) != 2 or sys.argv[1] not in TASKS:
- sys.exit(f"usage: python3 scripts/fetch-open-data.py {'|'.join(TASKS)}")
-TASKS[sys.argv[1]]()
+if __name__ == "__main__":
+ if len(sys.argv) != 2 or sys.argv[1] not in TASKS:
+ sys.exit(f"usage: python3 scripts/fetch-open-data.py {'|'.join(TASKS)}")
+ TASKS[sys.argv[1]]()
scripts/fetch-open-data.test.py +75 −0
@@ -0,0 +1,75 @@
+"""Regression checks for ClinicalTrials.gov start semantics (issue #188)."""
+import datetime
+import importlib.util
+from pathlib import Path
+import unittest
+from unittest.mock import patch
+from urllib.parse import parse_qs, urlparse
+
+spec = importlib.util.spec_from_file_location('fetch_open_data', Path(__file__).with_name('fetch-open-data.py'))
+data = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(data)
+AS_OF = datetime.date(2026, 10, 22)
+
+
+def status(date, kind='ACTUAL', overall='RECRUITING'):
+ return {'startDateStruct': {'date': date, 'type': kind}, 'overallStatus': overall}
+
+
+def study(nct, date, kind='ACTUAL', overall='RECRUITING', countries=('United States',)):
+ return {'protocolSection': {'identificationModule': {'nctId': nct},
+ 'statusModule': status(date, kind, overall),
+ 'contactsLocationsModule': {'locations': [{'country': c} for c in countries]}}}
+
+
+class ActualStartTests(unittest.TestCase):
+ def test_future_and_estimated_starts_are_not_started(self):
+ for record in [status('2026-11-01', 'ESTIMATED', 'NOT_YET_RECRUITING'),
+ status('2026-11-01'), status('2027-01-01'), status('2026-01-01', 'ESTIMATED'),
+ status('2026-01-01', overall='NOT_YET_RECRUITING'),
+ status('2026-01-01', overall='WITHDRAWN'),
+ {'startDateStruct': {'date': '2026-01-01'}}]:
+ with self.subTest(record=record):
+ self.assertIsNone(data.actual_start_year(record, AS_OF))
+
+ def test_actual_start_cutoff_inclusive_and_partial_dates_conservative(self):
+ self.assertEqual(data.actual_start_year(status('2026-10-22'), AS_OF), 2026)
+ self.assertIsNone(data.actual_start_year(status('2026-10-23'), AS_OF))
+ self.assertEqual(data.actual_start_year(status('2026-09'), AS_OF), 2026)
+ self.assertIsNone(data.actual_start_year(status('2026-10'), AS_OF))
+ self.assertEqual(data.actual_start_year(status('2025'), AS_OF), 2025)
+ self.assertIsNone(data.actual_start_year(status('2026'), AS_OF))
+ for date in ['', '2026-13', '2026-02-30', 'unknown']:
+ self.assertIsNone(data.actual_start_year(status(date), AS_OF))
+
+ def test_deduplication_cumulative_counts_and_zero_years(self):
+ actual = study('NCT00000001', '2024-01-01', countries=('United States', 'United States', 'Canada'))
+ rows, missed = data.trial_rows([actual, actual, study('NCT00000002', '2026-01-01'),
+ study('NCT07694466', '2026-11-01', 'ESTIMATED', 'NOT_YET_RECRUITING')],
+ {'United States': 'us', 'Canada': 'ca'}, AS_OF)
+ self.assertEqual(missed, set())
+ self.assertEqual([r for r in rows if r[0] == 'us'],
+ [['us', 2024, 1, 1, '2026-10-22'], ['us', 2025, 0, 1, '2026-10-22'],
+ ['us', 2026, 1, 2, '2026-10-22']])
+ self.assertEqual([r[-2] for r in rows if r[0] == 'ca'], [1, 1, 1])
+
+ def test_api_requests_start_type_and_consumes_all_pages(self):
+ pages = [{'studies': [study('NCT00000001', '2025-01-01')], 'nextPageToken': 'page2'},
+ {'studies': [study('NCT00000002', '2026-11-01', 'ESTIMATED')]}]
+ with patch.object(data, 'get', side_effect=pages) as get, patch.object(data, 'write') as write, \
+ patch.object(data.glob, 'glob', return_value=[]), \
+ patch.dict(data.TRIAL_NAMES, {'United States': 'us'}, clear=True), \
+ patch.dict(data.os.environ, {'TRIALS_SNAPSHOT': ''}):
+ data.trials(AS_OF)
+ self.assertEqual(get.call_count, 2)
+ for call in get.call_args_list:
+ fields = parse_qs(urlparse(call.args[0]).query)['fields'][0]
+ self.assertIn('StartDateType', fields)
+ self.assertIn('OverallStatus', fields)
+ self.assertEqual(parse_qs(urlparse(get.call_args_list[1].args[0]).query)['pageToken'], ['page2'])
+ self.assertEqual(write.call_args.args[1], ['country', 'year', 'trials', 'to_date', 'as_of'])
+ self.assertEqual(write.call_args.args[2][-1], ['us', 2026, 0, 1, '2026-10-22'])
+
+
+if __name__ == '__main__':
+ unittest.main()
scripts/profile-claims.test.mjs +15 −1
@@ -12,9 +12,10 @@ const root = path.join(path.dirname(fileURLToPath(import.meta.url)), "..");
const walk = (dir) => fs.readdirSync(dir, { withFileTypes: true }).flatMap((e) =>
e.isDirectory() ? walk(path.join(dir, e.name)) : /\.(ya?ml|md)$/.test(e.name) ? [path.join(dir, e.name)] : []);
-const files = [...walk(path.join(root, "countries")), ...walk(path.join(root, "articles")), path.join(root, "changes.yaml")];
+const files = [...walk(path.join(root, "countries")), ...walk(path.join(root, "articles")), ...walk(path.join(root, "culture")), path.join(root, "changes.yaml")];
// folded YAML breaks sentences across lines, so compare with the whitespace flattened
const texts = files.map((file) => [path.relative(root, file), fs.readFileSync(file, "utf8").replace(/\s+/g, " ")]);
const CLAIMS = [
+ { claim: /taxed(?: it)? from\s+\[?1798/i, why: "Bengal’s 1798 Board letter recommended taxes; culture pages must not turn it into enacted taxation (#182)" },
{ claim: /India taxed cannabis from \[?1798/i, why: "the 1798 record is a Board recommendation, including Egypt-article cross-references (#145)" },
{ claim: /Before Law 242 Panama had no lawful cannabis of any kind/i, why: "the 2021 framework did not establish operational supply (#128)" },
@@ -329,2 +330,15 @@ test('India federal bhang exclusion does not override state prohibition (#180)',
assert.equal(india.recreational, 'death');
});
+
+
+test('San Marino distinguishes authorised flower from verified dispensing (#187)', () => {
+ const sm = country('sm');
+ assert.doesNotMatch(sm.context.products.text + sm.context.medical.text, /no flower/i);
+ assert.match(sm.context.products.text, /medical framework includes cannabis flower/);
+ assert.match(sm.context.products.text, /dispensing of flower has not been verified/);
+ assert.match(sm.context.medical.text, /home growing remains illegal/);
+ assert.match(sm.sources.l113english.locator, /Articles 1–3 and 9–11/);
+ assert.match(sm.sources.asccfaq.locator, /Q3/);
+ assert.equal(sm.medical, 'limited');
… 3 more linessrc/content.config.ts +2 −1
Not shown: a picture, a generated file, or past the length shown per change.
src/lib/price-selection.js +3 −0
Not shown: a picture, a generated file, or past the length shown per change.
src/pages/charts.astro +4 −4
Not shown: a picture, a generated file, or past the length shown per change.