Which health and medical APIs work with no key?

Nine were checked with live requests on July 27, 2026: DailyMed, RxNorm and RxNav, ClinicalTrials.gov, openFDA drug labels, the WHO Global Health Observatory, PubChem PUG REST, the CMS NPI Registry, the public HAPI FHIR R4 test server and disease.sh. None required a key, a signup or a card. The HAPI FHIR server is a public test instance that anyone can write to and is listed as a protocol example, not as a data source.

How do I know if a health API record changed after I stored it?

Usually not from the record itself. You need a separate history endpoint, and only some providers publish one: DailyMed at /spls/{setid}/history.json, ClinicalTrials.gov at /api/int/studies/{nct}/history, and any FHIR server at resource/{id}/_history. Compare the number of versions rather than the value of a field, because a version field can move without the history growing. openFDA, the NPI Registry, PubChem, the WHO GHO and disease.sh published no history endpoint at all when checked on July 27, 2026.

Is spl_version the number of times a DailyMed drug label was revised?

No. In a sample of 80 setids taken on July 27, 2026, testing spl_version greater than 1 flagged 56 records as rewritten while the actual history endpoint showed 49, a disagreement on 7 of the 80. Three sampled records carried spl_version 100 with exactly one version in their history, and one carried spl_version 21 with eight versions in its history. The field is a publisher's own counter and is not comparable across setids.

Can I fetch an older version of a DailyMed SPL over the API?

No. Requesting the same document with ?spl_version=1, 3, 8 and 999 on July 27, 2026 returned four byte-identical responses of 266,520 bytes sharing md5 4ae91da54b97, every one of them carrying versionNumber 8 and HTTP 200, so the parameter is accepted and ignored without any error. The path form /spls/{setid}/3.xml returned HTTP 200 with 75,305 bytes of the DailyMed HTML homepage rather than a document. The history endpoint tells you which versions exist and when they were published, but the documents themselves are not retrievable. The same check repeated on ten further multi-version setids from a different index page returned an identical md5 for version 1 and for the latest version in all ten cases.

Why does an RxNorm rxcui return an empty object with HTTP 200?

Because the properties.json endpoint answers identically for three different situations. On July 27, 2026 an obsolete concept, a remapped concept and an identifier that never existed all returned HTTP 200 with a 2-byte empty object, while an active concept returned 270 bytes, giving 2 distinct responses for 4 distinct realities. The historystatus.json endpoint on the same identifiers returned four different answers: Active, Obsolete, Remapped and UNKNOWN. The information exists, it is simply not in the endpoint most clients call.

What is the right primary key for a scraped medical record?

The pair (id, version) plus the timestamp of your own observation. A setid, an rxcui or an NCT number identifies a folder rather than a document, which DailyMed makes literal: the setId element stays constant while the id element changes with every version, and openFDA ships both an id and a set_id whose values differ. Keying on the identifier alone makes a record that was rewritten seven times indistinguishable from one that was never touched.

Should I upsert scraped rows by source id?

Only when you are rewriting your own unfinished run, such as re-fetching rows after a crashed job, where an upsert correctly removes your duplicate. For a record the source can change, an upsert keyed on id alone deletes the only evidence that it changed. Key on (id, version) and append instead: replaying the same visits after a crash still produces no duplicates, because the deduplication comes from the key rather than from the upsert, and the version history survives.

Jul 29, 2026

9 Keyless Health APIs: You Asked for v1, You Got v8

I asked a keyless drug-label endpoint for version 1 of a document. Then version 3, version 8, and version 999. Here is what came back on July 27, 2026:

?spl_version=1    HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8
?spl_version=3    HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8
?spl_version=8    HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8
?spl_version=999  HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8

Four requests, one document, one md5. Not one of those lines is an error. The parameter is accepted, ignored, and never mentioned again.

And there really are eight versions under that id. The API will happily list them for you. It just will not serve you any of them except the newest.

The one idea

A keyless health API returns drug labels, trial registrations, provider records or compound data over plain HTTP with no key, no signup and no card. I checked nine of them with live requests on July 27, 2026, and the table is further down. The thing most of them share is not a missing field. It is that the id you store is stable while the record underneath it is not, and the response gives you no way to notice.

So the row in your database is not a fact about a drug label. It is a draft that the publisher can rewrite behind your back, under the same id, and your copy will keep looking exactly as correct as it did the day you fetched it.

The whole proof, and the controls that had to fail

Standard library, no key, no account. Copy it and you will get the same shape I did.

# runnable, standard library only, needs network. No key, no signup, no card.
import hashlib, json, re, urllib.request, urllib.error

DM = "https://dailymed.nlm.nih.gov/dailymed/services/v2"
RX = "https://rxnav.nlm.nih.gov/REST"
A = "10d36b32-0202-4c03-8a1a-cd5d53f231e7"   # one drug-label folder
B = "0cb0ba76-067f-46c4-a5d8-b585d5ecafe3"   # a different one, for controls
ZEROS = "00000000-0000-0000-0000-000000000000"

def fetch(url):
    try:
        with urllib.request.urlopen(url, timeout=30) as r:
            return r.status, r.read()
    except urllib.error.HTTPError as e:                 # fail loud, never silent
        return e.code, e.read()

def versions(setid):
    st, body = fetch(f"{DM}/spls/{setid}/history.json")
    return st, len(body), json.loads(body)["metadata"]["total_elements"], body

def version_in(doc):                                    # <versionNumber value="8"/>
    m = re.search(rb'<versionNumber\s+value="([^"]+)"', doc[:4000])
    return m.group(1).decode() if m else "<<ABSENT>>"

print("1. HOW MANY VERSIONS LIVE UNDER ONE id")
st, n, total, body = versions(A)
print(f"   setid {A[:8]}...  HTTP {st}, {n} bytes -> {total} versions")
for e in json.loads(body)["data"]["history"]:
    print(f"       v{e['spl_version']:<4} {e['published_date']}")
st, n, total_b, _ = versions(B)                         # control: data vs OTHER data
print(f"   CONTROL other real setid {B[:8]}... -> HTTP {st}, {n} bytes, {total_b} versions"
      f"  [{'PASS: the probe reads the document, not a constant' if total_b != total else 'FAIL'}]")
st, n, total_z, body_z = versions(ZEROS)                # an id that never existed
print(f"   an all-zeros setid       -> HTTP {st}, {n} bytes, {total_z} versions")
print(f"       body: {json.dumps(json.loads(body_z)['data'])[:74]}")

print("\n2. ASK FOR AN OLD VERSION. FINGERPRINT WHAT COMES BACK.")
seen = {}
for want in ("1", "3", "8", "999"):
    st, doc = fetch(f"{DM}/spls/{A}.xml?spl_version={want}")
    md5 = hashlib.md5(doc).hexdigest()
    seen.setdefault(md5, []).append(want)
    got = version_in(doc)
    note = "as asked" if got == want else f"<-- asked {want}, served {got}"
    print(f"   ?spl_version={want:<4} HTTP {st}  {len(doc):>7} B  md5 {md5[:12]}  "
          f"versionNumber={got:<11}{note}")
print(f"   distinct documents behind those 4 requests: {len(seen)}")
st, other = fetch(f"{DM}/spls/{B}.xml")                 # control: data vs OTHER data
print(f"   CONTROL a different setid -> {len(other):>7} B  md5 "
      f"{hashlib.md5(other).hexdigest()[:12]}  "
      f"[{'PASS: md5 can tell two documents apart' if hashlib.md5(other).hexdigest() not in seen else 'FAIL'}]")
st, doc = fetch(f"{DM}/spls/{A}/3.xml")
kind = "HTML homepage, not a document" if b"<!DOCTYPE html" in doc[:200] else "a document"
print(f"   path /3.xml       HTTP {st}  {len(doc):>7} B  "
      f"versionNumber={version_in(doc):<11}<-- {kind}")
impossible = b'<zzqqNotATag991 value="7"/>'
print(f"   CONTROL matcher on a tag that cannot exist -> "
      f"{version_in(impossible)}  [PASS: the matcher can miss]")
print("   Not one line above is an error. Every line is HTTP 200.")

print("\n3. 'OBSOLETE' AND 'NEVER EXISTED' ARE THE SAME RESPONSE")
cases = [("1801289", "in use"), ("105078", "obsolete"),
         ("351772", "remapped"), ("999999999", "never existed")]
print(f"   {'rxcui':<12}{'my label for it':<18}{'properties.json':<26}historystatus.json")
shapes = {}
for rxcui, what in cases:
    st, body = fetch(f"{RX}/rxcui/{rxcui}/properties.json")
    plain = f"HTTP {st}, {len(body)} bytes"
    _, hb = fetch(f"{RX}/rxcui/{rxcui}/historystatus.json")
    status = json.loads(hb)["rxcuiStatusHistory"]["metaData"]["status"]
    print(f"   {rxcui:<12}{what:<18}{plain:<26}status={status}")
    shapes.setdefault(plain, []).append(what)
for group in (v for v in shapes.values() if len(v) > 1):
    print(f"   COLLAPSED into one identical response: {', '.join(group)}")
print(f"   properties.json    -> {len(shapes)} distinct answers for 4 distinct realities")
print(f"   historystatus.json -> 4 distinct answers for the same 4 ids")

Output on my machine, July 27, 2026, pasted as it came out:

1. HOW MANY VERSIONS LIVE UNDER ONE id
   setid 10d36b32...  HTTP 200, 946 bytes -> 8 versions
       v8    Jun 26, 2026
       v7    Nov 22, 2024
       v6    Dec 15, 2023
       v5    Oct 07, 2022
       v4    Jun 29, 2021
       v3    Jul 31, 2020
       v2    May 07, 2019
       v1    Aug 01, 2017
   CONTROL other real setid 0cb0ba76... -> HTTP 200, 619 bytes, 1 versions  [PASS: the probe reads the document, not a constant]
   an all-zeros setid       -> HTTP 200, 433 bytes, 0 versions
       body: {"spl": {"title": "", "setid": ""}, "history": []}

2. ASK FOR AN OLD VERSION. FINGERPRINT WHAT COMES BACK.
   ?spl_version=1    HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8          <-- asked 1, served 8
   ?spl_version=3    HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8          <-- asked 3, served 8
   ?spl_version=8    HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8          as asked
   ?spl_version=999  HTTP 200   266520 B  md5 4ae91da54b97  versionNumber=8          <-- asked 999, served 8
   distinct documents behind those 4 requests: 1
   CONTROL a different setid ->   95825 B  md5 1dc13a19afa1  [PASS: md5 can tell two documents apart]
   path /3.xml       HTTP 200    75305 B  versionNumber=<<ABSENT>> <-- HTML homepage, not a document
   CONTROL matcher on a tag that cannot exist -> <<ABSENT>>  [PASS: the matcher can miss]
   Not one line above is an error. Every line is HTTP 200.

3. 'OBSOLETE' AND 'NEVER EXISTED' ARE THE SAME RESPONSE
   rxcui       my label for it   properties.json           historystatus.json
   1801289     in use            HTTP 200, 270 bytes       status=Active
   105078      obsolete          HTTP 200, 2 bytes         status=Obsolete
   351772      remapped          HTTP 200, 2 bytes         status=Remapped
   999999999   never existed     HTTP 200, 2 bytes         status=UNKNOWN
   COLLAPSED into one identical response: obsolete, remapped, never existed
   properties.json    -> 2 distinct answers for 4 distinct realities
   historystatus.json -> 4 distinct answers for the same 4 ids

A word about the controls, because I nearly shipped a weak one. My first version checked the history probe against a made-up setid and got zero versions back, which felt like proof the probe worked. It is not. That control only shows the probe can tell data from nothing. It says nothing about whether the probe can tell data from other data, which is the thing I was actually claiming. So both controls now compare two real documents: a second setid with a different version count, and an md5 that demonstrably differs across two real files before I trust it matching four times.

The all-zeros line is worth a second look on its own. A setid made of nothing but zeros returns HTTP 200 with a complete envelope and an empty history array. So “this record never existed” and “this record has no history” arrive as the same successful response. That is the article’s problem viewed from the other end.

How often does a label actually get rewritten?

Often enough that it is not an edge case. I sampled 80 setids from 8 random pages of the live index (seed 63, out of 158,178 records) and pulled the history of each:

history says rewritten (>1 version) : 49/80 = 61.3%
naive test spl_version > 1          : 56/80 = 70.0%
the two disagree on                 : 7/80 = 8.8%

Roughly six in ten of the labels in that sample had been rewritten at least once under an id that never moved.

Now look at the second line, because that is the test you would write. spl_version sits right there in the list response, so counting rows where it exceeds 1 looks like a free answer. It disagrees with the real history on 7 of the 80, always in the direction of overcounting.

spl_version is not a revision count. From that same sample:

spl_version=101     history has 2   version(s)   5c8cebcd...  PROPAFENONE HYDROCHLORIDE TABLET
spl_version=100     history has 1   version(s)   b4b7b936...  ENALAPRIL MALEATE TABLET
spl_version=100     history has 1   version(s)   c83daa7b...  DANTROLENE SODIUM CAPSULE
spl_version=100     history has 1   version(s)   1ebeb8ec...  RIVASTIGMINE TRANSDERMAL SYSTEM
spl_version=21      history has 8   version(s)   d1468c0f...  SODIUM CHLORIDE INJECTION

Three records carrying version 100 with exactly one version in their history. One carrying version 21 with eight. The number is a publisher’s own counter, and comparing it across setids means nothing. The titles above are record labels and nothing more; none of this says anything about the products themselves.

I want to be blunt about how I got here, because the wrong version of this section nearly went out. My first pass measured the rewrite rate with spl_version > 1 and produced a confident percentage. The very next probe killed the method: a record at version 100 with a single entry in its history proves the field does not mean what the name suggests. I threw the number away rather than publish it, and the 49 of 80 above comes from counting actual history entries instead.

The nine endpoints, checked live

Every row is a request I made on July 27, 2026 with no key, no account and no card. The last column is the point of the table: what, if anything, tells you the record has a past.

#	Provider	Keyless endpoint	What came back	Version signal
1	DailyMed (NLM)	`dailymed.nlm.nih.gov/dailymed/services/v2/spls/{setid}/history.json`	158,178 SPLs in the index; the sampled setid returned 8 versions in 946 bytes	full history, but old documents are not served
2	RxNorm / RxNav (NLM)	`rxnav.nlm.nih.gov/REST/rxcui/{id}/historystatus.json`	`Active`, `Obsolete`, `Remapped`, `UNKNOWN` on the four ids I tried	status only, and only on this endpoint
3	ClinicalTrials.gov	`clinicaltrials.gov/api/int/studies/{nct}/history`	NCT00001372 returned 395 versions in 86,963 bytes; NCT02576457 returned 27	full history with a status per version
4	openFDA drug/label	`api.fda.gov/drug/label.json?limit=1`	200, 3,933 bytes, 260,986 labels in the index, `version: 2`, and `id` differs from `set_id`	version number, no history
5	WHO GHO	`ghoapi.azureedge.net/api/WHOSIS_000001?$top=1`	200, 689 bytes, 26 leaf fields, 6 of them time related	none: every time field describes the observation, not the record
6	PubChem PUG	`pubchem.ncbi.nlm.nih.gov/rest/pug/compound/name/{name}/property/.../JSON`	200, 224 bytes, 4 leaf fields	none
7	NPI Registry (CMS)	`npiregistry.cms.hhs.gov/api/?version=2.1&number={npi}`	npi 1578736112: `enumeration_date` 2008-04-03, `last_updated` 2026-01-05	one timestamp, sitting 18 years after enumeration
8	HAPI FHIR R4 (public test server)	`hapi.fhir.org/baseR4/Patient/{id}/_history`	`_history` total 2: v1 POST at 13:06:56.678, v2 PUT at 13:06:57.560	versioning is in the protocol
9	disease.sh	`disease.sh/v3/covid-19/all`	200, 478 bytes, 21 keys, one of them time related (`updated`, epoch ms)	none

Nine services, no keys. Three of them list a record’s past version by version: DailyMed, ClinicalTrials.gov, and any FHIR server. Two more hint at it without enumerating anything, and they hint differently: RxNorm tells you a concept was retired or remapped, openFDA ships a version number with nothing behind it. NPPES gives you two dates eighteen years apart and nothing in between, so you can see that something happened without seeing how many times. The remaining three say nothing at all.

The same provider, two endpoints, two different truths

Block 3 of the output is the one I would tape to a wall. RxNorm’s properties.json returns 2 distinct responses for 4 distinct realities. An obsolete concept, a remapped concept and an id nobody ever issued all come back as HTTP 200 with a 2-byte body, {}.

Think about what that does to a pipeline. Your nightly job re-checks the ids you stored. One of them starts returning {}. There is no status code to branch on, no error field, no message. From inside the client, “this identifier was retired” is indistinguishable from “you sent garbage”, and both are indistinguishable from a typo in your own query.

The information is not missing. It is one endpoint away. historystatus.json answers all four cases separately and tells you the retirement month while it is at it. Nothing forces you to call it, and nothing warns you that you did not.

ClinicalTrials.gov shows the same split with a sharper edge. The public v2 endpoint gives you the current record. The history endpoint gives you every version, and for NCT02576457 the status field walks NOT_YET_RECRUITING to RECRUITING to ACTIVE_NOT_RECRUITING to TERMINATED across 27 versions between 2015-10-14 and 2017-09-14. If you stored that row in 2016 and keyed it by NCT id, you are holding a value the source stopped agreeing with nine years ago, and your row looks fine.

This is not the `latest` problem, and the usual fix does not apply

A few days ago I wrote about package registries and why you cannot pin latest. The prescription there was clear: never dereference a mutable alias in a build, resolve it once, and pin an exact version plus an integrity hash.

I want to be direct about this, because it is my own advice and it does not survive contact with these APIs.

On a package registry, version 1.2.3 is immutable and only the pointer moves, which is precisely why pinning rescues you. Here the pointer is all there is. DailyMed accepts ?spl_version=1 and serves you version 8 with no complaint, so there is nothing to pin. The path form /{setid}/3.xml is worse: HTTP 200, 75,305 bytes, and what arrives is the DailyMed homepage in HTML with no <document> element anywhere in it. RxNorm’s answer to an old identifier is {}. You can learn that version 3 existed and when it was published. You cannot obtain it.

So the two posts are not the same post. There, the archive exists and you have to aim at it correctly. Here the archive is announced and withheld, which means the only copy of version 3 you will ever have is the one you took on the day it was current.

The fix, and why my usual advice about upsert is wrong here

I have also argued, in a post about a scraper that died at row 12,000, that an upsert beats a blind append. I stand by that in its own context and it is the exact wrong move here, so let me draw the line rather than leave the two posts arguing with each other.

Upsert is right when you are rewriting your own unfinished run. The job crashed, you restart it, you re-fetch rows you already had, and you want one copy of each. Upsert is wrong when you are rewriting someone else’s changed record, because then the row you overwrite was evidence, and after the overwrite there is no trace that the source ever said anything different.

The distinction is not upsert versus append. It is what you put in the key.

# runnable, standard library only, no network, no keys.
# Replays the two real histories printed above: folder A was rewritten 7 times,
# folder B never was. A crawler happened to visit on each publication day.

A = [(8, "2026-06-26"), (7, "2024-11-22"), (6, "2023-12-15"), (5, "2022-10-07"),
     (4, "2021-06-29"), (3, "2020-07-31"), (2, "2019-05-07"), (1, "2017-08-01")]
B = [(1, "2026-07-24")]

def visits(setid, history):
    """What the crawler stores on each visit: whichever version was current."""
    return [(setid, ver, day) for ver, day in reversed(history)]

crawl = visits("A", A) + visits("B", B)

upsert, append = {}, {}
for setid, ver, day in crawl:
    upsert[setid] = (ver, day)                 # keyed by id alone
    append[(setid, ver)] = day                 # keyed by (id, version)

def rewrites(store, setid, keyed_by_version):
    rows = ([k for k in store if k[0] == setid] if keyed_by_version
            else [k for k in store if k == setid])
    return len(rows) - 1

print("after the crawl:")
print(f"  upsert by id        A: {sum(1 for k in upsert if k == 'A')} row   "
      f"B: {sum(1 for k in upsert if k == 'B')} row   "
      f"-> rewrites detected: A={rewrites(upsert,'A',False)}, B={rewrites(upsert,'B',False)}")
print(f"  append by (id, ver) A: {sum(1 for k in append if k[0]=='A')} rows  "
      f"B: {sum(1 for k in append if k[0]=='B')} row   "
      f"-> rewrites detected: A={rewrites(append,'A',True)}, B={rewrites(append,'B',True)}")

same = rewrites(upsert, "A", False) == rewrites(upsert, "B", False)
print(f"\n  the upsert store answers 'was this rewritten?' identically for a folder"
      f"\n  rewritten 7 times and one never touched: {same}")

# The control that matters: re-run the last visit, the way a crashed job does.
before = len(append)
for setid, ver, day in crawl[-3:]:
    append[(setid, ver)] = day
print(f"\n  replaying the last 3 visits after a crash: {before} -> {len(append)} rows"
      f"  [{'PASS: no duplicates' if len(append) == before else 'FAIL'}]")

# And a check that can fail, so it is a check and not decoration.
def assert_history_survives(store, setid, expected):
    got = len([k for k in store if k[0] == setid])
    if got != expected:
        raise AssertionError(f"{setid}: kept {got} versions, source published {expected}")

assert_history_survives(append, "A", len(A))
try:
    assert_history_survives({("A", 8): "2026-06-26"}, "A", len(A))
except AssertionError as e:
    print(f"  the same assertion on an upserted store -> {e}")

Output, same machine, July 27, 2026:

after the crawl:
  upsert by id        A: 1 row   B: 1 row   -> rewrites detected: A=0, B=0
  append by (id, ver) A: 8 rows  B: 1 row   -> rewrites detected: A=7, B=0

  the upsert store answers 'was this rewritten?' identically for a folder
  rewritten 7 times and one never touched: True

  replaying the last 3 visits after a crash: 9 -> 9 rows  [PASS: no duplicates]
  the same assertion on an upserted store -> A: kept 1 versions, source published 8

Read the middle block. The upsert store gives the same answer, zero rewrites, for a folder that was rewritten seven times and for one that was never touched. That is not a small loss of detail. It is the destruction of the only difference that mattered, and the store cannot even report that it happened.

Now read the line after it. Replaying the last three visits, the way a crashed job replays them, leaves the row count unchanged. The deduplication came from the key, not from the upsert. Once (id, version) is the key, the crash-safety argument from the row-12,000 post still holds and you keep the history as well. The two posts only conflict if you let the key stay at id.

The last line is the assertion doing its job in the direction that matters: fed an upserted store, it fails loudly and names the gap. A check that cannot fail is decoration.

Honest limits

This is a snapshot. Nine services, one machine, one IP, July 27, 2026. I did not read anyone’s terms and did not measure rate limits, which is why there is not a single limit number in this post.
80 setids is a sample, not a census. 80 out of 158,178, from 8 random pages, seed 63. I did not compute a confidence interval, so treat 49 of 80 as an order of magnitude, not a national statistic.
The title generalises from a small check, so here is the check. The four-request fingerprint above is one setid. I then repeated it on ten further multi-version setids drawn from a different page of the index, comparing version 1 against each document’s own latest version: 10 of 10 returned an identical md5 for both. Control: two different setids compared the same way do produce different md5s, so the comparison is capable of showing a difference. Eleven documents is still not the whole index of 158,178, and I have not tested a single one that behaved otherwise.
The rewrite rate is DailyMed only. I did not sample ClinicalTrials.gov or openFDA the same way, so nothing here says how often a trial record or an openFDA label changes.
HAPI FHIR is a public test server that anyone on the internet can write to. The two versions I found were created 0.882 seconds apart, so that is somebody’s test script, not clinical data. I include it because it shows the protocol carries versionId and _history natively, which is the counterexample to everything above, and for no other reason.
A search that returned nothing. Patient?_count=1 on that same server gave me HTTP 200, 455 bytes, a searchset bundle with a next link and no entry array at all. Reading a patient by id worked fine a second later. I have no explanation and I am not going to invent one.
I did not audit disease.sh past the top-level object. It answered with no key and it has no version field in that object. That is all I checked.
openFDA answered normally for me on the one call I made. I did not test it under repetition.
Nothing here is clinical. Every measurement in this post is about the behaviour of an API as a data source. Drug and trial names appear only as record identifiers, and no statement here concerns any product, treatment or study on its merits.
Provenance. I run scrapers and data pipelines in production, 32 published actors and something over 2,000 runs by my own count. The habit behind this post came from enrichment feeds, which lie with a confident 200 far more often than they fail honestly. These nine checks are one sitting, not a production sample, and I am not going to dress them up as one.

The open question

Here is the part I have not solved. DailyMed will tell me that version 3 of a document existed and was published on 2020-07-31. It will not give me version 3. So the only way to ever hold those bytes is to have been crawling on a day when version 3 was current, and to have kept them.

That turns a cheap decision into an expensive one. Storing (id, version, hash, seen_at) costs almost nothing and lets me prove that something changed. Storing the full document on every fetch lets me prove what changed, and at a quarter of a megabyte for the one label I measured, across an index of 158,178, that is a real bill for a question nobody has asked me yet.

I currently keep hashes for everything and full bodies only for the handful of sources I have been burned by. It feels like the wrong line, drawn from memory of past incidents rather than from anything principled.

If you pull records from a source that rewrites them in place: do you snapshot every fetch, or keep only the fingerprint and accept that you can prove a change without ever being able to show it? Tell me in the comments, I read every one. 👇

Written with AI assistance. Every status code, byte count, md5, row count and command output above comes from my own live requests and negative controls on July 27, 2026, printed as they came out.