Methodology

Each statement opens its document, with the passage highlighted

Each statement in an answer cites the original document: the study, FDA's review, the label, the report or the court opinion. Clicking the citation opens the document beside the answer with the supporting passage highlighted. Studies, FDA reviews and communications, label changes and court opinions are read once by a model that quotes them verbatim; code then checks that every quote appears in the source and every number appears in its quote. Those findings are labeled "Read by AI" and show the quote they rest on.

Each date carries its type

Where FDA held a document before it was public, the answer reads "Document dated [date]; public from [date]".

A study describing pigmentary maculopathy in patients taking pentosan polysulfate sodium appeared in Ophthalmology in 2018.^[1] The current Elmiron label warns of retinal pigmentary changes under Warnings.^[2] The interval between those two dates answers "what was knowable, and when", and it can be stated only if each date says what kind of date it is. The Elmiron analysis gives the full chronology. The same dated record gives the label version in force on a date the history held covers, such as the day a client started a drug, and says so when it does not.

Calculated figures come from a stored query run on a named snapshot

A calculated number in an answer is the result of a query over a dated snapshot of the data. The card names the query, its filters and the snapshot, and carries "Data as of" with its date. For a concept such as "eye damage", the answer lists the terms that make it up.

The Expert pack is a zip of the answer:

• exhibits.json: every card in the answer, each stamped with its data snapshot, code version and method version
• the methods appendix, in Markdown and PDF, which names the source paper of each measure and lists only the measures, sources and rules the answer used
• VERIFY.md: the command that opens each named snapshot read-only, checks its sha256, reruns each eligible query with its recorded arguments and compares the numbers, counts exactly and decimals to a relative difference of 1e-9

The check compares each exhibit's data, not every number in the written answer. Document, case-law and literature cards, and live lookups, are listed as not checked, because their figures do not come from a snapshot. Running it needs the code at the version named in the appendix, with its Python environment, and the data snapshot each exhibit names.

PRR, ROR, IC and EBGM, each with its interval

For one drug and one reaction, the signal card gives four disproportionality measures: the proportional reporting ratio (PRR), the reporting odds ratio (ROR), the information component (IC) and the empirical Bayes geometric mean (EBGM), the last being FDA's Bayesian shrinkage method. PRR, ROR and IC come with a 95% interval. EBGM comes with EB05 and EB95, its 5th and 95th percentiles, a central 90% interval. The card says whether the reports meet the signal threshold, as of the data date. The quarterly history chart plots PRR or EB05. It is a retrospective calculation: each quarter uses the current snapshot, with every case counted from the date FDA first received it, so a follow-up that adds a reaction later still counts from the first day. It does not rebuild the reports as FDA held them on each earlier date.

By default each measure compares the drug with all other reports. It can be restricted to drugs of the same indication or of the same ATC level-4 class, to reduce confounding by indication.

The measures count reports. A high value flags a drug and reaction for review, and FDA states that a report does not establish causation.^[3]

Duplicate reports: a sensitivity check by record linkage

FDA's page says its adverse event data contain duplicate and incomplete reports.^[3] FAERS keeps one report per case id, but one patient and event can arrive under several case ids: the manufacturer and the patient's doctor both report it, two manufacturers report one published case, or a follow-up is filed as a new case.

Counts stay one report per case id. For the reports behind a signal, Kelsey compares reports with different case ids field by field (age, sex, event date, country, drugs, reactions, sender, manufacturer report number, literature reference, weight, outcomes, reporter occupation), using Fellegi-Sunter record linkage, the method behind WHO-UMC's vigiMatch. The field weights were estimated from FAERS itself. A chain of likely duplicates counts as one cluster, of which the earliest received report is kept. The signal card then shows how the measures move without the likely duplicates.

Lawyer-filed reports, counted separately

Reports filed by lawyers are flagged. The measures leave them out by default, and the card gives their number beside the measures. A signal can be shown with them included, so the two versions sit side by side. When studies point both ways, the answer says so: "Studies disagree: N report an association; M found none or report evidence against it."

The four finding states

Every finding carries one of four states:

An earliest date is called earliest only when its source was searched completely: "Earliest public evidence found: [item] [date] ([source], searched completely)." Otherwise the answer reads "Earlier public evidence found ... so an earlier one may exist." An empty PubMed search reads "The PubMed search for [terms] through [date] returned no studies."

Limits of each source family

Adverse-event reports (FAERS)

FDA lists the limits of its own data.^[3]

• Duplicate and incomplete reports are in the system.
• The existence of a report does not establish causation. The event may have been related to the underlying disease, to another drug being taken, or to other causes.
• Information in reports has not been verified.
• Rates of occurrence cannot be established from reports.

FDA says the data by themselves are not an indicator of the safety profile of the drug. FDA's pages now use the name Adverse Event Monitoring System (AEMS) and show FAERS as the former name.^[3]

Device reports (MAUDE)

FDA says MDR data are not intended to evaluate rates of adverse events, to evaluate a change in event rates over time, or to compare event rates across devices. A report does not necessarily show that the device caused or contributed to the event. Certain information is withheld from public view under FOIA and replaced with "(b)(4)" or "(b)(6)". The downloadable files combine voluntary reports from June 1993, user facility reports from 1991, distributor reports from 1993 and manufacturer reports from August 1996.^[4]

Labels (DailyMed)

The "in use" labeling on DailyMed may not be identical to the most recent FDA-approved labeling, may not have been verified by FDA, and is not reviewed by the National Library of Medicine before publication.^[5] Kelsey keeps the label versions it holds or has fetched, with their dates, and answers a question about an earlier date from the version in force then, or says that the history does not reach that date. A version shows what DailyMed published, not which labeling came with a patient's dispensed product.

Studies and trials

Kelsey cites the published study or trial record and its date. A study's publication date marks when that study appeared, which can be later than the first report of the risk elsewhere. A trial's registry entry and its posted results are separate records, and the answer says which one it cites.

Regulators abroad

EMA, MHRA, Health Canada, Swissmedic and Canada Vigilance are cited with the date each acted or published. Health Canada's page says its adverse reaction database should not be used on its own to evaluate a health product's safety profile.^[6] A comparison of when a foreign regulator acted with when FDA acted states both dates and both kinds of action.

Court opinions and statutes

Kelsey cites an opinion or statute as written, with its date. A ruling shows how one court treated one record.

Watch lists and alerts

Choose a drug or a condition, optionally a side effect, and a look-back window. Each check compares the result with the previous one and adds one alert listing the evidence added or removed on the Watch lists page: a new reporting signal, a new study, a regulator acting abroad or a label change, each linked to its source. The first check stores the result as the baseline and raises no alert.

Sources

1. Pearce WA, Chen R, Jain N. Pigmentary Maculopathy Associated with Chronic Exposure to Pentosan Polysulfate Sodium. Ophthalmology. 2018 Nov;125(11):1793-1802. doi:10.1016/j.ophtha.2018.04.026. PMID 29801663. https://pubmed.ncbi.nlm.nih.gov/29801663/
2. ELMIRON (pentosan polysulfate sodium) capsules, label, Warnings: Retinal Pigmentary Changes. DailyMed, set id f0ba651e-3d8a-11df-8fbe-119855d89593. https://dailymed.nlm.nih.gov/dailymed/lookup.cfm?setid=f0ba651e-3d8a-11df-8fbe-119855d89593
3. FDA, Adverse Event Monitoring System (formerly FAERS) Public Dashboard, limitations of the data. https://www.fda.gov/drugs/fdas-adverse-event-reporting-system-faers/fda-adverse-event-reporting-system-faers-public-dashboard
4. FDA, About the Manufacturer and User Facility Device Experience (MAUDE) Database, limitations of medical device reports. https://www.fda.gov/medical-devices/mandatory-reporting-requirements-manufacturers-importers-and-device-user-facilities/manufacturer-and-user-facility-device-experience-database-maude
5. National Library of Medicine, DailyMed, About: Labeling on DailyMed and FDA-Approved Labeling. https://dailymed.nlm.nih.gov/dailymed/about-dailymed.cfm
6. Health Canada, Canada Vigilance Adverse Reaction Online Database. https://www.canada.ca/en/health-canada/services/drugs-health-products/medeffect-canada/adverse-reaction-database.html