Key Takeaways
- One way to test AI-enabled RWE is to have a human analyst implement the same protocol independently and compare results. We ran that test on Marmot™ in a rare disease cohort.
- Both pipelines produced identical counts at all eight cohort selection steps (30,221 patients down to 1,130) and identical age, region, and race and ethnicity distributions. 16 of 17 Charlson Comorbidity Index (CCI) components matched. Payer mix, one CCI component, and mean CCI score differed.
- The differences traced to derived-variable implementation, not patient selection. Agreement at one step does not establish correctness at the next, so specifications and human review matter at every step.
If an AI-enabled RWE workflow runs a study, would a human analyst working independently from the same protocol get the same answer? Earlier pieces in this series argued that validating AI-enabled RWE means evaluating every decision in the reasoning chain. This study tests one practical way to do that. My colleagues and I compared Marmot’s analysis of a non-ambulatory Duchenne muscular dystrophy (DMD) cohort with an independently built manual analyst pipeline. We presented the results as a poster at BioTechX 2026¹.
How Was the AI vs. Human Analyst Study Designed?
- Population: non-ambulatory DMD patients, index date November 14, 2025
- Data: Komodo Research Dataset, October 2018 through November 2025
- Design: two pipelines, one eight-step protocol, built by separate implementers blinded to each other’s outputs; a third reviewer adjudicated discrepancies
- Compared: counts at each selection step, baseline demographics, all 17 CCI components, and three repeated Marmot executions
November 14, 2025 is the date the FDA added a boxed warning to Elevidys and limited its indication to ambulatory patients, so the index date aligns the cohort with that regulatory change. The small cohort and multi-step phenotype definition made selection differences important to examine.
Did AI and a Human Analyst Get the Same Results?
Cohort selection: identical. All eight steps produced the same counts, from 30,221 patients at first diagnosis to a final cohort of 1,130 (Figure 1). All three Marmot executions returned identical outputs under the tested conditions.
Demographics: identical except payer. Age, region, and race and ethnicity matched exactly (V=0.000 each; Figure 2). Payer channel differed (V=0.056, p=0.072), with adjudication attributing the difference to payer reclassification, as patients moved across groups.
Comorbidities: 16 of 17 components matched. Diabetes without complications differed by three patients (V=0.007, p=0.75). Mean CCI was 1.5 in the manual analysis and 1.1 in the Marmot analysis; the difference in the score was traced to derived-variable implementation in Marmot.
Figure 1. Cohort Selection Waterfall
Figure 2. Baseline Demographics
Figure 3. CCI Components
What This Means for Validating AI in Real-World Evidence
Identical outputs across three repeated executions support reproducibility under the tested conditions.
A third reviewer traced the discrepancies to derived-variable implementation in Marmot. The hard part was not identifying the right patients. It was specifying derived variables precisely enough that two independent implementations agree, which is where explicit specification and human oversight matter most.
The two pipelines agreed wherever the specification was precise. Where they diverged, human review located the gap, which is the finding that a validation framework exists to surface.
Built for Implementation Concordance
A blinded comparison like this only works if every step can be inspected. In this study, Marmot made step-level outputs available at each decision node, and expert oversight ran from protocol specification through adjudication of intermediate outputs. This study covers cohort selection and baseline characteristics. Validation of downstream analyses, including healthcare resource utilization and cost outcomes, is ongoing. The analysis used Komodo Research Dataset, which covers de-identified individuals across linked medical and pharmacy claims and is enriched with imputed allowed amounts.
Grounded. Explainable. Reproducible. Verified. Governed.
This piece is part of The RWE Standard, Komodo’s column on HEOR and real-world evidence in the age of AI. Continue the series
About the Author
Ashis Das, MD, MPH, PhD is Senior Director of Evidence Intelligence at Komodo Health, where he works at the intersection of clinical medicine, AI, and real-world evidence. A physician and thought leader, he focuses on how the healthcare industry generates and applies evidence to drive better decisions across biopharma and payer stakeholders. Connect with Ashis
Reference
¹Noman A, Park C, Ma X, Wu Z, Das A. Validating Marmot AI Platform for Real-World Evidence Generation: Human and AI Concordance in Duchenne Muscular Dystrophy. Poster presented at: BioTechX; 2026.
Frequently Asked Questions
How Do You Validate AI for Real-World Evidence Generation?
One method is a blinded concordance study. An AI platform and a human analyst each run the same study protocol separately, without seeing each other’s outputs. A third reviewer then resolves any differences. Comparing results at every step, from cohort selection through derived variables, shows where the AI’s reasoning holds up and where the specifications need tightening. Komodo Health used this design to evaluate Marmot in a non-ambulatory Duchenne muscular dystrophy (DMD) cohort and presented the results at BioTechX 2026¹.
Can AI Match a Human Analyst on a Real-World Evidence Study?
In a blinded DMD study, Marmot and a separate human analyst got identical patient counts at all eight cohort selection steps, narrowing 30,221 patients to a final cohort of 1,130¹. Age, region, and race and ethnicity distributions matched exactly. So did 16 of 17 Charlson Comorbidity Index (CCI) components¹. Three repeated Marmot runs returned identical outputs under the tested conditions¹.
Why Did the AI and Human Analyst Results Differ?
The two analyses differed in payer mix, one CCI component (diabetes without complications, by three patients), and mean CCI score: 1.5 in the manual analysis vs. 1.1 in Marmot¹. A third reviewer traced the differences to how derived variables were built in Marmot. Patient selection matched at every step¹. The takeaway: derived variables need precise specifications and human review at each step of an AI-enabled RWE workflow.
