Key Takeaways
- Documentation isn’t inspection. Stating that a decision was made doesn’t show whether it was implemented as described.
- Code shows what happened, not whether it was the right choice. Reviewable AI-generated real-world evidence requires the rationale, implementation, and sensitivity to reasonable alternatives.
- Independent review requires more than code. Reviewers need executable logic, record-level traceability, reproducible execution, and sensitivity analyses.
Consider a hypothetical cost-of-care analysis for a rare disease cohort. One patient has a gene-therapy claim billed at $2.3 million. Should that claim be included in the cohort’s cost estimate?
There isn’t necessarily a universal right answer. It depends on the study’s cost definition, its prespecified objective, and whether billed charges or paid amounts are the appropriate measure.
What matters is whether a reviewer can see why the decision was made, how it was implemented, and how the estimate would change under a reasonable alternative.
From Documentation to Inspection
The previous column in this series called for decision-level traceability and explicit documentation of analytic assumptions. The goal is to surface the hidden stack of choices behind an analysis rather than force a regulator or payer to reconstruct them after the fact.
This piece takes the next step: once a decision is documented, how can a reviewer confirm that the documentation matches what actually happened?
A statement that “claims above a prespecified cost threshold were excluded in a sensitivity analysis” tells a reviewer that a rule existed.
It doesn’t show whether the rule was applied as written or how the final estimate would change under a different reasonable threshold. To answer those questions, reviewers need to compare the stated method with the executed code and inspect the resulting records and estimates.
What Reviewers Need
Was the choice appropriate?
Does the rule make sense for the study’s stated research question?
Was it implemented as described?
Does the executed code match the methodology as written?
Would the result change under a reasonable alternative?
How sensitive is the estimate to another defensible threshold, definition, or analytic choice?
Executable code answers the second question directly. It does not answer the first or third on its own. A pipeline can faithfully implement a rule that was not the right rule for the study.
What Makes an AI-Generated RWE Analysis Reviewable?
A reviewable analysis needs four things:
- Code that matches the method. A reviewer should be able to confirm that the executable logic matches the methodology as written. Code supports methodological review; it does not replace it.
- Record-level traceability. An authorized reviewer should be able to trace the reported aggregate back to the records that contributed to it.
- Reproducible execution. The same code, versioned data, and recorded execution settings should reproduce the reported result for an authorized reviewer.
- Sensitivity to alternative choices. A reviewer should be able to see how the estimate changes under another reasonable threshold or definition, not only the result produced by the selected rule.
Built for Independent Review
Marmot™ supports independent review by making SQL, scripts, filters, and code sets available for download alongside the report. These artifacts let a technical reviewer inspect how the analysis was implemented. In the hypothetical cost-of-care example, that review would need to establish how the threshold was applied and assess its effect on the estimate.
Marmot does not decide which threshold or cost definition is appropriate for the study. That remains a methodological judgment for the researcher.
What Marmot provides is a shared artifact for making that judgment reviewable: the same executable logic used to produce the result, rather than a separate description of what the analysis was supposed to do.
This distinction is central to Komodo’s approach to architecting trust in healthcare AI: a confident output is not necessarily a trustworthy one. Trust requires being able to inspect how a result was produced, trace it back to the underlying evidence, reproduce the analysis, and understand the choices that shaped it.
In this example, executable code makes an assumption Explainable, while guardrails for auditability and security create a Governed environment where the reported result can be traced back to the underlying records.
Grounded. Explainable. Reproducible. Verified. Governed.
This piece is part of The RWE Standard, Komodo’s column on HEOR and real-world evidence in the age of AI. Read part 2 here
About the Author
Ashis Das, MD, MPH, PhD is Senior Director of Evidence Intelligence at Komodo Health, where he works at the intersection of clinical medicine, AI, and real-world evidence. A physician and thought leader, he focuses on how the healthcare industry generates and applies evidence to drive better decisions across biopharma and payer stakeholders. Connect with Ashis
Frequently Asked Questions
Why Isn’t Documentation Enough to Validate an RWE Study?
Documentation establishes that a decision was made. Inspection establishes that it was carried out as described. A methods section can state a rule accurately while the executed analysis applies it in a way the text does not capture. Confirming that the stated method and the executed code agree requires access to both, along with the records the rule produced and the estimate it yielded.
Does Access to the Code Validate an RWE Result?
Not on its own. Executable code shows whether a rule was implemented as written. It does not establish that the rule was appropriate for the study’s research question, and it does not show how the estimate would change under another defensible choice. Validation depends on the rationale for the choice, the implementation, and the sensitivity of the result to reasonable alternatives.
What Do Reviewers Need to Validate an AI-Generated RWE Analysis?
A reviewable analysis needs four things: The executable logic should match the methodology as written. An authorized reviewer should be able to trace a reported aggregate back to the records that contributed to it. The same code, versioned data, and recorded execution settings should reproduce the reported result. And the analysis should show how the estimate changes under another reasonable threshold or definition. Code supports methodological review; it does not replace it. Whether a given rule suits the study remains a judgment for the researcher.
