Building a Scalable Data Ingestion Architecture | Komodo Health
Research: New Colorectal Cancer Diagnoses Fall by One-Third and Colonoscopies Drop 90%

Building a Scalable Data Ingestion Architecture

By Usha Periyanayagam, MD, MPH, and Lucy Chen

At Komodo Health, our Data Ingestion team is responsible for quickly incorporating new data sources into our expanding Healthcare Map™.  One of our engineers, Johannes Leppä, PhD, gave a talk at the DataCouncil conference in San Francisco in April about how we build our scalable data ingestion architecture and some of the lessons learned along the way.   Some of the technologies we use are Spark, Airflow and Kuberenetes.

Here is the video of that talk

Enjoy!  If you are interested in working on this and many other interesting technical problems while helping reduce the global burden of disease we are hiring for a number of roles.

New at Komodo Health

As healthcare providers (HCPs) and patients grapple with fears, policies, and practices during COVID-19, cancer patients are reporting significant disruptions […]
Read More →

Deferral of care during COVID-19 has emerged as a growing concern for healthcare professionals and public health leaders. We’ve seen […]
Read More →

Meet Jay Nallapothula, a Senior Full Stack Engineer and one of Komodo Health’s most tenured dragons. He describes the company’s […]
Read More →

Ready to find out more?
We'd love to hear about what challenges you face and how access to healthcare's ground truth can help.