.svg)
Accessing data on Dewey is easy. That's the point.
When a researcher comes to the Dewey platform, they can download the data they need with a few clicks. Their experience is fast, easy, and consistent across providers, always resulting in the most current data.
This post is about data ingest specifically, how data moves from dozens of providers into Dewey. There’s a lot we’re not covering here: how that data gets organized, kept queryable, sliced down to what a researcher actually needs, served back out. Not small footnotes, just topics for another time.
Data librarians already know that ingesting data is one of the hardest steps. What looks like "just receiving a file" is actually a moving target of formats, schemas, and delivery methods, and that’s exactly why it’s such a persistent headache for librarians and data teams everywhere.
The real problem: every data provider is its own puzzle
Onboarding hundreds of datasets from different providers is a data engineering problem, not a data entry problem, and it’s one librarians shouldn’t have to solve alone.
Each data provider ships differently: different file formats, different delivery mechanisms, different update cadences. A delivery that lands as clean Parquet in S3 is the exception, not the rule. Just as often, it’s something we have to pick apart by hand before it’s usable.
Every provider delivers data in a slightly different way which is kind of annoying" - anonymous Dewey data team member
Getting familiar with a new provider's format is step one. We've seen tab-separated files where we expected comma-separated. We've seen schemas change without notice. Spotting the pattern in one dataset is manageable– doing it across 40+ providers, each with its own quirks, is where it becomes a scale problem.
This isn't hypothetical. We hear it constantly, in almost identical words, from data teams outside of Dewey too: a provider drops a massive SFTP delivery, and keeping it current becomes its own part-time job. It's easy for a library to fall behind on ingesting and publishing feeds they're already paying for, simply because one large SFTP dump can eat a week of someone's time, and then the next one lands before the last one's done. The unglamorous daily reality (reconciling schemas, chasing down file drops, re-checking formats every time a provider changes something) doesn't scale with one person, and the stakes aren't small. A missed schema change can cost a researcher's credibility, breach data security, or strain a vendor relationship.
What Dewey actually built to solve data ingest
We built an ingestion tool that runs every provider pipeline through one system instead of a pile of one-off scripts scattered across the team. It runs on an always-on EC2 instance, firing scheduled cron jobs on a fixed early-morning cadence to check Snowflake and shared S3 buckets for new deliveries. When something new lands, the tool identifies the provider, kicks off that provider's ingest script, and runs a set of validation checks before the data is considered usable.
The architecture underneath hasn't changed much; it's roughly the same shape as before. What's changed is how much of it used to require a person watching a terminal. Detecting a new delivery, matching it to the right provider, validating the schema against what we expect, and flagging anomalies used to mean someone manually kicking off a script and eyeballing the output. Now that entire chain, from detection through validation runs automatically and with more rigorous checks in place.
The result for our team is visibility: every pipeline’s status lives in one place, so our lean team can stay on top of data deliveries with more confidence and less risk.
This has unlocked:
- Source-agnostic ingestion. Snowflake, S3, SFTP, GCP, or otherwise. Without one central intake point, we'd have to document file-level quirks separately for every dataset, by hand.
- Complex schemas at scale. A 26-table provider used to tie up our data engineers for a week. A 53-table provider now takes a couple of hours.
Some libraries do build something like this in house, usually leaning on their university's IT department. That's not a question of whether a library could pull it off. It's a question of whether reconciling schemas and babysitting cron jobs is the best use of a data librarian's time, especially once you factor in unpredictable warehouse costs, a pipeline per provider that needs its own ongoing upkeep, and enough SQL fluency on staff to keep it running. Even with all of that in place, most teams still end up without a single, current point of access for researchers, which just pushes the problem downstream instead of solving it. It's less about capability and more about whether that's really where a library's limited time should go.
Data quality isn't a one-time check, it's ongoing
Ingestion isn't a delivery you validate once and forget. Across hundreds of datasets (60+ billion rows!), staying current requires the same rigor every single time a provider pushes an update.
Catching problems at the source
Earlier this year we reorganized our backend file structure so that tracing a data quality issue back to its source no longer means digging through code by hand. Each provider has its own ingest script, seeded from a real sample file and built with checks for that provider's specific failure modes: tab versus comma mixups, schema drift, fields that occasionally show up empty. When something looks off, we can trace it straight to the script and the delivery that caused it, before it ever reaches a researcher's notebook.
Every pipeline is also wired into Sentry for error logging, with Slack alerts firing in real time when a refresh breaks so a fix can go out immediately. Fun fact: 8 of the last 10 alerts hit between the hours of 2am and 9am.
Refreshes that actually stay current
Dependability matters for academic researchers. They shouldn’t have to wonder if they’re looking at the latest version of a dataset. Our system doesn't sit on a refresh for days or weeks waiting for someone to notice.
Take a peek at the command we use for triggering a data provider refresh. It requires a dry run of the command first, shows the row count before and after the load so you can see the change in the dataset, and posts a summary to Slack of the tables loaded.
Across dozens of providers we've started to see the same handful of failure patterns repeat, which means we can build for them once instead of solving each one from scratch. In practice, that means fewer support tickets and fewer “this dataset looks broken” emails, for us and for the librarians relying on us.
Human-orchestrated, AI-accelerated
Our ingestion infrastructure runs on a fixed schedule and watches itself, but the infrastructure is only half of it. Every new provider feed is set up by hand and goes through a QA check from at least two data engineers before it ships. AI has sped up how fast we can stand up and QA a new pipeline, but people still make every call.
Here's specifically where AI is in the loop, and where it isn't:
Any AI we use has read-only access. It can study a file; it can't touch, edit, or move one.
Handling data, without an in-house engineering team
At Dewey, the ingestion pipeline is what powers our subscription: one annual payment that unlocks 40+ providers and hundreds of datasets, all ingested and validated the same way. If your team is still pulling files by hand over SFTP, standing up S3 buckets for vendor drops, or reconciling schemas on your own time, we're built to take some of that work off your plate.
Data ingestion is just step one in getting a dataset into the hands of an academic researcher. Other data engineering problems you may be encountering include:
- Storage — keeping billions of records across hundreds of datasets both current and queryable, without infrastructure costs scaling in lockstep with every new provider
- Filtering — letting thousands of researchers dynamically slice and filter across hundreds of datasets at once, without compute costs spiraling into something that forces us to raise prices.
- Egress — turning "I need this slice of data" into a downloaded, usable file in minutes, no matter what format or size the source data came in.
Which should we break down next?
If you're the primary data manager at your university, or you've hit blockers we might be able to help clear, we'd love to hear from you.
.svg)