How do you load years of history from an API that rate limits you?
Separate the backfill from the live sync and give each its own budget. Walk history in fixed time slices, checkpoint after every slice so a failure resumes instead of restarting, and cap the backfill at a share of the rate limit so ongoing sync keeps working. Expect it to take days rather than hours. That is normal, and far safer than saturating the limit and getting the whole integration throttled.
Two pipelines, one shared limit
Rate limits are almost always per account or per app, not per process. So the moment your backfill starts hammering the API, your live sync starts getting throttled, freshness collapses, and someone reports that the dashboard is broken while you are in the middle of a load that will run for two more days.
Give the backfill an explicit budget — a fixed fraction of the allowed request rate — and have it check that budget before every batch. The live pipeline keeps priority. This is basic courtesy to your own system and it is the difference between a load nobody notices and an outage.
Slice, checkpoint, resume
Break history into deterministic slices, usually a month or a week depending on volume. After a slice completes, write a checkpoint. When the process dies — and on a multi-day job it will — it restarts at the next slice rather than at the beginning.
Slices also make the load auditable. You can count records per slice, compare against the source's own totals for that period, and see immediately which month is short.
Load newest first
Chronological order feels natural and it is usually the wrong choice. Loading newest-first means the most useful data is available first, the team can start validating within hours instead of days, and if you decide two years of history is enough you simply stop. Loading oldest-first means nothing is usable until the whole job finishes.
Old records are not shaped like new ones
Document those boundaries as part of the load, not afterward. Every trend analysis built on top of the backfill depends on knowing where the data changes shape. Where an official bulk export or report endpoint exists, prefer it — it is usually orders of magnitude cheaper against the limit than walking records one page at a time, and we look for one early in any integration build.
- Fields that did not exist. Custom fields added last year are null for everything before that, and a report that treats null as zero will show a fake trend.
- Vocabulary that changed. Job types, business units and lead sources get renamed. The historical rows carry the old labels.
- Process changes. A company that started requiring an itemized invoice two years ago has structurally different data on each side of that date.
Topics: backfill · rate limits · ETL · historical data
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.