How do you load years of history from an API without hitting rate limits?

Two jobs that look similar and behave differently

A backfill is a bounded project. It runs for hours or days, moves a large volume once, and then is over. An incremental sync is a permanent service that runs on a schedule for years. Writing one piece of code to do both is tempting and usually produces something that is too slow for the backfill and too fragile for the ongoing sync.

Build them as separate paths that write to the same destination through the same idempotent write logic. Then a re-run of either is safe, which is the property that makes everything else recoverable.

The shape of a backfill that finishes

• Slice by date window. Month by month, or week by week for high-volume objects. Each slice is independently retryable.
• Checkpoint after each slice. Record which windows completed. A crash on month nineteen should resume at nineteen, not at one.
• Throttle deliberately below the published limit. Running at the ceiling means any transient burst pushes you over and triggers penalties that cost more time than you saved.
• Run it off-peak. The same rate limit is shared with the vendor's own application, and hammering the API during dispatch hours makes the software slow for the people using it.
• Write idempotently. Overlapping slices and re-runs should update rather than duplicate.

The modified-date rule, and the overlap window

Incremental syncs must filter on last-modified, not created. A job created in March and invoiced in May is a March record with a May change; a created-date filter will never bring you the invoice.

Then add overlap. Ask for changes since slightly before your last successful run rather than exactly at it. Clocks drift, transactions commit after their timestamp is assigned, and boundaries lose records. An overlap of an hour costs almost nothing when your writes are idempotent, and it closes a gap that is otherwise nearly impossible to diagnose. This is the same discipline behind reliable field service data integration.

How far back is far enough

For a seasonal service business, one year of history is not enough to say anything useful. You need at least two full cycles before a year-over-year comparison means more than noise, and more if you want to separate a genuine trend from one unusual summer.

Vendors sometimes limit how far back their API will go, or purge detail after a retention window. Find that out during planning, because it sets a hard floor on what your revenue analysis can ever look at.