Why store the raw API response instead of just the fields you need?
Because the fields you need will change, and the source may not let you go back. Raw payloads let you rebuild history when a definition changes, prove a number came from the vendor when someone disputes it, detect schema drift by comparing shapes over time, and replay a failed load without re-hitting a rate-limited API. Strip sensitive fields you have no business keeping.
Three things raw storage makes possible
Rebuilds come first. When someone decides that revenue should count at completion rather than at invoice, you re-run a transformation instead of re-extracting three years from an API that may throttle you or may no longer have the detail.
Then proof. When a manager insists the dashboard is wrong, you can show the exact payload the vendor returned at the time, with a received timestamp. That ends the conversation quickly, and in the cases where the vendor did change something, it is the only evidence you will have.
Then drift detection. Comparing the field set of today's payloads against last month's is how you notice a vendor quietly renamed a field or stopped populating one, usually before it has corrupted a full reporting period.
Replay is the underrated one
Loading is where bugs live. A parser mishandles a null, a mapping is wrong for one job type, a timezone conversion is off by an hour. If you kept the raw payloads, fixing the bug means re-running the parser over stored data, in minutes, with no API calls at all.
If you did not, the fix is a full re-extraction competing with rate limits and vendor retention. The cost difference between those two situations is the entire argument for a landing layer.
What the landing layer looks like
Minimal and boring: the payload itself, the source system, the endpoint, the request parameters, a received timestamp, and a hash so identical payloads are recognizable. Nothing is interpreted at this stage, which is the point.
Everything downstream is derived from it, which means the modeled tables can be dropped and rebuilt at will. That is the property that makes an integration layer maintainable rather than a stack of decisions nobody dares touch.
What not to keep
Raw does not mean everything forever. Payment instrument details, government identifiers and anything else you have no operational use for should be dropped or redacted at ingest rather than stored and forgotten. Set a retention window per data class and enforce it with a scheduled job.
Privacy and retention requirements vary by state and by the kind of data involved, so confirm the specifics with your own counsel rather than inheriting whatever default a vendor ships. The engineering position is simple: keep what the work requires, for as long as the work requires, and no longer. That principle applies equally to recordings handled through call analysis.
Topics: raw data · landing zone · schema drift · auditability
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.