The vendor has no API. What are the options, best to worst?
Roughly in order of durability: a supported data export or SFTP file drop; a scheduled emailed report parsed on arrival; a database or warehouse connector if the vendor offers one; browser automation driving the vendor's own interface; and last, scraping HTML. Everything below the file drop breaks when the vendor changes a screen. Take the highest option available, and design from day one for the day it stops working.
Ask for the report before you assume there is nothing
A surprising number of platforms with no public API do have a scheduled export — a nightly CSV to SFTP, a report emailed to an address you choose, or a data-sharing option that only appears when you ask an account manager. It is rarely marketed because it is not a product feature.
So the first move is not engineering. It is a specific request: can this system deliver a scheduled file containing these fields. That question has produced a clean data feed on projects where everyone assumed scraping was the only path.
What breaks each option
- File drop or export. Breaks when the vendor changes the column layout, which they will do without warning. Survivable if you validate the header on every run.
- Emailed report. Breaks on subject line changes, attachment format changes, and mail routing. Adds a delivery dependency you do not control.
- Direct database or connector. Excellent when offered, and usually only offered to enterprise accounts. Breaks on their schema changes with no version guarantee.
- Browser automation. Breaks on any UI change, login flow change, or new bot challenge. Do not build automation that attempts to defeat a human verification step.
- HTML scraping. Breaks constantly, and provides no signal that it broke — it usually returns fewer rows rather than an error.
The real cost is maintenance, not construction
Browser automation is cheap to build and expensive to own. A script that reliably logs in and downloads a report can be written in a day and will need attention several times a year forever. Judge these options on the maintenance commitment, not the build estimate, and be honest with yourself about who will do that maintenance in month nine.
Two things make it survivable: aggressive validation so failures are loud rather than silent, and a documented manual fallback. A person uploading a file for two days beats a robot that has been quietly returning half the data for two weeks.
Check the agreement, and design the exit
Terms of service vary, and some prohibit automated access to the interface even for your own data. Read the agreement and, where it matters, ask the vendor in writing. That is a business conversation as much as a technical one.
Finally, build the extraction behind a clean internal interface so the rest of the system does not know how the data arrived. When the vendor eventually ships an API — or you eventually replace the vendor — you swap one component instead of rebuilding the pipeline. That separation is standard in our custom development and it is what keeps a fragile source from contaminating an otherwise solid integration architecture.
Topics: no API · data extraction · automation · integration design
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.