Curious what everyone's approach is for handling cron jobs that depend on external APIs.

Had a data sync job that pulled from a vendor API every hour. Worked great for months until the vendor started rate-limiting us during peak hours. The job would fail, retry on the next cron tick, hit the rate limit again, and basically spin for 3-4 hours until traffic died down.

What I ended up doing was adding exponential backoff inside the job itself — first retry after 5 min, then 15, then 60. But it felt wrong because now the job's execution time is unpredictable and can overlap with the next scheduled run.

The other approach I considered was just letting it fail and having a separate "catch-up" job that runs less frequently and handles any gaps. Cleaner separation but more moving parts.

For those of you running scheduled jobs that depend on third-party services: do you build the retry logic into the job, handle it at the scheduler level, or just accept that some runs will fail and deal with it downstream?

Also wondering if anyone's hit issues with jobs that silently succeed but return partial data — like the API responds 200 but only gives you half the records because of an undocumented pagination change. That one took me a while to catch.