The site menudojuego.com compares game prices across stores. In July, 102 of its 3,346 game pages had a price on them. The rest were a price comparison with nothing to compare.
The prices come from a third-party API with a rate limit that is not documented anywhere: 15,780 records, fifty per request, and anything faster than about one request per second earns a 429 and a lockout of several minutes.
The part worth writing down
My sweep loop exited when a page came back empty: that is how you detect the end of the catalogue. It also exited when a request failed, because I had written both conditions the same way. So when the rate limiter kicked in at record 3,300, the job stopped, logged "sweep complete", and exited zero. Every signal said it had worked. It had seen 21% of the data.
That is now the first thing I look for in anyone's batch jobs. Not the one that crashes — a crash gets noticed the same day. The job that reports success on partial data gets noticed in October, when someone asks why a number looks low, and by then nobody trusts any of the numbers.
The fix
It was not only the loop condition. A failure now raises an explicit INCOMPLETE warning in the log; retries back off at 60, 120 and 180 seconds; and a mutex stops the scheduled run from starting on top of a manual one — which had already happened once, with the two runs rate-limiting each other for two hours.
Once the data was trustworthy, each game got its all-time low, so the page answers the question the visitor actually has: not what does this cost, but is this a good time to buy.
Where it stands today
As of 11 August 2026: 2,696 games with prices refreshed at 05:00 and 17:00 every day, up from 102 in July. 2,531 with an all-time low recorded, served with schema.org offers markup so a crawler sees it too.
What is still wrong, before you find it yourself
79 games show a current price below their own recorded "all-time low", because that figure comes from the API and is not yet recalculated against what I observe myself. It is on the list.
Stack: NestJS, Prisma, PostgreSQL, Next.js 15, Docker, cron on a self-managed VPS. My own project, nothing confidential — which is why it can be told in full.