22
Git scraping: track changes over time by scraping to a Git repository
(simonwillison.net)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 11 Aug 2023
22 points (100.0% liked)
Git
4740 readers
1 users here now
Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.
Resources
Rules
- Follow programming.dev rules
- Be excellent to each other, no hostility towards users for any reason
- No spam of tools/companies/advertisements. It’s OK to post your own stuff part of the time, but the primary use of the community should not be self-promotion.
Git Logo by Jason Long is licensed under the Creative Commons Attribution 3.0 Unported License.
founded 3 years ago
MODERATORS
Assuming each action takes 1 minute, it will use (assuming 30 days in the month) 2160 minutes exceeding the 2000 free tier monthly limit.
But apparently Gitea Actions are compatible so you could go with that.
And that's just a single repository. Although for a single JSON scrape it shouldn't take a full minute
According to the Actions log for that repository it generally takes about 11-16 seconds per scrape, nowhere near a full minute. Worst case, if it's 16 seconds every time, for a 31-day month that's 31 days * 72 actions/day * 16 sec/action = 595.2 minutes, well within the free tier limit.
And if it does become a problem it can easily be scaled down to once per hour or something with a single git commit.