22
Git scraping: track changes over time by scraping to a Git repository
(simonwillison.net)
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
this post was submitted on 11 Aug 2023
22 points (100.0% liked)
Git
4740 readers
1 users here now
Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.
Resources
Rules
- Follow programming.dev rules
- Be excellent to each other, no hostility towards users for any reason
- No spam of tools/companies/advertisements. It’s OK to post your own stuff part of the time, but the primary use of the community should not be self-promotion.
Git Logo by Jason Long is licensed under the Creative Commons Attribution 3.0 Unported License.
founded 3 years ago
MODERATORS
According to the Actions log for that repository it generally takes about 11-16 seconds per scrape, nowhere near a full minute. Worst case, if it's 16 seconds every time, for a 31-day month that's 31 days * 72 actions/day * 16 sec/action = 595.2 minutes, well within the free tier limit.
And if it does become a problem it can easily be scaled down to once per hour or something with a single git commit.