submitted 1 month ago by obbeel@kbin.earth to c/ask_experienced_devs@programming.dev

2 comments fedilink hide all child comments

I’m using university journal portal OAI endpoints for the third time in a row to fix a typo and handle DOI lookups.

Does anyone know how disrespectful or intrusive this is considered to be?

A few years ago, I scraped arXiv and they blocked my IP almost immediately; perhaps if I had used arXiv’s OAI interface, they wouldn't have blocked me.

But I really have no idea just how disrespectful or intrusive this is!

top 2 comments

sorted by: hot top new old

[-] xianjam@programming.dev 2 points 1 month ago* (last edited 1 month ago)

Basic etiquette, you'd probably want to ask the university portal maintainer.

Scraping is generally considered bad manners, and yes you would be immediately blocked if you ignore robots.txt. I assume by OAI you mean OpenAI? I'm unsure what kind of integration your university has, but like with many AI integrations, there is little oversight, which leads to... oversights.

[-] obbeel@kbin.earth 1 points 1 month ago

OAI are API endpoints for metadata access of articles. I imagine API access is more respectful in terms of bandwidth since it's organized specifically for access by the universities?

this post was submitted on 29 Aug 2026

5 points (85.7% liked)

Ask Experienced Devs

1517 readers

4 users here now

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 3 years ago

MODERATORS

Ategon@programming.dev