Baldur Bjarnason

... works as a web developer in Hveragerði, Iceland, and writes about the web, digital publishing, and web/product development

These are his notes

@fgtech Does robots.txt let you block by use case? Otherwise you’d have to preemptively block a potentially infinite list of user agents

Also inclusion in an ML training data set should be opt-in, especially if the download utility in question wants to comply with the GDPR.