Closes: #11199
@mekarpeles mentioned we were getting 403 nonstop from wikidata endpoint.
I recently noticed that we were often not loading data reliably.

Turns out it's because they have a user agent policy that is being enforced actively now here.

To test this I run the following code locally:

>>> import requests
>>> requests.get("https://www.wikidata.org/w/rest.php/wikibase/v1/entities/items/Q9152")
<Response [403]>
>>> 
>>> headers = {'User-Agent': 'OpenLibrary.org Wikidata Integration' }
>>> requests.get("https://www.wikidata.org/w/rest.php/wikibase/v1/entities/items/Q9152", headers=headers)
<Response [200]>

As we can see, with no headers we 403. But when you set headers it's 200.

If we were being blocked by wikidata for too many requests it would be a 429 (as I see locally when I use hey to make many requests)

Technical

Testing

Screenshot

Stakeholders