1 jun. 2023

Why the fuck did an Amazon crawler spend an entire month loading 200,000 pages from my client’s site (which has nowhere that much content), and with a regular browser’s user-agent to boot 🤬

Edit: I'm still mystified as to what the bot was looking for here, but the user-agent bit is probably my fault, I was storing it in a field that was too short.

Want to know when I post new content to my blog? It's a simple as registering for free to an RSS aggregator (Feedly, NewsBlur, Inoreader, …) and adding to your feeds (or if you want to subscribe to all my topics). We don't need newsletters, and we don't need Twitter; RSS still exists.

Legal information: This blog is hosted par OVH, 2 rue Kellermann, 59100 Roubaix, France,

Personal data about this blog's readers are not used nor transmitted to third-parties. Comment authors can request their deletion by e-mail.

All contents © the author or quoted under fair use.