You're not the only one that said they couldn't find a statement but I have tried archiving at archive.fo and it doesn't work. I'll try and look and see.
This is the same site that pulled all their pro pedophilia articles after attacking Milo's pederasty comments.
They can block the IP but it looks like in this case they are detecting and disabling a certain script. Robot.txt seems to be the culprit
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search …
TheReal_ND wrote:They can block the IP but it looks like in this case they are detecting and disabling a certain script. Robot.txt seems to be the culprit
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search …