AI Blocking in robots.txt

[Ragnar]

Which actually meant creating a robots.txt. Since to my surprise I found Xenforo apparently doesn't come with one by default.

I've ever relied on banning Google Analytics etc. to protect privacy; but as the Internet has changed in the last 5 years; decided to add some repellents to AI crap:

# The Common Crawl dataset. Original source for GPT and others.
User-agent: CCBot
Disallow: /

# The example for img2dataset, although the default is *None*
User-agent: img2dataset
Disallow: /

# GPTBot is OpenAI's web crawler
User-agent: GPTBot
Disallow: /

# ChatGPT-User takes direct actions on behalf of ChatGPT users
User-agent: ChatGPT-User
Disallow: /

# Google's Bard and Vertex AI generative APIs
User-agent: Google-Extended
Disallow: /

# Speculative blocks for Anthropic
User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

# webz.io - they sell data for training LLMs.
User-agent: Omgilibot
Disallow: /

User-agent: Omgili
Disallow: /

# Meta's bot that crawls public web pages to improve language models
User-agent: FacebookBot
Disallow: /

# ByteDance's bot used to gather data for their LLMs, including Doubao.
User-agent: Bytespider
Disallow: /

# Brandwatch - "AI to discover new trends"
User-agent: magpie-crawler
Disallow: /


User-agent: *
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /posts/
Disallow: /login/
Disallow: /search/
Disallow: /admin.php
Allow: /

User-agent: Baiduspider
Disallow: /

User-agent: Baiduspider-video
Disallow: /

User-agent: Baiduspider-image
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: *
Disallow: /account*
Disallow: /help*
Disallow: /misc/style*
Disallow: /misc/quick-navigation-menu*
Disallow: /login*
Disallow: /logout*
Disallow: /lost-password*
Disallow: /register*
Disallow: /reports*
Disallow: /search*
Disallow: /conversations*
Disallow: /css.php
Disallow: /cron.php
Disallow: /admin.php
Disallow: /js
Disallow: /styles
Disallow: /members/*
Disallow: /profile-posts/*
Disallow: /online/*
Disallow: /recent-activity/*



Disallow: /admin.php
Disallow: /js
Disallow: /styles
Disallow: /members/*
Disallow: /profile-posts/*
Disallow: /online/*
Disallow: /recent-activity/*

The AI stuff is at the top [ from here Github Training Data ]

Github is of course owned by vile old Microsoft, one of the biggest sinners --- in AI development as well.


The rest is standard stuff cobbled by Elves.

xxxxxxxxxxxxxxxxxxxxxxxxxxxxx


It was ftpped to what I believe was the absolute root of the installation, and not the other roots like public_html; so mistakes may have been made.

It will probably need to be updated at least annually; since advertisers etc. are tricky little scum; and the Tech Titans will always cheat.

 

[Black Rose]

The rest is standard stuff cobbled by Elves.

I was a machine elf once, but I got better.

 

[ZenRaiden]

Thanks for going to the trouble of doing so much for a forum that is barely populated.

Also surprised and did not realize you are still doing so much to make it function the way it does, much appreciated.

 

[birdsnestfern]

Ok, thank you Ragnar for keeping the bots down.

ps
If they have a way of enabling the ‘delete posts‘ again, that would also be an interest, but no need to explain or reply if you can’t. Just a vague wish that can be ignored.

 

[BurnedOut]

I had brought this up some time before. There are way too many crawlers running amok and that was a few years ago

 

[dr froyd]

ragnar you are the man

 

[Ragnar]

Thanks.

Just now unlocked pics in first post. For some forumish reason they hadn't developed.

One reason is that my regular browsers, Pale Moon and Basilisk, don't render many things very well and especially not on our Host HomepageUniverse ( HomepageUniverse took over our magnificent hosts UrlJet ) including cPanel and other tools

[ --- which is not the small browsers faults, but that most webpages are geared for Chrome which engine ( Blink, devolved from WebKit ) is now used by nearly all big browsers to enable Google spying --- ]

therefore on Linux I reluctantly have to use the degooglefied Chromium; to deal with web-hosting; and, I am neither familiar with Chrome nor capable with it.


Another reason may be that Xenforo still has many quirks.



__________________________


Do note this AI stuff are not the old familiar bots and spiders of yore; but machinery to suck up all your words into Large Learning Models so that they can train on them and regurgitate often in a muddled fashion for the sort of people who want Artificial Intelligence.

The only use of that to me would be for language translation.

Danbooru, * my main haunt, hates imprinted translations in manga etc., in order to maintain their Catholic monopoly on translations or something; but Danbooru has it's own quirks.


* Warning: Danbooru now has so much crap, it should only be used after registration and immediate pasting in of a fine blacklist.

Mine is magnificent, and I put it up on Pastebin ( code site ) for free use by all, but after a year or so, Pastebin closed it to private view only since many of the terms I objected to were deemed unsuitable for viewers --- which was very much the point...

 

[LOGICZOMBIE]

it would be a shame if our new AGI god turned out to be INTP