Summary: This wiki page shows how I overcame a DDOS attack by AI Bots on this wiki.
Date: 18 April 2026

I host this wiki on AWS lightsail as described here. This is mostly a hobby project of mine, to help people figure out technical challenges, but it also has a more serious purpose. I use it to showcase my skills, professionalism and passion about the stuff I work with. Potential customers or employers can see with their own eyes what I'm working on or have been working on in the past. So keeping this wiki healthy is important for me. Even if it wouldn't be used, it still matters to me. So when I woke up one day and noticed the wiki to be down, I wasn't happy, especially when I found out it was caused by a relentless number of page and media requests by AI Bots. The attack started around the 7th of March, and I just finished up with all the actions I had to do to keep my wiki safe for now, so it took me over a month of free time. I spent about 32 hours on it. And I'm still busy, but now, again, on writing a wiki page, this page, to share my knowledge. I hope you enjoy it.

One final remark I wanted to make. Hosting a website with actual content is nearly impossible for someone not tech-savvy nowadays. AI bots will keep on going requesting resources, and it just won't stop.

So why am I calling what happened a DDOS attack? Well, as from wikipedia:

In computing, a denial-of-service attack (DoS attack /dɒs/ doss) is a cyberattack in which the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services of a host connected to a network.

Even though all the AI bots maybe didn't intend to make my site unavailable, it was what happened. That is also caused because, my site being a hobby project, I'm using AWS lightsail. On AWS lightsail, you do not get to used the full CPU's available in the instance. You kind of earn a set of CPU burst capacity, which you can use when your site is heavily used for a short amount of time. However, when you use more burst capacity then you earned, you get limited to the performance baseline (if interested, see here for more information). As all the AI bots just went on with requests, my site was limited to it's performance baseline, which was too little too serve all the AI bots and legitimate users. Meaning, when visiting my wiki, all you got was errors.
For me, this meant I had to put up a maintenance page, so I could earn some burst capacity again. In the mean time, this would hopely scare of the AI bots. Which it didn't, as you can see in this burst capacity graph. Once the attach started, the burst capacity would decline, and climb up again once I enabled the maintenance mode. Then every time I tried to enable the wiki, the load would grow again:

Burst capacity over the first few days

After a few days I realized I needed more then just maintenance mode. In the month after, I implemented a more advanced robots.txt, a simpler sitemap.txt for more control, I enabled a botchecker, ditched apache htaccess for normal apache config files to improve performance and started blocking access to visitors who wouldn't come over cloudfront.

A robots.txt file is a file in the root of a website used to implement the Robots Exclusion Protocol. Unfortunately, this relies on voluntary compliance. So, as it does help against the bigger more reliable AI crawlers, it doesn't do anything against malicious bots. But it does help a bit. I started with a fairly simple robots.txt to just deny access to all, but as I do want my wiki to be crawled by google and such I eventually created this:

User-agent: *
Disallow: /wiki/templates/
Disallow: /wiki/playground/
Disallow: /*?*rev=
Disallow: /*?*do=login
Disallow: /*?*do=admin
Disallow: /*?*redirect=no
 
Sitemap: https://wiki.getshifting.com/sitemap.txt
 
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AhrefsBot
User-agent: Baiduspider
User-agent: barkrowler
User-agent: BitSightBot
User-agent: BLEXBot
User-agent: Bytespider
User-agent: CCBot
User-agent: coccocbot-web
User-agent: DataForSeoBot
User-agent: dataprovider
User-agent: dotbot
User-agent: facebookcatalog
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: HaloBot
User-agent: IbouBot
User-agent: ImagesiftBot
User-agent: keys-so-bot
User-agent: LinkupBot
User-agent: MauiBot
User-agent: Mediapartners-Google
User-agent: meta-externalads
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: MJ12bot
User-agent: MojeekBot
User-agent: netEstate NE Crawler
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: rogerbot
User-agent: serpstatbot
User-agent: SitesOverPagesBot
User-agent: Sogou web spider
User-agent: TerraCotta
User-agent: TikTokSpider
User-agent: YandexAdNet
User-agent: YandexBlogs
User-agent: YandexBot
User-agent: YandexComBot
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: YandexMarket
User-agent: YandexMedia
User-agent: YandexOntoDB
User-agent: YandexSitelinks
User-agent: YandexSpravBot
User-agent: YandexVerticals
User-agent: YandexVertis
User-agent: YandexVideo
User-agent: YandexWebmaster
Disallow: /
 
User-agent: bingbot
Crawl-delay: 10

This denies access to all dynamically build pages that should not be part of a crawler. It also provides a sitemap to provide a simple list of all pages, instead of all media and again all the dynamic pages. Last, it tries to enforce a crawl-delay to the bingbot who should support that.
In case this ever changes, the current version can be seen online here

As seen in the robots.txt, I also created a sitemap.txt. Before I relied on the built-in sitemap of dokuwiki. However, some bots don't really adhere to this, so I created a text based simple one. I also creates a script that updates it every time I add a page in the repository.

I also disabled the google sitemap creation in the dokuwiki settings.

You can check the sitemap online.

To reduce the load, and also block actions that should not be done by Bots, Users or who ever, I disabled the following dokuwiki actions:

- Recent Changes - Register - Sitemap - Old revisions - Other actions: media

While testing solutions I often found myself struggling with the interaction between htaccess and Apache config files. Finally, in the documentation I found it is better to avoid using htaccess when you have access to the server config files.

So instead of using htaccess as configured here I moved the rules into a apache conf file:

# DokuWiki Rewrite Rules
 
# File protection (originally from .htaccess)
<Files ~ "^([\._]ht|README$|VERSION$|COPYING$)">
    <IfModule mod_authz_core.c>
        Require all denied
    </IfModule>
    <IfModule !mod_authz_core.c>
        Order allow,deny
        Deny from all
    </IfModule>
</Files>
 
# Don't allow access to git directories
<IfModule alias_module>
    RedirectMatch 404 /\.git
</IfModule>
 
# DokuWiki specific rewrite rules
RewriteRule ^_media/(.*)              lib/exe/fetch.php?media=$1  [QSA,L]
RewriteRule ^_detail/(.*)             lib/exe/detail.php?media=$1  [QSA,L]
RewriteRule ^_export/([^/]+)/(.*)     doku.php?do=export_$1&id=$2  [QSA,L]
RewriteRule ^$                        doku.php  [L]
 
# Handle non-existent files/directories
RewriteCond %{REQUEST_FILENAME}       !-f
RewriteCond %{REQUEST_FILENAME}       !-d
RewriteRule (.*)                      doku.php?id=$1  [QSA,L]
RewriteRule ^index.php$               doku.php

Finally, I also reverted the htaccess overrides in the Apache main config files I configured before and added the new conf file:

DocumentRoot "/opt/bitnami/apache/htdocs"
<Directory "/opt/bitnami/apache/htdocs">
    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.4/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks
 
    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   AllowOverride FileInfo AuthConfig Limit
    #
    # AllowOverride None
    AllowOverride None
 
    # DokuWiki Configuration
    Include "/opt/bitnami/apache/conf/dokuwiki/dokuwiki.conf"
 
    #
    # Controls who can get stuff from this server.
    #
    Require all granted
</Directory>

When searching for solutions I also found many more dokuwiki users (here and here) to experience the same troubles. And not only the users, but als dokuwiki itself. The last link is from the main dokuwiki creator/maintainer and he created a botcheck, available on github. I decided to implement the botcheck, but I ran into some issues. Because I'm hosted behind cloudfront, all (valid - more on this later) requests come from cloudfront. Which means that the good bots also didn't have access anymore. So I implemented this PR myself. This extends the lookup utility to also allow IP addresses from good bots in the {HTTP:X-Forwarded-For} header.

Once I had botcheck working I had the idea to use botcheck as well to check if a request was valid. Remember, I am hosting on AWS lightsail but behind a Cloudfront. I mostly use Cloudfront to handle the certificates for me. So when you go to wiki.getshifting.com, you first go to cloudfront, and cloudfront then redirects you to the origin lightsail instance. Now, somehow, many requests were trying to access the origin lightsail instance directly.

To prevent this, I made an additional change to the botcheck.conf:

<IfDefine BOTCHECK_DIR>
    RewriteEngine On
 
    RewriteMap lookup prg:${BOTCHECK_DIR}/lookup
 
    # CloudFront IP restriction - block requests not coming from CloudFront
    RewriteCond ${lookup:cloudfrontaddresses.net.list;%{REMOTE_ADDR}|NOT_FOUND} =NOT_FOUND
    RewriteRule .* - [F,L]

The rewrite rule returns a 403 forbidden response [F] and stops processing further rules [L]. This of course meant I also needed to create and add all Cloudfront IP addresses to the new file cloudfrontaddresses.net.list.

To make the list complete, I've also taken the following steps:

• Implemented the update-bot-addresses.sh script to run after an update as well as weekly. Note that the script needed a small change to work for the common crawler
• I also implemented a script to automatically update the cloudfrontaddresses.net.list. This was created by copilot so I won't post it here.
• I also created systemd timers to run both scripts at a scheduled time. Again created by copilot.
• I created a pipeline to be able to manage all customizations from my repository. The pipeline can be checked in my public repo
• I also updated the apache logging in httpd.conf to include the {HTTP:X-Forwarded-For} IP address:

    # LogFormat "{ \"time\":\"%{%Y-%m-%d}tT%{%T}t.%{msec_frac}tZ\", \"process\":\"%D\", \"filename\":\"%f\", \"remoteIP\":\"%a\", \"host\":\"%V\", \"request\":\"%U\",\"query\":\"%q\",\"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\",\"referer\":\"%{Referer}i\"}" cloudwatch
    LogFormat "{ \"time\":\"%{%Y-%m-%d}tT%{%T}t.%{msec_frac}tZ\", \"process\":\"%D\", \"filename\":\"%f\", \"remoteIP\":\"%a\", \"xForwardedForIP\":\"%{X-Forwarded-For}i\", \"host\":\"%V\", \"request\":\"%U\",\"query\":\"%q\",\"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\",\"referer\":\"%{Referer}i\"}" cloudwatch

This wiki has been made possible by: