Jump to content

Manual:Nginx caching

From mediawiki.org

This page describes an alternative to Varnish caching for sites using Nginx. It supports multiple wikis on one server, and serving cached mobile pages from the same domain using MobileFrontend. Cache purging is supported, so anonymous visitors always get up to date content.

Note: The strategy used forces the Accept-Language header to be en-US, so the UI language for anonymous visitors will always be English. You can change this to any other language, if your wiki is primarily targeted at another demographic. Caching different versions of pages depending on the Accept-Language header sent by the visitor's browser is not supported.

This article assumes some basic familiarity with the configuration of Nginx, PHP, and MediaWiki.

Prelude

[edit]

When serving MediaWiki from Nginx using PHP-FPM, it's possible to use the caching capabilities of the Nginx FastCGI module to cache responses generated by MediaWiki's PHP code. This allows Nginx to serve anonymous visitors directly from memory, without needing to ask the PHP-FPM process to execute any code at all.

Visitors who are logged in, or have made edits (and thus receive a session token), should not be served from this cache, since MediaWiki's PHP code needs to customize the page for them, such as to display their user name, load user CSS/JS, display talk page notifications, and so on.

This caching strategy should be used in addition to, and not as a replacement for, the parser cache and other internal caching mechanisms of MediaWiki. Even if you use this strategy, you should still also use the various internal caching mechanisms supported by MediaWiki, such as with APCu, memcached, and so on.

However, a traditional "front-end HTTP cache" such as Varnish is essentially rendered obsolete when using this strategy. That being said, Varnish may be better suited for setups in which there are multiple back-end servers for load-balancing purposes. This strategy instead assumes a single Nginx server hosting one or more wikis.

On-disk or in-memory

[edit]

Although the Nginx FastCGI cache uses the filesystem, one can arrange for it to reside in a RAM filesystem such as /dev/shm to make sure it's entirely in-memory. If a persistent cache is desired, an SSD should also work well, especially since modern operating systems cache accessed files in memory anyway.

If you're using the wildcard purging strategy seen in the configuration example below, based on the third-party ngx_cache_purge module, then you should know that a wildcard purge operation walks through the entire on-disk file hierarchy of the cache, opening every single file to check if its key matches the wildcard. For this reason, a RAM filesystem is probably preferred.

Packages needed

[edit]

For Nginx itself, the package of choice doesn't matter, so long as FastCGI support is enabled. On Debian 12 and 13, the plain nginx package has it enabled. Metapackages such as nginx-core, nginx-full, or nginx-extra are not necessary; these merely pull in various selections of module packages that can just as well be installed one by one.

  • The libnginx-mod-http-brotli-filter package is recommended, as it allows you to enable support for in-stream Brotli compression.
  • The libnginx-mod-http-cache-purge package is required if you want to use cache purging, so that edited pages update more quickly for anonymous visitors.

Note: The Debian 12 package for the cache purge module does not support wildcard purging; the Debian 13 version does.

Warning: Up to version 2.5.3, the cache purge module leaks memory on wildcard purges. As of February 2026, Debian 13 has not yet updated to a newer version. You can compile and deploy the fix manually with these instructions.

You can also use the caching strategy explained on this page without any cache purging, but will have to live with the fact that anonymous visitors get outdated content until the cached version of a page expires on its own.

Nginx configuration

[edit]

Debian ships an /etc/nginx/nginx.conf file that automatically includes further configuration files from /etc/nginx/conf.d, and virtual-host configuration files from /etc/nginx/sites-enabled (a symlink farm to /etc/nginx/sites-available). It also ships /etc/nginx/fastcgi.conf which contains a standard set of FastCGI configuration parameters.

If you don't already have such a core structure under /etc/nginx, please make sure to create one first. The rest of this section assumes such a base setup, and won't teach you how to create it. If in doubt, just copy what Debian does, because this article is written with the assumption of that base Nginx setup.

The configuration is split up as follows:

  • /etc/nginx/conf.d/brotli.conf: Enables Brotli support.
  • /etc/nginx/conf.d/php-fpm.conf: Generic PHP-FPM settings.
  • /etc/nginx/conf.d/mediawiki-fcgi.conf: The "MediaWiki FastCGI / Cached" core setup.
  • /etc/nginx/snippets/run-php-direct: Snippet to include to run PHP code bypassing the cache.
  • /etc/nginx/snippets/run-php-mwfc: Snippet to include to run PHP code using the "MediaWiki FastCGI / Cached" setup.
  • /etc/nginx/sites-enabled/mwfc-purge: The purge request handler, so to speak.
  • /etc/nginx/sites-enabled/wiki1: Virtual-host configuration for wiki 1. (example)
  • /etc/nginx/sites-enabled/wiki2: Virtual-host configuration for wiki 2. (example)

You can have multiple wikis under sites-enabled and the caching configuration will ensure that their pages aren't confused for each other even if they happen to have identical titles.

brotli.conf

[edit]

/etc/nginx/conf.d/brotli.conf   

brotli on;
brotli_types
        # text/html is implicit
        text/css
        text/javascript
        text/plain
        text/xml
        application/javascript
        application/json
        application/activity+json
        application/manifest+json
        application/vnd.api+json
        application/xml
        application/xml+rss
        application/xhtml+xml
        application/xslt+xml
        application/rss+xml
        application/atom+xml
        image/svg+xml
        font/ttf
        font/otf
        font/opentype
        ;

If you've installed the Brotli module for Nginx, put this file in place to enable it. (This is not related to caching, but generally advisable.)

php-fpm.conf

[edit]

/etc/nginx/conf.d/php-fpm.conf   

upstream php_fpm {
        server unix:/run/php/php8.4-fpm.sock;
}

This file is very simple and merely defines the location of the PHP-FPM socket as an "upstream" in Nginx terms.

mediawiki-fcgi.conf

[edit]

/etc/nginx/conf.d/mediawiki-fcgi.conf   

#
# MediaWiki FastCGI with Response Caching (Setup)
#
# This file is meant to be included from within an http { } block.
# See 'snippets/run-php-mwfc' for usage within location { } blocks.
#
# Variables defined with map are prefixed to prevent clashes with other
# parts of your Nginx configuration in case you have a complex config.
#
# The prefix "mwfc" stands for "MediaWiki FastCGI / Cached".
#

# The cache is held in RAM and may use up to 16 GB of memory.
# The key zone may use an additional 250 MB of memory.
fastcgi_cache_path
        /dev/shm/nginx-mwfc
        levels=1:2
        keys_zone=mwfc:250m
        max_size=16000m
        inactive=2d;

# Check whether user is visiting from mobile.
# Use a simple 1/0 to be used as part of the cache key.
map $http_user_agent $mwfc_mobile {
        default 0;
        # Regexes based on:
        # https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/varnish/templates/text-frontend.inc.vcl.erb
        "~(SMART-TV.*SamsungBrowser)" 0;
        "~*^(lge?|sie|nec|sgh|pg)-" 1;
        "~*(mobi|240x240|240x320|320x320|alcatel|android|audiovox|bada|benq|blackberry|cdm-|compal-|docomo|ericsson|hiptop|htc[-_]|huawei|ipod|kddi-|kindle|meego|midp|mitsu|mmp\/|mot-|motor|ngm_|nintendo|opera.m|palm|panasonic|philips|phone|playstation|portalmmm|sagem-|samsung|sanyo|sec-|semc-browser|sendo|sharp|silk|softbank|symbian|teleca|up.browser|vodafone|webos)"
                1;
}

### OLD PURGE STRATEGY ###
#
# PURGE request purges cached responses for GET requests.
# Responses for other requests, like HEAD, can't be purged.
# We also can't purge mobile from desktop, and vice versa.
#
#map $request_method $mwfc_req_key {
#       default $request_method;
#       PURGE   GET;
#}
#
#fastcgi_cache_key "$host$request_uri$mwfc_req_key$mwfc_mobile";
#
### END OLD PURGE STRATEGY ###

### NEW PURGE STRATEGY ###
#
# Use a wildcard to purge responses for all HTTP methods, mobile and desktop.
#
# However, we don't want a PURGE for the URI /foo to also purge /foo/bar.
# Worst case would be "/" purging "/*" i.e. all URIs.
#
# For this reason, we terminate the URI with a pipe.  This should be safe,
# because the pipe should be percent-encoded in the received URI.
#

map $request_method $mwfc_cache_key {
        default "$host$request_uri|$request_method$mwfc_mobile";
        PURGE   "$host$request_uri|*";
}

fastcgi_cache_key "$mwfc_cache_key";

### END NEW PURGE STRATEGY ###

# We will imitate Apache's AMF headers, which use true/false.
map $mwfc_mobile $mwfc_mobile_amf {
        0 false;
        1 true;
}

# This is the simplest way to make Extension:MobileFrontend understand
# that we do our own UA detection so it doesn't attempt to do its own.
fastcgi_param AMF_DEVICE_IS_MOBILE $mwfc_mobile_amf;

# MediaWiki doesn't set Cache-Control for some Special:MyXyz pages.
# https://phabricator.wikimedia.org/T272431
map $request_uri $mwfc_bypass_uri {
        default            0;
        "~*/Special:My"    1;
        "~*/Special:AllMy" 1;
}

# Does the URI represent a resource that varies on cookies?
# To be safe, assume all URIs do, except those specified.
map $request_uri $mwfc_bypass_cookie_uri {
        default 1;
        "~^/w/(load|opensearch_desc|thumb)\.php" 0;
        "~^/php/" 0;
}

# Do we actually have a cookie that causes response variance?
map $http_cookie $mwfc_bypass_cookie_value {
        default                0;
        "~([sS]ession|Token)=" 1;
        "~mf_useformat"        1;
}

# Perform an AND on the previous two variables.
map $mwfc_bypass_cookie_uri$mwfc_bypass_cookie_value $mwfc_bypass_cookie {
        default 0;
        "11"    1;
}

log_format mwfc_stats
        "$upstream_cache_status $status $request_method $host$request_uri";

This is the most complex file, and contains the majority of the caching setup.

Please tweak values in this file to suit your needs. This one uses up to 16 GB of RAM!

This is also where you define the location of the cache; this example uses /dev/shm/nginx-mwfc which means it resides on an in-memory filesystem whose contents will be gone after a reboot.

For details on how the cache purging works, read the comments in this file under NEW PURGE STRATEGY. It explains how we arrange for the mobile version of a page to be purged as well when a purge for the URL is requested. It also makes sure that responses for HTTP requests other than GET, such as HEAD, are also purged.

run-php-direct

[edit]

/etc/nginx/snippets/run-php-direct   

include /etc/nginx/fastcgi.conf;

fastcgi_pass php_fpm;

This is a snippet file to be included from location { } blocks when PHP code should be executed unconditionally, without consulting the cache or attempting to store the result.

run-php-mwfc

[edit]

/etc/nginx/snippets/run-php-mwfc   

#
# MediaWiki FastCGI with Response Caching (Execution)
#
# This file is meant to be included from within location { } blocks,
# to actually pass a request to PHP-FPM.
#
# See conf.d/mediawiki-fcgi.conf for the setup this depends on.
#

include /etc/nginx/fastcgi.conf;

#access_log /var/log/nginx/mwcache-stats.log mwfc_stats gzip;

fastcgi_pass php_fpm;

fastcgi_cache mwfc;
fastcgi_cache_valid 1h;
fastcgi_cache_min_uses 2;
fastcgi_cache_use_stale error timeout updating http_500 http_503;

fastcgi_cache_lock on;
fastcgi_cache_lock_age 20s;
fastcgi_cache_lock_timeout 20s;

# Ignore Vary because we took care of everything ourselves.
fastcgi_ignore_headers Vary;

# Anon visitors only get English.
fastcgi_param HTTP_ACCEPT_LANGUAGE en-US;

# Bypass variables defined in MWFC setup file.
fastcgi_cache_bypass $mwfc_bypass_uri $mwfc_bypass_cookie;

add_header X-Cache $upstream_cache_status always;

This snippet file is included from location { } blocks when you want PHP code execution with the Nginx front-end HTTP caching.

mwfc-purge

[edit]

/etc/nginx/sites-enabled/mwfc-purge   

server {
        listen 1080 default_server;

        access_log /var/log/nginx/mwfc-purge-access.log;
        error_log /var/log/nginx/mwfc-purge-error.log;

        allow 127.0.0.0/8;
        deny all;

        location / {
                if ($request_method != "PURGE") {
                        return 405;
                }

                # Mandatory, for caching to be processed at all.
                fastcgi_pass php_fpm;

                # Select cache from which we will purge.
                fastcgi_cache mwfc;

                #
                # Note: The following differs from upstream docs!
                #
                # The functionality is provided by this package:
                #   libnginx-mod-http-cache-purge
                #
                # Whose documentation can be found here:
                #   https://github.com/nginx-modules/ngx_cache_purge
                #

                # Enable purge handling.
                fastcgi_cache_purge PURGE from 127.0.0.0/8;
                cache_purge_response_type text;
        }
}

This virtual-host configuration file is just to handle the PURGE requests coming from localhost, sent by MediaWiki itself.

Use of TCP port 1080 seems to be hardcoded into MediaWiki, so let's hope you don't use it for something else already!

wiki1 / wiki2 / etc.

[edit]

/etc/nginx/sites-enabled/bg3wiki (example)   

# Easily toggle logging on/off here
map "" $bg3log {
        default 0;
}

map $request_uri $bg3log_pagehits {
        default    0;
        "~^/wiki/" 1; # toggle page-hit logging
}

log_format bg3wiki_pagehits
        '$time_iso8601 $remote_addr $status '
        '$request_uri $http_referer "$http_user_agent"';

server {
        listen 443 ssl;
        server_name bg3.wiki;

        ssl_certificate     /etc/letsencrypt/live/bg3.wiki/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/bg3.wiki/privkey.pem;

        access_log /var/log/nginx/bg3wiki-access.log combined if=$bg3log;
        error_log /var/log/nginx/bg3wiki-error.log;

        access_log /var/log/nginx/bg3wiki-pagehits.log
                bg3wiki_pagehits if=$bg3log_pagehits;

        log_not_found off;

        root /var/www/bg3;

        # No cache for robots, sitemaps
        location ~ ^/(robots|sitemap) {
        }

        # Static assets outside of MediaWiki
        location ~ ^/(favicon|static) {
                # Cache 30 days
                add_header Cache-Control "max-age=2592000, public";
        }

        # Custom CSS and JS code
        location ~ ^/(css|js)/ {
                # Cache 5 min
                add_header Cache-Control "max-age=300, public";
        }

        # Custom PHP scripts
        location /php/ {
                include snippets/run-php-mwfc;
        }

        # Root URL is main page
        location = / {
                fastcgi_index w/index.php;
                include snippets/run-php-mwfc;
        }

        location = /wiki/Main_Page {
                return 301 /;
        }

        # Location for wiki's entry points
        location ~ ^/w/(index|load|api|thumb|opensearch_desc|rest)\.php$ {
                include snippets/run-php-mwfc;
        }

        # Block Discord's broken page preview tool
        location = /w/api.php {
                if ($http_user_agent ~ "Discordbot") {
                        return 403;
                }
                include snippets/run-php-mwfc;
        }

        # Images
        location /w/images {
                # Cache 24 h
                add_header Cache-Control "max-age=86400, public";
        }
        location /w/images/deleted {
                deny all;
        }

        # MediaWiki assets (usually images)
        location ~ ^/w/resources/(assets|lib|src) {
                # Cache 30 days
                add_header Cache-Control "max-age=2592000, public";
        }

        # Assets, scripts and styles from skins and extensions
        location ~ ^/w/(skins|extensions)/.+\.(css|js|gif|ico|jpg|jpeg|png|svg|webp|wasm|ttf|woff|woff2)$ {
                # Cache 30 days
                add_header Cache-Control "max-age=2592000, public";
        }

        # Handling for Mediawiki REST API, see [[mw:API:REST_API]]
        location /w/rest.php/ {
                try_files $uri $uri/ /w/rest.php?$query_string;
        }

        # Handling for the article path (pretty URLs)
        location /wiki/ {
                rewrite ^/wiki/(?<pagename>.*)$ /w/index.php;
        }

        # Every other entry point will be disallowed.
        # Add specific rules for other entry points/images as needed above this
        location / {
                return 404;
        }
}

# On port 80, handle LetsEncrypt requests, otherwise redirect to HTTPS.
server {
        listen 80;
        server_name bg3.wiki www.bg3.wiki;

        access_log /var/log/nginx/bg3wiki-redirect.log combined if=$bg3log;
        error_log /var/log/nginx/bg3wiki-error.log;

        location /.well-known/acme-challenge {
                root /var/www/html;
                try_files $uri =404;
        }

        location / {
                return 301 https://bg3.wiki$request_uri;
        }
}

# Redirect erroneous www. subdomain requests to main domain.
server {
        listen 443 ssl;
        server_name www.bg3.wiki;

        ssl_certificate     /etc/letsencrypt/live/bg3.wiki/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/bg3.wiki/privkey.pem;

        access_log /var/log/nginx/bg3wiki-redirect.log combined if=$bg3log;
        error_log /var/log/nginx/bg3wiki-error.log;

        return 301 https://bg3.wiki$request_uri;
}

This is an example wiki virtual-host configuration. This one is taken from bg3.wiki and simplified down a bit for demonstration.

Note that the Cache-Control headers set in some cases have nothing to do with our PHP caching strategy. These just tell the visitor's browser that it can cache certain files, like images or CSS, for a certain duration.

MediaWiki configuration

[edit]

LocalSettings.php (excerpt)   

# Allow caching via reverse proxy; in our case the Nginx FastCGI cache.
$wgUseCdn = true;
$wgCdnMaxAge = 3600;

# Make MediaWiki send PURGE requests to Nginx
# Note that this implicitly uses port 1080
$wgCdnServers = [ '127.0.0.1' ];
$wgInternalServer = 'http://bg3.wiki';

Finally, we need some tweaks in LocalSettings.php to make sure that:

  1. MediaWiki's PHP code sets the correct response headers for Nginx to actually cache responses.
  2. MediaWiki sends PURGE requests for page whose cache is invalidated, like when it's edited, or when a template it uses is edited.

Automatic cache refresh

[edit]

Note: This section is entirely optional.

There's a potential issue a wiki could run into if a large number of pages have their cache purged. For example, if you edit a template used by thousands of pages, MediaWiki will start running htmlCacheUpdate jobs for all those pages, which will invalidate their parser cache and also send PURGE requests to Nginx. Although these jobs are spread out over time, they could complete within a few hours, leaving your caches empty. Later, if a sudden surge of traffic (especially from bots/crawlers) causes all those pages to be requested again, the server runs into a worst-case scenario: Both the Nginx HTTP cache and the MediaWiki parser cache are empty or invalidated, so all the pages need to be parsed from scratch.

If the wiki doesn't have too many pages, and the server has sufficient disk storage or RAM (depending on whether your Nginx cache is in a RAM filesystem or not), then an elegant solution is to have any page re-cached immediately after htmlCacheUpdate invalidates its parser cache and asks the HTTP front-end (Nginx) to purge it from its cache.

To achieve this, the following trick can be used:

  • The actual Nginx PURGE handling is moved to another port number of your liking, say 1081. We will invoke it ourselves.
  • A new Nginx server block responsible for handling PURGE requests from MediaWiki (port 1080) is set up to execute a PHP script of ours. This script:
    1. Forwards the PURGE request to the real handler (port 1081).
    2. Uses a regular GET request to immediately force the page to be parsed and cached again.
  • In fact, the script can send a second GET request with a spoofed mobile User-Agent string, to also have the mobile version pre-cached.

A naive implementation of this seems to suffer certain issues: Requesting the page again immediately and synchronously, when an htmlCacheUpdate job triggers our script, can cause an outdated version of the page to be cached. The reason is unclear, but the solution is to perform this asynchronously and after a small delay.

Cache refresh strategy 1

[edit]

The following pair of PHP scripts can implement the aforementioned solution. The associated update to the Nginx configuration is shown further below.

Dispatcher script:

/var/www/mwfc-purge-dispatch.php   

<?php

#
# Important notes:
#
# This script is invoked by MediaWiki right after it invokes an SQL command
# to invalidate the parser cache by updating the page_touched column in the
# database, which has a resolution of seconds.  This happens synchronously,
# i.e., MediaWiki waits for the response of this script.
#
# Also, this may be invoked in rapid succession by any script or job that
# mass-purges the caches of large numbers of pages.
#
# To mitigate any potential issues with with re-entrancy of MediaWiki code,
# race conditions, and off-by-one errors in timestamp comparisons, we will
# immediately send a response to the client, and wait 1 second.
#
# (The exact reason why this is needed has not been verified; it's most
# likely because the page_touched update isn't committed to the database
# until after MediaWiki receives a response from this script.)
#
# We perform the 1-second delay, and the actual work, in a separate process,
# so we don't keep the PHP-FPM worker busy.  This is achieved by starting a
# new PHP process, which forks and exits, like a double-forking daemon.
#

fastcgi_finish_request();

$script = '/var/www/mwfc-purge-process.php';
$uri = $_SERVER['URI'];

$p = proc_open(
        # Need '-f' for '--' to be handled correctly; observed in 8.4.16
        [ 'php', '-f', $script, '--', $uri ],
        [
                0 => [ 'file', '/dev/null', 'r' ],
                1 => [ 'file', '/dev/null', 'w' ],
                2 => [ 'file', '/dev/null', 'w' ],
        ],
        $pipes
);

# We have to wait for the child process to exit, to prevent a defunct aka
# zombie process, but that's fine because it will exit immediately after a
# fork, creating a disowned child process.  We can't do that here because
# we're in a PHP-FPM execution context.
proc_close($p);

Handler script:

/var/www/mwfc-purge-process.php   

<?php

#
# See mwfc-purge-dispatch.php for relevant documentation.
#

$pid = pcntl_fork();
if ($pid !== 0) {
        exit();
}

$uri = $argv[1];
$ua = 'mwfc-purge';

sleep(1);

# Run the real Nginx purger
$c = curl_init('http://127.0.0.1:1081');
curl_setopt_array($c, [
        CURLOPT_CUSTOMREQUEST => 'PURGE',
        CURLOPT_REQUEST_TARGET => $uri,
        CURLOPT_WRITEFUNCTION => fn($c, $d) => strlen($d),
        CURLOPT_USERAGENT => $ua,
]);
curl_exec($c);

# Requests a page to warm the cache
function get($uri, $ua) {
        $c = curl_init($uri);
        curl_setopt_array($c, [
                CURLOPT_SSL_VERIFYHOST => 0,
                CURLOPT_CONNECT_TO => [ '::127.0.0.1:' ],
                CURLOPT_WRITEFUNCTION => fn($c, $d) => strlen($d),
                CURLOPT_USERAGENT => $ua,
        ]);
        curl_exec($c);
}

get($uri, "$ua-phone");
get($uri, "$ua-desktop");

Updated 'mwfc-purge' in Nginx configuration:

/etc/nginx/sites-enabled/mwfc-purge (excerpt)   

server {
        listen 1080 default_server;

        access_log /var/log/nginx/mwfc-purge-dispatch.log;
        error_log /var/log/nginx/mwfc-purge-dispatch.log;

        allow 127.0.0.0/8;
        deny all;

        location / {
                if ($request_method != "PURGE") {
                        return 405;
                }

                include snippets/run-php-direct;
                fastcgi_param URI https://$host$request_uri;
                fastcgi_param SCRIPT_FILENAME /var/www/mwfc-purge-dispatch.php;
        }
}

server {
        listen 1081 default_server;

        # ... original 'mwfc-purge' config for port 1080 goes here ...
}

There are some issues to consider with this strategy:

  • If MediaWiki sends a large number of PURGE requests, triggering mwfc-purge-dispatch.php repeatedly, it will spawn a large number of mwfc-purge-process.php instances in the background, which may take several seconds each. This has the potential of spawning thousands of OS processes, which could become a problem.
  • Although the dispatch script runs very quickly, it briefly occupies a PHP-FPM worker process. The GET requests later invoked also occupy PHP-FPM workers, and for a more significant duration, since they trigger a page parse. This can lead to high PHP pressure, causing the problem we were trying to prevent in the first place.

To prevent MediaWiki's own htmlCacheUpdate jobs from causing this, we spread out their execution with a configuration such as the following in LocalSettings.php. This should neatly spread out the cache invalidation & refresh work, applying only a gentle amount of PHP-FPM/page-parse pressure over time, properly solving the original problem. 

# Invalidate at most 5 pages per second
$wgUpdateRowsPerJob = 5;
$wgJobBackoffThrottling['htmlCacheUpdate'] = 1;

Note: It seems that even without any special configuration, MediaWiki makes sure not to flood itself with a large surge of htmlCacheUpdate jobs all at once. It spreads them out even with default settings. So, the above tweak in LocalSettings.php may not be necessary. Test for yourself, based on your server's resources.

Note 2: The way in which the job queue is processed also affects this. For example, using a non-zero value for $wgJobRunRate, vs. setting it to zero and using a cron job or systemd service to call runJobs.php instead, as laid out in Manual:Job queue. If the runJobs.php approach is used, then the --maxJobs setting, and the frequency of runs / sleep duration between runs, are also important.

Warning: There are still ways in which the server could be hammered with PURGE requests, leading to immense sudden PHP execution pressure, even if htmlCacheUpdate jobs are limited. One example is invoking purgeList.php maintenance script on an entire namespace or all namespaces. Avoid invoking such scripts when using this automatic cache refresh strategy, without a way to mitigate this issue. A proper solution would require a task queue similar to MediaWiki's own job queue, but for our custom purge refresh script.

Cache refresh strategy 2

[edit]

This strategy solves the shortcomings of the previous one, at the cost of significantly higher complexity. Use at your own risk.

We implement a daemon that continuously listens on a System V message queue for the URLs to purge, and handles them at a set pace. This way, no amount of PURGE requests --be it from MediaWiki's htmlCacheUpdate jobs, a maintenance script such as purgeList, or any other mechanism-- has any chance of overloading the system, because the real work will always be performed at a set pace.

The maximum size for a System V message queue in Linux is 16 KiB by default, at the time of this writing. To increase it, put the following lines in a boot-time script, such as /etc/rc.local on Debian-based systems; you can create it if it doesn't exist: 

# Set maximum size for newly created SysV message queues to 64 MiB
echo $(( 64 * 1024 * 1024 )) > /proc/sys/kernel/msgmnb

This needs to happen before any System V message queue is created, because the upper size limit of each queue is set at creation. They are still dynamically grown as messages arrive, not statically allocated, so don't worry about memory waste.

Tests on bg3.wiki seem to indicate that about a quarter million purge requests take up about 8 MiB of memory, so the limit of 64 MiB would allow for up to 2 million queued requests; more than enough.

Next, we install the daemon that will receive messages from the queue and process them.

/var/www/mwfc-purge-process.php   

<?php

#
# See mwfc-purge-dispatch.php for related documentation.
#

$workers = 8;
$workLimit = 8192;
$workDelayMs = 1000;

$mqKey = ftok('/var/www', 'p');

$msgTypeId = 1;
$msgSizeLimit = 8192;

pcntl_async_signals(true);

pcntl_signal(SIGTERM, 'shutdown', false);
pcntl_signal(SIGHUP, 'shutdown', false);
pcntl_signal(SIGINT, 'shutdown', false);

$q = msg_get_queue($mqKey);

$name = 'mwfc-purger';
$pids = [];

function errlog($msg) {
        global $name;
        $ts = date('Y-m-d H:i:s');
        error_log("$ts $name: $msg");
}

cli_set_process_title($name);

for ($i = 0; $i < $workers; ++$i) {
        spawn($i);
}

errlog("Started $i workers.");

while (true) {
        $i = wait($s);
        if (pcntl_wifexited($s)) {
                errlog("Respawning worker $i.");
                spawn($i);
        } else if (pcntl_wifsignaled($s)) {
                $status = pcntl_wexitstatus($s);
                $signal = pcntl_wtermsig($s);
                errlog("Worker $i died. Status: $status Signal: $signal");
                errlog("Respawning after a short delay.");
                sleep(2);
                spawn($i);
        }
        sleep(1);
        gc_collect_cycles();
}

function spawn($i) {
        global $pids;

        switch ($p = pcntl_fork()) {
        case -1:
                errlog("FATAL: Couldn't fork.");
                shutdown(false);
        case 0:
                worker_main($i);
                exit();
        }
        $pids[$p] = $i;
}

function wait(&$s) {
        global $pids;

        while ( ($p = pcntl_wait($s)) === -1 ) {
                $e = pcntl_strerror(pcntl_errno());
                errlog("Unexpected wait() error: $e");
                sleep(1);
        }
        $i = $pids[$p];
        unset($pids[$p]);
        return $i;
}

function shutdown($s) {
        global $pids;

        if ($s !== false) {
                errlog("Caught signal $s. Shutting down.");
        }

        foreach ($pids as $p => $_) {
                posix_kill($p, SIGTERM);
        }

        while ($pids) wait($_);

        exit();
}

function worker_main($i) {
        global $name;
        global $workLimit;

        $name = "mwfc-purger/worker-$i";

        cli_set_process_title($name);
        if (posix_isatty(STDIN)) {
                posix_setpgid(0, 0);
        }

        $stop = false;
        pcntl_signal(SIGTERM, function() use (&$stop) {
                errlog("Caught SIGTERM. Finishing.");
                $stop = true;
        });

        for ($i = 0; $i < $workLimit; ++$i) {
                $uri = receive($stop);
                if ($uri) handle($uri);
                if ($stop) break;
                gc_collect_cycles();
        }

        errlog("Exiting after $i tasks.");
}

function receive(&$stop) {
        global $q;
        global $msgTypeId;
        global $msgSizeLimit;

        //
        // The SysV message queue API has some nasty limitations:
        //
        // * A naive "if (!$stop) receive()" pattern would be vulnerable to
        //   race conditions, as the signal setting $stop could run the exact
        //   moment before the receive() starts waiting.  The elegant solution
        //   would be to block signals, and have receive() atomically unblock
        //   and handle them.  This is not supported by the API.  The only
        //   solution is to always use NOWAIT and sleep between calls.
        //
        // * A too-large message triggers an error, but is *not* discarded, and
        //   there's simply no API to have it discarded atomically; an explicit
        //   discard (through a bogus receive call with size = 1 and NOERROR)
        //   would once again be vulnerable to race conditions, since multiple
        //   processes may do it in parallel.  The only solution is to always
        //   use NOERROR and assume messages of maximum length to be truncated;
        //   the sender should then refuse to send messages of length equal to,
        //   not only greater than, the size limit.
        //

        $mt = $msgTypeId;
        $ms = $msgSizeLimit;
        $fl = MSG_IPC_NOWAIT | MSG_NOERROR;
        while (!msg_receive($q, $mt, $_, $ms, $msg, false, $fl, $e)) {
                if ($e !== MSG_ENOMSG) {
                        $e = pcntl_strerror($e);
                        errlog("Couldn't read message. Error: $e");
                }
                usleep(100 * 1000);
                if ($stop) return false;
        }
        // Shouldn't happen if sender is configured correctly, but still.
        if (strlen($msg) === $ms) {
                errlog("Discarding truncated message: $msg");
                return false;
        }
        return $msg;
}

function handle($uri) {
        global $workDelayMs;
        global $name;

        usleep($workDelayMs * 1000);

        purge($uri, $name);
        get($uri, "$name/phone");
        get($uri, "$name/desktop");
}

// Runs the real Nginx purger
function purge($uri, $ua) {
        $c = curl_init('http://127.0.0.1:1081');
        curl_setopt_array($c, [
                CURLOPT_CUSTOMREQUEST => 'PURGE',
                CURLOPT_REQUEST_TARGET => $uri,
                CURLOPT_WRITEFUNCTION => ignore_write(...),
                CURLOPT_USERAGENT => $ua,
        ]);
        curl_exec($c);
}

// Requests a page to warm the cache
function get($uri, $ua) {
        $c = curl_init($uri);
        curl_setopt_array($c, [
                CURLOPT_SSL_VERIFYHOST => 0,
                CURLOPT_CONNECT_TO => [ '::127.0.0.1:' ],
                CURLOPT_WRITEFUNCTION => ignore_write(...),
                CURLOPT_USERAGENT => $ua,
        ]);
        curl_exec($c);
}

function ignore_write($c, $d) {
        return strlen($d);
}

We can turn this into a systemd service by dropping a service unit file into a location such as ~/etc and loading it from there:

/root/etc/mwfc-purger.service   

[Unit]

Description = MediaWiki Purge Handler

After = mariadb.service
After = php8.4-fpm.service
After = nginx.service

[Service]

User = www-data
ProtectSystem = full
WorkingDirectory = /var/www

# We don't want it to restart on OOM.
Restart = on-success
OOMScoreAdjust = 200

ExecStart = php mwfc-purge-process.php

MemoryMax = 400M
MemoryHigh = 200M

Nice = 15

StandardOutput = append:/var/log/mwfc-purger.log
StandardError = append:/var/log/mwfc-purger.log

[Install]

WantedBy = multi-user.target

Then run the following in the shell: 

# Load the service description and register it for automatic start at boot
sytemctl enable /root/etc/mwfc-purger.service
# Start it now
sytemctl start mwfc-purger

Since the service logs to a text file, it's best to add a logrotate entry, or it will grow indefinitely:

/etc/logrotate.d/mwfc-purger   

/var/log/mwfc-purger.log {
    weekly
    notifempty
    missingok
    rotate 4
    compress
    delaycompress
    prerotate
        systemctl stop mwfc-purger
    endscript
    postrotate
        systemctl start mwfc-purger
    endscript
}

Next, we have to update our dispatcher to send messages into the queue.

/var/www/mwfc-purge-dispatch.php   

<?php

#
# Important notes:
#
# This script is invoked by MediaWiki right after it invokes an SQL command
# to invalidate the parser cache by updating the page_touched column in the
# database, which has a resolution of seconds.  This happens synchronously,
# i.e., MediaWiki waits for the response of this script.
#
# Also, this may be invoked in rapid succession by any script or job that
# mass-purges the caches of large numbers of pages.
#
# To mitigate any potential issues with with re-entrancy of MediaWiki code,
# race conditions, and off-by-one errors in timestamp comparisons, we will
# immediately send a response to the client, and wait 1 second.
#
# (The exact reason why this is needed has not been verified; it's most
# likely because the page_touched update isn't committed to the database
# until after MediaWiki receives a response from this script.)
#
# We perform the 1-second delay, and the actual work, in a separate process,
# so we don't keep the PHP-FPM worker busy.  This is achieved by sending a
# message to a System V message queue that a daemon listens on.
#

fastcgi_finish_request();

$mqKey = ftok('/var/www', 'p');

$msgTypeId = 1;
$msgSizeLimit = 8192;

$msg = $_SERVER['URI'];
$log = $_SERVER['LOG'];

ini_set('error_log', $log);

# The greater-than-or-equal is intentional!  See comments in the receive()
# function of mwfc-purge-process.php, explaining the truncation logic.
if (strlen($msg) >= $msgSizeLimit) {
        error_log("URI too long: $msg");
        exit();
}

$q = msg_get_queue($mqKey);
if (!msg_send($q, $msgTypeId, $msg, false, false, $e)) {
        if ($e === MSG_EAGAIN) {
                $e = "Queue is full.";
        } else {
                $e = posix_strerror($e);
        }
        error_log("ERROR: Couldn't send message: $e");
        error_log("Message was: $msg");
}

And finally, an associated update to the Nginx configuration, just to ensure we get proper logging on errors.

/etc/nginx/sites-enabled/mwfc-purge (excerpt)   

server {
        listen 1080 default_server;

        access_log /var/log/nginx/mwfc-purge-dispatch.log;
        error_log /var/log/nginx/mwfc-purge-dispatch.log;

        allow 127.0.0.0/8;
        deny all;

        location / {
                if ($request_method != "PURGE") {
                        return 405;
                }

                include snippets/run-php-direct;
                fastcgi_param URI https://$host$request_uri;
                fastcgi_param LOG /var/log/nginx/mwfc-purge-dispatch.log;
                fastcgi_param SCRIPT_FILENAME /var/www/mwfc-purge-dispatch.php;
        }
}

server {
        listen 1081 default_server;

        # ... original 'mwfc-purge' config for port 1080 goes here ...
}

There is one significant downside to this strategy, apart from the implementation complexity:

The purge tasks being queued, then performed at a set pace, means that they could be delayed for quite a bit when many purge requests come in. Let's assume that only 4 can be handled each second, because the daemon uses 8 worker processes, waits 1 second before handling each request, and the request itself takes 1 second to process. That would mean 25 K requests need about 25,000 Requests ÷ (4 Requests/Second × 60 Seconds) ≈ 100 Minutes to complete.

In other words, if you frequently have to rely on MediaWiki's purge mechanism to swiftly deal with vandalized page content that ended up in the HTTP cache, and this happens at an unfortunate time where the queue is quite overloaded already, it could take up to several hours for the page to update for visitors.

(That being said, the previous strategy doesn't fare much better under such a scenario. As mentioned in the warning, a burst of thousands of PURGE requests would lead to immense CPU pressure, and most likely cause the server to become unresponsive to visitors. This strategy prevents that risk, and simply replaces it with the risk of PURGE requests being deferred for up to several hours in the future.)

It's possible to observe and manipulate System V message queues from the command-line. For example, use ipcs -q to display the queues: 

root@host # ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x70020d3d 1          www-data   666        4702370      74675

root@host #

In a pinch, you could destroy and recreate the queue. (Emptying it without destroying isn't possible from the CLI.) First, stop the purger daemon, so it doesn't begin to spam error messages due to the queue being gone. Then run ipcrm -Q KEY with the key displayed by ipcs -q, and restart the purger daemon. The queue will be recreated automatically. 

root@host # ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0x70020d3d 1          www-data   666        4702370      74675

root@host # systemctl stop mwfc-purger
root@host # ipcrm -Q 0x70020d3d
root@host # systemctl start mwfc-purger
root@host #

Of course this means that all the previously queued purge requests will be gone and not processed anymore. So, only do this if you urgently need to deal with severe vandalism, and can accept the cost of having many other pages remain outdated until their Nginx cache duration runs out naturally or they happen to be caught by another purge request later.

As an extreme measure, it's also possible to nuke the Nginx cache directly, which is quick and takes effect immediately. For example: rm -rf /dev/shm/nginx-mwfc/*

Alternative strategies

[edit]

One can imagine further variants of the cache refresh strategy. One prominent example that comes to mind is:

  1. Perform the actual PURGE operations immediately. I.e., forward the PURGE request to Nginx without delay.
  2. Defer the work of sending GET requests to trigger cache refresh. (E.g. by placing them on a System V message queue, as in the previous strategy.)

This involves a trade-off: Visitors are less likely to be served outdated content, as the real PURGE operations are performed early; in exchange, we once again create the potential of time-windows in which the HTTP front-end cache is starved of content. (Since the deferred GET requests to force HTTP cache updates are processed slowly over time.) As such, we would once again be exposed to the risk of a surge of bot/crawler traffic causing immense page parse pressure, causing the website to become unresponsive.

See also

[edit]
Retrieved from "https://www.mediawiki.org/w/index.php?title=Manual:Nginx_caching&oldid=8203174"