Over half of all web traffic in 2024 was automated. That is the headline number from the Imperva 2025 Bad Bot Report, and it is the first time bots have outnumbered humans in more than a decade. Drupal sites sit squarely in that traffic mix, and the old defensive playbook — block an IP, ban a user agent, drop a robots.txt entry, lean on Fail2ban — does not hold up anymore.

This is the companion post to my DrupalSouth Wellington 2026 talk, Bots, scrapers, and proxies: defending Drupal sites in an automated internet. The talk walked through the defences I actually use at amazee.io and recommend on client sites. The post covers the same ground, with a bit more room to show config and link out to the projects.

What actually changed

The technical context underneath bot defence has shifted in three ways that matter:

• Residential proxy networks. Scrapers no longer come from a handful of cloud subnets you can block. They route through real consumer IP addresses, often unwittingly donated by free-VPN users or piggy-backed off shady SDKs in mobile apps.
• Headless browsers everywhere. Playwright and Puppeteer have made it trivial to render JavaScript-heavy pages at scale. A page that needed a real human five years ago can be scraped today by anyone with a laptop.
• AI-driven scraping. Volume is up sharply because every new LLM needs training data, and there is now a steady drip of new crawlers showing up. Meta's externalagent is one recent example. There will be more.

Mimicry is now the baseline, not the edge case. A modern scraper will rotate IPs, randomise user agents, replay realistic TLS fingerprints, and pace itself slowly enough to look like a real user. You cannot rely on signal that lives in one HTTP header.

The scale of it

If this still sounds like a niche problem, the numbers say otherwise.

• 51% - share of web traffic that was automated in 2024, per the Imperva 2025 Bad Bot Report.
• +96% - year-on-year growth in some popular bot services across Pantheon's hosting fleet in their July 2025 data.
• 1B+ - unique monthly visitors Pantheon sees across its platform, which is the size of dataset those numbers are coming from.

On the amazee.io platform globally, 13% of incoming requests can be flagged as non-human based on the user agent alone. That is the lazy bots. The actual share of automated traffic is higher once you account for the ones that try to blend in. In absolute terms it adds up to hundreds of millions of requests every month.

The goal is not to block all bots

Before going through the defences, one thing I am careful to say up front, both on stage and here: the goal is not to block all bots. That is unwinnable, and the closer you get to it the more real users you break.

Search crawlers, RSS readers, uptime monitors, link-preview generators in Slack and iMessage, accessibility tooling - all bots, all wanted. The goal is to reduce abuse where it hurts most, on the endpoints that cost you real money or real performance, while leaving everything else alone.

Drupal-native defences

The defences closest to your application are the smartest. They can see the path, the user, the form, the cache state. They are also the most expensive per blocked request, because every block at this layer has already cost you a full PHP bootstrap.

Perimeter

The Perimeter module drops requests matching known-bad patterns: /wp-admin, /.env, xmlrpc.php, all the WordPress scanner noise that hits every Drupal site daily. It is the cheapest win on the list. It will not stop a serious scraper, but it will keep your logs clean and your error rate honest.

CrowdSec and AbuseIPDB

CrowdSec is a local agent plus a community blocklist. Every site running CrowdSec contributes detected attacks back to a shared signal, and pulls down the latest list of bad actors. It is the closest thing the open-source world has to a distributed reputation system.

AbuseIPDB is a reputation lookup service. You query an IP, you get a confidence score. It is most useful on the forms and login flows where you can afford the latency of an external API call. Both are available as Drupal modules.

Facet Bot Blocker

If you run Search API with facets, this is the single cheapest huge win available to you. Faceted search URLs are catnip for scrapers: every combination of filters is a new URL, every URL is uncached, every uncached request hits the database. A bot that crawls a faceted listing can take a site down without trying.

The Facet Bot Blocker module acts as a rate limit on requests that include at least one facet in the URL. Configure it to use Redis or memcache for the counter so you are not making the problem worse by hitting the database to record the request. On one of our hosting customers, this one module cut Search API load by more than half.

Form-side defences

Logins, registrations, password resets and contact forms all need their own treatment, separate from page-level defence:

• Honeypot - invisible field plus a time-based check. Cheap, fast, surprisingly effective against the dumb half of form spam.
• Antibot - requires JavaScript to submit, blocks the bots that do not run JS.
• CAPTCHA, reCAPTCHA, or Cloudflare Turnstile - full challenge. Use the lightest option that works, and ideally only after Honeypot and Antibot have already rejected the easy cases.
• Hidden CAPTCHA - bridges the gap when you want a CAPTCHA-style check without the accessibility cost of a visible challenge.

The trade-off the project pages don't mention

Every block at the Drupal layer has already cost you a PHP bootstrap. That is fine when the absolute volume is small. It is not fine when you are eating hundreds of millions of bot requests and bootstrapping PHP for each one. This is why you cannot stop at the application layer.

Web server and infrastructure

One layer out, the web server can drop requests before PHP ever runs. The trade-off flips: you save the bootstrap cost, but you lose access to application context.

Rate limiting and geo blocking

nginx ships with limit_req_zone, Apache has mod_ratelimit. Both are blunt but effective on volume. A starting point for nginx looks roughly like this:

limit_req_zone $binary_remote_addr zone=search:10m rate=10r/m;

location /search {
    limit_req zone=search burst=5 nodelay;
    proxy_pass http://drupal;
}

Ten search requests per minute, per IP, with a burst of five. Tune to taste. The $binary_remote_addr key is cheap on memory; a 10MB zone holds around 160,000 IPs.

Geo blocking is the other infrastructure-level lever. It is pragmatic and occasionally controversial. If your audience is the New Zealand public sector, blocking inbound from regions you do not serve is a defensible call. If your audience is global, it is not. Know your traffic before reaching for it.

ModSecurity and the OWASP CRS

ModSecurity with the OWASP Core Rule Set is a proper WAF you can self-host. Once tuned, it is real protection. The tuning is the catch — out of the box it will flag Drupal admin actions, file uploads, anything that looks like SQL in a body or a query string. Expect to spend real time pruning rules and adding exceptions for legitimate site behaviour before you stop generating false positives.

Cache discipline

A request that hits the cache costs you nothing. Whatever else you do, get your cache headers right. Vary on the bits that need to vary, cache aggressively on the bits that do not, and lean on the page cache or the reverse proxy in front of Drupal. The cheapest bot is the one that asks for a page you have already served.

(shamless plug) you can also use my site Caching Score to review your current caching setup, to see if there is anything better you can be doing.

Caching Score

Assess how strong the caching capabilities of any given site is.

Caching Scoresean.hamlinamazee.io

What this layer is bad at

Rate limits, ModSecurity rules and geo blocks are great at volume and bad at quality. They cannot tell a scraper trickling one request per minute apart from a real user. For that you need either the edge or the application.

Edge and paid bot management

The edge is where the big vendors live, and it is where you push the cheapest blocks. A scraper rejected by Cloudflare at the network edge never gets to your origin at all.

Cloudflare

The free tier already includes Bot Fight Mode, basic challenges, and Turnstile. For most small-to-medium Drupal sites, this is a good baseline at zero extra cost. The paid Bot Management product adds custom rule logic, JA3 and JA4 TLS fingerprinting, and machine-learning-based bot scoring you can wire into firewall rules. The jump from free to paid is significant in price; the jump in capability is also significant.

Fastly, Akamai, and the rest

Fastly offers the Next-Gen WAF (originally Signal Sciences) with a Bot Management add-on. Akamai sits at the enterprise tier with the most sophisticated fingerprinting available, and a price tag to match. Beyond those, there is AWS WAF with Bot Control, DataDome, HUMAN, and Imperva — all credible, all paid, all priced for sites where bot abuse is costing real money.

The trade-offs nobody puts on the sales deck

Bot Management at the edge solves real problems. It also comes with real costs that the vendor demos skip past:

• Cost. Bot Management is almost always an add-on to the core WAF subscription, and the pricing escalates fast with traffic.
• Vendor lock-in. Your rules, your dashboards, your observability all live in the vendor's UI. Migrating off is painful.
• Accessibility and SEO. Aggressive challenges break real users, and search bots that fail a challenge will hurt your rankings. Test both before turning anything up.
• Rules live outside your application codebase. They drift, they are not versioned alongside the code that depends on them, and a rule change can break a feature without any commit to point to.
• False positives are invisible to you. By default, blocked requests do not reach your logs. You will not know which real users were turned away unless you specifically ask for that signal.

Anubis

The newest piece in the picture, and the one that has me genuinely interested.

What Anubis is

Anubis is an open-source reverse proxy (MIT licensed) that sits in front of your site and issues a proof-of-work challenge to clients before letting them through. It was built specifically for the AI scraper era — for the case where the scraper is mimicking a real browser well enough that classifying it on signal alone has stopped working.

Why proof-of-work, not CAPTCHA

The interesting move with Anubis is who pays the cost. A real user pays a few hundred milliseconds of CPU once when they first arrive, and never sees it again for the lifetime of the cookie. A scraper hitting you a million times pays the cost a million times.

That asymmetry is the whole point. CAPTCHAs put the cost on humans (the people who lose patience trying to identify traffic lights). Anubis puts it on whoever is doing the hammering. That is closer to the right shape of the trade.

Where to put it

You do not want Anubis in front of your whole site. You want it in front of the endpoints that are expensive and uncacheable. From the talk, my shortlist:

• Search endpoints
• Facet and filter URLs
• Pagination tails - ?page=2348 is not a real user
• Login, register, password reset
• Spicy forms (contact, anything that triggers an email)
• Authenticated user flows
• Anything expensive and uncacheable

Static pages stay fast. The cache stays warm. The PoW cost only applies on the routes where it earns its keep.

But what about Googlebot?

This is the first question every site owner asks, and the answer is good. Anubis ships with allowlists for known good crawlers, matching IP ranges against the published lists from Google, Bing, and the rest. The allowlist is maintained upstream, which means you need to keep Anubis deployed on a reasonable cadence to pull in the latest changes. New legitimate crawlers do show up.

Demo site

You can see Anubis in action with a demo Drupal 11 site I put together, the login form has Anubis in front of it, the homepage does not.

Log in | Drush Site-Install

Drush Site-Install

Putting the layers together

None of these defences is a silver bullet on its own. Each layer is cheap at one thing and bad at another, and the trick is matching the layer to the threat.

Block the cheap traffic at the edge. Block the lazy bots with rate limits and ModSecurity at the web server. Put Anubis in front of the endpoints that are expensive and uncacheable. Let Drupal-native modules handle the application-aware decisions where you actually need to see the user, the form, or the facet state.

Five things to take away

1. No single layer is enough. Stack them. The edge handles raw volume, the web server handles patterns, and the application is the only thing that can see real user and form context.
2. Match the protection to the threat. A login form needs different defence to a faceted search results page.
3. Measure before you defend. Look at your actual traffic. Find your most-hit uncacheable endpoints. Defend those first.
4. Watch accessibility and SEO. Every challenge you add is a tax on a real user or a real crawler. The cost of false positives is invisible unless you go looking.
5. Plan for adversarial improvement. Whatever you deploy today, the scrapers get a turn next. Pick defences you can iterate on.

One last thing

You do not need to win the bot war. You just need to make your site a worse target than the next one.

The slides from the talk are on the DrupalSouth schedule page. The recording will be posted here once the DrupalSouth team have edited and uploaded it — check back in a few weeks.

This is the complementary blog post for my DrupalSouth Sydney 2024 session, and also v2.0 follow up of sorts from the original 2022 version. The full presentation was much longer than this blog post, this blog post is just going to highlight the core statistics and findings.

Have you ever wondered how popular Drupal is in your local state and at the Australian Federal Government level? This blog post will help to answer that question, using open source tooling. The hope is that you gain some insight to the relative popularity of Drupal and appreciate more the impact you and Drupal have in Australia.

As this blog post is a follow up, you can also now start to see trends (data is around 13 months newer than the last time I did this).

Just show me the graphs

Disclaimer:

• This is based on Oct 20, 2023 data
• The scoring is based off PageRank data, so the percentages are not raw counts of websites, but an approximation of how important the respective sites are compared to others (assumes a logarithmic base of 5).
• Wappalyzer detection is not perfect (see the end of this blog post for upstreamed PRs), and there is still a fairly large portion of sites where the CMS cannot be identified
• MoGs make this tricky (PageRank relies on incoming links, which break due to MoGs)
• Only source *.gov.au domains considered (some Government sites use other TLDs)
• Unlikely newly created websites are in the top 10 million just yet (due to how the algorithm of PageRank works)

All sites (*.gov.au)

Federal sites (not state based domains)

Programmes like GovCMS are having an impact here. Also interesting that if you are not using Drupal, the chances are you have written something entirely custom.

Victoria *.vic.gov.au

The Single Digital Presence (SDP) programme makes a mark in Victoria.

New South Wales *.nsw.gov.au

Large Drupal sites like https://www.nsw.gov.au/ and https://www.service.nsw.gov.au/ help to make Drupal dominant in NSW.

South Australia *.sa.gov.au

Squiz Matrix increasing in market share ↑5.4% over 2022. There is a clear state led mandate here.

Western Australia *.wa.gov.au

A lot of sites this time around are now identified (decrease of ↓30.8% of unknown sites). Drupal also increased market share by ↑9.9%.

Tasmania *.tas.gov.au

The lowest usage of Drupal for any Australian state or territory and the highest percentage of Wordpress.

Queensland *.qld.gov.au

Australian Capital Territory *.act.gov.au

The highest percentage of Squiz compared to any other Australia state or territory.

Northern Territory *.nt.gov.au

Open Source Software (OSS) CMS vs Proprietary CMS

For the CMS' that can be identified, splitting them into 2 categories, OSS and Proprietary. OSS is determined on whether the source code is freely available, and there is a licence that allows me to run it without paying someone.

Drupal sites by major version

For sites reporting as Drupal, Drupal 10 is the most popular. Still 5.4% of Drupal sites running Drupal 7 to which will be End-of-Life (EOL) in early 2025.

Score by state and territory

This is weighted by total score, broken down by federal/state/territory.

Observations and other unusual findings

Drupal usage

“Drupal powers ~29.9% of all digital experiences that you use in the Australian government. This is↑2.7% compared to 2022”

Drupal Growth

“Relative to the growth of Australian government sites, Drupal adoption is growing faster”

Top contender

“Squiz Matrix is the top contender with 15.6%, and has a clear state lead mandate in 5 states/territories. This is↑3.5% compared to 2022”

Drupal 7 usage

“Drupal 7 usage dropped from 15.8% in 2022 to 5.4% in 2024. This is↓65.8% compared to 2022. Most popular Drupal 7 site is https://www.sl.nsw.gov.au/”

Also after my presentation I got to meet the team behind the State Library of NSW, and they advised that they are due to upgrade to Drupal 10 anytime soon.

TLS coverage is still not 100%

83 sites with HTTP only (a drop of ↓46 since 2022)

If in doubt, add a number

14 sites with ww[0-9] in the domain name (a drop of ↓5 since 2022)

I think this is often used like a form of poor mans version control, often archiving the previous version of the site. For some reason it is archived publicly.

I want the raw data!

If you want to make your own visualisations of the data, or even just do random queries “how popular is Spark CMS in Western Australia”, you can download a CSV from https://bit.ly/dsau2024csv. Slides are https://bit.ly/dsau2024.

Upstreamed enhancements

I found a lot of software running in certain states, so I upstreamed some changes to better detect these software packages:

• Better SilverStripe detection #120
• Datascape detection #122
• Spark CMS detection #123
• Jadu detection #126
• Engagement HQ detection #124
• Social Pinpoint detection #125
• Citizen Space detection #121

Comments

I am keen to hear feedback on this data, and what can be done to improve the scoring. Also, if you can help fill in some of the 'unknown' data, let me know, I am happy to craft another PR into WebAppAnalyzer.

This is the complementary blog post for my DrupalSouth Brisbane 2022 session.

Have you ever wondered how popular Drupal is in your local state and at the Australian Federal Government level? This blog post will help to answer that question, using open source tooling. The hope is that you gain some insight to the relative popularity of Drupal and appreciate more the impact you and Drupal have in Australia.

Existing solutions

There are a number of websites that will claim to be able to give you this information. However they all will likely want you at some point to pay them money.

• wappalyzer.com
• semrush.com
• builtwith.com
• whatcms.com
• similartech.com
• larger.io

“I wanted an open way to do this”

As it turns out, you can plug a few things together to scrape the technologies in use

• Wappalyzer library (open source)
• Puppeteer (open source)

Problem #1: How to get a list of all Australia Government domains

If anything, there are too many sources of this information:

• https://github.com/tb0hdan/domains
• Let's Encrypt Certificate Transparency (CT) logs
• SecurityTrails

A crawling method could also be done, loads of suitable seed sites, e.g.:

• https://www.australia.gov.au/
• https://www.directory.gov.au/departments-and-agencies

The main issue is that just having a list of sites, does not convey the importance of the site relative to another site.

Enter DomCop to which publishes a list of the top 10 million domains on the internet, including a rank and Open PageRank.

On top of suppling 5,795 Australian Government domains, there also is an "Open Page Rank" field. The PageRanks are calculated based on the Open data provided by Common Crawl and Common Search.

Problem #2: What is PageRank?

PageRank is a system for ranking web pages that Google's founders developed in 1996. A PageRank score of 0 is typically a low-quality website, whereas, a score of 10 would represent only the most authoritative sites on the web. It is logarithmic (with a base of 5).

A site with PageRank 3 is 5 times more authoritative than a site with PageRank 2.

Problem #3: Machinery of Government (MoG) and link rot

Australian Governments sites are never static, they are constantly evolving. Sometimes several sites merge into 1, or sometimes 1 site splits into move sites.

Just show me the graphs

Disclaimer:

• This is based on Sept 22, 2022 data
• The scoring is based off PageRank data, so the percentages are not raw counts of websites, but an approximation of how important the respective sites are compared to others (assumes a logarithmic base of 5).
• Wappalyzer detection is not perfect (see the end of this blog post for upstreamed PRs), and there is still a fairly large portion of sites where the CMS cannot be identified
• MoGs make this tricky (PageRank relies on incoming links, which break due to MoGs)
• Only *.gov.au domains considered (some Government sites use other TLDs)
• Unlikely newly created websites are in the top 10 million just yet (due to how PageRank works)

All sites (*.gov.au)

Federal sites (not state based domains)

Programmes like GovCMS are having an impact here.

Victoria *.vic.gov.au

The Single Digital Presence (SDP) programme makes a mark in Victoria.

New South Wales *.nsw.gov.au

Large Drupal sites like https://www.nsw.gov.au/ and https://www.service.nsw.gov.au/ help to make Drupal dominant in NSW.

South Australia *.sa.gov.au

Western Australia *.wa.gov.au

A lot of unknown CMSs in WA, including sites like https://ww2.health.wa.gov.au/ which I still have no idea what the CMS used is. Edit - a keen eyed developer has told me that because this URL exists, so the WA Health site is generated by SiteCore.

Tasmania *.tas.gov.au

The lowest usage of Drupal for any Australian state or territory and the highest percentage of Wordpress.

Queensland *.qld.gov.au

Australian Capital Territory *.act.gov.au

The highest percentage of Squiz compared to any other Australia state or territory.

Northern Territory *.nt.gov.au

Open Source Software (OSS) CMS vs Proprietary CMS

For the CMS' that can be identified, splitting them into 2 categories, OSS and Proprietary.

Drupal sites by major version

For sites reporting as Drupal, Drupal 9 and 7 are the most popular.

Observations and other unusual findings

#1 - Drupal usage

“Drupal powers roughly 27% of all digital experiences that you use in the Australian government”

#2 - Top contender

“Squiz Matrix is the top contender with 15%, and has a clear state led mandate in certain states/territories”

#3 - TLS coverage

TLS coverage is not 100% - 129 domains found with no TLS

#4 - If in doubt, add a number

19 domains found with ww[number] as a subdomain.

#5 - You cannot kill Dreamweaver

15 sites found in 2022.

Extending this for the future

• Crawl other domain spaces, e.g. the New Zealand government domain space *.govt.nz
• Make a website and publish this data quarterly (DomCop's data updates around this frequency)
• Measure trends over time

Upstreamed enhancements

These are all to make the detection of CMS' and Javascript frameworks more accurate for Australia Government sites.

• Ripple #6827
• Dreamweaver #6828
• HCL DX #6829
• MODX #6832
• Squiz Matrix #6833
• Optimizely CMS #6834
• Umbraco #6835
• Elcom #6836
• Lagoon #6861
• OSS flags and links #6902
• Squiz Matrix #2 #6925
• Ektron CMS #6924
• SiteCore #6926

Raw data

If you want to do your own analysis, here is a link to a full CSV dump.

Comments

I am keen to hear feedback on this data, and what can be done to improve the scoring. Also, if you can help fill in some of the 'unknown' data, let me know, I am happy to craft another PR into Wappalyzer.

This topic came up a while ago when Acquia was looking to introduce Soft Purge as an option for their Acquia purge module in Drupal (N.B. not to be confused by the Fastly Drupal module).

There appears to be some confusion about what Fastly Soft Purge is, when it is useful, and how to use it. This post will help clear this topic up.

What is Fastly Soft Purge?

In Varnish, to force a miss for an object in cache you have a couple of options

• evict it completely (e.g. PURGE, or BAN with a lurker). The Varnish documentation cover this. This is the same as Fastly Instant Purge.
• or simply mark the object as expired. I imagine Fastly achieve this by tinkering with the TTL of the cached object. This is what Fastly Soft Purge is.

So the main difference with Fastly Soft Purge is that the object persists in cache, and can be used in a stale context. This can prove to be extremely useful.

When is Fastly Soft Purge useful?

I would say Fastly Soft Purge should be used in 99.99% of all purges. There are extremely limited situations in which I would recommend Fastly Instant Purge:

• You accidentally published embargoed content, or there is some legal issue with the content
• You accidentally published incorrect or incomplete content
• You accidentally published test content (we have all been there)
• You need to destroy the entire cache for a given Fastly service 😱 (which I never really recommend)

It boils down to making a whoopsie really, and I would imagine the vast majority of your purges should (hopefully) not fall into these categories.

Having a stale object is cache is useful as it:

• soft purges to high traffic pages (e.g. the homepage) will not result in slow downs for new users, this is because "Stale while revalidate" will perform a background (asynchronous) refresh of the cached object, and will serve the cached stale object in the meantime.
• if your origin is completely down (or in maintenance) stale objects can be served to users (which is often better than a generic error page). This is the "Stale if error" feature.

How to use Fastly Soft Purge with the Drupal module

Just because you execute a purge to Fastly does not mean you will instantly get any benefit, you also need to combo this with some "Stale Content Options" to which I will go through.

If you are using the Fastly Drupal module, Soft Purge is configured like so:

And as for Stale Content Options, you need to ensure that both "Stale while revalidate" and "Stale if error" are enabled. These are some basic tuning for the values as well:

If you are not using Drupal (or using the Acquia purge module), then you can always set the above HTTP headers manually. See the Fastly documentation on this.

How to use Fastly Soft Purge with the API only

If you don't run Drupal, or just want to execute a soft purge via the Fastly API, then this is simple as well, just pass in the Fastly-Soft-Purge HTTP header:

curl -sXPOST https://api.fastly.com/service/SERVICE_ID/purge/TAG \
 -H 'Fastly-Soft-Purge:1' \
 -H "Fastly-Key:$FASTLY_TOKEN" | jq
{
  "status": "ok",
  "id": "10427-1615460322-6"
}

Wait, I want to learn more

If you want to go deeper in Varnish (what powers Fastly), and how objects, TTL, stale and grace (stale-while-revalidate) are involved, please see their documentation.

There is also this excellent graphic to which explains how an object goes from fresh to stale

Gotchas

If you use Fastly shielding (which you probably should), then there is the potential for soft purged content on the edge PoP to end up being re-cached as fresh 😱. See the Fastly documentation on this. There is a VCL snippet to mitigate this:

if (fastly.ff.visits_this_service > 0) {
  set req.max_stale_while_revalidate = 0s;
}

If you are not already using Fastly shielding, and you care about caching, and cache hit rates, you should really start to look at this. The benefits far outweigh the tiny edge cases.

P.S. if you are not already using the Fastly purge auth snippet, get this installed ASAP. By not doing this, a bad actor is able to purge any single URL on your site, in an instant fashion. This could create a DoS.

# Fastly's URL purge feature allows you to purge individual URLs on your
# website. By default, authentication is not required to purge a URL with the
# Fastly API, but you can enable API token authentication with this snippet.
#
# @see https://docs.fastly.com/en/guides/authenticating-api-purge-requests
if ( req.request == "FASTLYPURGE" ) {
    set req.http.Fastly-Purge-Requires-Auth = "1";
}

Comments

If you have any other pro tips when it comes to soft purging in Fastly, please let me know.

Background

I was recently part of a migration from AWS (Amazon Aurora - MySQL-Compatible) to Azure (Azure database for MariaDB 10.3) for a large suite of applications. This platform contains a number of Drupal 8 sites, which surface content through JSONAPI (now part of core in Drupal 8).

The issue

Drupal is extremely flexible, and creates a highly normalised table structure, 2 tables per field on a piece of content (1 for revisions, and another for active data). A given piece of content can contain dozens of fields. When loading a piece of content, it is not uncommon in Drupal to have 20+ joins on a single SQL query.

These SQL queries the end developers do not write by hand, Drupal abstracts this detail away through the entity API.

We were seeing SQL queries appear to never complete when they had lots of JOINs in them. One query I found that was 'stuck' had 53 joins. In saying that, the database overall data size was tiny, with only 228 pieces of content in Drupal (this is very low, some Drupal sites can have millions of items of content).

Running a SHOW FULL PROCESSLIST showed these queries were all stuck in a Statistics phase.

The issue seemed to disproportionately impact SQL queries with > 40 joins in a single query.

These queries did not appear to ever complete, and the CPU was pegged at 100%.

Upsizing the vCPU count in the database cluster had no impact, as the 'stuck' queries just consume all the CPU available. We even had trouble trying to connect to the database cluster via the MySQL client, due to timeouts.

Research

After seeing the seeing queries stuck in Statistics phase, we did some digging to see what other content had to say on this topic:

• https://dev.mysql.com/doc/refman/8.0/en/general-thread-states.html seemed to indicate that Statistics phase hanging could be due to a lack of IOPS
• https://stackoverflow.com/questions/17797191/sql-query-stuck-in-statistics-state gave the first clue about optimizer_search_depth and how this can blow out query times
• https://www.drupal.org/project/ubercart/issues/636574 indicated that tuning optimizer_search_depth to a value lower than 62 would be a good idea.
• https://mariadb.com/resources/blog/setting-optimizer-search-depth-in-mysql/ contains really useful information on tuning optimizer_search_depth in general
• https://lists.mysql.com/internals/37635 has the background on why 62 was chosen as the default for optimizer_search_depth

Actions we did

After reading the above literature, we ended up settling on the 'automatic' tuning for optimizer_search_depth.

 optimizer_search_depth = 0

This will mean that queries that do > 7 JOINS may not run the best query path, but at least they will actually complete. This is a win in my books.

I also killed every running query manually that was stuck in the Statistics phase. This brought down the CPU. The CPU never went up to the same levels due to the change in optimizer_search_depth.

Result and final thoughts

The 53 JOIN query that used to fail to complete, now completes in 268ms. Not fast, but a damn sight faster than several days.

I still don't have a good explanation as to why the 53 join query had no issues executing on AWS Aurora MySQL, optimizer_search_depth is set to the default of 62 on there. I assume (like most of Aurora) that there is some special AWS sauce helping here. If anyone can shed more information on this, please let me know in the comments.

Ordinarily, sessions in Drupal are created when you log into Drupal, e.g. when you want to create content, or perform some other function that requires authentication.

You can however (just like in any PHP application) start a session without first authenticating. This is known as an anonymous session.

This is almost always a poor decision, because as soon as you start a session, this generates a session cookie, and all reverse proxy caches will refuse to cache the responses from Drupal. Drupal will also add caching headers to indicate the response is not cacheable as well.

There are often better places to store state for anonymous users, e.g. the browsers localStorage, these have the advantage of not interfering with Drupal or caching.

This post will attempt to highlight some paths and avenues you can explore, the next time you come across this in an application you are working on.

Known contributed module offenders

Acquia publishes a fairly broad list of modules to which are known to break caching, some examples include

• global_filter
• flag (when voting as anonymous users is allowed)
• textsize
• ip_geoloc
• smart_ip

If you are looking to store state (e.g. flag, textsize) then localStorage is better suited to that. If you are doing geolocation based things, most CDNs have the ability to do this lookup for you.

If you are using these modules, use with caution, and ensure you are not breaking the caching of your entire site.

Known Drupal core offenders

If you set a message using drupal_set_message() (in Drupal 7) or \Drupal::messenger()->addStatus(); (in Drupal 8/9) this will store the message in a PHP session. What sounds like a cool idea at the time - e.g. "show a message to all users on the homepage" - will end up making the site completely un-cacheable.

This can (and has) taken down high traffic sites.

Custom code

Custom code is another place you will find code that may set anonymous sessions. Doing a quick grep will speed things up:

grep -nrI --exclude-dir=web/vendor --exclude-dir=web/core '$_SESSION' web/

Check the sessions table

It is fairly easy to spot the culprit when you examine the sessions table in Drupal's database. Here is a quick query to show the current count of anonymous sessions. This number ideally should be 0 or close to it.

MySQL [example]> SELECT count(*) FROM sessions WHERE uid = 0;
+----------+
| count(*) |
+----------+
|    10853 |
+----------+
1 row in set (0.004 sec)

If you see you have a lot of sessions, then the next step is to read the contents of one:

MySQL [example]> SELECT SUBSTR(session, 1, 650) as '' FROM sessions WHERE uid = 0 ORDER BY timestamp DESC LIMIT 1;

_symfony_flashes|a:1:{s:5:"error";a:149:{i:0;O:25:"Drupal\Core\Render\Markup":1:{s:9:" * string";s:2078:"<em class="placeholder">Notice</em>: Object of class Drupal\Core\Field\FieldItemList could not be converted to int in <em class="placeholder">example_preprocess_node()</em> (line <em class="placeholder">146</em> of <em class="placeholder">themes/example/example.theme</em>). <pre class="backtrace">example_preprocess_node(Array, &#039;node&#039;, Array) (Line: 287)
Drupal\Core\Theme\ThemeManager-&gt;render(&#039;node&#039;, Array) (Line: 437)
Drupal\Core\Render\Renderer-&gt;doRender(Array, ) (Line: 195)
Drupal\Core\Render\Renderer-&gt;render(Array, ) (Line: 

1 row in set (0.001 sec)

So now you know that there is a bug in the code example_preprocess_node() around line 146. This is the source of the messages in the first place.

After reading this great answer on stackoverflow this is likely due to the fact that the messages block is not rendered in the theme.

After checking the block layout page, the block is listed there

However checking the page templates, the variable {{ page.highlighted }} was completely omitted. This was the root cause of the messages not being able to sent to the browser, and thus they keep piling up in the sessions table.

If you want to see how much data you are storing for these sessions:

MySQL [example]> SELECT uid, CONCAT(SUM(LENGTH(session)) / 1048576.0, ' MB') as size FROM sessions  WHERE uid = 0 GROUP BY uid;
+-----+--------------+
| uid | size         |
+-----+--------------+
|   0 | 2433.6637 MB |
+-----+--------------+
1 row in set (0.513 sec)

Around 2.4GB for this one site 😱. That is a lot of messages.

Comments

If you have any other tips and tricks for hunting down anonymous sessions, or even any (anonymized) war stories - please let me know in the comments.

The steps that any new launch or high traffic event should go through in order to have the best chance of success. This post is aimed at the project management level, so will try to stay out of the weeds, and focus on the high level topics you need to think about. There is a ~18 minute recording at the end of this post where I presented this topic at Drupalsouth 2019.

Preamble: What could be considered a high traffic event

Launching a brand new site

• Re-platforming (e.g. moving CMS version or type, or between hosting providers)
• eDM or other marketing event (e.g. Adwords)
• Planned traffic event (e.g. black Friday)
• Unplanned traffic event (e.g. news and media site)

Step 1) Ensure you have some basic Drupal configuration in place

• Disable known problem child modules dblog, devel, statistics, radioactivity, page_cache
• Enable dynamic_page_cache (if you have authenticated traffic)
• Set minimum cache lifetime to something sensible
• JS and CSS aggregation enabled
• Automate these checks with Drutiny

Step 2) Content Delivery Network (CDN)

Additional insurance against a lot of traffic is distributing your cached content to all corners of the globe.

Tiered caching should be used to ensure the highest offload rate. Most CDN providers will support this at a given price point.

Step 3) Cache tuning and minimising origin requests

Every request that bypasses your CDN layer adds load to the platform. In order to have the best chance of surviving a high traffic event, origin traffic needs to be carefully considered and reduced where possible.

Requests to origin that are often overlooked

• 404s
• Marketing based parameters (e.g. utm_campaign)
• Redirects (especially if re-platforming)
• WAF to block silly requests (e.g. Wordpress URLs like wp-login.php)

It you are interested in WAF tuning, you should check out my talk last year on using Cloudflare to secure your Drupal site.

Step 4) Load testing

If you are building a new site, or are expecting a substantially different traffic profile than what you have currently, then you should look to load test the system.

• Production hardware replica (scaled up if appropriate)
• Emulate expected user behaviour, use existing analytics, or expected flows
• Emulate what the browser would be doing (download all assets, including any HTTP 404s)
• Ensure complex tasks are also simulated at the same time (e.g. editorial, searching, form submissions, feeds ingestions)

At the end of this task (they you may need to run several times), you should have the confidence that you can handle the traffic expected.

Step 5) Hardware (auto) scaling

Now that you have the hardware you need to have in place with load testing, ensure you have autoscaling in place to deal with the peaks and troughs (it is unlikely you need to run your peak hardware for the entire duration of the event).

Autoscaling can also help if the origin traffic that you experience is higher than anticipated.

Test the autoscaler, set limits that you are comfortable with, and ensure you know how quickly the new resources take to come to life.

Step 6) Have a good fallback

Say the worst does happen, and you site does go down, or a critical API drops off the face of the internet, what does the end user see? Can you offer at least a better experience than a generic web server error page?

Most CDNs will have the ability to load balance origins (hot DR), and even fallback to a static version of the site if all origins are down.

It would make sense to test this prior to the high load event as well.

Step 7) Warm your cache

If you have a rather long tail website, it will be worth warming your cache prior to the event. An excellent module called warmer has been written, to which allows warming all sorts of caches. It can for instance load every page in the XML sitemap. So this is fairly low effort, high reward.

Step 8) Third party API dependencies

This is more of a fundamental design decision likely made much earlier on in the project. Say the content of your page is dependent on the content in an API response.  If you request the API content during page generation time, then you are tying the speed and availability of your site to another site (often outside your control).

This can lead to slow page load times, and worse case scenario can tie up your server's resources.

New Relic APM has "external requests" to which allow you to visualise this.

There are ways to mitigate this:

• Fetch the data in the background and cache locally in Drupal for as long as the data is considered 'good'. e.g. using Drush and a cronjob.
• Use a client side application (e.g. React) and request the API response in the client side
• Use a CDN on the API and see Step #6 above

Step 9) Realtime analytics

During the event, having access to realtime (or near realtime) analytics to find out

• how the system is currently performing
• requests/sec
• where the traffic is coming from
• cache offload rate from the CDN

Is extremely valuable. Even more valuable is being able to respond to this data in a quick and efficient matter. Having access to technical people can help. The types of logs and analytics you should be looking to get a hold of:

• Web analytics tools (e.g. Google Analytics)
• APM tools (e.g. New Relic)
• CDN analytics (e.g. Cloudflare Logs)
• Log stream from hosting provider (e.g. PHP error log)

To see where you can take this, you might also be interested in reading this blog post that shows off some dashboards that were purpose built for a high traffic event.

Step 10) Application changes in a pinch

If you do spot something in your analytics, knowing what tools you have at your disposal to mitigate issues quickly and easily is worth knowing.

• Cloudflare page rules (redirect a broken path, increase the WAF presence on a route)
• Nginx or Apache configuration
• Application hotfix (avoid clearing the cache)

Knowing what tool will solve what problem, how long each option takes to deploy, how safe it is, how easy is the rollback is is absolutely critical.

Step 11) Letting your hosting provider and their support team know

No-one likes surprises, so plan ahead. Ensure there are people available or on call during your traffic event. This goes for both your hosting provider, to CDN provider to support staff.

Postamble: What success looks like

So after your high traffic event has ended, here are some simple things to check in order to see how successful you were:

• Minimal origin requests and a high CDN offload
• Boring origin hardware graphs
• No rants on twitter
• No trending hashtag on twitter that is negative
• Users remember the event for it's content, and not the problems with it

Drupalsouth 2019 video

Let me know in the comments if this was of use, and also if you have any other words of wisdom for anyone else.

On a number of Drupal sites that I am involved with recently I see the error message when either trying to dump or restore certain databases:

ERROR 2020 (HY000) at line 1: Got packet bigger than 'max_allowed_packet'

It is important to note that with MySQL there are 2 settings for max_allowed_packet, the client (what you are connecting from), and the server (what you are connecting to).

In order to find out the client's current settings, you can run:

$ mysql --help | grep max-allowed-packet | grep -v '#' | awk '{print $2/(1024*1024)}'
16

So 16MB on the client side.

In order to find out the server's current setting, you can run:

SHOW VARIABLES LIKE 'max_allowed_packet';
+--------------------+----------+
| Variable_name      | Value    |
+--------------------+----------+
| max_allowed_packet | 67108864 |
+--------------------+----------+

So 64MB on the server side.

So at the moment:

• The client cannot send or receive more than 16MB in a single statement
• The server cannot send or receive more than 64MB in a single statement

So you can end up in a position where there is content in your database, that is working fine, but you cannot dump the database, nor can you restore from a dump (with your current client configuration).

Take the search_api_attachments module in Drupal, it uses the key_value table in Drupal 8 to store extracted text from documents.

SELECT name, length(value) as size FROM key_value ORDER BY size DESC limit 5;
+------------------------------+----------+
| name                         | size     |
+------------------------------+----------+
| search_api_attachments:14646 | 33623803 |
| search_api_attachments:14288 |  3394023 |
| search_api_attachments:13146 |  2356958 |
| search_api_attachments:4921  |  1554830 |
| search_api_attachments:1586  |  1549981 |
+------------------------------+----------+

Solution 1 - alter max_allowed_packet

The first solution is simply to alter the max_allowed_packet size to accommodate the size needed (on both the client and server). The only issue is that this is a game of cat and mouse. As soon as you tune the size to be larger, a content editor will upload a larger document.

It also means your database size will grow fairly unchecked, especially if you have heavy editorial where documents are commonly uploaded.

While I do advocate for sensible defaults, I think having > 64MB in a single cell in a table as potentially overkill for the benefits it provides.

Solution 2 - reduce the amount in the database

The point of indexing attachments is to ensure that you can still find content that is buried in binary files. I would argue that most of the important keywords of these documents will be in the first few pages. Indexing the entire document often will provide limited added value (if any).

Reading through the code of the search_api_attachments module, I spot these useful lines:

  /**
   * Limit the indexed text to first N bytes.
   *
   * @param string $extracted_text
   *   The hole extracted text.
   *
   * @return string
   *   The first N bytes of the extracted text that will be indexed and cached.
   */
  public function limitBytes($extracted_text) {
    $bytes = 0;
    if (isset($this->configuration['number_first_bytes'])) {
      $bytes = Bytes::toInt($this->configuration['number_first_bytes']);
    }
    // If $bytes is 0 return all items.
    if ($bytes == 0) {
      return $extracted_text;
    }
    else {
      $extracted_text = mb_strcut($extracted_text, 0, $bytes);
    }
    return $extracted_text;
  }

So it turns out, baked into the module, is a way to effectively limit the number of characters stored in the database (drupal.org issue).

In order to enable this feature, you need to edit the Processors for a given index:

Inside this page, is a configuration form that allows you to set a max limit for the amount stored in the database.

After setting the size to 100 KB which seems like a reasonable number, and then re-indexing the appropriate index, you see the results

SELECT name, length(value) as size FROM key_value ORDER BY size DESC limit 5;
+--------------------------------+--------+
| name                           | size   |
+--------------------------------+--------+
| node.field_storage_definitions | 116308 |
| search_api_attachments:831     | 102412 |
| search_api_attachments:1536    | 102412 |
| search_api_attachments:1571    | 102412 |
| search_api_attachments:1576    | 102412 |
+--------------------------------+--------+

So this will mean that the database is a lot smaller, allowing faster database backups, restores, and rollbacks. It also will save a lot of issues around forever tuning max_allowed_packet.

It is also worth noting that this key_value storage is actually a caching system for search_api_attachments, and that this table will be used, even if your only search servers are external to Drupal (e.g. Solr), and you make no use of database searching.

Update 21 November 2019

In the hopes that the module maintainers make a more sensible default limit I have also raised this issue to get a default value set. Having any size limit would be better than having no limit. A patch is now uploaded so you can test this out.

Update 23 November 2019

The patch has been committed, and a new beta released for Drupal 8 🎉. Thanks so much to Ismaeil (module maintainer) for the prompt reply and action.

Comments

If you have come across this in the past, what was your go to solution? I am keen to understand how others have solved this issue, and how big your key_value table got in the mean time.

I am writing this quick tutorial in the hopes it helps someone else out there. There are a few guides out there to do similar tasks to this. They just are not quite what I wanted.

To give everyone an idea on the desired outcome, this is what I wanted to achieve:

Before I dive into this, I will mention that you can do this with views, if all that you want to produce is content supplied by views. Ivan wrote a nice article on this. In my situation, I wanted a completely custom route, controller and theme function. I wanted full control over the output.

Steps to add sub tabs

Step 1 - create a new module

If you don't already have a module to house this code, you will need one. These commands make use of Drupal console, so ensure you have this installed first.

drupal generate:module --module='Example module' --machine-name='example' --module-path='modules/custom' --description='My example module' --package='Custom' --core='8.x'

Step 2 - create a new controller

Now that you have a base module, you need a route

drupal generate:controller --module='example' --class='ExampleController' --routes='"title":"Content", "name":"example.user.contentlist", "method":"contentListUser", "path":"/user/{user}/content"'

Step 3 - alter your routes

In order to use magic autoloading, and also proper access control, you can alter your routes to look like this. This is covered in the official documentation.

# Content user tab.
example.user.contentlist:
  path: '/user/{user}/content'
  defaults:
    _controller: '\Drupal\example\Controller\ExampleController::contentListUser'
    _title: 'Content'
  requirements:
    _permission: 'access content'
    _entity_access: 'user.view'
    user: \d+
  options:
    parameters:
      user:
        type: entity:user

# Reports user tab.
example.user.reportList:
  path: '/user/{user}/reports'
  defaults:
    _controller: '\Drupal\example\Controller\ExampleController::reportListUser'
    _title: 'Reports'
  requirements:
    _permission: 'access content'
    _entity_access: 'user.view'
    user: \d+
  options:
    parameters:
      user:
        type: entity:user

Step 4 - create example.links.task.yml

This is the code that actually creates the tabs in the user profile. No Drupal console command for this unfortunately. The key part of this is defining base_route: entity.user.canonical.

example.user.content_task:
  title: 'Content'
  route_name: example.user.contentlist
  base_route: entity.user.canonical
  weight: 1

example.user.reports_task:
  title: 'Reports'
  route_name: example.user.reportList
  base_route: entity.user.canonical
  weight: 2

Step 5 - enable the module

Don't forget to actually turn on your custom module, nothing will work until then.

drush en example

Example module

The best (and simplest) example module I could find that demonstrates this is the Tracker module in Drupal core. The Tracker module adds a tab to the user profile.

This is a short story on an interesting problem we were having with the Feeds module and Feeds directory fetcher module in Drupal 7.

Background on the use of Feeds

Feeds for this site is being used to ingest XML from a third party source (Reuters). The feed perhaps ingests a couple of hundred articles per day. There can be updates to the existing imported articles as well, but typically they are only updated the day the article is ingested.

Feeds was working well for over a few years, and then all of a sudden, the ingests started to fail. The failure was only on production, whilst the other (lower environments) the ingestion worked as expected.

The bizarre error

On production we were experiencing the error during import:

PDOStatement::execute(): MySQL server has gone away database.inc:2227 [warning] 
PDOStatement::execute(): Error reading result set's header [warning] 
database.inc:2227PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has [error]

The error is not so much that the database server is not alive, more so that PHP's connection to the database has been severed due to exceeding MySQL's wait_timeout value.

The reason why this would occur on only production happens on Acquia typically when you need to read and write to the shared filesystem a lot. As lower environments, the filesystem is local disk (as the environments are not clustered) the access is a lot faster. On production, the public filesystem is a network file share (which is slower).

Going down the rabbit hole

Working out why Feeds was wanting to read and/or write many files from the filesystem was the next question, and immediately one thing stood out. The shear size of the config column in the feeds_source table:

mysql> SELECT id,SUM(char_length(config))/1048576 AS size FROM feeds_source GROUP BY id;
+-------------------------------------+---------+
| id                                  | size    |
+-------------------------------------+---------+
| apworldcup_article                  |  0.0001 |
| blogs_photo_import                  |  0.0003 |
| csv_infographics                    |  0.0002 |
| photo_feed                          |  0.0002 |
| po_feeds_prestige_article           |  1.5412 |
| po_feeds_prestige_gallery           |  1.5410 |
| po_feeds_prestige_photo             |  0.2279 |
| po_feeds_reuters_article            | 21.5086 |
| po_feeds_reuters_composite          | 41.9530 |
| po_feeds_reuters_photo              | 52.6076 |
| example_line_feed_article           |  0.0002 |
| example_line_feed_associate_article |  0.0001 |
| example_line_feed_blogs             |  0.0003 |
| example_line_feed_gallery           |  0.0002 |
| example_line_feed_photo             |  0.0001 |
| example_line_feed_video             |  0.0002 |
| example_line_youtube_feed           |  0.0003 |
+-------------------------------------+---------+

Having to deserialize 52 MB of ASCII in PHP is bad enough.

The next step was dumping the value of the config column for a single row:

drush --uri=www.example.com sqlq 'SELECT config FROM feeds_source WHERE id = "po_feeds_reuters_photo"' > /tmp/po_feeds_reuters_photo.txt

Then open the resulting file in vim:

"/tmp/po_feeds_reuters_photo.txt" 1L, 55163105C

And sure enough, inside this config column was a reference to every single XML file ever imported, a cool ~450,000 files.

a:2:{s:31:"feeds_fetcher_directory_fetcher";a:3:{s:6:"source";s:23:"private://reuters/pass1";s:5:"reset";i:0;
s:18:"feed_files_fetched";a:457065:{
s:94:"private://reuters/pass1/topnews/2018-07-04T083557Z_1_KBN1JU0WQ_RTROPTC_0_US-CHINA-AUTOS-GM.XML";i:1530693632;
s:94:"private://reuters/pass1/topnews/2018-07-04T083557Z_1_KBN1JU0WR_RTROPTT_0_US-CHINA-AUTOS-GM.XML";i:1530693632;
s:96:"private://reuters/pass1/topnews/2018-07-04T083557Z_1_LYNXMPEE630KJ_RTROPTP_0_USA-TRADE-CHINA.XML";i:1530693632;
s:97:"private://reuters/pass1/topnews/2018-07-04T083617Z_147681_KBE99T04E_RTROPTT-LNK_0_OUSBSM-LINK.XML";i:1530693632;
s:102:"private://reuters/pass1/topnews/2018-07-04T083658Z_1_KBN1JU0X2_RTROPTT_0_JAPAN-RETAIL-ZOZOTOWN-INT.XML";i:1530693632

So this is the root cause of the problem, Drupal is attempting to stat() ~450,000 files that do not exist, and these files are mounted on a network file share. This process took longer than MySQL's wait_timeout and MySQL closed the connection. When Drupal finally wanted to talk to the database, it was not to be found.

Interesting enough, the problem of the config column running out of space came up in 2012, and "the solution" was just to change the type of the column. Now you can store 4GB of content in this 1 column. In hindsight, perhaps this was not the smartest solution.

Also in 2012, you see the comment from @valderama:

However, as feed_files_fetched saves all items which were already imported, it grows endless if you have a periodic import.

Great to see we are not the only people having this pain.

The solution

The simple solution to limp by is to increase the wait_timeout value of your Database connection. This gives Drupal more time to scan for the previously imported files prior to importing the new ones.

$databases['default']['default']['init_commands'] = [
  'wait_timeout' => "SET SESSION wait_timeout=2500",
];

As you might guess, this is not a good long term solution for sites with a lot of imported content, or content that is continually being imported.

Instead we opted to do a fairly quick update hook that would loop though all of the items in the feed_files_fetched key, and unset the older items.

<?php

/**
 * @file
 * Install file.
 */

/**
 * Function to iterate through multiple strings.
 *
 * @see https://www.sitepoint.com/community/t/strpos-with-multiple-characters/2004/2
 * @param $haystack
 * @param $needles
 * @param int $offset
 * @return bool|int
 */
function multi_strpos($haystack, $needles, $offset = 0) {
  foreach ($needles as $n) {
    if (strpos($haystack, $n, $offset) !== FALSE) {
      return strpos($haystack, $n, $offset);
    }
  }
  return false;
}

/**
 * Implements hook_update_N().
 */
function example_reuters_update_7001() {
  $feedsSource = db_select("feeds_source", "fs")
    ->fields('fs', ['config'])
    ->condition('fs.id', 'po_feeds_reuters_photo')
    ->execute()
    ->fetchObject();

  $config = unserialize($feedsSource->config);

  // We only want to keep the last week's worth of imported articles in the
  // database for content updates.
  $cutoff_date = [];
  for ($i = 0; $i < 7; $i++) {
    $cutoff_date[] = date('Y-m-d', strtotime("-$i days"));
  }

  watchdog('FeedSource records - Before trimmed at ' . time(), count($config['feeds_fetcher_directory_fetcher']['feed_files_fetched']));

  // We attempt to match based on the filename of the imported file. This works
  // as the files have a date in their filename.
  // e.g. '2018-07-04T083557Z_1_KBN1JU0WQ_RTROPTC_0_US-CHINA-AUTOS-GM.XML'
  foreach ($config['feeds_fetcher_directory_fetcher']['feed_files_fetched'] as $key => $source) {
    if (multi_strpos($key, $cutoff_date) === FALSE) {
      unset($config['feeds_fetcher_directory_fetcher']['feed_files_fetched'][$key]);
    }
  }

  watchdog('FeedSource records - After trimmed at ' . time(), count($config['feeds_fetcher_directory_fetcher']['feed_files_fetched']));

  // Save back to the database.
  db_update('feeds_source')
    ->fields([
      'config' => serialize($config),
    ])
    ->condition('id', 'po_feeds_reuters_photo', '=')
    ->execute();
}

Before the code ran, there were > 450,000 items in the array, and after we are below 100. So a massive decrease in database size.

More importantly, the importer now runs a lot quicker (as it is not scanning the shared filesystem for non-existent files).

What this means for Feeds in Drupal 8

It came to my attention from Dinesh, that this same issue may likely impact Drupal 8 feeds. To make things slightly more interesting, is that the functionality of the Drupal 7 module feeds_fetcher_directory is now moved into the main feeds module.

An issue has been opened on Drupal.org to track this. I will update this blog post once we know the outcome.

Update - 27 September 2019

The above update hook has been run on production (to where Gluster is used). Feeds used to take upwards of 30 minutes to run there (even if there was no new files to process). Post the update hook running, it is now under 1 minute. We were also able to remove the wait_timeout setting as well. So this is a nice result.

PHP 7.3.0 was released in December 2018, and brings with it a number of improvements in both performance and the language. As always with Drupal you need to strike a balance between adopting these new improvements early and running into issues that are not yet fixed by the community.

Why upgrade PHP to 7.3 over 7.2?

• It is around 10% faster compared to PHP 7.2 - some basic benchmarks for Drupal on https://kinsta.com/blog/php-benchmarks/#drupal-benchmarks
• A bunch of quality of life improvements to the language - e.g. flexible heredoc and nowdoc syntaxes, allowing a trailing comma in function calls and better JSON parsing error messages (just to name a few). I would recommend reading this great blog post on the topic if you want to know more.

What hosting providers support PHP 7.3?

All the major players have support, here is how you configure it for each.

Acquia

Somewhere around April 2019 the option to choose PHP 7.3 was released. You can opt into this version by changing a value in Acquia Cloud. This can be done on a per environment basis.

Pantheon

Pantheon have had support since the April 2019 as well (see the announcement post). To change the version, you update your pantheon.yml file (see the docs on this).

# Put overrides to your pantheon.upstream.yml file here.
# For more information, see: https://pantheon.io/docs/pantheon-yml/
api_version: 1
php_version: 7.3

On a side note, it is interesting that PHP 5.3 is still offered on Pantheon (end of life for nearly 5 years).

Platform.sh

Unsure when Platform.sh released PHP 7.3, but the process to enable it is very similar to Pantheon, you update your .platform.app.yaml file (see the docs on this).

# .platform.app.yaml
type: "php:7.3"

Dreamhost

PHP 7.3 is also available on Dreamhost, and can be chosen in a dropdown in their UI (see the docs on this).

Dreamhost also win some award for also allowing the oldest version of PHP that I have seen in a while (PHP 5.2).

When can you upgrade PHP 7.3

Drupal 8

As of Drupal 8.6.4 (6^th December 2018), PHP 7.3 is fully supported in Drupal core (change record). I have been running PHP 7.3 with Drupal 8 for a while now and have seen no issues, and this includes running some complex installation profiles such as Thunder and Lightning.

Any Drupal 8 site that is reasonably up to date, should be fine with PHP 7.3.

Drupal 7

Slated for support in the next release of Drupal 7 - being Drupal 7.68 (see the drupal.org issue), however there are a number of related tasks that seem like deal breakers. There also is not PHP 7.3 and Drupal 7 tests running daily either.

It seems like for the mean time, it is probably best to hold off on the PHP 7.3 upgrade until 7.68 is out the door, and also contributed modules have had a chance to upgrade and release a new stable release.

A simple search on Drupal.org yields the following modules that look like they may need work (more are certainly possible):

• composer_manager (issue)
• scald (issue) [now fixed and released]
• video (issue)
• search_api (issue) [now fixed and released]

Most of the issues seem to be related to this deprecation - Deprecate and remove continue targeting switch. If you know of any other modules that have issues, please let me know in the comments.

Drupal 6

For all you die hard Drupal 6 fans out there (I know a few large websites still running this), you are going to be in for a rough ride. There is a PHP 7 branch of the d6lts Github repo, so this is promising, however the last commit was September 2018, so this does not bode well for PHP 7.3 support. I also doubt contributed modules are going to be up to scratch (drupal.org does not even list D6 versions of modules anymore).

To test this theory, I audited the current 6.x-2.x branch of views

$ phpcs -p ~/projects/views --standard=PHPCompatibility --runtime-set testVersion 7.3
................................................W.W.WW.W....  60 / 261 (23%)
................................E........................... 120 / 261 (46%)
...................................................EE....... 180 / 261 (69%)
............................................................ 240 / 261 (92%)
.....................                                        261 / 261 (100%)

3 errors in views alone. The errors are show stoppers too

Function split() is deprecated since PHP 5.3 and removed since PHP 7.0; Use preg_split() instead

If this is the state of one of the most popular modules for Drupal 7, then I doubt the lesser known modules will be any better.

If you are serious about supporting Drupal 6, it would pay to get in contact with My Drop Wizard, as they at least at providing support for people looking to adopt PHP 7.

Part of my day job is to help tune the Cloudflare WAF for several customers. This blog post helps to summarise some of the default rules I will deploy to every Drupal (7 or 8) site as a base line.

The format of the custom WAF rules in this blog post are YAML format (for humans to read), if you do want to create these rules via the API, then you will need them in JSON format (see the end of this blog post for a sample API command).

Default custom WAF rules

Unfriendly Drupal 7 URLs

I often see bots trying to hit URLs like /?q=node/add and /?q=user/register. This is the default unfriendly URL to hit on Drupal 7 to see if user registration or someone has messed up the permissions table (and you can create content as an anonymous user). Needless to say, these requests are rubbish and add no value to your site, let's block them.

description: 'Drupal 7 Unfriendly URLs (bots)'
action: block
filter:
  expression: '(http.request.uri.query matches "q=user/register") or (http.request.uri.query matches "q=node/add")'

Autodiscover

If your organisation has bought Microsoft Exchange, then likely your site will receive loads of requests (GET and POST) to which is likely to just tie up resources on your application server serving these 404s. I am yet to meet anyone that actually serves back real responses from a Drupal site for Autodiscover URLs. Blocking is a win here.

description: Autodiscover
action: block
filter:
  expression: '(http.request.uri.path matches "/autodiscover\.xml$") or (http.request.uri.path matches "/autodiscover\.src/")'

Wordpress

Seeing as Wordpress has a huge market share (34% of all websites) a lot of Drupal sites get caught up in the mindless (and endless) crawling. These rules will effectively remove all of this traffic from your site.

description: 'Wordpress PHP scripts'
action: block
filter:
  expression: '(http.request.uri.path matches "/wp-.*\.php$")'

description: 'Wordpress common folders (excluding content)'
action: block
filter:
  expression: '(http.request.uri.path matches "/wp-(admin|includes|json)/")'

I separate wp-content into it's own rule as you may want to disable this rule if you are migrating from a old Wordpress site (and want to put in place redirects for instance).

description: 'Wordpress content folder'
action: block
filter:
  expression: '(http.request.uri.path matches "/wp-content/")'

SQLi

I have seen several instanced in the past where obvious SQLi was being attempted and the default WAF rules by Cloudflare were not intercepting them. This custom WAF rule is an attempt to fill in this gap.

description: 'SQLi in URL'
action: block
filter:
  expression: '(http.request.uri.path contains "select unhex") or (http.request.uri.path contains "select name_const") or (http.request.uri.path contains "unhex(hex(version()))") or (http.request.uri.path contains "union select") or (http.request.uri.path contains "select concat")'

Drupal 8 install script

Drupal 8's default install script will expose your major, minor and patch version of Drupal you are running. This is bad for a lot of reasons.

It is better to just remove these requests from your Drupal site altogether. Note, this is not a replacement for upgrading Drupal, it is just to make fingerprinting a little harder.

description: 'Install script'
action: block
filter:
  expression: '(http.request.uri.path eq "/core/install.php")'

Microsoft Office and Skype for Business

Microsoft sure is good at making lots of products that attempt to DoS its own customers websites. These requests are always POST requests, often to your homepage, and you require partial string matching to match the user agent, as it changes with the version of Office/Skype you are running.

In large organisation, I have seen the number of requests here number in the hundreds of thousands per day.

description: 'Microsoft Office/Skype for Business POST requests'
action: block
filter:
  expression: '(http.request.method eq "POST") and (http.user_agent matches "Microsoft Office" or http.user_agent matches "Skype for Business")'

Microsoft ActiveSync

Yet another Microsoft product that you don't why it is trying to hit another magic endpoint that doesn't exist.

description: 'Microsoft Active Sync'
action: block
filter:
  expression: '(http.request.uri.path eq "/Microsoft-Server-ActiveSync")'

Using the Cloudflare API to import custom WAF rules

It can be a pain to have to manually point and click a few hundred times per zone to import the above rules. Instead you would be better off to use the API. Here is a sample cURL command you can use do import all of the above rules in one easy go.

You will need to replace the redacted sections with your details.

curl 'https://api.cloudflare.com/client/v4/zones/XXXXXXXXXXXXXX/firewall/rules' \
  -H 'X-Auth-Email: XXXXXXXXXXXXXX' \
  -H 'X-Auth-Key: XXXXXXXXXXXXXX'
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json'
  -H 'Accept-Encoding: gzip'
  -X POST \
  -d '[{"ref":"","description":"Autodiscover","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/autodiscover\\.xml$\") or (http.request.uri.path matches \"\/autodiscover\\.src\/\")"}},{"ref":"","description":"Drupal 7 Unfriendly URLs (bots)","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.query matches \"q=user\/register\") or (http.request.uri.query matches \"q=node\/add\")"}},{"ref":"","description":"Install script","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path eq \"\/core\/install.php\")"}},{"ref":"","description":"Microsoft Active Sync","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path eq \"\/Microsoft-Server-ActiveSync\")"}},{"ref":"","description":"Microsoft Office\/Skype for Business POST requests","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.method eq \"POST\") and (http.user_agent matches \"Microsoft Office\" or http.user_agent matches \"Skype for Business\")"}},{"ref":"","description":"SQLi in URL","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path contains \"select unhex\") or (http.request.uri.path contains \"select name_const\") or (http.request.uri.path contains \"unhex(hex(version()))\") or (http.request.uri.path contains \"union select\") or (http.request.uri.path contains \"select concat\")"}},{"ref":"","description":"Wordpress common folders (excluding content)","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/wp-(admin|includes|json)\/\")"}},{"ref":"","description":"Wordpress content folder","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/wp-content\/\")"}},{"ref":"","description":"Wordpress PHP scripts","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/wp-.*\\.php$\")"}}]'

How do you know the above rules are working

Visit the firewall overview tab in Cloudflare's UI to see how many requests are being intercepted by the above rules.

Final thoughts

The above custom WAF rules are likely not the only custom WAF rules you will need for any given Drupal site, but it should at least be a good start. Let me know in the comments if you have any custom WAF rules that you always deploy. I would be keen to update this blog post with additional rules from the community.

This is likely the first post in a series of blog posts on customising Cloudflare to suit your Drupal site. If you want to stay up to date - subscribe to the RSS feed, sign up for email updates, or follow us on Twitter.

I am working with a customer now that is looking to go through a JSON:API upgrade, from version 1.x on Drupal 8.6.x to 2.x and then ultimately to Drupal 8.7.x (where it is bundled into core).

As this upgrade will involve many moving parts, and it is critical to not break any existing integrations (e.g. mobile applications etc), having basic end-to-end tests over the API endpoints is essential.

In the past I have written a lot about CasperJS, and since then a number of more modern frameworks have emerged for end-to-end testing. For the last year or so, I have been involved with Cypress.

I won't go too much in depth about Cypress in this blog post (I will likely post more in the coming months), instead I want to focus specifically on JSON:API testing using Cypress.

In this basic test, I just wanted to hit some known valid endpoints, and ensure the response was roughly OK.

Rather than have to rinse and repeat a lot of boiler plate code for every API end point, I wrote a custom Cypress command, to which abstracts all of this away in a convenient function.

Below is what the spec file looks like (the test definition), it is very clean, and is mostly just the JSON:API paths.

describe('JSON:API tests.', () => {

    it('Agents JSON:API tests.', () => {
        cy.expectValidJsonWithMinimumLength('/jsonapi/node/agent?_format=json&include=field_agent_containers,field_agent_containers.field_cont_storage_conditions&page[limit]=18', 6);
        cy.expectValidJsonWithMinimumLength('/jsonapi/node/agent?_format=json&include=field_agent_containers,field_agent_containers.field_cont_storage_conditions&page[limit]=18&page[offset]=72', 0);
    });
    
    it('Episodes JSON:API tests.', () => {
        cy.expectValidJsonWithMinimumLength('/jsonapi/node/episode?fields[file--file]=uri,url&filter[field_episode_podcast.nid][value]=4976&include=field_episode_podcast,field_episode_audio,field_episode_audio.field_media_audio_file,field_episode_audio.thumbnail,field_image,field_image.image', 6);
    });

});

And as for the custom function implementation, it is fairly straight forward. Basic tests are done like:

• Ensure the response is an HTTP 200
• Ensure the content-type is valid for JSON:API
• Ensure there is a response body and it is valid JSON
• Enforce a minimum number of entities you expect to be returned
• Check for certain properties in those returned entities.

Cypress.Commands.add('expectValidJsonWithMinimumLength', (url, length) => {
    return cy.request({
        method: 'GET',
        url: url,
        followRedirect: false,
        headers: {
            'accept': 'application/json'
        }
    })
    .then((response) => {
        // Parse JSON the body.
        let body = JSON.parse(response.body);
        expect(response.status).to.eq(200);
        expect(response.headers['content-type']).to.eq('application/vnd.api+json');
        cy.log(body);
        expect(response.body).to.not.be.null;
        expect(body.data).to.have.length.of.at.least(length);

        // Ensure certain properties are present.
        body.data.forEach(function (item) {
            expect(item).to.have.all.keys('type', 'id', 'attributes', 'relationships', 'links');
            ['changed', 'created', 'default_langcode', 'langcode', 'moderation_state', 'nid', 'path', 'promote', 'revision_log', 'revision_timestamp', 'status', 'sticky', 'title', 'uuid', 'vid'].forEach((key) => {
                expect(item['attributes']).to.have.property(key);
            });
        });
    });

});

Some of the neat things in this function is that it does log the parsed JSON response with cy.log(body); this allows you to inspect the response in Chrome. This allows you to extend the test function rather easily to meet you own needs (as you can see the full entity properties and fields.

Using Cypress is like having an extra pair of eyes on the Drupal upgrade. Over time Cypress will end up saving us a lot of developer time (and therefore money). The tests will be in place forever, and so regressions can be spotted much sooner (ideally in local development) and therefore fixed much faster.

Comments

If you do JSON:API testing with Cypress I would be keen to know if you have any tips and tricks.

I have had the chance to be involved with 2 fresh builds with Drupal 8 now, I thought I would describe some of the neat things I have found during this time and some of my lessons learned. My hope is that blog post will help you in your journey with Drupal 8.

1. Drupal Console is awesome

Every time you need to generate a custom module, or a new block in a custom module, you can quickly and easily use Drupal Console to produce the code scaffolding for you. This quite easily makes the job of a developer a lot less stressful, and allows you to focus on actually writing code that delivers functionality.

I plucked these example commands that I use frequently from my bash history:

drupal site:mode dev
drupal generate:module
drupal generate:plugin:block
drupal generate:routesubscriber
drupal generate:form:config

Documentation is online but for the most part, the commands are self documenting, if you use the --help option, then you get a great summary on the command, and the other options you can pass in.

The other nice thing is that this is a Symfony Console application, so it should feel very familiar to you if you used another tool written in the same framework.

2. Custom block types are amazing

In Drupal 7 land there was bean which was an attempt to stop making ‘meta’ nodes to fill in content editable parts of complex landing pages. Now, fast forward to Drupal 8, and custom block types are now in Drupal Core.

This basically means as a site builder you now have another really powerful tool at your disposal in order to model content effectively in Drupal 8.

Each custom block type can have it’s own fields, it’s own display settings, and form displays.

Here are the final custom block types on a recent Drupal 8 build:

One downside is that there is no access control per custom block type (just a global permission “administer blocks”), no doubt contrib will step in to fill this hole in the future (does anyone know a module that can help here?). In the mean time there is drupal.org issue on the subject.

I also found it weird that the custom blocks administration section was not directly under the ‘structure’ section of the site, there is another drupal.org issue about normalising this as well. Setting up some default shortcuts really helped me save some time.

3. View modes on all the things

To create custom view modes in Drupal 7 required either a custom module or Dave Reid’s entity_view_mode contrib module. Now this is baked into Drupal 8 core.

View modes on your custom block types takes things to yet another level still as well. This is one more feather in the Drupal site builder’s cap.

4. Twig is the best

In Drupal 7 I always found it weird that you could not unleash a front end developer upon your site and expect to have a pleasant result. In order to be successful the themer would need to know PHP, preprocess hooks, template naming standards, the mystical specific order in which the templates apply and so on. This often meant that a backend and front end developer would need to work together in order to create a good outcome.

With the introduction of Twig, I now feel that theming is back in the hands of the front end developer, and knowledge of PHP is no longer needed in order to override just about any markup that Drupal 8 produces.

Pro tip - use the Drupal Console command drupal site:mode dev to enable Twig development options, and disable Drupal caching. Another positive side effect is that Twig will then render the entire list of templates that you could be using, and which one you actually are using (and where that template is located).

Pro tip: - If you want to use a template per custom block type (to which I did), then you can use this PHP snippet in your theme’s .theme file (taken from drupal.org):

<?php
/**
 * Implements hook_theme_suggestions_HOOK_alter() for form templates.
 *
 * @param array $suggestions
 * @param array $variables
 */
function THEMENAME_theme_suggestions_block_alter(array &$suggestions, array $variables) {
  if (isset($variables['elements']['content']['#block_content'])) {
    array_splice($suggestions, 1, 0, 'block__bundle__' . $variables['elements']['content']['#block_content']->bundle());
  }
}

5. Panelizer + panels IPE is a formidable site building tool

When looking for a layout manager to help build the more complex landing pages, I came across panelizer + panels IPE. Using panelizer you are able to:

• create per node layout variants
• apply a single layout to all nodes of a particular bundle (e.g. all your news articles have the same layout)

The other neat thing is that the layouts themselves are now standardised between all the various layout managers using a contrib module called layout_plugin. Also they are just YAML and Twig. Simple. There is even an effort to get this merged into Drupal 8.2 which I think would be a great idea.

Downside - all JS is still rendered on the page even though the user (e.g. anonymous users) have no access to panelizer. There is a patch on drupal.org to help fix this.

Since starting this build there has also been a stable release of display suite come out for Drupal 8 as well giving you even more options.

6. You can build a rather complex site with very little contributed modules

For this most recent site I build I got away with using only 10 contributed modules (one of which - devel was purely for debugging purposes).

• ctools
• google_analytics
• metatag
• panels
• token
• contact_block
• devel
• layout_plugin
• panelizer
• pathauto

This means you are inherently building a more stable and supportable site, as most of the functionality now comes out of Drupal core.

7. The contact module now is supercharged

In Drupal 7, the contact module was one of those modules to which I never turned on, as it was rather inflexible. You could not change the fields in a UI, nor add email recipients, or have more than 1 form. Now in Drupal 8 you can have as many “contact” forms as you want, each one is fieldable, and can send emails to as many people as needed.

You can also enhance the core module with:

• contact_block - allows you to place the contact form in a block
• contact_storage - allows you to store the submissions in the database, rather than firing an email and forgetting about it

There is still a place for webform, namely:

• large complex form with lots of fields
• multi-step forms
• forms you want to ‘save draft’

You can read more about this in the OS training blog post on the contact module.

Downside - I wanted to have a plain page use the path /contact but the contact module registers this path, so pathauto gave my contact page a path of /contact-0. Luckily creating a route subscriber with Drupal Console was painless, so altering the contact module route was very simple to do. I can paste the code here if needed, but most of it is the code that Drupal Console generates for you.

8. PHPunit is bundled into core

Now that Drupal 8 is largely Object Oriented (OO), you are able to test classes using PHPunit. I have wrote about phpunit in the past if you want to know more.

9. Views is in core

This was the main reason why adoption of Drupal 7 was so slow after it’s initial 7.0 release, as everyone needed views to be stable before jumping ship. Now with views bundled into core, views plugins are also being ported at a great rate of knots too.

10. CKEditor is in core

I often found that this was one library that never (or hardly ever) got updated on sites that had been around for a while. More worryingly, CKEditor (the library) would from time to time fix security related issues. Now that this comes with Drupal 8 core, it is just one less thing to worry about.

Also I would love to shout out to Wim Leers (and other contributors) for revamping the image dialog with alignment and caption options. I cannot tell you how much pain and suffering this caused me in Drupal 7.

Comments

If you have built a site recently in Drupal 8 and have found anything interesting or exciting, please let me know in the comments. Also keen to see what sites people have built, so post a link to it if it is public.