Over half of all web traffic in 2024 was automated. That is the headline number from the Imperva 2025 Bad Bot Report, and it is the first time bots have outnumbered humans in more than a decade. Drupal sites sit squarely in that traffic mix, and the old defensive playbook — block an IP, ban a user agent, drop a robots.txt entry, lean on Fail2ban — does not hold up anymore.

This is the companion post to my DrupalSouth Wellington 2026 talk, Bots, scrapers, and proxies: defending Drupal sites in an automated internet. The talk walked through the defences I actually use at amazee.io and recommend on client sites. The post covers the same ground, with a bit more room to show config and link out to the projects.

What actually changed

The technical context underneath bot defence has shifted in three ways that matter:

• Residential proxy networks. Scrapers no longer come from a handful of cloud subnets you can block. They route through real consumer IP addresses, often unwittingly donated by free-VPN users or piggy-backed off shady SDKs in mobile apps.
• Headless browsers everywhere. Playwright and Puppeteer have made it trivial to render JavaScript-heavy pages at scale. A page that needed a real human five years ago can be scraped today by anyone with a laptop.
• AI-driven scraping. Volume is up sharply because every new LLM needs training data, and there is now a steady drip of new crawlers showing up. Meta's externalagent is one recent example. There will be more.

Mimicry is now the baseline, not the edge case. A modern scraper will rotate IPs, randomise user agents, replay realistic TLS fingerprints, and pace itself slowly enough to look like a real user. You cannot rely on signal that lives in one HTTP header.

The scale of it

If this still sounds like a niche problem, the numbers say otherwise.

• 51% - share of web traffic that was automated in 2024, per the Imperva 2025 Bad Bot Report.
• +96% - year-on-year growth in some popular bot services across Pantheon's hosting fleet in their July 2025 data.
• 1B+ - unique monthly visitors Pantheon sees across its platform, which is the size of dataset those numbers are coming from.

On the amazee.io platform globally, 13% of incoming requests can be flagged as non-human based on the user agent alone. That is the lazy bots. The actual share of automated traffic is higher once you account for the ones that try to blend in. In absolute terms it adds up to hundreds of millions of requests every month.

The goal is not to block all bots

Before going through the defences, one thing I am careful to say up front, both on stage and here: the goal is not to block all bots. That is unwinnable, and the closer you get to it the more real users you break.

Search crawlers, RSS readers, uptime monitors, link-preview generators in Slack and iMessage, accessibility tooling - all bots, all wanted. The goal is to reduce abuse where it hurts most, on the endpoints that cost you real money or real performance, while leaving everything else alone.

Drupal-native defences

The defences closest to your application are the smartest. They can see the path, the user, the form, the cache state. They are also the most expensive per blocked request, because every block at this layer has already cost you a full PHP bootstrap.

Perimeter

The Perimeter module drops requests matching known-bad patterns: /wp-admin, /.env, xmlrpc.php, all the WordPress scanner noise that hits every Drupal site daily. It is the cheapest win on the list. It will not stop a serious scraper, but it will keep your logs clean and your error rate honest.

CrowdSec and AbuseIPDB

CrowdSec is a local agent plus a community blocklist. Every site running CrowdSec contributes detected attacks back to a shared signal, and pulls down the latest list of bad actors. It is the closest thing the open-source world has to a distributed reputation system.

AbuseIPDB is a reputation lookup service. You query an IP, you get a confidence score. It is most useful on the forms and login flows where you can afford the latency of an external API call. Both are available as Drupal modules.

Facet Bot Blocker

If you run Search API with facets, this is the single cheapest huge win available to you. Faceted search URLs are catnip for scrapers: every combination of filters is a new URL, every URL is uncached, every uncached request hits the database. A bot that crawls a faceted listing can take a site down without trying.

The Facet Bot Blocker module acts as a rate limit on requests that include at least one facet in the URL. Configure it to use Redis or memcache for the counter so you are not making the problem worse by hitting the database to record the request. On one of our hosting customers, this one module cut Search API load by more than half.

Form-side defences

Logins, registrations, password resets and contact forms all need their own treatment, separate from page-level defence:

• Honeypot - invisible field plus a time-based check. Cheap, fast, surprisingly effective against the dumb half of form spam.
• Antibot - requires JavaScript to submit, blocks the bots that do not run JS.
• CAPTCHA, reCAPTCHA, or Cloudflare Turnstile - full challenge. Use the lightest option that works, and ideally only after Honeypot and Antibot have already rejected the easy cases.
• Hidden CAPTCHA - bridges the gap when you want a CAPTCHA-style check without the accessibility cost of a visible challenge.

The trade-off the project pages don't mention

Every block at the Drupal layer has already cost you a PHP bootstrap. That is fine when the absolute volume is small. It is not fine when you are eating hundreds of millions of bot requests and bootstrapping PHP for each one. This is why you cannot stop at the application layer.

Web server and infrastructure

One layer out, the web server can drop requests before PHP ever runs. The trade-off flips: you save the bootstrap cost, but you lose access to application context.

Rate limiting and geo blocking

nginx ships with limit_req_zone, Apache has mod_ratelimit. Both are blunt but effective on volume. A starting point for nginx looks roughly like this:

limit_req_zone $binary_remote_addr zone=search:10m rate=10r/m;

location /search {
    limit_req zone=search burst=5 nodelay;
    proxy_pass http://drupal;
}

Ten search requests per minute, per IP, with a burst of five. Tune to taste. The $binary_remote_addr key is cheap on memory; a 10MB zone holds around 160,000 IPs.

Geo blocking is the other infrastructure-level lever. It is pragmatic and occasionally controversial. If your audience is the New Zealand public sector, blocking inbound from regions you do not serve is a defensible call. If your audience is global, it is not. Know your traffic before reaching for it.

ModSecurity and the OWASP CRS

ModSecurity with the OWASP Core Rule Set is a proper WAF you can self-host. Once tuned, it is real protection. The tuning is the catch — out of the box it will flag Drupal admin actions, file uploads, anything that looks like SQL in a body or a query string. Expect to spend real time pruning rules and adding exceptions for legitimate site behaviour before you stop generating false positives.

Cache discipline

A request that hits the cache costs you nothing. Whatever else you do, get your cache headers right. Vary on the bits that need to vary, cache aggressively on the bits that do not, and lean on the page cache or the reverse proxy in front of Drupal. The cheapest bot is the one that asks for a page you have already served.

(shamless plug) you can also use my site Caching Score to review your current caching setup, to see if there is anything better you can be doing.

Caching Score

Assess how strong the caching capabilities of any given site is.

Caching Scoresean.hamlinamazee.io

What this layer is bad at

Rate limits, ModSecurity rules and geo blocks are great at volume and bad at quality. They cannot tell a scraper trickling one request per minute apart from a real user. For that you need either the edge or the application.

Edge and paid bot management

The edge is where the big vendors live, and it is where you push the cheapest blocks. A scraper rejected by Cloudflare at the network edge never gets to your origin at all.

Cloudflare

The free tier already includes Bot Fight Mode, basic challenges, and Turnstile. For most small-to-medium Drupal sites, this is a good baseline at zero extra cost. The paid Bot Management product adds custom rule logic, JA3 and JA4 TLS fingerprinting, and machine-learning-based bot scoring you can wire into firewall rules. The jump from free to paid is significant in price; the jump in capability is also significant.

Fastly, Akamai, and the rest

Fastly offers the Next-Gen WAF (originally Signal Sciences) with a Bot Management add-on. Akamai sits at the enterprise tier with the most sophisticated fingerprinting available, and a price tag to match. Beyond those, there is AWS WAF with Bot Control, DataDome, HUMAN, and Imperva — all credible, all paid, all priced for sites where bot abuse is costing real money.

The trade-offs nobody puts on the sales deck

Bot Management at the edge solves real problems. It also comes with real costs that the vendor demos skip past:

• Cost. Bot Management is almost always an add-on to the core WAF subscription, and the pricing escalates fast with traffic.
• Vendor lock-in. Your rules, your dashboards, your observability all live in the vendor's UI. Migrating off is painful.
• Accessibility and SEO. Aggressive challenges break real users, and search bots that fail a challenge will hurt your rankings. Test both before turning anything up.
• Rules live outside your application codebase. They drift, they are not versioned alongside the code that depends on them, and a rule change can break a feature without any commit to point to.
• False positives are invisible to you. By default, blocked requests do not reach your logs. You will not know which real users were turned away unless you specifically ask for that signal.

Anubis

The newest piece in the picture, and the one that has me genuinely interested.

What Anubis is

Anubis is an open-source reverse proxy (MIT licensed) that sits in front of your site and issues a proof-of-work challenge to clients before letting them through. It was built specifically for the AI scraper era — for the case where the scraper is mimicking a real browser well enough that classifying it on signal alone has stopped working.

Why proof-of-work, not CAPTCHA

The interesting move with Anubis is who pays the cost. A real user pays a few hundred milliseconds of CPU once when they first arrive, and never sees it again for the lifetime of the cookie. A scraper hitting you a million times pays the cost a million times.

That asymmetry is the whole point. CAPTCHAs put the cost on humans (the people who lose patience trying to identify traffic lights). Anubis puts it on whoever is doing the hammering. That is closer to the right shape of the trade.

Where to put it

You do not want Anubis in front of your whole site. You want it in front of the endpoints that are expensive and uncacheable. From the talk, my shortlist:

• Search endpoints
• Facet and filter URLs
• Pagination tails - ?page=2348 is not a real user
• Login, register, password reset
• Spicy forms (contact, anything that triggers an email)
• Authenticated user flows
• Anything expensive and uncacheable

Static pages stay fast. The cache stays warm. The PoW cost only applies on the routes where it earns its keep.

But what about Googlebot?

This is the first question every site owner asks, and the answer is good. Anubis ships with allowlists for known good crawlers, matching IP ranges against the published lists from Google, Bing, and the rest. The allowlist is maintained upstream, which means you need to keep Anubis deployed on a reasonable cadence to pull in the latest changes. New legitimate crawlers do show up.

Demo site

You can see Anubis in action with a demo Drupal 11 site I put together, the login form has Anubis in front of it, the homepage does not.

Log in | Drush Site-Install

Drush Site-Install

Putting the layers together

None of these defences is a silver bullet on its own. Each layer is cheap at one thing and bad at another, and the trick is matching the layer to the threat.

Block the cheap traffic at the edge. Block the lazy bots with rate limits and ModSecurity at the web server. Put Anubis in front of the endpoints that are expensive and uncacheable. Let Drupal-native modules handle the application-aware decisions where you actually need to see the user, the form, or the facet state.

Five things to take away

1. No single layer is enough. Stack them. The edge handles raw volume, the web server handles patterns, and the application is the only thing that can see real user and form context.
2. Match the protection to the threat. A login form needs different defence to a faceted search results page.
3. Measure before you defend. Look at your actual traffic. Find your most-hit uncacheable endpoints. Defend those first.
4. Watch accessibility and SEO. Every challenge you add is a tax on a real user or a real crawler. The cost of false positives is invisible unless you go looking.
5. Plan for adversarial improvement. Whatever you deploy today, the scrapers get a turn next. Pick defences you can iterate on.

One last thing

You do not need to win the bot war. You just need to make your site a worse target than the next one.

The slides from the talk are online too. The recording is here:

The JSON Formatter Chrome extension has over 2 million users. I was one of them for years. A few weeks ago it went closed-source, and shortly after, started injecting adware into web pages.

The extension was modifying the DOM on checkout pages, hijacking click and submit events to overlay donation popups. It was also running geolocation lookups against MaxMind's GeoIP2 API with a hardcoded key, profiling users by country so that people in high-income regions got more aggressive prompts. For something that was supposed to format JSON.

This is a supply chain problem, and it keeps happening.

Chrome extensions are a trust problem

The pattern is well documented at this point. A popular extension with millions of installs gets sold or transferred to a new owner. The new owner pushes an update that adds tracking and ad injection. Sometimes it goes further into outright credential theft. Users get the update automatically and have no idea anything changed.

The JSON Formatter incident is almost tame by comparison. But it hit home because I had it installed. I trusted it. And I had no real way to know when the ownership changed or the code was modified.

What I did about it

Around the same time, Pieter Levels posted about replacing all his Chrome extensions with a single one he built himself using Claude Code. His reasoning was straightforward: if the code is yours, nobody can sell it to an ad company.

I liked the idea, so I did the same thing.

I sat down with Claude Code and built a Chrome extension from scratch. The whole thing took under an hour. The result is a single Manifest V3 extension that currently replaces JSON Formatter for me, and it is extensible so new features can be added easily.

The repo is public: github.com/seanhamlin/chrome-tweaks (not that I expect you to run this, but you can see what Claude can make with a little effort).

Why build your own instead of finding another extension

You could swap JSON Formatter for another extension. There are plenty of alternatives. But that misses the point. Every third-party extension you install is a bet that the current maintainer will stay honest, won't sell it, and won't get phished. You're making that bet for every extension, and Chrome will silently auto-update them all.

Building your own means:

• You control the code. No surprises in the next update.
• You only include the permissions you actually need.
• You can combine several small utilities into one extension instead of running five separate ones with five separate attack surfaces.

Getting started with your own extension

If you have Claude (or any decent AI coding tool), this is a good afternoon project. Chrome extensions are well documented by Google, and the structure is small enough that an LLM can generate the whole thing from a prompt.

Can u make a chrome extension for chrome called "Tab Tweaks" - this extension will grow in functionality over time, so ensure it is extensible. The current functionality should include:

* Able to inject a request header to every request. The default should be the Fastly debug header `Fastly-Debug:1`. The user should be allowed to edit this, and change it, or add another header.
* A button to scan the site in www.cachingscore.com. The current URL should be URL encoded, and a new tab opened to point to `https://www.cachingscore.com/scan?q=https%3A%2F%2Fwww.example.com%2F&followRedirects=on` (replace the example URL).

Claude generated the manifest, the content script, and the CSS. I reviewed it, tested it, tweaked it, loaded it as an unpacked extension in chrome://extensions, and it worked.

Audit your extensions

Open chrome://extensions right now and look at what you have installed. For each one, ask yourself:

• Do I actually use this?
• Who maintains it? Is it still the original author?
• What permissions does it have?
• Could I replace this with a simpler version?

You will probably find at least two or three extensions you forgot about. Remove them.

For the ones you keep, consider whether the functionality is simple enough to replicate yourself. A lot of developer-focused extensions (JSON formatters, cookie viewers) are doing things that a content script or devtools panel can do in under 100 lines.

My extension is at github.com/seanhamlin/chrome-tweaks if you want to see how it is structured, but the real suggestion is to build your own. You will trust it more than anything in the Chrome Web Store.

Have you ever wondered why your docker images are so large, and what can be done to drill down into the layers one by one. Well, you can.

Introducing Dive

dive is a cool command line tool for inspecting the layers one by one of your docker images.

A tool for exploring a Docker image, layer contents, and discovering ways to shrink the size of your Docker/OCI image.

Running dive on an image

This is simple, and can be run with docker (meta):

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker.io/wagoodman/dive:latest <registry>/<repo>@sha256:<hash>

Example report

Here is a real life docker image, that came in at 1.3GB, which is on the high side of things. Scrolling through the layers I find this one, to which contributes 560MB all by itself.

The cause jumps out and smacks you in the face, the layer is copying the entire git repo into the image (see the right hand panel, green indicates a file was added in this layer). This folder never really needed in a docker image, and should not be copied if possible. Finding the offending line in the docker file:

# Copy the rest of the application code
COPY . /app

So the fix here is to use .dockerignore and add in .git as a line in that file. Simple.

The use of .dockerignore is covered in more detail on my other blog post that deals with making best practice docker images.

In years past, when I wanted to analyse a bunch of JSON logs, I would use jq. jq is an amazing tool, but suffers from some pretty massive drawbacks for large log analysis:

• jq is slow. It is single threaded. Parsing a several GB file will take minutes
• If you want to run multiple different reports, then you need jq to parse it all again. So the first point becomes even more painful

Enter OpenObserve (O2). O2 is an awesome, open source (GPLv3), all in one application that is able to ingest vast quantities of logs (and other things) and be able to query them extremely quickly, and even draw visualisations and dashboards.

Think of O2 like an extremely lightweight ElasticSearch.

In my particular case, I often need to visualise JSON logs that come from Fastly. These logs are vast in nature, and being able to analyse millions of logs quickly is key.

The best thing (IMO) is that you can run it all locally, with a single container, and your computer will not grind to a halt in the process.

Here is a docker-compose file to get you up and running:

services:
  openobserve:
    image: public.ecr.aws/zinclabs/openobserve:latest
    restart: unless-stopped
    environment:
      ZO_ROOT_USER_EMAIL: "admin@example.com"
      ZO_ROOT_USER_PASSWORD: "password"
      ZO_INGEST_ALLOWED_UPTO: "48"
      ZO_PAYLOAD_LIMIT: "2000000000"
      ZO_JSON_LIMIT: "2000000000"
    ports:
      - "5080:5080"
    volumes:
      - data:/data
volumes:
  data:

I had to tweak a few settings (all through environment variables) to allow me to ingest hundreds of MBs in a single payload, and also allow me to ingest data up to 2 days old.

In order to get my JSON logs into a format that O2 would like, I needed to massage them a little.

O2 will automatically parse timestamps that have the name @timestamp, in my case, the field was just called timestamp. So a little sed to the rescue:

sed -i -e 's/"timestamp/"@timestamp/g' $FILE

I then needed to format the JSON lines into a giant JSON array, luckily someone on stackoverflow sorted this:

sed -i -e '1s/^/[/; $!s/$/,/; $s/$/]/' $FILE

Finally, you need to send the formatted JSON array into O2

curl -u admin@example.com:password -k http://127.0.0.1:5080/api/default/default/_json --data-binary "@$FILE"

If it all goes well, then you should see something like:

{"code":200,"status":[{"name":"default","successful":195001,"failed":0}]}

You can now view the UI at http://localhost:5080/ and start to use the power of O2.

Happy graphing and log visualising.

This is the complementary blog post for my DrupalSouth Sydney 2024 session, and also v2.0 follow up of sorts from the original 2022 version. The full presentation was much longer than this blog post, this blog post is just going to highlight the core statistics and findings.

Have you ever wondered how popular Drupal is in your local state and at the Australian Federal Government level? This blog post will help to answer that question, using open source tooling. The hope is that you gain some insight to the relative popularity of Drupal and appreciate more the impact you and Drupal have in Australia.

As this blog post is a follow up, you can also now start to see trends (data is around 13 months newer than the last time I did this).

Just show me the graphs

Disclaimer:

• This is based on Oct 20, 2023 data
• The scoring is based off PageRank data, so the percentages are not raw counts of websites, but an approximation of how important the respective sites are compared to others (assumes a logarithmic base of 5).
• Wappalyzer detection is not perfect (see the end of this blog post for upstreamed PRs), and there is still a fairly large portion of sites where the CMS cannot be identified
• MoGs make this tricky (PageRank relies on incoming links, which break due to MoGs)
• Only source *.gov.au domains considered (some Government sites use other TLDs)
• Unlikely newly created websites are in the top 10 million just yet (due to how the algorithm of PageRank works)

All sites (*.gov.au)

Federal sites (not state based domains)

Programmes like GovCMS are having an impact here. Also interesting that if you are not using Drupal, the chances are you have written something entirely custom.

Victoria *.vic.gov.au

The Single Digital Presence (SDP) programme makes a mark in Victoria.

New South Wales *.nsw.gov.au

Large Drupal sites like https://www.nsw.gov.au/ and https://www.service.nsw.gov.au/ help to make Drupal dominant in NSW.

South Australia *.sa.gov.au

Squiz Matrix increasing in market share ↑5.4% over 2022. There is a clear state led mandate here.

Western Australia *.wa.gov.au

A lot of sites this time around are now identified (decrease of ↓30.8% of unknown sites). Drupal also increased market share by ↑9.9%.

Tasmania *.tas.gov.au

The lowest usage of Drupal for any Australian state or territory and the highest percentage of Wordpress.

Queensland *.qld.gov.au

Australian Capital Territory *.act.gov.au

The highest percentage of Squiz compared to any other Australia state or territory.

Northern Territory *.nt.gov.au

Open Source Software (OSS) CMS vs Proprietary CMS

For the CMS' that can be identified, splitting them into 2 categories, OSS and Proprietary. OSS is determined on whether the source code is freely available, and there is a licence that allows me to run it without paying someone.

Drupal sites by major version

For sites reporting as Drupal, Drupal 10 is the most popular. Still 5.4% of Drupal sites running Drupal 7 to which will be End-of-Life (EOL) in early 2025.

Score by state and territory

This is weighted by total score, broken down by federal/state/territory.

Observations and other unusual findings

Drupal usage

“Drupal powers ~29.9% of all digital experiences that you use in the Australian government. This is↑2.7% compared to 2022”

Drupal Growth

“Relative to the growth of Australian government sites, Drupal adoption is growing faster”

Top contender

“Squiz Matrix is the top contender with 15.6%, and has a clear state lead mandate in 5 states/territories. This is↑3.5% compared to 2022”

Drupal 7 usage

“Drupal 7 usage dropped from 15.8% in 2022 to 5.4% in 2024. This is↓65.8% compared to 2022. Most popular Drupal 7 site is https://www.sl.nsw.gov.au/”

Also after my presentation I got to meet the team behind the State Library of NSW, and they advised that they are due to upgrade to Drupal 10 anytime soon.

TLS coverage is still not 100%

83 sites with HTTP only (a drop of ↓46 since 2022)

If in doubt, add a number

14 sites with ww[0-9] in the domain name (a drop of ↓5 since 2022)

I think this is often used like a form of poor mans version control, often archiving the previous version of the site. For some reason it is archived publicly.

I want the raw data!

If you want to make your own visualisations of the data, or even just do random queries “how popular is Spark CMS in Western Australia”, you can download a CSV from https://bit.ly/dsau2024csv. Slides are https://bit.ly/dsau2024.

Upstreamed enhancements

I found a lot of software running in certain states, so I upstreamed some changes to better detect these software packages:

• Better SilverStripe detection #120
• Datascape detection #122
• Spark CMS detection #123
• Jadu detection #126
• Engagement HQ detection #124
• Social Pinpoint detection #125
• Citizen Space detection #121

Comments

I am keen to hear feedback on this data, and what can be done to improve the scoring. Also, if you can help fill in some of the 'unknown' data, let me know, I am happy to craft another PR into WebAppAnalyzer.

GitHub recommends that git repositories be < 1 GiB is size. The most important reasons why I think this matters:

• It takes longer to clone and fetch larger git repositories. This may impact the time for Continuous Integration (CI) jobs to run, also it may impact the speed of deployments to your hosting infrastructure. It will definitely slow down developers looking to get setup for the first time.
• It adds to space needed to house the git repository. Both GitHub and GitLab will reach out to you if you are storing too much data. Not to mention the lost GB on your laptop.

Find the current size of your git repo

There is a command built in to git (since 2013) that does this very easily:

$ git count-objects -vH

count: 0
size: 0 bytes
in-pack: 4024
packs: 1
size-pack: 4.27 GiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

Find the large files in your git repository

Now that you know your git repository is excessively large, the next step is to work out why.

Using this amazing script on stackoverflow:

git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  sed -n 's/^blob //p' |
  sort --numeric-sort --key=2 |
  cut -c 1-12,41- |
  $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

N.B. on Mac OSX, you will need brew install coreutils for the magic number formatting to work.

This script will list every single file in your git repository, with the largest files at the bottom, in the example git repository I was looking into, these were the largest files:

26aa9bdfd920  564KiB themes/site/hypejs/hypejs.css
1d8f89f0d18a  564KiB themes/site/hypejs/hypejs.css
cf4d7ceb1d4f  684KiB themes/site/css/fonts/fontawesome/fa-brands-400 2.svg
401b7f7b3219  823KiB themes/site/css/fonts/fontawesome/fa-solid-900 2.svg
989b349bb493  4.3GiB files.zip

You can spot a single large zip file files.zip in the repository, coming in at a whooping 4.3GiB. In this particular case, this was due to a developer accidentally committing the file in one commit, and then thinking they can remove it by deleting it in the next. Git however, remembers data in all commits for all time.

So how do we actually properly delete files.zip and reclaim that space?

Remove the large files by rewriting git history - using the BFG repo cleaner

The BFG repo cleaner is an excellent example of a tool to make complex git operations more approachable for the average punter. I use docker to run it, but if you have java installed locally, then running it natively might be easier for you.

docker run -it --rm \
  --volume "$PWD:/home/bfg/workspace" \
  koenrh/bfg \
  --strip-blobs-bigger-than 100M

The output is amazing from the BFG repo cleaner as well, there is a report supplied showing the files it removed, and which commits needed to be altered.

You also need to run git garbage collection to reclaim the space

  git reflog expire --expire=now --all && git gc --prune=now --aggressive

After you are all done, re-run the command to find the new size

$ git count-objects -vH

count: 0
size: 0 bytes
in-pack: 4024
packs: 1
size-pack: 8.29 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

If you are happy then force push over your remote

git push

N.B. all rewritten commits will have a different SHA, and all other developers using the same repo will need to do steps to ensure they have the most recent version of the git repo.

git fetch origin
git reset --hard origin/master

Hope this helps somebody out there

I was recently to supply a list of non-superuser user accounts that could access a given Service ID in Fastly, and what their role and permission was.

The Fastly UI is not terribly useful for doing this, if your account is sufficiently large. At present, you would need to enumerate every single user, visit their access controls page, and then review which services they can access, and the permission level granted.

Users can be granted a range of roles, and within the engineer role can be granted permissions as a service ID level.

Rather than do this by hand (on top of taking loads of time, it would be prone to human error), I wanted to work out how to achieve this through programmatic means.

Step 1 - Setup Fastly CLI

This tool is invaluable for dealing with Fastly on the command line. Installation instructions can be found on Fastly's website.

I then configure the CLI tool with environment variables in my ~/.zshrc:

FASTLY_API_TOKEN=notarealtokenyo
export FASTLY_API_TOKEN
FASTLY_SERVICE_ID=banana
export FASTLY_SERVICE_ID
FASTLY_CUSTOMER_ID=notarealcustomerid
export FASTLY_CUSTOMER_ID

Step 2 - List service auth

This will list all user accounts that have access to a particular service ID $FASTLY_SERVICE_ID :

N.B. This does not include users with access to all services (such as superuser).

fastly service-auth list --json | jq --arg SERVICE_ID "$FASTLY_SERVICE_ID" -r '.Items[] | select( .Service.ID | contains($SERVICE_ID)) | {id: .User.ID, permission: .Permission, created: .CreatedAt}' | jq -s '.' > auth.json

The output will be similar to:

[
  {
    "id": "16Sw3VMRzrXXXXLQfL6aen",
    "permission": "purge_all",
    "created": "2022-09-20T13:37:43Z"
  },
  {
    "id": "ZFVW99lVnYXXXXbohCv82",
    "permission": "purge_all",
    "created": "2022-05-10T23:29:30Z"
  }
]

Step 3 - List users

Having access to only the user ID is not that great, we still need to match this to a users email address

fastly user list --customer-id $FASTLY_CUSTOMER_ID --json | jq -r '.[] | {id: .ID, email: .Login, role: .Role, locked: .Locked, tfa: .TwoFactorAuthEnabled}' | jq -s '.' > user.json

The output will be similar to:

[
  {
    "id": "16Sw3VMRzrXXXXLQfL6aen",
    "email": "somebody@example.com",
    "role": "engineer",
    "locked": false,
    "tfa": true
  },
  {
    "id": "ZFVW99lVnYXXXXbohCv82",
    "email": "anotherperson@example.com",
    "role": "engineer",
    "locked": false,
    "tfa": true
  }
]

Step 4 - Joining the 2 JSON files together

We now have 2 JSON files, that we need to merge, based on a shared key id and it's value. We can use native JOIN and INDEX in jq to do this (see the stackoverflow post):

jq 'JOIN(INDEX(inputs[];.id);.[];.id;add)' auth.json user.json | jq -s '.' 

The output is now pretty much perfect

$ jq 'JOIN(INDEX(inputs[];.id);.[];.id;add)' auth.json user.json | jq -s '.'
[
  {
    "id": "16Sw3VMRzrXXXXLQfL6aen",
    "permission": "purge_all",
    "created": "2022-09-20T13:37:43Z",
    "email": "somebody@example.com",
    "role": "engineer",
    "locked": false,
    "tfa": true
  },
  {
    "id": "ZFVW99lVnYXXXXbohCv82",
    "permission": "purge_all",
    "created": "2022-05-10T23:29:30Z",
    "email": "anotherperson@example.com",
    "role": "engineer",
    "locked": false,
    "tfa": true
  }
]

You can make a CSV (with header row) with some additional effort (see the stackoverflow post):

$ jq 'JOIN(INDEX(inputs[];.id);.[];.id;add)' auth.json user.json | jq -s '.' | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv'
"created","email","id","locked","permission","role","tfa"
"2022-09-20T13:37:43Z","somebody@example.com","16Sw3VMRzrXXXXLQfL6aen",false,"purge_all","engineer",true
"2022-05-10T23:29:30Z","anotherperson@example.com","ZFVW99lVnYXXXXbohCv82",false,"purge_all","engineer",true

You could even turn this into a giant single one liner if you want, but I am happy with this being a multi-step process.

This is the complementary blog post for my DrupalSouth Brisbane 2022 session.

Have you ever wondered how popular Drupal is in your local state and at the Australian Federal Government level? This blog post will help to answer that question, using open source tooling. The hope is that you gain some insight to the relative popularity of Drupal and appreciate more the impact you and Drupal have in Australia.

Existing solutions

There are a number of websites that will claim to be able to give you this information. However they all will likely want you at some point to pay them money.

• wappalyzer.com
• semrush.com
• builtwith.com
• whatcms.com
• similartech.com
• larger.io

“I wanted an open way to do this”

As it turns out, you can plug a few things together to scrape the technologies in use

• Wappalyzer library (open source)
• Puppeteer (open source)

Problem #1: How to get a list of all Australia Government domains

If anything, there are too many sources of this information:

• https://github.com/tb0hdan/domains
• Let's Encrypt Certificate Transparency (CT) logs
• SecurityTrails

A crawling method could also be done, loads of suitable seed sites, e.g.:

• https://www.australia.gov.au/
• https://www.directory.gov.au/departments-and-agencies

The main issue is that just having a list of sites, does not convey the importance of the site relative to another site.

Enter DomCop to which publishes a list of the top 10 million domains on the internet, including a rank and Open PageRank.

On top of suppling 5,795 Australian Government domains, there also is an "Open Page Rank" field. The PageRanks are calculated based on the Open data provided by Common Crawl and Common Search.

Problem #2: What is PageRank?

PageRank is a system for ranking web pages that Google's founders developed in 1996. A PageRank score of 0 is typically a low-quality website, whereas, a score of 10 would represent only the most authoritative sites on the web. It is logarithmic (with a base of 5).

A site with PageRank 3 is 5 times more authoritative than a site with PageRank 2.

Problem #3: Machinery of Government (MoG) and link rot

Australian Governments sites are never static, they are constantly evolving. Sometimes several sites merge into 1, or sometimes 1 site splits into move sites.

Just show me the graphs

Disclaimer:

• This is based on Sept 22, 2022 data
• The scoring is based off PageRank data, so the percentages are not raw counts of websites, but an approximation of how important the respective sites are compared to others (assumes a logarithmic base of 5).
• Wappalyzer detection is not perfect (see the end of this blog post for upstreamed PRs), and there is still a fairly large portion of sites where the CMS cannot be identified
• MoGs make this tricky (PageRank relies on incoming links, which break due to MoGs)
• Only *.gov.au domains considered (some Government sites use other TLDs)
• Unlikely newly created websites are in the top 10 million just yet (due to how PageRank works)

All sites (*.gov.au)

Federal sites (not state based domains)

Programmes like GovCMS are having an impact here.

Victoria *.vic.gov.au

The Single Digital Presence (SDP) programme makes a mark in Victoria.

New South Wales *.nsw.gov.au

Large Drupal sites like https://www.nsw.gov.au/ and https://www.service.nsw.gov.au/ help to make Drupal dominant in NSW.

South Australia *.sa.gov.au

Western Australia *.wa.gov.au

A lot of unknown CMSs in WA, including sites like https://ww2.health.wa.gov.au/ which I still have no idea what the CMS used is. Edit - a keen eyed developer has told me that because this URL exists, so the WA Health site is generated by SiteCore.

Tasmania *.tas.gov.au

The lowest usage of Drupal for any Australian state or territory and the highest percentage of Wordpress.

Queensland *.qld.gov.au

Australian Capital Territory *.act.gov.au

The highest percentage of Squiz compared to any other Australia state or territory.

Northern Territory *.nt.gov.au

Open Source Software (OSS) CMS vs Proprietary CMS

For the CMS' that can be identified, splitting them into 2 categories, OSS and Proprietary.

Drupal sites by major version

For sites reporting as Drupal, Drupal 9 and 7 are the most popular.

Observations and other unusual findings

#1 - Drupal usage

“Drupal powers roughly 27% of all digital experiences that you use in the Australian government”

#2 - Top contender

“Squiz Matrix is the top contender with 15%, and has a clear state led mandate in certain states/territories”

#3 - TLS coverage

TLS coverage is not 100% - 129 domains found with no TLS

#4 - If in doubt, add a number

19 domains found with ww[number] as a subdomain.

#5 - You cannot kill Dreamweaver

15 sites found in 2022.

Extending this for the future

• Crawl other domain spaces, e.g. the New Zealand government domain space *.govt.nz
• Make a website and publish this data quarterly (DomCop's data updates around this frequency)
• Measure trends over time

Upstreamed enhancements

These are all to make the detection of CMS' and Javascript frameworks more accurate for Australia Government sites.

• Ripple #6827
• Dreamweaver #6828
• HCL DX #6829
• MODX #6832
• Squiz Matrix #6833
• Optimizely CMS #6834
• Umbraco #6835
• Elcom #6836
• Lagoon #6861
• OSS flags and links #6902
• Squiz Matrix #2 #6925
• Ektron CMS #6924
• SiteCore #6926

Raw data

If you want to do your own analysis, here is a link to a full CSV dump.

Comments

I am keen to hear feedback on this data, and what can be done to improve the scoring. Also, if you can help fill in some of the 'unknown' data, let me know, I am happy to craft another PR into Wappalyzer.

On a recent project we were looking to take advantage of Amazon EC2 spot instances. The main issue was narrowing down the enormous list of instance types so they would be suitable to run our workloads. There are so many instance types with AWS, and they are growing all the time. To compound the problems, not all instance types are in every region.

Our key requirements were:

• 8 vCPU
• 32 GB RAM
• 58 pods per node (we use the nodes in EKS)
• Available in Sydney (ap-southeast-2)

Rather than go through manually all the instances to ensure they meet the above minimum requirements (to which there are 565 instance types 😲) to find the most optimal instances, I thought there must be a better way (read: faster).

Introducing Amazon EC2 Instance Selector

This is a neat little CLI tool that seems like is a great fit for this exact problem.

GitHub - aws/amazon-ec2-instance-selector: A CLI tool and go library which recommends instance types based on resource criteria like vcpus and memory

A CLI tool and go library which recommends instance types based on resource criteria like vcpus and memory - GitHub - aws/amazon-ec2-instance-selector: A CLI tool and go library which recommends in...

GitHubaws

Installation is simple on a mac:

brew tap aws/tap
brew install ec2-instance-selector

Here is the command that I ended up using to select the instances. Of note, I am asking for:

• 8 vCPU exactly and 32 GB RAM exactly. This seems to always yield 58 pods/node density when run in EKS. This is desirable for us.
• Current generation instances only
• Available in Sydney
• Must be spot capable
• No GPU instances, we don't have a use for these, and they cost loads
• Deny the t series (not suitable for our workloads), deny the i series (not suitable for our workloads), deny the r series (not suitable for our workloads)
• x86 architecture (perhaps ARM in the future)
• No more than 30 cents an hour

ec2-instance-selector \
  --profile $PROFILE_NAME \
  --vcpus 8 \
  --memory-min 32 \
  --current-generation true \
  --region ap-southeast-2 \
  --usage-class spot \
  --gpus 0 \
  --max-results 50 \
  --deny-list '^(t|i|r)' \
  --cpu-architecture x86_64 \
  --price-per-hour-max 0.3 \
  --output table-wide

This yields the following table:

Changing the output format to one-line for easy copy/paste.

m4.2xlarge,m5.2xlarge,m5a.2xlarge,m5ad.2xlarge,m5d.2xlarge,m5zn.2xlarge,m6i.2xlarge

And there you have it, this is how we were able to filter the giant list of AWS instance types to just 7.

I hope this ends up helping someone else in the future.

This came up recently, when a customer supplied a text file with several hundred CIDRs, and wanted these inputted into Fastly as an ACL (to control access to a certain site).

This was beyond my ability to do by hand, so here is a simple script to help automate this.

Step 1 - Setup Fastly CLI

This tool is invaluable for dealing with Fastly on the command line. Installation instructions can be found on Fastly's website.

I then configure the CLI tool with environment variables in my ~/.zshrc:

FASTLY_API_TOKEN=notarealtokenyo
export FASTLY_API_TOKEN
FASTLY_SERVICE_ID=banana
export FASTLY_SERVICE_ID

Step 2 - Add entries to the ACL

You first need to actually create the ACL, you can use the Fastly UI for this, or the CLI. Once you have done that, you need to find the ID of the newly created ACL:

fastly acl list --version=latest

To actually input the ACL entries, here is a small bash script to loop over each line in a text file, split by the / and then add the ACL entries with the subnet one at a time:

cat /tmp/ip.txt | while read LINE; do \
  IP=$(echo $LINE | cut -d '/' -f1)
  SUBNET=$(echo $LINE | cut -d '/' -f2)
  fastly acl-entry create --acl-id=$ACL_ID --ip=$IP --subnet=$SUBNET
done

This yields something like:

Duplicate ACL entries are ignored, so you can run this script multiple times (if say you get an updated version with additional entries).

Hope you find this of some value.

This is a collection of things I have come across using Docker over the years and working with kubernetes across thousands of projects. The hope is that there is a nugget of wisdom somewhere in this post, and that it can help on your next project.

I framed the post as a series of best practices, each one has a much larger story behind it, but I decided to keep this post positive 😁. I will likely continue to update this post as new information comes to light.

1. Create tiny images

Your ideal image should be as small as possible to do its job. No unnecessary binaries, temporary files, development tools, uncompiled code, documentation etc should be in the final image.

Your ideal image should be as small as possible to do its job.
- me, just now

Why this is important? Because size matters when it comes to docker images. The time to deploy your application into kubernetes will be largely determined by the image size. Smaller overall images push and pull faster into registries. Deployments are faster, applications scale horizontally faster in kubernetes. Life is generally better.

You can find the sizes of your local docker images with this handy command (source):

$ docker images --format "{{.ID}}\t{{.Size}}\t{{.Repository}}:{{.Tag}}" | sort -k 2 -h -r
307b0123a4dd	936MB	cachingheaders:latest
c28b5b41a4dd	405MB	ghost-example_node:latest
dd54221ac341	393MB	ghost-pixelite_node:latest
441ccef7ac50	318MB	uselagoon/solr-7.7-drupal:latest
416cb5fc3d92	257MB	uselagoon/mariadb-10.5-drupal:latest
5e977a444840	244MB	uselagoon/mariadb:latest
d9300475ba78	236MB	uselagoon/mariadb-drupal:latest

In kubernetes, this is a little trickier, as the kubernetes pod API does not return image size, the node API does however, and with some awk magic, you can get a list of images sorted by size descending in MiB (inspired by this post):

$ kubectl get nodes -o json \
  | jq -r '.items[].status.images[] | "\(.sizeBytes) \(.names[-1])"' \
  | sort -k 1 -n -r \
  | awk '{ printf "%d %s\n", $1/1024/1024, $2 }'
771 harbor.example.com/example-2020/deploy-lagoon-development/cli@sha256:77947585cbf58bdae7e6a9182b104c2b2e30f535d9238cae0683dd571e3a3411
768 harbor.example.com/example2-2020/deploy-lagoon-staging/cli@sha256:6569474568b04f9bddb2194a66c4bf4a40bac79d936dc3bd864afd351f3f2200
768 harbor.example.com/example3-2020/deploy-lagoon-production/cli@sha256:95c3d048f50e33ea12d0b68b64034e0fc81d1b7b6d3f15b84e161a35e5417d72

In general, I consider images in the low hundreds of MB to be OK, anything over 1GB to be large, and anything over 2 GB likely needs optimisation 😱. This obviously depends a lot on the language you are using (e.g. Go can build native executable binaries with no runtime), the size of the application (e.g. lines of code) and the Operating System required to run that code (e.g. Alpine Linux vs Ubuntu).

Avoid adding development and build related resources to the final image

The main issue with adding tools like gcc (which is needed to compile code) into your images is that you don't need this binary in production.

There are several techniques to deal with this ranging from virtual dependencies to builder images (covered later in this post).

# Add build tools.
RUN apk update \
    && apk add pv python3 make gcc g++ \
    && ln -sf python3 /usr/bin/python \
    && rm -rf /var/cache/apk/*

Most package managers/languages also have flags you can use to toggle a production build. Production builds omit development dependencies, and will overall lead to smaller images.

# PHP
RUN composer install --no-dev

# Nodejs
ENV NODE_ENV production
RUN npm ci --only=production

Go builds can also be optimised a lot, and have things like cross platform bits removed, and debugging information. See this blog post on some tips and tricks to reduce Go binary size.

Virtual packages

In Alpine Linux you can also make use of virtual packages, to which allow you to install build related packages, and then clean them up. This is extremely useful for compilation.

# Install libsvm, compile from source, and then remove dependencies.
COPY resources/libsvm.sh /tmp/libsvm.sh
RUN apk add --no-cache --virtual .build-deps \
    g++ \
    make \
    && chmod +x /tmp/libsvm.sh \
    && /tmp/libsvm.sh \
    && rm /tmp/libsvm.sh \
    && apk del .build-deps

N.B. you should run all commands in the same RUN command to ensure the layer stays lightweight.

Builder images (multi-stage builds)

Probably the most popular way to keep your resulting images small where compilation is involved is to make use of a builder image (AKA multi-stage builds).

Say you have a Nodejs application, that requires a compilation step, but you don't want Nodejs installed in your production running container (just Nginx serving the static output).

This is sometimes called multi-stage dockerfiles, and has been a feature of docker since 2017.

For a real life example of this (based on actual code), first you need to create your 'builder' intermediate image, this image will have the full development tools installed in it, and will be used to compile the application:

FROM uselagoon/node-14-builder:latest as builder
RUN npm install --pure-lockfile
RUN npm install -y yarn
RUN yarn build

After you have compiled the application (e.g. with yarn or something as above), you copy the resulting (static) artifacts into a clean final image:

FROM uselagoon/nginx:latest
COPY --from=builder /app/dist /app

The resulting Nginx image is fairly tiny, this particular one is around 133MB. This also has the nice side effect of being completely static, nothing to hack, no dynamic languages. A nice side effect of tiny images is a tiny attack surface. This will also make your security people happy.

For more information see https://codefresh.io/docker-tutorial/node_docker_multistage/ and https://docs.docker.com/develop/develop-images/multistage-build/#use-multi-stage-builds

Select the best base image

A rookie docker mistake is to start from a fairly generic base image, and then customise it to the nth degree. There are vast libraries of images out there already that you can elect to start from to give yourself a head start.

Using another organisation's images does come with certain points to ponder:

• Who maintains the images?
• How fast do they release new versions when a security issue is identified?
• Do you trust them?
• Is the code open source?

At amazee.io, the Lagoon team looks after a suite of upstream docker images in the uselagoon namespace. These images in turn (for the most part) inherit from specialised builds of Alpine Linux. e.g. this is a line from the PHP 8.0 FPM dockerfile:

FROM php:8.0.14-fpm-alpine3.14

By selecting the best starting point for your images, will mean less code you need to maintain, less layers in your dockerfiles, and faster builds.

Most applications also function perfectly well on Alpine Linux (a lightweight Linux distribution) to which can run on a 8MB image. If you have not checked this out, do it, your dockerhost thanks you in advance.

If you are using Rust or Go, you should consider using scratch (blog post on this topic) or distroless static (blog post on this topic) as your base image (which are extremely basic and lightweight images). Distroless static is the same as scratch but with a few niceties such as CA certificates installed, a functioning /tmp directory etc. All of this for 1 additional MB, sounds pretty good.

2. Docker Layer Caching (DLC)

During a build, docker steps through each line one at a time. As each line is read, docker attempts to match this against it's cache to see if the step can be re-used from cache.

Making effective use of layer caching can speed up your build times a lot. It is important to note that only certain commands create layers, namely ADD, COPY and RUN.

See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#leverage-build-cache

Ordering layers to increase cache hits

The order in which you create layers matters. In general, you want to have the most static things at the beginning of the docker file (e.g. environment variables) and more volatile things towards the end (e.g. code changes by a developer).

As soon as you have a cache miss on a layer, then all subsequent layers will need to be rebuilt.

This is especially useful for COPY commands. You want to structure the dockerfile such that the files that are most frequently changes are copied in as late as possible.

Inlining commands to reduce layers

Rather that create a layer for each similar command, you can chain lines with a backslash \ to ensure only a single layer is created.

RUN apk --no-cache add \
    bash \
    bind-tools \
    ca-certificates \
    curl \
    git \
    unzip

A pro tip is to ensure only 1 package per line, and keep them in alphabetical order. This helps to ensure the next PR to update the list is a tiny bit easier.

See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#minimize-the-number-of-layers

Breaking layer cache

You will run into situations where Operating System packages, NPM packages or a Git repo are updated to newer versions but as your Dockerfile or package.json hasn't updated, docker will continue using the cache. This may be less than ideal for your particular circumstance.

A quick way to 'bust the cache' is to define the use of a build time variable, and run a simple command to use it. Lagoon injects the SHA of the commit as such a variable, so if you do want to ensure the layer is built fresh, you can quite easily:

ARG LAGOON_GIT_SHA
RUN echo $LAGOON_GIT_SHA

Other build systems will likely have a similar way to obtain the SHA of the commit.

3. .dockerignore files

This will prevent certain local files and debug logs from being copied onto your Docker image and possibly overwriting files installed within your image during the build.

It also is a good idea to not copy your dockerfiles themselves into the docker image.

# PHP
.git
.idea
.ahoy.yml
.circleci
.dockerignore
.gitignore
Dockerfile
docker-compose.yml
README.md
vendor
web/sites/default/files

# Nodejs
.dockerignore
node_modules
npm-debug.log
Dockerfile
.git
.gitignore
.npmrc

My best advice here is to SSH into a running container and inspect the files you have left in there. If you see anything that is not essential to production runtime, then consider not COPY'ing it, or adding it to .dockerignore.

See https://docs.docker.com/engine/reference/builder/#dockerignore-file

Comments

If you have any neat tips or tricks, please let me know!

This topic came up a while ago when Acquia was looking to introduce Soft Purge as an option for their Acquia purge module in Drupal (N.B. not to be confused by the Fastly Drupal module).

There appears to be some confusion about what Fastly Soft Purge is, when it is useful, and how to use it. This post will help clear this topic up.

What is Fastly Soft Purge?

In Varnish, to force a miss for an object in cache you have a couple of options

• evict it completely (e.g. PURGE, or BAN with a lurker). The Varnish documentation cover this. This is the same as Fastly Instant Purge.
• or simply mark the object as expired. I imagine Fastly achieve this by tinkering with the TTL of the cached object. This is what Fastly Soft Purge is.

So the main difference with Fastly Soft Purge is that the object persists in cache, and can be used in a stale context. This can prove to be extremely useful.

When is Fastly Soft Purge useful?

I would say Fastly Soft Purge should be used in 99.99% of all purges. There are extremely limited situations in which I would recommend Fastly Instant Purge:

• You accidentally published embargoed content, or there is some legal issue with the content
• You accidentally published incorrect or incomplete content
• You accidentally published test content (we have all been there)
• You need to destroy the entire cache for a given Fastly service 😱 (which I never really recommend)

It boils down to making a whoopsie really, and I would imagine the vast majority of your purges should (hopefully) not fall into these categories.

Having a stale object is cache is useful as it:

• soft purges to high traffic pages (e.g. the homepage) will not result in slow downs for new users, this is because "Stale while revalidate" will perform a background (asynchronous) refresh of the cached object, and will serve the cached stale object in the meantime.
• if your origin is completely down (or in maintenance) stale objects can be served to users (which is often better than a generic error page). This is the "Stale if error" feature.

How to use Fastly Soft Purge with the Drupal module

Just because you execute a purge to Fastly does not mean you will instantly get any benefit, you also need to combo this with some "Stale Content Options" to which I will go through.

If you are using the Fastly Drupal module, Soft Purge is configured like so:

And as for Stale Content Options, you need to ensure that both "Stale while revalidate" and "Stale if error" are enabled. These are some basic tuning for the values as well:

If you are not using Drupal (or using the Acquia purge module), then you can always set the above HTTP headers manually. See the Fastly documentation on this.

How to use Fastly Soft Purge with the API only

If you don't run Drupal, or just want to execute a soft purge via the Fastly API, then this is simple as well, just pass in the Fastly-Soft-Purge HTTP header:

curl -sXPOST https://api.fastly.com/service/SERVICE_ID/purge/TAG \
 -H 'Fastly-Soft-Purge:1' \
 -H "Fastly-Key:$FASTLY_TOKEN" | jq
{
  "status": "ok",
  "id": "10427-1615460322-6"
}

Wait, I want to learn more

If you want to go deeper in Varnish (what powers Fastly), and how objects, TTL, stale and grace (stale-while-revalidate) are involved, please see their documentation.

There is also this excellent graphic to which explains how an object goes from fresh to stale

Gotchas

If you use Fastly shielding (which you probably should), then there is the potential for soft purged content on the edge PoP to end up being re-cached as fresh 😱. See the Fastly documentation on this. There is a VCL snippet to mitigate this:

if (fastly.ff.visits_this_service > 0) {
  set req.max_stale_while_revalidate = 0s;
}

If you are not already using Fastly shielding, and you care about caching, and cache hit rates, you should really start to look at this. The benefits far outweigh the tiny edge cases.

P.S. if you are not already using the Fastly purge auth snippet, get this installed ASAP. By not doing this, a bad actor is able to purge any single URL on your site, in an instant fashion. This could create a DoS.

# Fastly's URL purge feature allows you to purge individual URLs on your
# website. By default, authentication is not required to purge a URL with the
# Fastly API, but you can enable API token authentication with this snippet.
#
# @see https://docs.fastly.com/en/guides/authenticating-api-purge-requests
if ( req.request == "FASTLYPURGE" ) {
    set req.http.Fastly-Purge-Requires-Auth = "1";
}

Comments

If you have any other pro tips when it comes to soft purging in Fastly, please let me know.

One of the cheapest and easiest methods for logging in Fastly is to use Amazon S3. S3 is extremely low cost, and suitable for long term storage and archival of logs.

This post will examine how to set this logging up, some of the common gotchas I came across after running this in production for a while now. The post will also explore how to download and analyse the logs.

Create a new logging endpoint

In the desired service, visit the Logging section, and add a new Amazon S3 endpoint.

Creating a custom JSON log format

Fastly comes with a default, I choose to log specific fields that are useful for debugging and analysis later.

{ "timestamp":%{time.start.sec}V, "service_id":"%{req.service_id}V", "time_elapsed":%{time.elapsed.usec}V, "is_edge":%{if(fastly.ff.visits_this_service == 0, "true", "false")}V, "client_ip":"%{req.http.Fastly-Client-IP}V", "client_as":"%{if(client.as.number!=0, client.as.number, "null")}V", "geo_city":"%{client.geo.city}V", "geo_country_code":"%{client.geo.country_code}V", "request":"%{req.request}V", "host":"%{json.escape(req.http.Host)}V", "url":"%{json.escape(req.url)}V", "request_referer":"%{json.escape(req.http.Referer)}V", "request_user_agent":"%{json.escape(req.http.User-Agent)}V", "resp_status":"%s", "resp_message":"%{resp.response}V", "resp_body_size":%{resp.body_bytes_written}V, "server_datacenter":"%{server.datacenter}V", "cache_status":"%{fastly_info.state}V", "waf_block":"%{req.http.X-Waf-Block}V", "waf_block_id":"%{req.http.X-Waf-Block-Id}V" }

An example request will look like this:

{
  "timestamp": 1629796513,
  "service_id": "4gJgt7cCs3QChTi2evKdke",
  "time_elapsed": 1064614,
  "is_edge": true,
  "client_ip": "107.170.227.23",
  "client_as": "14061",
  "geo_city": "san francisco",
  "geo_country_code": "US",
  "request": "GET",
  "host": "www.example.com",
  "url": "/",
  "request_referer": "http://www.example.com/",
  "request_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36",
  "resp_status": "200",
  "resp_message": "OK",
  "resp_body_size": 16362,
  "server_datacenter": "PAO",
  "cache_status": "MISS-CLUSTER",
  "waf_block": "0",
  "waf_block_id": "0"
}

Extending the format

You can log pretty much anything you want here, there is a comprehensive list in the Fastly docs about other variables you can log.

Gotchas

During the development of our logging format several things have cropped up that I have had to solve:

Duplicate logs with shielding

If you enable shielding, then every request from an end user could potentially be 2 logs (edge PoP and shield PoP). In order to counter this, you can filter logs by is_edge being true. The below jq examples will show how to do this.

Escape user supplied fields

If you find this out the hard way when your JSON is no longer valid. An easy rule of thumb is - if it is supplied by the end user, then you must escape it

"host":"%{json.escape(req.http.Host)}V"

Truncate user supplied fields

I also saw examples where the user agent supplied was thousands of characters long, the same with the referrer. Fastly has some hidden truncation logic of it's own (around 1,500 characters or something), so the full log line will not be written. The solution was to deploy this VCL snippet in vcl_recv:

if ( req.http.request_referer ~ ".{1000,}" ) {
  set req.http.request_referer = regsub(req.http.request_referer, "(.{0,125}).*", "\1");
}
if ( req.http.User-Agent ~ ".{1000,}" ) {
  set req.http.User-Agent = regsub(req.http.User-Agent, "(.{0,125}).*", "\1");
}

Set the S3 domain correctly

Unless your S3 bucket is in us-east-1 it is likely you will need to customise the domain for the S3 bucket. Ensure this is correct, else you will not see a single

Log line format should be blank

This is now the default (it was not a while back), just ensure this stays on Blank, else Fastly will write 'helpful' timestamps to each line, and mess up the JSON syntax.

Downloading the logs

Now that logs are flowing into your S3 bucket, if you actually want to interact with them, it is likely better to download them. This is simple with the AWS CLI:

aws --profile PROFILE_NAME s3 sync s3://BUCKET_NAME/ . --exclude '*' --include '2021-09-25*'

This will produce a lot of log files (1 per PoP per hour). In order to do analysis on them, it is easier to have a giant log file:

find . -maxdepth 1 -name "*.log" -print0 | xargs -0 cat > logs.json

Analyse your Fastly logs

Now that you have the raw data, you should look to turn it into something you can make business decisions with.

Here are some simple analysis you can do with the jq tool (ensure you install this first if you have not already).

Top URIs

jq -r 'select(.is_edge) | .url' ${FILENAME} | sort -n | uniq -c | sort -nr | head -n 3

704990 /
670605 /sitewide_alert/load
109435 /ajax/site_alert

Top user agents

jq -r 'select(.is_edge) | .request_user_agent' ${FILENAME} | sort -n | uniq -c | sort -nr | head -n 3


580342 Elastic-Heartbeat/7.14.0
418879 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36
290797 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36

Top HTTP 404s

jq 'select(.is_edge and (.resp_status == "404")) | "\(.host)\(.url)"' ${FILENAME} | sort -n | uniq -c | sort -nr | head -n 3

58791 "www.example.com/sites/default/files/logo.png"
58117 "www.example.com/sites/default/files/social/Facebook.png"
58021 "www.example.com/sites/default/files/social/LinkedIn.png"

Comments

If you have any neat tips and tricks around logging with Fastly to S3, please let me know in the comments.

Ghost currently is geared towards only hosting 1 copy of Ghost per site, in fact this is explicitly written in the FAQs:

Ghost doesn’t support load-balanced clustering or multi-server setups of any description, there should only be one Ghost instance per site.

and also the hosting page:

Clustering or sharding is not supported in any way.

At my work amazee.io we specialise in hosting applications in Kubernetes (both native Kubernetes and Openshift), so I thought of this like a challenge to see how well Ghost would sit on a completely different architecture then a single VM in DigitalOcean (Ghost recommends this as a web host).

Why is Kubernetes so different from a traditional VM

Kubernetes runs your applications inside containers, and those containers run in pods. Your pod at any one time may be running, or it may not (e.g. it may be in the process of being moved to another node). Building resilient architectures in Kubernetes often involves ensuring your application can run in a Highly Available (HA) setup, which often means running 2 or more pods at any one time. It also means avoiding any single point of failures.

This way your application can survive the loss of a single node, or even Availability Zone with zero changes (it is transparent to your application).

This is a radical deviation from Ghost's recommended hosting approach.

Approach to getting Ghost running on Kubernetes

The approach I took was:

• Inherit from an existing official docker image for Ghost (to avoid me having to learn how to install Ghost). Github repo is here. I used this image as a builder image.
• Use a Node.js 14 based image with Alpine Linux as the Operating System (to ensure the image is as small as possible). Node.js 14 is also an LTS version (with security support to April 2023), it is also Ghost's recommended version to run.
• Use a Persistent Volume (PV) for all uploaded files (e.g. images, themes etc) in Ghost (path is /var/lib/ghost/content/). This way these files will survive pod restarts. The PV will be ReadWriteMany to ensure multiple pods can read and write to the PV at the same time.
• Use a MySQL database through the Cloud provider in question (in my case AWS RDS Aurora). You never want to host a database yourself in Kubernetes, if you want it to be HA. You certainly do not want to use SQLite.
• Shim the supplied environment variables into the names that Ghost expects.

The above approach ended up working quite well, as you will see later on, the code to get this done is quite tiny.

What this ends up looking like

Here is a screenshot of LensIDE to which lists the pods running for this site (meta am I right):

I also created an Horizontal Pod Autoscaler (HPA) to show that you can scale this deployment with load automatically

The HPA likely will not be used (Ghost is not heavy on the CPU), but it is there in any case.

N.B. there appears to be some in-memory caching that Ghost does, this means that running 2 pods minimum will cause issues (e.g. some URLs will respond with HTTP 404 on newly created posts for example). Likely best to run with 1 pod for now.

Example repository

I created an example to show this in action, that you can use to give you a headstart:

seanhamlin/ghost-example

Contribute to seanhamlin/ghost-example development by creating an account on GitHub.

GitHubseanhamlin

Things still to do/gotchas

A few things I still need to solve, and/or write about:

• Email configuration - not essential for this blog, but it would be nice to actually get this working. Seems like just a few more environment variables to configure.
• The PV mount replaces the default files at /var/lib/ghost/content/, so the theme Casper will not be available the first time the application runs. Ghost gets grumpy about that. This likely can be fixed with another entrypoint to copy the files into the correct place during pod startup, if the files are missing.
• How I setup caching in Fastly. Ghost does not play nicely with CDNs out of the box. I had to tune this.
• Fix Systemd complaints - does not seem to be in Alpine Linux

I will update this blog post as further details come in and I solve the above things.

Comments

Keen to hear from anyone else that in interested in this, or has questions or concerns about this approach.

How we configure Fastly at work is typically to use a single Fastly service and add multiple domains to each service. This works well for a number of reasons:

• Configuration - each service has it's own configuration and VCL snippets. It is common to have the same configuration extent to multiple sites.
• Cost - number of services is a factor in Fastly costs

One of the current limitations out of the box, is that you cannot purge a single domain (and its contents) from a service, without dropping the entire cache. This post will explore a solution to this.

Altering Surrogate-Key in Fastly VCL

It turns out you can add a custom snippet in vcl_fetch to which will adjust the response from origin, to create (or append to) the current domain to HTTP header Surrogate-Key.

# We want to append the host of the request to the response Surrogate-Key
# header, or create it, if it does not already exist. We only want to do this
# once, and this includes if shielding is in use.
#
# This is so we can purge the domain, of all requests, including static files,
# without touching other domains on the service.
if ( !beresp.http.Surrogate-Key ) {
  set beresp.http.Surrogate-Key = req.http.host;
}
else {
  if ( !std.strstr(beresp.http.Surrogate-Key, req.http.host) ){
    set beresp.http.Surrogate-Key = beresp.http.Surrogate-Key " " req.http.host;
  }
}

N.B. This new logic will only apply to new objects in cache only, existing cached objects will be unaffected. It pays to think ahead for this logic. You might also consider dropping the entire service cache after you have the VCL in place (if this is an option).

How to test if it is working

You can use cURL to test this snippet is working, you need to pass a debug header Fastly-Debug in order to see this:

curl -sLIXGET https://www.example.com/ -H 'Fastly-Debug:1' | grep -i surrogate-key
surrogate-key: www.example.com

If you have existing cache tags in the response, then you should see the custom cache tag appended at the end:

curl -sLIXGET https://www.exampletwo.com/ -H 'Fastly-Debug:1' | grep -i surrogate-key
surrogate-key: aZw7 f57P eygW 45lo gq5V Czwh cvlF /+/A 7KwZ eth0 www.exampletwo.com

How to purge an entire domain using the Fastly API

Now armed with the knowledge that every single cached object in Fastly has a custom cache tag attached to it (being the domain) we can easily purge a single domain using the Fastly API:

curl -sXPOST https://api.fastly.com/service/SERVICE_ID/purge/www.example.com -H 'Fastly-Soft-Purge:1' -H "Fastly-Key:$FASTLY_TOKEN" | jq
{
  "status": "ok",
  "id": "10427-1615460322-6"
}

N.B. this is executing a 'Soft Purge', so the object is still in Fastly cache, it is just marked as stale. This is useful if you have configured Serve While Stale (SWS), as stale objects can be served in the event the origin is down or otherwise not returning happy responses. So this is a much safer purge. If you want to read more about this, see my blog post on Soft Purge vs Instant Purge.

Further thoughts

I did ask Fastly if they would consider adding another option to the purge dropdown (my attempt at a mockup below), without having to muck around with custom VCL snippets.

Apparently this is not a popular feature request. If this blog post was helpful for you, I would encourage you to contact Fastly to let them know this would make your life easier.

Stock Varnish can BAN by domain, so this is not a limitation of Varnish.

Comments

If this has helped you, or you have an improvement to the VCL, please let me know in the comments.