The steps that any new launch or high traffic event should go through in order to have the best chance of success. This post is aimed at the project management level, so will try to stay out of the weeds, and focus on the high level topics you need to think about. There is a ~18 minute recording at the end of this post where I presented this topic at Drupalsouth 2019.

Preamble: What could be considered a high traffic event

Launching a brand new site

• Re-platforming (e.g. moving CMS version or type, or between hosting providers)
• eDM or other marketing event (e.g. Adwords)
• Planned traffic event (e.g. black Friday)
• Unplanned traffic event (e.g. news and media site)

Step 1) Ensure you have some basic Drupal configuration in place

• Disable known problem child modules dblog, devel, statistics, radioactivity, page_cache
• Enable dynamic_page_cache (if you have authenticated traffic)
• Set minimum cache lifetime to something sensible
• JS and CSS aggregation enabled
• Automate these checks with Drutiny

Step 2) Content Delivery Network (CDN)

Additional insurance against a lot of traffic is distributing your cached content to all corners of the globe.

Tiered caching should be used to ensure the highest offload rate. Most CDN providers will support this at a given price point.

Step 3) Cache tuning and minimising origin requests

Every request that bypasses your CDN layer adds load to the platform. In order to have the best chance of surviving a high traffic event, origin traffic needs to be carefully considered and reduced where possible.

Requests to origin that are often overlooked

• 404s
• Marketing based parameters (e.g. utm_campaign)
• Redirects (especially if re-platforming)
• WAF to block silly requests (e.g. Wordpress URLs like wp-login.php)

It you are interested in WAF tuning, you should check out my talk last year on using Cloudflare to secure your Drupal site.

Step 4) Load testing

If you are building a new site, or are expecting a substantially different traffic profile than what you have currently, then you should look to load test the system.

• Production hardware replica (scaled up if appropriate)
• Emulate expected user behaviour, use existing analytics, or expected flows
• Emulate what the browser would be doing (download all assets, including any HTTP 404s)
• Ensure complex tasks are also simulated at the same time (e.g. editorial, searching, form submissions, feeds ingestions)

At the end of this task (they you may need to run several times), you should have the confidence that you can handle the traffic expected.

Step 5) Hardware (auto) scaling

Now that you have the hardware you need to have in place with load testing, ensure you have autoscaling in place to deal with the peaks and troughs (it is unlikely you need to run your peak hardware for the entire duration of the event).

Autoscaling can also help if the origin traffic that you experience is higher than anticipated.

Test the autoscaler, set limits that you are comfortable with, and ensure you know how quickly the new resources take to come to life.

Step 6) Have a good fallback

Say the worst does happen, and you site does go down, or a critical API drops off the face of the internet, what does the end user see? Can you offer at least a better experience than a generic web server error page?

Most CDNs will have the ability to load balance origins (hot DR), and even fallback to a static version of the site if all origins are down.

It would make sense to test this prior to the high load event as well.

Step 7) Warm your cache

If you have a rather long tail website, it will be worth warming your cache prior to the event. An excellent module called warmer has been written, to which allows warming all sorts of caches. It can for instance load every page in the XML sitemap. So this is fairly low effort, high reward.

Step 8) Third party API dependencies

This is more of a fundamental design decision likely made much earlier on in the project. Say the content of your page is dependent on the content in an API response.  If you request the API content during page generation time, then you are tying the speed and availability of your site to another site (often outside your control).

This can lead to slow page load times, and worse case scenario can tie up your server's resources.

New Relic APM has "external requests" to which allow you to visualise this.

There are ways to mitigate this:

• Fetch the data in the background and cache locally in Drupal for as long as the data is considered 'good'. e.g. using Drush and a cronjob.
• Use a client side application (e.g. React) and request the API response in the client side
• Use a CDN on the API and see Step #6 above

Step 9) Realtime analytics

During the event, having access to realtime (or near realtime) analytics to find out

• how the system is currently performing
• requests/sec
• where the traffic is coming from
• cache offload rate from the CDN

Is extremely valuable. Even more valuable is being able to respond to this data in a quick and efficient matter. Having access to technical people can help. The types of logs and analytics you should be looking to get a hold of:

• Web analytics tools (e.g. Google Analytics)
• APM tools (e.g. New Relic)
• CDN analytics (e.g. Cloudflare Logs)
• Log stream from hosting provider (e.g. PHP error log)

To see where you can take this, you might also be interested in reading this blog post that shows off some dashboards that were purpose built for a high traffic event.

Step 10) Application changes in a pinch

If you do spot something in your analytics, knowing what tools you have at your disposal to mitigate issues quickly and easily is worth knowing.

• Cloudflare page rules (redirect a broken path, increase the WAF presence on a route)
• Nginx or Apache configuration
• Application hotfix (avoid clearing the cache)

Knowing what tool will solve what problem, how long each option takes to deploy, how safe it is, how easy is the rollback is is absolutely critical.

Step 11) Letting your hosting provider and their support team know

No-one likes surprises, so plan ahead. Ensure there are people available or on call during your traffic event. This goes for both your hosting provider, to CDN provider to support staff.

Postamble: What success looks like

So after your high traffic event has ended, here are some simple things to check in order to see how successful you were:

• Minimal origin requests and a high CDN offload
• Boring origin hardware graphs
• No rants on twitter
• No trending hashtag on twitter that is negative
• Users remember the event for it's content, and not the problems with it

Drupalsouth 2019 video

Let me know in the comments if this was of use, and also if you have any other words of wisdom for anyone else.

On a number of Drupal sites that I am involved with recently I see the error message when either trying to dump or restore certain databases:

ERROR 2020 (HY000) at line 1: Got packet bigger than 'max_allowed_packet'

It is important to note that with MySQL there are 2 settings for max_allowed_packet, the client (what you are connecting from), and the server (what you are connecting to).

In order to find out the client's current settings, you can run:

$ mysql --help | grep max-allowed-packet | grep -v '#' | awk '{print $2/(1024*1024)}'
16

So 16MB on the client side.

In order to find out the server's current setting, you can run:

SHOW VARIABLES LIKE 'max_allowed_packet';
+--------------------+----------+
| Variable_name      | Value    |
+--------------------+----------+
| max_allowed_packet | 67108864 |
+--------------------+----------+

So 64MB on the server side.

So at the moment:

• The client cannot send or receive more than 16MB in a single statement
• The server cannot send or receive more than 64MB in a single statement

So you can end up in a position where there is content in your database, that is working fine, but you cannot dump the database, nor can you restore from a dump (with your current client configuration).

Take the search_api_attachments module in Drupal, it uses the key_value table in Drupal 8 to store extracted text from documents.

SELECT name, length(value) as size FROM key_value ORDER BY size DESC limit 5;
+------------------------------+----------+
| name                         | size     |
+------------------------------+----------+
| search_api_attachments:14646 | 33623803 |
| search_api_attachments:14288 |  3394023 |
| search_api_attachments:13146 |  2356958 |
| search_api_attachments:4921  |  1554830 |
| search_api_attachments:1586  |  1549981 |
+------------------------------+----------+

Solution 1 - alter max_allowed_packet

The first solution is simply to alter the max_allowed_packet size to accommodate the size needed (on both the client and server). The only issue is that this is a game of cat and mouse. As soon as you tune the size to be larger, a content editor will upload a larger document.

It also means your database size will grow fairly unchecked, especially if you have heavy editorial where documents are commonly uploaded.

While I do advocate for sensible defaults, I think having > 64MB in a single cell in a table as potentially overkill for the benefits it provides.

Solution 2 - reduce the amount in the database

The point of indexing attachments is to ensure that you can still find content that is buried in binary files. I would argue that most of the important keywords of these documents will be in the first few pages. Indexing the entire document often will provide limited added value (if any).

Reading through the code of the search_api_attachments module, I spot these useful lines:

  /**
   * Limit the indexed text to first N bytes.
   *
   * @param string $extracted_text
   *   The hole extracted text.
   *
   * @return string
   *   The first N bytes of the extracted text that will be indexed and cached.
   */
  public function limitBytes($extracted_text) {
    $bytes = 0;
    if (isset($this->configuration['number_first_bytes'])) {
      $bytes = Bytes::toInt($this->configuration['number_first_bytes']);
    }
    // If $bytes is 0 return all items.
    if ($bytes == 0) {
      return $extracted_text;
    }
    else {
      $extracted_text = mb_strcut($extracted_text, 0, $bytes);
    }
    return $extracted_text;
  }

So it turns out, baked into the module, is a way to effectively limit the number of characters stored in the database (drupal.org issue).

In order to enable this feature, you need to edit the Processors for a given index:

Inside this page, is a configuration form that allows you to set a max limit for the amount stored in the database.

After setting the size to 100 KB which seems like a reasonable number, and then re-indexing the appropriate index, you see the results

SELECT name, length(value) as size FROM key_value ORDER BY size DESC limit 5;
+--------------------------------+--------+
| name                           | size   |
+--------------------------------+--------+
| node.field_storage_definitions | 116308 |
| search_api_attachments:831     | 102412 |
| search_api_attachments:1536    | 102412 |
| search_api_attachments:1571    | 102412 |
| search_api_attachments:1576    | 102412 |
+--------------------------------+--------+

So this will mean that the database is a lot smaller, allowing faster database backups, restores, and rollbacks. It also will save a lot of issues around forever tuning max_allowed_packet.

It is also worth noting that this key_value storage is actually a caching system for search_api_attachments, and that this table will be used, even if your only search servers are external to Drupal (e.g. Solr), and you make no use of database searching.

Update 21 November 2019

In the hopes that the module maintainers make a more sensible default limit I have also raised this issue to get a default value set. Having any size limit would be better than having no limit. A patch is now uploaded so you can test this out.

Update 23 November 2019

The patch has been committed, and a new beta released for Drupal 8 🎉. Thanks so much to Ismaeil (module maintainer) for the prompt reply and action.

Comments

If you have come across this in the past, what was your go to solution? I am keen to understand how others have solved this issue, and how big your key_value table got in the mean time.

I am writing this quick tutorial in the hopes it helps someone else out there. There are a few guides out there to do similar tasks to this. They just are not quite what I wanted.

To give everyone an idea on the desired outcome, this is what I wanted to achieve:

Before I dive into this, I will mention that you can do this with views, if all that you want to produce is content supplied by views. Ivan wrote a nice article on this. In my situation, I wanted a completely custom route, controller and theme function. I wanted full control over the output.

Steps to add sub tabs

Step 1 - create a new module

If you don't already have a module to house this code, you will need one. These commands make use of Drupal console, so ensure you have this installed first.

drupal generate:module --module='Example module' --machine-name='example' --module-path='modules/custom' --description='My example module' --package='Custom' --core='8.x'

Step 2 - create a new controller

Now that you have a base module, you need a route

drupal generate:controller --module='example' --class='ExampleController' --routes='"title":"Content", "name":"example.user.contentlist", "method":"contentListUser", "path":"/user/{user}/content"'

Step 3 - alter your routes

In order to use magic autoloading, and also proper access control, you can alter your routes to look like this. This is covered in the official documentation.

# Content user tab.
example.user.contentlist:
  path: '/user/{user}/content'
  defaults:
    _controller: '\Drupal\example\Controller\ExampleController::contentListUser'
    _title: 'Content'
  requirements:
    _permission: 'access content'
    _entity_access: 'user.view'
    user: \d+
  options:
    parameters:
      user:
        type: entity:user

# Reports user tab.
example.user.reportList:
  path: '/user/{user}/reports'
  defaults:
    _controller: '\Drupal\example\Controller\ExampleController::reportListUser'
    _title: 'Reports'
  requirements:
    _permission: 'access content'
    _entity_access: 'user.view'
    user: \d+
  options:
    parameters:
      user:
        type: entity:user

Step 4 - create example.links.task.yml

This is the code that actually creates the tabs in the user profile. No Drupal console command for this unfortunately. The key part of this is defining base_route: entity.user.canonical.

example.user.content_task:
  title: 'Content'
  route_name: example.user.contentlist
  base_route: entity.user.canonical
  weight: 1

example.user.reports_task:
  title: 'Reports'
  route_name: example.user.reportList
  base_route: entity.user.canonical
  weight: 2

Step 5 - enable the module

Don't forget to actually turn on your custom module, nothing will work until then.

drush en example

Example module

The best (and simplest) example module I could find that demonstrates this is the Tracker module in Drupal core. The Tracker module adds a tab to the user profile.

This is a short story on an interesting problem we were having with the Feeds module and Feeds directory fetcher module in Drupal 7.

Background on the use of Feeds

Feeds for this site is being used to ingest XML from a third party source (Reuters). The feed perhaps ingests a couple of hundred articles per day. There can be updates to the existing imported articles as well, but typically they are only updated the day the article is ingested.

Feeds was working well for over a few years, and then all of a sudden, the ingests started to fail. The failure was only on production, whilst the other (lower environments) the ingestion worked as expected.

The bizarre error

On production we were experiencing the error during import:

PDOStatement::execute(): MySQL server has gone away database.inc:2227 [warning] 
PDOStatement::execute(): Error reading result set's header [warning] 
database.inc:2227PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has [error]

The error is not so much that the database server is not alive, more so that PHP's connection to the database has been severed due to exceeding MySQL's wait_timeout value.

The reason why this would occur on only production happens on Acquia typically when you need to read and write to the shared filesystem a lot. As lower environments, the filesystem is local disk (as the environments are not clustered) the access is a lot faster. On production, the public filesystem is a network file share (which is slower).

Going down the rabbit hole

Working out why Feeds was wanting to read and/or write many files from the filesystem was the next question, and immediately one thing stood out. The shear size of the config column in the feeds_source table:

mysql> SELECT id,SUM(char_length(config))/1048576 AS size FROM feeds_source GROUP BY id;
+-------------------------------------+---------+
| id                                  | size    |
+-------------------------------------+---------+
| apworldcup_article                  |  0.0001 |
| blogs_photo_import                  |  0.0003 |
| csv_infographics                    |  0.0002 |
| photo_feed                          |  0.0002 |
| po_feeds_prestige_article           |  1.5412 |
| po_feeds_prestige_gallery           |  1.5410 |
| po_feeds_prestige_photo             |  0.2279 |
| po_feeds_reuters_article            | 21.5086 |
| po_feeds_reuters_composite          | 41.9530 |
| po_feeds_reuters_photo              | 52.6076 |
| example_line_feed_article           |  0.0002 |
| example_line_feed_associate_article |  0.0001 |
| example_line_feed_blogs             |  0.0003 |
| example_line_feed_gallery           |  0.0002 |
| example_line_feed_photo             |  0.0001 |
| example_line_feed_video             |  0.0002 |
| example_line_youtube_feed           |  0.0003 |
+-------------------------------------+---------+

Having to deserialize 52 MB of ASCII in PHP is bad enough.

The next step was dumping the value of the config column for a single row:

drush --uri=www.example.com sqlq 'SELECT config FROM feeds_source WHERE id = "po_feeds_reuters_photo"' > /tmp/po_feeds_reuters_photo.txt

Then open the resulting file in vim:

"/tmp/po_feeds_reuters_photo.txt" 1L, 55163105C

And sure enough, inside this config column was a reference to every single XML file ever imported, a cool ~450,000 files.

a:2:{s:31:"feeds_fetcher_directory_fetcher";a:3:{s:6:"source";s:23:"private://reuters/pass1";s:5:"reset";i:0;
s:18:"feed_files_fetched";a:457065:{
s:94:"private://reuters/pass1/topnews/2018-07-04T083557Z_1_KBN1JU0WQ_RTROPTC_0_US-CHINA-AUTOS-GM.XML";i:1530693632;
s:94:"private://reuters/pass1/topnews/2018-07-04T083557Z_1_KBN1JU0WR_RTROPTT_0_US-CHINA-AUTOS-GM.XML";i:1530693632;
s:96:"private://reuters/pass1/topnews/2018-07-04T083557Z_1_LYNXMPEE630KJ_RTROPTP_0_USA-TRADE-CHINA.XML";i:1530693632;
s:97:"private://reuters/pass1/topnews/2018-07-04T083617Z_147681_KBE99T04E_RTROPTT-LNK_0_OUSBSM-LINK.XML";i:1530693632;
s:102:"private://reuters/pass1/topnews/2018-07-04T083658Z_1_KBN1JU0X2_RTROPTT_0_JAPAN-RETAIL-ZOZOTOWN-INT.XML";i:1530693632

So this is the root cause of the problem, Drupal is attempting to stat() ~450,000 files that do not exist, and these files are mounted on a network file share. This process took longer than MySQL's wait_timeout and MySQL closed the connection. When Drupal finally wanted to talk to the database, it was not to be found.

Interesting enough, the problem of the config column running out of space came up in 2012, and "the solution" was just to change the type of the column. Now you can store 4GB of content in this 1 column. In hindsight, perhaps this was not the smartest solution.

Also in 2012, you see the comment from @valderama:

However, as feed_files_fetched saves all items which were already imported, it grows endless if you have a periodic import.

Great to see we are not the only people having this pain.

The solution

The simple solution to limp by is to increase the wait_timeout value of your Database connection. This gives Drupal more time to scan for the previously imported files prior to importing the new ones.

$databases['default']['default']['init_commands'] = [
  'wait_timeout' => "SET SESSION wait_timeout=2500",
];

As you might guess, this is not a good long term solution for sites with a lot of imported content, or content that is continually being imported.

Instead we opted to do a fairly quick update hook that would loop though all of the items in the feed_files_fetched key, and unset the older items.

<?php

/**
 * @file
 * Install file.
 */

/**
 * Function to iterate through multiple strings.
 *
 * @see https://www.sitepoint.com/community/t/strpos-with-multiple-characters/2004/2
 * @param $haystack
 * @param $needles
 * @param int $offset
 * @return bool|int
 */
function multi_strpos($haystack, $needles, $offset = 0) {
  foreach ($needles as $n) {
    if (strpos($haystack, $n, $offset) !== FALSE) {
      return strpos($haystack, $n, $offset);
    }
  }
  return false;
}

/**
 * Implements hook_update_N().
 */
function example_reuters_update_7001() {
  $feedsSource = db_select("feeds_source", "fs")
    ->fields('fs', ['config'])
    ->condition('fs.id', 'po_feeds_reuters_photo')
    ->execute()
    ->fetchObject();

  $config = unserialize($feedsSource->config);

  // We only want to keep the last week's worth of imported articles in the
  // database for content updates.
  $cutoff_date = [];
  for ($i = 0; $i < 7; $i++) {
    $cutoff_date[] = date('Y-m-d', strtotime("-$i days"));
  }

  watchdog('FeedSource records - Before trimmed at ' . time(), count($config['feeds_fetcher_directory_fetcher']['feed_files_fetched']));

  // We attempt to match based on the filename of the imported file. This works
  // as the files have a date in their filename.
  // e.g. '2018-07-04T083557Z_1_KBN1JU0WQ_RTROPTC_0_US-CHINA-AUTOS-GM.XML'
  foreach ($config['feeds_fetcher_directory_fetcher']['feed_files_fetched'] as $key => $source) {
    if (multi_strpos($key, $cutoff_date) === FALSE) {
      unset($config['feeds_fetcher_directory_fetcher']['feed_files_fetched'][$key]);
    }
  }

  watchdog('FeedSource records - After trimmed at ' . time(), count($config['feeds_fetcher_directory_fetcher']['feed_files_fetched']));

  // Save back to the database.
  db_update('feeds_source')
    ->fields([
      'config' => serialize($config),
    ])
    ->condition('id', 'po_feeds_reuters_photo', '=')
    ->execute();
}

Before the code ran, there were > 450,000 items in the array, and after we are below 100. So a massive decrease in database size.

More importantly, the importer now runs a lot quicker (as it is not scanning the shared filesystem for non-existent files).

What this means for Feeds in Drupal 8

It came to my attention from Dinesh, that this same issue may likely impact Drupal 8 feeds. To make things slightly more interesting, is that the functionality of the Drupal 7 module feeds_fetcher_directory is now moved into the main feeds module.

An issue has been opened on Drupal.org to track this. I will update this blog post once we know the outcome.

Update - 27 September 2019

The above update hook has been run on production (to where Gluster is used). Feeds used to take upwards of 30 minutes to run there (even if there was no new files to process). Post the update hook running, it is now under 1 minute. We were also able to remove the wait_timeout setting as well. So this is a nice result.

PHP 7.3.0 was released in December 2018, and brings with it a number of improvements in both performance and the language. As always with Drupal you need to strike a balance between adopting these new improvements early and running into issues that are not yet fixed by the community.

Why upgrade PHP to 7.3 over 7.2?

• It is around 10% faster compared to PHP 7.2 - some basic benchmarks for Drupal on https://kinsta.com/blog/php-benchmarks/#drupal-benchmarks
• A bunch of quality of life improvements to the language - e.g. flexible heredoc and nowdoc syntaxes, allowing a trailing comma in function calls and better JSON parsing error messages (just to name a few). I would recommend reading this great blog post on the topic if you want to know more.

What hosting providers support PHP 7.3?

All the major players have support, here is how you configure it for each.

Acquia

Somewhere around April 2019 the option to choose PHP 7.3 was released. You can opt into this version by changing a value in Acquia Cloud. This can be done on a per environment basis.

Pantheon

Pantheon have had support since the April 2019 as well (see the announcement post). To change the version, you update your pantheon.yml file (see the docs on this).

# Put overrides to your pantheon.upstream.yml file here.
# For more information, see: https://pantheon.io/docs/pantheon-yml/
api_version: 1
php_version: 7.3

On a side note, it is interesting that PHP 5.3 is still offered on Pantheon (end of life for nearly 5 years).

Platform.sh

Unsure when Platform.sh released PHP 7.3, but the process to enable it is very similar to Pantheon, you update your .platform.app.yaml file (see the docs on this).

# .platform.app.yaml
type: "php:7.3"

Dreamhost

PHP 7.3 is also available on Dreamhost, and can be chosen in a dropdown in their UI (see the docs on this).

Dreamhost also win some award for also allowing the oldest version of PHP that I have seen in a while (PHP 5.2).

When can you upgrade PHP 7.3

Drupal 8

As of Drupal 8.6.4 (6^th December 2018), PHP 7.3 is fully supported in Drupal core (change record). I have been running PHP 7.3 with Drupal 8 for a while now and have seen no issues, and this includes running some complex installation profiles such as Thunder and Lightning.

Any Drupal 8 site that is reasonably up to date, should be fine with PHP 7.3.

Drupal 7

Slated for support in the next release of Drupal 7 - being Drupal 7.68 (see the drupal.org issue), however there are a number of related tasks that seem like deal breakers. There also is not PHP 7.3 and Drupal 7 tests running daily either.

It seems like for the mean time, it is probably best to hold off on the PHP 7.3 upgrade until 7.68 is out the door, and also contributed modules have had a chance to upgrade and release a new stable release.

A simple search on Drupal.org yields the following modules that look like they may need work (more are certainly possible):

• composer_manager (issue)
• scald (issue) [now fixed and released]
• video (issue)
• search_api (issue) [now fixed and released]

Most of the issues seem to be related to this deprecation - Deprecate and remove continue targeting switch. If you know of any other modules that have issues, please let me know in the comments.

Drupal 6

For all you die hard Drupal 6 fans out there (I know a few large websites still running this), you are going to be in for a rough ride. There is a PHP 7 branch of the d6lts Github repo, so this is promising, however the last commit was September 2018, so this does not bode well for PHP 7.3 support. I also doubt contributed modules are going to be up to scratch (drupal.org does not even list D6 versions of modules anymore).

To test this theory, I audited the current 6.x-2.x branch of views

$ phpcs -p ~/projects/views --standard=PHPCompatibility --runtime-set testVersion 7.3
................................................W.W.WW.W....  60 / 261 (23%)
................................E........................... 120 / 261 (46%)
...................................................EE....... 180 / 261 (69%)
............................................................ 240 / 261 (92%)
.....................                                        261 / 261 (100%)

3 errors in views alone. The errors are show stoppers too

Function split() is deprecated since PHP 5.3 and removed since PHP 7.0; Use preg_split() instead

If this is the state of one of the most popular modules for Drupal 7, then I doubt the lesser known modules will be any better.

If you are serious about supporting Drupal 6, it would pay to get in contact with My Drop Wizard, as they at least at providing support for people looking to adopt PHP 7.

If you have an enterprise zone with Cloudflare, there is the ability to request the raw request logs using 'Cloudflare Logs' (formally called Enterprise Log Share or ELS for short).

Cloudflare logs comes in 2 flavours, "Log Push" (e.g. to an S3 bucket) and "Log Pull" (using the REST API). In this blog post I will be covering the REST API, as I find analyzing the data easier on my local laptop.

If you have Splunk or Sumologic (or similar), then likely Log Push will be better suited to you.

How to download your Cloudflare logs using the REST API

Step 1: Ensure Cloudflare Logs are enabled for your zone

This is a manual step, and it requires you to raise a ticket with Cloudflare in order to enable it. It would be super if there was an API endpoint to both read and write this feature flag, but alas, manual it is for now.

You can en-masse enable Cloudflare logs for many zones in the same ticket, so save some time and enable it for all your enterprise zones now would be advice. There is literally no downside to enabling it (assuming you don't store silly things like credit card numbers in the URL).

It is also important to note that the logs are only captured from the point you enable the service, they do not retroactively appear. So if you are experiencing issues, and you don't have Cloudflare Logs already enabled, you may have missed out on collecting critical data.

Step 2: Get your Cloudflare Global API key

You get this from your user profile in Cloudflare

Treat this API key like you would your password, keep it safe.

Step 3: Use your favourite tool or language to download the logs

Now that you have your email address and Global API key, you can start to use the Cloudflare REST API to retrieve the logs.

Here is an example to download 1 hour's worth of logs. There is an offset of 5 minutes in the past (to ensure you get a full 1 hours worth of logs, as there is a delay). This will work so long as the zone does not have loads of traffic (as there is a 1GB limit on the download).

ZONE_ID=XXXX
CLOUDFLARE_EMAIL=bob@example.com
CLOUDFLARE_KEY=XXXXXX
STARTDATE=$(($(date +%s)-3900))
ENDDATE=$((STARTDATE+3600))
FILENAME="/tmp/${ZONE_ID}-${STARTDATE}-${ENDDATE}.log"
FIELDS=$(curl -s -H "X-Auth-Email: ${CLOUDFLARE_EMAIL}" -H "X-Auth-Key: ${CLOUDFLARE_KEY}" "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/logs/received/fields" | jq '. | to_entries[] | .key' -r | paste -sd "," -)

curl -s \
  -H "X-Auth-Email: ${CLOUDFLARE_EMAIL}" \
  -H "X-Auth-Key: ${CLOUDFLARE_KEY}" \
  "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/logs/received?start=${STARTDATE}&end=${ENDDATE}&fields=${FIELDS}" \
  > "${FILENAME}" \
  && echo "Logs written to ${FILENAME}"

There is actually a lot of information in a single request, and a large JSON object is returned. You can get a sense of this below (dummy data has been substituted):

$ head -n1 ${FILENAME} | jq
{
  "CacheCacheStatus": "hit",
  "CacheResponseBytes": 89846,
  "CacheResponseStatus": 200,
  "CacheTieredFill": false,
  "ClientASN": 9304,
  "ClientCountry": "hk",
  "ClientDeviceType": "desktop",
  "ClientIP": "118.143.70.210",
  "ClientIPClass": "noRecord",
  "ClientRequestBytes": 1928,
  "ClientRequestHost": "www.example.com",
  "ClientRequestMethod": "GET",
  "ClientRequestPath": "/scripts/app.built.js",
  "ClientRequestProtocol": "HTTP/1.1",
  "ClientRequestReferer": "https://www.example.com/",
  "ClientRequestURI": "/scripts/app.built.js?puhl4d",
  "ClientRequestUserAgent": "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko",
  "ClientSSLCipher": "ECDHE-RSA-AES128-SHA",
  "ClientSSLProtocol": "TLSv1.2",
  "ClientSrcPort": 33177,
  "EdgeColoID": 23,
  "EdgeEndTimestamp": 1563271736447000000,
  "EdgePathingOp": "wl",
  "EdgePathingSrc": "macro",
  "EdgePathingStatus": "nr",
  "EdgeRateLimitAction": "",
  "EdgeRateLimitID": 0,
  "EdgeRequestHost": "www.example.com",
  "EdgeResponseBytes": 89194,
  "EdgeResponseCompressionRatio": 0,
  "EdgeResponseContentType": "application/javascript",
  "EdgeResponseStatus": 200,
  "EdgeServerIP": "",
  "EdgeStartTimestamp": 1563271736404000000,
  "FirewallMatchesActions": [],
  "FirewallMatchesSources": [],
  "FirewallMatchesRuleIDs": [],
  "OriginIP": "",
  "OriginResponseBytes": 0,
  "OriginResponseHTTPExpires": "",
  "OriginResponseHTTPLastModified": "",
  "OriginResponseStatus": 0,
  "OriginResponseTime": 0,
  "OriginSSLProtocol": "unknown",
  "ParentRayID": "00",
  "RayID": "4f732d708c26d1ee",
  "SecurityLevel": "med",
  "WAFAction": "unknown",
  "WAFFlags": "0",
  "WAFMatchedVar": "",
  "WAFProfile": "unknown",
  "WAFRuleID": "",
  "WAFRuleMessage": "",
  "WorkerCPUTime": 0,
  "WorkerStatus": "unknown",
  "WorkerSubrequest": false,
  "WorkerSubrequestCount": 0,
  "ZoneID": 12345
}

Analyse your Cloudflare logs

Now that you have the raw data, you should look to turn it into something you can make business decisions with.

Here are some simple analysis you can do with the jq tool (ensure you install this first if you have not already).

Top URIs

jq -r .ClientRequestURI ${FILENAME} | sort -n | uniq -c | sort -nr | head -n 3

3716 /scripts/app.built.js
1331 /images/sample.png
 642 /

Top user agents

jq -r .ClientRequestUserAgent ${FILENAME} | sort -n | uniq -c | sort -nr | head -n 3

1507 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
1364 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
1014 Mozilla/5.0 (iPhone; CPU iPhone OS 12_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.1 Mobile/15E148 Safari/604.1

Top HTTP 404s

Slightly more complex query, but certainly still readable.

jq 'select(.EdgeResponseStatus == 404) | "\(.ClientRequestHost)\(.ClientRequestURI)"' ${FILENAME} | sort -n | uniq -c | sort -nr | head -n 3

 197 "www.example.com/images/globalnav/example.gif"
 195 "www.example.com/images/excelmark200_blue_300dpi.png"
  49 "www.example.com/includes/fonts/318130/AFD64F04666D9047C.css"

Top IPs triggering the WAF

jq -r 'select(.WAFAction == "drop") | .ClientIP' ${FILENAME} | sort -n | uniq -c | sort -nr | head -n 3

   1 58.11.157.113
   1 18.212.21.164

There are other examples on Cloudflare's own documentation site if you wish to pursue this further. Mostly this is just a matter of knowing how to use jq.

Extra for experts

Using jq is fun for some basic analysis, at some point you will want something more comprehensive. Here are some of the more unique things I have done to help show off this data. This likely will involve using a programming language, and having some form of presentation output (e.g. HTML) from it.

HTML tables

Having the data in HTML makes it more presentable.

On a side note, I often find the offload of the CDN is one number that is the most critical for caching, that is missing from the Cloudflare UI. Here the offload of 64.38% indicates that Cloudflare has removed around two thirds of all requests from your origin platform. Given enough time, energy and traffic you can tune this number to be > 99.9%.

Integration with Highcharts

Highcharts is a Javascript powered graphing library. It supports zooming, and removing series by clicking on them. Fairly fancy, and great for graphing time based data. Here is 24 hours of data, broken down by minute (the Cloudflare UI does not allow this granularity).

Integration with Geckoboard

If you need something more realtime, then Geckoboard is a simple solution. Geckoboard supports custom datasets, and allows you to send arbitrary data to it. Here is a real dashboard for a high traffic event

Logstalgia

If you convert the JSON format into Apache format you can use this rather unique visualization. Logstalgia ends up producing a pong-like representation of the traffic and the paddles are the virtual hosts. Fun stuff to have on the TV on your office wall if you have one free. See the official site for more information on how to install and use this.

Comments

If you have done something unique with Cloudflare Logs (and are allowed to share it), please let me know in the comments.

I noticed that out of the box, Ghost comes with AMP support (as has done since 2016), however there is no built in way to in the administration UI to add Google Analytics to your AMP theme. This means you will be missing out on a portion of your mobile traffic in your analytics. Not great.

Fortunately, implementing this is relatively simple.

Download the starter AMP page template

You can grab Ghost's start AMP page template from their Github repository. Place amp.hbs into the root of your theme.

Add in the required Google Analytics code

There are 2 sections

In the <head> section, near the end of it, place the Javascript include:

    {{!-- Load amp-analytics --}}
    <script async custom-element="amp-analytics" src="https://cdn.ampproject.org/v0/amp-analytics-0.1.js"></script>

After the <body> tag is opened add the following gtag configuration section. You will need to replace the UA-XXXXXXX-X with your own Google Analytics property ID.

    {{!-- Configure analytics to use gtag --}}
    <amp-analytics type="gtag" data-credentials="include">
        <script type="application/json">
            {
                "vars" : {
                    "gtag_id": "UA-XXXXXXX-X",
                    "config" : {
                        "UA-XXXXXXX-X": { "groups": "default" }
                    }
                }
            }
        </script>
    </amp-analytics>

You will then need to upload your theme to your Ghost blog, to which is covered nicely in this blog post.

Verify your AMP posts are valid

Option #1 - Use your browser's console

Visit any AMP post and add on to the end to the end of the URL #development=1 this will trigger the AMP validator to run. If you open the console of your browser, you should see something like this:

If you see AMP validation successful. then you know at least the theme is still valid.

Option #2 - Use a Chrome extension

Another option (if the console is not your friend) is to use the dedicated plugin for AMP validation. There is one for chrome that I am aware of.

Option #3 - Use a website

Likely the simplest option if you content has a valid public URL, visit https://search.google.com/test/amp and plug in your URL to your AMP post, and then see the results:

Verify Google Analytics is working

Next steps is verifying that the tracking is enabled, you can prove this in the Network tab of your browser's console and filtering by google-analytics, you should see a request to a URL that starts off like collect?v=1&_v=a1&ds=AMP&aip&_s=1...

You can also just go checkout the realtime view inside Google Analytics itself:

Further reading and extra for experts

For more advanced topics like tracking custom events, using a custom URL, please see the official documentation from Google on this.

I recent project required me to generate a CSR programmatically in PHP. There are lots of tutorials on the generation of CSR with just a Common Name (with no SANs). To my surprise, as soon as you add in a list of SANs, the process is much more complex, and the tutorials are quite thin.

In this post I hope to make this process as simple as possible for everyone, in the hopes it helps at least one other person.

Step 1 - Create a Distinguished Name

An SSL certificate contains your Distinguished Name information to help your users trust your certificate. You should replace the dummy data with your own

// The Distinguished Name to be used in the certificate.
$dn = [
  'commonName' => example.com,
  'organizationName' => 'ACME Inc',
  'organizationalUnitName' => 'IT',
  'localityName' => 'Seattle',
  'stateOrProvinceName' => 'Washington',
  'countryName' => 'US',
  'emailAddress' => 'foo@example.com',
];

Step 2 - Generate a new private key

Here is where you generate your private key. 4096 bits is used as this is stronger than the default of 2048 bits.

// Generates a new private key
$privateKey = openssl_pkey_new([
  'private_key_type' => OPENSSL_KEYTYPE_RSA,
  'private_key_bits' => 4096
]);

Step 3 - Generate OpenSSL config file

For some reason, OpenSSL in PHP requires you to create a file to supply arguments to add in SANs, they cannot be supplied in PHP.

[ req ]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[ req_distinguished_name ]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @san

[ san ]
{% for san in sans %}
DNS.{{ loop.index }} = {{ san }}
{% endfor %}

I render the above template using twig, and then create a file with the contents of the compiled template.

file_put_contents('/tmp/openssl.cnf', $contents);

If you did not want to use twig, the format of the [ san ] section down the bottom is very simple to replicate in raw PHP, or another templating language, after being compiled, it should look something like this (replace the SANs with your own):

[ san ]
DNS.1 = www.example.com
DNS.2 = shop.example.com
DNS.3 = foo.example.com
DNS.4 = *.foo.com

Essentially, a 1 indexed, newline separated, list of SANs you want to list on your certificate.

N.B. You should not include your Common Name in the SAN list.

Step 4 - Generate the CSR

Here is the magic that pulls in all the above code, and exports your CSR and private key into files on your filesystem.

$csrResource = openssl_csr_new($dn, $privateKey, [
  'digest_alg' => 'sha256',
  'config' => '/tmp/openssl.cnf',
]);

openssl_csr_export($csrResource, $csrString);
openssl_pkey_export($privateKey, $privateKeyString);

file_put_contents('/tmp/private.key', $privateKeyString);
file_put_contents('/tmp/public.csr', $csrString);

Final thoughts

This process seemed fairly complex for what I thought to be a simple process. I found no PHP libraries thought would help make this process a bit more Object Oriented. If you happen to know of a CSR generation PHP library let me know in the comments.

Part of my day job is to help tune the Cloudflare WAF for several customers. This blog post helps to summarise some of the default rules I will deploy to every Drupal (7 or 8) site as a base line.

The format of the custom WAF rules in this blog post are YAML format (for humans to read), if you do want to create these rules via the API, then you will need them in JSON format (see the end of this blog post for a sample API command).

Default custom WAF rules

Unfriendly Drupal 7 URLs

I often see bots trying to hit URLs like /?q=node/add and /?q=user/register. This is the default unfriendly URL to hit on Drupal 7 to see if user registration or someone has messed up the permissions table (and you can create content as an anonymous user). Needless to say, these requests are rubbish and add no value to your site, let's block them.

description: 'Drupal 7 Unfriendly URLs (bots)'
action: block
filter:
  expression: '(http.request.uri.query matches "q=user/register") or (http.request.uri.query matches "q=node/add")'

Autodiscover

If your organisation has bought Microsoft Exchange, then likely your site will receive loads of requests (GET and POST) to which is likely to just tie up resources on your application server serving these 404s. I am yet to meet anyone that actually serves back real responses from a Drupal site for Autodiscover URLs. Blocking is a win here.

description: Autodiscover
action: block
filter:
  expression: '(http.request.uri.path matches "/autodiscover\.xml$") or (http.request.uri.path matches "/autodiscover\.src/")'

Wordpress

Seeing as Wordpress has a huge market share (34% of all websites) a lot of Drupal sites get caught up in the mindless (and endless) crawling. These rules will effectively remove all of this traffic from your site.

description: 'Wordpress PHP scripts'
action: block
filter:
  expression: '(http.request.uri.path matches "/wp-.*\.php$")'

description: 'Wordpress common folders (excluding content)'
action: block
filter:
  expression: '(http.request.uri.path matches "/wp-(admin|includes|json)/")'

I separate wp-content into it's own rule as you may want to disable this rule if you are migrating from a old Wordpress site (and want to put in place redirects for instance).

description: 'Wordpress content folder'
action: block
filter:
  expression: '(http.request.uri.path matches "/wp-content/")'

SQLi

I have seen several instanced in the past where obvious SQLi was being attempted and the default WAF rules by Cloudflare were not intercepting them. This custom WAF rule is an attempt to fill in this gap.

description: 'SQLi in URL'
action: block
filter:
  expression: '(http.request.uri.path contains "select unhex") or (http.request.uri.path contains "select name_const") or (http.request.uri.path contains "unhex(hex(version()))") or (http.request.uri.path contains "union select") or (http.request.uri.path contains "select concat")'

Drupal 8 install script

Drupal 8's default install script will expose your major, minor and patch version of Drupal you are running. This is bad for a lot of reasons.

It is better to just remove these requests from your Drupal site altogether. Note, this is not a replacement for upgrading Drupal, it is just to make fingerprinting a little harder.

description: 'Install script'
action: block
filter:
  expression: '(http.request.uri.path eq "/core/install.php")'

Microsoft Office and Skype for Business

Microsoft sure is good at making lots of products that attempt to DoS its own customers websites. These requests are always POST requests, often to your homepage, and you require partial string matching to match the user agent, as it changes with the version of Office/Skype you are running.

In large organisation, I have seen the number of requests here number in the hundreds of thousands per day.

description: 'Microsoft Office/Skype for Business POST requests'
action: block
filter:
  expression: '(http.request.method eq "POST") and (http.user_agent matches "Microsoft Office" or http.user_agent matches "Skype for Business")'

Microsoft ActiveSync

Yet another Microsoft product that you don't why it is trying to hit another magic endpoint that doesn't exist.

description: 'Microsoft Active Sync'
action: block
filter:
  expression: '(http.request.uri.path eq "/Microsoft-Server-ActiveSync")'

Using the Cloudflare API to import custom WAF rules

It can be a pain to have to manually point and click a few hundred times per zone to import the above rules. Instead you would be better off to use the API. Here is a sample cURL command you can use do import all of the above rules in one easy go.

You will need to replace the redacted sections with your details.

curl 'https://api.cloudflare.com/client/v4/zones/XXXXXXXXXXXXXX/firewall/rules' \
  -H 'X-Auth-Email: XXXXXXXXXXXXXX' \
  -H 'X-Auth-Key: XXXXXXXXXXXXXX'
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json'
  -H 'Accept-Encoding: gzip'
  -X POST \
  -d '[{"ref":"","description":"Autodiscover","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/autodiscover\\.xml$\") or (http.request.uri.path matches \"\/autodiscover\\.src\/\")"}},{"ref":"","description":"Drupal 7 Unfriendly URLs (bots)","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.query matches \"q=user\/register\") or (http.request.uri.query matches \"q=node\/add\")"}},{"ref":"","description":"Install script","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path eq \"\/core\/install.php\")"}},{"ref":"","description":"Microsoft Active Sync","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path eq \"\/Microsoft-Server-ActiveSync\")"}},{"ref":"","description":"Microsoft Office\/Skype for Business POST requests","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.method eq \"POST\") and (http.user_agent matches \"Microsoft Office\" or http.user_agent matches \"Skype for Business\")"}},{"ref":"","description":"SQLi in URL","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path contains \"select unhex\") or (http.request.uri.path contains \"select name_const\") or (http.request.uri.path contains \"unhex(hex(version()))\") or (http.request.uri.path contains \"union select\") or (http.request.uri.path contains \"select concat\")"}},{"ref":"","description":"Wordpress common folders (excluding content)","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/wp-(admin|includes|json)\/\")"}},{"ref":"","description":"Wordpress content folder","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/wp-content\/\")"}},{"ref":"","description":"Wordpress PHP scripts","paused":false,"action":"block","priority":null,"filter":{"expression":"(http.request.uri.path matches \"\/wp-.*\\.php$\")"}}]'

How do you know the above rules are working

Visit the firewall overview tab in Cloudflare's UI to see how many requests are being intercepted by the above rules.

Final thoughts

The above custom WAF rules are likely not the only custom WAF rules you will need for any given Drupal site, but it should at least be a good start. Let me know in the comments if you have any custom WAF rules that you always deploy. I would be keen to update this blog post with additional rules from the community.

This is likely the first post in a series of blog posts on customising Cloudflare to suit your Drupal site. If you want to stay up to date - subscribe to the RSS feed, sign up for email updates, or follow us on Twitter.

PHP 7.4 is scheduled for release in November of this year, with it bring some performance improvements along with some new features. Here are a couple of the new features that are coming that I'm excited about.

Spread operator updates

PHP has had support for the spread operator for a while. It's functionality has been limited to unpacking functional arguments. For example. You've been able to do the following since PHP 5.6.

<?php
function spreadArgs(...$args) {
  print_r($args);
}

spreadArgs(...[1, 2, 3, 4]);
spreadArgs(1, 2, 3, 4);

/*
Array
(
    [0] => 1
    [1] => 2
    [2] => 3
    [3] => 4
)
*/

This new RFC will add spread functionality to the array expression. That means unpacking arrays inline like so

<?php
$vine_veges = ['cucumber', 'pumpkin'];
$ground_veges = ['carrots', 'potatos'];

print_r(['eggplant', ...$vine_veges, ...$ground_veges]);

/*
Array
(
    [0] => eggplant
    [1] => cucumber
    [2] => pumpkin
    [3] => carrots
    [4] => potatos
)
*/

This unpacking works with both array() and [] syntax. You can also unpack arrays returned directly from functions.

<?php
function get_colours($additional_colours = []) {
  return ['red', 'green', 'blue', ...$additional_colours];
}

print_r(['yellow', ...get_colours(['purple', 'green']), 'black']);

/*
Array
(
    [0] => yellow
    [1] => red
    [2] => green
    [3] => blue
    [4] => purple
    [5] => green
    [6] => black
)
*/

The unpacking syntax doesn't work with associative arrays. So it's not as flexible as it's Javascript counterpart.

Arrow functions

While PHP has supported closures for some time, they tend to be quite verbose. This RFC adds arrow functions and short function syntax to PHP. Take this code example from the RFC.

function array_values_from_keys($arr, $keys) {
    return array_map(function ($x) use ($arr) { return $arr[$x]; }, $keys);
}

With the new syntax this can be shortened to be

function array_values_from_keys($arr, $keys) {
    return array_map(fn($x) => $arr[$x], $keys);
}

Few things to notice there, there's a new shortened function fn operator, also the scoping has been simplified so that the variable $arr is in scope to the function without the need for the use statement.

I'm genuinely excited about this as I find this code.

/* inline */
array_filter(range(0, 1024), fn($b) => $b % 64 === 0);

/* variable function */
$factor = fn($number) => $number % 64 === 0;
array_filter(range(0, 1024), $factor);

A lot cleaner than these variants.

/* inline */
array_filter(range(0, 1024), function($b) { 
  return $b % 64 === 0;
});

/* variable function */
$factor = function($number) {
  return $number % 64 === 0;
}
array_filter(range(0, 1024), $factor);

Typed properties

PHP has supported types in some form or another for quite a while. Argument types have been a staple since version 5 and since version 7, PHP also supports return types. It'spretty standard to see class definitions like this.

class MyClass {

  /* int */
  protected $count = null;
  
  /* MyClass */
  protected $sibling;
  
  /**
   * @param $sibling MyClass
   * @return array
   */
  public function addSibling(MyClass $sibling): array
  {
  	$this->sibling = $sibling;
    return [
      'count' => $this->count++,
      'current_sibling' => $this->sibling
    ];
  }
}

With typed properties we'll also be able to define the types of the class properties.

class MyClass {

  protected ?int $count = null;
  
  protected MyClass $sibling;
  
  /* ... */
}

This improves code readability and will also help debugging and IDE support.

Currently the supported types are.

• int
• bool
• float
• string
• array
• object
• iterable
• self
• parent
• any class or interface name
• ?type where type may be any of the above.

The preceding ? tells the run-time interpreter that the property can be null.

There's a lot more to typed properties than I've covered here so go and have a read over at the RFC page.

Null Coalescing Assignment

PHP 7 introduced the Null Coalescing Operator as a shorthand for common usage of the ternary operator. As per the documentation:

The null coalescing operator (??) has been added as syntactic sugar for the common case of needing to use a ternary in conjunction with isset(). It returns its first operand if it exists and is not NULL; otherwise it returns its second operand.

<?php
// Fetches the value of $_GET['user'] and returns 'nobody'
// if it does not exist.
$username = $_GET['user'] ?? 'nobody';
// This is equivalent to:
$username = isset($_GET['user']) ? $_GET['user'] : 'nobody';

// Coalescing can be chained: this will return the first
// defined value out of $_GET['user'], $_POST['user'], and
// 'nobody'.
$username = $_GET['user'] ?? $_POST['user'] ?? 'nobody';
?>

The Null Coalescing Assignment Operator takes this a step further. Consider the case of defining defaults.

/* Standard Conditional */
function setName($name = null) {
  if (!$name) {
    $name = 'default';
  }
  /* ... */
}

/* Ternary (Elvis Operator) */
function setName($name = null) {
  $name = $name ?: 'default';
  /* ... */
}

/* Null Coalescing */
function setName($name = null) {
  $name = $name ?? 'default';
  /* ... */
}

/* Null Coalescing Assignment */
function setName($name = null) {
  $name ??= 'default';
  /* ... */
}

As you can see the syntax is a lot cleaner. Another benefit is that it's safe to undefined values.

<?php
$details = [[], ['category' => 'Red']];

/* Ternary/Elvis */
$details[0]['category'] = $details[0]['category'] ?: 'Blue';

// PHP Notice:  Undefined index: category in php shell code on line 1


/* Null Coalescing */
$details[0]['category'] = $details[0]['category'] ?? 'Blue';

// 'Blue'


/* Null Coalescing Assignment */
$details[0]['category'] ??= 'Blue';

// 'Blue'

One more

I'm not a massive fan of all the new features. Take, for example the Numeric Literal Separator.

This feature enables the ability to make numeric literals easier to read for developers. By adding an underscore separator to Numerical Literals. The underscore is removed during runtime so is ignored by the interpreter. It's completely optional.

// a billion!
(1000000000 === 1_000_000_000) // true

// scale is hundreds of millions
(107925284.88 === 107_925_284.88) // true

// $135, stored as cents
(13500 === 135_00) // true

While I appreciate that it makes the intention of the original developer easier to decode, this one seems a tad strange to me.

Conclusion

These are the features I'm most looking forward to, let me know in the comments what you think, or if there are other features you'd like to see in PHP in the future.

7.4 isn't due till November 28th 2019. In the meantime if you want to have a play with some of the new features you can use the interactive PHP interpreter via docker.

docker run -it php:7.4-rc-cli -a

Have fun.

This website www.pixelite.co.nz has gone through a few iterations over the years. I thought I would go through a few of the various CMSs and hosting providers we have used, and what went well, and what lessons we learned.

Drupal 7 (2012 -2015)

Pixelite started off as a Drupal 7 site, with a basic responsive theme applied over the top

What worked well

• Drupal is FOSS (GPLv2), and thus no licensing costs to get up and running
• Drupal modules meant that adding functionality was simple (e.g. Disqus comments, tag cloud)
• The theme was more or less out of the box (Arctica)

Lessons learnt

• We were terrible at keeping Drupal up to date. The blog ended up getting hacked by Drupalgeddon. This was less than ideal. Very much the 'mechanics car never gets fixed' type of situation.

We figured we didn't actually need a dynamic CMS at this point, and were wondering what other options were out there.

Jekyll (2015 - 2019)

Enter Jekyll, I guess the most popular static site generator around 4 years ago.

What worked well

• Jekyll is also FOSS (MIT license), and thus no licensing costs to get up and running
• Loads of free themes available, the one we ended up using was called clean-blog
• Markdown is a nice and easy way to write content, although often at the cost of spelling and grammar checking
• We were able to extend Jekyll to support authors (with author pages), tags, article types, RSS
• Hosting on Github (as a static site) using Github Pages which was entirely free
• The site was un-hackable really as a result

Lessons learnt

• At the time Github Pages did not support SSL (it does now)
• Ruby is not really our strong suit, so adding complexity in Ruby was not really a great long term move for us
• The customizations we had done to Jekyll meant that Github was no longer able to compile the site for us, and we had to push a compiled branch ourselves
• No real support for draft posts (it does now by the looks) this is handy, as often a well researched blog post will take a few stabs at to get right.

Ultimately the friction in creating content caused a massive drop in articles over the last 3 years. In order to compile the site you required a certain version of Ruby and certain gems installed. It was clear we needed a simpler solution.

Ghost (Pro) (2019 - now)

In June of this year I decided I wanted to shake things up, and move the content over to a new CMS, called Ghost.

Ghost struck me as a happy medium between having some flexibility to customise the theme, and yet still have the core platform looked after by experts.

What is working well

• Ghost is FOSS (MIT license), and thus no licensing costs to get up and running.
• If we wanted to move to self hosted Ghost in the future (I don't) you can export your posts and users as a giant JSON file.

• The editing interface for authors is great, it reminds me of Medium or Gutenberg
• Someone else looks after Ghost. I remember logging into the administration interface one day to discover Ghost 2.25 had been released, and things looked a little nicer, with no effort on my part.
• The post header images automatically get responsive versions created (e.g. for thumbnails etc)
• The default Casper theme looks pretty slick out of the box
• I was able to customise the theme very easily. For instance here is a quick post on how I added PrismJS to the default Casper theme
• Built in CDN with Cloudflare, very useful for a global audience
• SSL out of the box

What could be better

• You cannot extend content types or users with additional fields. The schema appears to be quite locked. If you example you wish to add a LinkedIn field on your user account, so you can display a link on your author page, then you are out of luck. [link to the feature request]
• The date picker on the post is some silly Javascript widget, rather than the standard HTML5 date picker. When we migrated a lot of our old content, do you know how many times you need to click the "back 1 month" button to get back to 2012? [link to the feature request]
• US date formats in the post settings [now fixed]
• Ghost Pro has restrictions around the number of authors you can have at any one time, this makes it very difficult to have guest blog posts by other contributors. You are forced to author in another platform and then migrate the content in.

Final thoughts

Pixelite is a work in progress and has been for a number of years. It is finally at a stage where it is fairly frictionless to create new content, and I am happy with the fact that the core platform is being looked after by people far more qualified to look after Ghost than I am.

Expect a lot of content to be coming out in the coming months. Subscribe to the RSS feed, sign up for email updates, or follow us on Twitter.

Rancher is an open-source self-hosted Kubernetes user interface. I'm going to show you how easy it is to get up and running with Rancher so that you can have a play. This will just be a single node install of Rancher so not recommended for production environments.

Provision server

The requirements are well documented on the Rancher site. We're going to  a small single node server, so we'll need a server with 4G of memory and at lease 1 VCPU.

For this example I'll also be running Ubuntu Server 18.04.

I'm going to use a Digital Ocean droplet, but you can use whatever provider you'd like so long as it meets the requirements.

Once that's done, if you want to have SSL that's not self signed. You can optionally point a DNS record at the boxes IP address.For this example I'm going to use rancher.craigpearson.co.nz.

Install docker

We need to install docker so log into the newly created server and run the following commands.

sudo apt-get update && sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get -y install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

This removes any previously installed versions of Docker and adds the official Docker apt repositories.

Start Rancher container

Now we've got Docker installed all we need to do is start up the rancher container. If you're happy with self signed SSL cert you can run the following.

docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher

If however you want to use Lets Encrypt and you've already set up a DNS record which is ready to go you can use the following to get Rancher to issue a SSL cert via the Lets Encrypt HTTP challenge. (Remember to change your --acme-domain).

sudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher:latest --acme-domain rancher.craigpearson.co.nz

Since this is for demo purposes and not meant for production there's no data persistence. If you'd like to persist the rancher data you can add a docker volume to the above command.

-v /opt/rancher:/var/lib/rancher

It should work away and download the latest image the output should look something like this.

Typing docker ps you should see there is now a rancher/rancher:latest container running.

Now at this point if you are considering running this in a publicly accessible manner I'd suggest taking a look at the Rancher hardening guide. Since this is just a simple tutorial, I'm not going to bother.

Configure and sign in

Now we're up and running. In a browser, navigate to your server's IP address or your configured DNS record and you should get something that looks like this.

Set an administrator password and confirm your URL on the next page. Then, you're all logged in.

Next steps

This is a quick and dirty install to show you the basics of Rancher. In the next post I'll talk about some of the key features of rancher.

I'm going to assume you have some basic knowledge of docker. I also assume you have installed docker and docker-compose locally. You can read more about docker here. I'm going to be doing this for an example Laravel site however this should work if you're using Drupal or some similar framework.

What we need

We're going to build a local stack for development, so we're going to need Apache, PHP, and MariaDB database.

Setup

I've generated an example project called mysite it's a typical composer based PHP site, with a public web root.

In the root of your project we're going to want to create some files and folder structures, run the following command.

mkdir -p docker/php-apache2 && touch docker-compose.yml docker/php-apache2/Dockerfile

This will create the following:

docker-compose.yml
docker
docker/php-apache2
docker/php-apache2/Dockerfile

The Web/PHP container

First up we'll get the web container configured as it's the only one we'll need a Dockerfile for. In your favourite editor open up docker/php-apache2/Dockerfile and paste in the following.

# Base image
FROM php:7.2-apache

# Fix debconf warnings upon build
ARG DEBIAN_FRONTEND=noninteractive

# Run apt update and install some dependancies needed for docker-php-ext
RUN apt update && apt install -y apt-utils sendmail mariadb-client pngquant unzip zip libpng-dev libmcrypt-dev git \
  curl libicu-dev libxml2-dev libssl-dev libcurl3 libcurl3-dev libsqlite3-dev libsqlite3-0

# Install PHP extensions
RUN docker-php-ext-install mysqli bcmath gd intl xml curl pdo_mysql pdo_sqlite hash zip dom session opcache

# Update web root to public
# See: https://hub.docker.com/_/php#changing-documentroot-or-other-apache-configuration
ENV APACHE_DOCUMENT_ROOT /var/www/html/public
RUN sed -ri -e 's!/var/www/html!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's!/var/www/!${APACHE_DOCUMENT_ROOT}!g' /etc/apache2/apache2.conf /etc/apache2/conf-available/*.conf

# Enable mod_rewrite
RUN a2enmod rewrite

There's a lot going on in there however, the key points to know are, we're using the official PHP-Apache2 docker images as a base (found here), we also install some other packages needed for enabling some PHP extensions.

We're not going to customise the database container so we're ready to put it all together.

...and the rest

In your editor of choice open the docker-compose.yml file we created earlier. Paste the following into it.

version: "3.1"
services:
  database:
    image: mariadb:10.1
    container_name: mysite-mariadb
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - MYSQL_DATABASE=mysite
      - MYSQL_USER=mysite
      - MYSQL_PASSWORD=password
    ports:
      - "8083:3306"
  web:
    build: docker/php-apache2
    container_name: mysite-web
    volumes:
      - .:/var/www/html
    ports:
      - "8080:80"

This uses the off-the-shelf MariaDB docker image and specifies the build directory for the web container. docker/php-apache2. The environment section in the database service specifies the credentials we want MariaDB to start up with.

Lets start it up

Now we've put it all together, lets start up our stack and see what we have. Run the following to build and start up the containers.

docker-compose up -d

It might take a while to build, especially if you have to download the base images but ones completed you should be able to run docker ps and see something like the following.

So we now have an Apache2 webserver running mod_php listening on http://localhost:8080 if you navigate there now you'll likely get an error, cause we still have to tell Laravel about the correct database settings. Open up the .env file in your editor and update the database settings to read:

DB_CONNECTION=mysql
DB_HOST=mysite-mariadb
DB_PORT=3306
DB_DATABASE=mysite
DB_USERNAME=mysite
DB_PASSWORD=password

These are the environment values provided to the MariaDB container in the docker-compose.yml file. Now if we navigate to http://localhost:8080 we should see the default Laravel homepage.

Some final thoughts

Having this setup makes it pretty easy to update software versions like PHP and MariaDB. Try it. Update the version of MariaDB in the docker-compose.yml to 10.2 and rebuilding. Or even changing the base image in the Dockerfile from php:7.2-apache to php:7.3-apache. It's nice not having to upgrade distributions or VirtualBox images in order to see if your site will work on a newer software. It also makes adding things like Redis, Memcache, Mailhog etc a lot easier.

Now this is a pretty cut back version of what I'd use normally as I wanted to keep this post as simple as possible. Normally in my PHP/Apache Dockerfile I'd install some extra things I find useful for development like Xdebug, PHPUnit, NodeJS, Composer etc. Let me know in the comments if you have any questions.

Out of the box Ghost will not syntax highlight your code snippets. In this blog post I will explain the changes I made to the Casper theme, and how you can look to implement these in your own Ghost site.

There are 2 options, and which option you use is up to you. The code snippets are the same, it is only the method of embedding the snippets that change. Only second option can be version controlled (which I prefer).

Option #1 - Use Code Injection

This is likely the simplest way if you do not have your theme in code, or run Ghost pro, and don't really care about code at all.

You simply add the following code to your Site Header in your blog settings:

    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/themes/prism-okaidia.min.css" integrity="sha256-Ykz0nNWK7w4QWJUYR7OraN4773aMB/11aMt1nZyrhuQ=" crossorigin="anonymous" />

    <style type="text/css" media="screen">
        .post-full-content pre strong {
            color: white;
        }
        .post-full-content pre {
            line-height: 1;
        }
        .post-full-content pre code {
            white-space: pre-wrap;
            hyphens: auto;
            line-height: 0.7;
            font-size: 0.7em;
        }
    </style>

I have also added some extra CSS to reduce the size of PrismJS code blocks within the Casper theme in Ghost (as it can be quite large), and also allow long strings to wrap to newlines (to avoid horizontal scrolling).

In Site Footer you add something like:

    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/prism.min.js" integrity="sha256-NFZVyNmS1YlmiklazBA+TALYJlJtZj/y/i/oADk6CVE=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-markup-templating.min.js" integrity="sha256-41PtHfb57czcvRtAYtUhYcSaLDZ3ahSDmVZarE0uWPo=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-javascript.min.js" integrity="sha256-KxieZ8/m0L2wDwOE1+F76U3TMFw4wc55EzHvzTC6Ej8=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-css.min.js" integrity="sha256-49Y45o2obU1Yv4zkYDpMDyAa+D9sgKNbNy4ZYGRl/ls=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-php.min.js" integrity="sha256-gJj4RKQeXyXlVFu2I8jQACQZsii/YzVMhcDT99lr45I=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-sql.min.js" integrity="sha256-zgHnuWPEbzVKrT72LUtMObJgbwkv0VESwRfz7jpdsq0=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-yaml.min.js" integrity="sha256-JoqiKM2GipZjbGjNyl62d6qjQY1F9QTLriWOe4N76wE=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-sass.min.js" integrity="sha256-3oigyyaPovKMS9Ktg4ahAD1R6fOSMGASuA03DT8IrvU=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-json.min.js" integrity="sha256-18m89UBQcWGjPHHo64UD+sQx4SpMxiRI1F0MbefKXWw=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-bash.min.js" integrity="sha256-0W9ddRPtgrjvZVUxGhU/ShLxFi3WGNV2T7A7bBTuDWo=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-python.min.js" integrity="sha256-zXSwQE9cCZ8HHjjOoy6sDGyl5/3i2VFAxU8XxJWfhC0=" crossorigin="anonymous"></script>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/prism/1.16.0/components/prism-ruby.min.js" integrity="sha256-SGBXZakPP3Fv0P4U6jksuwZQU5FlC22ZAANstHSSp3k=" crossorigin="anonymous"></script>

The only real gotcha here is that you must add prism.min.js first, then prism-markup-templating.min.js followed by any and all languages you want to syntax highlight for.

If you want to find the links to more libraries, or newer versions of the above, check out https://cdnjs.com/libraries/prism.

Option #2 - Edit the theme templates

This is just like the above, except you edit the template file default.hbs, and insert the code in the appropriate sections. The end result is the same, except this method allows you to version control the theme (which I prefer).

Final thoughts

The above are the libraries that we run on pixelite, but your needs might be different (i.e. you may need different languages etc). This should serve as a head start for any other people looking to share code snippets, and have them look great on your Ghost site.

Please let me know if you have any other pro tips that I may have missed.

I am working with a customer now that is looking to go through a JSON:API upgrade, from version 1.x on Drupal 8.6.x to 2.x and then ultimately to Drupal 8.7.x (where it is bundled into core).

As this upgrade will involve many moving parts, and it is critical to not break any existing integrations (e.g. mobile applications etc), having basic end-to-end tests over the API endpoints is essential.

In the past I have written a lot about CasperJS, and since then a number of more modern frameworks have emerged for end-to-end testing. For the last year or so, I have been involved with Cypress.

I won't go too much in depth about Cypress in this blog post (I will likely post more in the coming months), instead I want to focus specifically on JSON:API testing using Cypress.

In this basic test, I just wanted to hit some known valid endpoints, and ensure the response was roughly OK.

Rather than have to rinse and repeat a lot of boiler plate code for every API end point, I wrote a custom Cypress command, to which abstracts all of this away in a convenient function.

Below is what the spec file looks like (the test definition), it is very clean, and is mostly just the JSON:API paths.

describe('JSON:API tests.', () => {

    it('Agents JSON:API tests.', () => {
        cy.expectValidJsonWithMinimumLength('/jsonapi/node/agent?_format=json&include=field_agent_containers,field_agent_containers.field_cont_storage_conditions&page[limit]=18', 6);
        cy.expectValidJsonWithMinimumLength('/jsonapi/node/agent?_format=json&include=field_agent_containers,field_agent_containers.field_cont_storage_conditions&page[limit]=18&page[offset]=72', 0);
    });
    
    it('Episodes JSON:API tests.', () => {
        cy.expectValidJsonWithMinimumLength('/jsonapi/node/episode?fields[file--file]=uri,url&filter[field_episode_podcast.nid][value]=4976&include=field_episode_podcast,field_episode_audio,field_episode_audio.field_media_audio_file,field_episode_audio.thumbnail,field_image,field_image.image', 6);
    });

});

And as for the custom function implementation, it is fairly straight forward. Basic tests are done like:

• Ensure the response is an HTTP 200
• Ensure the content-type is valid for JSON:API
• Ensure there is a response body and it is valid JSON
• Enforce a minimum number of entities you expect to be returned
• Check for certain properties in those returned entities.

Cypress.Commands.add('expectValidJsonWithMinimumLength', (url, length) => {
    return cy.request({
        method: 'GET',
        url: url,
        followRedirect: false,
        headers: {
            'accept': 'application/json'
        }
    })
    .then((response) => {
        // Parse JSON the body.
        let body = JSON.parse(response.body);
        expect(response.status).to.eq(200);
        expect(response.headers['content-type']).to.eq('application/vnd.api+json');
        cy.log(body);
        expect(response.body).to.not.be.null;
        expect(body.data).to.have.length.of.at.least(length);

        // Ensure certain properties are present.
        body.data.forEach(function (item) {
            expect(item).to.have.all.keys('type', 'id', 'attributes', 'relationships', 'links');
            ['changed', 'created', 'default_langcode', 'langcode', 'moderation_state', 'nid', 'path', 'promote', 'revision_log', 'revision_timestamp', 'status', 'sticky', 'title', 'uuid', 'vid'].forEach((key) => {
                expect(item['attributes']).to.have.property(key);
            });
        });
    });

});

Some of the neat things in this function is that it does log the parsed JSON response with cy.log(body); this allows you to inspect the response in Chrome. This allows you to extend the test function rather easily to meet you own needs (as you can see the full entity properties and fields.

Using Cypress is like having an extra pair of eyes on the Drupal upgrade. Over time Cypress will end up saving us a lot of developer time (and therefore money). The tests will be in place forever, and so regressions can be spotted much sooner (ideally in local development) and therefore fixed much faster.

Comments

If you do JSON:API testing with Cypress I would be keen to know if you have any tips and tricks.