Integrating with AMP Dev Mode in WordPress

tl;dr In v1.3 the AMP plugin for WordPress no longer has to remove the Admin Bar to keep pages valid AMP.

The AMP plugin allows WordPress themes to be developed as usual and have their templates and stylesheets used to serve valid AMP pages. It does as much as possible to prevent serving invalid AMP pages, no matter what WordPress is outputting. One standard component of WordPress pages is the Admin Bar (aka Toolbar) which appears on the frontend once a user has logged-in to the Admin. The Admin Bar provides tools for administering a site, including links to create new posts, moderate comments, and access the Customizer. This Admin Bar has been a challenge for the AMP plugin to accommodate, but this is now changing.

When the AMP plugin first introduced theme support in v0.7, the Admin Bar was disabled entirely on AMP responses because of the 20KB+ of CSS that it adds to the page. In v1.0 the plugin restored the Admin Bar on AMP pages thanks in part to the CSS tree shaker, but still a checkbox was needed to turn off the Admin Bar when not enough CSS could be removed. In v1.2 the plugin was enhanced to eliminate this checkbox by automatically removing the Admin Bar when its CSS was too much for the 50KB limit on the page. As part of this, we had to fork core’s admin-bar.css to make it work with JavaScript turned off. Nevertheless, plugins also extend the Admin Bar with functionality which often requires JavaScript:

These Admin Bar integrations would be largely broken on AMP pages or at least limited in functionality (as if JS is turned off in the browser, since the plugin removes custom scripts).

In addition to the Admin Bar being important for normal site administration, it is also vital in an AMP context because the plugin adds an Admin Bar menu item to show the validation status, letting users know if markup was removed through sanitization to make the page valid AMP:

Admin Bar on AMP page that has invalid markup being sanitized.

So the Admin Bar is important. But again, it would unfortunately often get removed due to excessive CSS, and even when it was included it could be broken or functionally limited due to plugins’ JS being sanitized out of the page. Not great!

All of this is about to change with the release of the AMP plugin v1.3 (now available as release candidate).

Introducing AMP Dev Mode

Last month I was going through the recurring task of updating the plugin to the latest version of the AMP Validator specification. I noticed something unusual at the very end of a diff of the protoascii files:

@@ -6847,6 +6874,14 @@ error_specificity {
   code: DOCUMENT_SIZE_LIMIT_EXCEEDED
   specificity: 120
 }
+error_specificity {
+  code: DEV_MODE_ONLY
+  # This should always trump any other error. It is asserting
+  # that the developer intends for this tag to be an error but
+  # that no additional errors should be reported for this tag.
+  specificity: 1000
+}
+
 # Error formats
 error_formats {
   code: UNKNOWN_CODE
@@ -7311,3 +7346,8 @@ error_formats {
   code: DOCUMENT_SIZE_LIMIT_EXCEEDED
   format: "Document exceeded %1 bytes limit. Actual size %2 bytes."
 }
+error_formats {
+  code: DEV_MODE_ONLY
+  format: "Tag 'html' marked with attribute 'ampdevmode'. Validator "
+          "will suppress errors regarding any other tag with this attribute."
+}

What is this “AMP Dev Mode”? I searched through the issues in the AMP project and I found it introduced in amphtml#20974:

Allow non-AMP script tags in the JS validator if the [data-ampdevmode] attribute is present on the script tag.

This is for development environments that may want to run additional code (e.g. to do hot-reloading on code changes) while otherwise being compatible with AMP tools that do validation.

The scope for this Dev Mode expanded beyond just non-AMP script tags to apply to any element that is not valid in AMP! When the root html element has a data-ampdevmode attribute, any AMP validation errors that would normally be reported on an invalid element (or its attributes) will be suppressed if the element also has a data-ampdevmode attribute. This is exactly what we have needed for the Admin Bar.

In #3084/#3187 the AMP plugin added support for Dev Mode. The plugin’s sanitizers were updated to skip processing any elements that have the data-ampdevmode attribute, if the data-ampdevmode attribute is also at the root. The plugin adds this attribute to the link element for the admin-bar.css stylesheet which results in it not being added to style[amp-custom] and thus no longer counting against the 50KB limit for custom CSS. Similarly, the admin-bar.js script also gets this same attribute, preventing it from being removed by the sanitizers. Lastly, every element under the div#wpadminbar element DOM tree also gets this data-ampdevmode attribute; this prevents a user’s Gravatar img from being flagged as a validation error, for example. So, by adding these attributes WordPress core’s Admin Bar can be passed through without anything special done to it.

Note that an AMP page in Dev Mode is not a valid AMP page, as it’s explicitly not intending to be. The html element will only get the data-ampdevmode attribute added to it if the user is authenticated into WordPress and the Admin Bar is showing (source). This authentication requirement ensures that a crawler (like Googlebot) never encounters an AMP page in Dev Mode, thus ensuring the pages will be available on an AMP Cache and you won’t get Google Search Console complaining about AMP validation errors. There is an amp_dev_mode_enabled filter allowing you to override this. The amp_is_dev_mode() function can be used to determine if the current page is in Dev Mode, though if you’re adding code for the Admin Bar you can just assume this is the case, and add the data-ampdevmode attribute (which is just ignored if the page as a whole is not in Dev Mode).

When a page is in Dev Mode, you will currently see one single AMP validation error being reported by the AMP Validator Extension; opening the extension will show the following Dev Mode validation error:

Soon the extension will be updated to replace this error badge with something indicating Dev Mode. In any case, the AMP menu item in the Admin Bar indicates () that there is no validation problems, that is, there are no unaccepted validation error sanitizations. If the AMP plugin’s sanitizers do happen to leak through a validation error, then you’ll still see a validation error count greater than one in the AMP Validator extension. Again, only elements which explicitly have the data-ampdevmode attribute can have their validation errors ignored by the AMP Validator.

How to Integrate with Dev Mode

There are a few ways that plugins can integrate with Dev Mode. First of all, if all of the plugin’s added markup is located inside the Admin Bar menu item, nothing has to be done because all descendant elements of div#wpadminbar will get the required Dev Mode attributes, as noted above.

Secondly, the AMP plugin will automatically add data-ampdevmode attributes to any enqueued stylesheets that have the admin-bar stylesheet as a dependency (recursively). This is all that was required for Yoast SEO. For example:

wp_enqueue_style(
	'my-admin-bar',
	plugin_dir_url( __FILE__ ) . 'admin-bar.css',
	array( 'admin-bar' ), // 👈👈👈
	'1.0'
);

Similarly, if a plugin adds inline styles for the Admin Bar, all it have to do is use the wp_add_inline_style() for the admin-bar stylesheet, as was done in Pantheon HUD. For example:

wp_add_inline_style( 'admin-bar', 'wp-admin-bar-foo { /* ... */ }' );

Similarly to enqueued styles, an enqueued script can also be marked for Dev Mode automatically by just registering it with a dependency on the admin-bar script in core:

wp_enqueue_script(
	'my-admin-bar',
	plugin_dir_url( __FILE__ ) . 'admin-bar.js',
	array( 'admin-bar' ), // 👈👈👈
	'1.0'
);

The AMP plugin recursively checks registered scripts and styles for a dependency on admin-bar and injects the data-ampdevmode attribute via the script_loader_tag and style_loader_tag filters, respectively. Using these filters is useful if you need to enqueue styles/scripts that don’t depend on admin-bar, for example see a pull request for Site Kit which uses these filters to mark react and wp-api-fetch for Dev Mode. This can also be seen in a Jetpack pull request to ensure mustache, backbone, and other scripts are included in Dev Mode; this is done using a script_loader_tag filter like this:

$script_handles = array( /* Handles for Dev Mode */ );
add_filter(
	'script_loader_tag',
	function ( $tag, $handle ) use ( $script_handles ) {
		if ( in_array( $handle, $script_handles, true ) ) {
			$tag = preg_replace(
				'/(?<=<script)(?=\s|>)/i',
				' data-ampdevmode',
				$tag
			);
		}
		return $tag;
	},
	10,
	2
);

Now, you can’t always easily add this data-ampdevmode attribute to the elements being added to the page. For example, when you call wp_localize_script() or wp_add_inline_script() there’s no way to filter the script tag that is printed. For such cases, there is also an amp_dev_mode_element_xpaths filter which allows you to provide XPath expressions to query the elements that you want to add the attribute to. While waiting for plugins to release direct support for AMP Dev Mode, it is straightforward to add the attribute to the required elements, as someone needed for Jetpack and Yoast SEO stylesheets (which won’t be necessary soon):

add_filter(
	'amp_dev_mode_element_xpaths',
	function ( $xpaths ) {
		$ids = array(
			'yoast-seo-adminbar-css',
			'wpcom-notes-admin-bar-css',
			'noticons-css',
			// Add more element IDs as desired.
		);
		foreach ( $ids as $id ) {
			$xpaths[] = sprintf( '//*[ @id = "%s" ]', $id );
		}
		return $xpaths;
	}
);

Similarly, in the Site Kit pull request, each of the added inline scripts get the string “googlesitekit” included. This allows all such inline scripts (added via wp_add_inline_script()) to be targeted with XPath via:

add_filter(
	'amp_dev_mode_element_xpaths',
	function ( $xpaths ) {
		$xpaths[] = '//script[ contains( text(), "googlesitekit" ) ]';
		return $xpaths;
	}
);

So there are several ways to add this data-ampdevmode attribute to elements on pages generated by WordPress and the AMP plugin.

Query Monitor on AMP Pages

Going back to v0.7 when the AMP plugin was suppressing the Admin Bar entirely, I proposed a pull request for the excellent Query Monitor plugin to prevent it from enqueueing the scripts and styles which would break AMP pages. I ultimately withdrew the PR from consideration because disabling Query Monitor functionality on AMP pages was fixing a symptom of broken AMP validity but not the underlying problem of being able to include its important functionality on AMP pages. Now with AMP Dev Mode, this can now be easily achieved! I’ve written a simple plugin called AMP Query Monitor Compat which adds data-ampdevmode to the scripts and styles that Query Monitor depends on, and behold the result:

Query Monitor interface shown on an AMP page in Dev Mode.

Perhaps this code would make sense to be merged directly into Query Monitor.

Conclusion: AMP Hybrid Documents

One reason why I’m excited about AMP Dev Mode is that it’s an example of a hybrid document. As described in AMP as your framework, work is underway on a initiative called “Bento AMP” which intends to allow you to freely use AMP components standalone, outside the context of pages being managed by the AMP framework. AMP Dev Mode is an example of the inverse: a framework-official way of using non-valid markup in AMP pages. While this was technically possible before, you either had to remove the amp attribute from the html element or else live with the AMP Validator inundating you with validation errors (and wrestle with the AMP plugin’s sanitizers by rejecting sanitization for those errors). Now that AMP Dev Mode is a thing, the framework has a built-in way to describe these hybrid documents and still get benefits from the AMP Validator to catch problems with markup not explicitly marked as being part of Dev Mode.

Give it a try by testing 1.3-RC2!

AMP for JavaScripters

Today at the JavaScript for WordPress Conference (#JSforWPConf), Felix Arntz and I gave a talk called AMP for JavaScripters about implementing interactive interfaces in AMP. Here’s the abstract we submitted for the talk:

As we all know, adding JavaScript to a web page allows for dynamic page modifications. However, with that flexibility comes great responsibility: When used excessively or independently of user interaction, JavaScript can seriously hurt performance and UX.

AMP has a somewhat curious relationship with JavaScript. Due to its restrictions on custom scripting, AMP may be disdained by JavaScript developers. In spite of this, AMP itself is an HTML framework written in JavaScript and powered by Web Components, so in no way does AMP consider JavaScript to be inherently bad.

The problem is that JavaScript is extensively abused on the web today, harming user experience in ways such as blocking page rendering, creating janky experiences, and reducing battery life. So AMP encourages developers to focus back on writing markup, using the large library of performant web components that allow pages to be created declaratively. Custom dynamic functionality is made possible through dedicated components and the usage of AMP actions/events and amp-bind.

This session dives into the paradigms for developing highly dynamic interfaces with AMP, while providing guidance to AMP’s approaches compared to traditional JavaScript. It also looks at the upcoming amp-script component that will allow developers to implement even the most custom interactions in a performant and developer-friendly way.

Here’s a recording of the talk on YouTube, also available on Crowdcast (requires login):

Also available are the slides:

The code examples are available on GitHub.

To learn more about AMP in general, please check out amp.dev as well as the AMP YouTube channel. For more about the official AMP WordPress plugin, see amp-wp.org.

To get an introduction to amp-bind, see this talk by William Chou at AMPConf 2017:

Also, to learn about amp-script, check out Kristofer Baxter‘s talk at AMPConf 2019:

Using the AMP Plugin to Protect Site Visitors and Debug Security Vulnerabilities

Recently I’ve been testing compatibility for all of Jetpack‘s various widgets when used on pages served by the AMP plugin. In the process I ran across a security vulnerability in Jetpack (which I responsibly disclosed and is now fixed), but I never would have noticed the issue if it weren’t for the AMP plugin’s internal validator.

As you may be aware, AMP is both a subset and superset of HTML. The standard HTML elements which can have problems with performance and privacy are not allowed in AMP. At the same time, AMP is also a web components library which provides custom elements that implement performance best practices and support privacy-preserving prerendering. All of the elements and attributes that AMP allows are codified in a specification which is used to programmatically validate AMP pages. Valid AMP pages can be distributed via an AMP Cache and safely prerendered to a user (e.g. in search results).

The AMP plugin internalizes the AMP specification and it uses the spec to catch invalid AMP markup to prevent it from leaking out onto the frontend. The plugin does its best to ensure your site serves valid AMP pages, not only so that Google Search Console doesn’t complain about AMP validation errors, but also in order to give you immediate feedback without having to wait for Googlebot to crawl your site. In contrast to the plugin’s Classic mode, the plugin no longer silently sanitizes the invalid AMP markup when in the Paired/Native modes; you can now be informed of what markup it is removing. This is particularly important when you have a site running ads or analytics, as you need to be alerted when the related script tags are getting stripped out (as AMP doesn’t allow custom scripts, at least not quite yet, though never like this).

So, back to the Jetpack plugin. When I tested the My Community widget, I noticed some strange new AMP validation errors reported by the AMP plugin, including unrecognized attributes: bencowboy, and alman:

New validation errors appearing after adding Jetpack’s My Community widget.

The AMP plugin’s validator stripped out these invalid attributes—being “accepted” for sanitization—so they would not have shown up on the frontend of the site. But where did they come from? Here also the AMP plugin provides a key tool. As shown above, the plugin already identified that Jetpack was the source of the errors. Then by expanding a validation error, the full context for the error including its source information is provided:

Details for an AMP validation error as provided by the plugin’s internal validator.

Here it is clear that the invalid markup is coming from that My Community widget in Jetpack, as can be seen in the source function (Jetpack_My_Community_Widget::display_callback). When I looked at the widget output in a non-AMP version of the page, the issue became clear:

<li>
    <a 
        href="https://en.gravatar.com/978a1a2a80394217a0e39c84f07a7c16" 
        title=""Cowboy" Ben Alman"
    >
        <img alt="" src="https://0.gravatar.com/avatar/978a1a2a80394217a0e39c84f07a7c16?s=96&d=https%3A%2F%2F0.gravatar.com%2Favatar%2Fad516503a11cd5ca435acc9bb6523536%3Fs%3D96&r=G" class="avatar avatar-240" height="48" width="48" originals="240" scale="1" />
    </a>
</li>

The problem was title=""Cowboy" Ben Alman". The title attribute (among other attributes) was not being escaped with esc_attr() in Jetpack_My_Community_Widget::get_community():

foreach ( $members as $member ) {
    $my_community .= sprintf(
        '<li><a href="%s" %s><img alt="" src="%s" class="avatar avatar-240" height="48" width="48" originals="240" scale="1" /></a></li>',
        $member->profile_URL,
        empty( $member->name ) ? '' : 'title="' . $member->name . '"',
        $member->avatar_URL
    );
}

This is a persistent cross-site scripting vulnerability (stored XSS). An attacker could have exploited the vulnerability to run arbitrary JavaScript on a site that uses the My Community widget. All they’d have to do to exploit it is change their account “name” to something like:

John Smith"><script>doSomething("EVIL")</script><a class="

Then since the widget lists users who have recently interacted with the site, the attacker would just have to leave a comment and then wait 10 minutes for the transient to flush. At this point the malicious doSomething('evil') would run for every visitor to the site.

I responsibly disclosed this Jetpack security vulnerability to Automattic’s HackerOne, and I got approval to blog about the find. Many thanks to the Jetpack team for being so responsive and including the fix in a release so quickly.

Remember: Never trust external input. Always validate/sanitize all inputs early and escape all output late.

However, this vulnerability would not have been exploitable on an AMP-first site. In the plugin’s native mode there is no non-AMP version of the site (no paired AMP). The AMP plugin removes all custom script (including script tags and on-event handler attributes), so on a fully AMP site the AMP plugin would have prevented this stored XSS vulnerability from being exploited. Furthermore, the AMP plugin also informs the site owner of such invalid markup being removed and where it came from in the first place.

So the AMP plugin is useful for protecting visitors to your site, as well as providing you with tools for finding and debugging security vulnerabilities. To learn more about the plugin, check out amp-wp.org.

Creating Gutenberg Blocks without a Build Step via HTM

If you’ve ever looked into developing a block for the new WordPress editor (Gutenberg), you’ve seen that it’s recommended to code it up with JSX. Blocks are powered by React and the JSX syntax is significantly more readable and less verbose than the ES5-compatible syntax. For example, compare this ES5 code:

function save( props ) {
    return wp.element.createElement(
        'div',
        {
            id: props.attributes.id 
        },
        props.attributes.content
    );
}

With this equivalent in JSX:

function save( { attributes } ) {
    return <div id={ attributes.id }>
        { attributes.content }
    </div>;
}

The difference is clear. However, a major downside to JSX is that it requires a build step. As the Gutenberg handbook states regarding JavaScript versions and build step:

Additionally, the ESNext code examples in the Gutenberg handbook include JSX syntax, a syntax that blends HTML and JavaScript. It makes it easier to read and write markup code, but likewise requires the build step using webpack and babel to transform into compatible code.

I believe this build step is one of the biggest sources of hesitation toward Gutenberg by the WordPress developer community. WordPress developers have historically developed PHP and JS without any build step, and this has made developing themes and plugins very accessible to newcomers. Without a build step you just edit a file and immediately reload the browser to see the changes; you can also locate and understand the source code more easily since it is not compiled and no source maps are needed. All of the tooling around JavaScript is also very intimidating and an impediment to getting started.

With this in mind, I was excited to see Jason Miller‘s announcement of HTM (Hyperscript Tagged Markup):

While it is possible to write JSX without a build step by loading a standalone Babel into the browser, it is very expensive to do this runtime transpilation and so it’s not recommended in production. In contrast, HTM is small and fast:

It’s built using Tagged Templates and the browser’s HTML parser. Works in all modern browsers.

So HTM offers a third way to write blocks beyond ES5 and JSX. As with ES5 it doesn’t require a build step, while like JSX it has a much more pleasant syntax. Compare the JSX above with the following HTM:

function save( { attributes } ) {
    return html`<div id=${ attributes.id }>
        ${ attributes.content }
    </div>`;
}

I’ve given it a try in my syntax-highlighting Code block which extends the core Code block (forked from mkaz/code-syntax-block). Take a look at code-syntax-block.js for the editor JS file which is enqueued straight into WordPress without any build step.

P.S. Do you realize that you just read an AMP page?

WordCamp Europe 2018 Recap: AMP and PWA

Recently I attended WCEU 2018 in Belgrade with quite a few colleagues from XWP. We were there in large part to promote the adoption of progressive technologies in WordPress. We spent a lot of our time at the Google booth where we had an area to talk about contributing to WordPress across a wide range of roles. I spent most of the time in the booth stationed at the AMP area talking about the new capabilities we recently published in the plugin’s v1.0-alpha1 release, and since then we’ve followed up by releasing v1.0-beta1.

I’m really excited about how the AMP plugin is turning out. It now enables you to create AMP-compatible themes in the WordPress way; your theme can render your site in AMP using the same templates and stylesheets you would use normally on a non-AMP site. There is complete visual parity between your AMP pages and your non-AMP pages, aside for some differences in embeds (compare this post with AMP and without AMP). This being the case, you don’t even need to have a non-AMP version of your site anymore (the Paired mode), as the Native mode can just serve your entire site in AMP (such as xwp.co). AMP restricts what HTML you can use in order to guarantee performance and security, and the plugin never serves a response that contains invalid AMP in it. The plugin has a validation workflow to identify what the AMP validation errors are, where they are coming from in the page, and which theme/plugin is to blame. Please try it out and refer to the wiki for all the details on how to leverage the new features, especially Adding Theme Support and Implementing Interactivity.

My colleagues Alberto Medina (of Google) and Thierry Muller (of XWP) gave a great talk on Progressive WordPress which also dove into why AMP is important and how the AMP plugin brings its benefits to WordPress:

And we were interviewed on the topic of Progressive WordPress by Sarah Gooding at WP Tavern:

WCEU Panel Discusses Progressive WordPress Themes, AMP, and Gutenberg

On the topic of Progressive Web Apps, after Matt Mullenweg’s keynote someone asked during the Q&A about about a future where WordPress could be used to create to create apps. Matt responded:

There’s very exciting technologies coming out of two big corporations, one of which is a sponsor of WordCamp Europe, thank you Google. […] The other thing coming from Google which is very very exciting since they also contribute to the largest open source browser which is chrome—progressive web apps. So there’s lots of technologies around there that I think could drastically speed up how wp-admin works in a way that makes it a much more app-like experience without it actually needing to be completely rewritten from the ground up. So I feel like there is an opportunity to get… sort of almost like a JavaScript app-like performance increases from the wp-admin in a mostly backwards-compatible way, with progressive web app technology. So that’s very very exciting. It’s also open standards. They’re being supported by many places. So I feel like there’s a time there when it looked pretty dark to be honest for the web, particularly around performance. Things were just going slower and of course we know users start to tend toward things that are faster. So the apps were winning. But between AMP, progressive web apps, and just all the improvements and optimizations that we’re making including things coming under the hood like PHP7, basically doubling performance of WordPress overnight. […] These things are making it so that we can a really competitive experience and I’m excited about the future of the web.

As I just tweeted, there is now a PWA feature plugin on the WordPress.org directory. Its purpose is to curate Progressive Web App capabilities for proposed merging into WordPress core: service workers, the web app manifest, and improved HTTPS support.

PWA

This PWA feature plugin is intended to equip and facilitate other plugins which implement PWA features. It’s not intended to negate any existing plugins with these features, but rather to allow such plugins (and themes) to work together seamlessly and expand upon them. The plugin’s first release (v0.1.0) includes support for the web app manifest and an API for themes and plugins to register scripts for service workers, of which two are installed: one for the frontend (scope: ~/) and one for the admin (scope: ~/wp-admin/). A next step for service workers in the PWA feature plugin is to integrate Workbox to provide a declarative WordPress PHP abstraction for managing the caching strategies for routes, with support for detecting conflicts. You can follow development and contribute to the plugin on GitHub.

Photos not taken by me are courtesy of Ryan Kienstra, Alberto Medina, and Paul Bakaus.

“Building with JavaScript in the Customizer” at WCUS 2017

At WordCamp US 2017 I gave a talk on “Building with JavaScript in the Customizer”. I was happy to have the opportunity to share the technical details on the Customizer’s architecture and JavaScript API, which saw many improvements in 4.9, in addition to being able to share the Customizer’s new user-facing features during State of the Word.

The video has been posted on WordPress.tv:

Some photos taken during my talk:

Here are the slides:

I want to convert the talk into a series of blog posts to dig into more of the details and provide more examples, but I probably won’t get to that until 2018.

Mel Choyce’s Recap on WordPress 4.9

Whereas my recap post about WordPress 4.9 focused mostly on the new features and enhancements, my co-release lead Mel Choyce just published a great post that gets into more of the process aspects of the release, including the key contributors:

WordPress 4.9 Released

I think that 4.9 went really well by having essentially three co-release leads: Mel Choyce leading design, myself leading the development, and Jeff Paul leading project management as deputy release lead.