Running the WordPress Plugin Directory Slurper

As part of my work on the WordPress Core Performance Team, I need to do analyses on the plugins in the WordPress Plugin Directory. For example, in WordPress 6.3 we’re finally shipping first-party support for adding async and defer to scripts (a 13-year old ticket). In this case, the excellent WordPress Directory Searcher (WPdirectory) is all I needed to find usages of inline after scripts across the ecosystem. However, not all code searches can be easily performed with a regular expression for a single-line match.

In order to facilitate more extensive searching where necessary going forward, I’ve just used Mark Jaquith’s WordPress Plugin Directory Slurper to download the entire directory.

According to the FAQ How long will it take?:

Your first update will take a while (at least a couple of hours, and potentially overnight, depending on your connection and disk speeds).

And How much disk space do I need?:

As of December 2017, the plugin repository contains over 70,000 plugins. The script will download around 20 GB of zip files which, when unpacked, will use around 45 GB of disk space.

Things have changed since December 2017, which was now 6½ years ago.

My call to time ./update finished as follows:

[SUCCESS] Done updating plugins!
It took 1580 minutes and 3 seconds to update 87873 plugins (11610 failed).
[DONE]

real	1580m4.203s
user	92m31.046s
sys	94m57.713sCode language: plaintext (plaintext)

It took an astonishing 26⅓ hours to finish on a system with a blazing 2 Gbps internet connection (granted I’m not immediately aware of the disk write speed). The total size of the downloaded directory ended up being 109 GB.

Fair warning for anyone else wanting to slurp down the entire plugin directory!