wiki.techinc.nl/tests/phpunit/structure/ResourcesTest.php
Timo Tijhof f37cee996e resourceloader: Replace timestamp system with version hashing
Modules now track their version via getVersionHash() instead of getModifiedTime().

== Background ==

While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.

For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.

Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.

For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.

More scenarios can be found in the associated Phabricator tasks.

== Version hash ==

Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.

All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.

For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.

As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.

* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule

The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).

== ETag ==

Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).

Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).

Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.

While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).

== File cache ==

If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.

For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.

This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.

== Misc ==

* Remove redundant modifiedTime cache in ResourceLoaderFileModule.

* Change bugzilla references to Phabricator.

* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
  logic was duplicated in each place the modified timestamp was used.

* It's easy to forget calling the parent class in getDefinitionSummary().
  Previously this method only tracked 'class' by default. As such, various
  extensions hardcoded that one value instead of calling the parent and extending
  the array. To better prevent this in the future, getVersionHash() now asserts
  that the '_cacheEpoch' property made it through.

* tests: Don't use getDefinitionSummary() as an API.
  Fix ResourceLoaderWikiModuleTest to call getPages properly.

* In tests, the default timestamp used to be 1388534400000 (which is the unix time
  of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
  modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
  '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'

* Add sha1.js library for client-side hash generation.
  Compared various different implementations for code size (after minfication/gzip),
  and speed (when used for short hexidecimal strings).
  https://jsperf.com/sha1-implementations
  - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
    http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
    Chrome: 45k, Firefox: 89k, Safari: 92k
  - jsSHA <https://github.com/Caligatio/jsSHA>
    https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
    Chrome: 65k, Firefox: 53k, Safari: 69k
  - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
    https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
    Chrome: 200k, Firefox: 280k, Safari: 78k

  Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
  only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
  code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
  needed in the module loader, we'd have to load the fallback regardless. Considering
  this is not used in a critical path for performance, it's not worth shipping two
  implementations for this optimisation.

May also resolve:
* T44094
* T90411
* T94810

Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-05-19 22:28:17 +00:00

280 lines
7.6 KiB
PHP

<?php
/**
* Sanity checks for making sure registered resources are sane.
*
* @file
* @author Antoine Musso
* @author Niklas Laxström
* @author Santhosh Thottingal
* @author Timo Tijhof
* @copyright © 2012, Antoine Musso
* @copyright © 2012, Niklas Laxström
* @copyright © 2012, Santhosh Thottingal
* @copyright © 2012, Timo Tijhof
*
*/
class ResourcesTest extends MediaWikiTestCase {
/**
* @dataProvider provideResourceFiles
*/
public function testFileExistence( $filename, $module, $resource ) {
$this->assertFileExists( $filename,
"File '$resource' referenced by '$module' must exist."
);
}
/**
* @dataProvider provideMediaStylesheets
*/
public function testStyleMedia( $moduleName, $media, $filename, $css ) {
$cssText = CSSMin::minify( $css->cssText );
$this->assertTrue(
strpos( $cssText, '@media' ) === false,
'Stylesheets should not both specify "media" and contain @media'
);
}
public function testVersionHash() {
$data = self::getAllModules();
foreach ( $data['modules'] as $moduleName => $module ) {
$version = $module->getVersionHash( $data['context'] );
$this->assertEquals( 8, strlen( $version ), "$moduleName must use ResourceLoader::makeHash" );
}
}
/**
* Verify that nothing explicitly depends on the 'jquery' and 'mediawiki' modules.
* They are always loaded, depending on them is unsupported and leads to unexpected behaviour.
*/
public function testIllegalDependencies() {
$data = self::getAllModules();
$illegalDeps = array( 'jquery', 'mediawiki' );
/** @var ResourceLoaderModule $module */
foreach ( $data['modules'] as $moduleName => $module ) {
foreach ( $illegalDeps as $illegalDep ) {
$this->assertNotContains(
$illegalDep,
$module->getDependencies(),
"Module '$moduleName' must not depend on '$illegalDep'"
);
}
}
}
/**
* Verify that all modules specified as dependencies of other modules actually exist.
*/
public function testMissingDependencies() {
$data = self::getAllModules();
$validDeps = array_keys( $data['modules'] );
/** @var ResourceLoaderModule $module */
foreach ( $data['modules'] as $moduleName => $module ) {
foreach ( $module->getDependencies() as $dep ) {
$this->assertContains(
$dep,
$validDeps,
"The module '$dep' required by '$moduleName' must exist"
);
}
}
}
/**
* Verify that all dependencies of all modules are always satisfiable with the 'targets' defined
* for the involved modules.
*
* Example: A depends on B. A has targets: mobile, desktop. B has targets: desktop. Therefore the
* dependency is sometimes unsatisfiable: it's impossible to load module A on mobile.
*/
public function testUnsatisfiableDependencies() {
$data = self::getAllModules();
$validDeps = array_keys( $data['modules'] );
/** @var ResourceLoaderModule $module */
foreach ( $data['modules'] as $moduleName => $module ) {
$moduleTargets = $module->getTargets();
foreach ( $module->getDependencies() as $dep ) {
if ( !isset( $data['modules'][$dep] ) ) {
// Missing dependencies reported by testMissingDependencies
continue;
}
$targets = $data['modules'][$dep]->getTargets();
foreach ( $moduleTargets as $moduleTarget ) {
$this->assertContains(
$moduleTarget,
$targets,
"The module '$moduleName' must not have target '$moduleTarget' "
. "because its dependency '$dep' does not have it"
);
}
}
}
}
/**
* Get all registered modules from ResouceLoader.
* @return array
*/
protected static function getAllModules() {
global $wgEnableJavaScriptTest;
// Test existance of test suite files as well
// (can't use setUp or setMwGlobals because providers are static)
$org_wgEnableJavaScriptTest = $wgEnableJavaScriptTest;
$wgEnableJavaScriptTest = true;
// Initialize ResourceLoader
$rl = new ResourceLoader();
$modules = array();
foreach ( $rl->getModuleNames() as $moduleName ) {
$modules[$moduleName] = $rl->getModule( $moduleName );
}
// Restore settings
$wgEnableJavaScriptTest = $org_wgEnableJavaScriptTest;
return array(
'modules' => $modules,
'resourceloader' => $rl,
'context' => new ResourceLoaderContext( $rl, new FauxRequest() )
);
}
/**
* Get all stylesheet files from modules that are an instance of
* ResourceLoaderFileModule (or one of its subclasses).
*/
public static function provideMediaStylesheets() {
$data = self::getAllModules();
$cases = array();
foreach ( $data['modules'] as $moduleName => $module ) {
if ( !$module instanceof ResourceLoaderFileModule ) {
continue;
}
$reflectedModule = new ReflectionObject( $module );
$getStyleFiles = $reflectedModule->getMethod( 'getStyleFiles' );
$getStyleFiles->setAccessible( true );
$readStyleFile = $reflectedModule->getMethod( 'readStyleFile' );
$readStyleFile->setAccessible( true );
$styleFiles = $getStyleFiles->invoke( $module, $data['context'] );
$flip = $module->getFlip( $data['context'] );
foreach ( $styleFiles as $media => $files ) {
if ( $media && $media !== 'all' ) {
foreach ( $files as $file ) {
$cases[] = array(
$moduleName,
$media,
$file,
// XXX: Wrapped in an object to keep it out of PHPUnit output
(object)array( 'cssText' => $readStyleFile->invoke( $module, $file, $flip ) ),
);
}
}
}
}
return $cases;
}
/**
* Get all resource files from modules that are an instance of
* ResourceLoaderFileModule (or one of its subclasses).
*
* Since the raw data is stored in protected properties, we have to
* overrride this through ReflectionObject methods.
*/
public static function provideResourceFiles() {
$data = self::getAllModules();
$cases = array();
// See also ResourceLoaderFileModule::__construct
$filePathProps = array(
// Lists of file paths
'lists' => array(
'scripts',
'debugScripts',
'loaderScripts',
'styles',
),
// Collated lists of file paths
'nested-lists' => array(
'languageScripts',
'skinScripts',
'skinStyles',
),
);
foreach ( $data['modules'] as $moduleName => $module ) {
if ( !$module instanceof ResourceLoaderFileModule ) {
continue;
}
$reflectedModule = new ReflectionObject( $module );
$files = array();
foreach ( $filePathProps['lists'] as $propName ) {
$property = $reflectedModule->getProperty( $propName );
$property->setAccessible( true );
$list = $property->getValue( $module );
foreach ( $list as $key => $value ) {
// 'scripts' are numeral arrays.
// 'styles' can be numeral or associative.
// In case of associative the key is the file path
// and the value is the 'media' attribute.
if ( is_int( $key ) ) {
$files[] = $value;
} else {
$files[] = $key;
}
}
}
foreach ( $filePathProps['nested-lists'] as $propName ) {
$property = $reflectedModule->getProperty( $propName );
$property->setAccessible( true );
$lists = $property->getValue( $module );
foreach ( $lists as $list ) {
foreach ( $list as $key => $value ) {
// We need the same filter as for 'lists',
// due to 'skinStyles'.
if ( is_int( $key ) ) {
$files[] = $value;
} else {
$files[] = $key;
}
}
}
}
// Get method for resolving the paths to full paths
$method = $reflectedModule->getMethod( 'getLocalPath' );
$method->setAccessible( true );
// Populate cases
foreach ( $files as $file ) {
$cases[] = array(
$method->invoke( $module, $file ),
$moduleName,
( $file instanceof ResourceLoaderFilePath ? $file->getPath() : $file ),
);
}
}
return $cases;
}
}