wiki.techinc.nl/tests/phpunit/ResourceLoaderTestCase.php

188 lines
4.9 KiB
PHP
Raw Normal View History

<?php
use MediaWiki\MediaWikiServices;
use Psr\Log\LoggerInterface;
use Psr\Log\NullLogger;
abstract class ResourceLoaderTestCase extends MediaWikiTestCase {
// Version hash for a blank file module.
// Result of ResourceLoader::makeHash(), ResourceLoaderTestModule
// and ResourceLoaderFileModule::getDefinitionSummary().
const BLANK_VERSION = '09p30q0';
/**
* @param array|string $options Language code or options array
* - string 'lang' Language code
* - string 'dir' Language direction (ltr or rtl)
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
* - string 'modules' Pipe-separated list of module names
* - string|null 'only' "scripts" (unwrapped script), "styles" (stylesheet), or null
* (mw.loader.implement).
* @param ResourceLoader|null $rl
* @return ResourceLoaderContext
*/
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
protected function getResourceLoaderContext( $options = [], ResourceLoader $rl = null ) {
if ( is_string( $options ) ) {
// Back-compat for extension tests
$options = [ 'lang' => $options ];
}
$options += [
'lang' => 'en',
'dir' => 'ltr',
'skin' => 'vector',
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
'modules' => 'startup',
'only' => 'scripts',
'safemode' => null,
];
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
$resourceLoader = $rl ?: new ResourceLoader();
$request = new FauxRequest( [
'lang' => $options['lang'],
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
'modules' => $options['modules'],
'only' => $options['only'],
'safemode' => $options['safemode'],
'skin' => $options['skin'],
'target' => 'phpunit',
] );
$ctx = $this->getMockBuilder( ResourceLoaderContext::class )
->setConstructorArgs( [ $resourceLoader, $request ] )
->setMethods( [ 'getDirection' ] )
->getMock();
$ctx->method( 'getDirection' )->willReturn( $options['dir'] );
return $ctx;
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
public static function getSettings() {
return [
// For ResourceLoader::inDebugMode since it doesn't have context
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
'ResourceLoaderDebug' => true,
// Avoid influence from wgInvalidateCacheOnLocalSettingsChange
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
'CacheEpoch' => '20140101000000',
// For ResourceLoader::__construct()
'ResourceLoaderSources' => [],
// For wfScript()
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
'ScriptPath' => '/w',
'Script' => '/w/index.php',
'LoadScript' => '/w/load.php',
];
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
}
protected function setUp() {
parent::setUp();
ResourceLoader::clearCache();
$globals = [];
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
foreach ( self::getSettings() as $key => $value ) {
$globals['wg' . $key] = $value;
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
}
$this->setMwGlobals( $globals );
}
}
/* Stubs */
class ResourceLoaderTestModule extends ResourceLoaderModule {
protected $messages = [];
protected $dependencies = [];
protected $group = null;
protected $source = 'local';
protected $script = '';
protected $styles = '';
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
protected $skipFunction = null;
protected $isRaw = false;
resourceloader: Move queue formatting out of OutputPage HTML formatting of the queue was distributed over several OutputPage methods. Each method demanding a snippet of HTML by calling makeResourceLoaderLink() with a limited amount of information. As such, makeResourceLoaderLink() was unable to provide the client with the proper state information. Centralising it also allows it to better reduce duplication in HTML output and maintain a more accurate state. Problems fixed by centralising: 1. The 'user' module is special (due to per-user 'version' and 'user' params). It is manually requested via script-src. To avoid a separate (and wrong) request from something that requires it, we set state=loading directly. However, because the module is in the bottom, the old HTML formatter could only put state=loading in the bottom also. This sometimes caused a wrong request to be fired for modules=user if something in the top queue triggered a requirement for it. 2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose of allowing dependencies on style modules without risking duplicate loading on pages where the styles are loaded already. This didn't work, because the state information about page-style modules is output near the stylesheet, which is after the script tag with mw.loader.load(). That runs first, and mw.loader would still make a duplicate request before it learns the state. Changes: * Document reasons for style/script tag order in getHeadHtml (per 09537e83). * Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed before since a duplicate check in makeResourceLoaderLink() verified the origin a second time. * Declare explicit position 'top' on 'user.options' and 'user.tokens' module. Previously, OutputPage hardcoded them in the top. The new formatter doesn't. * Remove getHeadScripts(). * Remove getInlineHeadScripts(). * Remove getExternalHeadScripts(). * Remove buildCssLinks(). * Remove getScriptsForBottomQueue(). * Change where Skin::setupSkinUserCss() is called. This methods lets the skin add modules to the queue. Previously it was called from buildCssLinks(), via headElement(), via prepareQuickTemplate(), via OutputPage::output(). It's now in OutputPage::output() directly (slightly earlier). This is needed because prepareQuickTemplate() calls bottomScripts() before headElement(). And bottomScript() would lazy-initialise the queue and lock it before setupSkinUserCss() is called from headElement(). This makes execution order more predictable instead of being dependent on the arbitrary order of data extraction in prepareQuickTemplate (which varies from one skin to another). * Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early on so. This avoids wrongful loading and fixes problem 1. Effective changes in output: * mw.loader.state() is now before mw.loader.load(). This fixes problem 2. * mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading". * mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1. * The <script async src> tag for 'startup' changed position (slightly). Previously it was after all inline scripts and stylesheets. It's still after all inline scripts and after most stylesheets, but before any user styles. Since the queue is now formatted outside OutputPage, it can't inject the meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle of existing output. This shouldn't have any noticable impact. Bug: T87871 Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
2016-07-15 14:13:09 +00:00
protected $isKnownEmpty = false;
protected $type = ResourceLoaderModule::LOAD_GENERAL;
protected $targets = [ 'phpunit' ];
protected $shouldEmbed = null;
public function __construct( $options = [] ) {
foreach ( $options as $key => $value ) {
$this->$key = $value;
}
}
public function getScript( ResourceLoaderContext $context ) {
return $this->validateScriptFile( 'input', $this->script );
}
public function getStyles( ResourceLoaderContext $context ) {
return [ '' => $this->styles ];
}
public function getMessages() {
return $this->messages;
}
resourceloader: Add context param to ResourceLoaderModule::getDependencies By providing context as a parameter in getDependencies, we allow modules to dyanamically determine dependencies based on context. Note: To ease rollout, the parameter is optional in this patch. It is expected that it will be made non-optional in the near future. The use case is for CentralNotice campaigns to be able to add special modules ahead of deciding which banner to show a user. The dynamically chosen RL modules would replace ad-hoc JS currently sent with some banners. A list of possible campaigns and banners is already sent as a PHP- implemented RL module; that's the module that will dynamically choose other modules as dependencies when appropriate. This approach will save a round trip as compared to dynamically loading the modules client-side. For compatibility, extensions that override ResourceLoaderModule::getDependencies() should be updated with the new method signature. Here are changes for extensions currently deployed on Wikimedia wikis: * CentralNotice: I816bffa3815e2eab7e88cb04d1b345070e6aa15f * Gadgets: I0a10fb0cbf17d095ece493e744296caf13dcee02 * EventLogging: I67e957f74d6ca48cfb9a41fb5144bcc78f885e50 * PageTriage: Ica3ba32aa2fc76d11a44f391b6edfc871e7fbe0d * UniversalLanguageSelector: Ic63e617f51702c27104e123d4bed91983a726b7f * VisualEditor: I0ac775ca286e64825e31a9213b94648e41a5bc30 For more on the CentralNotice use case, please see I9f80edcbcacca2. Bug: T98924 Change-Id: Iee61e5b527321d01287baa03ad9b4d4f526ff3ef
2015-04-08 21:34:08 +00:00
public function getDependencies( ResourceLoaderContext $context = null ) {
return $this->dependencies;
}
public function getGroup() {
return $this->group;
}
public function getSource() {
return $this->source;
}
resourceloader: Move queue formatting out of OutputPage HTML formatting of the queue was distributed over several OutputPage methods. Each method demanding a snippet of HTML by calling makeResourceLoaderLink() with a limited amount of information. As such, makeResourceLoaderLink() was unable to provide the client with the proper state information. Centralising it also allows it to better reduce duplication in HTML output and maintain a more accurate state. Problems fixed by centralising: 1. The 'user' module is special (due to per-user 'version' and 'user' params). It is manually requested via script-src. To avoid a separate (and wrong) request from something that requires it, we set state=loading directly. However, because the module is in the bottom, the old HTML formatter could only put state=loading in the bottom also. This sometimes caused a wrong request to be fired for modules=user if something in the top queue triggered a requirement for it. 2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose of allowing dependencies on style modules without risking duplicate loading on pages where the styles are loaded already. This didn't work, because the state information about page-style modules is output near the stylesheet, which is after the script tag with mw.loader.load(). That runs first, and mw.loader would still make a duplicate request before it learns the state. Changes: * Document reasons for style/script tag order in getHeadHtml (per 09537e83). * Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed before since a duplicate check in makeResourceLoaderLink() verified the origin a second time. * Declare explicit position 'top' on 'user.options' and 'user.tokens' module. Previously, OutputPage hardcoded them in the top. The new formatter doesn't. * Remove getHeadScripts(). * Remove getInlineHeadScripts(). * Remove getExternalHeadScripts(). * Remove buildCssLinks(). * Remove getScriptsForBottomQueue(). * Change where Skin::setupSkinUserCss() is called. This methods lets the skin add modules to the queue. Previously it was called from buildCssLinks(), via headElement(), via prepareQuickTemplate(), via OutputPage::output(). It's now in OutputPage::output() directly (slightly earlier). This is needed because prepareQuickTemplate() calls bottomScripts() before headElement(). And bottomScript() would lazy-initialise the queue and lock it before setupSkinUserCss() is called from headElement(). This makes execution order more predictable instead of being dependent on the arbitrary order of data extraction in prepareQuickTemplate (which varies from one skin to another). * Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early on so. This avoids wrongful loading and fixes problem 1. Effective changes in output: * mw.loader.state() is now before mw.loader.load(). This fixes problem 2. * mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading". * mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1. * The <script async src> tag for 'startup' changed position (slightly). Previously it was after all inline scripts and stylesheets. It's still after all inline scripts and after most stylesheets, but before any user styles. Since the queue is now formatted outside OutputPage, it can't inject the meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle of existing output. This shouldn't have any noticable impact. Bug: T87871 Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
2016-07-15 14:13:09 +00:00
public function getType() {
return $this->type;
}
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
public function getSkipFunction() {
return $this->skipFunction;
}
public function isRaw() {
return $this->isRaw;
}
resourceloader: Move queue formatting out of OutputPage HTML formatting of the queue was distributed over several OutputPage methods. Each method demanding a snippet of HTML by calling makeResourceLoaderLink() with a limited amount of information. As such, makeResourceLoaderLink() was unable to provide the client with the proper state information. Centralising it also allows it to better reduce duplication in HTML output and maintain a more accurate state. Problems fixed by centralising: 1. The 'user' module is special (due to per-user 'version' and 'user' params). It is manually requested via script-src. To avoid a separate (and wrong) request from something that requires it, we set state=loading directly. However, because the module is in the bottom, the old HTML formatter could only put state=loading in the bottom also. This sometimes caused a wrong request to be fired for modules=user if something in the top queue triggered a requirement for it. 2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose of allowing dependencies on style modules without risking duplicate loading on pages where the styles are loaded already. This didn't work, because the state information about page-style modules is output near the stylesheet, which is after the script tag with mw.loader.load(). That runs first, and mw.loader would still make a duplicate request before it learns the state. Changes: * Document reasons for style/script tag order in getHeadHtml (per 09537e83). * Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed before since a duplicate check in makeResourceLoaderLink() verified the origin a second time. * Declare explicit position 'top' on 'user.options' and 'user.tokens' module. Previously, OutputPage hardcoded them in the top. The new formatter doesn't. * Remove getHeadScripts(). * Remove getInlineHeadScripts(). * Remove getExternalHeadScripts(). * Remove buildCssLinks(). * Remove getScriptsForBottomQueue(). * Change where Skin::setupSkinUserCss() is called. This methods lets the skin add modules to the queue. Previously it was called from buildCssLinks(), via headElement(), via prepareQuickTemplate(), via OutputPage::output(). It's now in OutputPage::output() directly (slightly earlier). This is needed because prepareQuickTemplate() calls bottomScripts() before headElement(). And bottomScript() would lazy-initialise the queue and lock it before setupSkinUserCss() is called from headElement(). This makes execution order more predictable instead of being dependent on the arbitrary order of data extraction in prepareQuickTemplate (which varies from one skin to another). * Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early on so. This avoids wrongful loading and fixes problem 1. Effective changes in output: * mw.loader.state() is now before mw.loader.load(). This fixes problem 2. * mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading". * mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1. * The <script async src> tag for 'startup' changed position (slightly). Previously it was after all inline scripts and stylesheets. It's still after all inline scripts and after most stylesheets, but before any user styles. Since the queue is now formatted outside OutputPage, it can't inject the meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle of existing output. This shouldn't have any noticable impact. Bug: T87871 Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
2016-07-15 14:13:09 +00:00
public function isKnownEmpty( ResourceLoaderContext $context ) {
return $this->isKnownEmpty;
}
resourceloader: Enable module content version for data modules This greatly simplifies logic required to compute module versions. It also makes it significantly less error-prone. Since f37cee996e, we support hashes as versions (instead of timestamps). This means we can build a hash of the content directly, instead of compiling a large array with all values that may influence the module content somehow. Benefits: * Remove all methods and logic related to querying database and disk for timestamps, revision numbers, definition summaries, cache epochs, and more. * No longer needlessly invalidate cache as a result of no-op changes to implementation datails. Due to inclusion of absolute file paths in the definition summary, cache was always invalidated when moving wikis to newer MediaWiki branches; even if the module observed no actual changes. * When changes are reverted within a certain period of time, old caches can now be re-used. The module would produce the same version hash as before. Previously when a change was deployed and then reverted, all web clients (even those that never saw the bad version) would have re-fetch modules because the version increased. Updated unit tests to account for the change in version. New default version of empty test modules is: "mvgTPvXh". For the record, this comes from the base64 encoding of the SHA1 digest of the JSON serialised form of the module content: > $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}'; > echo base64_encode(sha1($str, true)); > FEb3+VuiUm/fOMfod1bjw/te+AQ= Enabled content versioning for the data modules in MediaWiki core: * EditToolbarModule * JqueryMsgModule * LanguageDataModule * LanguageNamesModule * SpecialCharacterDataModule * UserCSSPrefsModule * UserDefaultsModule * UserOptionsModule The FileModule and base class explicitly disable it for now and keep their current behaviour of using the definition summary. We may remove it later, but that requires more performance testing first. Explicitly disable it in the WikiModule class to avoid breakage when the default changes. Ref T98087. Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-02 17:27:23 +00:00
public function shouldEmbedModule( ResourceLoaderContext $context ) {
return $this->shouldEmbed ?? parent::shouldEmbedModule( $context );
}
resourceloader: Enable module content version for data modules This greatly simplifies logic required to compute module versions. It also makes it significantly less error-prone. Since f37cee996e, we support hashes as versions (instead of timestamps). This means we can build a hash of the content directly, instead of compiling a large array with all values that may influence the module content somehow. Benefits: * Remove all methods and logic related to querying database and disk for timestamps, revision numbers, definition summaries, cache epochs, and more. * No longer needlessly invalidate cache as a result of no-op changes to implementation datails. Due to inclusion of absolute file paths in the definition summary, cache was always invalidated when moving wikis to newer MediaWiki branches; even if the module observed no actual changes. * When changes are reverted within a certain period of time, old caches can now be re-used. The module would produce the same version hash as before. Previously when a change was deployed and then reverted, all web clients (even those that never saw the bad version) would have re-fetch modules because the version increased. Updated unit tests to account for the change in version. New default version of empty test modules is: "mvgTPvXh". For the record, this comes from the base64 encoding of the SHA1 digest of the JSON serialised form of the module content: > $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}'; > echo base64_encode(sha1($str, true)); > FEb3+VuiUm/fOMfod1bjw/te+AQ= Enabled content versioning for the data modules in MediaWiki core: * EditToolbarModule * JqueryMsgModule * LanguageDataModule * LanguageNamesModule * SpecialCharacterDataModule * UserCSSPrefsModule * UserDefaultsModule * UserOptionsModule The FileModule and base class explicitly disable it for now and keep their current behaviour of using the definition summary. We may remove it later, but that requires more performance testing first. Explicitly disable it in the WikiModule class to avoid breakage when the default changes. Ref T98087. Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-02 17:27:23 +00:00
public function enableModuleContentVersion() {
return true;
}
}
class ResourceLoaderFileTestModule extends ResourceLoaderFileModule {
protected $lessVars = [];
public function __construct( $options = [], $test = [] ) {
parent::__construct( $options );
foreach ( $test as $key => $value ) {
$this->$key = $value;
}
}
public function getLessVars( ResourceLoaderContext $context ) {
return $this->lessVars;
}
}
class ResourceLoaderFileModuleTestModule extends ResourceLoaderFileModule {
}
class EmptyResourceLoader extends ResourceLoader {
// TODO: This won't be needed once ResourceLoader is empty by default
// and default registrations are done from ServiceWiring instead.
public function __construct( Config $config = null, LoggerInterface $logger = null ) {
$this->setLogger( $logger ?: new NullLogger() );
$this->config = $config ?: MediaWikiServices::getInstance()->getMainConfig();
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
// Source "local" is required by StartupModule
$this->addSource( 'local', $this->config->get( 'LoadScript' ) );
$this->setMessageBlobStore( new MessageBlobStore( $this, $this->getLogger() ) );
}
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
public function getErrors() {
return $this->errors;
}
}