wiki.techinc.nl/includes/resourceloader/ResourceLoaderStartUpModule.php

414 lines
14 KiB
PHP
Raw Normal View History

<?php
/**
* Module for ResourceLoader initialization.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
* http://www.gnu.org/copyleft/gpl.html
*
* @file
* @author Trevor Parscal
* @author Roan Kattouw
*/
class ResourceLoaderStartUpModule extends ResourceLoaderModule {
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Cache for getConfigSettings() as it's called by multiple methods
protected $configVars = [];
protected $targets = [ 'desktop', 'mobile' ];
/**
* @param ResourceLoaderContext $context
* @return array
*/
protected function getConfigSettings( $context ) {
$hash = $context->getHash();
if ( isset( $this->configVars[$hash] ) ) {
return $this->configVars[$hash];
}
global $wgContLang;
$conf = $this->getConfig();
// We can't use Title::newMainPage() if 'mainpage' is in
// $wgForceUIMsgAsContentMsg because that will try to use the session
// user's language and we have no session user. This does the
// equivalent but falling back to our ResourceLoaderContext language
// instead.
$mainPage = Title::newFromText( $context->msg( 'mainpage' )->inContentLanguage()->text() );
if ( !$mainPage ) {
$mainPage = Title::newFromText( 'Main Page' );
}
/**
* Namespace related preparation
* - wgNamespaceIds: Key-value pairs of all localized, canonical and aliases for namespaces.
* - wgCaseSensitiveNamespaces: Array of namespaces that are case-sensitive.
*/
$namespaceIds = $wgContLang->getNamespaceIds();
$caseSensitiveNamespaces = [];
foreach ( MWNamespace::getCanonicalNamespaces() as $index => $name ) {
2011-06-14 22:51:47 +00:00
$namespaceIds[$wgContLang->lc( $name )] = $index;
if ( !MWNamespace::isCapitalized( $index ) ) {
$caseSensitiveNamespaces[] = $index;
}
}
$illegalFileChars = $conf->get( 'IllegalFileChars' );
// Build list of variables
$vars = [
'wgLoadScript' => wfScript( 'load' ),
'debug' => $context->getDebug(),
'skin' => $context->getSkin(),
'stylepath' => $conf->get( 'StylePath' ),
'wgUrlProtocols' => wfUrlProtocols(),
'wgArticlePath' => $conf->get( 'ArticlePath' ),
'wgScriptPath' => $conf->get( 'ScriptPath' ),
'wgScriptExtension' => '.php',
'wgScript' => wfScript(),
'wgSearchType' => $conf->get( 'SearchType' ),
'wgVariantArticlePath' => $conf->get( 'VariantArticlePath' ),
// Force object to avoid "empty" associative array from
// becoming [] instead of {} in JS (bug 34604)
'wgActionPaths' => (object)$conf->get( 'ActionPaths' ),
'wgServer' => $conf->get( 'Server' ),
'wgServerName' => $conf->get( 'ServerName' ),
'wgUserLanguage' => $context->getLanguage(),
'wgContentLanguage' => $wgContLang->getCode(),
'wgTranslateNumerals' => $conf->get( 'TranslateNumerals' ),
'wgVersion' => $conf->get( 'Version' ),
'wgEnableAPI' => $conf->get( 'EnableAPI' ),
'wgEnableWriteAPI' => $conf->get( 'EnableWriteAPI' ),
'wgMainPageTitle' => $mainPage->getPrefixedText(),
'wgFormattedNamespaces' => $wgContLang->getFormattedNamespaces(),
'wgNamespaceIds' => $namespaceIds,
'wgContentNamespaces' => MWNamespace::getContentNamespaces(),
'wgSiteName' => $conf->get( 'Sitename' ),
'wgDBname' => $conf->get( 'DBname' ),
'wgExtraSignatureNamespaces' => $conf->get( 'ExtraSignatureNamespaces' ),
'wgAvailableSkins' => Skin::getSkinNames(),
'wgExtensionAssetsPath' => $conf->get( 'ExtensionAssetsPath' ),
// MediaWiki sets cookies to have this prefix by default
'wgCookiePrefix' => $conf->get( 'CookiePrefix' ),
'wgCookieDomain' => $conf->get( 'CookieDomain' ),
'wgCookiePath' => $conf->get( 'CookiePath' ),
'wgCookieExpiration' => $conf->get( 'CookieExpiration' ),
'wgResourceLoaderMaxQueryLength' => $conf->get( 'ResourceLoaderMaxQueryLength' ),
'wgCaseSensitiveNamespaces' => $caseSensitiveNamespaces,
'wgLegalTitleChars' => Title::convertByteClassToUnicodeClass( Title::legalChars() ),
'wgIllegalFileChars' => Title::convertByteClassToUnicodeClass( $illegalFileChars ),
'wgResourceLoaderStorageVersion' => $conf->get( 'ResourceLoaderStorageVersion' ),
'wgResourceLoaderStorageEnabled' => $conf->get( 'ResourceLoaderStorageEnabled' ),
'wgResourceLoaderLegacyModules' => self::getLegacyModules(),
'wgForeignUploadTargets' => $conf->get( 'ForeignUploadTargets' ),
'wgEnableUploads' => $conf->get( 'EnableUploads' ),
];
Hooks::run( 'ResourceLoaderGetConfigVars', [ &$vars ] );
$this->configVars[$hash] = $vars;
return $this->configVars[$hash];
}
/**
* Recursively get all explicit and implicit dependencies for to the given module.
*
* @param array $registryData
* @param string $moduleName
* @return array
*/
protected static function getImplicitDependencies( array $registryData, $moduleName ) {
static $dependencyCache = [];
// The list of implicit dependencies won't be altered, so we can
// cache them without having to worry.
if ( !isset( $dependencyCache[$moduleName] ) ) {
if ( !isset( $registryData[$moduleName] ) ) {
// Dependencies may not exist
$dependencyCache[$moduleName] = [];
} else {
$data = $registryData[$moduleName];
$dependencyCache[$moduleName] = $data['dependencies'];
foreach ( $data['dependencies'] as $dependency ) {
// Recursively get the dependencies of the dependencies
$dependencyCache[$moduleName] = array_merge(
$dependencyCache[$moduleName],
self::getImplicitDependencies( $registryData, $dependency )
);
}
}
}
return $dependencyCache[$moduleName];
}
/**
* Optimize the dependency tree in $this->modules.
*
* The optimization basically works like this:
* Given we have module A with the dependencies B and C
* and module B with the dependency C.
* Now we don't have to tell the client to explicitly fetch module
* C as that's already included in module B.
*
* This way we can reasonably reduce the amount of module registration
* data send to the client.
*
* @param array &$registryData Modules keyed by name with properties:
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* - string 'version'
* - array 'dependencies'
* - string|null 'group'
* - string 'source'
*/
public static function compileUnresolvedDependencies( array &$registryData ) {
foreach ( $registryData as $name => &$data ) {
$dependencies = $data['dependencies'];
foreach ( $data['dependencies'] as $dependency ) {
$implicitDependencies = self::getImplicitDependencies( $registryData, $dependency );
$dependencies = array_diff( $dependencies, $implicitDependencies );
}
// Rebuild keys
$data['dependencies'] = array_values( $dependencies );
}
}
/**
* Get registration code for all modules.
*
* @param ResourceLoaderContext $context
* @return string JavaScript code for registering all modules with the client loader
*/
public function getModuleRegistrations( ResourceLoaderContext $context ) {
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
$resourceLoader = $context->getResourceLoader();
$target = $context->getRequest()->getVal( 'target', 'desktop' );
// Bypass target filter if this request is Special:JavaScriptTest.
// To prevent misuse in production, this is only allowed if testing is enabled server-side.
$byPassTargetFilter = $this->getConfig()->get( 'EnableJavaScriptTest' ) && $target === 'test';
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
$out = '';
$registryData = [];
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
// Get registry data
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
foreach ( $resourceLoader->getModuleNames() as $name ) {
$module = $resourceLoader->getModule( $name );
$moduleTargets = $module->getTargets();
if ( !$byPassTargetFilter && !in_array( $target, $moduleTargets ) ) {
continue;
}
if ( $module->isRaw() ) {
// Don't register "raw" modules (like 'jquery' and 'mediawiki') client-side because
// depending on them is illegal anyway and would only lead to them being reloaded
// causing any state to be lost (like jQuery plugins, mw.config etc.)
continue;
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
$versionHash = $module->getVersionHash( $context );
if ( strlen( $versionHash ) !== 7 ) {
$context->getLogger()->warning(
"Module '{module}' produced an invalid version hash: '{version}'.",
[
'module' => $name,
'version' => $versionHash,
]
);
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Module implementation either broken or deviated from ResourceLoader::makeHash
// Asserted by tests/phpunit/structure/ResourcesTest.
$versionHash = ResourceLoader::makeHash( $versionHash );
}
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
$skipFunction = $module->getSkipFunction();
if ( $skipFunction !== null && !ResourceLoader::inDebugMode() ) {
$skipFunction = ResourceLoader::filter( 'minify-js', $skipFunction );
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
}
$registryData[$name] = [
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
'version' => $versionHash,
resourceloader: Add context param to ResourceLoaderModule::getDependencies By providing context as a parameter in getDependencies, we allow modules to dyanamically determine dependencies based on context. Note: To ease rollout, the parameter is optional in this patch. It is expected that it will be made non-optional in the near future. The use case is for CentralNotice campaigns to be able to add special modules ahead of deciding which banner to show a user. The dynamically chosen RL modules would replace ad-hoc JS currently sent with some banners. A list of possible campaigns and banners is already sent as a PHP- implemented RL module; that's the module that will dynamically choose other modules as dependencies when appropriate. This approach will save a round trip as compared to dynamically loading the modules client-side. For compatibility, extensions that override ResourceLoaderModule::getDependencies() should be updated with the new method signature. Here are changes for extensions currently deployed on Wikimedia wikis: * CentralNotice: I816bffa3815e2eab7e88cb04d1b345070e6aa15f * Gadgets: I0a10fb0cbf17d095ece493e744296caf13dcee02 * EventLogging: I67e957f74d6ca48cfb9a41fb5144bcc78f885e50 * PageTriage: Ica3ba32aa2fc76d11a44f391b6edfc871e7fbe0d * UniversalLanguageSelector: Ic63e617f51702c27104e123d4bed91983a726b7f * VisualEditor: I0ac775ca286e64825e31a9213b94648e41a5bc30 For more on the CentralNotice use case, please see I9f80edcbcacca2. Bug: T98924 Change-Id: Iee61e5b527321d01287baa03ad9b4d4f526ff3ef
2015-04-08 21:34:08 +00:00
'dependencies' => $module->getDependencies( $context ),
'group' => $module->getGroup(),
'source' => $module->getSource(),
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
'skip' => $skipFunction,
];
}
self::compileUnresolvedDependencies( $registryData );
// Register sources
$out .= ResourceLoader::makeLoaderSourcesScript( $resourceLoader->getSources() );
// Figure out the different call signatures for mw.loader.register
$registrations = [];
foreach ( $registryData as $name => $data ) {
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Call mw.loader.register(name, version, dependencies, group, source, skip)
$registrations[] = [
$name,
$data['version'],
$data['dependencies'],
$data['group'],
// Swap default (local) for null
$data['source'] === 'local' ? null : $data['source'],
$data['skip']
];
}
// Register modules
$out .= "\n" . ResourceLoader::makeLoaderRegisterScript( $registrations );
return $out;
}
/**
* @return bool
*/
public function isRaw() {
return true;
}
/**
* Base modules required for the base environment of ResourceLoader
*
* @return array
*/
public static function getStartupModules() {
return [ 'jquery', 'mediawiki' ];
}
public static function getLegacyModules() {
global $wgIncludeLegacyJavaScript;
$legacyModules = [];
if ( $wgIncludeLegacyJavaScript ) {
$legacyModules[] = 'mediawiki.legacy.wikibits';
}
return $legacyModules;
}
/**
* Get the load URL of the startup modules.
*
* This is a helper for getScript(), but can also be called standalone, such
* as when generating an AppCache manifest.
*
* @param ResourceLoaderContext $context
* @return string
*/
public static function getStartupModulesUrl( ResourceLoaderContext $context ) {
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
$rl = $context->getResourceLoader();
$derivative = new DerivativeResourceLoaderContext( $context );
$derivative->setModules( self::getStartupModules() );
$derivative->setOnly( 'scripts' );
// Must setModules() before makeVersionQuery()
$derivative->setVersion( $rl->makeVersionQuery( $derivative ) );
return $rl->createLoaderURL( 'local', $derivative );
}
/**
* @param ResourceLoaderContext $context
* @return string
*/
public function getScript( ResourceLoaderContext $context ) {
global $IP;
if ( $context->getOnly() !== 'scripts' ) {
return '/* Requires only=script */';
}
Split /resources into /resources/lib and /resources/src Currently, MediaWiki has a mess of resources, some but not all of which are upstream libraries. This means that occasionally people will attempt to make local modifications of the libraries by accident, or deliberately but do not flag it, making dependency management a challenge (and fixing upstream-related bugs a headache). Instead, we now split the /resources directory into a /resources/lib directory for those items which are used purely as intact upstream libraries, and a /resources/src directroy for those which are either local libraries written for MediaWiki and mastered here, or downstream forks of upstream libraries which require particular care and attention when being upgraded. Also, some incidental files which are not to be loaded as part of a module have been moved to better locations. Upstream libaries * jquery resources/jquery/jquery.js * jquery.appear resources/jquery/jquery.appear.js * jquery.async resources/jquery/jquery.async.js * jquery.chosen resources/jquery.chosen/* * jquery.cookie resources/jquery/jquery.cookie.js * jquery.form resources/jquery/jquery.form.js * jquery.fullscreen.js resources/jquery/jquery.fullscreen.js * jquery.hoverIntent resources/jquery/jquery.hoverIntent.js * jquery.json resources/jquery/jquery.json.js * jquery.mockjax resources/jquery/jquery.mockjax.js * jquery.qunit resources/jquery/jquery.qunit.css resources/jquery/jquery.qunit.js * jquery.jStorage resources/jquery/jquery.jStorage.js * jquery.throttle-debounce resources/jquery/jquery.ba-throttle-debounce.js * jquery.validate resources/jquery/jquery.validate.js * jquery.xmldom resources/jquery/jquery.xmldom.js * moment resources/moment/* * OOjs resource/oojs/* * OOjs UI resources/oojs-ui/* * jquery.ui.* resources/jquery.ui/* resources/jquery.ui/themes/default moved to resources/lib/jquery.ui/themes/ resources/jquery.ui/themes/vector moved to resources/src/jquery.ui-themes/ * jquery.effects.* resources/jquery.effects/* * sinon resources/sinonjs/* Upstream libraries forked for MediaWiki * jquery.color resources/jquery/jquery.color.js * jquery.tipsy resources/jquery.tipsy/* Local libraries * jquery.arrowSteps resources/jquery/jquery.arrowSteps.css resources/jquery/jquery.arrowSteps.js resources/jquery/images/jquery.arrowSteps.divider-ltr.png resources/jquery/images/jquery.arrowSteps.divider-rtl.png resources/jquery/images/jquery.arrowSteps.head-ltr.png resources/jquery/images/jquery.arrowSteps.head-rtl.png resources/jquery/images/jquery.arrowSteps.tail-ltr.png resources/jquery/images/jquery.arrowSteps.tail-rtl.png * jquery.autoEllipsis resources/jquery/jquery.autoEllipsis.js * jquery.badge resources/jquery/jquery.badge.css resources/jquery/jquery.badge.js * jquery.byteLength resources/jquery/jquery.byteLength.js * jquery.byteLimit resources/jquery/jquery.byteLimit.js * jquery.checkboxShiftClick resources/jquery/jquery.checkboxShiftClick.js * jquery.client resources/jquery/jquery.client.js * jquery.colorUtil resources/jquery/jquery.colorUtil.js * jquery.delayedBind resources/jquery/jquery.delayedBind.js * jquery.expandableField resources/jquery/jquery.expandableField.js * jquery.farbtastic resources/jquery/jquery.farbtastic.css resources/jquery/jquery.farbtastic.js resources/jquery/images/marker.png resources/jquery/images/mask.png resources/jquery/images/wheel.png * jquery.footHovzer resources/jquery/jquery.footHovzer.css resources/jquery/jquery.footHovzer.js * jquery.getAttrs resources/jquery/jquery.getAttrs.js * jquery.hidpi resources/jquery/jquery.hidpi.js * jquery.highlightText resources/jquery/jquery.highlightText.js * jquery.localize resources/jquery/jquery.localize.js * jquery.makeCollapsible resources/jquery/jquery.makeCollapsible.css resources/jquery/jquery.makeCollapsible.js * jquery.mw-jump resources/jquery/jquery.mw-jump.js * jquery.mwExtension resources/jquery/jquery.mwExtension.js * jquery.placeholder resources/jquery/jquery.placeholder.js * jquery.spinner resources/jquery/jquery.spinner.css resources/jquery/jquery.spinner.js resources/jquery/images/spinner-large.gif resources/jquery/images/spinner.gif * jquery.suggestions resources/jquery/jquery.suggestions.css resources/jquery/jquery.suggestions.js * jquery.tabIndex resources/jquery/jquery.tabIndex.js * jquery.tablesorter resources/jquery/jquery.tablesorter.css resources/jquery/jquery.tablesorter.js resources/jquery/images/sort_both.gif resources/jquery/images/sort_down.gif resources/jquery/images/sort_none.gif resources/jquery/images/sort_up.gif * jquery.textSelection resources/jquery/jquery.textSelection.js * jquery.qunit.completenessTest resources/jquery/jquery.qunit.completenessTest.js * mediawiki resources/mediawiki/mediawiki.js resources/mediawiki/mediawiki.log.js * mediawiki.api resources/mediawiki.api/mediawiki.api.js * mediawiki.api.category resources/mediawiki.api/mediawiki.api.category.js * mediawiki.api.edit resources/mediawiki.api/mediawiki.api.edit.js * mediawiki.api.login resources/mediawiki.api/mediawiki.api.login.js * mediawiki.api.parse resources/mediawiki.api/mediawiki.api.parse.js * mediawiki.api.watch resources/mediawiki.api/mediawiki.api.watch.js * mediawiki.debug resources/mediawiki/mediawiki.debug.js resources/mediawiki/mediawiki.debug.less resources/mediawiki/mediawiki.debug.profile.css resources/mediawiki/mediawiki.debug.profile.js * mediawiki.debug.init resources/mediawiki/mediawiki.debug.init.js * mediawiki.feedback resources/mediawiki/mediawiki.feedback.css resources/mediawiki/mediawiki.feedback.js resources/mediawiki/mediawiki.feedback.spinner.gif * mediawiki.hidpi resources/mediawiki/mediawiki.hidpi.js * mediawiki.hlist resources/mediawiki/mediawiki.hlist.css resources/mediawiki/mediawiki.hlist.js * mediawiki.htmlform resources/mediawiki/mediawiki.htmlform.js * mediawiki.icon resources/mediawiki/mediawiki.icon.less resources/mediawiki/images/arrow-collapsed-ltr.png resources/mediawiki/images/arrow-collapsed-ltr.svg resources/mediawiki/images/arrow-collapsed-rtl.png resources/mediawiki/images/arrow-collapsed-rtl.svg resources/mediawiki/images/arrow-expanded.png resources/mediawiki/images/arrow-expanded.svg * mediawiki.inspect resources/mediawiki/mediawiki.inspect.js * mediawiki.notification resources/mediawiki/mediawiki.notification.css resources/mediawiki/mediawiki.notification.hideForPrint.css resources/mediawiki/mediawiki.notification.js * mediawiki.notify resources/mediawiki/mediawiki.notify.js * mediawiki.searchSuggest resources/mediawiki/mediawiki.searchSuggest.css resources/mediawiki/mediawiki.searchSuggest.js * mediawiki.Title resources/mediawiki/mediawiki.Title.js * mediawiki.toc resources/mediawiki/mediawiki.toc.js * mediawiki.Uri resources/mediawiki/mediawiki.Uri.js * mediawiki.user resources/mediawiki/mediawiki.user.js * mediawiki.util resources/mediawiki/mediawiki.util.js * mediawiki.action.edit resources/mediawiki.action/mediawiki.action.edit.js * mediawiki.action.edit.styles resources/mediawiki.action/mediawiki.action.edit.styles.css * mediawiki.action.edit.collapsibleFooter resources/mediawiki.action/mediawiki.action.edit.collapsibleFooter.css resources/mediawiki.action/mediawiki.action.edit.collapsibleFooter.js * mediawiki.action.edit.preview resources/mediawiki.action/mediawiki.action.edit.preview.js * mediawiki.action.history resources/mediawiki.action/mediawiki.action.history.js * mediawiki.action.history.diff resources/mediawiki.action/mediawiki.action.history.diff.css * mediawiki.action.view.dblClickEdit resources/mediawiki.action/mediawiki.action.view.dblClickEdit.js * mediawiki.action.view.metadata resources/mediawiki.action/mediawiki.action.view.metadata.js * mediawiki.action.view.postEdit resources/mediawiki.action/mediawiki.action.view.postEdit.css resources/mediawiki.action/mediawiki.action.view.postEdit.js resources/mediawiki.action/images/green-checkmark.png * mediawiki.action.view.redirectToFragment resources/mediawiki.action/mediawiki.action.view.redirectToFragment.js * mediawiki.action.view.rightClickEdit resources/mediawiki.action/mediawiki.action.view.rightClickEdit.js * mediawiki.action.edit.editWarning resources/mediawiki.action/mediawiki.action.edit.editWarning.js * mediawiki.action.watch.ajax No files associated with this resource. * mediawiki.language resources/mediawiki.language/mediawiki.language.js resources/mediawiki.language/mediawiki.language.numbers.js resources/mediawiki.language/languages/bs.js resources/mediawiki.language/languages/dsb.js resources/mediawiki.language/languages/fi.js resources/mediawiki.language/languages/ga.js resources/mediawiki.language/languages/he.js resources/mediawiki.language/languages/hsb.js resources/mediawiki.language/languages/hu.js resources/mediawiki.language/languages/hy.js resources/mediawiki.language/languages/la.js resources/mediawiki.language/languages/os.js resources/mediawiki.language/languages/ru.js resources/mediawiki.language/languages/sl.js resources/mediawiki.language/languages/uk.js * mediawiki.cldr resources/mediawiki.language/mediawiki.cldr.js * mediawiki.libs.pluralruleparser Possibly should be moved to /lib resources/mediawiki.libs/CLDRPluralRuleParser.js * mediawiki.language.init resources/mediawiki.language/mediawiki.language.init.js * mediawiki.jqueryMsg resources/mediawiki/mediawiki.jqueryMsg.js resources/mediawiki/mediawiki.jqueryMsg.peg * mediawiki.language.months resources/mediawiki.language/mediawiki.language.months.js * mediawiki.libs.jpegmeta resources/mediawiki.libs/mediawiki.libs.jpegmeta.js * mediawiki.page.gallery resources/mediawiki.page/mediawiki.page.gallery.js * mediawiki.page.ready resources/mediawiki.page/mediawiki.page.ready.js * mediawiki.page.startup resources/mediawiki.page/mediawiki.page.startup.js * mediawiki.page.patrol.ajax resources/mediawiki.page/mediawiki.page.patrol.ajax.js * mediawiki.page.watch.ajax resources/mediawiki.page/mediawiki.page.watch.ajax.js * mediawiki.page.image.pagination resources/mediawiki.page/mediawiki.page.image.pagination.js * mediawiki.special resources/mediawiki.special/mediawiki.special.css resources/mediawiki.special/mediawiki.special.js * mediawiki.special.block resources/mediawiki.special/mediawiki.special.block.css resources/mediawiki.special/mediawiki.special.block.js * mediawiki.special.changeemail resources/mediawiki.special/mediawiki.special.changeemail.css resources/mediawiki.special/mediawiki.special.changeemail.js * mediawiki.special.changeslist resources/mediawiki.special/mediawiki.special.changeslist.css * mediawiki.special.changeslist.legend resources/mediawiki.special/mediawiki.special.changeslist.legend.css * mediawiki.special.changeslist.legend.js resources/mediawiki.special/mediawiki.special.changeslist.legend.js * mediawiki.special.changeslist.enhanced resources/mediawiki.special/mediawiki.special.changeslist.enhanced.css * mediawiki.special.movePage resources/mediawiki.special/mediawiki.special.movePage.js * mediawiki.special.pagesWithProp resources/mediawiki.special/mediawiki.special.pagesWithProp.css * mediawiki.special.preferences resources/mediawiki.special/mediawiki.special.preferences.css resources/mediawiki.special/mediawiki.special.preferences.js * mediawiki.special.recentchanges resources/mediawiki.special/mediawiki.special.recentchanges.js * mediawiki.special.search resources/mediawiki.special/mediawiki.special.search.css resources/mediawiki.special/mediawiki.special.search.js * mediawiki.special.undelete resources/mediawiki.special/mediawiki.special.undelete.js * mediawiki.special.upload resources/mediawiki.special/mediawiki.special.upload.js * mediawiki.special.userlogin.common.styles resources/mediawiki.special/mediawiki.special.userlogin.common.css resources/mediawiki.special/images/icon-lock.png * mediawiki.special.userlogin.signup.styles resources/mediawiki.special/mediawiki.special.userlogin.signup.css resources/mediawiki.special/images/icon-edits.png resources/mediawiki.special/images/icon-pages.png resources/mediawiki.special/images/icon-contributors.png * mediawiki.special.userlogin.login.styles resources/mediawiki.special/mediawiki.special.userlogin.login.css resources/mediawiki.special/images/glyph-people-large.png * mediawiki.special.userlogin.common.js resources/mediawiki.special/mediawiki.special.userlogin.common.js * mediawiki.special.userlogin.signup.js resources/mediawiki.special/mediawiki.special.userlogin.signup.js * mediawiki.special.javaScriptTest resources/mediawiki.special/mediawiki.special.javaScriptTest.js * mediawiki.special.version resources/mediawiki.special/mediawiki.special.version.css * mediawiki.ui resources/mediawiki.ui/default.less resources/mediawiki.ui/vector.less resources/mediawiki.ui/components/utilities.less resources/mediawiki.ui/components/default/forms.less resources/mediawiki.ui/components/vector/containers.less resources/mediawiki.ui/components/vector/forms.less resources/mediawiki.ui/mixins/effects.less resources/mediawiki.ui/mixins/forms.less resources/mediawiki.ui/mixins/type.less resources/mediawiki.ui/mixins/utilities.less resources/mediawiki.ui/settings/colors.less resources/mediawiki.ui/settings/typography.less resources/mediawiki.ui/styleguide.md * mediawiki.ui.button resources/mediawiki.ui/components/default/buttons.less resources/mediawiki.ui/components/vector/buttons.less Legacy modules, untouched: * mediawiki.legacy.ajax * mediawiki.legacy.commonPrint * mediawiki.legacy.config * mediawiki.legacy.protect * mediawiki.legacy.shared * mediawiki.legacy.oldshared * mediawiki.legacy.upload * mediawiki.legacy.wikibits Miscellaneous files moved elsewhere * A currently-failing (unused?) script attempting to build some kss style documentation. Previously in the root of the resources directory, now moved to a new folder, /docs/kss. Makefile README.txt package.json scripts/nodecheck.sh styleguide-template/index.html styleguide-template/public/kss.less styleguide-template/public/less.js * ResourceLoader startup JS code was also moved to /src resources/startup.js * mediawiki.mixins, hard-coded to be loaded by ResourceLoader, also moved to /src resources/mediawiki.less/mediawiki.mixins.animation.less resources/mediawiki.less/mediawiki.mixins.less resources/mediawiki.less/mediawiki.mixins.rotation.less * jquery.cycle.all.js appears not to be refered to (?!): (added in 2f1f0abc6b) resources/jquery/jquery.cycle.all.js Change-Id: Id5ead1453412e763bdb3763084e43694517088fd
2014-04-03 16:17:09 +00:00
$out = file_get_contents( "$IP/resources/src/startup.js" );
$pairs = array_map( function ( $value ) {
$value = FormatJson::encode( $value, ResourceLoader::inDebugMode(), FormatJson::ALL_OK );
// Fix indentation
$value = str_replace( "\n", "\n\t", $value );
return $value;
}, [
'$VARS.wgLegacyJavaScriptGlobals' => $this->getConfig()->get( 'LegacyJavaScriptGlobals' ),
'$VARS.configuration' => $this->getConfigSettings( $context ),
resourceloader: Async all the way Page startup: * Due to the startup module and top queue being asynchronous now, move client-nojs/client-js class handling to OutputPage to ensure there is no flashes of wrongly styled or unstyled content. To preserve compatibility for unsupported browsers, undo the class swap at runtime after the isCompatible() check. ResourceLoader startup module: * Load the startup module with <script async>. * Use DOM methods instead of 'document.write' to create base module request (jquery|mediawiki). mw.loader: * Drop 'async' parameter from mw.loader.load(). * Remove the now-unused code paths for synchronous requests. OutputPage: * Drop '$loadCall' parameter from makeResourceLoaderLink(). Asynchronous is now the default and only way to load JavaScript. This means the 'user' module "conditional document-write scripts" are now a simple "mw.loader.load( url )" call. * Fix incorrect @return of makeResourceLoaderLink(). This returns an array not a string. * Improve documentation of makeResourceLoaderLink(). * Drop '$inHead' parameter from getScriptsForBottomQueue(). No longer used. Compatibility with the $wgResourceLoaderExperimentalAsyncLoading feature is maintained. It just no longer needs to change the way the queue works since it's always asynchronous. The feature flag now only controls whether the bottom queue starts at the bottom or starts at the top. * Remove jQuery.ready() optimisation. This was mostly there to avoid the setTimeout() loop jQuery does to detect dom-ready in IE6/IE7 (which we no longer serve JavaScript at all). And for a bug in Firefox with document.write (which is no longer used as of this commit). Bug: T107399 Change-Id: Icba6d7a87b239bf127a221bc6bc432cfa71a4a72
2015-07-28 02:46:00 +00:00
'$VARS.baseModulesUri' => self::getStartupModulesUrl( $context ),
] );
$pairs['$CODE.registrations()'] = str_replace(
"\n",
"\n\t",
trim( $this->getModuleRegistrations( $context ) )
);
return strtr( $out, $pairs );
}
/**
* @return bool
*/
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
public function supportsURLLoading() {
return false;
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Get the definition summary for this module.
*
* @param ResourceLoaderContext $context
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @return array
*/
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
public function getDefinitionSummary( ResourceLoaderContext $context ) {
global $IP;
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
$summary = parent::getDefinitionSummary( $context );
$summary[] = [
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Detect changes to variables exposed in mw.config (T30899).
'vars' => $this->getConfigSettings( $context ),
// Changes how getScript() creates mw.Map for mw.config
'wgLegacyJavaScriptGlobals' => $this->getConfig()->get( 'LegacyJavaScriptGlobals' ),
// Detect changes to the module registrations
'moduleHashes' => $this->getAllModuleHashes( $context ),
'fileMtimes' => [
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
filemtime( "$IP/resources/src/startup.js" ),
],
];
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
return $summary;
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Helper method for getDefinitionSummary().
*
* @param ResourceLoaderContext $context
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @return string SHA-1
*/
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
protected function getAllModuleHashes( ResourceLoaderContext $context ) {
$rl = $context->getResourceLoader();
// Preload for getCombinedVersion()
$rl->preloadModuleInfo( $rl->getModuleNames(), $context );
// ATTENTION: Because of the line below, this is not going to cause infinite recursion.
// Think carefully before making changes to this code!
// Pre-populate versionHash with something because the loop over all modules below includes
// the startup module (this module).
// See ResourceLoaderModule::getVersionHash() for usage of this cache.
$this->versionHash[$context->getHash()] = null;
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
return $rl->getCombinedVersion( $context, $rl->getModuleNames() );
}
/**
* @return string
*/
public function getGroup() {
return 'startup';
}
}