wiki.techinc.nl/includes/resourceloader/ResourceLoaderModule.php

945 lines
29 KiB
PHP
Raw Normal View History

<?php
/**
* Abstraction for ResourceLoader modules.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
* http://www.gnu.org/copyleft/gpl.html
2010-09-04 12:53:01 +00:00
*
2010-09-05 13:31:34 +00:00
* @file
* @author Trevor Parscal
* @author Roan Kattouw
*/
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
use Psr\Log\LoggerAwareInterface;
use Psr\Log\LoggerInterface;
use Psr\Log\NullLogger;
/**
* Abstraction for ResourceLoader modules, with name registration and maxage functionality.
*/
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
abstract class ResourceLoaderModule implements LoggerAwareInterface {
# Type of resource
const TYPE_SCRIPTS = 'scripts';
const TYPE_STYLES = 'styles';
const TYPE_COMBINED = 'combined';
# sitewide core module like a skin file or jQuery component
const ORIGIN_CORE_SITEWIDE = 1;
# per-user module generated by the software
const ORIGIN_CORE_INDIVIDUAL = 2;
# sitewide module generated from user-editable files, like MediaWiki:Common.js, or
# modules accessible to multiple users, such as those generated by the Gadgets extension.
const ORIGIN_USER_SITEWIDE = 3;
# per-user module generated from user-editable files, like User:Me/vector.js
const ORIGIN_USER_INDIVIDUAL = 4;
# an access constant; make sure this is kept as the largest number in this group
const ORIGIN_ALL = 10;
# script and style modules form a hierarchy of trustworthiness, with core modules like
# skins and jQuery as most trustworthy, and user scripts as least trustworthy. We can
# limit the types of scripts and styles we allow to load on, say, sensitive special
# pages like Special:UserLogin and Special:Preferences
protected $origin = self::ORIGIN_CORE_SITEWIDE;
/* Protected Members */
2010-09-04 12:53:01 +00:00
protected $name = null;
protected $targets = array( 'desktop' );
// In-object cache for file dependencies
protected $fileDeps = array();
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
// In-object cache for message blob (keyed by language)
protected $msgBlobs = array();
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// In-object cache for version hash
protected $versionHash = array();
// In-object cache for module content
protected $contents = array();
2010-09-04 12:53:01 +00:00
/**
* @var Config
*/
protected $config;
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
/**
* @var LoggerInterface
*/
protected $logger;
/* Methods */
2010-09-04 12:53:01 +00:00
/**
* Get this module's name. This is set when the module is registered
* with ResourceLoader::register()
2010-09-05 13:31:34 +00:00
*
* @return string|null Name (string) or null if no name was set
*/
public function getName() {
return $this->name;
}
2010-09-04 12:53:01 +00:00
/**
* Set this module's name. This is called by ResourceLoader::register()
* when registering the module. Other code should not call this.
2010-09-05 13:31:34 +00:00
*
* @param string $name Name
*/
public function setName( $name ) {
$this->name = $name;
}
2010-09-04 12:53:01 +00:00
/**
* Get this module's origin. This is set when the module is registered
* with ResourceLoader::register()
*
* @return int ResourceLoaderModule class constant, the subclass default
* if not set manually
*/
public function getOrigin() {
return $this->origin;
}
/**
* Set this module's origin. This is called by ResourceLoader::register()
* when registering the module. Other code should not call this.
*
* @param int $origin Origin
*/
public function setOrigin( $origin ) {
$this->origin = $origin;
}
/**
* @param ResourceLoaderContext $context
* @return bool
*/
public function getFlip( $context ) {
global $wgContLang;
return $wgContLang->getDir() !== $context->getDirection();
}
2010-09-04 12:53:01 +00:00
/**
* Get all JS for this module for a given language and skin.
* Includes all relevant JS except loader scripts.
2010-09-05 13:31:34 +00:00
*
* @param ResourceLoaderContext $context
* @return string JavaScript code
*/
public function getScript( ResourceLoaderContext $context ) {
// Stub, override expected
return '';
}
/**
* Takes named templates by the module and returns an array mapping.
*
* @return array of templates mapping template alias to content
*/
public function getTemplates() {
// Stub, override expected.
return array();
}
/**
* @return Config
* @since 1.24
*/
public function getConfig() {
if ( $this->config === null ) {
// Ugh, fall back to default
$this->config = ConfigFactory::getDefaultInstance()->makeConfig( 'main' );
}
return $this->config;
}
/**
* @param Config $config
* @since 1.24
*/
public function setConfig( Config $config ) {
$this->config = $config;
}
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
/**
* @since 1.27
* @param LoggerInterface $logger
* @return null
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
*/
public function setLogger( LoggerInterface $logger ) {
$this->logger = $logger;
}
/**
* @since 1.27
* @return LoggerInterface
*/
protected function getLogger() {
if ( !$this->logger ) {
$this->logger = new NullLogger();
}
return $this->logger;
}
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
/**
* Get the URL or URLs to load for this module's JS in debug mode.
* The default behavior is to return a load.php?only=scripts URL for
* the module, but file-based modules will want to override this to
* load the files directly.
*
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
* This function is called only when 1) we're in debug mode, 2) there
* is no only= parameter and 3) supportsURLLoading() returns true.
* #2 is important to prevent an infinite loop, therefore this function
* MUST return either an only= URL or a non-load.php URL.
*
* @param ResourceLoaderContext $context
* @return array Array of URLs
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
*/
public function getScriptURLsForDebug( ResourceLoaderContext $context ) {
$resourceLoader = $context->getResourceLoader();
$derivative = new DerivativeResourceLoaderContext( $context );
$derivative->setModules( array( $this->getName() ) );
$derivative->setOnly( 'scripts' );
$derivative->setDebug( true );
$url = $resourceLoader->createLoaderURL(
$this->getSource(),
$derivative
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
);
return array( $url );
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
}
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
/**
* Whether this module supports URL loading. If this function returns false,
* getScript() will be used even in cases (debug mode, no only param) where
* getScriptURLsForDebug() would normally be used instead.
* @return bool
*/
public function supportsURLLoading() {
return true;
}
2010-09-04 12:53:01 +00:00
/**
* Get all CSS for this module for a given skin.
2010-09-05 13:31:34 +00:00
*
* @param ResourceLoaderContext $context
* @return array List of CSS strings or array of CSS strings keyed by media type.
ResourceLoader: Refactor style loading Fixes: * bug 31676: Work around IE stylesheet limit. * bug 35562: @import styles broken in modules that combine multiple stylesheets. * bug 40498: Don't output empty "@media print { }" blocks. * bug 40500: Don't ignore media-type for urls in debug mode. Approach: * Re-use the same <style> tag so that we stay under the 31 stylesheet limit in IE. Unless the to-be-added css text from the being-loaded module contains @import, in which case we do create a new <style> tag and then re-use that one from that point on (bug 31676). * Return stylesheets as arrays, instead of a concatenated string. This fixes bug 35562, because @import only works when at the top of a stylesheet. By not unconditionally concatenating files within a module on the server side already, @import will work in e.g. module 'site' that contains 2 wiki pages. This is normalized in ResourceLoader::makeCombinedStyles(), so far only ResourceLoaderWikiModule makes use of this. Misc. clean up and bug fixes: * Reducing usage of jQuery() and mw.html.element() where native DOM would be very simple and faster. Aside from simplicity and speed, this is also working towards a more stand-alone ResourceLoader. * Trim server output a little bit more - Redundant new line after minify-css (it is now an array, so no need to keep space afterwards) - Redundant semi-colon after minify-js if it ends in a colon * Allow space in styleTest.css.php * Clean up and extend unit tests to cover for these features and bug fixes. * Don't set styleEl.rel = 'stylesheet'; that has no business on a <style> tag. * Fix bug in mw.loader's addStyleTag(). It turns out IE6 has an odd security measure that does not allow manipulation of elements (at least style tags) that are created by a different script (even if that script was served from the same domain/origin etc.). We didn't ran into this before because we only created new style tags, never appended to them. Now that we do, this came up. Took a while to figure out because it was created by mediawiki.js but it calls jQuery which did the actual dom insertion. Odd thing is, we load jquery.js and mediawiki.js in the same request even... Without this all css-url related mw.loader tests would fail in IE6. * mediawiki.js and mediawiki.test.js now pass jshint again. Tested (and passing qunit/?module=mediawiki; 123 of 123): * Chrome 14, 21 * Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta * IE 6, 7, 8, 9 * Safari 4.0, 5.0, 5.1 * Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta * iPhone 3GS / iOS 3.0 / Mobile Safari 4.0 iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5 iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0 Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
2012-07-25 21:20:21 +00:00
* like array( 'screen' => '.foo { width: 0 }' );
* or array( 'screen' => array( '.foo { width: 0 }' ) );
*/
public function getStyles( ResourceLoaderContext $context ) {
// Stub, override expected
return array();
}
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
/**
* Get the URL or URLs to load for this module's CSS in debug mode.
* The default behavior is to return a load.php?only=styles URL for
* the module, but file-based modules will want to override this to
* load the files directly. See also getScriptURLsForDebug()
*
* @param ResourceLoaderContext $context
* @return array Array( mediaType => array( URL1, URL2, ... ), ... )
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
*/
public function getStyleURLsForDebug( ResourceLoaderContext $context ) {
$resourceLoader = $context->getResourceLoader();
$derivative = new DerivativeResourceLoaderContext( $context );
$derivative->setModules( array( $this->getName() ) );
$derivative->setOnly( 'styles' );
$derivative->setDebug( true );
$url = $resourceLoader->createLoaderURL(
$this->getSource(),
$derivative
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
);
return array( 'all' => array( $url ) );
Fix the fixme on r88053: dependency handling was broken in debug mode in certain cases. More specifically, if A is a file module that depends on B, B is a wiki module that depends on C and C is a file module, the loading order is CBA (correct) in production mode but was BCA (wrong) in debug mode. Fixed this by URL-ifying scripts and styles for those modules in debug mode, as I said to on CR. What this means is that the initial debug=true request for a module will now always return arrays of URLs, never the JS or CSS itself. This was already the case for file modules (which returned arrays of URLs to the raw files), but not for other modules (which returned the JS and CSS itself). So for non-file modules, load.php?modules=foo&debug=true now returns some JS that instructs the loader to fetch the module's JS from load.php?modules=foo&debug=true&only=scripts and the CSS from ...&only=styles . * Removed the magic behavior where ResourceLoaderModule::getScripts() and getStyles() could return an array of URLs where the documentation said they should return a JS/CSS string. Because I didn't restructure the calling code too much, the old magical behavior should still work. * Instead, move this behavior to getScriptURLsForDebug() and getStyleURLsForDebug(). The default implementation constructs a single URL for a load.php request for the module with debug=true&only=scripts (or styles). The URL building code duplicates some things from OutputPage::makeResourceLoaderLink(), I'll clean that up later. ResourceLoaderFileModule overrides this method to return URLs to the raw files, using code that I removed from getScripts()/getStyles() * Add ResourceLoaderModule::supportsURLLoading(), which returns true by default but may return false to indicate that a module does not support loading via a URL. This is needed to respect $this->debugRaw in ResourceLoaderFileModule (set to true for jquery and mediawiki), and obviously for the startup module as well, because we get bootstrapping problems otherwise (can't call mw.loader.implement() when the code for mw.loader isn't loaded yet)
2011-09-13 17:13:53 +00:00
}
2010-09-04 12:53:01 +00:00
/**
* Get the messages needed for this module.
*
* To get a JSON blob with messages, use MessageBlobStore::get()
2010-09-05 13:31:34 +00:00
*
* @return array List of message keys. Keys may occur more than once
*/
public function getMessages() {
// Stub, override expected
return array();
}
/**
* Get the group this module is in.
*
* @return string Group name
*/
public function getGroup() {
// Stub, override expected
return null;
}
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
/**
* Get the origin of this module. Should only be overridden for foreign modules.
*
* @return string Origin name, 'local' for local modules
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
*/
public function getSource() {
// Stub, override expected
return 'local';
}
/**
* Where on the HTML page should this module's JS be loaded?
* - 'top': in the "<head>"
* - 'bottom': at the bottom of the "<body>"
*
* @return string
*/
public function getPosition() {
return 'bottom';
}
2010-09-04 12:53:01 +00:00
/**
* Whether this module's JS expects to work without the client-side ResourceLoader module.
* Returning true from this function will prevent mw.loader.state() call from being
* appended to the bottom of the script.
*
* @return bool
*/
public function isRaw() {
return false;
}
/**
* Get a list of modules this module depends on.
*
* Dependency information is taken into account when loading a module
* on the client side.
*
resourceloader: Add context param to ResourceLoaderModule::getDependencies By providing context as a parameter in getDependencies, we allow modules to dyanamically determine dependencies based on context. Note: To ease rollout, the parameter is optional in this patch. It is expected that it will be made non-optional in the near future. The use case is for CentralNotice campaigns to be able to add special modules ahead of deciding which banner to show a user. The dynamically chosen RL modules would replace ad-hoc JS currently sent with some banners. A list of possible campaigns and banners is already sent as a PHP- implemented RL module; that's the module that will dynamically choose other modules as dependencies when appropriate. This approach will save a round trip as compared to dynamically loading the modules client-side. For compatibility, extensions that override ResourceLoaderModule::getDependencies() should be updated with the new method signature. Here are changes for extensions currently deployed on Wikimedia wikis: * CentralNotice: I816bffa3815e2eab7e88cb04d1b345070e6aa15f * Gadgets: I0a10fb0cbf17d095ece493e744296caf13dcee02 * EventLogging: I67e957f74d6ca48cfb9a41fb5144bcc78f885e50 * PageTriage: Ica3ba32aa2fc76d11a44f391b6edfc871e7fbe0d * UniversalLanguageSelector: Ic63e617f51702c27104e123d4bed91983a726b7f * VisualEditor: I0ac775ca286e64825e31a9213b94648e41a5bc30 For more on the CentralNotice use case, please see I9f80edcbcacca2. Bug: T98924 Change-Id: Iee61e5b527321d01287baa03ad9b4d4f526ff3ef
2015-04-08 21:34:08 +00:00
* Note: It is expected that $context will be made non-optional in the near
* future.
*
* @param ResourceLoaderContext $context
* @return array List of module names as strings
*/
resourceloader: Add context param to ResourceLoaderModule::getDependencies By providing context as a parameter in getDependencies, we allow modules to dyanamically determine dependencies based on context. Note: To ease rollout, the parameter is optional in this patch. It is expected that it will be made non-optional in the near future. The use case is for CentralNotice campaigns to be able to add special modules ahead of deciding which banner to show a user. The dynamically chosen RL modules would replace ad-hoc JS currently sent with some banners. A list of possible campaigns and banners is already sent as a PHP- implemented RL module; that's the module that will dynamically choose other modules as dependencies when appropriate. This approach will save a round trip as compared to dynamically loading the modules client-side. For compatibility, extensions that override ResourceLoaderModule::getDependencies() should be updated with the new method signature. Here are changes for extensions currently deployed on Wikimedia wikis: * CentralNotice: I816bffa3815e2eab7e88cb04d1b345070e6aa15f * Gadgets: I0a10fb0cbf17d095ece493e744296caf13dcee02 * EventLogging: I67e957f74d6ca48cfb9a41fb5144bcc78f885e50 * PageTriage: Ica3ba32aa2fc76d11a44f391b6edfc871e7fbe0d * UniversalLanguageSelector: Ic63e617f51702c27104e123d4bed91983a726b7f * VisualEditor: I0ac775ca286e64825e31a9213b94648e41a5bc30 For more on the CentralNotice use case, please see I9f80edcbcacca2. Bug: T98924 Change-Id: Iee61e5b527321d01287baa03ad9b4d4f526ff3ef
2015-04-08 21:34:08 +00:00
public function getDependencies( ResourceLoaderContext $context = null ) {
// Stub, override expected
return array();
}
/**
* Get target(s) for the module, eg ['desktop'] or ['desktop', 'mobile']
*
* @return array Array of strings
*/
public function getTargets() {
return $this->targets;
}
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
/**
* Get the skip function.
*
* Modules that provide fallback functionality can provide a "skip function". This
* function, if provided, will be passed along to the module registry on the client.
* When this module is loaded (either directly or as a dependency of another module),
* then this function is executed first. If the function returns true, the module will
* instantly be considered "ready" without requesting the associated module resources.
*
* The value returned here must be valid javascript for execution in a private function.
* It must not contain the "function () {" and "}" wrapper though.
*
* @return string|null A JavaScript function body returning a boolean value, or null
*/
public function getSkipFunction() {
return null;
}
/**
* Get the files this module depends on indirectly for a given skin.
*
* These are only image files referenced by the module's stylesheet.
*
* @param ResourceLoaderContext $context
* @return array List of files
*/
protected function getFileDependencies( ResourceLoaderContext $context ) {
$vary = $context->getSkin() . '|' . $context->getLanguage();
// Try in-object cache first
if ( !isset( $this->fileDeps[$vary] ) ) {
$dbr = wfGetDB( DB_SLAVE );
$deps = $dbr->selectField( 'module_deps',
'md_deps',
array(
'md_module' => $this->getName(),
'md_skin' => $vary,
),
__METHOD__
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
);
if ( !is_null( $deps ) ) {
$this->fileDeps[$vary] = self::expandRelativePaths(
(array)FormatJson::decode( $deps, true )
);
} else {
$this->fileDeps[$vary] = array();
}
}
return $this->fileDeps[$vary];
}
/**
* Set in-object cache for file dependencies.
*
* This is used to retrieve data in batches. See ResourceLoader::preloadModuleInfo().
* To save the data, use saveFileDependencies().
*
* @param ResourceLoaderContext $context
* @param string[] $files Array of file names
*/
public function setFileDependencies( ResourceLoaderContext $context, $files ) {
$vary = $context->getSkin() . '|' . $context->getLanguage();
$this->fileDeps[$vary] = $files;
}
/**
* Set the files this module depends on indirectly for a given skin.
*
* @since 1.27
* @param ResourceLoaderContext $context
* @param array $localFileRefs List of files
*/
protected function saveFileDependencies( ResourceLoaderContext $context, $localFileRefs ) {
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
// Normalise array
$localFileRefs = array_values( array_unique( $localFileRefs ) );
sort( $localFileRefs );
try {
// If the list has been modified since last time we cached it, update the cache
if ( $localFileRefs !== $this->getFileDependencies( $context ) ) {
$cache = ObjectCache::getLocalClusterInstance();
$key = $cache->makeKey( __METHOD__, $this->getName() );
$scopeLock = $cache->getScopedLock( $key, 0 );
if ( !$scopeLock ) {
return; // T124649; avoid write slams
}
$vary = $context->getSkin() . '|' . $context->getLanguage();
$dbw = wfGetDB( DB_MASTER );
$dbw->replace( 'module_deps',
array( array( 'md_module', 'md_skin' ) ),
array(
'md_module' => $this->getName(),
'md_skin' => $vary,
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
// Use relative paths to avoid ghost entries when $IP changes (T111481)
'md_deps' => FormatJson::encode( self::getRelativePaths( $localFileRefs ) ),
)
);
$dbw->onTransactionIdle( function () use ( &$scopeLock ) {
ScopedCallback::consume( $scopeLock ); // release after commit
} );
}
} catch ( Exception $e ) {
wfDebugLog( 'resourceloader', __METHOD__ . ": failed to update DB: $e" );
}
}
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
/**
* Make file paths relative to MediaWiki directory.
*
* This is used to make file paths safe for storing in a database without the paths
* becoming stale or incorrect when MediaWiki is moved or upgraded (T111481).
*
* @since 1.27
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
* @param array $filePaths
* @return array
*/
public static function getRelativePaths( Array $filePaths ) {
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
global $IP;
return array_map( function ( $path ) use ( $IP ) {
return RelPath\getRelativePath( $path, $IP );
}, $filePaths );
}
/**
* Expand directories relative to $IP.
*
* @since 1.27
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
* @param array $filePaths
* @return array
*/
public static function expandRelativePaths( Array $filePaths ) {
resourceloader: Store relative instead of absolute paths in module_deps Make paths stored in the module_deps table relative to $IP. This ensures that when the MediaWiki install path changes and/or if the location of the extension or skins directory changes, that ResourceLoader's internal model is still accurate. Previously when these paths change, ResourceLoader would exhibit various bugs. 1. Unable to detect changes in the module (if the directory no longer exists). 2. Point #1 is usually preceeded by one last cache invalidation as the content hash of the file path changes (from a valid hash to an empty string). 3. Unnecessary cache invalidation (if both old and new directories exist). This happens when a file is both an entry point (in the 'scripts' or 'styles' array) and also a file dependency. At first they are de-duplicated by array_unique. But after the disk path changes, the next check will result in the old path being fetched from module_deps, and the new path from the live configuration. This causes two changes that result in needless cache invalidation: - The hash list contains one more item (T111481). - The hash list contains both the old and new version of a file. (or even alternate versions, e.g. when a change is backported to the old wmf branch; it also affects wikis on the new branch due to the stale file path still in the database). It seems unusual to move a MediaWiki install, and usually we recommend third parties to run update.php if site administrators do move their wiki. However Wikimedia's deployment system implicitly moves the MediaWiki install continously from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6". This caused virtually all ResourceLoader caching layers to get invalidated every week when another wmf-branch is deployed, thus breaking these file paths, which changes the version hash, which then invalidates the cache. Bug: T111481 Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
2015-09-23 01:35:38 +00:00
global $IP;
return array_map( function ( $path ) use ( $IP ) {
return RelPath\joinPath( $IP, $path );
}, $filePaths );
}
/**
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
* Get the hash of the message blob.
*
* @since 1.27
* @param ResourceLoaderContext $context
* @return string|null JSON blob or null if module has no messages
*/
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
protected function getMessageBlob( ResourceLoaderContext $context ) {
if ( !$this->getMessages() ) {
// Don't bother consulting MessageBlobStore
return null;
}
// Message blobs may only vary language, not by context keys
$lang = $context->getLanguage();
if ( !isset( $this->msgBlobs[$lang] ) ) {
$this->getLogger()->warning( 'Message blob for {module} should have been preloaded', array(
'module' => $this->getName(),
) );
$store = $context->getResourceLoader()->getMessageBlobStore();
$this->msgBlobs[$lang] = $store->getBlob( $this, $lang );
}
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
return $this->msgBlobs[$lang];
}
/**
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
* Set in-object cache for message blobs.
*
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
* Used to allow fetching of message blobs in batches. See ResourceLoader::preloadModuleInfo().
*
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
* @since 1.27
* @param string|null $blob JSON blob or null
* @param string $lang Language code
*/
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
public function setMessageBlob( $blob, $lang ) {
$this->msgBlobs[$lang] = $blob;
}
2015-09-25 17:57:35 +00:00
/**
* Get module-specific LESS variables, if any.
*
* @since 1.27
2015-09-25 17:57:35 +00:00
* @param ResourceLoaderContext $context
* @return array Module-specific LESS variables.
*/
protected function getLessVars( ResourceLoaderContext $context ) {
return array();
}
/**
* Get an array of this module's resources. Ready for serving to the web.
*
* @since 1.26
* @param ResourceLoaderContext $context
* @return array
*/
public function getModuleContent( ResourceLoaderContext $context ) {
$contextHash = $context->getHash();
// Cache this expensive operation. This calls builds the scripts, styles, and messages
// content which typically involves filesystem and/or database access.
if ( !array_key_exists( $contextHash, $this->contents ) ) {
$this->contents[$contextHash] = $this->buildContent( $context );
}
return $this->contents[$contextHash];
}
/**
* Bundle all resources attached to this module into an array.
*
* @since 1.26
* @param ResourceLoaderContext $context
* @return array
*/
final protected function buildContent( ResourceLoaderContext $context ) {
$rl = $context->getResourceLoader();
$stats = RequestContext::getMain()->getStats();
$statStart = microtime( true );
// Only include properties that are relevant to this context (e.g. only=scripts)
// and that are non-empty (e.g. don't include "templates" for modules without
// templates). This helps prevent invalidating cache for all modules when new
// optional properties are introduced.
$content = array();
// Scripts
if ( $context->shouldIncludeScripts() ) {
// If we are in debug mode, we'll want to return an array of URLs if possible
// However, we can't do this if the module doesn't support it
// We also can't do this if there is an only= parameter, because we have to give
// the module a way to return a load.php URL without causing an infinite loop
if ( $context->getDebug() && !$context->getOnly() && $this->supportsURLLoading() ) {
$scripts = $this->getScriptURLsForDebug( $context );
} else {
$scripts = $this->getScript( $context );
// rtrim() because there are usually a few line breaks
// after the last ';'. A new line at EOF, a new line
// added by ResourceLoaderFileModule::readScriptFiles, etc.
if ( is_string( $scripts )
&& strlen( $scripts )
&& substr( rtrim( $scripts ), -1 ) !== ';'
) {
// Append semicolon to prevent weird bugs caused by files not
// terminating their statements right (bug 27054)
$scripts .= ";\n";
}
}
$content['scripts'] = $scripts;
}
// Styles
if ( $context->shouldIncludeStyles() ) {
$styles = array();
// Don't create empty stylesheets like array( '' => '' ) for modules
// that don't *have* any stylesheets (bug 38024).
$stylePairs = $this->getStyles( $context );
if ( count( $stylePairs ) ) {
// If we are in debug mode without &only= set, we'll want to return an array of URLs
// See comment near shouldIncludeScripts() for more details
if ( $context->getDebug() && !$context->getOnly() && $this->supportsURLLoading() ) {
$styles = array(
'url' => $this->getStyleURLsForDebug( $context )
);
} else {
// Minify CSS before embedding in mw.loader.implement call
// (unless in debug mode)
if ( !$context->getDebug() ) {
foreach ( $stylePairs as $media => $style ) {
// Can be either a string or an array of strings.
if ( is_array( $style ) ) {
$stylePairs[$media] = array();
foreach ( $style as $cssText ) {
if ( is_string( $cssText ) ) {
$stylePairs[$media][] =
ResourceLoader::filter( 'minify-css', $cssText );
}
}
} elseif ( is_string( $style ) ) {
$stylePairs[$media] = ResourceLoader::filter( 'minify-css', $style );
}
}
}
// Wrap styles into @media groups as needed and flatten into a numerical array
$styles = array(
'css' => $rl->makeCombinedStyles( $stylePairs )
);
}
}
$content['styles'] = $styles;
}
// Messages
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
$blob = $this->getMessageBlob( $context );
if ( $blob ) {
$content['messagesBlob'] = $blob;
}
$templates = $this->getTemplates();
if ( $templates ) {
$content['templates'] = $templates;
}
$statTiming = microtime( true ) - $statStart;
$statName = strtr( $this->getName(), '.', '_' );
$stats->timing( "resourceloader_build.all", 1000 * $statTiming );
$stats->timing( "resourceloader_build.$statName", 1000 * $statTiming );
return $content;
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Get a string identifying the current version of this module in a given context.
2010-09-05 13:31:34 +00:00
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Whenever anything happens that changes the module's response (e.g. scripts, styles, and
* messages) this value must change. This value is used to store module responses in cache.
* (Both client-side and server-side.)
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* It is not recommended to override this directly. Use getDefinitionSummary() instead.
* If overridden, one must call the parent getVersionHash(), append data and re-hash.
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* This method should be quick because it is frequently run by ResourceLoaderStartUpModule to
* propagate changes to the client and effectively invalidate cache.
*
* For backward-compatibility, the following optional data providers are automatically included:
*
* - getModifiedTime()
* - getModifiedHash()
*
* @since 1.26
* @param ResourceLoaderContext $context
* @return string Hash (should use ResourceLoader::makeHash)
*/
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
public function getVersionHash( ResourceLoaderContext $context ) {
// The startup module produces a manifest with versions representing the entire module.
// Typically, the request for the startup module itself has only=scripts. That must apply
// only to the startup module content, and not to the module version computed here.
$context = new DerivativeResourceLoaderContext( $context );
$context->setModules( array() );
// Version hash must cover all resources, regardless of startup request itself.
$context->setOnly( null );
// Compute version hash based on content, not debug urls.
$context->setDebug( false );
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Cache this somewhat expensive operation. Especially because some classes
// (e.g. startup module) iterate more than once over all modules to get versions.
$contextHash = $context->getHash();
if ( !array_key_exists( $contextHash, $this->versionHash ) ) {
resourceloader: Enable module content version for data modules This greatly simplifies logic required to compute module versions. It also makes it significantly less error-prone. Since f37cee996e, we support hashes as versions (instead of timestamps). This means we can build a hash of the content directly, instead of compiling a large array with all values that may influence the module content somehow. Benefits: * Remove all methods and logic related to querying database and disk for timestamps, revision numbers, definition summaries, cache epochs, and more. * No longer needlessly invalidate cache as a result of no-op changes to implementation datails. Due to inclusion of absolute file paths in the definition summary, cache was always invalidated when moving wikis to newer MediaWiki branches; even if the module observed no actual changes. * When changes are reverted within a certain period of time, old caches can now be re-used. The module would produce the same version hash as before. Previously when a change was deployed and then reverted, all web clients (even those that never saw the bad version) would have re-fetch modules because the version increased. Updated unit tests to account for the change in version. New default version of empty test modules is: "mvgTPvXh". For the record, this comes from the base64 encoding of the SHA1 digest of the JSON serialised form of the module content: > $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}'; > echo base64_encode(sha1($str, true)); > FEb3+VuiUm/fOMfod1bjw/te+AQ= Enabled content versioning for the data modules in MediaWiki core: * EditToolbarModule * JqueryMsgModule * LanguageDataModule * LanguageNamesModule * SpecialCharacterDataModule * UserCSSPrefsModule * UserDefaultsModule * UserOptionsModule The FileModule and base class explicitly disable it for now and keep their current behaviour of using the definition summary. We may remove it later, but that requires more performance testing first. Explicitly disable it in the WikiModule class to avoid breakage when the default changes. Ref T98087. Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-02 17:27:23 +00:00
if ( $this->enableModuleContentVersion() ) {
// Detect changes directly
$str = json_encode( $this->getModuleContent( $context ) );
} else {
// Infer changes based on definition and other metrics
$summary = $this->getDefinitionSummary( $context );
if ( !isset( $summary['_cacheEpoch'] ) ) {
throw new LogicException( 'getDefinitionSummary must call parent method' );
resourceloader: Enable module content version for data modules This greatly simplifies logic required to compute module versions. It also makes it significantly less error-prone. Since f37cee996e, we support hashes as versions (instead of timestamps). This means we can build a hash of the content directly, instead of compiling a large array with all values that may influence the module content somehow. Benefits: * Remove all methods and logic related to querying database and disk for timestamps, revision numbers, definition summaries, cache epochs, and more. * No longer needlessly invalidate cache as a result of no-op changes to implementation datails. Due to inclusion of absolute file paths in the definition summary, cache was always invalidated when moving wikis to newer MediaWiki branches; even if the module observed no actual changes. * When changes are reverted within a certain period of time, old caches can now be re-used. The module would produce the same version hash as before. Previously when a change was deployed and then reverted, all web clients (even those that never saw the bad version) would have re-fetch modules because the version increased. Updated unit tests to account for the change in version. New default version of empty test modules is: "mvgTPvXh". For the record, this comes from the base64 encoding of the SHA1 digest of the JSON serialised form of the module content: > $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}'; > echo base64_encode(sha1($str, true)); > FEb3+VuiUm/fOMfod1bjw/te+AQ= Enabled content versioning for the data modules in MediaWiki core: * EditToolbarModule * JqueryMsgModule * LanguageDataModule * LanguageNamesModule * SpecialCharacterDataModule * UserCSSPrefsModule * UserDefaultsModule * UserOptionsModule The FileModule and base class explicitly disable it for now and keep their current behaviour of using the definition summary. We may remove it later, but that requires more performance testing first. Explicitly disable it in the WikiModule class to avoid breakage when the default changes. Ref T98087. Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-02 17:27:23 +00:00
}
$str = json_encode( $summary );
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
resourceloader: Enable module content version for data modules This greatly simplifies logic required to compute module versions. It also makes it significantly less error-prone. Since f37cee996e, we support hashes as versions (instead of timestamps). This means we can build a hash of the content directly, instead of compiling a large array with all values that may influence the module content somehow. Benefits: * Remove all methods and logic related to querying database and disk for timestamps, revision numbers, definition summaries, cache epochs, and more. * No longer needlessly invalidate cache as a result of no-op changes to implementation datails. Due to inclusion of absolute file paths in the definition summary, cache was always invalidated when moving wikis to newer MediaWiki branches; even if the module observed no actual changes. * When changes are reverted within a certain period of time, old caches can now be re-used. The module would produce the same version hash as before. Previously when a change was deployed and then reverted, all web clients (even those that never saw the bad version) would have re-fetch modules because the version increased. Updated unit tests to account for the change in version. New default version of empty test modules is: "mvgTPvXh". For the record, this comes from the base64 encoding of the SHA1 digest of the JSON serialised form of the module content: > $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}'; > echo base64_encode(sha1($str, true)); > FEb3+VuiUm/fOMfod1bjw/te+AQ= Enabled content versioning for the data modules in MediaWiki core: * EditToolbarModule * JqueryMsgModule * LanguageDataModule * LanguageNamesModule * SpecialCharacterDataModule * UserCSSPrefsModule * UserDefaultsModule * UserOptionsModule The FileModule and base class explicitly disable it for now and keep their current behaviour of using the definition summary. We may remove it later, but that requires more performance testing first. Explicitly disable it in the WikiModule class to avoid breakage when the default changes. Ref T98087. Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-02 17:27:23 +00:00
$mtime = $this->getModifiedTime( $context );
if ( $mtime !== null ) {
// Support: MediaWiki 1.25 and earlier
$str .= strval( $mtime );
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
resourceloader: Enable module content version for data modules This greatly simplifies logic required to compute module versions. It also makes it significantly less error-prone. Since f37cee996e, we support hashes as versions (instead of timestamps). This means we can build a hash of the content directly, instead of compiling a large array with all values that may influence the module content somehow. Benefits: * Remove all methods and logic related to querying database and disk for timestamps, revision numbers, definition summaries, cache epochs, and more. * No longer needlessly invalidate cache as a result of no-op changes to implementation datails. Due to inclusion of absolute file paths in the definition summary, cache was always invalidated when moving wikis to newer MediaWiki branches; even if the module observed no actual changes. * When changes are reverted within a certain period of time, old caches can now be re-used. The module would produce the same version hash as before. Previously when a change was deployed and then reverted, all web clients (even those that never saw the bad version) would have re-fetch modules because the version increased. Updated unit tests to account for the change in version. New default version of empty test modules is: "mvgTPvXh". For the record, this comes from the base64 encoding of the SHA1 digest of the JSON serialised form of the module content: > $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}'; > echo base64_encode(sha1($str, true)); > FEb3+VuiUm/fOMfod1bjw/te+AQ= Enabled content versioning for the data modules in MediaWiki core: * EditToolbarModule * JqueryMsgModule * LanguageDataModule * LanguageNamesModule * SpecialCharacterDataModule * UserCSSPrefsModule * UserDefaultsModule * UserOptionsModule The FileModule and base class explicitly disable it for now and keep their current behaviour of using the definition summary. We may remove it later, but that requires more performance testing first. Explicitly disable it in the WikiModule class to avoid breakage when the default changes. Ref T98087. Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-02 17:27:23 +00:00
$mhash = $this->getModifiedHash( $context );
if ( $mhash !== null ) {
// Support: MediaWiki 1.25 and earlier
$str .= strval( $mhash );
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
}
$this->versionHash[$contextHash] = ResourceLoader::makeHash( $str );
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
}
return $this->versionHash[$contextHash];
}
resourceloader: Enable module content version for data modules This greatly simplifies logic required to compute module versions. It also makes it significantly less error-prone. Since f37cee996e, we support hashes as versions (instead of timestamps). This means we can build a hash of the content directly, instead of compiling a large array with all values that may influence the module content somehow. Benefits: * Remove all methods and logic related to querying database and disk for timestamps, revision numbers, definition summaries, cache epochs, and more. * No longer needlessly invalidate cache as a result of no-op changes to implementation datails. Due to inclusion of absolute file paths in the definition summary, cache was always invalidated when moving wikis to newer MediaWiki branches; even if the module observed no actual changes. * When changes are reverted within a certain period of time, old caches can now be re-used. The module would produce the same version hash as before. Previously when a change was deployed and then reverted, all web clients (even those that never saw the bad version) would have re-fetch modules because the version increased. Updated unit tests to account for the change in version. New default version of empty test modules is: "mvgTPvXh". For the record, this comes from the base64 encoding of the SHA1 digest of the JSON serialised form of the module content: > $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}'; > echo base64_encode(sha1($str, true)); > FEb3+VuiUm/fOMfod1bjw/te+AQ= Enabled content versioning for the data modules in MediaWiki core: * EditToolbarModule * JqueryMsgModule * LanguageDataModule * LanguageNamesModule * SpecialCharacterDataModule * UserCSSPrefsModule * UserDefaultsModule * UserOptionsModule The FileModule and base class explicitly disable it for now and keep their current behaviour of using the definition summary. We may remove it later, but that requires more performance testing first. Explicitly disable it in the WikiModule class to avoid breakage when the default changes. Ref T98087. Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
2015-06-02 17:27:23 +00:00
/**
* Whether to generate version hash based on module content.
*
* If a module requires database or file system access to build the module
* content, consider disabling this in favour of manually tracking relevant
* aspects in getDefinitionSummary(). See getVersionHash() for how this is used.
*
* @return bool
*/
public function enableModuleContentVersion() {
return false;
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Get the definition summary for this module.
*
* This is the method subclasses are recommended to use to track values in their
* version hash. Call this in getVersionHash() and pass it to e.g. json_encode.
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Subclasses must call the parent getDefinitionSummary() and build on that.
* It is recommended that each subclass appends its own new array. This prevents
* clashes or accidental overwrites of existing keys and gives each subclass
* its own scope for simple array keys.
*
* @code
* $summary = parent::getDefinitionSummary( $context );
* $summary[] = array(
* 'foo' => 123,
* 'bar' => 'quux',
* );
* return $summary;
* @endcode
*
* Return an array containing values from all significant properties of this
* module's definition.
*
* Be careful not to normalise too much. Especially preserve the order of things
* that carry significance in getScript and getStyles (T39812).
*
* Avoid including things that are insiginificant (e.g. order of message keys is
* insignificant and should be sorted to avoid unnecessary cache invalidation).
*
* This data structure must exclusively contain arrays and scalars as values (avoid
* object instances) to allow simple serialisation using json_encode.
*
* If modules have a hash or timestamp from another source, that may be incuded as-is.
*
* A number of utility methods are available to help you gather data. These are not
* called by default and must be included by the subclass' getDefinitionSummary().
*
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
* - getMessageBlob()
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
*
* @since 1.23
* @param ResourceLoaderContext $context
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @return array|null
*/
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
public function getDefinitionSummary( ResourceLoaderContext $context ) {
return array(
'_class' => get_class( $this ),
'_cacheEpoch' => $this->getConfig()->get( 'CacheEpoch' ),
);
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Get this module's last modification timestamp for a given context.
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @deprecated since 1.26 Use getDefinitionSummary() instead
* @param ResourceLoaderContext $context Context object
* @return int|null UNIX timestamp
*/
public function getModifiedTime( ResourceLoaderContext $context ) {
return null;
}
/**
* Helper method for providing a version hash to getVersionHash().
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @deprecated since 1.26 Use getDefinitionSummary() instead
* @param ResourceLoaderContext $context
* @return string|null Hash
*/
public function getModifiedHash( ResourceLoaderContext $context ) {
return null;
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Back-compat dummy for old subclass implementations of getModifiedTime().
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* This method used to use ObjectCache to track when a hash was first seen. That principle
* stems from a time that ResourceLoader could only identify module versions by timestamp.
* That is no longer the case. Use getDefinitionSummary() directly.
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @deprecated since 1.26 Superseded by getVersionHash()
* @param ResourceLoaderContext $context
* @return int UNIX timestamp
*/
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
public function getHashMtime( ResourceLoaderContext $context ) {
if ( !is_string( $this->getModifiedHash( $context ) ) ) {
return 1;
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Dummy that is > 1
return 2;
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Back-compat dummy for old subclass implementations of getModifiedTime().
*
* @since 1.23
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @deprecated since 1.26 Superseded by getVersionHash()
* @param ResourceLoaderContext $context
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @return int UNIX timestamp
*/
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
public function getDefinitionMtime( ResourceLoaderContext $context ) {
if ( $this->getDefinitionSummary( $context ) === null ) {
return 1;
}
// Dummy that is > 1
return 2;
}
/**
* Check whether this module is known to be empty. If a child class
* has an easy and cheap way to determine that this module is
* definitely going to be empty, it should override this method to
* return true in that case. Callers may optimize the request for this
* module away if this function returns true.
* @param ResourceLoaderContext $context
* @return bool
*/
public function isKnownEmpty( ResourceLoaderContext $context ) {
return false;
}
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
/** @var JSParser Lazy-initialized; use self::javaScriptParser() */
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
private static $jsParser;
private static $parseCacheVersion = 1;
/**
* Validate a given script file; if valid returns the original source.
* If invalid, returns replacement JS source that throws an exception.
*
* @param string $fileName
* @param string $contents
* @return string JS with the original, or a replacement error
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
*/
protected function validateScriptFile( $fileName, $contents ) {
if ( $this->getConfig()->get( 'ResourceLoaderValidateJS' ) ) {
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
// Try for cache hit
$cache = ObjectCache::getMainWANInstance();
$key = $cache->makeKey(
'resourceloader',
'jsparse',
self::$parseCacheVersion,
md5( $contents )
);
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
$cacheEntry = $cache->get( $key );
if ( is_string( $cacheEntry ) ) {
return $cacheEntry;
}
$parser = self::javaScriptParser();
try {
$parser->parse( $contents, $fileName, 1 );
$result = $contents;
} catch ( Exception $e ) {
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
// We'll save this to cache to avoid having to validate broken JS over and over...
$err = $e->getMessage();
$result = "mw.log.error(" .
Xml::encodeJsVar( "JavaScript parse error: $err" ) . ");";
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
}
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
$cache->set( $key, $result );
return $result;
} else {
return $contents;
}
}
/**
* @return JSParser
*/
* (bug 28626) Validate JavaScript files and pages loaded via ResourceLoader before minification, protecting separate modules from interference This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits. - $wgResourceLoaderValidateJs on by default (can be disabled) - when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser. - parsing results are cached based on md5 of content to avoid re-parsing identical text - for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log Issues: - the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget). - have not done performance testing on the JSParser - have not done thorough unit testing with the JSParser
2011-07-06 21:48:09 +00:00
protected static function javaScriptParser() {
if ( !self::$jsParser ) {
self::$jsParser = new JSParser();
}
return self::$jsParser;
}
/**
* Safe version of filemtime(), which doesn't throw a PHP warning if the file doesn't exist.
* Defaults to 1.
*
* @param string $filePath File path
* @return int UNIX timestamp
*/
protected static function safeFilemtime( $filePath ) {
MediaWiki\suppressWarnings();
$mtime = filemtime( $filePath ) ?: 1;
MediaWiki\restoreWarnings();
return $mtime;
}
/**
* Compute a non-cryptographic string hash of a file's contents.
* If the file does not exist or cannot be read, returns an empty string.
*
* @since 1.26 Uses MD4 instead of SHA1.
* @param string $filePath File path
* @return string Hash
*/
protected static function safeFileHash( $filePath ) {
return FileContentsHasher::getFileContentsHash( $filePath );
}
}