wiki.techinc.nl/includes/resourceloader/ResourceLoader.php

1775 lines
55 KiB
PHP
Raw Normal View History

<?php
/**
* Base class for resource loading system.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
* http://www.gnu.org/copyleft/gpl.html
*
2010-09-05 13:31:34 +00:00
* @file
* @author Roan Kattouw
* @author Trevor Parscal
*/
use MediaWiki\MediaWikiServices;
use Psr\Log\LoggerAwareInterface;
use Psr\Log\LoggerInterface;
use Psr\Log\NullLogger;
use Wikimedia\Rdbms\DBConnectionError;
use Wikimedia\WrappedString;
2010-09-05 13:31:34 +00:00
/**
* Dynamic JavaScript and CSS resource loading system.
*
* Most of the documentation is on the MediaWiki documentation wiki starting at:
* https://www.mediawiki.org/wiki/ResourceLoader
*/
class ResourceLoader implements LoggerAwareInterface {
/** @var int */
const CACHE_VERSION = 8;
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
/** @var bool */
protected static $debugMode = null;
/**
* Module name/ResourceLoaderModule object pairs
* @var array
*/
protected $modules = [];
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
/**
* Associative array mapping module name to info associative array
* @var array
*/
protected $moduleInfos = [];
/** @var Config $config */
protected $config;
/**
* Associative array mapping framework ids to a list of names of test suite modules
* like [ 'qunit' => [ 'mediawiki.tests.qunit.suites', 'ext.foo.tests', ... ], ... ]
* @var array
*/
protected $testModuleNames = [];
2010-09-04 12:53:01 +00:00
/**
* E.g. [ 'source-id' => 'http://.../load.php' ]
* @var array
*/
protected $sources = [];
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
/**
* Errors accumulated during current respond() call.
* @var array
*/
protected $errors = [];
/**
* List of extra HTTP response headers provided by loaded modules.
*
* Populated by makeModuleResponse().
*
* @var array
*/
protected $extraHeaders = [];
/**
* @var MessageBlobStore
*/
protected $blobStore;
/**
* @var LoggerInterface
*/
private $logger;
/** @var string JavaScript / CSS pragma to disable minification. **/
const FILTER_NOMIN = '/*@nomin*/';
/**
* Load information stored in the database about modules.
*
* This method grabs modules dependencies from the database and updates modules
* objects.
*
* This is not inside the module code because it is much faster to
* request all of the information at once than it is to have each module
* requests its own information. This sacrifice of modularity yields a substantial
* performance improvement.
*
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
* @param array $moduleNames List of module names to preload information for
* @param ResourceLoaderContext $context Context to load the information within
*/
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
public function preloadModuleInfo( array $moduleNames, ResourceLoaderContext $context ) {
if ( !$moduleNames ) {
// Or else Database*::select() will explode, plus it's cheaper!
return;
}
$dbr = wfGetDB( DB_REPLICA );
$skin = $context->getSkin();
$lang = $context->getLanguage();
// Batched version of ResourceLoaderModule::getFileDependencies
$vary = "$skin|$lang";
$res = $dbr->select( 'module_deps', [ 'md_module', 'md_deps' ], [
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
'md_module' => $moduleNames,
'md_skin' => $vary,
], __METHOD__
);
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
// Prime in-object cache for file dependencies
$modulesWithDeps = [];
foreach ( $res as $row ) {
$module = $this->getModule( $row->md_module );
if ( $module ) {
$module->setFileDependencies( $context, ResourceLoaderModule::expandRelativePaths(
json_decode( $row->md_deps, true )
) );
$modulesWithDeps[] = $row->md_module;
}
}
// Register the absence of a dependency row too
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
foreach ( array_diff( $moduleNames, $modulesWithDeps ) as $name ) {
$module = $this->getModule( $name );
if ( $module ) {
$this->getModule( $name )->setFileDependencies( $context, [] );
}
}
// Batched version of ResourceLoaderWikiModule::getTitleInfo
ResourceLoaderWikiModule::preloadTitleInfo( $context, $dbr, $moduleNames );
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
// Prime in-object cache for message blobs for modules with messages
$modules = [];
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
foreach ( $moduleNames as $name ) {
$module = $this->getModule( $name );
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
if ( $module && $module->getMessages() ) {
$modules[$name] = $module;
}
}
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
$store = $this->getMessageBlobStore();
$blobs = $store->getBlobs( $modules, $lang );
foreach ( $blobs as $name => $blob ) {
$modules[$name]->setMessageBlob( $blob, $lang );
}
}
/**
* Run JavaScript or CSS data through a filter, caching the filtered result for future calls.
*
* Available filters are:
*
* - minify-js \see JavaScriptMinifier::minify
* - minify-css \see CSSMin::minify
*
* If $data is empty, only contains whitespace or the filter was unknown,
* $data is returned unmodified.
*
* @param string $filter Name of filter to run
* @param string $data Text to filter, such as JavaScript or CSS text
* @param array $options Keys:
* - (bool) cache: Whether to allow caching this data. Default: true.
* @return string Filtered data, or a comment containing an error message
*/
public static function filter( $filter, $data, array $options = [] ) {
if ( strpos( $data, self::FILTER_NOMIN ) !== false ) {
return $data;
}
2010-09-04 12:53:01 +00:00
if ( isset( $options['cache'] ) && $options['cache'] === false ) {
return self::applyFilter( $filter, $data );
}
2010-09-04 12:53:01 +00:00
$stats = MediaWikiServices::getInstance()->getStatsdDataFactory();
$cache = ObjectCache::getLocalServerInstance( CACHE_ANYTHING );
$key = $cache->makeGlobalKey(
'resourceloader',
'filter',
$filter,
self::CACHE_VERSION,
md5( $data )
);
$result = $cache->get( $key );
if ( $result === false ) {
$stats->increment( "resourceloader_cache.$filter.miss" );
$result = self::applyFilter( $filter, $data );
$cache->set( $key, $result, 24 * 3600 );
} else {
$stats->increment( "resourceloader_cache.$filter.hit" );
}
if ( $result === null ) {
// Cached failure
$result = $data;
}
return $result;
}
private static function applyFilter( $filter, $data ) {
$data = trim( $data );
if ( $data ) {
try {
$data = ( $filter === 'minify-css' )
? CSSMin::minify( $data )
: JavaScriptMinifier::minify( $data );
} catch ( Exception $e ) {
MWExceptionHandler::logException( $e );
return null;
}
}
return $data;
}
2010-09-04 12:53:01 +00:00
/**
* Register core modules and runs registration hooks.
* @param Config|null $config [optional]
* @param LoggerInterface|null $logger [optional]
*/
public function __construct( Config $config = null, LoggerInterface $logger = null ) {
global $IP;
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
$this->logger = $logger ?: new NullLogger();
if ( !$config ) {
// TODO: Deprecate and remove.
$this->logger->debug( __METHOD__ . ' was called without providing a Config instance' );
$config = MediaWikiServices::getInstance()->getMainConfig();
}
$this->config = $config;
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
// Add 'local' source first
resourceloader: Move queue formatting out of OutputPage HTML formatting of the queue was distributed over several OutputPage methods. Each method demanding a snippet of HTML by calling makeResourceLoaderLink() with a limited amount of information. As such, makeResourceLoaderLink() was unable to provide the client with the proper state information. Centralising it also allows it to better reduce duplication in HTML output and maintain a more accurate state. Problems fixed by centralising: 1. The 'user' module is special (due to per-user 'version' and 'user' params). It is manually requested via script-src. To avoid a separate (and wrong) request from something that requires it, we set state=loading directly. However, because the module is in the bottom, the old HTML formatter could only put state=loading in the bottom also. This sometimes caused a wrong request to be fired for modules=user if something in the top queue triggered a requirement for it. 2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose of allowing dependencies on style modules without risking duplicate loading on pages where the styles are loaded already. This didn't work, because the state information about page-style modules is output near the stylesheet, which is after the script tag with mw.loader.load(). That runs first, and mw.loader would still make a duplicate request before it learns the state. Changes: * Document reasons for style/script tag order in getHeadHtml (per 09537e83). * Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed before since a duplicate check in makeResourceLoaderLink() verified the origin a second time. * Declare explicit position 'top' on 'user.options' and 'user.tokens' module. Previously, OutputPage hardcoded them in the top. The new formatter doesn't. * Remove getHeadScripts(). * Remove getInlineHeadScripts(). * Remove getExternalHeadScripts(). * Remove buildCssLinks(). * Remove getScriptsForBottomQueue(). * Change where Skin::setupSkinUserCss() is called. This methods lets the skin add modules to the queue. Previously it was called from buildCssLinks(), via headElement(), via prepareQuickTemplate(), via OutputPage::output(). It's now in OutputPage::output() directly (slightly earlier). This is needed because prepareQuickTemplate() calls bottomScripts() before headElement(). And bottomScript() would lazy-initialise the queue and lock it before setupSkinUserCss() is called from headElement(). This makes execution order more predictable instead of being dependent on the arbitrary order of data extraction in prepareQuickTemplate (which varies from one skin to another). * Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early on so. This avoids wrongful loading and fixes problem 1. Effective changes in output: * mw.loader.state() is now before mw.loader.load(). This fixes problem 2. * mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading". * mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1. * The <script async src> tag for 'startup' changed position (slightly). Previously it was after all inline scripts and stylesheets. It's still after all inline scripts and after most stylesheets, but before any user styles. Since the queue is now formatted outside OutputPage, it can't inject the meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle of existing output. This shouldn't have any noticable impact. Bug: T87871 Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
2016-07-15 14:13:09 +00:00
$this->addSource( 'local', $config->get( 'LoadScript' ) );
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
// Add other sources
$this->addSource( $config->get( 'ResourceLoaderSources' ) );
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
// Register core modules
$this->register( include "$IP/resources/Resources.php" );
// Register extension modules
$this->register( $config->get( 'ResourceModules' ) );
// Avoid PHP 7.1 warning from passing $this by reference
$rl = $this;
Hooks::run( 'ResourceLoaderRegisterModules', [ &$rl ] );
if ( $config->get( 'EnableJavaScriptTest' ) === true ) {
$this->registerTestModules();
}
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
$this->setMessageBlobStore( new MessageBlobStore( $this, $this->logger ) );
}
2010-09-04 12:53:01 +00:00
/**
* @return Config
*/
public function getConfig() {
return $this->config;
}
/**
* @since 1.26
* @param LoggerInterface $logger
*/
public function setLogger( LoggerInterface $logger ) {
$this->logger = $logger;
}
/**
* @since 1.27
* @return LoggerInterface
*/
public function getLogger() {
return $this->logger;
}
/**
* @since 1.26
* @return MessageBlobStore
*/
public function getMessageBlobStore() {
return $this->blobStore;
}
/**
* @since 1.25
* @param MessageBlobStore $blobStore
*/
public function setMessageBlobStore( MessageBlobStore $blobStore ) {
$this->blobStore = $blobStore;
}
/**
* Register a module with the ResourceLoader system.
*
* @param mixed $name Name of module as a string or List of name/object pairs as an array
* @param array|null $info Module info array. For backwards compatibility with 1.17alpha,
* this may also be a ResourceLoaderModule object. Optional when using
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
* multiple-registration calling style.
* @throws MWException If a duplicate module registration is attempted
* @throws MWException If a module name contains illegal characters (pipes or commas)
* @throws MWException If something other than a ResourceLoaderModule is being registered
* @return bool False if there were any errors, in which case one or more modules were
* not registered
*/
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
public function register( $name, $info = null ) {
$moduleSkinStyles = $this->config->get( 'ResourceModuleSkinStyles' );
// Allow multiple modules to be registered in one call
$registrations = is_array( $name ) ? $name : [ $name => $info ];
foreach ( $registrations as $name => $info ) {
// Warn on duplicate registrations
if ( isset( $this->moduleInfos[$name] ) ) {
// A module has already been registered by this name
$this->logger->warning(
'ResourceLoader duplicate registration warning. ' .
'Another module has already been registered as ' . $name
);
}
// Check $name for validity
if ( !self::isValidModuleName( $name ) ) {
throw new MWException( "ResourceLoader module name '$name' is invalid, "
. "see ResourceLoader::isValidModuleName()" );
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
}
// Attach module
if ( $info instanceof ResourceLoaderModule ) {
$this->moduleInfos[$name] = [ 'object' => $info ];
$info->setName( $name );
$this->modules[$name] = $info;
} elseif ( is_array( $info ) ) {
// New calling convention
$this->moduleInfos[$name] = $info;
} else {
throw new MWException(
'ResourceLoader module info type error for module \'' . $name .
'\': expected ResourceLoaderModule or array (got: ' . gettype( $info ) . ')'
);
}
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
// Last-minute changes
// Apply custom skin-defined styles to existing modules.
if ( $this->isFileModule( $name ) ) {
foreach ( $moduleSkinStyles as $skinName => $skinStyles ) {
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
// If this module already defines skinStyles for this skin, ignore $wgResourceModuleSkinStyles.
if ( isset( $this->moduleInfos[$name]['skinStyles'][$skinName] ) ) {
continue;
}
// If $name is preceded with a '+', the defined style files will be added to 'default'
// skinStyles, otherwise 'default' will be ignored as it normally would be.
if ( isset( $skinStyles[$name] ) ) {
$paths = (array)$skinStyles[$name];
$styleFiles = [];
} elseif ( isset( $skinStyles['+' . $name] ) ) {
$paths = (array)$skinStyles['+' . $name];
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
$styleFiles = isset( $this->moduleInfos[$name]['skinStyles']['default'] ) ?
(array)$this->moduleInfos[$name]['skinStyles']['default'] :
[];
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
} else {
continue;
}
// Add new file paths, remapping them to refer to our directories and not use settings
// from the module we're modifying, which come from the base definition.
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
list( $localBasePath, $remoteBasePath ) =
ResourceLoaderFileModule::extractBasePaths( $skinStyles );
foreach ( $paths as $path ) {
$styleFiles[] = new ResourceLoaderFilePath( $path, $localBasePath, $remoteBasePath );
}
$this->moduleInfos[$name]['skinStyles'][$skinName] = $styleFiles;
}
}
}
}
2010-09-04 12:53:01 +00:00
public function registerTestModules() {
global $IP;
if ( $this->config->get( 'EnableJavaScriptTest' ) !== true ) {
throw new MWException( 'Attempt to register JavaScript test modules '
. 'but <code>$wgEnableJavaScriptTest</code> is false. '
. 'Edit your <code>LocalSettings.php</code> to enable it.' );
}
$testModules = [
'qunit' => [],
];
// Get test suites from extensions
// Avoid PHP 7.1 warning from passing $this by reference
$rl = $this;
Hooks::run( 'ResourceLoaderTestModules', [ &$testModules, &$rl ] );
$extRegistry = ExtensionRegistry::getInstance();
// In case of conflict, the deprecated hook has precedence.
$testModules['qunit'] += $extRegistry->getAttribute( 'QUnitTestModules' );
// Add the QUnit testrunner as implicit dependency to extension test suites.
foreach ( $testModules['qunit'] as &$module ) {
// Shuck any single-module dependency as an array
if ( is_string( $module['dependencies'] ) ) {
$module['dependencies'] = [ $module['dependencies'] ];
}
$module['dependencies'][] = 'test.mediawiki.qunit.testrunner';
}
// Get core test suites
$testModules['qunit'] =
( include "$IP/tests/qunit/QUnitTestResources.php" ) + $testModules['qunit'];
foreach ( $testModules as $id => $names ) {
// Register test modules
$this->register( $testModules[$id] );
// Keep track of their names so that they can be loaded together
$this->testModuleNames[$id] = array_keys( $testModules[$id] );
}
}
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
/**
* Add a foreign source of modules.
*
* Source IDs are typically the same as the Wiki ID or database name (e.g. lowercase a-z).
*
* @param array|string $id Source ID (string), or [ id1 => loadUrl, id2 => loadUrl, ... ]
* @param string|array|null $loadUrl load.php url (string), or array with loadUrl key for
* backwards-compatibility.
* @throws MWException
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
*/
public function addSource( $id, $loadUrl = null ) {
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
// Allow multiple sources to be registered in one call
if ( is_array( $id ) ) {
foreach ( $id as $key => $value ) {
$this->addSource( $key, $value );
}
return;
}
// Disallow duplicates
if ( isset( $this->sources[$id] ) ) {
throw new MWException(
'ResourceLoader duplicate source addition error. ' .
'Another source has already been registered as ' . $id
);
}
// Pre 1.24 backwards-compatibility
if ( is_array( $loadUrl ) ) {
if ( !isset( $loadUrl['loadScript'] ) ) {
throw new MWException(
__METHOD__ . ' was passed an array with no "loadScript" key.'
);
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
}
$loadUrl = $loadUrl['loadScript'];
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
}
$this->sources[$id] = $loadUrl;
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
}
/**
* Get a list of module names.
*
* @return array List of module names
*/
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
public function getModuleNames() {
return array_keys( $this->moduleInfos );
}
/**
* Get a list of test module names for one (or all) frameworks.
*
* If the given framework id is unknkown, or if the in-object variable is not an array,
* then it will return an empty array.
*
* @param string $framework Get only the test module names for one
* particular framework (optional)
* @return array
*/
public function getTestModuleNames( $framework = 'all' ) {
/** @todo api siteinfo prop testmodulenames modulenames */
if ( $framework == 'all' ) {
return $this->testModuleNames;
} elseif ( isset( $this->testModuleNames[$framework] )
&& is_array( $this->testModuleNames[$framework] )
) {
return $this->testModuleNames[$framework];
} else {
return [];
}
}
2010-09-04 12:53:01 +00:00
/**
* Check whether a ResourceLoader module is registered
*
* @since 1.25
* @param string $name
* @return bool
*/
public function isModuleRegistered( $name ) {
return isset( $this->moduleInfos[$name] );
}
/**
* Get the ResourceLoaderModule object for a given module name.
2010-09-05 13:31:34 +00:00
*
* If an array of module parameters exists but a ResourceLoaderModule object has not
* yet been instantiated, this method will instantiate and cache that object such that
* subsequent calls simply return the same object.
*
* @param string $name Module name
* @return ResourceLoaderModule|null If module has been registered, return a
* ResourceLoaderModule instance. Otherwise, return null.
*/
public function getModule( $name ) {
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
if ( !isset( $this->modules[$name] ) ) {
if ( !isset( $this->moduleInfos[$name] ) ) {
// No such module
return null;
}
// Construct the requested object
$info = $this->moduleInfos[$name];
/** @var ResourceLoaderModule $object */
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
if ( isset( $info['object'] ) ) {
// Object given in info array
$object = $info['object'];
} elseif ( isset( $info['factory'] ) ) {
$object = call_user_func( $info['factory'], $info );
$object->setConfig( $this->getConfig() );
$object->setLogger( $this->logger );
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
} else {
$class = $info['class'] ?? ResourceLoaderFileModule::class;
/** @var ResourceLoaderModule $object */
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
$object = new $class( $info );
$object->setConfig( $this->getConfig() );
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
$object->setLogger( $this->logger );
* Made Resources.php return a pure-data array instead of an ugly mix of data and code. This allows the class code to be lazy-loaded with the autoloader, for a performance advantage especially on non-APC installs. And using the convention where if the class is omitted, ResourceLoaderFileModule is assumed, the registration code becomes shorter and simpler. * Modified ResourceLoader to lazy-initialise module objects, for a further performance advantage. * Deleted ResourceLoader::getModules(), provided getModuleNames() instead. Although the startup module needs this functionality, it's slow to generate, so to avoid misuse, it's better to provide a foolproof fast interface and let the startup module do the slow thing itself. * Modified ResourceLoader::register() to optionally accept an info array instead of an object. * Added $wgResourceModules, allowing extensions to efficiently define their own resource loader modules. The trouble with hooks is that they contain code, and code is slow. We've been through all this before with i18n. Hooks are useful as a performance tool only if you call them very rarely. * Moved ResourceLoader settings to their own section in DefaultSettings.php * Added options to ResourceLoaderFileModule equivalent to the $localBasePath and $remoteBasePath parameters, to allow it to be instantiated via the new array style. Also added remoteExtPath, which allows modules to be registered before $wgExtensionAssetsPath is known. * Added OutputPage::getResourceLoader(), mostly for debugging. * The time saving at the moment is about 5ms per request with no extensions, which is significant already with 6 load.php requests for a cold cache page view. This is a much more scalable interface; the relative saving will grow as more extensions are added which use this interface, especially for non-APC installs. Although the interface is backwards compatible, extension updates will follow in a subsequent commit.
2010-11-19 10:41:06 +00:00
}
$object->setName( $name );
$this->modules[$name] = $object;
}
return $this->modules[$name];
}
2010-09-04 12:53:01 +00:00
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
/**
* Whether the module is a ResourceLoaderFileModule (including subclasses).
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
*
* @param string $name Module name
* @return bool
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
*/
protected function isFileModule( $name ) {
if ( !isset( $this->moduleInfos[$name] ) ) {
return false;
}
$info = $this->moduleInfos[$name];
if ( isset( $info['object'] ) ) {
return false;
}
return (
// The implied default for 'class' is ResourceLoaderFileModule
!isset( $info['class'] ) ||
// Explicit default
$info['class'] === ResourceLoaderFileModule::class ||
is_subclass_of( $info['class'], ResourceLoaderFileModule::class )
);
resourceloader: Allow skins to provide additional styles for any module The newly introduced $wgResourceModuleSkinStyles global enables skins to provide additional stylesheets to existing ResourceLoader module. This both makes it easier (or at all possible) to override default styles and lowers the style footprint by making it possible not to load styles unused on most pages. ---- Example: Use the file 'foo-styles.css' for the 'mediawiki.foo' module when using the MySkin skin: $wgResourceModuleSkinStyles['myskin'] = array( 'mediawiki.foo' => 'foo-styles.css', 'remoteSkinPath' => 'MySkin', 'localBasePath' => __DIR__, ); For detailed documentation, see the doc comment in DefaultSettings.php. For a practical usage example, see Vector.php. ---- Implementation notes: * The values defined in $wgResourceModuleSkinStyles are embedded into the modules as late as possible (in ResourceLoader::register()). * Only plain file modules are supported, setting module skin styles for other module types has no effect. * ResourceLoader and ResourceLoaderFileModule now support loading files from arbitrary paths to make this possible, defined using ResourceLoaderFilePath objects. * This required some adjustments in seemingly unrelated places for code which didn't handle the paths fully correctly before. * ResourceLoader and ResourceLoaderFileModule are now a bit more tightly coupled than before :( * Included a tiny example change for the Vector skin, a lot more of similar cleanup is possible and planned for the future. * Many of the non-essential mediawiki.* modules defined in Resources.php should be using `'skinStyles' => array( 'default' => … )` instead of `'styles' => …` to allow more customizations, this is also planned for the future after auditing which ones would actually benefit from this. Change-Id: Ica4ff9696b490e35f60288d7ce1295766c427e87
2014-06-26 14:29:31 +00:00
}
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
/**
* Get the list of sources.
*
* @return array Like [ id => load.php url, ... ]
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
*/
public function getSources() {
return $this->sources;
}
/**
* Get the URL to the load.php endpoint for the given
* ResourceLoader source
*
* @since 1.24
* @param string $source
* @throws MWException On an invalid $source name
* @return string
*/
public function getLoadScript( $source ) {
if ( !isset( $this->sources[$source] ) ) {
throw new MWException( "The $source source was never registered in ResourceLoader." );
}
return $this->sources[$source];
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
/**
* @since 1.26
* @param string $value
* @return string Hash
*/
public static function makeHash( $value ) {
$hash = hash( 'fnv132', $value );
return Wikimedia\base_convert( $hash, 16, 36, 7 );
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
}
/**
* Add an error to the 'errors' array and log it.
*
resourceloader: Use 'enableModuleContentVersion' for startup module This significantly simplifies the getVersionHash implementation for StartupModule, and fixes a couple of bugs. Previously, the startup module's E-Tag was determined by the 'getDefinitionSummary' method, which combined the E-Tag values from all registered modules, plus what we thought is all information used by 'getScript' (config vars, embedded script files, list of base modules, ...) However, this were various things part of the manifest that it forgot about, including: * Changes to the list of dependencies of a module. * Changes to the name of module. * Changes to the cache group of module. * Adding or removing a foreign module source (mw.loader.addSource). These are all quite rare, and when they do change, they usually also involve a change that *was* tracked already. But, sometimes they don't and that's when bugs happened. Instead of the tracking array of getDefinitionSummary, we now use the 'enableModuleContentVersion' option for StartupModule, which simply calls the actual getScript() method and hashes that. Of note: When an exception happens with the version computation of any individual module, we catch it, log it, and continue with the rest. Previously, the first time such error was discovered at run-time would be in the getCombinedVersion() call from StartupModule::getAllModuleHashes(). That public getCombinedVersion() method of ResourceLoader had the benefit of also outputting details of that exception in the HTTP response output. In order to keep that behaviour, I made outputErrorAndLog() public so that StartupModule can call it directly now. This is covered by ResourceLoaderTest::testMakeModuleResponseStartupError. Bug: T201686 Change-Id: I8e8d3a2cd2ccd68d2d78e988bcdd0d77fbcbf1d4
2018-08-30 02:52:39 +00:00
* @private For internal use by ResourceLoader and ResourceLoaderStartUpModule.
* @since 1.29
* @param Exception $e
* @param string $msg
* @param array $context
*/
resourceloader: Use 'enableModuleContentVersion' for startup module This significantly simplifies the getVersionHash implementation for StartupModule, and fixes a couple of bugs. Previously, the startup module's E-Tag was determined by the 'getDefinitionSummary' method, which combined the E-Tag values from all registered modules, plus what we thought is all information used by 'getScript' (config vars, embedded script files, list of base modules, ...) However, this were various things part of the manifest that it forgot about, including: * Changes to the list of dependencies of a module. * Changes to the name of module. * Changes to the cache group of module. * Adding or removing a foreign module source (mw.loader.addSource). These are all quite rare, and when they do change, they usually also involve a change that *was* tracked already. But, sometimes they don't and that's when bugs happened. Instead of the tracking array of getDefinitionSummary, we now use the 'enableModuleContentVersion' option for StartupModule, which simply calls the actual getScript() method and hashes that. Of note: When an exception happens with the version computation of any individual module, we catch it, log it, and continue with the rest. Previously, the first time such error was discovered at run-time would be in the getCombinedVersion() call from StartupModule::getAllModuleHashes(). That public getCombinedVersion() method of ResourceLoader had the benefit of also outputting details of that exception in the HTTP response output. In order to keep that behaviour, I made outputErrorAndLog() public so that StartupModule can call it directly now. This is covered by ResourceLoaderTest::testMakeModuleResponseStartupError. Bug: T201686 Change-Id: I8e8d3a2cd2ccd68d2d78e988bcdd0d77fbcbf1d4
2018-08-30 02:52:39 +00:00
public function outputErrorAndLog( Exception $e, $msg, array $context = [] ) {
MWExceptionHandler::logException( $e );
$this->logger->warning(
$msg,
$context + [ 'exception' => $e ]
);
$this->errors[] = self::formatExceptionNoComment( $e );
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
/**
* Helper method to get and combine versions of multiple modules.
*
* @since 1.26
* @param ResourceLoaderContext $context
* @param string[] $moduleNames List of known module names
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @return string Hash
*/
public function getCombinedVersion( ResourceLoaderContext $context, array $moduleNames ) {
if ( !$moduleNames ) {
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
return '';
}
$hashes = array_map( function ( $module ) use ( $context ) {
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
try {
return $this->getModule( $module )->getVersionHash( $context );
} catch ( Exception $e ) {
resourceloader: Use 'enableModuleContentVersion' for startup module This significantly simplifies the getVersionHash implementation for StartupModule, and fixes a couple of bugs. Previously, the startup module's E-Tag was determined by the 'getDefinitionSummary' method, which combined the E-Tag values from all registered modules, plus what we thought is all information used by 'getScript' (config vars, embedded script files, list of base modules, ...) However, this were various things part of the manifest that it forgot about, including: * Changes to the list of dependencies of a module. * Changes to the name of module. * Changes to the cache group of module. * Adding or removing a foreign module source (mw.loader.addSource). These are all quite rare, and when they do change, they usually also involve a change that *was* tracked already. But, sometimes they don't and that's when bugs happened. Instead of the tracking array of getDefinitionSummary, we now use the 'enableModuleContentVersion' option for StartupModule, which simply calls the actual getScript() method and hashes that. Of note: When an exception happens with the version computation of any individual module, we catch it, log it, and continue with the rest. Previously, the first time such error was discovered at run-time would be in the getCombinedVersion() call from StartupModule::getAllModuleHashes(). That public getCombinedVersion() method of ResourceLoader had the benefit of also outputting details of that exception in the HTTP response output. In order to keep that behaviour, I made outputErrorAndLog() public so that StartupModule can call it directly now. This is covered by ResourceLoaderTest::testMakeModuleResponseStartupError. Bug: T201686 Change-Id: I8e8d3a2cd2ccd68d2d78e988bcdd0d77fbcbf1d4
2018-08-30 02:52:39 +00:00
// If modules fail to compute a version, don't fail the request (T152266)
// and still compute versions of other modules.
$this->outputErrorAndLog( $e,
resourceloader: Don't let module exception break startup When getScript (or some other method used in a module response) throws an error, only that module fails (by outputting mw.loader.state instead of mw.loader.implement). Other modules will work. This has always been the case and is working fine. For example, "load.php?modules=foo|bar", where 'foo' throws, will return: ```js /* exception message: .. */ mw.loader.implement('bar', ..) mw.loader.state('foo', 'error') ``` The problem, however, is that during the generation of the startup module, we iterate over all other modules. In 2011, the getVersionHash method (then: getModifiedTime) was fairly simple and unlikely to throw errors. Nowadays, some modules use enableModuleContentVersion which will involve the same code path as for regular module responses. The try/catch in ResourceLoader::makeModuleResponse() suffices for the case of loading modules other than startup. But when loading the startup module, and an exception happens in getVersionHash, then the entire startup response is replaced with an exception comment. Example case: * A file not existing for a FileModule subclass that uses enableModuleContentVersion. * A database error from a data module, like CiteDataModule or CNChoiceData. Changes: * Ensure E-Tag is still useful while an error happens in production because we respond with 200 OK and one error isn't the same as another. Fixed by try/catch in getCombinedVersion. * Ensure start manifest isn't disrupted by one broken module. Fixed by try/catch in StartupModule::getModuleRegistrations(). Tests: * testMakeModuleResponseError: The case that already worked fined. * testMakeModuleResponseStartupError: The case fixed in this commit. * testGetCombinedVersion: The case fixed in this commit for E-Tag. Bug: T152266 Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
2016-12-03 00:48:14 +00:00
'Calculating version for "{module}" failed: {exception}',
[
'module' => $module,
]
);
return '';
}
}, $moduleNames );
return self::makeHash( implode( '', $hashes ) );
}
/**
* Get the expected value of the 'version' query parameter.
*
* This is used by respond() to set a short Cache-Control header for requests with
* information newer than the current server has. This avoids pollution of edge caches.
* Typically during deployment. (T117587)
*
* This MUST match return value of `mw.loader#getCombinedVersion()` client-side.
*
* @since 1.28
* @param ResourceLoaderContext $context
* @return string Hash
*/
public function makeVersionQuery( ResourceLoaderContext $context ) {
// As of MediaWiki 1.28, the server and client use the same algorithm for combining
// version hashes. There is no technical reason for this to be same, and for years the
// implementations differed. If getCombinedVersion in PHP (used for StartupModule and
// E-Tag headers) differs in the future from getCombinedVersion in JS (used for 'version'
// query parameter), then this method must continue to match the JS one.
$moduleNames = [];
foreach ( $context->getModules() as $name ) {
if ( !$this->getModule( $name ) ) {
// If a versioned request contains a missing module, the version is a mismatch
// as the client considered a module (and version) we don't have.
return '';
}
$moduleNames[] = $name;
}
return $this->getCombinedVersion( $context, $moduleNames );
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
}
2010-09-05 13:31:34 +00:00
/**
* Output a response to a load request, including the content-type header.
*
* @param ResourceLoaderContext $context Context in which a response should be formed
*/
public function respond( ResourceLoaderContext $context ) {
// Buffer output to catch warnings. Normally we'd use ob_clean() on the
// top-level output buffer to clear warnings, but that breaks when ob_gzhandler
// is used: ob_clean() will clear the GZIP header in that case and it won't come
// back for subsequent output, resulting in invalid GZIP. So we have to wrap
// the whole thing in our own output buffer to be sure the active buffer
// doesn't use ob_gzhandler.
// See https://bugs.php.net/bug.php?id=36514
ob_start();
$this->measureResponseTime( RequestContext::getMain()->getTiming() );
// Find out which modules are missing and instantiate the others
$modules = [];
$missing = [];
foreach ( $context->getModules() as $name ) {
$module = $this->getModule( $name );
if ( $module ) {
* (bug 34212) ApiBlock/ApiUnblock allow action to take place without a token parameter present * (bug 35317) CSRF in Special:Upload Revert r56793, which removed the CSRF check for Special:Upload for normal file uploads. Cross-site posting of file uploads without user interaction has been possible since at least as early as Chrome 8 (late 2010) and Firefox 6 (mid 2011). Commonist has used api.php since version 0.4.0 (April 2010), and the API already requires an edit token, so Commonist 0.4.0+ is not affected by this change. * (bug 34907) Fix for CSRF vulnerability due to mw.user.tokens. Patch by Roan Kattouw and Tim Starling. * Filter out private modules early in ResourceLoader::makeResponse() and just pretend they weren't specified. This means these modules cannot be loaded through load.php . This filtering must not happen in makeModuleResponse(), because that would break inlining. * Force inlining of private modules in OutputPage::makeResourceLoaderLink(), disregarding $wgResourceLoaderInlinePrivateModules * Remove $wgResourceLoaderInlinePrivateModules * Remove special treatment of private modules ($private) in ResourceLoader::makeResponse() and sendResponseHeaders(), because we're not allowing private modules to be loaded through here any more * Remove identity checks in ResourceLoaderUserOptionsModule and ResourceLoaderUserCSSPrefsModule, they didn't make a lot of sense before but they're certainly useless now. * Factored out error comment construction in ResourceLoader.php and stripped comment terminations from exception messages. I didn't find an XSS vulnerability but it looked scary. Patchset2: Removes whitespace error that prevented automatic merge by Gerrit: includes/resourceloader/ResourceLoaderUserOptionsModule.php Change-Id: I2dec8b8caf9db3c64919763865cc10cccdd6a1a3
2012-03-22 19:52:37 +00:00
// Do not allow private modules to be loaded from the web.
// This is a security issue, see T36907.
* (bug 34212) ApiBlock/ApiUnblock allow action to take place without a token parameter present * (bug 35317) CSRF in Special:Upload Revert r56793, which removed the CSRF check for Special:Upload for normal file uploads. Cross-site posting of file uploads without user interaction has been possible since at least as early as Chrome 8 (late 2010) and Firefox 6 (mid 2011). Commonist has used api.php since version 0.4.0 (April 2010), and the API already requires an edit token, so Commonist 0.4.0+ is not affected by this change. * (bug 34907) Fix for CSRF vulnerability due to mw.user.tokens. Patch by Roan Kattouw and Tim Starling. * Filter out private modules early in ResourceLoader::makeResponse() and just pretend they weren't specified. This means these modules cannot be loaded through load.php . This filtering must not happen in makeModuleResponse(), because that would break inlining. * Force inlining of private modules in OutputPage::makeResourceLoaderLink(), disregarding $wgResourceLoaderInlinePrivateModules * Remove $wgResourceLoaderInlinePrivateModules * Remove special treatment of private modules ($private) in ResourceLoader::makeResponse() and sendResponseHeaders(), because we're not allowing private modules to be loaded through here any more * Remove identity checks in ResourceLoaderUserOptionsModule and ResourceLoaderUserCSSPrefsModule, they didn't make a lot of sense before but they're certainly useless now. * Factored out error comment construction in ResourceLoader.php and stripped comment terminations from exception messages. I didn't find an XSS vulnerability but it looked scary. Patchset2: Removes whitespace error that prevented automatic merge by Gerrit: includes/resourceloader/ResourceLoaderUserOptionsModule.php Change-Id: I2dec8b8caf9db3c64919763865cc10cccdd6a1a3
2012-03-22 19:52:37 +00:00
if ( $module->getGroup() === 'private' ) {
$this->logger->debug( "Request for private module '$name' denied" );
$this->errors[] = "Cannot show private module \"$name\"";
* (bug 34212) ApiBlock/ApiUnblock allow action to take place without a token parameter present * (bug 35317) CSRF in Special:Upload Revert r56793, which removed the CSRF check for Special:Upload for normal file uploads. Cross-site posting of file uploads without user interaction has been possible since at least as early as Chrome 8 (late 2010) and Firefox 6 (mid 2011). Commonist has used api.php since version 0.4.0 (April 2010), and the API already requires an edit token, so Commonist 0.4.0+ is not affected by this change. * (bug 34907) Fix for CSRF vulnerability due to mw.user.tokens. Patch by Roan Kattouw and Tim Starling. * Filter out private modules early in ResourceLoader::makeResponse() and just pretend they weren't specified. This means these modules cannot be loaded through load.php . This filtering must not happen in makeModuleResponse(), because that would break inlining. * Force inlining of private modules in OutputPage::makeResourceLoaderLink(), disregarding $wgResourceLoaderInlinePrivateModules * Remove $wgResourceLoaderInlinePrivateModules * Remove special treatment of private modules ($private) in ResourceLoader::makeResponse() and sendResponseHeaders(), because we're not allowing private modules to be loaded through here any more * Remove identity checks in ResourceLoaderUserOptionsModule and ResourceLoaderUserCSSPrefsModule, they didn't make a lot of sense before but they're certainly useless now. * Factored out error comment construction in ResourceLoader.php and stripped comment terminations from exception messages. I didn't find an XSS vulnerability but it looked scary. Patchset2: Removes whitespace error that prevented automatic merge by Gerrit: includes/resourceloader/ResourceLoaderUserOptionsModule.php Change-Id: I2dec8b8caf9db3c64919763865cc10cccdd6a1a3
2012-03-22 19:52:37 +00:00
continue;
}
$modules[$name] = $module;
} else {
$missing[] = $name;
}
}
2010-09-04 12:53:01 +00:00
try {
resourceloader: Migrate from msg_resource table to object cache MessageBlobStore class: * Make logger aware. * Log an error if json encoding fails. * Stop using the DB table. WANObjectCache supports everything we need: - Batch retrieval. - Invalidate keys with wildcard selects or cascading check keys. * Update tests slightly since the actual update now happens on-demand as part of get() instead of within updateMessage(). ResourceLoader class: * Remove all interaction with the msg_resource table. Remove db table later. * Refactor code to use a hash of the blob instead of a timestamp. Timestamps are unreliable and roll over too frequently for message blob store because there is no authoritative source. The timestamps were inferred based on when a change is observed. Message overrides from the local wiki have an explicit update event when the page is edited. All other messages, such as from MediaWiki core and extensions using LocalisationCache, have a single timestamp for all messages which rolls over every time the cache is rebuilt. A hash is deterministic, and won't cause needless invalidation (T102578). * Remove redundant pre-fetching in makeModuleResponse(). This is already done by preloadModuleInfo() in respond(). * Don't bother storing and retreiving empty "{}" objects. Instead, detect whether a module's message list is empty at runtime. ResourceLoaderModule class: * Make logger aware. * Log if a module's message blob was not preloaded. cleanupRemovedModules: * Now that blobs have a TTL, there's no need to prune old entries. Bug: T113092 Bug: T92357 Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-11-13 00:04:12 +00:00
// Preload for getCombinedVersion() and for batch makeModuleResponse()
$this->preloadModuleInfo( array_keys( $modules ), $context );
} catch ( Exception $e ) {
$this->outputErrorAndLog( $e, 'Preloading module info failed: {exception}' );
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Combine versions to propagate cache invalidation
$versionHash = '';
try {
$versionHash = $this->getCombinedVersion( $context, array_keys( $modules ) );
} catch ( Exception $e ) {
$this->outputErrorAndLog( $e, 'Calculating version hash failed: {exception}' );
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// See RFC 2616 § 3.11 Entity Tags
// https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.11
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
$etag = 'W/"' . $versionHash . '"';
// Try the client-side cache first
if ( $this->tryRespondNotModified( $context, $etag ) ) {
return; // output handled (buffers cleared)
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// Use file cache if enabled and available...
if ( $this->config->get( 'UseFileCache' ) ) {
$fileCache = ResourceFileCache::newFromContext( $context );
if ( $this->tryRespondFromFileCache( $fileCache, $context, $etag ) ) {
return; // output handled
}
}
// Generate a response
$response = $this->makeModuleResponse( $context, $modules, $missing );
// Capture any PHP warnings from the output buffer and append them to the
// error list if we're in debug mode.
if ( $context->getDebug() ) {
$warnings = ob_get_contents();
if ( strlen( $warnings ) ) {
$this->errors[] = $warnings;
}
}
* (bug 34212) ApiBlock/ApiUnblock allow action to take place without a token parameter present * (bug 35317) CSRF in Special:Upload Revert r56793, which removed the CSRF check for Special:Upload for normal file uploads. Cross-site posting of file uploads without user interaction has been possible since at least as early as Chrome 8 (late 2010) and Firefox 6 (mid 2011). Commonist has used api.php since version 0.4.0 (April 2010), and the API already requires an edit token, so Commonist 0.4.0+ is not affected by this change. * (bug 34907) Fix for CSRF vulnerability due to mw.user.tokens. Patch by Roan Kattouw and Tim Starling. * Filter out private modules early in ResourceLoader::makeResponse() and just pretend they weren't specified. This means these modules cannot be loaded through load.php . This filtering must not happen in makeModuleResponse(), because that would break inlining. * Force inlining of private modules in OutputPage::makeResourceLoaderLink(), disregarding $wgResourceLoaderInlinePrivateModules * Remove $wgResourceLoaderInlinePrivateModules * Remove special treatment of private modules ($private) in ResourceLoader::makeResponse() and sendResponseHeaders(), because we're not allowing private modules to be loaded through here any more * Remove identity checks in ResourceLoaderUserOptionsModule and ResourceLoaderUserCSSPrefsModule, they didn't make a lot of sense before but they're certainly useless now. * Factored out error comment construction in ResourceLoader.php and stripped comment terminations from exception messages. I didn't find an XSS vulnerability but it looked scary. Patchset2: Removes whitespace error that prevented automatic merge by Gerrit: includes/resourceloader/ResourceLoaderUserOptionsModule.php Change-Id: I2dec8b8caf9db3c64919763865cc10cccdd6a1a3
2012-03-22 19:52:37 +00:00
// Save response to file cache unless there are errors
if ( isset( $fileCache ) && !$this->errors && $missing === [] ) {
// Cache single modules and images...and other requests if there are enough hits
if ( ResourceFileCache::useFileCache( $context ) ) {
if ( $fileCache->isCacheWorthy() ) {
$fileCache->saveText( $response );
} else {
$fileCache->incrMissesRecent( $context->getRequest() );
}
}
}
$this->sendResponseHeaders( $context, $etag, (bool)$this->errors, $this->extraHeaders );
// Remove the output buffer and output the response
ob_end_clean();
if ( $context->getImageObj() && $this->errors ) {
// We can't show both the error messages and the response when it's an image.
$response = implode( "\n\n", $this->errors );
} elseif ( $this->errors ) {
$errorText = implode( "\n\n", $this->errors );
$errorResponse = self::makeComment( $errorText );
if ( $context->shouldIncludeScripts() ) {
$errorResponse .= 'if (window.console && console.error) {'
. Xml::encodeJsCall( 'console.error', [ $errorText ] )
. "}\n";
}
// Prepend error info to the response
$response = $errorResponse . $response;
}
$this->errors = [];
echo $response;
}
protected function measureResponseTime( Timing $timing ) {
DeferredUpdates::addCallableUpdate( function () use ( $timing ) {
$measure = $timing->measure( 'responseTime', 'requestStart', 'requestShutdown' );
if ( $measure !== false ) {
$stats = MediaWikiServices::getInstance()->getStatsdDataFactory();
$stats->timing( 'resourceloader.responseTime', $measure['duration'] * 1000 );
}
} );
}
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Send main response headers to the client.
*
* Deals with Content-Type, CORS (for stylesheets), and caching.
*
* @param ResourceLoaderContext $context
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @param string $etag ETag header value
* @param bool $errors Whether there are errors in the response
* @param string[] $extra Array of extra HTTP response headers
* @return void
*/
protected function sendResponseHeaders(
ResourceLoaderContext $context, $etag, $errors, array $extra = []
) {
\MediaWiki\HeaderCallback::warnIfHeadersSent();
$rlMaxage = $this->config->get( 'ResourceLoaderMaxage' );
// Use a short cache expiry so that updates propagate to clients quickly, if:
// - No version specified (shared resources, e.g. stylesheets)
// - There were errors (recover quickly)
// - Version mismatch (T117587, T47877)
if ( is_null( $context->getVersion() )
|| $errors
|| $context->getVersion() !== $this->makeVersionQuery( $context )
) {
$maxage = $rlMaxage['unversioned']['client'];
$smaxage = $rlMaxage['unversioned']['server'];
// If a version was specified we can use a longer expiry time since changing
// version numbers causes cache misses
} else {
$maxage = $rlMaxage['versioned']['client'];
$smaxage = $rlMaxage['versioned']['server'];
}
if ( $context->getImageObj() ) {
// Output different headers if we're outputting textual errors.
if ( $errors ) {
header( 'Content-Type: text/plain; charset=utf-8' );
} else {
$context->getImageObj()->sendResponseHeaders( $context );
}
} elseif ( $context->getOnly() === 'styles' ) {
header( 'Content-Type: text/css; charset=utf-8' );
header( 'Access-Control-Allow-Origin: *' );
} else {
header( 'Content-Type: text/javascript; charset=utf-8' );
}
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// See RFC 2616 § 14.19 ETag
// https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
header( 'ETag: ' . $etag );
if ( $context->getDebug() ) {
// Do not cache debug responses
header( 'Cache-Control: private, no-cache, must-revalidate' );
header( 'Pragma: no-cache' );
} else {
header( "Cache-Control: public, max-age=$maxage, s-maxage=$smaxage" );
* (bug 34212) ApiBlock/ApiUnblock allow action to take place without a token parameter present * (bug 35317) CSRF in Special:Upload Revert r56793, which removed the CSRF check for Special:Upload for normal file uploads. Cross-site posting of file uploads without user interaction has been possible since at least as early as Chrome 8 (late 2010) and Firefox 6 (mid 2011). Commonist has used api.php since version 0.4.0 (April 2010), and the API already requires an edit token, so Commonist 0.4.0+ is not affected by this change. * (bug 34907) Fix for CSRF vulnerability due to mw.user.tokens. Patch by Roan Kattouw and Tim Starling. * Filter out private modules early in ResourceLoader::makeResponse() and just pretend they weren't specified. This means these modules cannot be loaded through load.php . This filtering must not happen in makeModuleResponse(), because that would break inlining. * Force inlining of private modules in OutputPage::makeResourceLoaderLink(), disregarding $wgResourceLoaderInlinePrivateModules * Remove $wgResourceLoaderInlinePrivateModules * Remove special treatment of private modules ($private) in ResourceLoader::makeResponse() and sendResponseHeaders(), because we're not allowing private modules to be loaded through here any more * Remove identity checks in ResourceLoaderUserOptionsModule and ResourceLoaderUserCSSPrefsModule, they didn't make a lot of sense before but they're certainly useless now. * Factored out error comment construction in ResourceLoader.php and stripped comment terminations from exception messages. I didn't find an XSS vulnerability but it looked scary. Patchset2: Removes whitespace error that prevented automatic merge by Gerrit: includes/resourceloader/ResourceLoaderUserOptionsModule.php Change-Id: I2dec8b8caf9db3c64919763865cc10cccdd6a1a3
2012-03-22 19:52:37 +00:00
$exp = min( $maxage, $smaxage );
header( 'Expires: ' . wfTimestamp( TS_RFC2822, $exp + time() ) );
}
foreach ( $extra as $header ) {
header( $header );
}
}
2010-09-04 12:53:01 +00:00
/**
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* Respond with HTTP 304 Not Modified if appropiate.
*
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* If there's an If-None-Match header, respond with a 304 appropriately
* and clear out the output buffer. If the client cache is too old then do nothing.
*
* @param ResourceLoaderContext $context
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @param string $etag ETag header value
* @return bool True if HTTP 304 was sent and output handled
*/
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
protected function tryRespondNotModified( ResourceLoaderContext $context, $etag ) {
// See RFC 2616 § 14.26 If-None-Match
// https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
$clientKeys = $context->getRequest()->getHeader( 'If-None-Match', WebRequest::GETHEADER_LIST );
// Never send 304s in debug mode
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
if ( $clientKeys !== false && !$context->getDebug() && in_array( $etag, $clientKeys ) ) {
// There's another bug in ob_gzhandler (see also the comment at
// the top of this function) that causes it to gzip even empty
// responses, meaning it's impossible to produce a truly empty
// response (because the gzip header is always there). This is
// a problem because 304 responses have to be completely empty
// per the HTTP spec, and Firefox behaves buggily when they're not.
// See also https://bugs.php.net/bug.php?id=51579
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
// To work around this, we tear down all output buffering before
// sending the 304.
wfResetOutputBuffers( /* $resetGzipEncoding = */ true );
HttpStatus::header( 304 );
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
$this->sendResponseHeaders( $context, $etag, false );
return true;
}
return false;
}
/**
* Send out code for a response from file cache if possible.
*
* @param ResourceFileCache $fileCache Cache object for this request URL
* @param ResourceLoaderContext $context Context in which to generate a response
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
* @param string $etag ETag header value
* @return bool If this found a cache file and handled the response
*/
protected function tryRespondFromFileCache(
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
ResourceFileCache $fileCache,
ResourceLoaderContext $context,
$etag
) {
$rlMaxage = $this->config->get( 'ResourceLoaderMaxage' );
// Buffer output to catch warnings.
ob_start();
// Get the maximum age the cache can be
$maxage = is_null( $context->getVersion() )
? $rlMaxage['unversioned']['server']
: $rlMaxage['versioned']['server'];
// Minimum timestamp the cache file must have
$good = $fileCache->isCacheGood( wfTimestamp( TS_MW, time() - $maxage ) );
if ( !$good ) {
try { // RL always hits the DB on file cache miss...
wfGetDB( DB_REPLICA );
} catch ( DBConnectionError $e ) { // ...check if we need to fallback to cache
$good = $fileCache->isCacheGood(); // cache existence check
}
}
if ( $good ) {
$ts = $fileCache->cacheTimestamp();
// Send content type and cache headers
resourceloader: Replace timestamp system with version hashing Modules now track their version via getVersionHash() instead of getModifiedTime(). == Background == While some resources have observeable timestamps (e.g. files stored on disk), many other resources do not. E.g. config variables, and module definitions. For static file modules, one can e.g. revert one of more files in a module to a previous version and not affect the max timestamp. Wiki modules include pages only if they exist. The user module supports common.js and skin.js. By default neither exists. If a user has both, and then the less-recently modified one is deleted, the max-timestamp remains unchanged. For client-side caching, batch requests use "Math.max" on the relevant timestamps. Again, if a module changes but another module is more recent (e.g. out-of-order deployment, or out-of-order discovery), the change would not result in a cache miss. More scenarios can be found in the associated Phabricator tasks. == Version hash == Previously we virtually mapped these variables to a timestamp by storing the current time alongside a hash of the value in ObjectCache. Considering the number of possible request contexts (wikis * modules * users * skins * languages) this doesn't work well. It results in needless cache invalidation when the first time observation is purged due to LRU algorithms. It also has other minor bugs leading to fewer cache hits. All modules automatically get the benefits of version hashing with this change. The old getDefinitionMtime() and getHashMtime() have been replaced with dummies that return 1. These functions are often called from getModifiedTime() in subclasses. For backward-compatibility, their respective values (definition summary and hash) are now included in getVersionHash directly. As examples, the following modules have been updated to use getVersionHash directly. Other modules still work fine and can be updated later. * ResourceLoaderFileModule * ResourceLoaderEditToolbarModule * ResourceLoaderStartUpModule * ResourceLoaderWikiModule The presence of hashes in place of timestamps increases the startup module size on a default MediaWiki install from 4.4k to 5.8k (after gzip and minification). == ETag == Since timestamps are no longer tracked, we need a different way to implement caching for cache proxies (e.g. Varnish) and web browsers. Previously we used the Last-Modified header (in combination with Cache-Control and Expires). Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match). Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default. They instruct browsers to allow usage of partial Range requests. Since our responses are dynamically generated, we need to use the Weak version of ETag. While this sounds bad, it's no different than Last-Modified. As reassured by RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the specified behaviour behind Last-Modified follows the same "Weak" caching logic as Entity tags. It's just that entity tags are capable of a stricter mode (whereas Last-Modified is inherently weak). == File cache == If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache load.php responses. While the blind TTL handling (during the allowed expiry period) is still maxage/timestamp based, tryRespondNotModified() now requires the caller to know the expected ETag. For this to work, the FileCache handling had to be moved from the top of ResoureLoader::respond() to after the expected ETag is computed. This also allows us to remove the duplicate tryRespondNotModified() handling since that's is already handled by ResourceLoader::respond() meanwhile. == Misc == * Remove redundant modifiedTime cache in ResourceLoaderFileModule. * Change bugzilla references to Phabricator. * Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this logic was duplicated in each place the modified timestamp was used. * It's easy to forget calling the parent class in getDefinitionSummary(). Previously this method only tracked 'class' by default. As such, various extensions hardcoded that one value instead of calling the parent and extending the array. To better prevent this in the future, getVersionHash() now asserts that the '_cacheEpoch' property made it through. * tests: Don't use getDefinitionSummary() as an API. Fix ResourceLoaderWikiModuleTest to call getPages properly. * In tests, the default timestamp used to be 1388534400000 (which is the unix time of 20140101000000; the unit tests' CacheEpoch). The new version hash of these modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of: '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}' * Add sha1.js library for client-side hash generation. Compared various different implementations for code size (after minfication/gzip), and speed (when used for short hexidecimal strings). https://jsperf.com/sha1-implementations - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k) http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js Chrome: 45k, Firefox: 89k, Safari: 92k - jsSHA <https://github.com/Caligatio/jsSHA> https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k) Chrome: 65k, Firefox: 53k, Safari: 69k - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k) https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js Chrome: 200k, Firefox: 280k, Safari: 78k Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous, only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being needed in the module loader, we'd have to load the fallback regardless. Considering this is not used in a critical path for performance, it's not worth shipping two implementations for this optimisation. May also resolve: * T44094 * T90411 * T94810 Bug: T94074 Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-04-29 22:53:24 +00:00
$this->sendResponseHeaders( $context, $etag, false );
$response = $fileCache->fetchText();
// Capture any PHP warnings from the output buffer and append them to the
// response in a comment if we're in debug mode.
if ( $context->getDebug() ) {
$warnings = ob_get_contents();
if ( strlen( $warnings ) ) {
$response = self::makeComment( $warnings ) . $response;
}
}
// Remove the output buffer and output the response
ob_end_clean();
echo $response . "\n/* Cached {$ts} */";
return true; // cache hit
}
// Clear buffer
ob_end_clean();
return false; // cache miss
}
/**
* Generate a CSS or JS comment block.
*
* Only use this for public data, not error message details.
*
* @param string $text
* @return string
*/
public static function makeComment( $text ) {
* (bug 34212) ApiBlock/ApiUnblock allow action to take place without a token parameter present * (bug 35317) CSRF in Special:Upload Revert r56793, which removed the CSRF check for Special:Upload for normal file uploads. Cross-site posting of file uploads without user interaction has been possible since at least as early as Chrome 8 (late 2010) and Firefox 6 (mid 2011). Commonist has used api.php since version 0.4.0 (April 2010), and the API already requires an edit token, so Commonist 0.4.0+ is not affected by this change. * (bug 34907) Fix for CSRF vulnerability due to mw.user.tokens. Patch by Roan Kattouw and Tim Starling. * Filter out private modules early in ResourceLoader::makeResponse() and just pretend they weren't specified. This means these modules cannot be loaded through load.php . This filtering must not happen in makeModuleResponse(), because that would break inlining. * Force inlining of private modules in OutputPage::makeResourceLoaderLink(), disregarding $wgResourceLoaderInlinePrivateModules * Remove $wgResourceLoaderInlinePrivateModules * Remove special treatment of private modules ($private) in ResourceLoader::makeResponse() and sendResponseHeaders(), because we're not allowing private modules to be loaded through here any more * Remove identity checks in ResourceLoaderUserOptionsModule and ResourceLoaderUserCSSPrefsModule, they didn't make a lot of sense before but they're certainly useless now. * Factored out error comment construction in ResourceLoader.php and stripped comment terminations from exception messages. I didn't find an XSS vulnerability but it looked scary. Patchset2: Removes whitespace error that prevented automatic merge by Gerrit: includes/resourceloader/ResourceLoaderUserOptionsModule.php Change-Id: I2dec8b8caf9db3c64919763865cc10cccdd6a1a3
2012-03-22 19:52:37 +00:00
$encText = str_replace( '*/', '* /', $text );
return "/*\n$encText\n*/\n";
}
/**
* Handle exception display.
*
* @param Exception $e Exception to be shown to the user
* @return string Sanitized text in a CSS/JS comment that can be returned to the user
*/
public static function formatException( $e ) {
return self::makeComment( self::formatExceptionNoComment( $e ) );
}
/**
* Handle exception display.
*
* @since 1.25
* @param Exception $e Exception to be shown to the user
* @return string Sanitized text that can be returned to the user
*/
protected static function formatExceptionNoComment( $e ) {
global $wgShowExceptionDetails;
if ( !$wgShowExceptionDetails ) {
return MWExceptionHandler::getPublicLogMessage( $e );
}
return MWExceptionHandler::getLogMessage( $e ) .
"\nBacktrace:\n" .
MWExceptionHandler::getRedactedTraceAsString( $e );
}
/**
* Generate code for a response.
*
* Calling this method also populates the `errors` and `headers` members,
* later used by respond().
*
* @param ResourceLoaderContext $context Context in which to generate a response
* @param ResourceLoaderModule[] $modules List of module objects keyed by module name
* @param string[] $missing List of requested module names that are unregistered (optional)
* @return string Response data
*/
public function makeModuleResponse( ResourceLoaderContext $context,
array $modules, array $missing = []
) {
$out = '';
$states = [];
if ( $modules === [] && $missing === [] ) {
return <<<MESSAGE
/* This file is the Web entry point for MediaWiki's ResourceLoader:
<https://www.mediawiki.org/wiki/ResourceLoader>. In this request,
no modules were requested. Max made me put this here. */
MESSAGE;
}
$image = $context->getImageObj();
if ( $image ) {
$data = $image->getImageData( $context );
if ( $data === false ) {
$data = '';
$this->errors[] = 'Image generation failed';
}
return $data;
}
foreach ( $missing as $name ) {
$states[$name] = 'missing';
}
// Generate output
$isRaw = false;
$filter = $context->getOnly() === 'styles' ? 'minify-css' : 'minify-js';
foreach ( $modules as $name => $module ) {
try {
$content = $module->getModuleContent( $context );
$implementKey = $name . '@' . $module->getVersionHash( $context );
$strContent = '';
if ( isset( $content['headers'] ) ) {
$this->extraHeaders = array_merge( $this->extraHeaders, $content['headers'] );
}
// Append output
switch ( $context->getOnly() ) {
case 'scripts':
$scripts = $content['scripts'];
if ( is_string( $scripts ) ) {
// Load scripts raw...
$strContent = $scripts;
} elseif ( is_array( $scripts ) ) {
// ...except when $scripts is an array of URLs or an associative array
$strContent = self::makeLoaderImplementScript( $implementKey, $scripts, [], [], [] );
}
break;
case 'styles':
$styles = $content['styles'];
// We no longer separate into media, they are all combined now with
ResourceLoader: Refactor style loading Fixes: * bug 31676: Work around IE stylesheet limit. * bug 35562: @import styles broken in modules that combine multiple stylesheets. * bug 40498: Don't output empty "@media print { }" blocks. * bug 40500: Don't ignore media-type for urls in debug mode. Approach: * Re-use the same <style> tag so that we stay under the 31 stylesheet limit in IE. Unless the to-be-added css text from the being-loaded module contains @import, in which case we do create a new <style> tag and then re-use that one from that point on (bug 31676). * Return stylesheets as arrays, instead of a concatenated string. This fixes bug 35562, because @import only works when at the top of a stylesheet. By not unconditionally concatenating files within a module on the server side already, @import will work in e.g. module 'site' that contains 2 wiki pages. This is normalized in ResourceLoader::makeCombinedStyles(), so far only ResourceLoaderWikiModule makes use of this. Misc. clean up and bug fixes: * Reducing usage of jQuery() and mw.html.element() where native DOM would be very simple and faster. Aside from simplicity and speed, this is also working towards a more stand-alone ResourceLoader. * Trim server output a little bit more - Redundant new line after minify-css (it is now an array, so no need to keep space afterwards) - Redundant semi-colon after minify-js if it ends in a colon * Allow space in styleTest.css.php * Clean up and extend unit tests to cover for these features and bug fixes. * Don't set styleEl.rel = 'stylesheet'; that has no business on a <style> tag. * Fix bug in mw.loader's addStyleTag(). It turns out IE6 has an odd security measure that does not allow manipulation of elements (at least style tags) that are created by a different script (even if that script was served from the same domain/origin etc.). We didn't ran into this before because we only created new style tags, never appended to them. Now that we do, this came up. Took a while to figure out because it was created by mediawiki.js but it calls jQuery which did the actual dom insertion. Odd thing is, we load jquery.js and mediawiki.js in the same request even... Without this all css-url related mw.loader tests would fail in IE6. * mediawiki.js and mediawiki.test.js now pass jshint again. Tested (and passing qunit/?module=mediawiki; 123 of 123): * Chrome 14, 21 * Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta * IE 6, 7, 8, 9 * Safari 4.0, 5.0, 5.1 * Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta * iPhone 3GS / iOS 3.0 / Mobile Safari 4.0 iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5 iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0 Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
2012-07-25 21:20:21 +00:00
// custom media type groups into @media .. {} sections as part of the css string.
// Module returns either an empty array or a numerical array with css strings.
$strContent = isset( $styles['css'] ) ? implode( '', $styles['css'] ) : '';
break;
default:
$scripts = $content['scripts'] ?? '';
if ( is_string( $scripts ) ) {
if ( $name === 'site' || $name === 'user' ) {
// Legacy scripts that run in the global scope without a closure.
// mw.loader.implement will use globalEval if scripts is a string.
// Minify manually here, because general response minification is
// not effective due it being a string literal, not a function.
if ( !self::inDebugMode() ) {
$scripts = self::filter( 'minify-js', $scripts ); // T107377
}
} else {
$scripts = new XmlJsCode( $scripts );
}
}
$strContent = self::makeLoaderImplementScript(
$implementKey,
$scripts,
$content['styles'] ?? [],
isset( $content['messagesBlob'] ) ? new XmlJsCode( $content['messagesBlob'] ) : [],
$content['templates'] ?? []
);
break;
}
if ( !$context->getDebug() ) {
$strContent = self::filter( $filter, $strContent );
}
if ( $context->getOnly() === 'scripts' ) {
// Use a linebreak between module scripts (T162719)
$out .= $this->ensureNewline( $strContent );
} else {
$out .= $strContent;
}
} catch ( Exception $e ) {
$this->outputErrorAndLog( $e, 'Generating module package failed: {exception}' );
// Respond to client with error-state instead of module implementation
$states[$name] = 'error';
unset( $modules[$name] );
}
$isRaw |= $module->isRaw();
}
2010-09-04 12:53:01 +00:00
// Update module states
if ( $context->shouldIncludeScripts() && !$context->getRaw() && !$isRaw ) {
if ( count( $modules ) && $context->getOnly() === 'scripts' ) {
// Set the state of modules loaded as only scripts to ready as
// they don't have an mw.loader.implement wrapper that sets the state
foreach ( $modules as $name => $module ) {
$states[$name] = 'ready';
}
}
// Set the state of modules we didn't respond to with mw.loader.implement
if ( count( $states ) ) {
$stateScript = self::makeLoaderStateScript( $states );
if ( !$context->getDebug() ) {
$stateScript = self::filter( 'minify-js', $stateScript );
}
// Use a linebreak between module script and state script (T162719)
$out = $this->ensureNewline( $out ) . $stateScript;
}
} else {
if ( $states ) {
$this->errors[] = 'Problematic modules: '
. self::encodeJsonForScript( $states );
}
}
2010-09-04 12:53:01 +00:00
return $out;
}
/**
* Ensure the string is either empty or ends in a line break
* @param string $str
* @return string
*/
private function ensureNewline( $str ) {
$end = substr( $str, -1 );
if ( $end === false || $end === "\n" ) {
return $str;
}
return $str . "\n";
}
/**
* Get names of modules that use a certain message.
*
* @param string $messageKey
* @return array List of module names
*/
public function getModulesByMessage( $messageKey ) {
$moduleNames = [];
foreach ( $this->getModuleNames() as $moduleName ) {
$module = $this->getModule( $moduleName );
if ( in_array( $messageKey, $module->getMessages() ) ) {
$moduleNames[] = $moduleName;
}
}
return $moduleNames;
}
/**
* Return JS code that calls mw.loader.implement with given module properties.
*
* @param string $name Module name or implement key (format "`[name]@[version]`")
* @param XmlJsCode|array|string $scripts Code as XmlJsCode (to be wrapped in a closure),
* list of URLs to JavaScript files, string of JavaScript for `$.globalEval`, or array with
* 'files' and 'main' properties (see ResourceLoaderModule::getScript())
* @param mixed $styles Array of CSS strings keyed by media type, or an array of lists of URLs
* to CSS files keyed by media type
* @param mixed $messages List of messages associated with this module. May either be an
* associative array mapping message key to value, or a JSON-encoded message blob containing
* the same data, wrapped in an XmlJsCode object.
* @param array $templates Keys are name of templates and values are the source of
* the template.
* @throws MWException
* @return string JavaScript code
*/
protected static function makeLoaderImplementScript(
$name, $scripts, $styles, $messages, $templates
) {
if ( $scripts instanceof XmlJsCode ) {
if ( $scripts->value === '' ) {
$scripts = null;
} elseif ( self::inDebugMode() ) {
$scripts = new XmlJsCode( "function ( $, jQuery, require, module ) {\n{$scripts->value}\n}" );
} else {
$scripts = new XmlJsCode( 'function($,jQuery,require,module){' . $scripts->value . '}' );
}
} elseif ( is_array( $scripts ) && isset( $scripts['files'] ) ) {
$files = $scripts['files'];
foreach ( $files as $path => &$file ) {
// $file is changed (by reference) from a descriptor array to the content of the file
// All of these essentially do $file = $file['content'];, some just have wrapping around it
if ( $file['type'] === 'script' ) {
// Multi-file modules only get two parameters ($ and jQuery are being phased out)
if ( self::inDebugMode() ) {
$file = new XmlJsCode( "function ( require, module ) {\n{$file['content']}\n}" );
} else {
$file = new XmlJsCode( 'function(require,module){' . $file['content'] . '}' );
}
} else {
$file = $file['content'];
}
}
$scripts = XmlJsCode::encodeObject( [
'main' => $scripts['main'],
'files' => XmlJsCode::encodeObject( $files, self::inDebugMode() )
], self::inDebugMode() );
} elseif ( !is_string( $scripts ) && !is_array( $scripts ) ) {
throw new MWException( 'Invalid scripts error. Array of URLs or string of code expected.' );
}
// mw.loader.implement requires 'styles', 'messages' and 'templates' to be objects (not
// arrays). json_encode considers empty arrays to be numerical and outputs "[]" instead
// of "{}". Force them to objects.
$module = [
$name,
$scripts,
(object)$styles,
(object)$messages,
(object)$templates
];
self::trimArray( $module );
return Xml::encodeJsCall( 'mw.loader.implement', $module, self::inDebugMode() );
}
/**
* Returns JS code which, when called, will register a given list of messages.
*
* @param mixed $messages Either an associative array mapping message key to value, or a
* JSON-encoded message blob containing the same data, wrapped in an XmlJsCode object.
* @return string JavaScript code
*/
public static function makeMessageSetScript( $messages ) {
return Xml::encodeJsCall(
'mw.messages.set',
[ (object)$messages ],
self::inDebugMode()
);
}
/**
* Combines an associative array mapping media type to CSS into a
* single stylesheet with "@media" blocks.
*
* @param array $stylePairs Array keyed by media type containing (arrays of) CSS strings
* @return array
*/
public static function makeCombinedStyles( array $stylePairs ) {
$out = [];
ResourceLoader: Refactor style loading Fixes: * bug 31676: Work around IE stylesheet limit. * bug 35562: @import styles broken in modules that combine multiple stylesheets. * bug 40498: Don't output empty "@media print { }" blocks. * bug 40500: Don't ignore media-type for urls in debug mode. Approach: * Re-use the same <style> tag so that we stay under the 31 stylesheet limit in IE. Unless the to-be-added css text from the being-loaded module contains @import, in which case we do create a new <style> tag and then re-use that one from that point on (bug 31676). * Return stylesheets as arrays, instead of a concatenated string. This fixes bug 35562, because @import only works when at the top of a stylesheet. By not unconditionally concatenating files within a module on the server side already, @import will work in e.g. module 'site' that contains 2 wiki pages. This is normalized in ResourceLoader::makeCombinedStyles(), so far only ResourceLoaderWikiModule makes use of this. Misc. clean up and bug fixes: * Reducing usage of jQuery() and mw.html.element() where native DOM would be very simple and faster. Aside from simplicity and speed, this is also working towards a more stand-alone ResourceLoader. * Trim server output a little bit more - Redundant new line after minify-css (it is now an array, so no need to keep space afterwards) - Redundant semi-colon after minify-js if it ends in a colon * Allow space in styleTest.css.php * Clean up and extend unit tests to cover for these features and bug fixes. * Don't set styleEl.rel = 'stylesheet'; that has no business on a <style> tag. * Fix bug in mw.loader's addStyleTag(). It turns out IE6 has an odd security measure that does not allow manipulation of elements (at least style tags) that are created by a different script (even if that script was served from the same domain/origin etc.). We didn't ran into this before because we only created new style tags, never appended to them. Now that we do, this came up. Took a while to figure out because it was created by mediawiki.js but it calls jQuery which did the actual dom insertion. Odd thing is, we load jquery.js and mediawiki.js in the same request even... Without this all css-url related mw.loader tests would fail in IE6. * mediawiki.js and mediawiki.test.js now pass jshint again. Tested (and passing qunit/?module=mediawiki; 123 of 123): * Chrome 14, 21 * Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta * IE 6, 7, 8, 9 * Safari 4.0, 5.0, 5.1 * Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta * iPhone 3GS / iOS 3.0 / Mobile Safari 4.0 iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5 iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0 Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
2012-07-25 21:20:21 +00:00
foreach ( $stylePairs as $media => $styles ) {
// ResourceLoaderFileModule::getStyle can return the styles
// as a string or an array of strings. This is to allow separation in
// the front-end.
$styles = (array)$styles;
ResourceLoader: Refactor style loading Fixes: * bug 31676: Work around IE stylesheet limit. * bug 35562: @import styles broken in modules that combine multiple stylesheets. * bug 40498: Don't output empty "@media print { }" blocks. * bug 40500: Don't ignore media-type for urls in debug mode. Approach: * Re-use the same <style> tag so that we stay under the 31 stylesheet limit in IE. Unless the to-be-added css text from the being-loaded module contains @import, in which case we do create a new <style> tag and then re-use that one from that point on (bug 31676). * Return stylesheets as arrays, instead of a concatenated string. This fixes bug 35562, because @import only works when at the top of a stylesheet. By not unconditionally concatenating files within a module on the server side already, @import will work in e.g. module 'site' that contains 2 wiki pages. This is normalized in ResourceLoader::makeCombinedStyles(), so far only ResourceLoaderWikiModule makes use of this. Misc. clean up and bug fixes: * Reducing usage of jQuery() and mw.html.element() where native DOM would be very simple and faster. Aside from simplicity and speed, this is also working towards a more stand-alone ResourceLoader. * Trim server output a little bit more - Redundant new line after minify-css (it is now an array, so no need to keep space afterwards) - Redundant semi-colon after minify-js if it ends in a colon * Allow space in styleTest.css.php * Clean up and extend unit tests to cover for these features and bug fixes. * Don't set styleEl.rel = 'stylesheet'; that has no business on a <style> tag. * Fix bug in mw.loader's addStyleTag(). It turns out IE6 has an odd security measure that does not allow manipulation of elements (at least style tags) that are created by a different script (even if that script was served from the same domain/origin etc.). We didn't ran into this before because we only created new style tags, never appended to them. Now that we do, this came up. Took a while to figure out because it was created by mediawiki.js but it calls jQuery which did the actual dom insertion. Odd thing is, we load jquery.js and mediawiki.js in the same request even... Without this all css-url related mw.loader tests would fail in IE6. * mediawiki.js and mediawiki.test.js now pass jshint again. Tested (and passing qunit/?module=mediawiki; 123 of 123): * Chrome 14, 21 * Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta * IE 6, 7, 8, 9 * Safari 4.0, 5.0, 5.1 * Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta * iPhone 3GS / iOS 3.0 / Mobile Safari 4.0 iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5 iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0 Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
2012-07-25 21:20:21 +00:00
foreach ( $styles as $style ) {
$style = trim( $style );
// Don't output an empty "@media print { }" block (T42498)
ResourceLoader: Refactor style loading Fixes: * bug 31676: Work around IE stylesheet limit. * bug 35562: @import styles broken in modules that combine multiple stylesheets. * bug 40498: Don't output empty "@media print { }" blocks. * bug 40500: Don't ignore media-type for urls in debug mode. Approach: * Re-use the same <style> tag so that we stay under the 31 stylesheet limit in IE. Unless the to-be-added css text from the being-loaded module contains @import, in which case we do create a new <style> tag and then re-use that one from that point on (bug 31676). * Return stylesheets as arrays, instead of a concatenated string. This fixes bug 35562, because @import only works when at the top of a stylesheet. By not unconditionally concatenating files within a module on the server side already, @import will work in e.g. module 'site' that contains 2 wiki pages. This is normalized in ResourceLoader::makeCombinedStyles(), so far only ResourceLoaderWikiModule makes use of this. Misc. clean up and bug fixes: * Reducing usage of jQuery() and mw.html.element() where native DOM would be very simple and faster. Aside from simplicity and speed, this is also working towards a more stand-alone ResourceLoader. * Trim server output a little bit more - Redundant new line after minify-css (it is now an array, so no need to keep space afterwards) - Redundant semi-colon after minify-js if it ends in a colon * Allow space in styleTest.css.php * Clean up and extend unit tests to cover for these features and bug fixes. * Don't set styleEl.rel = 'stylesheet'; that has no business on a <style> tag. * Fix bug in mw.loader's addStyleTag(). It turns out IE6 has an odd security measure that does not allow manipulation of elements (at least style tags) that are created by a different script (even if that script was served from the same domain/origin etc.). We didn't ran into this before because we only created new style tags, never appended to them. Now that we do, this came up. Took a while to figure out because it was created by mediawiki.js but it calls jQuery which did the actual dom insertion. Odd thing is, we load jquery.js and mediawiki.js in the same request even... Without this all css-url related mw.loader tests would fail in IE6. * mediawiki.js and mediawiki.test.js now pass jshint again. Tested (and passing qunit/?module=mediawiki; 123 of 123): * Chrome 14, 21 * Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta * IE 6, 7, 8, 9 * Safari 4.0, 5.0, 5.1 * Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta * iPhone 3GS / iOS 3.0 / Mobile Safari 4.0 iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5 iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0 Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
2012-07-25 21:20:21 +00:00
if ( $style !== '' ) {
// Transform the media type based on request params and config
// The way that this relies on $wgRequest to propagate request params is slightly evil
$media = OutputPage::transformCssMedia( $media );
if ( $media === '' || $media == 'all' ) {
$out[] = $style;
} elseif ( is_string( $media ) ) {
ResourceLoader: Refactor style loading Fixes: * bug 31676: Work around IE stylesheet limit. * bug 35562: @import styles broken in modules that combine multiple stylesheets. * bug 40498: Don't output empty "@media print { }" blocks. * bug 40500: Don't ignore media-type for urls in debug mode. Approach: * Re-use the same <style> tag so that we stay under the 31 stylesheet limit in IE. Unless the to-be-added css text from the being-loaded module contains @import, in which case we do create a new <style> tag and then re-use that one from that point on (bug 31676). * Return stylesheets as arrays, instead of a concatenated string. This fixes bug 35562, because @import only works when at the top of a stylesheet. By not unconditionally concatenating files within a module on the server side already, @import will work in e.g. module 'site' that contains 2 wiki pages. This is normalized in ResourceLoader::makeCombinedStyles(), so far only ResourceLoaderWikiModule makes use of this. Misc. clean up and bug fixes: * Reducing usage of jQuery() and mw.html.element() where native DOM would be very simple and faster. Aside from simplicity and speed, this is also working towards a more stand-alone ResourceLoader. * Trim server output a little bit more - Redundant new line after minify-css (it is now an array, so no need to keep space afterwards) - Redundant semi-colon after minify-js if it ends in a colon * Allow space in styleTest.css.php * Clean up and extend unit tests to cover for these features and bug fixes. * Don't set styleEl.rel = 'stylesheet'; that has no business on a <style> tag. * Fix bug in mw.loader's addStyleTag(). It turns out IE6 has an odd security measure that does not allow manipulation of elements (at least style tags) that are created by a different script (even if that script was served from the same domain/origin etc.). We didn't ran into this before because we only created new style tags, never appended to them. Now that we do, this came up. Took a while to figure out because it was created by mediawiki.js but it calls jQuery which did the actual dom insertion. Odd thing is, we load jquery.js and mediawiki.js in the same request even... Without this all css-url related mw.loader tests would fail in IE6. * mediawiki.js and mediawiki.test.js now pass jshint again. Tested (and passing qunit/?module=mediawiki; 123 of 123): * Chrome 14, 21 * Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta * IE 6, 7, 8, 9 * Safari 4.0, 5.0, 5.1 * Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta * iPhone 3GS / iOS 3.0 / Mobile Safari 4.0 iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5 iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0 Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
2012-07-25 21:20:21 +00:00
$out[] = "@media $media {\n" . str_replace( "\n", "\n\t", "\t" . $style ) . "}";
}
// else: skip
}
}
}
return $out;
}
/**
* Wrapper around json_encode that avoids needless escapes,
* and pretty-prints in debug mode.
*
* @internal
* @since 1.32
* @param bool|string|array $data
* @return string JSON
*/
public static function encodeJsonForScript( $data ) {
// Keep output as small as possible by disabling needless escape modes
// that PHP uses by default.
// However, while most module scripts are only served on HTTP responses
// for JavaScript, some modules can also be embedded in the HTML as inline
// scripts. This, and the fact that we sometimes need to export strings
// containing user-generated content and labels that may genuinely contain
// a sequences like "</script>", we need to encode either '/' or '<'.
// By default PHP escapes '/'. Let's escape '<' instead which is less common
// and allows URLs to mostly remain readable.
$jsonFlags = JSON_UNESCAPED_SLASHES |
JSON_UNESCAPED_UNICODE |
JSON_HEX_TAG |
JSON_HEX_AMP;
if ( self::inDebugMode() ) {
$jsonFlags |= JSON_PRETTY_PRINT;
}
return json_encode( $data, $jsonFlags );
}
/**
* Returns a JS call to mw.loader.state, which sets the state of one
* ore more modules to a given value. Has two calling conventions:
*
* - ResourceLoader::makeLoaderStateScript( $name, $state ):
* Set the state of a single module called $name to $state
*
* - ResourceLoader::makeLoaderStateScript( [ $name => $state, ... ] ):
* Set the state of modules with the given names to the given states
*
* @param array|string $states
* @param string|null $state
* @return string JavaScript code
*/
public static function makeLoaderStateScript( $states, $state = null ) {
if ( !is_array( $states ) ) {
$states = [ $states => $state ];
}
return Xml::encodeJsCall(
'mw.loader.state',
[ $states ],
self::inDebugMode()
);
}
private static function isEmptyObject( stdClass $obj ) {
foreach ( $obj as $key => $value ) {
return false;
}
return true;
}
/**
* Remove empty values from the end of an array.
*
* Values considered empty:
*
* - null
* - []
* - new XmlJsCode( '{}' )
* - new stdClass() // (object) []
*
* @param array $array
*/
private static function trimArray( array &$array ) {
$i = count( $array );
while ( $i-- ) {
if ( $array[$i] === null
|| $array[$i] === []
|| ( $array[$i] instanceof XmlJsCode && $array[$i]->value === '{}' )
|| ( $array[$i] instanceof stdClass && self::isEmptyObject( $array[$i] ) )
) {
unset( $array[$i] );
} else {
break;
}
}
}
/**
* Returns JS code which calls mw.loader.register with the given
* parameter.
*
* @par Example
* @code
*
* ResourceLoader::makeLoaderRegisterScript( [
* [ $name1, $version1, $dependencies1, $group1, $source1, $skip1 ],
* [ $name2, $version2, $dependencies1, $group2, $source2, $skip2 ],
* ...
* ] ):
* @endcode
*
* @internal
* @since 1.32
* @param array $modules Array of module registration arrays, each containing
* - string: module name
* - string: module version
* - array|null: List of dependencies (optional)
* - string|null: Module group (optional)
* - string|null: Name of foreign module source, or 'local' (optional)
* - string|null: Script body of a skip function (optional)
* @return string JavaScript code
*/
public static function makeLoaderRegisterScript( array $modules ) {
// Optimisation: Transform dependency names into indexes when possible
// to produce smaller output. They are expanded by mw.loader.register on
// the other end using resolveIndexedDependencies().
$index = [];
foreach ( $modules as $i => &$module ) {
// Build module name index
$index[$module[0]] = $i;
}
foreach ( $modules as &$module ) {
if ( isset( $module[2] ) ) {
foreach ( $module[2] as &$dependency ) {
if ( isset( $index[$dependency] ) ) {
// Replace module name in dependency list with index
$dependency = $index[$dependency];
}
}
}
}
array_walk( $modules, [ 'self', 'trimArray' ] );
return Xml::encodeJsCall(
'mw.loader.register',
[ $modules ],
self::inDebugMode()
);
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
}
/**
* Returns JS code which calls mw.loader.addSource() with the given
* parameters. Has two calling conventions:
*
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
* - ResourceLoader::makeLoaderSourcesScript( $id, $properties ):
* Register a single source
*
* - ResourceLoader::makeLoaderSourcesScript( [ $id1 => $loadUrl, $id2 => $loadUrl, ... ] );
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
* Register sources with the given IDs and properties.
*
* @param string|array $sources Source ID
* @param string|null $loadUrl load.php url
* @return string JavaScript code
[ResourceLoader 2]: Add support for multiple loadScript sources Front-end: * New mw.loader method: addSource(). Call with two arguments or an object as first argument for multiple registrations * New property in module registry: "source". Optional for local modules (falls back to 'local'). When loading/using one or more modules, the worker will group the request by source and make separate requests to the sources as needed. * Re-arranging object properties in mw.loader.register to match the same order all other code parts use. * Adding documentation for 'source' and where missing updating it to include 'group' as well. * Refactor of mw.loader.work() by Roan Kattouw and Timo Tijhof:' -- Additional splitting layer by source (in addition to splitting by group), renamed 'groups' to 'splits' -- Clean up of the loop, and removing a no longer needed loop after the for-in-loop -- Much more function documentation in mw.loader.work() -- Moved caching of wgResourceLoaderMaxQueryLength out of the loop and renamed 'limit' to 'maxQueryLength Back-end changed provided through patch by Roan Kattouw (to avoid broken code between commits): * New method in ResourceLoader: addSource(). During construction of ResourceLoader this will be called by default for 'local' with loadScript property set to $wgLoadScript. Additional sources can be registered through $wgResourceLoaderSources (empty array by default) * Calling mw.loader.addSource from the startup module * Passing source to mw.loader.register from startup module * Some new static helper methods Use: * By default nothing should change in core, all modules simply default to 'local'. This info originates from the getSource()-method of the ResourceLoaderModule class, which is inherited to all core ResourceLoaderModule-implementations (none override it) * Third-party users and/or extensions can create new classes extending ResourceLoaderModule, re-implementing the getSource-method to return something else. Basic example: $wgResourceLoaderSources['mywiki'] = array( 'loadScript' => 'http://example.org/w/load.php' ); class MyCentralWikiModule extends ResourceLoaderModule { function getSource(){ return 'mywiki'; } } $wgResourceModules['cool.stuff'] => array( 'class' => 'MyCentralWikiModule' ); More complicated example // imagine some stuff with a ForeignGadgetRepo class, putting stuff in $wgResourceLoaderSources in the __construct() method class ForeignGadgetRepoGadget extends ResourceLoaderModule { function getSource(){ return $this->source; } } Loading: Loading is completely transparent, stuff like $wgOut->addModules() or mw.loader.loader/using both take it as any other module and load from the right source accordingly. -- This commit is part of the ResourceLoader 2 project.
2011-07-26 21:10:34 +00:00
*/
public static function makeLoaderSourcesScript( $sources, $loadUrl = null ) {
if ( !is_array( $sources ) ) {
$sources = [ $sources => $loadUrl ];
}
return Xml::encodeJsCall(
'mw.loader.addSource',
[ $sources ],
self::inDebugMode()
);
}
/**
resourceloader: Combine base modules and page modules requests This commit implements step 4 and step 5 of the plan outlined at T192623. Before this task began, the typical JavaScript execution flow was: * HTML triggers request for startup module (js req 1). * Startup module contains registry, site config, and triggers a request for the base modules (js req 2). * After the base modules arrive (which define jQuery and mw.loader), the startup module invokes a callback that processes RLQ, which is what will request modules for this page (js req 3). In past weeks, we have: * Made mediawiki.js independent of jQuery. * Spun off 'mediawiki.base' from mediawiki.js – for everything that wasn't needed for defining `mw.loader`. * Moved mediawiki.js from the base module request to being embedded as part of startup.js. The concept of dependencies is native to ResourceLoader, and thanks to the use of closures in mw.loader.implement() responses, we can download any number of interdependant modules in a single request (or parallel requests). Then, when a response arrives, mw.loader takes care to pause or resume execution as-needed. It is normal for ResourceLoader to batch several modules together, including their dependencies. As such, we can eliminate one of the two roundtrips required before a page can request modules. Specifically, we can eliminate "js req 2" (above), by making the two remaining base modules ("jquery" and "mediawiki.base") an implied dependency for all other modules, which ResourceLoader will naturally fetch and execute in the right order as part of the batch request. Bug: T192623 Change-Id: I17cd13dffebd6ae476044d8d038dc3974a1fa176
2018-07-12 20:09:28 +00:00
* Wraps JavaScript code to run after the startup module.
*
* @param string $script JavaScript code
* @return string JavaScript code
*/
public static function makeLoaderConditionalScript( $script ) {
// Adds a function to lazy-created RLQ
return '(window.RLQ=window.RLQ||[]).push(function(){' .
trim( $script ) . '});';
}
/**
* Wraps JavaScript code to run after a required module.
*
* @since 1.32
* @param string|string[] $modules Module name(s)
* @param string $script JavaScript code
* @return string JavaScript code
*/
public static function makeInlineCodeWithModule( $modules, $script ) {
// Adds an array to lazy-created RLQ
return '(window.RLQ=window.RLQ||[]).push(['
. self::encodeJsonForScript( $modules ) . ','
. 'function(){' . trim( $script ) . '}'
. ']);';
}
/**
* Returns an HTML script tag that runs given JS code after startup and base modules.
*
* The code will be wrapped in a closure, and it will be executed by ResourceLoader's
* startup module if the client has adequate support for MediaWiki JavaScript code.
*
* @param string $script JavaScript code
* @param string|null $nonce [optional] Content-Security-Policy nonce
* (from OutputPage::getCSPNonce)
* @return string|WrappedString HTML
*/
Initial support for Content Security Policy, disabled by default The primary goal here is a defense in depth measure to stop an attacker who found a bug in the parser allowing them to insert malicious attributes. This wouldn't stop someone who could insert a full script tag (since at current it can't distinguish between malicious and legit user js). It also would not prevent DOM-based or reflected XSS for anons, as the nonce value is guessable for anons when receiving a response cached by varnish. However, the limited protection of just stopping stored XSS where the attacker only has control of attributes, is still a big win in my opinion. (But it wouldn't prevent someone who has that type of xss from abusing things like data-ooui attribute). This will likely break many gadgets. Its expected that any sort of rollout on Wikimedia will be done very slowly, with lots of testing and the report-only option to begin with. This is behind feature flags that are off by default, so merging this patch should not cause any change in default behaviour. This may break some extensions (The most obvious one is charinsert (See fe648d41005), but will probably need some testing in report-only mode to see if anything else breaks) This uses the unsafe-eval option of CSP, in order to support RL's local storage thingy. For better security, we may want to remove some of the sillier uses of eval (e.g. jquery.ui.datepicker.js). For more info, see spec: https://www.w3.org/TR/CSP2/ Additionally see: https://www.mediawiki.org/wiki/Requests_for_comment/Content-Security-Policy Bug: T135963 Change-Id: I80f6f469ba4c0b608385483457df96ccb7429ae5
2016-02-29 04:13:10 +00:00
public static function makeInlineScript( $script, $nonce = null ) {
$js = self::makeLoaderConditionalScript( $script );
Initial support for Content Security Policy, disabled by default The primary goal here is a defense in depth measure to stop an attacker who found a bug in the parser allowing them to insert malicious attributes. This wouldn't stop someone who could insert a full script tag (since at current it can't distinguish between malicious and legit user js). It also would not prevent DOM-based or reflected XSS for anons, as the nonce value is guessable for anons when receiving a response cached by varnish. However, the limited protection of just stopping stored XSS where the attacker only has control of attributes, is still a big win in my opinion. (But it wouldn't prevent someone who has that type of xss from abusing things like data-ooui attribute). This will likely break many gadgets. Its expected that any sort of rollout on Wikimedia will be done very slowly, with lots of testing and the report-only option to begin with. This is behind feature flags that are off by default, so merging this patch should not cause any change in default behaviour. This may break some extensions (The most obvious one is charinsert (See fe648d41005), but will probably need some testing in report-only mode to see if anything else breaks) This uses the unsafe-eval option of CSP, in order to support RL's local storage thingy. For better security, we may want to remove some of the sillier uses of eval (e.g. jquery.ui.datepicker.js). For more info, see spec: https://www.w3.org/TR/CSP2/ Additionally see: https://www.mediawiki.org/wiki/Requests_for_comment/Content-Security-Policy Bug: T135963 Change-Id: I80f6f469ba4c0b608385483457df96ccb7429ae5
2016-02-29 04:13:10 +00:00
$escNonce = '';
if ( $nonce === null ) {
wfWarn( __METHOD__ . " did not get nonce. Will break CSP" );
} elseif ( $nonce !== false ) {
// If it was false, CSP is disabled, so no nonce attribute.
// Nonce should be only base64 characters, so should be safe,
// but better to be safely escaped than sorry.
$escNonce = ' nonce="' . htmlspecialchars( $nonce ) . '"';
}
return new WrappedString(
Initial support for Content Security Policy, disabled by default The primary goal here is a defense in depth measure to stop an attacker who found a bug in the parser allowing them to insert malicious attributes. This wouldn't stop someone who could insert a full script tag (since at current it can't distinguish between malicious and legit user js). It also would not prevent DOM-based or reflected XSS for anons, as the nonce value is guessable for anons when receiving a response cached by varnish. However, the limited protection of just stopping stored XSS where the attacker only has control of attributes, is still a big win in my opinion. (But it wouldn't prevent someone who has that type of xss from abusing things like data-ooui attribute). This will likely break many gadgets. Its expected that any sort of rollout on Wikimedia will be done very slowly, with lots of testing and the report-only option to begin with. This is behind feature flags that are off by default, so merging this patch should not cause any change in default behaviour. This may break some extensions (The most obvious one is charinsert (See fe648d41005), but will probably need some testing in report-only mode to see if anything else breaks) This uses the unsafe-eval option of CSP, in order to support RL's local storage thingy. For better security, we may want to remove some of the sillier uses of eval (e.g. jquery.ui.datepicker.js). For more info, see spec: https://www.w3.org/TR/CSP2/ Additionally see: https://www.mediawiki.org/wiki/Requests_for_comment/Content-Security-Policy Bug: T135963 Change-Id: I80f6f469ba4c0b608385483457df96ccb7429ae5
2016-02-29 04:13:10 +00:00
Html::inlineScript( $js, $nonce ),
"<script$escNonce>(window.RLQ=window.RLQ||[]).push(function(){",
'});</script>'
);
}
/**
* Returns JS code which will set the MediaWiki configuration array to
* the given value.
*
* @param array $configuration List of configuration values keyed by variable name
* @return string JavaScript code
* @throws Exception
*/
public static function makeConfigSetScript( array $configuration ) {
$js = Xml::encodeJsCall(
'mw.config.set',
[ $configuration ],
self::inDebugMode()
);
if ( $js === false ) {
$e = new Exception(
'JSON serialization of config data failed. ' .
'This usually means the config data is not valid UTF-8.'
);
MWExceptionHandler::logException( $e );
$js = Xml::encodeJsCall( 'mw.log.error', [ $e->__toString() ] );
}
return $js;
}
/**
* Convert an array of module names to a packed query string.
*
* For example, `[ 'foo.bar', 'foo.baz', 'bar.baz', 'bar.quux' ]`
* becomes `'foo.bar,baz|bar.baz,quux'`.
*
* This process is reversed by ResourceLoaderContext::expandModuleNames().
* See also mw.loader#buildModulesString() which is a port of this, used
* on the client-side.
*
* @param array $modules List of module names (strings)
* @return string Packed query string
*/
public static function makePackedModulesString( $modules ) {
$moduleMap = []; // [ prefix => [ suffixes ] ]
foreach ( $modules as $module ) {
$pos = strrpos( $module, '.' );
$prefix = $pos === false ? '' : substr( $module, 0, $pos );
$suffix = $pos === false ? $module : substr( $module, $pos + 1 );
$moduleMap[$prefix][] = $suffix;
}
$arr = [];
foreach ( $moduleMap as $prefix => $suffixes ) {
$p = $prefix === '' ? '' : $prefix . '.';
$arr[] = $p . implode( ',', $suffixes );
}
return implode( '|', $arr );
}
/**
* Determine whether debug mode was requested
* Order of priority is 1) request param, 2) cookie, 3) $wg setting
* @return bool
*/
public static function inDebugMode() {
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
if ( self::$debugMode === null ) {
global $wgRequest, $wgResourceLoaderDebug;
self::$debugMode = $wgRequest->getFuzzyBool( 'debug',
$wgRequest->getCookie( 'resourceLoaderDebug', '', $wgResourceLoaderDebug )
);
}
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
return self::$debugMode;
}
/**
* Reset static members used for caching.
*
* Global state and $wgRequest are evil, but we're using it right
* now and sometimes we need to be able to force ResourceLoader to
* re-evaluate the context because it has changed (e.g. in the test suite).
*
* @internal For use by unit tests
* @codeCoverageIgnore
resourceloader: Implement "skip function" feature A module can be registered with a skip function. Such function, if provided, will be invoked by the client when a module is queued for loading. If the function returns true, the client will bypass any further loading action and mark the module as 'ready'. This can be used to implement a feature test for a module providing a shim or polyfill. * Change visibility of method ResourceLoader::filter to public. So that it can be invoked by ResourceLoaderStartupModule. * Add option to suppress the cache key report in ResourceLoader::filter. We usually only call the minifier once on an entire request reponse (because it's all concatenated javascript or embedded javascript in various different closures, still valid as one large script) and only add a little bottom line for the cache key. When embedding the skip function we have to run the minifier on them separately as they're output as strings (not actual functions). These strings are typically quite small and blowing up the response with loads of cache keys is not desirable in production. * Add method to clear the static cache of ResourceLoader::inDebugMode. Global static state is evil but, as long as we have it, we at least need to clear it after switching contexts in the test suite. Also: * Remove obsolete setting of 'debug=true' in the FauxRequest in ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug in the setUp() method. Bug: 66390 Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
2014-04-30 21:06:51 +00:00
*/
public static function clearCache() {
self::$debugMode = null;
}
/**
* Build a load.php URL
*
* @since 1.24
* @param string $source Name of the ResourceLoader source
* @param ResourceLoaderContext $context
* @param array $extraQuery
* @return string URL to load.php. May be protocol-relative if $wgLoadScript is, too.
*/
public function createLoaderURL( $source, ResourceLoaderContext $context,
$extraQuery = []
) {
$query = self::createLoaderQuery( $context, $extraQuery );
$script = $this->getLoadScript( $source );
return wfAppendQuery( $script, $query );
}
/**
* Helper for createLoaderURL()
*
* @since 1.24
* @see makeLoaderQuery
* @param ResourceLoaderContext $context
* @param array $extraQuery
* @return array
*/
protected static function createLoaderQuery( ResourceLoaderContext $context, $extraQuery = [] ) {
return self::makeLoaderQuery(
$context->getModules(),
$context->getLanguage(),
$context->getSkin(),
$context->getUser(),
$context->getVersion(),
$context->getDebug(),
$context->getOnly(),
$context->getRequest()->getBool( 'printable' ),
$context->getRequest()->getBool( 'handheld' ),
$extraQuery
);
}
/**
* Build a query array (array representation of query string) for load.php. Helper
* function for createLoaderURL().
*
* @param array $modules
* @param string $lang
* @param string $skin
* @param string|null $user
* @param string|null $version
* @param bool $debug
* @param string|null $only
* @param bool $printable
* @param bool $handheld
* @param array $extraQuery
*
* @return array
*/
public static function makeLoaderQuery( $modules, $lang, $skin, $user = null,
$version = null, $debug = false, $only = null, $printable = false,
$handheld = false, $extraQuery = []
) {
$query = [
'modules' => self::makePackedModulesString( $modules ),
'lang' => $lang,
'skin' => $skin,
'debug' => $debug ? 'true' : 'false',
];
if ( $user !== null ) {
$query['user'] = $user;
}
if ( $version !== null ) {
$query['version'] = $version;
}
if ( $only !== null ) {
$query['only'] = $only;
}
if ( $printable ) {
$query['printable'] = 1;
}
if ( $handheld ) {
$query['handheld'] = 1;
}
$query += $extraQuery;
// Make queries uniform in order
ksort( $query );
return $query;
}
/**
* Check a module name for validity.
*
* Module names may not contain pipes (|), commas (,) or exclamation marks (!) and can be
* at most 255 bytes.
*
* @param string $moduleName Module name to check
* @return bool Whether $moduleName is a valid module name
*/
public static function isValidModuleName( $moduleName ) {
return strcspn( $moduleName, '!,|', 0, 255 ) === strlen( $moduleName );
}
/**
* Returns LESS compiler set up for use with MediaWiki
*
* @since 1.27
* @param array $vars Associative array of variables that should be used
* for compilation. Since 1.32, this method no longer automatically includes
* global LESS vars from ResourceLoader::getLessVars (T191937).
* @throws MWException
* @return Less_Parser
*/
public function getLessCompiler( $vars = [] ) {
global $IP;
// When called from the installer, it is possible that a required PHP extension
// is missing (at least for now; see T49564). If this is the case, throw an
// exception (caught by the installer) to prevent a fatal error later on.
if ( !class_exists( 'Less_Parser' ) ) {
throw new MWException( 'MediaWiki requires the less.php parser' );
}
$parser = new Less_Parser;
$parser->ModifyVars( $vars );
$parser->SetImportDirs( [
"$IP/resources/src/mediawiki.less/" => '',
] );
$parser->SetOption( 'relativeUrls', false );
return $parser;
}
/**
* Get global LESS variables.
*
* @since 1.27
* @deprecated since 1.32 Use ResourceLoderModule::getLessVars() instead.
* @return array Map of variable names to string CSS values.
*/
public function getLessVars() {
return [];
}
}