=== Why
* More speed
In debug mode, the server should regenerate the startup manifest
on each page view to ensure immediate effect of changes. But,
this also means more version recomputation work on the server.
For most modules, this was already quite fast on repeat views
because of OS-level file caches, and our file-hash caches and
LESS compile caches in php-apcu from ResourceLoader.
But, this makes it even faster.
* Better integration with browser devtools.
Breakpoints stay more consistently across browsers when the
URL stays the same even after you have changed the file and
reloaded the page. For static files, I believe most browsers ignore
query parameters. But for package files that come from load.php,
this was harder for browsers to guess correctly which old script URL
is logically replaced by a different one on the next page view.
=== How
Change Module::getVersionHash to return empty strings in debug mode.
I considered approaching this from StartupModule::getModuleRegistrations
instead to make the change apply only to the client-side manifest.
I decided against this because we have other calls to getVersionHash
on the server-side (such as for E-Tag calculation, and formatting
cross-wiki URLs) which would then not match the version queries that
mw.loader formats in debug mode.
Also, those calls would still be incurring some the avoidable costs.
=== Notes
* The two test cases for verifying the graceful fallback in production
if version hash computations throw an exception, were moved to a
non-debug test case as no longer happen now during the debug
(unminified) test cases.
* Avoid "PHP Notice: Undefined offset 0" in testMakeModuleResponseStartupError
by adding a fallback to empty string so that if the test fails,
it fails in a more useful way instead of aborting with this error
before the assertion happens. (Since PHPUnit generally stops on the
first error.)
* In practice, there are still "version" query parameters and E-Tag
headers in debug mode. These are not module versions, but URL
"combined versions" crafted by getCombinedVersion() in JS and PHP.
These return the constant "ztntf" in debug mode, which is the hash
of an empty string. We could alter these methods to special-case
when all inputs are and join to a still-empty string, or maybe we
just leave them be. I've done the latter for now.
Bug: T235672
Bug: T85805
Change-Id: I0e63eef4f85b13089a0aa3806a5b6f821d527a92
In debug mode of a packageFiles module there is now only one newline
between the JavaScript code and the closing }.
This change reuses ResourceLoader::ensureNewline and make this to a
interal public static function.
Change-Id: I89519896e3dc56d966c4a63102904686bff6fac9
The native "foreign module source" feature, as used by the GlobalCssJs
extension, did not work correctly in debug mode as the urls returned
by the remote wiki were formatted as "/w/load.php...", which would
be interpreted by the browser relative to the host document, instead
of relative to the parent script.
For example:
1. Page view on en.wikipedia.org.
2. Script call to
meta.wikimedia.org/w/load.php?debug=true&modules=ext.globalCssJs.user&user
This URL is formatted by getScriptURLsForDebug on en.wikipedia.org,
when building the article HTML. It knows the modules is on Meta, and
formats it as such.
So far so good.
3. meta.wikimedia.org responds with an array of urls for sub resources.
That array contained URLs like "/w/load.php...only=scripts".
These were formatted by getScriptURLsForDebug running on Meta,
no longer with a reason to make it a Meta-Wiki URL as it isn't
perceived as cross-wiki. It is indistinguishable from debugging
a Meta-Wiki page view from its perspective.
This patch affects scenario 3 by always expanding it relative to the
current-request's wgServer. We still only do this in debug mode. There
is not yet a need to do this in non-debug mode, and if there was we'd
likely want to find a way to avoid it in the common case to keep
embedded URLs short.
The ResourceLoader::expandUrl() method is similar to the one in
Wikimedia\Minify\CSSMin.
Test Plan:
* view-source:http://mw.localhost:8080/w/load.php?debug=1&modules=site
For Module base class.
Before, the array entries were relative. After, they are full.
* view-source:http://mw.localhost:8080/w/load.php?debug=1&modules=jquery
For FileModule.
Before, the array entries were relative. After, they are full.
* view-source:http://mw.localhost:8080/wiki/Main_Page?debug=true
Unchanged.
* view-source:http://mw.localhost:8080/wiki/Main_Page
Unchanged.
Bug: T255367
Change-Id: I83919744b2677c7fb52b84089ecc60b89957d32a
This was covered in small ways already, but we did not have coverage
of the registration logic where the skin attribute is injected into
the module info, and this then correctly getting read and composed
into a single stylesheet.
This is prep for an optimisation in a subsequent commit.
Change-Id: Ia6fff44a345bb4f5811956638cada8b3e6b6ea71
There is a fallback in Module->getConfig() to the global services
container. This is not meant to be used in practice, but there were
two places where this was missing: WebInstallerOutput, and various
PHPUnit tests.
* Add missing setConfig() to WebInstallerOutput.
* Add missing setConfig() to various tests. Most tests were already
doing this correctly and using the standard mock from
ResourceLoaderTestCase. Upon switching the last few tests as well,
I uncovered various errors due to the mock missing some settings
that the tested code uses, so these have been added now to
ResourceLoaderTestCase.
Bug: T277728
Change-Id: I44f16ec4e00423fb6f641e58fffc1d40e4571f01
Modules that set "es6": true in their module definition will error when
a non-ES6 client tries to load them.
To detect ES6 support, this looks for native Promise support,
RegExp.prototype.flags, and non-BMP characters in variable names. All
browsers that lack full ES6 support fail at least one of those checks.
To flag modules as requiring ES6, this adds a ! to the end of their
version string. This takes up much less space than adding another
register() parameter (which would have to be at the end). It's hacky,
but we expect this feature to be relatively temporary, until we require
ES6 for running any JS at all (probably in about a year).
For distinguishing different types of errors thrown from
sortDependencies(), use e.name. We can't subclass Error properly because
that requires ES6.
Bug: T272104
Change-Id: I45670c910ff12eb422ae54c9fcf372e45c7b2bf1
* Move the method to SkinFactory and replaces usages.
* Inject $wgSkipSkins into the SkinFactory
Bug: T257993
Change-Id: I9869cf34c5e87cbad963f48db0649b3b7a252a4a
This fixes an issue with HTML tags inside the <script> tag.
Remex also doesn't throw errors on attributes like @click, although it
does mangle them when producing DOM. To work around this, don't use DOM
serialization for the template HTML, but parse everything again using a
Remex parse+serialize pipeline that extracts the template and
(optionally) removes comments and strips whitespace.
One important effect of this change is that we'll have to forbid
self-closing tags in Vue templates, because Remex doesn't handle those
correctly (or rather, handles them *too* correctly). But on the up side,
we can now allow shorthands for v-bind/v-on/v-slot again.
Bug: T253334
Bug: T255587
Depends-On: I2253a2317187fe0d781ba5bfefab95e0f97d0a80
Change-Id: Id9a9728b7163601cc60bc587be07b70977d41970
The old way of providing a callable to SkinFactory::register is
still supported. Those callables expected the skin name as their
first argument. Coincidentally, so does the constructor of Skin.
Some skins might not define any constructor parameters at all,
which is acceptable to PHP, as it will just discard the argument.
The registration using $wgValidSkinNames has not been changed,
and skins that want to define services to be injected will still
need to manually register their skin to the skin factory.
CodeSearch did not indicate any extensions or skins manually
constructing a SkinFactory in tests, but for posterity, the old
way of creating a SkinFactory for testing can be replaced with
new SkinFactory( new ObjectFactory(
$this->createMock( ContainerInterface::class )
) );
Note that the constructor for SkinFactory for internal use only,
in accordance with the Stable interface policy.
You should use MediaWikiServices::getInstance()->getSkinFactory
instead.
Bug: T244466
Change-Id: I8ba9d869bddd9b6124e47697b789d752c0620b02
Allows .vue files to be used in package modules as if they were .js
files: they can be added to the 'packageFiles' array in module
definitions, and require()d from JS files.
In the load.php output, each .vue file is transformed to a function that
contains the JS from the <script> tag, then a line that sets
module.exports.template to the contents of the <template> tag (encoded
as a string). The contents of the <style> tag are added to the module's
styles.
Internally, the type of a .vue file is inferred as 'script-vue', and the
file is parsed with VueComponentParser, which extracts the three parts.
After the transformation, the file's type is set to 'script+style', and
files of this type contribute to both getScript() and getStyles().
This change also adds caching to getPackageFiles(), because it now needs
to be called twice (in getScript() and getStyles()).
Change-Id: Ic0a5c771901450a518eb7d24456053956507e1ed
If a callback for a virtual file returns a ResourceLoaderFilePath
object, we will load that file from disk and use it in that file's
place. This enables us to port modules that use languageScripts or
skinScripts, by writing a virtual file callback that returns a
ResourceLoaderFilePath pointing to different files depending on the
language/skin.
This also makes the base path parameters to the ResourceLoaderFilePath
constructor optional. Callbacks would construct a ResourceLoaderFilePath
with only one parameter (the file path, but no base paths), and the
RL infrastructure ignores the object's base paths anyway, in favor of
the module's base paths.
Bug: T239371
Change-Id: I8e24f667b4e1944c20f3354a9f7382d5c486055e
Avoid use of wfTimestamp and wfDebugLog global functions.
Also simplify some of the error messages around processing of
'packageFiles' definitions and throw generic LogicException
instead of MWException.
Change-Id: I55ce1f107f53dfdfe673cbe4411b0a7c4e24b2ea
Also widen up @covers for the integration tests to let the
coverage percentage increase from those.
Change-Id: Ib9d42b124774b4c968c52f34e587059c63e8ffc2
Currently packageFiles callbacks take 2 parameters, $context and $config.
This patch allows for specifying an extra parameter in the packageFiles
definition that will be passed to the callback. Example:
'callback' => function ( $context, $config, $extra ) { ... },
'callbackParam => [ 'this is val 1', 'this is val 2' ],
The callback will be called with the usual $context and $config
parameters, and the extra array is passed as third parameter.
Bug: T233634
Change-Id: Ie11874665f4f9a557d4e394dcab3a972887e8126
This change follows I39cc2a735d9625c87bf4ede6f5fb0ec441d47dcc.
docs/extension.schema.v1.json
docs/extension.schema.v2.json
includes/registration/ExtensionProcessor.php
* The new extension attribute 'OOUIThemePaths' can be used to define
custom OOUI themes. See I9187a63e509b601b8558ea82850fa828e5c8cc0a
for an example usage.
includes/resourceloader/ResourceLoaderOOUIModule.php
* Add support for 'OOUIThemePaths'.
* Defining 'images' is now optional. I figure custom themes are
unlikely to have or need them.
* Use ResourceLoaderFilePath objects to allow skin-/extension-defined
OOUI module files to use skin/extension's base paths.
This was previously used to support $wgResourceModuleSkinStyles,
but only for 'skinStyles' - now ResourceLoaderFileModule needs
to also handle it for 'skinScripts', and ResourceLoaderImageModule
for 'images').
includes/resourceloader/ResourceLoaderFilePath.php
* Add getters for local/remote base paths, for when we need to
construct a new ResourceLoaderFilePath based on existing one.
includes/resourceloader/ResourceLoaderFileModule.php
includes/resourceloader/ResourceLoaderImageModule.php
includes/resourceloader/ResourceLoaderOOUIImageModule.php
* Add or improve handling of ResourceLoaderFilePaths:
* Replace `(array)` casts with explicit array wrapping, to avoid
casting objects into associative arrays.
* Use getLocalPath() instead of string concatenation.
tests/phpunit/includes/resourceloader/ResourceLoaderFileModuleTest.php
tests/phpunit/includes/resourceloader/ResourceLoaderImageModuleTest.php
* Some basic checks for the above.
Bug: T100896
Change-Id: I74362f0fc215b26f1f104ce7bdbbac1e106736ad
Also, for the unit test, disable the two methods we use there
that can get called. The unintended side-effects of these two
methods was the only reason it used `@group Database`.
Removing that makes the test a bit faster as well.
Enforce this via MediaWikiServices for this suite to avoid an
untracked dependency slipping back in in the future.
Bug: T225730
Change-Id: I6c54466e9517d9899bc39f8f9bb946369c0a526d
The use cases we've seen for using computed (or virtual) package files,
involve expensive computations to expand and transform data for the client
that we don't want to evaluate in full just to compute the module's version
hash (e.g. in the StartupModule, where we need to do this for 1000s of
modules).
For such cases, the module can specify a 'versionCallback' of which the
return value will be used to seed the module's version hash. The default
remains the same as before, which is to use the full content to seed the
version hash (via getDefinitionSummary).
Bug: T223260
Change-Id: I76f573239e6bd429287e7adb33a92ffd5e260c20
The module definition format for 'packageFiles', as initially designed,
mixes sequential and associative arrays. This works in PHP, but not in
JSON. To make the format JSON compatible, use a 'name' key instead of
using the key in the main array.
Leave backwards compatibility in place so that extensions using the old
format can be migrated. This will be removed in the next commit.
Before:
'packageFiles' => [
'script.js',
'script2.js',
'config.json' => [ 'config' => [ 'Foo', 'Bar' ] ],
'data.json' => [ 'callback' => function () { ... } ],
],
After:
'packageFiles' => [
'script.js',
'script2.js',
[ 'name' => 'config.json', 'config' => [ 'Foo', 'Bar' ] ],
[ 'name' => 'data.json', 'callback' => function () { ... } ],
],
This can then be written in extension.json as:
"packageFiles": [
"script.js",
"script2.js",
[ "name": "config.json", "config": [ "Foo", "Bar" ] ],
[ "name": "data.json", "callback: [ "MyExtHooks", "getData" ] ]
]
Change-Id: Ic566a1cd7efd075c380bc50ba0cc2c329a2041d7
Package files are files that are part of a module, but are not
immediately executed when the module executes. Instead, they are
lazy-excecuted when require() is called on them. Package files can be
scripts (JS) or data (JSON), and can be real files on the file system,
or virtual files generated by a callback.
Using virtual data files, server-side data and config variables can be
bundled with a module. Support for file-based require() allows us to
import npm modules into ResourceLoader more easily.
The require function passed to each script execution context, which was
previously a reference to the global mw.loader.require() function, is
changed to one that is scoped to the module and the file being executed.
This is needed to support relative paths: require( '../foo.js' ) can
mean a different file depending on the path of the calling file.
The results of require()ing each file (i.e. the value of module.exports
after executing it) are stored, and calling require() on the same file a
second time won't execute it again, but will return the stored value.
Miscellaneous changes:
- Add XmlJsCode::encodeObject(), which combines an associative array of
XmlJsCode objects into one larger XmlJsCode object. This is needed for
encoding the packageFiles parameter in mw.loader.implement() calls.
Bug: T133462
Change-Id: I78cc86e626de0720397718cd2bed8ed279579112
The use of global variables was deprecated in favour of
ResourceLoaderModule::getLessVars() on a per-module basis.
Also moved testLessFileCompilation case to the appropiate file as it
covers ResourceLoaderFileModule.php, not ResourceLoader.php.
Bug: T140804
Depends-On: Ib1b2808df2384473bfac47f53a5d25d7c9bbca2b
Depends-On: I96047f69d01c4736306df2719267e6347daf556f
Change-Id: If708087c85c80355c7e78f1768529b5f2e16ed07
Add @covers for various helper methods used by public methods, where the helper
methods actually contain most of the logic being tested in FileModuleTest.
I've changed these methods from protected to private (confirmed no usage)
to further pin down that their contract doesn't matter beyond making the
public methods work.
Change-Id: I2aef0d322b38bc3595e7d2c2339112b16fc66b8d
This already worked as expected for any module that uses the new
enableModuleContentVersion model, but for the majority of file modules
this is not yet the case for performance reasons. As such, make
sure lessVars are included in our manual tracking.
Include it conditionally to avoid changing the array for other modules,
which would needlessly invalidate their cache.
Bug: T171809
Change-Id: Ib250068e0ecfc29a09ca33c23bef901ee0482bf2
Some used a string value, others an array with 'message' property.
Standardise on the string value, which seems more intuitive.
Change-Id: I5caead7b7017d2bad660db02fb45a54a26bf3728
This makes it easier to add other options in the future,
such as setting 'modules' in the context to something else.
Change-Id: I53c25fa7ad705cc34e44f95e4f87eb53612d800e
ResourceLoader modules can now carry a 'deprecated' option which can
be a boolean or an object with message key. This message or a default
deprecation message will be show whenever that module is used in production.
Note: This will not work in debug mode for ResourceLoaderFile modules
and this is deemed acceptable for the time being. We can revisit later.
Bug: T137772
Change-Id: Ib9ebd2d39a59fd41d8537e06884699f77b03580c
We read files and concatenate their contents. Files may start with a BOM character.
BOM characters are only allowed at the beginning of a file, not half way.
Stripping it should be safe, since we already assume that everything is UTF-8.
Change-Id: I14ad698a684e78976e873e9ae2c367475550a063
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
These tests all involve database access in some way,
and thus need @group Database tags.
These failed when setting a bogus database password
and then running the tests.
Change-Id: I7f113a79ac44d09d88ec607f76b8ec22bc1ebcf1
* Move testMixedCssAnnotations to ResourceLoaderFileModuleTest.
* Re-order data providers before test methods.
* Add relevant @covers annotations in resourceloader/ tests.
* Make test helper function private.
* Add a few @covers for methods called from OutputPage::makeResourceLoaderLink
(only one level deep, we should have separate unit tests for
the more internal helpers).
Change-Id: I2cc1757126214ed28059d4566ca813a86bcd95a7
Mock calls to ResourceLoaderContext::getDirection(), which creates
Language objects to get the directionality of a language.
Change-Id: Ibe6da3013e658aa7cf596c1da2f8ca1314b7cdd3
A base ResourceLoaderModule::getTemplates() exists for subclasses
to override. An implementation is provided for ResourceLoaderFileModule.
For file modules, templates can be specified in the following manner:
'example' => array(
'templates' => array(
'bar' => 'templates/foo.html',
),
'scripts' => 'example.js',
),
The delivery system is template language agnostic, and currently
only supports "compiling" plain HTML templates.
This also adds template support to the following modules as a POC:
* mediawiki.feedback
* mediawiki.action.view.postEdit
* mediawiki.special.upload
Works with $wgResourceLoaderStorageEnabled
Change-Id: Ia0c5c8ec960aa6dff12c9626cee41ae9a3286b76
