Package files are files that are part of a module, but are not
immediately executed when the module executes. Instead, they are
lazy-excecuted when require() is called on them. Package files can be
scripts (JS) or data (JSON), and can be real files on the file system,
or virtual files generated by a callback.
Using virtual data files, server-side data and config variables can be
bundled with a module. Support for file-based require() allows us to
import npm modules into ResourceLoader more easily.
The require function passed to each script execution context, which was
previously a reference to the global mw.loader.require() function, is
changed to one that is scoped to the module and the file being executed.
This is needed to support relative paths: require( '../foo.js' ) can
mean a different file depending on the path of the calling file.
The results of require()ing each file (i.e. the value of module.exports
after executing it) are stored, and calling require() on the same file a
second time won't execute it again, but will return the stored value.
Miscellaneous changes:
- Add XmlJsCode::encodeObject(), which combines an associative array of
XmlJsCode objects into one larger XmlJsCode object. This is needed for
encoding the packageFiles parameter in mw.loader.implement() calls.
Bug: T133462
Change-Id: I78cc86e626de0720397718cd2bed8ed279579112
The use of global variables was deprecated in favour of
ResourceLoaderModule::getLessVars() on a per-module basis.
Also moved testLessFileCompilation case to the appropiate file as it
covers ResourceLoaderFileModule.php, not ResourceLoader.php.
Bug: T140804
Depends-On: Ib1b2808df2384473bfac47f53a5d25d7c9bbca2b
Depends-On: I96047f69d01c4736306df2719267e6347daf556f
Change-Id: If708087c85c80355c7e78f1768529b5f2e16ed07
Add @covers for various helper methods used by public methods, where the helper
methods actually contain most of the logic being tested in FileModuleTest.
I've changed these methods from protected to private (confirmed no usage)
to further pin down that their contract doesn't matter beyond making the
public methods work.
Change-Id: I2aef0d322b38bc3595e7d2c2339112b16fc66b8d
This already worked as expected for any module that uses the new
enableModuleContentVersion model, but for the majority of file modules
this is not yet the case for performance reasons. As such, make
sure lessVars are included in our manual tracking.
Include it conditionally to avoid changing the array for other modules,
which would needlessly invalidate their cache.
Bug: T171809
Change-Id: Ib250068e0ecfc29a09ca33c23bef901ee0482bf2
Some used a string value, others an array with 'message' property.
Standardise on the string value, which seems more intuitive.
Change-Id: I5caead7b7017d2bad660db02fb45a54a26bf3728
This makes it easier to add other options in the future,
such as setting 'modules' in the context to something else.
Change-Id: I53c25fa7ad705cc34e44f95e4f87eb53612d800e
ResourceLoader modules can now carry a 'deprecated' option which can
be a boolean or an object with message key. This message or a default
deprecation message will be show whenever that module is used in production.
Note: This will not work in debug mode for ResourceLoaderFile modules
and this is deemed acceptable for the time being. We can revisit later.
Bug: T137772
Change-Id: Ib9ebd2d39a59fd41d8537e06884699f77b03580c
We read files and concatenate their contents. Files may start with a BOM character.
BOM characters are only allowed at the beginning of a file, not half way.
Stripping it should be safe, since we already assume that everything is UTF-8.
Change-Id: I14ad698a684e78976e873e9ae2c367475550a063
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
These tests all involve database access in some way,
and thus need @group Database tags.
These failed when setting a bogus database password
and then running the tests.
Change-Id: I7f113a79ac44d09d88ec607f76b8ec22bc1ebcf1
* Move testMixedCssAnnotations to ResourceLoaderFileModuleTest.
* Re-order data providers before test methods.
* Add relevant @covers annotations in resourceloader/ tests.
* Make test helper function private.
* Add a few @covers for methods called from OutputPage::makeResourceLoaderLink
(only one level deep, we should have separate unit tests for
the more internal helpers).
Change-Id: I2cc1757126214ed28059d4566ca813a86bcd95a7
Mock calls to ResourceLoaderContext::getDirection(), which creates
Language objects to get the directionality of a language.
Change-Id: Ibe6da3013e658aa7cf596c1da2f8ca1314b7cdd3
A base ResourceLoaderModule::getTemplates() exists for subclasses
to override. An implementation is provided for ResourceLoaderFileModule.
For file modules, templates can be specified in the following manner:
'example' => array(
'templates' => array(
'bar' => 'templates/foo.html',
),
'scripts' => 'example.js',
),
The delivery system is template language agnostic, and currently
only supports "compiling" plain HTML templates.
This also adds template support to the following modules as a POC:
* mediawiki.feedback
* mediawiki.action.view.postEdit
* mediawiki.special.upload
Works with $wgResourceLoaderStorageEnabled
Change-Id: Ia0c5c8ec960aa6dff12c9626cee41ae9a3286b76
Not ready for merging, and Roan says that the +2 was
most likely accidental and meant to be a -1.
This reverts commit d146934f94.
Change-Id: I3926c9ae9e3c8026fceb3aeedd3b1f1d9b91667b
Preparation work for templating in core.
RL should allow us to ship HTML / template markup from server to client.
Use in Special:Upload and mediawiki.feedback as a proof of concept.
Separation of concerns etc...
See Also:
Ia63d6b6868f23a773e4a41daa0036d4bf2cd6724
Change-Id: I6ff38c12897e3164969a1090449e626001926c3b
