If a type=general module is enqueued, don't try to load it as a
stylesheet.
* Per a464d1d41d, state tracking is already disabled for
these loads (as otherwise we wrongly claim state=ready, when in
fact only the styles and not the scripts were loaded).
* The warning was added in a464d1d41d.
* Default install (tested in Vagrant), Wikimedia Beta cluster, and
Wikimedia production have seen zero violations of this warning
in the past 7 days.
Raise severity to ERROR and add the 'continue' statement so that
these are now not loaded at all.
Bug: T92459
Change-Id: I211d56ac2df479ebf5b98667c613ecf81489539b
The example for a mixed user module is outdated. Current user modules
all separate styles and scripts.
To make the effective change by the next commit easier to review,
update these tests first seperately.
Change-Id: I76a8a96eed8a8c39863c535362b1d0144ef2f866
This fixes two bugs:
* 1) When two modules are requested, and the first one ends with ";"
inside a comment, the second module might become part of
that comment.
* 2) A request with script=only where the requested module content
ends in a statement without ";", but has a comment after it
that does ends with a semicolon, then in debug=false, mw.loader.state()
would be appended directly after the semicolon-less statement because
the check is performed before minification.
Previously:
script> foo()
script> // bar();
states> mw.loader.state( {} );
Became (minified separately):
script> foo()
states> mw.loader.state({});
Became (concatenated)
> foo()mw.loader.state();
Which is invalid code.
Both of these are now fixed.
Bug: T162719
Change-Id: Ic8114c46ce232f5869400eaa40d3027003550533
The used phpcs has a bug, so the version 0.9.0 could not be enforced at the moment.
Will be fixed in next version, see T167168
Changed:
- Remove duplicate newline at end of file
- Add space between function and ( for closures
- and -> &&, or -> ||
Change-Id: I4172fb08861729bccd55aecbd07e029e2638d311
Using names that match real OOjs UI icons is confusing when trying to
find usages of the real icons and when debugging the tests.
Bug: T166730
Change-Id: Ibb97c0347476efc95f1a50c97822d7dab19737f2
This should work the same way as registering API modules via a factory callback.
Point in case: Ifb8611473a971 could avoid global state using this mechanism.
Change-Id: Ifbf29006141ce2a2dff42efa352f406502a06bc6
Replaces \TestingAccessWrapper (defined in core) with
\Wikimedia\TestingAccessWrapper (defined in the composer package
wikimedia/testing-access-wrapper).
See https://gerrit.wikimedia.org/r/#/q/topic:librarize-testing-access-wrapper
for downstream patches.
The core version of the class is kept around for a while to avoid
circular dependency problems.
Bug: T163434
Change-Id: I52cc257e593da3d6c3b01a909e554a950225aec8
* Missing cases for StartupModule::getModuleRegistrations
(now 100% covered)
- Raw modules are omitted from the manifest.
E.g. The base modules ('jquery', 'mediawiki') are raw modules
that we don't register client side (they can't load themselves).
- Exceptions from getVersionHash() are caught.
- Oversized versions are re-hashed.
* Missing cases for ResourceLoader::makeLoaderRegisterScript.
(now 100% covered)
* Missing cases for ResourceLoader::getModule.
(now 100% covered)
Change-Id: If9717a48195fc6ae776da5d0e86f323d7f60426d
preloadTitleInfo:
* Add missing case for empty $moduleNames.
* Add missing case for invalid page names.
getContent:
* Add missing case for bad title
* Add missing case for dead redirect.
* Add missing case for no content found.
Change-Id: I44dde13cb0db19d91c4ff15a5abefd17353cad90
Some used a string value, others an array with 'message' property.
Standardise on the string value, which seems more intuitive.
Change-Id: I5caead7b7017d2bad660db02fb45a54a26bf3728
When the SVG is too big to be embeded it is now included via URL.
Previously it would produce an empty/broken 'url()' value.
Bug: T160532
Change-Id: I158781f9430cfa35737397ac7537a471634c4480
> 1) ResourceLoaderTest::testMakeModuleResponseError
> Failed asserting that '[e08c982d974548127cb5d7ce] Fatal exception of type Exception'
> matches PCRE pattern "/Ferry not found/".
> .../ResourceLoaderTest.php:519
This happened on Travis CI, because ResourceLoader::formatException() behaves
differently based on $wgShowExceptionDetails. Which is enabled in Vagrant
and Jenkins, but disabled by default (and thus in Travis CI builds).
Bug: T75176
Change-Id: If15dd03213703b7b6ff899cad5e5569e2515b378
Previously, style modules were only in a predictable order for production mode.
In debug mode, the order was determined by order in which modules were added
to queue at run time. This made it sometimes hard to debug, especially when
dealing with gadgets that apply in a different order among each other.
Change-Id: I4bff0c91d127e4ad8015cd8c1775220fe460cbc3
When getScript (or some other method used in a module response)
throws an error, only that module fails (by outputting mw.loader.state
instead of mw.loader.implement). Other modules will work.
This has always been the case and is working fine. For example,
"load.php?modules=foo|bar", where 'foo' throws, will return:
```js
/* exception message: .. */
mw.loader.implement('bar', ..)
mw.loader.state('foo', 'error')
```
The problem, however, is that during the generation of the startup
module, we iterate over all other modules. In 2011, the
getVersionHash method (then: getModifiedTime) was fairly simple
and unlikely to throw errors.
Nowadays, some modules use enableModuleContentVersion which will
involve the same code path as for regular module responses.
The try/catch in ResourceLoader::makeModuleResponse() suffices
for the case of loading modules other than startup. But when
loading the startup module, and an exception happens in getVersionHash,
then the entire startup response is replaced with an exception comment.
Example case:
* A file not existing for a FileModule subclass that uses
enableModuleContentVersion.
* A database error from a data module, like CiteDataModule or
CNChoiceData.
Changes:
* Ensure E-Tag is still useful while an error happens in production
because we respond with 200 OK and one error isn't the same as
another.
Fixed by try/catch in getCombinedVersion.
* Ensure start manifest isn't disrupted by one broken module.
Fixed by try/catch in StartupModule::getModuleRegistrations().
Tests:
* testMakeModuleResponseError: The case that already worked fined.
* testMakeModuleResponseStartupError: The case fixed in this commit.
* testGetCombinedVersion: The case fixed in this commit for E-Tag.
Bug: T152266
Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
This makes it easier to add other options in the future,
such as setting 'modules' in the context to something else.
Change-Id: I53c25fa7ad705cc34e44f95e4f87eb53612d800e
It's not explicitly supported anywhere, but I don't see a point in explicitly
disallowing it. Add unit tests to verify that this works.
Bug: T28804
Change-Id: I876ac43885bb27da54ef6e59b6416868ff636b84
* The styles queue has always been top-only
(except for a few months in 2015).
* The top queue loads asynchronous since mid-2015. (T107399)
And LocalStorage eval, previously the last remaining non-async part
of module loading, is also async as of October 2016. (T142129)
* This change merges the bottom 'mw.loader.load()' queue with the top queue.
It also moves any other snippets potentially in the bottom queue still:
- embed: I couldn't find any private modules with position=bottom
(doesn't make sense due to their blocking nature). If any do exist,
(third-party extensions?), they'll now be embedded in the <head>.
- scripts: Any legacy 'only=scripts' requests will now initiate
from the <head>.
Bug: T109837
Change-Id: I6c21e3e47c23df33a04c42ce94bd4c1964599c7f
This is one of the top three DB queries showing up in xenon
reverse flamegraph profiling.
It works via a per-wiki check key that is bumped whenever
someone changes a .js or .css page on that wiki.
Change-Id: I73f419558864ba3403b4601a098f6aaf84a3e7c1
Follows-up 6fa1e56. This is already fixed for http caches by
shortening the Cache-Control max-age in case of a version mismatch.
However the client still cached it blindly in mw.loader.store.
Resolve this by communicating to the client what version of the module
was exported. The client can then compare this version to the version
it originally requested and decide not to cache it.
Adopt the module key format (name@version) from mw.loader.store
in mw.loader.implement() as well.
Bug: T117587
Change-Id: I1a7c44d0222893afefac20bef507bdd1a1a87ecd
* Keep this out of makeLoaderImplementScript() to keep it more generic
and to simplify future refactoring.
* Remove now-broken test case that asserted that the output varies
by global debug mode.
* Make the test responsible for wrapping in XmlJsCode. Previously
this magically happened because the module name was "user" in the
last test case.
* Make makeLoaderImplementScript protected. It's not used anywhere
outside ResourceLoader and we should keep it that way.
Test plan:
* Verify output unchanged:
- load.php?modules=user&only=scripts&user=Admin (raw code)
- load.php?modules=user&user=Admin (implement with unwrapped string)
- load.php?modules=jquery.client (implement with closure)
Change-Id: I527d01926fb6e4ce68c931695d830cdb9ceb608c
Follows-up 6f8dc27ca2 and dbd11e04aa. You'd expect a bug in preloading
to fallback to fetching it on-demand but due to a title format mismatch
the preload method did use the same title format for the cache key, but
not the same title format for discovering the results. As such, it set
the right cache key to an empty array.
* Make relevant methods testable and mockable.
* Add regression test.
* Change call to array_interect_key to use the same format as
fetchTitleInfo(): Normalised title keys from Title::getPrefixedText.
Previously, the intersect used the declared titles from getPages() which
are not localised - causing an empty array to be returned from the
intersect on wikis where the namespace name is localised.
Bug: T145673
Change-Id: Ibe788157724d73c727b9e2127b6828db32ca9420
While rev_sha1 was preferred to rev_id (page_latest) to allow
client-cache to be re-used in case of a bad change and revert,
this is no longer possible since we recently added support for
explicit purging by considering page_touched.
As such, use of rev_sha1 is no longer useful.
Change-Id: Iddb65305ca4655098fdea9fcf736fd4046035dd7
* Fix up one last use of global config vars in this class.
Other places in this class already used $rl->getConfig().
This way we don't inherit all of MediaWikiTestCase.
* Add unit tests covering all of ResourceLoaderContext
except expandModuleNames and getImageObj (tested in better
places already with the right @covers).
* Increase coverage for expandModuleNames(), add missing case
of when modules are not in alphabetical order.
Change-Id: Id19b084d37a6c3a77b36e03509adffb6b156fee1
* Fix signature of makeLoaderSourcesScript() to match
the change in behaviour since e103ba265.
* Consistently order providers before the test.
* Simplify testRegisterValid() and remove needless @depends.
* Remove unused private method stripNoflip().
Coverage:
* Expand test coverage for register().
* Add tests for getModuleNames().
* Add tests for getModule().
* Expand test coverage for addSource().
(case of invalid array)
* Expand test coverage for makeLoaderImplementScript().
(case of unwrapped user script, and case of invalid scripts)
* Add tests for makeLoaderSourcesScript().
Change-Id: Ibca3e486fcd3664f171f135327a0f340ee6da9ee
HTML formatting of the queue was distributed over several OutputPage methods.
Each method demanding a snippet of HTML by calling makeResourceLoaderLink()
with a limited amount of information. As such, makeResourceLoaderLink() was
unable to provide the client with the proper state information.
Centralising it also allows it to better reduce duplication in HTML output
and maintain a more accurate state.
Problems fixed by centralising:
1. The 'user' module is special (due to per-user 'version' and 'user' params).
It is manually requested via script-src. To avoid a separate (and wrong)
request from something that requires it, we set state=loading directly.
However, because the module is in the bottom, the old HTML formatter could
only put state=loading in the bottom also. This sometimes caused a wrong
request to be fired for modules=user if something in the top queue
triggered a requirement for it.
2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose
of allowing dependencies on style modules without risking duplicate loading
on pages where the styles are loaded already. This didn't work, because the
state information about page-style modules is output near the stylesheet,
which is after the script tag with mw.loader.load(). That runs first, and
mw.loader would still make a duplicate request before it learns the state.
Changes:
* Document reasons for style/script tag order in getHeadHtml (per 09537e83).
* Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed
before since a duplicate check in makeResourceLoaderLink() verified the
origin a second time.
* Declare explicit position 'top' on 'user.options' and 'user.tokens' module.
Previously, OutputPage hardcoded them in the top. The new formatter doesn't.
* Remove getHeadScripts().
* Remove getInlineHeadScripts().
* Remove getExternalHeadScripts().
* Remove buildCssLinks().
* Remove getScriptsForBottomQueue().
* Change where Skin::setupSkinUserCss() is called. This methods lets the skin
add modules to the queue. Previously it was called from buildCssLinks(),
via headElement(), via prepareQuickTemplate(), via OutputPage::output().
It's now in OutputPage::output() directly (slightly earlier). This is needed
because prepareQuickTemplate() calls bottomScripts() before headElement().
And bottomScript() would lazy-initialise the queue and lock it before
setupSkinUserCss() is called from headElement().
This makes execution order more predictable instead of being dependent on
the arbitrary order of data extraction in prepareQuickTemplate (which varies
from one skin to another).
* Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early
on so. This avoids wrongful loading and fixes problem 1.
Effective changes in output:
* mw.loader.state() is now before mw.loader.load(). This fixes problem 2.
* mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading".
* mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1.
* The <script async src> tag for 'startup' changed position (slightly).
Previously it was after all inline scripts and stylesheets. It's still after
all inline scripts and after most stylesheets, but before any user styles.
Since the queue is now formatted outside OutputPage, it can't inject the
meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle
of existing output. This shouldn't have any noticable impact.
Bug: T87871
Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
ResourceLoader modules can now carry a 'deprecated' option which can
be a boolean or an object with message key. This message or a default
deprecation message will be show whenever that module is used in production.
Note: This will not work in debug mode for ResourceLoaderFile modules
and this is deemed acceptable for the time being. We can revisit later.
Bug: T137772
Change-Id: Ib9ebd2d39a59fd41d8537e06884699f77b03580c
SHA-1 is not secure enough to be used as a cryptographic hash function, and its
implementation in JavaScript is too long and too slow for it to be a good
general-purpose hash function. And we currently throw away most of the work:
SHA-1 produces 160-bit hash values, of which we keep 48.
Although the JavaScript implementation is not exported, SHA-1 is a well-known
hash function, and I'm willing to bet that sooner or later someone will move to
make it accessible to other modules, at which point usage will start to spread.
For ResourceLoader, the qualities we're looking for in a hash function are:
* Already implemented in PHP
* Easy to implement in JavaScript
* Fast
* Collision-resistant
The requirement that hashes be cheap to compute in JavaScript narrows the field
to 32-bit hash functions, because in JavaScript bitwise operators treat their
operands as 32 bits, and arithmetic uses double-precision floats, which have a
total precision of 53 bits. It's possible to work around these limitations, but
it's a lot of extra work.
The best match I found is the 32-bit variant of FNV-1, which is available in
PHP as of version 5.4 (as 'fnv1a32'). The fnv132 JavaScript function is
around ten times faster and eight times shorter than sha1.
Change-Id: I1e4fb08d17948538d96f241b2464d594fdc14578
No longer needed per doing Ic137cb494ba23 in a different way.
This may be useful to revisit, but for now preferring to keep
simplicity and removing this unused option.
This reverts commit 9e217bf42d.
Change-Id: I9c0c316a3b58a3d0a3d3282dd74c7fa4eef8e378
To be used for the 'site' module. This will make it easier to split up
'site' into 'site' and 'site.styles' (per T92459).
Change-Id: Iaac3e458d5107e4c10c2826bd64608d5c47e8b87
We read files and concatenate their contents. Files may start with a BOM character.
BOM characters are only allowed at the beginning of a file, not half way.
Stripping it should be safe, since we already assume that everything is UTF-8.
Change-Id: I14ad698a684e78976e873e9ae2c367475550a063
* Send 'module' and 'require' parameters to module closures.
This depends on Ia925844cc22f143 being deployed one cycle earlier.
* Patch Moment and OOjs to ensure these libraries continue to expose
their module as globals as well. AMD/UMD-compatible libraries
only expose a global *OR* an export, not both. We need both
for back-compat.
* Update pluralRuleParser to make use of module export to allow
usage via require().
To test, check out the patch and run:
> mw.loader.load('moment');
> mw.loader.require('moment')()
> mw.loader.require('moment')('2011-04-01').fromNow()
Bug: T108655
Change-Id: Idbd054880ee70d659ec760aef8fcb38d0704a394
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
- Batch retrieval.
- Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
part of get() instead of within updateMessage().
ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
Timestamps are unreliable and roll over too frequently for message blob store
because there is no authoritative source. The timestamps were inferred based on
when a change is observed. Message overrides from the local wiki have an
explicit update event when the page is edited. All other messages, such as
from MediaWiki core and extensions using LocalisationCache, have a single
timestamp for all messages which rolls over every time the cache is rebuilt.
A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
Instead, detect whether a module's message list is empty at runtime.
ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.
cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.
Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
* Also added getBlob() and getBlobs() methods. These don't
require passing ResourceLoader as argument.
In preparation for T113092.
Change-Id: I3239c45f0243eca1eaf85bc53fca736e4c9a209c
This parameter was introduced last year as a way to extract
bare scripts from modules without any 'mw.loader.state()' suffix.
When ResourceFileCache is used ($wgUseFileCache) this causes
cache pollution as it didn't include getRaw() in the hash.
Change-Id: I7b9f9b6a5756777462395b911abafb62468cbefa
* No effective change. Only code reformatting.
* Logically group related members.
* Break up items one per line in getHash().
Change-Id: I8ccbe9d071a5c39f5c3d36d0d990574fb0ed8d72
The direction is derived from the language code, which is included
already. The method was added for convenience to consuming code,
but including it in the cache key seems pointless.
Main rationale here is runtime performance. getDirection() incurs
Language::factory() and Language::getDir() which require loading
of LCStore files.
Change-Id: I397a1c483203ec2c4903046c9494cae1c9480f8c
To avoid repeating this too much and make it easier to update when
underlying components incorporated into the version hash change.
Change-Id: I986988c07f5f888046b1d93c7e72f2e9585893cf
Make paths stored in the module_deps table relative to $IP. This ensures that when
the MediaWiki install path changes and/or if the location of the extension or skins
directory changes, that ResourceLoader's internal model is still accurate.
Previously when these paths change, ResourceLoader would exhibit various bugs.
1. Unable to detect changes in the module (if the directory no longer exists).
2. Point #1 is usually preceeded by one last cache invalidation as the content hash
of the file path changes (from a valid hash to an empty string).
3. Unnecessary cache invalidation (if both old and new directories exist). This
happens when a file is both an entry point (in the 'scripts' or 'styles' array)
and also a file dependency. At first they are de-duplicated by array_unique.
But after the disk path changes, the next check will result in the old path
being fetched from module_deps, and the new path from the live configuration.
This causes two changes that result in needless cache invalidation:
- The hash list contains one more item (T111481).
- The hash list contains both the old and new version of a file.
(or even alternate versions, e.g. when a change is backported to the old
wmf branch; it also affects wikis on the new branch due to the stale
file path still in the database).
It seems unusual to move a MediaWiki install, and usually we recommend third
parties to run update.php if site administrators do move their wiki. However
Wikimedia's deployment system implicitly moves the MediaWiki install continously
from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6".
This caused virtually all ResourceLoader caching layers to get invalidated every
week when another wmf-branch is deployed, thus breaking these file paths, which
changes the version hash, which then invalidates the cache.
Bug: T111481
Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
Deprecated in 1.24, for reasons explained in a0c41ae39d. I don't see any
usage in core or extensions.
Change-Id: I46f9e04ae633e7ff1ee112b652e1865731172f1f
The code is easier to maintain in an actual JavaScript file.
Especially with how variables were declared and concatenated in
a different order.
Change-Id: I758acb78de1cdf2128e81c86f992807ef0dbf444
getDefinitionSummary isn't the authoritive method to detect changes.
Using it as such may false cause something to appear detected or
undetected. Use getVersionHash() instead.
Thanks to Gilles for uncovering this bug.
Bug: T105476
Change-Id: Ibefc9fa8ffd9d45e29901d726801e8d4e008b66f
This greatly simplifies logic required to compute module versions.
It also makes it significantly less error-prone.
Since f37cee996e, we support hashes as versions (instead of timestamps).
This means we can build a hash of the content directly, instead of compiling a
large array with all values that may influence the module content somehow.
Benefits:
* Remove all methods and logic related to querying database and disk for
timestamps, revision numbers, definition summaries, cache epochs, and more.
* No longer needlessly invalidate cache as a result of no-op changes to
implementation datails. Due to inclusion of absolute file paths in the
definition summary, cache was always invalidated when moving wikis to newer
MediaWiki branches; even if the module observed no actual changes.
* When changes are reverted within a certain period of time, old caches can now
be re-used. The module would produce the same version hash as before.
Previously when a change was deployed and then reverted, all web clients (even
those that never saw the bad version) would have re-fetch modules because the
version increased.
Updated unit tests to account for the change in version. New default version of
empty test modules is: "mvgTPvXh". For the record, this comes from the base64
encoding of the SHA1 digest of the JSON serialised form of the module content:
> $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}';
> echo base64_encode(sha1($str, true));
> FEb3+VuiUm/fOMfod1bjw/te+AQ=
Enabled content versioning for the data modules in MediaWiki core:
* EditToolbarModule
* JqueryMsgModule
* LanguageDataModule
* LanguageNamesModule
* SpecialCharacterDataModule
* UserCSSPrefsModule
* UserDefaultsModule
* UserOptionsModule
The FileModule and base class explicitly disable it for now and keep their
current behaviour of using the definition summary. We may remove it later, but
that requires more performance testing first.
Explicitly disable it in the WikiModule class to avoid breakage when the
default changes.
Ref T98087.
Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
getDirection() isn't a simple getter value like the others. It actually
is tightly coupled with getLanguage() and lazy-initialised.
When calling setLanguage(), we shouldn't reset direction back to the
parent class but make sure getDirection() will recompute it based
on the local value.
Added regression test (which fails without this patch).
The parent getDirection() looks for $this->request, but the subclass
doesn't assign that member in the constructor. getRequest() forwards
it accordingly, so make sure getRequest() is also used internally.
Change-Id: Ifec703647368c3bb58748288ed754aaaf3730e19
The ResourceLoaderContext class used null to determine absence of
an overridde in the derivative object.
However three of the members in question allow null as legitimate value.
(Namely 'only', 'user', and 'version').
This makes is impossible for a derivative context to remove one
of those values if the parent context has them set.
Use case: I782df43c needs to create a derivative context of
load.php?only=scripts&modules=startup without 'only'.
Use -1 instead as internal placeholder value.
Also:
* ResourceLoaderContext::getSkin() was documented as returning 'string|null' when in
fact it always has a default value. Never returns null.
* DerivativeResourceLoaderContext::setOnly() and setVersion() were missing
type hint for 'null' (as it was incompatible with their getter). Adding 'false'.
* Swap if/else statements to handle the special case first (inheriting).
Allowing the rest of the function body to handle the local value.
In preparation for further development.
Change-Id: I058884525237effe8aef35469ed7693bb7cea591
Wiki modules are special due to their isKnownEmpty implementation and support
for foreign databases. MediaWiki doesn't have convenient ways of making
Revision objects for remote wikis. As such, wiki modules will keep using meta
data to generate the hash.
However minimise needless cache invalidation by refining the implementation.
Impact:
* Remove use of getMsgBlobMtime(). This module doesn't support getMessages().
* In the title info, use the revision content sha1 and size for tracking.
The page_touched previously used updates too often. It's updated both on edits
for various types of purges. Using the rev_sha1 means old versions return
when the content is the same. Regardless of how the content changed via
revert or actual edits resulting in the same contnet.
* Change in-process cache to be keyed by page list instead of entire
ResourceLoaderContext.
Because of this, getTitleInfo() was previously performing its batch query
twice on the same page. Once for only=styles (top) and only=scripts (bottom).
Both operate on the full getPages() set but had different context keys.
Clean up:
* Better document the support for foreign databases.
* Move Title construction to getContent to reduce duplication.
* Remove use of getDefinitionMtime(). That method is a no-op since the switch
to version hashing.
* Remove remaining use of mtime in getModifiedTime(). This is now covered by
hashing the title info in getDefinitionSummary().
Also refactor the code to be more readable. No intended change in behaviour.
Bug: T98087
Change-Id: Id46740db04c0c42bc5ca87d1487230a32feb34df
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
Follows-up ebeb29723, 1f393b6da, 0e719ce23.
Also:
* Add tests for ResourceLoader::makeLoaderImplementScript().
* Apply ResourceLoader::trimArray to makeLoaderImplementScript (new in c0c221bf).
This commit changes the load.php response to omit empty parameters.
These parameters were required until recently. The client has been
updated (1f393b6da and 0e719ce23) to make these optional, thus supporting
both the old server format and the change this commit makes
Clients with a tab open from before 0e719ce23 are naturally not
compatible with load.php responses from after this commit. Ensure
this is deployed several days after 0e719ce23 to reduce race
conditions of this nature.
(This is a re-submitted version of 4ce0c0da4)
Bug: T88879
Change-Id: I9e998261ee9b0b745e3339bc3493755c0cb04b6a
array( "en,de,fr" => "foo.svg" ) now means the same as
array( "en" => "foo.svg", "de" => "foo.svg", "fr" => "foo.svg" ).
Bug: T76539
Change-Id: I0bf82e06be3c5f94b6ac88bbc0437b5229ceb284
These tests all involve database access in some way,
and thus need @group Database tags.
These failed when setting a bogus database password
and then running the tests.
Change-Id: I7f113a79ac44d09d88ec607f76b8ec22bc1ebcf1
Also remove some wrappers from ResourceLoaderImageModule tests that
are no longer necessary, since the files they were mocking now exist.
Bug: T86334
Change-Id: If02e58716ce8e6c8327c8939c6c6425bd48bb560
Provides no value and makes the definitions uglier. Also, the
implementation sucks.
This is a breaking change, but ResourceLoaderImageModule was not in
any public release yet.
Submitted patches to fix two usages in extensions:
* Gather: I371209afe7b48e7c215ea9912826d4eb2cacf4e5
* MobileFrontend: I1963f5fe759c3a031220157d3a6f0f3b42bc5426
Bug: T94073
Change-Id: I36cfdb09bb203b8d9958e6016447e446dd6ff78b
Instead of a 'prefix', a 'selector' can be specified, with a string
containing a simple template to use for CSS selector. For example:
'selector' => '.mw-ui-icon-{name}'
'prefix' continues to work as before.
When using variants, one might want to provide separate
'selectorWithoutVariant' and 'selectorWithVariant' options.
Available variables are {prefix}, {type}, {name}, and {variant}.
Bug: T78215
Change-Id: I99ccaf25e8d24fed5afd0c4b770d2f389789ce4b
* Move testMixedCssAnnotations to ResourceLoaderFileModuleTest.
* Re-order data providers before test methods.
* Add relevant @covers annotations in resourceloader/ tests.
* Make test helper function private.
* Add a few @covers for methods called from OutputPage::makeResourceLoaderLink
(only one level deep, we should have separate unit tests for
the more internal helpers).
Change-Id: I2cc1757126214ed28059d4566ca813a86bcd95a7
Currently if code wants to check whether a module is registered it
has to call getModule() and see if the response !== null.
Change-Id: I4b470083ddaa5d8cd6be50d5c5b690d4b99b6c4a
This caused a database error due to NULL being inserted as name, which is illegal.
> Function: DatabaseSqlite::replace/single-row
> NOT NULL constraint failed: unittest_module_deps.md_module
> Stack trace:
> #3 includes/resourceloader/ResourceLoaderFileModule.php(420): DatabaseSqlite->replace()
> #4 tests/phpunit/includes/resourceloader/ResourceLoaderTest.php(88): ResourceLoaderFileModule->getStyles()
> #5 (): ResourceLoaderTest->testLessFileCompilation()
This test shouldn't be trigggering database updates, but that's for
a later change to stub out or refactor.
Bug: T91567
Change-Id: Ic451bd41e2ffc188d2efd6b7ce61b03b9de61296
This makes it easier for subclasses to use ResourceLoaderWikiModule. Currently
many subclasses of this simply need to override the getPages() method.
UserModule and SiteModule keep their getPages override due to the set of pages
being dependent on context.
Change-Id: I388531398671afacfec36c6c5746d72267b5bdac
Works as intended, but didn't account for the first implement() parameter
being omitted client-side. Revert until that is accounted for, then re-try after
that fix is rolled out for > 1 week.
This reverts commit 4ce0c0da42.
Change-Id: I36c1619991663c0303636d1d3f037b0021ac79bf
Follows-up ebeb297236.
Also:
* Add tests for ResourceLoader::makeLoaderImplementScript().
* Apply ResourceLoader::trimArray to makeLoaderImplementScript (new in c0c221bf).
As always, the client (updated in Ie32e7d6a3c) is backward-compatible with old
(cached) load.php module responses. However, the old client is not compatible
new load.php responses after this commit.
That's generally not an issue, as we don't cache the client very long (~ 5 min).
However people with their browser open and mw.loader clients initialised can
still make new module requests (e.g. modules loaded on-demand, such as when
previewing edits, clicking buttons etc.). That can easily be several hours after
initial page load. As such, client/server bound changes should always be
back-compat and deployed a reasonable time apart to reduce chances of active
sessions making subsequent requests. Ideally we'd find some solution to this in
the long-term, but handling this at all is better than what we usually do...
For deployment: Ensure this is deployed several days after Ie32e7d6a3c09f.
Change-Id: Ic8d7efe49b5d45e3f95a2f04e3a26a014b10af16
Mock calls to ResourceLoaderContext::getDirection(), which creates
Language objects to get the directionality of a language.
Change-Id: Ibe6da3013e658aa7cf596c1da2f8ca1314b7cdd3
We already did this, but it was rather convoluted with lots of
if/elseif sequences checking all the possible values.
Remove this logic from ResourceLoaderStartUpModule. Simplying it
simply create the array and pass it to ResourceLoader::makeLoaderRegisterScript.
In makeLoaderRegisterScript, we apply a filter to the array(s) that
trim empty values.
While at it:
* As with other registration properties' default values (like for dependencies,
group, and skip) also use 'null' for the default value of 'source'.
The mediawiki.js client was already compatible with this, and the server omitted
it if it was the last value in the list. But in all other cases it explicitly
outputs "local". Use null instead of simplicity sake. This also gains us a
few characters in the output, and a relatively larger win after gzip since
there's lots more re-using of "null".
* Remove stray casting of $version to int. This only happened in case of
registering a single module (which don't do anywhere), and is redundant.
Change-Id: I1f321e7b8bd3b5cffc550b51169957a3da9b971d
By using the existing indexes of modules in the array being passed to
mw.loader.register we can reduce the size of the startup module by about
6% after gzip (nearly 20% before) on a wiki with very few modules (such
as my localhost). Comparing data from en.wikipedia.org shows about 9%
after gzip (nearly 30% before).
The technique adds a function to mediawiki.js which resolves the indexes
before registering the modules, which costs a little bit of data in that
payload, but it's negligible (118 bytes after gzip) in comparison to the
overall reduction.
Also, cleaned up lies in documentation and strange use of "m" as an
iterator variable.
Bonus: fix ISO8601 timestamp instead of UNIX timestamp being passed
to custom loader scripts.
Change-Id: If12991413fa6129cd20ceab0e59a3a30a4fdf5ce
* Use time() instead of:
- wfTimestamp()
- wfTimestamp( TS_UNIX )
- wfTimestamp( TS_UNIX, 0 )
- wfTimestamp( TS_UNIX, time() )
- intval( wfTimestamp( TS_UNIX, time() ) )
* Consistently use 1 as default instead of 0. Previously the
unwritten convention was that anything "final" max()'ed with 1,
and any internal method would use 0, but this wasn't applied
consistently made it fragile. There doesn't seem to be any
value in returning 0 only to have it maxed up to 1 (because if
the 0 would ever make it out alive, we'd be in trouble).
* wfTimestamp returns a string for TS_UNIX. In PHP this doesn't
matter much. In fact, max() takes number-like integers so
transparently, it even preserves it:
> max( 1, 3, '2' );
< 3
> max( 1, '3', 2 );
< "3"
Just cast it in one place at the very end (StartupModule)
instead of doing intval( wfTimestamp() ).
* Fix weird documentation claiming getModifiedTime can return
an array, or mixed.
* Remove 'version > 1 ? version : 1' logic in
ResourceLoader::makeLoaderRegisterScript. The client doesn't
have "0 means now" behaviour so this isn't needed. And the method
was only doing it for variadic argument calls.
Removal of quotes around timestamps reduced the size of the startup
module from 26.8KB to 25.9KB before gzip. After gzip the size was
and still is 5.7KB, though. (From 5456 bytes to 5415 bytes.)
Change-Id: If92ca3e7511e78fa779f2f2701e2ab24db78c8a8
optimizeDependencies got renamed to compileUnresolvedDependencies
but the covers tag was not changed accordingly.
Change-Id: I63a3d5454dd69b545b63150b50430e65b3928192
A base ResourceLoaderModule::getTemplates() exists for subclasses
to override. An implementation is provided for ResourceLoaderFileModule.
For file modules, templates can be specified in the following manner:
'example' => array(
'templates' => array(
'bar' => 'templates/foo.html',
),
'scripts' => 'example.js',
),
The delivery system is template language agnostic, and currently
only supports "compiling" plain HTML templates.
This also adds template support to the following modules as a POC:
* mediawiki.feedback
* mediawiki.action.view.postEdit
* mediawiki.special.upload
Works with $wgResourceLoaderStorageEnabled
Change-Id: Ia0c5c8ec960aa6dff12c9626cee41ae9a3286b76
Not ready for merging, and Roan says that the +2 was
most likely accidental and meant to be a -1.
This reverts commit d146934f94.
Change-Id: I3926c9ae9e3c8026fceb3aeedd3b1f1d9b91667b
Preparation work for templating in core.
RL should allow us to ship HTML / template markup from server to client.
Use in Special:Upload and mediawiki.feedback as a proof of concept.
Separation of concerns etc...
See Also:
Ia63d6b6868f23a773e4a41daa0036d4bf2cd6724
Change-Id: I6ff38c12897e3164969a1090449e626001926c3b
This reverts most of commit 2d842f1425,
leaving only the test added in it, and reimplements the same
functionality better.
Instead of stripping /*@noflip*/ annotations in CSSJanus, which is
incompatible with other implementations that preserve it, extend
CSSMin to allow other CSS comments to be present before the
rule-global @embed annotation. (This required making the regex logic
in it even worse than it was, but it's actually slightly less terrible
than I expected it would be. Good thing we have tests!)
Bug: 69698
Change-Id: I58603ef64f7d7cdc6461b34721a4d6b15f15ad79
Custom LESS functions are problematic for us for a number of reasons,
as outlined by Timo on bug 67368. We should get rid of them.
The only use case was implementing CSSMin data: URI embedding in LESS,
which used to be impossible due to lessc not preserving comments (bug
54673). However, thanks to new syntax added in f3779e06 we can insert
the annotations in such a way that the compiler won't mess with them.
The same technique is used in OOjs UI since 584ed144.
The LESS-function-based embedding implementation also meant that we
were unable to flip images for RTL (bug 66091 and friends: bug 66773,
bug 68326). The annotation one doesn't have this limitation.
Bug: 67368
Bug: 66091
Bug: 66773
Bug: 68326
Change-Id: I3062346ed63272a1c22b5df27b4cc1de2a699d9a
Follows-up b36d883.
By far most data providers are static (and PHPUnit expects them
to be static and calls them that way).
Most of these classes already had their data providers static
but additional commits sloppily introduced non-static ones.
* ResourceLoaderWikiModuleTest, 8968d8787f.
* TitleTest, 545f1d3a73.
Odd unused method 'dataTestIsValidMoveOperation' was introduced
in 550b878e63.
* GlobalVarConfigTest, a3e18c3670.
Change-Id: I5da99f7cd3da68c550ae507ffe1f725d31e7666f
Previously ResourceLoader would store any arbitrary data about a
source, provided it had a 'loadScript' key. It would register
the 'local' source with an additional 'apiScript' key, which was
also documented in DefaultSettings.php. However, it was
completely unused outside of the ForeignAPIGadgetRepo class in
Gadgets 2.0, which should be changed to take an API url as a
parameter. This was not useful as it was not ever formally
exposed, and it could not be depended upon that a source had
registered an 'apiScript' key.
For backwards compatability, both ResourceLoader::addSource()
and mw.loader.addSource() will both take an array/object, but
discard all parameters except for 'loadScript'.
Also added tests for ResourceLoader::addSource().
Bug: 69878
Change-Id: I4205cf788cddeec13b619be0c3576197dec1b8bf
In most cases, we just check whether the pages exist before saying
the module is not empty to avoid generating cached HTML without
the appropriate <script> or <link> tags.
However, for modules in the 'user' group, normal users cannot
delete their personal JavaScript/CSS pages, causing needless
extra requests, even though we know the pages are empty.
ResourceLoader::isKnownEmpty() now checks the page_len field
for modules in the 'user' group to check that there is
some actual content.
Bug: 68488
Change-Id: I0570f62887fd4642fd60367ae0b51d7dc19488ca
- Added space after reserved words: function, foreach, if
- Combined 'else if' into elseif
- Added braces to one-line statements
- Added spaces after comma, before parentheses
Change-Id: Ie5bbf680d6fbe0f0872dab2700c16b1394906a72
To do it, just remove /*@noflip*/ annotations in CSSJanus after
we're done processing. They are not needed anymore and some obscure
interactions with CSSMin logic for preserving comments caused
`/*@noflip*/ /*@embed*/ background-image: url(…)` not to work
correctly (it would not be embedded).
This also requires us to always do CSSJanus processing, even when we
don't need flipping, to consistently handle the annotations.
I'm not entirely sure if this is worth it, but I still greatly prefer
doing it to documenting this stupid limitation. :)
Bug: 69698
Change-Id: I311b12b08b2dff9d45efb584db08cf4a11318f59
Given that we have entirely separate code for handling one and the
other, and given the nature of code comments stuffed inside other
structures, this isn't really obvious that they work.
And indeed, "/*@noflip*/ /*@embed*/" doesn't work (filed bug 69698).
Amazingly, all other combinations do.
Change-Id: Ie30bab251eb4abee122c783d057de4102e53d1fc
To do so, created ResourceLoader::createLoaderURL(), which takes a
ResourceLoaderContext object. ResourceLoader::makeLoaderURL() was
deprecated.
While reviewing usage of the old function, many of the callers only
differed by one or two parameters from their respective
ResourceLoaderContext object. To simplify that use case, I created
DerivativeResourceLoaderContext, based of off DerivativeContext for
IContextSource.
Change-Id: I961c641ab953153057be3e3b8cf6c07435e9a0b0
* Add the ResourceLoaderFileModule#getAllSkinStyleFiles method,
which returns all of the skinStyles files for a given module
* The LessFileCompilationTest and checkLess.php script, which use
the #getAllStyleFile method, now validate all LESS style files
Bug: 63343
Change-Id: Ib2eb5761919c648aea4ae58f8d0531799fe7982b
A module can be registered with a skip function. Such function,
if provided, will be invoked by the client when a module is
queued for loading. If the function returns true, the client will
bypass any further loading action and mark the module as 'ready'.
This can be used to implement a feature test for a module
providing a shim or polyfill.
* Change visibility of method ResourceLoader::filter to public.
So that it can be invoked by ResourceLoaderStartupModule.
* Add option to suppress the cache key report in ResourceLoader::filter.
We usually only call the minifier once on an entire request
reponse (because it's all concatenated javascript or embedded
javascript in various different closures, still valid as one
large script) and only add a little bottom line for the cache
key. When embedding the skip function we have to run the minifier
on them separately as they're output as strings (not actual
functions). These strings are typically quite small and blowing
up the response with loads of cache keys is not desirable in
production.
* Add method to clear the static cache of ResourceLoader::inDebugMode.
Global static state is evil but, as long as we have it, we at
least need to clear it after switching contexts in the test suite.
Also:
* Remove obsolete setting of 'debug=true' in the FauxRequest in
ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug
in the setUp() method.
Bug: 66390
Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
The optimization basically works like this:
* Given module A with the dependencies B and C and module B with the
dependency C.
* Don't tell the client that A depends on C, as that's already included
in module B.
This way we can reduce the amount of data for module registration sent
to the client.
The code here isn't polished yet, but it works and should be good enough
to demonstrate my idea and implementation.
Change-Id: I7732a3b1d879c5eef059e136a5241d6d48046872
The icons in Flow and the Compact Personal Bar beta feature are missing,
their generated background-image 7 is empty.
The lessphp implementation passes full data frames into the helper
functions. Utilize the now-public compileValue method of less
to evaluate the data frame into a string.
Additionally adds a small abstraction to run pairs of less/css files
as unit tests by executing the less file and comparing it to the
equivalent css file.
Change-Id: I1704f84638d86a0e6e6b9c190972ab19180bd484
