We read files and concatenate their contents. Files may start with a BOM character.
BOM characters are only allowed at the beginning of a file, not half way.
Stripping it should be safe, since we already assume that everything is UTF-8.
Change-Id: I14ad698a684e78976e873e9ae2c367475550a063
* Send 'module' and 'require' parameters to module closures.
This depends on Ia925844cc22f143 being deployed one cycle earlier.
* Patch Moment and OOjs to ensure these libraries continue to expose
their module as globals as well. AMD/UMD-compatible libraries
only expose a global *OR* an export, not both. We need both
for back-compat.
* Update pluralRuleParser to make use of module export to allow
usage via require().
To test, check out the patch and run:
> mw.loader.load('moment');
> mw.loader.require('moment')()
> mw.loader.require('moment')('2011-04-01').fromNow()
Bug: T108655
Change-Id: Idbd054880ee70d659ec760aef8fcb38d0704a394
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
- Batch retrieval.
- Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
part of get() instead of within updateMessage().
ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
Timestamps are unreliable and roll over too frequently for message blob store
because there is no authoritative source. The timestamps were inferred based on
when a change is observed. Message overrides from the local wiki have an
explicit update event when the page is edited. All other messages, such as
from MediaWiki core and extensions using LocalisationCache, have a single
timestamp for all messages which rolls over every time the cache is rebuilt.
A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
Instead, detect whether a module's message list is empty at runtime.
ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.
cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.
Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
* Also added getBlob() and getBlobs() methods. These don't
require passing ResourceLoader as argument.
In preparation for T113092.
Change-Id: I3239c45f0243eca1eaf85bc53fca736e4c9a209c
This parameter was introduced last year as a way to extract
bare scripts from modules without any 'mw.loader.state()' suffix.
When ResourceFileCache is used ($wgUseFileCache) this causes
cache pollution as it didn't include getRaw() in the hash.
Change-Id: I7b9f9b6a5756777462395b911abafb62468cbefa
* No effective change. Only code reformatting.
* Logically group related members.
* Break up items one per line in getHash().
Change-Id: I8ccbe9d071a5c39f5c3d36d0d990574fb0ed8d72
The direction is derived from the language code, which is included
already. The method was added for convenience to consuming code,
but including it in the cache key seems pointless.
Main rationale here is runtime performance. getDirection() incurs
Language::factory() and Language::getDir() which require loading
of LCStore files.
Change-Id: I397a1c483203ec2c4903046c9494cae1c9480f8c
To avoid repeating this too much and make it easier to update when
underlying components incorporated into the version hash change.
Change-Id: I986988c07f5f888046b1d93c7e72f2e9585893cf
Make paths stored in the module_deps table relative to $IP. This ensures that when
the MediaWiki install path changes and/or if the location of the extension or skins
directory changes, that ResourceLoader's internal model is still accurate.
Previously when these paths change, ResourceLoader would exhibit various bugs.
1. Unable to detect changes in the module (if the directory no longer exists).
2. Point #1 is usually preceeded by one last cache invalidation as the content hash
of the file path changes (from a valid hash to an empty string).
3. Unnecessary cache invalidation (if both old and new directories exist). This
happens when a file is both an entry point (in the 'scripts' or 'styles' array)
and also a file dependency. At first they are de-duplicated by array_unique.
But after the disk path changes, the next check will result in the old path
being fetched from module_deps, and the new path from the live configuration.
This causes two changes that result in needless cache invalidation:
- The hash list contains one more item (T111481).
- The hash list contains both the old and new version of a file.
(or even alternate versions, e.g. when a change is backported to the old
wmf branch; it also affects wikis on the new branch due to the stale
file path still in the database).
It seems unusual to move a MediaWiki install, and usually we recommend third
parties to run update.php if site administrators do move their wiki. However
Wikimedia's deployment system implicitly moves the MediaWiki install continously
from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6".
This caused virtually all ResourceLoader caching layers to get invalidated every
week when another wmf-branch is deployed, thus breaking these file paths, which
changes the version hash, which then invalidates the cache.
Bug: T111481
Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
Deprecated in 1.24, for reasons explained in a0c41ae39d. I don't see any
usage in core or extensions.
Change-Id: I46f9e04ae633e7ff1ee112b652e1865731172f1f
The code is easier to maintain in an actual JavaScript file.
Especially with how variables were declared and concatenated in
a different order.
Change-Id: I758acb78de1cdf2128e81c86f992807ef0dbf444
getDefinitionSummary isn't the authoritive method to detect changes.
Using it as such may false cause something to appear detected or
undetected. Use getVersionHash() instead.
Thanks to Gilles for uncovering this bug.
Bug: T105476
Change-Id: Ibefc9fa8ffd9d45e29901d726801e8d4e008b66f
This greatly simplifies logic required to compute module versions.
It also makes it significantly less error-prone.
Since f37cee996e, we support hashes as versions (instead of timestamps).
This means we can build a hash of the content directly, instead of compiling a
large array with all values that may influence the module content somehow.
Benefits:
* Remove all methods and logic related to querying database and disk for
timestamps, revision numbers, definition summaries, cache epochs, and more.
* No longer needlessly invalidate cache as a result of no-op changes to
implementation datails. Due to inclusion of absolute file paths in the
definition summary, cache was always invalidated when moving wikis to newer
MediaWiki branches; even if the module observed no actual changes.
* When changes are reverted within a certain period of time, old caches can now
be re-used. The module would produce the same version hash as before.
Previously when a change was deployed and then reverted, all web clients (even
those that never saw the bad version) would have re-fetch modules because the
version increased.
Updated unit tests to account for the change in version. New default version of
empty test modules is: "mvgTPvXh". For the record, this comes from the base64
encoding of the SHA1 digest of the JSON serialised form of the module content:
> $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}';
> echo base64_encode(sha1($str, true));
> FEb3+VuiUm/fOMfod1bjw/te+AQ=
Enabled content versioning for the data modules in MediaWiki core:
* EditToolbarModule
* JqueryMsgModule
* LanguageDataModule
* LanguageNamesModule
* SpecialCharacterDataModule
* UserCSSPrefsModule
* UserDefaultsModule
* UserOptionsModule
The FileModule and base class explicitly disable it for now and keep their
current behaviour of using the definition summary. We may remove it later, but
that requires more performance testing first.
Explicitly disable it in the WikiModule class to avoid breakage when the
default changes.
Ref T98087.
Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
getDirection() isn't a simple getter value like the others. It actually
is tightly coupled with getLanguage() and lazy-initialised.
When calling setLanguage(), we shouldn't reset direction back to the
parent class but make sure getDirection() will recompute it based
on the local value.
Added regression test (which fails without this patch).
The parent getDirection() looks for $this->request, but the subclass
doesn't assign that member in the constructor. getRequest() forwards
it accordingly, so make sure getRequest() is also used internally.
Change-Id: Ifec703647368c3bb58748288ed754aaaf3730e19
The ResourceLoaderContext class used null to determine absence of
an overridde in the derivative object.
However three of the members in question allow null as legitimate value.
(Namely 'only', 'user', and 'version').
This makes is impossible for a derivative context to remove one
of those values if the parent context has them set.
Use case: I782df43c needs to create a derivative context of
load.php?only=scripts&modules=startup without 'only'.
Use -1 instead as internal placeholder value.
Also:
* ResourceLoaderContext::getSkin() was documented as returning 'string|null' when in
fact it always has a default value. Never returns null.
* DerivativeResourceLoaderContext::setOnly() and setVersion() were missing
type hint for 'null' (as it was incompatible with their getter). Adding 'false'.
* Swap if/else statements to handle the special case first (inheriting).
Allowing the rest of the function body to handle the local value.
In preparation for further development.
Change-Id: I058884525237effe8aef35469ed7693bb7cea591
Wiki modules are special due to their isKnownEmpty implementation and support
for foreign databases. MediaWiki doesn't have convenient ways of making
Revision objects for remote wikis. As such, wiki modules will keep using meta
data to generate the hash.
However minimise needless cache invalidation by refining the implementation.
Impact:
* Remove use of getMsgBlobMtime(). This module doesn't support getMessages().
* In the title info, use the revision content sha1 and size for tracking.
The page_touched previously used updates too often. It's updated both on edits
for various types of purges. Using the rev_sha1 means old versions return
when the content is the same. Regardless of how the content changed via
revert or actual edits resulting in the same contnet.
* Change in-process cache to be keyed by page list instead of entire
ResourceLoaderContext.
Because of this, getTitleInfo() was previously performing its batch query
twice on the same page. Once for only=styles (top) and only=scripts (bottom).
Both operate on the full getPages() set but had different context keys.
Clean up:
* Better document the support for foreign databases.
* Move Title construction to getContent to reduce duplication.
* Remove use of getDefinitionMtime(). That method is a no-op since the switch
to version hashing.
* Remove remaining use of mtime in getModifiedTime(). This is now covered by
hashing the title info in getDefinitionSummary().
Also refactor the code to be more readable. No intended change in behaviour.
Bug: T98087
Change-Id: Id46740db04c0c42bc5ca87d1487230a32feb34df
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
Follows-up ebeb29723, 1f393b6da, 0e719ce23.
Also:
* Add tests for ResourceLoader::makeLoaderImplementScript().
* Apply ResourceLoader::trimArray to makeLoaderImplementScript (new in c0c221bf).
This commit changes the load.php response to omit empty parameters.
These parameters were required until recently. The client has been
updated (1f393b6da and 0e719ce23) to make these optional, thus supporting
both the old server format and the change this commit makes
Clients with a tab open from before 0e719ce23 are naturally not
compatible with load.php responses from after this commit. Ensure
this is deployed several days after 0e719ce23 to reduce race
conditions of this nature.
(This is a re-submitted version of 4ce0c0da4)
Bug: T88879
Change-Id: I9e998261ee9b0b745e3339bc3493755c0cb04b6a
array( "en,de,fr" => "foo.svg" ) now means the same as
array( "en" => "foo.svg", "de" => "foo.svg", "fr" => "foo.svg" ).
Bug: T76539
Change-Id: I0bf82e06be3c5f94b6ac88bbc0437b5229ceb284
These tests all involve database access in some way,
and thus need @group Database tags.
These failed when setting a bogus database password
and then running the tests.
Change-Id: I7f113a79ac44d09d88ec607f76b8ec22bc1ebcf1
Also remove some wrappers from ResourceLoaderImageModule tests that
are no longer necessary, since the files they were mocking now exist.
Bug: T86334
Change-Id: If02e58716ce8e6c8327c8939c6c6425bd48bb560
Provides no value and makes the definitions uglier. Also, the
implementation sucks.
This is a breaking change, but ResourceLoaderImageModule was not in
any public release yet.
Submitted patches to fix two usages in extensions:
* Gather: I371209afe7b48e7c215ea9912826d4eb2cacf4e5
* MobileFrontend: I1963f5fe759c3a031220157d3a6f0f3b42bc5426
Bug: T94073
Change-Id: I36cfdb09bb203b8d9958e6016447e446dd6ff78b
Instead of a 'prefix', a 'selector' can be specified, with a string
containing a simple template to use for CSS selector. For example:
'selector' => '.mw-ui-icon-{name}'
'prefix' continues to work as before.
When using variants, one might want to provide separate
'selectorWithoutVariant' and 'selectorWithVariant' options.
Available variables are {prefix}, {type}, {name}, and {variant}.
Bug: T78215
Change-Id: I99ccaf25e8d24fed5afd0c4b770d2f389789ce4b
* Move testMixedCssAnnotations to ResourceLoaderFileModuleTest.
* Re-order data providers before test methods.
* Add relevant @covers annotations in resourceloader/ tests.
* Make test helper function private.
* Add a few @covers for methods called from OutputPage::makeResourceLoaderLink
(only one level deep, we should have separate unit tests for
the more internal helpers).
Change-Id: I2cc1757126214ed28059d4566ca813a86bcd95a7
Currently if code wants to check whether a module is registered it
has to call getModule() and see if the response !== null.
Change-Id: I4b470083ddaa5d8cd6be50d5c5b690d4b99b6c4a
This caused a database error due to NULL being inserted as name, which is illegal.
> Function: DatabaseSqlite::replace/single-row
> NOT NULL constraint failed: unittest_module_deps.md_module
> Stack trace:
> #3 includes/resourceloader/ResourceLoaderFileModule.php(420): DatabaseSqlite->replace()
> #4 tests/phpunit/includes/resourceloader/ResourceLoaderTest.php(88): ResourceLoaderFileModule->getStyles()
> #5 (): ResourceLoaderTest->testLessFileCompilation()
This test shouldn't be trigggering database updates, but that's for
a later change to stub out or refactor.
Bug: T91567
Change-Id: Ic451bd41e2ffc188d2efd6b7ce61b03b9de61296
This makes it easier for subclasses to use ResourceLoaderWikiModule. Currently
many subclasses of this simply need to override the getPages() method.
UserModule and SiteModule keep their getPages override due to the set of pages
being dependent on context.
Change-Id: I388531398671afacfec36c6c5746d72267b5bdac
Works as intended, but didn't account for the first implement() parameter
being omitted client-side. Revert until that is accounted for, then re-try after
that fix is rolled out for > 1 week.
This reverts commit 4ce0c0da42.
Change-Id: I36c1619991663c0303636d1d3f037b0021ac79bf
Follows-up ebeb297236.
Also:
* Add tests for ResourceLoader::makeLoaderImplementScript().
* Apply ResourceLoader::trimArray to makeLoaderImplementScript (new in c0c221bf).
As always, the client (updated in Ie32e7d6a3c) is backward-compatible with old
(cached) load.php module responses. However, the old client is not compatible
new load.php responses after this commit.
That's generally not an issue, as we don't cache the client very long (~ 5 min).
However people with their browser open and mw.loader clients initialised can
still make new module requests (e.g. modules loaded on-demand, such as when
previewing edits, clicking buttons etc.). That can easily be several hours after
initial page load. As such, client/server bound changes should always be
back-compat and deployed a reasonable time apart to reduce chances of active
sessions making subsequent requests. Ideally we'd find some solution to this in
the long-term, but handling this at all is better than what we usually do...
For deployment: Ensure this is deployed several days after Ie32e7d6a3c09f.
Change-Id: Ic8d7efe49b5d45e3f95a2f04e3a26a014b10af16
Mock calls to ResourceLoaderContext::getDirection(), which creates
Language objects to get the directionality of a language.
Change-Id: Ibe6da3013e658aa7cf596c1da2f8ca1314b7cdd3
We already did this, but it was rather convoluted with lots of
if/elseif sequences checking all the possible values.
Remove this logic from ResourceLoaderStartUpModule. Simplying it
simply create the array and pass it to ResourceLoader::makeLoaderRegisterScript.
In makeLoaderRegisterScript, we apply a filter to the array(s) that
trim empty values.
While at it:
* As with other registration properties' default values (like for dependencies,
group, and skip) also use 'null' for the default value of 'source'.
The mediawiki.js client was already compatible with this, and the server omitted
it if it was the last value in the list. But in all other cases it explicitly
outputs "local". Use null instead of simplicity sake. This also gains us a
few characters in the output, and a relatively larger win after gzip since
there's lots more re-using of "null".
* Remove stray casting of $version to int. This only happened in case of
registering a single module (which don't do anywhere), and is redundant.
Change-Id: I1f321e7b8bd3b5cffc550b51169957a3da9b971d
By using the existing indexes of modules in the array being passed to
mw.loader.register we can reduce the size of the startup module by about
6% after gzip (nearly 20% before) on a wiki with very few modules (such
as my localhost). Comparing data from en.wikipedia.org shows about 9%
after gzip (nearly 30% before).
The technique adds a function to mediawiki.js which resolves the indexes
before registering the modules, which costs a little bit of data in that
payload, but it's negligible (118 bytes after gzip) in comparison to the
overall reduction.
Also, cleaned up lies in documentation and strange use of "m" as an
iterator variable.
Bonus: fix ISO8601 timestamp instead of UNIX timestamp being passed
to custom loader scripts.
Change-Id: If12991413fa6129cd20ceab0e59a3a30a4fdf5ce
* Use time() instead of:
- wfTimestamp()
- wfTimestamp( TS_UNIX )
- wfTimestamp( TS_UNIX, 0 )
- wfTimestamp( TS_UNIX, time() )
- intval( wfTimestamp( TS_UNIX, time() ) )
* Consistently use 1 as default instead of 0. Previously the
unwritten convention was that anything "final" max()'ed with 1,
and any internal method would use 0, but this wasn't applied
consistently made it fragile. There doesn't seem to be any
value in returning 0 only to have it maxed up to 1 (because if
the 0 would ever make it out alive, we'd be in trouble).
* wfTimestamp returns a string for TS_UNIX. In PHP this doesn't
matter much. In fact, max() takes number-like integers so
transparently, it even preserves it:
> max( 1, 3, '2' );
< 3
> max( 1, '3', 2 );
< "3"
Just cast it in one place at the very end (StartupModule)
instead of doing intval( wfTimestamp() ).
* Fix weird documentation claiming getModifiedTime can return
an array, or mixed.
* Remove 'version > 1 ? version : 1' logic in
ResourceLoader::makeLoaderRegisterScript. The client doesn't
have "0 means now" behaviour so this isn't needed. And the method
was only doing it for variadic argument calls.
Removal of quotes around timestamps reduced the size of the startup
module from 26.8KB to 25.9KB before gzip. After gzip the size was
and still is 5.7KB, though. (From 5456 bytes to 5415 bytes.)
Change-Id: If92ca3e7511e78fa779f2f2701e2ab24db78c8a8
optimizeDependencies got renamed to compileUnresolvedDependencies
but the covers tag was not changed accordingly.
Change-Id: I63a3d5454dd69b545b63150b50430e65b3928192
