Deprecated since 1.26. No subclasses in Wikimedia Git define these methods,
no calls to methods by this name in Wikimedia Git.
If a module subclass were to still define such a method, it is simply
not called anymore. The version hash system introduced in 1.26 will
still invalidate modules based on wgCacheEpoch.
Bug: T94074
Change-Id: I65b2a625a30f22c8a9d14a3505605546fa5bab83
This effectively applies safemode to the mw.loader client,
without the client itself needing specific knowledge of safemode.
Test Plan:
* Unchanged: When viewing a page in safemode, the 'user' and
'site' modules are still not queued by OutputPage.
* New: mw.loader.getState('site'), previously would yield
'registered', but will now yield null.
* New: mw.loader.load('site'), previously loaded the module,
it now logs a dependency warning for unknown module, like for
any other unknown module.
* New: mw.loader.using('site'), previously resolved, it is now
rejected.
Bug: T185303
Change-Id: I672e3891c8e1b3c2d13655fa134d0f1d031b8247
The deprecation warning for the module 'mediawiki.ui' (used
e.g. on Special:UserLogin) is now actually shown.
Change-Id: If35a106c77622dbf7e8b5628fbea28f9e7ffd76d
These comments do not add anything. I argue they are worse than having
no comments, because I have to read them first to understand they
actually don't explain anything. Removing them makes room for actual
improvements in the future (if needed).
Change-Id: Iee70aad681b3385e9af282d5581c10addbb91ac4
I can see that "parent::__construct" literally calls the parent
constructor. I can see that stuff preceeded by the keyword "protected"
is protected. I really (really) don't need comments explaining such.
Change-Id: I7458e714976a6acd3ba6a7c93fdc27d03903df83
ResourceLoaderModule objects gain a new method getPreloadLinks() which
returns an array with the meta data required to build a Link rel=preload
header according to the current draft for W3C Preload.
<https://w3c.github.io/preload/>
Another implementation of this is already in use in OutputPage for
preloading the logo image.
This array is formatted by the ResourceLoaderModule::getHeaders method,
which is implemented as "final" at this time, thus restricting use to
the Link rel=preload header.
Headers are exposed and process-cached, like all other content
(scripts, styles, etc.), through ResourceLoaderModule::getModuleContent,
and aggregated by ResoureLoader::makeModuleResponse.
I had hoped for the getPreloadLinks to be stateless (not vary on $context).
Whether something should be preloaded and what, should not vary on the
skin or language. However, while that conceptually holds true, the exact
url for any given resource may still vary. Even the main use case for this
feature (T164299, preloading base modules request) require $context to pass
down skin and lang to the load.php url.
Add full test coverage and example documentation.
Bug: T164299
Change-Id: I2bfe0796ceaa0c82579c501f5b10e931f2175681
Rather than only the 'private' group triggering embedding, allow modules
to explicitly specify if they should be embedded.
The default is still to only embed when the group is 'private', and the
'private' group is still special in that ResourceLoader::respond() will
still refuse to serve it from load.php.
Change-Id: Ib9a043c566822e278baecc15e87f9c5cebc2eb98
This fixes two bugs:
* 1) When two modules are requested, and the first one ends with ";"
inside a comment, the second module might become part of
that comment.
* 2) A request with script=only where the requested module content
ends in a statement without ";", but has a comment after it
that does ends with a semicolon, then in debug=false, mw.loader.state()
would be appended directly after the semicolon-less statement because
the check is performed before minification.
Previously:
script> foo()
script> // bar();
states> mw.loader.state( {} );
Became (minified separately):
script> foo()
states> mw.loader.state({});
Became (concatenated)
> foo()mw.loader.state();
Which is invalid code.
Both of these are now fixed.
Bug: T162719
Change-Id: Ic8114c46ce232f5869400eaa40d3027003550533
* Add fileName to cache key to fix T52919. The cached parsed error
message contains the filename, this should be part of the cache
key as otherwise two identical user scripts may report the same
error message, including " on line X of page Y" where Y is whichever
of the two pages first created the cache entry.
* Make the cache key global instead of per-wiki. There is no need
for this to be per-wiki.
Bug: T52919
Change-Id: I6c2718c53be7f6384a6486a4a8718ae7f423d216
* Simplify by using early return and getWithSetCallback.
* Add TTL (previously indefinite, now 1 week).
Bug: T52919
Change-Id: Ic95ba392cdb3bcc8081c77d2c2a3240548bed366
Some used a string value, others an array with 'message' property.
Standardise on the string value, which seems more intuitive.
Change-Id: I5caead7b7017d2bad660db02fb45a54a26bf3728
Follows-up 047b60b96d (ref T111481).
The if-condition compared the expanded paths, not the relative paths.
This meant there were two conditions under which the code will perform
a useless write that inserts *literally* the exact same JSON value.
1. The base directory ($IP) changes after a branch upgrade.
2. Paths contain '../', '//' or other unnormalized paths.
The latter caused various Echo and ULS methods to keep writing the
same value because one of their images is referenced in CSS using
'../'. When inserted in the database as relative path and then
expanded again at run-time and compared to the input value, they
don't match ("$IP/foo/../bar.png" != "$IP/bar.png") and cause a write.
Bug: T158813
Change-Id: I223c232d3a8c4337d09ecf7ec6e5cd7cf7effbff
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.
Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
Follows-up 1d15085bb3.
The column has a unique index for module name and skin/language pair.
Previously the write lock was on module name, which meant that
shortly after deployment, the following happens:
* Files change on disk.
* (1-5min pass)
* First startup module request after 5min http-cache expires. Detects
one or more changes and updates the version hash of that module.
* Web client subsequently requests this module (if used on that page).
The first time that request comes in, it's a varnish cache miss
and will make RL load all files from disk related to that module
and update the cache index in the module_deps table. At this point
most popular skin/lang pairs fail, except one. As a result, the
other rows remain stale.
* (7-30 days varnish expiry pass OR another change to the module deploys)
* Web client requests this module and tries to update its skin/lang pair
for that module.
One simple change in January 2016 changes jquery.tablesorter to load
a PNG file instead of a GIF file. Now, over a year later, there are
still a dozen skin/lang pairs in enwiki.module_deps with stale data,
which is causing various suble bugs, as well as filesystem calls for
files that don't exist.
Ref T113916 (refactor module_deps).
Ref T158105 (stale cache bug).
Bug: T158105
Change-Id: Ib6c024bfa8d35ce2d622ba4242291daedb507d5e
This should perform better and reduce internal lock contention on the
database server.
Bug: T158105
Change-Id: I1acfb0630946283b317cb929e8d7c3b2af757ecf
* The styles queue has always been top-only
(except for a few months in 2015).
* The top queue loads asynchronous since mid-2015. (T107399)
And LocalStorage eval, previously the last remaining non-async part
of module loading, is also async as of October 2016. (T142129)
* This change merges the bottom 'mw.loader.load()' queue with the top queue.
It also moves any other snippets potentially in the bottom queue still:
- embed: I couldn't find any private modules with position=bottom
(doesn't make sense due to their blocking nature). If any do exist,
(third-party extensions?), they'll now be embedded in the <head>.
- scripts: Any legacy 'only=scripts' requests will now initiate
from the <head>.
Bug: T109837
Change-Id: I6c21e3e47c23df33a04c42ce94bd4c1964599c7f
This is more consistent with LoadBalancer, modern, and inclusive
of master/master mysql, NDB cluster, and MariaDB galera cluster.
The old constant is an alias now.
Change-Id: I0b37299ecb439cc446ffbe8c341365d1eef45849
ResourceLoader modules can now carry a 'deprecated' option which can
be a boolean or an object with message key. This message or a default
deprecation message will be show whenever that module is used in production.
Note: This will not work in debug mode for ResourceLoaderFile modules
and this is deemed acceptable for the time being. We can revisit later.
Bug: T137772
Change-Id: Ib9ebd2d39a59fd41d8537e06884699f77b03580c
This allows dynamically loaded modules to depend on page-style modules
without it causing the page-style module to be loaded a second time.
* New method Module::getType() indicates whether a module is
a page-style module or supposed to be dynamically loaded.
* Emit warning from addModuleStyles() when given a module that is
not a page-style module (to be enforced later)
Bug: T92459
Bug: T87871
Change-Id: I8b6c6a10d965e73965f877c42e995d04202524f3
Follows-up da36f65433 which added this method with intent to use in
ResourceLoader::register(). However, the feature was redesigned to
not need this and the method was left behind.
Change-Id: I5ebc93805d0df6605bec94094bcd4eb2b70ff18d
* Fix errors spotted by new release
* Introduce "composer fix", which uses phpcbf to automatically fix some
errors spotted by phpcs.
* Drop $PHPCS_ARGS variable that didn't work on Windows, and add -s flag
* Remove rules from phpcs.xml that are now in MW-CS ruleset.
Change-Id: I13e2155695918c918b67497ac65b85a03897095e
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
- Batch retrieval.
- Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
part of get() instead of within updateMessage().
ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
Timestamps are unreliable and roll over too frequently for message blob store
because there is no authoritative source. The timestamps were inferred based on
when a change is observed. Message overrides from the local wiki have an
explicit update event when the page is edited. All other messages, such as
from MediaWiki core and extensions using LocalisationCache, have a single
timestamp for all messages which rolls over every time the cache is rebuilt.
A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
Instead, detect whether a module's message list is empty at runtime.
ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.
cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.
Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
This effectively reverts d6b4d3c537 and declines T97420.
This was previously attempted in b7c0e537eb.
Drop support for position "bottom" for addModuleStyles().
This feature was only recently introduced with the intent
to optimise page load performance, but had an adverse effect.
It increases chances of FOUC due to late discovery of these styles.
It also caused minor problems for some gadgets and extensions
that did not or were unable to set these flags. Some mobile
code was introduced around the same time as this feature and
was never given position=top.
Stylesheets that don't affect initial render or are only needed
on interaction should be loaded via addModules() instead; which
is handled by the asynchronous loader in JavaScript.
Change-Id: Ib9821b7b87cfc8a59062bb6ca358974fdb01ced1
Use BagOStuff::makeKey() and BagOStuff::makeGlobalKey() instead of
wfMemcKey() and wfGlobalCacheKey().
Change-Id: Id4e58c0e616b74de220faf13ba7c9ea606fef8c1
Using the following command line, I have found doc comments (and
a wfDeprecated() call) mentioning "1.26" when they should mention
"1.27" instead, which I have fixed manually:
git diff -M REL1_26 | grep --color=always -C 10 -iP \
'^\+.*\D1\.26(\D|$)' | aha > oldver.html
Follows-up these commits:
* 047b60b96d
* 169b7b98b5
* 25a44aa3e4
* 2fb2a3f14b
* 3f1e9fa268 [1]
* 8c84af70b6
* c0cb80beac
* d04a92a551
* d3b85592ea
[1] Release notes moved in I195dd1cf.
Because a release note stating that the UserRights hook is deprecated
was added in 37062a0c0d (before the branch cut), this commit does not
change the Hooks::run call that had been changed in 21206c8fbe.
Change-Id: I5a427f003e7e3b4559fe377bcdfdca466a570708
* Add support for a '/* @nomin */' annotation in ResourceLoader. If present in
JavaScript or CSS, the code will not be minified or cached. This allows
modules like the ResourceLoaderUserTokensModule to declare themselves unfit
for minification / caching without requiring a complicated refactor.
* Make ResourceLoader::filter() static, at the cost of not having minifier
errors in the ResourceLoader log bucket. (They will continue to be logged as
exceptions, however).
Change-Id: Ic1d802ee20565e61046bfbd8fd209bc56a4cbd6c
Re-use the md_skin database field for now as to not need a
schema change, beause this table is going away soon (T113916).
Bug: T113868
Change-Id: I7c7546ec58fd9be0447604989b908dd2084b0fe3
Make paths stored in the module_deps table relative to $IP. This ensures that when
the MediaWiki install path changes and/or if the location of the extension or skins
directory changes, that ResourceLoader's internal model is still accurate.
Previously when these paths change, ResourceLoader would exhibit various bugs.
1. Unable to detect changes in the module (if the directory no longer exists).
2. Point #1 is usually preceeded by one last cache invalidation as the content hash
of the file path changes (from a valid hash to an empty string).
3. Unnecessary cache invalidation (if both old and new directories exist). This
happens when a file is both an entry point (in the 'scripts' or 'styles' array)
and also a file dependency. At first they are de-duplicated by array_unique.
But after the disk path changes, the next check will result in the old path
being fetched from module_deps, and the new path from the live configuration.
This causes two changes that result in needless cache invalidation:
- The hash list contains one more item (T111481).
- The hash list contains both the old and new version of a file.
(or even alternate versions, e.g. when a change is backported to the old
wmf branch; it also affects wikis on the new branch due to the stale
file path still in the database).
It seems unusual to move a MediaWiki install, and usually we recommend third
parties to run update.php if site administrators do move their wiki. However
Wikimedia's deployment system implicitly moves the MediaWiki install continously
from e.g. "/srv/mediawiki/php-1.26wmf5" to "/srv/mediawiki/php-1.26wmf6".
This caused virtually all ResourceLoader caching layers to get invalidated every
week when another wmf-branch is deployed, thus breaking these file paths, which
changes the version hash, which then invalidates the cache.
Bug: T111481
Change-Id: I173a9820b3067c4a6598a4c8d77e239797d2659c
* Remove ResourceLoaderFileModule::getLessCompiler(). There is no reason for a
module to need to get a compiler in a different manner than
ResourceLoader::getLessCompiler().
* Add ResourceLoaderModule::getLessVars(). This method provides a means for
subclasses to easily inject custom LESS variables. The default implementation
simply returns an empty array.
* Make the $context parameter for ResourceLoaderFileModule::readStyleFiles()
non-optional (via graceful deprecation). The only callers I found were either
already calling it with a ResourceLoader context, or had a perfectly usable
ResourceLoaderContext in local scope.
* Make ResourceLoaderFileModule::{readStyleFile,getLessCompiler} require a
context. These methods are protected, so I didn't bother with a deprecation.
* Call ksort() on the LESS variables array in the only place it matters -- when
hashing its serialized representation to construct a cache lookup key. This
relieves getLessVars() subclasses from having to remember to re-sort the
variables array if they modify it.
* These changes make it possible to substantially simplify
ResourceLoaderEditToolbarModule, because the only thing it needs to do now is
implement its own getLessVars() method.
* This also allows it to be versioned like any other ResourceLoaderFileModule,
rather than having to use enableModuleContentVersion().
Change-Id: Ic3eab71691e502bfe19bdf4eb6f82cc679a7782f
Most of it was already there (SELECT in preloadInfo, and in-class holder via
setFileDependencies) but the logic to write to the database was within
the file module class.
Moving it the other way around may make more sense in the future, but for
the moment ResourceLoader.php makes assumptions about all modules being
able to store setFileDependencies as read from the database. Whether or not
a class currently implements a way to write to that table.
Change-Id: I06b5da4144798197478c66e8b9ccd4cc62cf6fb6
Replace the one-off code for APC-assisted file content hashing from I5ceb8537c
with a call to the FileContentsHasher class introduced in I1ff61153.
Change-Id: I8e4c3d5a5e7cdf6455a43769261d742b0255622c
Cache the content hash of module files in APC, if available. Include the
modification time in the cached value, so that touching the file invalidates
the cache.
Does this take us back to a world in which updates to a file's mtime cause a
module's version to change? No. The content hash cache will be invalidated,
and the content hash will be computed, but the actual value should remain
the same.
Change-Id: I5ceb8537c3cdb120beae19740635f085ee128272
The hash value generated by ResourceLoaderModule::safeFileHash() is used for
versioning and cache invalidation, so a cryptographic hash function is not
necessary. We can get better performance by using MD4.
Wikimedia on-CPU time over 24 hours of safeFileHash() in load.php:
* 6.68% with safeFileHash using SHA1 (day 1)
* 7.07% with safeFileHash using SHA1 (day 2)
* 2.84% with safeFileHash using MD4 (day 3)
Change-Id: I6ff728f1240268517c0f03e0823129316bc901cb
Follows-up 28f6d7fbde.
This was causing versions to stay stuck at their current version.
It worked fine for the initial deployment since the rest of the
definition summary is still fine, but subsequent deploys that changed
only the file were ineffective due to these silently becoming
empty strings.
Thanks to suppressWarnings() we never saw these anywhere in the
tests or in production.
Change-Id: Ib4371255fe970442bcc0041219bce633a7f6f6dd
Enabling the module content versionining is not feasible for FileModule as that
would involve Lessc and CSSJanus just to compute the version hash.
Instead, we can keep the existing logic that exists for the timestamp-based
versioning (which already has a comprehensive grip on tracking all involved
factors that cause a module to change) and convert it to use hashing instead.
This way the version hashes will be deterministic. Currently module versions
tend to be invalidated too often (and sometimes not often enough) due to Git and
other transport mechanisms not preserving file timestamps.
== Research ==
* <https://phabricator.wikimedia.org/T98087#1412712>
It'd take upto 10 seconds to run startup if FileModule enables build versioning.
* <https://phabricator.wikimedia.org/T104950#1433142>
Checking all files in resources/** took only ~30m longer in total with
sha1_file compared to filemtime.
Bug: T104950
Change-Id: I732fa4db32258c634e32b507952f76eac7fc9395
strtr() is marginally faster as it runs through the string only
once. A better fit for one-for-one character translation.
The strtr() function also supports an associative array as second
parameter for entire string replacements. This, too, has the same
performance and predictable behaviour (starts with the longest key).
Whereas str_replace is for more aggressive needs where you want
multiple passes until there are no further matches.
The associative array form is arguably also easier to understand
and harder to mess up since the needle/replacement pairs are
explicitly connected instead of two separate arrays.
Also:
* Use getFormattedNsText instead of strtr( getNsText, .. ) which
reduces duplication of this fact through a more semantic intent.
Change-Id: Ie23e4210a5b6908dd79eebc8a2b931d12fe31af6
This greatly simplifies logic required to compute module versions.
It also makes it significantly less error-prone.
Since f37cee996e, we support hashes as versions (instead of timestamps).
This means we can build a hash of the content directly, instead of compiling a
large array with all values that may influence the module content somehow.
Benefits:
* Remove all methods and logic related to querying database and disk for
timestamps, revision numbers, definition summaries, cache epochs, and more.
* No longer needlessly invalidate cache as a result of no-op changes to
implementation datails. Due to inclusion of absolute file paths in the
definition summary, cache was always invalidated when moving wikis to newer
MediaWiki branches; even if the module observed no actual changes.
* When changes are reverted within a certain period of time, old caches can now
be re-used. The module would produce the same version hash as before.
Previously when a change was deployed and then reverted, all web clients (even
those that never saw the bad version) would have re-fetch modules because the
version increased.
Updated unit tests to account for the change in version. New default version of
empty test modules is: "mvgTPvXh". For the record, this comes from the base64
encoding of the SHA1 digest of the JSON serialised form of the module content:
> $str = '{"scripts":"","styles":{"css":[]},"messagesBlob":"{}"}';
> echo base64_encode(sha1($str, true));
> FEb3+VuiUm/fOMfod1bjw/te+AQ=
Enabled content versioning for the data modules in MediaWiki core:
* EditToolbarModule
* JqueryMsgModule
* LanguageDataModule
* LanguageNamesModule
* SpecialCharacterDataModule
* UserCSSPrefsModule
* UserDefaultsModule
* UserOptionsModule
The FileModule and base class explicitly disable it for now and keep their
current behaviour of using the definition summary. We may remove it later, but
that requires more performance testing first.
Explicitly disable it in the WikiModule class to avoid breakage when the
default changes.
Ref T98087.
Change-Id: I782df43c50dfcfb7d7592f744e13a3a0430b0dc6
The startup module produces a manifest with versions representing the
entire module.
Typically, the request for the startup module itself has only=scripts.
However, that "only" must only apply to what resources of the startup module are
output in the request.
The context passed to getModifiedTime() and getVersionHash() must not suggest
any restriction of measuring only the scripts, or otherwise ignoring some
aspects of the module (such as stylesheets, or messages).
Most existing getModifiedTime() implementations compute timestamps of all
files, regardless of the context. So this bug didn't surface before.
However that will change for modules that compute the version hash based on
getModuleContent(), which does honour the getOnly() and shouldIncludeScripts()
methods of the request context.
Change-Id: Ib8f09c39d10724d146b53b6d53d82da18944a12b
- Removed space after casts
- Removed spaces in array index
- Added spaces around string concat
- Added space after words: switch, foreach
- else if -> elseif
- Removed parentheses around require_once, because it is not a function
- Added newline at end of file
- Removed double spaces
- Added spaces around operations
- Removed repeated newlines
Bug: T102609
Change-Id: Ib860222b24f8ad8e9062cd4dc42ec88dc63fb49e
wfSuppressWarnings() and wfRestoreWarnings() were split out into a
separate library. All usages in core were replaced with the new
functions, and the wf* global functions are marked as deprecated.
Additionally, some uses of @ were replaced due to composer's autoloader
being loaded even earlier.
Ie1234f8c12693408de9b94bf6f84480a90bd4f8e adds the library to
mediawiki/vendor.
Bug: T100923
Change-Id: I5c35079a0a656180852be0ae6b1262d40f6534c4
By providing context as a parameter in getDependencies, we allow
modules to dyanamically determine dependencies based on context.
Note: To ease rollout, the parameter is optional in this patch. It is expected
that it will be made non-optional in the near future.
The use case is for CentralNotice campaigns to be able to add special
modules ahead of deciding which banner to show a user. The dynamically
chosen RL modules would replace ad-hoc JS currently sent with some banners.
A list of possible campaigns and banners is already sent as a PHP-
implemented RL module; that's the module that will dynamically choose other
modules as dependencies when appropriate. This approach will save a round
trip as compared to dynamically loading the modules client-side.
For compatibility, extensions that override
ResourceLoaderModule::getDependencies() should be updated with the new
method signature. Here are changes for extensions currently deployed on
Wikimedia wikis:
* CentralNotice: I816bffa3815e2eab7e88cb04d1b345070e6aa15f
* Gadgets: I0a10fb0cbf17d095ece493e744296caf13dcee02
* EventLogging: I67e957f74d6ca48cfb9a41fb5144bcc78f885e50
* PageTriage: Ica3ba32aa2fc76d11a44f391b6edfc871e7fbe0d
* UniversalLanguageSelector: Ic63e617f51702c27104e123d4bed91983a726b7f
* VisualEditor: I0ac775ca286e64825e31a9213b94648e41a5bc30
For more on the CentralNotice use case, please see I9f80edcbcacca2.
Bug: T98924
Change-Id: Iee61e5b527321d01287baa03ad9b4d4f526ff3ef
Centralise the building of module packages into this method so
that it can be easily re-used.
This is in preparation for providing the option for modules to use
content-based hashing (instead of based on meta-data) in the version
string of modules. Having a getModuleContent method allows that to
be implemented in a way that is well-cached (as we'll be calling it
from multiple multiple code paths) and without duplicating this logic.
Bug: T98087
Change-Id: I376233caaabe44b6101565b70a50904abdf8ab4f
For scripts loaded from load.php using mw.loader.implement(), an
exception is handled fine and results in the module state progressing
from 'loading' to 'errror'.
However, for wikis modules loaded with only=scripts from the HTML directly,
there is no wrapper. As such, the state() call appended to the script is
never reached and the module indefinitely stays "loading".
VisualEditor (and others) already explicitly ignore the error state
of the wiki modules (with inline mw.loader.using and casting the Promise to resolved).
However in this case the Promise never completed to neither success nor failure.
As such, this was causing users with syntax errors in their user script to
encounter a stuck VisualEditor loading bar that never goes beyond 70%.
Change-Id: I091a020bdd3392b965e0e25d03b216037fadc48f
Currently all styles modules added to the page using
addModuleStyles are put into the head, regardless
of their "position" value.
Bug: T97420
Change-Id: Ie4287e17d6f298cc63f42f257b1f67ee36961b77
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
Similar to what getDefinitionMtime() does already. No need to
repeat the hash. No need for an array that needs serialising and
unserialising internally.
Change the hash key to avoid using old cache values.
Also moved the comment about hashes being included in the key to
this method. getDefinitionMtime() is a later method that performs
the same logic but doesn't need the entire story again.
Follows-up 044713c4, d3bdda32.
Change-Id: Idd83de5ac27138a2dbf2ec49d81ea9188bd6ad57
Xhprof generates this data now. Custom profiling of various
sub-function units are kept.
Calls to profiler represented about 3% of page execution
time on Special:BlankPage (1.5% in/out); after this change
it's down to about 0.98% of page execution time.
Change-Id: Id9a1dc9d8f80bbd52e42226b724a1e1213d07af7
* Use time() instead of:
- wfTimestamp()
- wfTimestamp( TS_UNIX )
- wfTimestamp( TS_UNIX, 0 )
- wfTimestamp( TS_UNIX, time() )
- intval( wfTimestamp( TS_UNIX, time() ) )
* Consistently use 1 as default instead of 0. Previously the
unwritten convention was that anything "final" max()'ed with 1,
and any internal method would use 0, but this wasn't applied
consistently made it fragile. There doesn't seem to be any
value in returning 0 only to have it maxed up to 1 (because if
the 0 would ever make it out alive, we'd be in trouble).
* wfTimestamp returns a string for TS_UNIX. In PHP this doesn't
matter much. In fact, max() takes number-like integers so
transparently, it even preserves it:
> max( 1, 3, '2' );
< 3
> max( 1, '3', 2 );
< "3"
Just cast it in one place at the very end (StartupModule)
instead of doing intval( wfTimestamp() ).
* Fix weird documentation claiming getModifiedTime can return
an array, or mixed.
* Remove 'version > 1 ? version : 1' logic in
ResourceLoader::makeLoaderRegisterScript. The client doesn't
have "0 means now" behaviour so this isn't needed. And the method
was only doing it for variadic argument calls.
Removal of quotes around timestamps reduced the size of the startup
module from 26.8KB to 25.9KB before gzip. After gzip the size was
and still is 5.7KB, though. (From 5456 bytes to 5415 bytes.)
Change-Id: If92ca3e7511e78fa779f2f2701e2ab24db78c8a8
A base ResourceLoaderModule::getTemplates() exists for subclasses
to override. An implementation is provided for ResourceLoaderFileModule.
For file modules, templates can be specified in the following manner:
'example' => array(
'templates' => array(
'bar' => 'templates/foo.html',
),
'scripts' => 'example.js',
),
The delivery system is template language agnostic, and currently
only supports "compiling" plain HTML templates.
This also adds template support to the following modules as a POC:
* mediawiki.feedback
* mediawiki.action.view.postEdit
* mediawiki.special.upload
Works with $wgResourceLoaderStorageEnabled
Change-Id: Ia0c5c8ec960aa6dff12c9626cee41ae9a3286b76
Not ready for merging, and Roan says that the +2 was
most likely accidental and meant to be a -1.
This reverts commit d146934f94.
Change-Id: I3926c9ae9e3c8026fceb3aeedd3b1f1d9b91667b
Preparation work for templating in core.
RL should allow us to ship HTML / template markup from server to client.
Use in Special:Upload and mediawiki.feedback as a proof of concept.
Separation of concerns etc...
See Also:
Ia63d6b6868f23a773e4a41daa0036d4bf2cd6724
Change-Id: I6ff38c12897e3164969a1090449e626001926c3b
- Swap "$variable type" to "type $variable"
- Added missing types
- Fixed spacing inside docs
- Makes beginning of @param/@return/@var/@throws in capital
- Changed some types to match the more common spelling
Change-Id: I8ebfbcea0e2ae2670553822acedde49c1aa7e98d
To do so, created ResourceLoader::createLoaderURL(), which takes a
ResourceLoaderContext object. ResourceLoader::makeLoaderURL() was
deprecated.
While reviewing usage of the old function, many of the callers only
differed by one or two parameters from their respective
ResourceLoaderContext object. To simplify that use case, I created
DerivativeResourceLoaderContext, based of off DerivativeContext for
IContextSource.
Change-Id: I961c641ab953153057be3e3b8cf6c07435e9a0b0
A module can be registered with a skip function. Such function,
if provided, will be invoked by the client when a module is
queued for loading. If the function returns true, the client will
bypass any further loading action and mark the module as 'ready'.
This can be used to implement a feature test for a module
providing a shim or polyfill.
* Change visibility of method ResourceLoader::filter to public.
So that it can be invoked by ResourceLoaderStartupModule.
* Add option to suppress the cache key report in ResourceLoader::filter.
We usually only call the minifier once on an entire request
reponse (because it's all concatenated javascript or embedded
javascript in various different closures, still valid as one
large script) and only add a little bottom line for the cache
key. When embedding the skip function we have to run the minifier
on them separately as they're output as strings (not actual
functions). These strings are typically quite small and blowing
up the response with loads of cache keys is not desirable in
production.
* Add method to clear the static cache of ResourceLoader::inDebugMode.
Global static state is evil but, as long as we have it, we at
least need to clear it after switching contexts in the test suite.
Also:
* Remove obsolete setting of 'debug=true' in the FauxRequest in
ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug
in the setUp() method.
Bug: 66390
Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Change-Id: Ifbb1da2a6278b0bde2a6f6ce2e7bd383ee3fb28a
We were continuously invalidating the cache for the
ResourceLoaderLanguageDataModule every time a request for a langauge
other than the previous request came in as they all resulted in
the same cache key.
This could be resolved by adding $context->getLang() to the key
(which would be enough for langdata module, but not enough for
other uses of hashmtime), or by adding $context->getHash() (which
will fragment it for all possible context factor combinations,
slightly overkill for most cases, but would work).
Instead resolving it the same way we did in d3bdda3 for a similar
issue in #getDefinitionMtime: by adding the hash itself to the
key. That way it will solve the problem, including two nice things:
* Not fragment anymore than necessary (if it doesn't vary by
skin, it won't vary by skin, and same for language etc.).
* Always separate keys when the value is different (if something
can very based on other factors, e.g. LocalSettings globals or
whatever untracked method may exist for influencing anything,
it will all work as expected).
* Don't keep alternating and incrementing the timestamp while a
multi-server infrastructure is syncing code (e.g. old code and
new code for the same request context will not result in the
same key and each old/new server alternatingly increasing the
timestamp, instead the first new server will make the new key
and the other servers will gradually start finding the new key
with the timestamp of when it was first seen).
Change-Id: Ifa9088c11c388ca6c912e62efc43494e91702603
I added the profiling calls in PS13 of I00cf086c9 but forgot to add
wfProfileOuts for the early returns.
Change-Id: Ibcd22daba8da0e087376553bca271796e0d207e5
Currently we invalidate module caches based on timestamps from
various places (including message blob, file mtimes and more).
This meant that when a module changes such that the maximum
detectable timestamp is still the same, the cache would not
invalidate.
Module classes can now implement a method to build a summary.
In most cases this should be (a normalised version of) the
definition array originally given to ResourceLoader::register
(such as $wgResourceModules items).
The most common scenarios this addresses:
* File lists (scripts, styles) being re-ordered.
* Files being removed when more recently changed files remain
part of the module (e.g. removing an older file).
* Files or messages being added to a module while other more
recently changed things are already part of the module.
E.g. you create a message and use it in a js file, but forget
to add the message to the module. Then later you add the msg
key. This last update would be ignored, because the mtime of
the message is no newer than the total which already included
the (same) mtime of the js file added in the previous update.
* Change the concatenation order of pages in a Gadget definition.
Bug: 37812
Change-Id: I00cf086c981a84235623bf58fb83c9c23aa2d619
For now ResourceLoader will still internally deal with timestamps,
especially because they are easier to max() on the client side
(on the server we could make a hash of different hashes, on the
client that's a bit annoying).
However ResourceLoaderModule#getHashMtime will abstract the logic
we already use in ResourceLoaderLanguageDataModule and have
encouraged others to use, which is to put the timestamp and
hash in Memcached, and use that to observer the hash change and
update the timestamp when it does.
Updated ResourceLoaderLanguageDataModule to make use of this.
Change-Id: Ib051ef41ba239084671c30fd275b8d94099d5d52
A reworked version of script from I068686854ad79e2f63a08d81b1af02f373110613
Move leccs instantiation code to ResourceLoader because it doesn't depend on any
particular module's state.
Change-Id: I733b53171dca77f50a30e5bd0bd5f1b456e4c85d
Follows-up b67b9e1, lessphp now has a public method to add to the
list of files for compilation cache.
Change-Id: I62a6c7cdeb34ea742fa7547e3ca10e24ee484b97
