This effectively applies safemode to the mw.loader client,
without the client itself needing specific knowledge of safemode.
Test Plan:
* Unchanged: When viewing a page in safemode, the 'user' and
'site' modules are still not queued by OutputPage.
* New: mw.loader.getState('site'), previously would yield
'registered', but will now yield null.
* New: mw.loader.load('site'), previously loaded the module,
it now logs a dependency warning for unknown module, like for
any other unknown module.
* New: mw.loader.using('site'), previously resolved, it is now
rejected.
Bug: T185303
Change-Id: I672e3891c8e1b3c2d13655fa134d0f1d031b8247
* Remove left-over mention of the .php5 entry points in docs.
* Remove dead logic in NoLocalSettings for php5 entry points.
* Remove dead match in WebRequest for php5 entry points (they'd
redirect since 1.25, and not seen by PHP).
Change-Id: Ia0ee8588591860b8fe34030c8503f38e9bce31f3
The siteinfo API response's 'writeapi' value is now hard-set to true,
as are the ResourceLoader variables wgEnableAPI and wgEnableWriteAPI,
to be deprecated later.
Bug: T115414
Change-Id: I54ff9428b247ba203d67aba079149393f323d5a9
For the same reasons as in ResourceLoaderContext. The only valid values
here are known strings like "desktop" and "mobile". No Language/UTF
normalisation required.
Ensures that Language instance won't get initialised just for this.
Change-Id: If219463d80a66132786710184329388080fbe713
For unfortunate historical reasons, browsers' native maxlength counts
the number of UTF-16 code units rather than Unicode codepoints [1],
which means that codepoints outside the Basic Multilingual Plane
(e.g. many emojis) count as 2 characters each. That could be good
enough, but we want our software to be excellent rather than merely
good enough.
1. Introduce a few new functions, added to the existing modules:
* mediawiki.String:
* codePointLength() counts the length of a string in Unicode
codepoints (characters).
* trimCodePointLength() trims a string to the desired length in
Unicode codepoints (characters).
* jquery.lengthLimit:
* $.fn.codePointLimit() enforces the specified maximum length in
codepoints of an input field.
* mediawiki.widgets.visibleLengthLimit:
* mw.widgets.visibleCodePointLimit() enforces the maximum length
in codepoints of an OOUI widget and displays the remaining
length in an inline label.
2. Add client-side mw.config variables:
* wgCommentByteLimit for the old byte limit, equal to 255.
* wgCommentCodePointLimit for the new codepoint limit, equal to 1000.
Only one of them may be set, depending on which limit should be applied.
3. Make use of both of these new features. For the sake of an example,
I updated the forms shown on action=edit (using visibleCodePointLimit)
and on action=protect (using codePointLimit). Many remain to be updated.
[1] https://www.w3.org/TR/html5/sec-forms.html#limiting-user-input-length-the-maxlength-attribute
Bug: T185948
Change-Id: Ia1269fd898dabbcf1582618eab46cef97e10a3b1
This allows browsers that support W3C Preload <https://www.w3.org/TR/preload/>,
to start downloading and parsing the second load.php JavaScript request
as soon as the headers of the startup module arrive.
Before:
- HTML request, response start.
|\- Discover <script src async> for startup.js, request, response start.
| \ - Startup response end, JS parsing.
| |
|__|- HTML parsing and rendering yields for async JS execution.
\- JavaScript inserts <script>, Base-modules request starts. <<<
After:
- HTML request, response start.
|\- Discover <script src async> for startup.js, request, response start
\ - Base-modules request starts. <<<
| |- Startup response end, JS parsing.
| |
|__|- HTML parsing and rendering yields for async JS execution.
\- JavaScript inserts <script>. (Base-modules req on-going, or done)
On local testing with "Slow 3G" network throttling in Chrome,
this reduced page load time (window.onload, time to interaction for JS)
by about 1 second (from 14s to 13s).
See also https://phabricator.wikimedia.org/T164299#3572231
Bug: T164299
Change-Id: I7047f4ab881947cf3392256087cc5a0cb177dd3a
Follows-up 0ac4f998 (restore "blocking" legacy modules).
After d790562, legacy modules in the top queue were no longer consistently
loaded before the bottom queue due to the top queue being async.
The implied dependency was made explicit by 0ac4f998 by forcing all modules
to wait for legacy modules before executing.
This had the negative side-effect of putting an extra HTTP request between
the startup module request, base modules request, and actual execution
of page modules.
(Indentation aligns with when a request is triggered.)
Before:
1. Request: Startup module.
2. Request: Base modules
3. Request: Legacy modules
4. Page module request (or local store hit) and execution
After:
1. Request: Startup module.
2. Request: Base+legacy modules
3. Page module request (or local store hit) and execution
This could alternatively be fixed by moving the top queue to be before
the embedded modules and enforcing the embed in a different way.
It could also be fixed by debouncing module load calls so they naturally
end up in the same request as page modules.
However for now I'm addressing this by adding legacy modules to the
list of modules in the initial load request from the startup module.
This was not possible before because the legacy wikibits had dependencies
and base modules cannot have dependencies. Fixed in I7f9f61ea81ad1ef.
Bug: T159911
Change-Id: I54f087655e1cde1b8ff1ca5fe56e82f7f7d80965
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.
Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
When getScript (or some other method used in a module response)
throws an error, only that module fails (by outputting mw.loader.state
instead of mw.loader.implement). Other modules will work.
This has always been the case and is working fine. For example,
"load.php?modules=foo|bar", where 'foo' throws, will return:
```js
/* exception message: .. */
mw.loader.implement('bar', ..)
mw.loader.state('foo', 'error')
```
The problem, however, is that during the generation of the startup
module, we iterate over all other modules. In 2011, the
getVersionHash method (then: getModifiedTime) was fairly simple
and unlikely to throw errors.
Nowadays, some modules use enableModuleContentVersion which will
involve the same code path as for regular module responses.
The try/catch in ResourceLoader::makeModuleResponse() suffices
for the case of loading modules other than startup. But when
loading the startup module, and an exception happens in getVersionHash,
then the entire startup response is replaced with an exception comment.
Example case:
* A file not existing for a FileModule subclass that uses
enableModuleContentVersion.
* A database error from a data module, like CiteDataModule or
CNChoiceData.
Changes:
* Ensure E-Tag is still useful while an error happens in production
because we respond with 200 OK and one error isn't the same as
another.
Fixed by try/catch in getCombinedVersion.
* Ensure start manifest isn't disrupted by one broken module.
Fixed by try/catch in StartupModule::getModuleRegistrations().
Tests:
* testMakeModuleResponseError: The case that already worked fined.
* testMakeModuleResponseStartupError: The case fixed in this commit.
* testGetCombinedVersion: The case fixed in this commit for E-Tag.
Bug: T152266
Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
makeVersionQuery and getCombinedVersion return the same string in
most cases, but there are exceptions, and it could diverge further
in the future. Use the semantically correct method.
Before 6fa1e56 it didn't matter how 'version' was computed as long
as it's deterministic and sufficiently unique. Now that we validate
the hashes it's important all methods use the same logic.
Rename method to makeVersionQuery since it's no longer used just
for comparison against the expected value.
Change-Id: I19f5818e27c8a0920d3d1374b40aeb0b6ba0b614
Avoid manually crafting a url. The packing logic and key sorting logic
is already in this method and keeps it central that way.
Test plan:
* Check output of /w/load.php?modules=startup&only=scripts.
This is used to construct the url to 'jquery%2Cmediawiki'.
No change in output.
Change-Id: I51ba51b6e74d78761108341c0ee318699ca7952f
mw.Title.newFromFileName() will now correctly use $wgIllegalFileChars,
rather than partially and incorrectly duplicating it.
$wgIllegalFileChars is currently defined to contain ':', '/' and '\'.
Previously we incorrectly did not filter out backslashes '\'. The hash
'#' is already disallowed by $wgLegalTitleChars.
Bug: T140222
Change-Id: I17e3ef61e04ae002ef576c9416c8f265ee000e89
$wgForceUIMsgAsContentMsg forces Message::inContentLanguage() to not
actually do anything, thus falling back to the session user's language
which we can't use in ResourceLoader. And 'mainpage' is added to that
array on various multi-lingual wikis.
Bug: T127233
Change-Id: I9dc1e57922641881b36c70658083c2c8c8a608a0
This was always false since introduction in 3c72b527 released in MediaWiki
1.19, and deprecated in 0ac4f998, released as part of MediaWiki 1.26. Any
code that still needs the mediawiki.util module can continue to use it via
expressing this dependency through their ResourceLoader manifest as usual.
Bug: T111077
Change-Id: Ic838af8727476c047f01ef0dbbeb952c85e263e1
We don't want it to accidentally end up in MediaWiki 1.27 release.
It can be restored again when needed.
This reverts commit d0e47d475c.
Bug: T120867
Change-Id: Ie1a90ad2d2ccdecf189313e18c3c5b24576363f4
It turns out that people click the checkbox affirming that they are
they author of the file and that they release it under CC BY-SA even
when neither of those is true. So we're trying some interfaces that
require a modicum of thought rather than just a click on "I agree".
Option 1: The form we have right now, with a single checkbox.
Option 2: Four checkboxes, each with a label explaining one facet of
the requirements (own work; no pictures of copyrighted work;
educational/useful; irrevocably released as CC BY-SA).
Option 3: Some Yes/No questions structured so that 'Yes' is not
always the right answer to continue uploading.
Option 4: Longer introduction before a single checkbox (as in option
1), with examples of good and unacceptable content.
As only logged in users are able to upload files, we're able to bucket
them into four groups by user ID number. When the user completes a
file upload, the bucket number is saved server-side in a change tag by
the companion patch I90cb12c505b2581f36113ec6b4f7bf732f0971b7 (we could
match the user IDs cross-wiki by username, but that sounds painful).
For testing and debugging, add '?uploadbucket=N' to the URL to force
given interface option to appear. Any completed upload won't count
towards the bucket.
Note that for expediency, the tested options all assume uploads to
'shared' repository (that is, Wikimedia Commons). The winner's
messages will be tweaked to work with 'local' and other targets too.
This patch DOES NOT ENABLE THE TEST yet, it just implements the options.
Enabling it on specific wikis can be done via config:
* $wgForeignUploadTestEnabled = true/false (defaults to 'false')
Whether the test is running.
* $wgForeignUploadTestDefault = 1/2/3/4 (defaults to '1')
Interface to use when the test is not running (and for anons).
Bug: T120867
Bug: T121021
Change-Id: I557056b867c6a55ef2c9af321eb48893312632a3
* Add logger to ResourceLoaderContext for convenient use within modules.
* Group related members within ResourceLoaderContext.
Change-Id: Ifbc3de1b6e0838386735f1573df328d9b331ac37
Follows-up e9b9e4df7a, which started using target=test as reserved for
Special:JavaScriptTest. However that conflicted with this.
Without this change, the unit tests fail if the MediaWiki install has
$wgEnableJavaScriptTest enabled.
Change-Id: I471017e7aedb74dddb4dc18ee4e89c1706ff09b5
* The '.php5' entrypoints were deprecated in I68b1ae842, $wgScriptExtension
in I3690f78bc.
* Drop the associated ResourceLoader configuration variable, too. `mwgrep`
shows no usage in the MediaWiki namespace.
* Keep the scriptExtension configuration parameter for FileRepo for people who
would like to interoperate with older MediaWiki installations that still use
'.php5'.
Change-Id: I17c8a15484b7e82cd5970d34e688109a2aae3840
This seems like a (slightly) cleaner system. We'll need to change
some documentation and add some configuration, but at least this will
be easier to handle.
Bug: T114765
Change-Id: If962fd3066e25e43e745efd29058eae82195bfb1
* Add support for a '/* @nomin */' annotation in ResourceLoader. If present in
JavaScript or CSS, the code will not be minified or cached. This allows
modules like the ResourceLoaderUserTokensModule to declare themselves unfit
for minification / caching without requiring a complicated refactor.
* Make ResourceLoader::filter() static, at the cost of not having minifier
errors in the ResourceLoader log bucket. (They will continue to be logged as
exceptions, however).
Change-Id: Ic1d802ee20565e61046bfbd8fd209bc56a4cbd6c
This ensures 'wikibits' and 'mediawiki.util' (if so configured)
finish loading before executing other modules.
This was previously implicitly provided for the bottom queue by
being in the top-queue, but since this is now asynchronous that
is no lower guaranteed. Fix this by making this explicit instead
of implicit.
Keep them in the top queue as-is to ensure consistency with cached
pages and it allows them to preload and batch in the same request
instead of being discovered later at run-time as a separate request.
Bug: T108124
Change-Id: I74e0cbe616404da927ea46d06308a7bae930eb69
Page startup:
* Due to the startup module and top queue being asynchronous now,
move client-nojs/client-js class handling to OutputPage to ensure
there is no flashes of wrongly styled or unstyled content.
To preserve compatibility for unsupported browsers, undo the
class swap at runtime after the isCompatible() check.
ResourceLoader startup module:
* Load the startup module with <script async>.
* Use DOM methods instead of 'document.write' to create base module request (jquery|mediawiki).
mw.loader:
* Drop 'async' parameter from mw.loader.load().
* Remove the now-unused code paths for synchronous requests.
OutputPage:
* Drop '$loadCall' parameter from makeResourceLoaderLink().
Asynchronous is now the default and only way to load JavaScript.
This means the 'user' module "conditional document-write scripts"
are now a simple "mw.loader.load( url )" call.
* Fix incorrect @return of makeResourceLoaderLink(). This returns
an array not a string.
* Improve documentation of makeResourceLoaderLink().
* Drop '$inHead' parameter from getScriptsForBottomQueue(). No longer used.
Compatibility with the $wgResourceLoaderExperimentalAsyncLoading
feature is maintained. It just no longer needs to change the
way the queue works since it's always asynchronous. The feature
flag now only controls whether the bottom queue starts at the bottom
or starts at the top.
* Remove jQuery.ready() optimisation.
This was mostly there to avoid the setTimeout() loop jQuery does to detect
dom-ready in IE6/IE7 (which we no longer serve JavaScript at all).
And for a bug in Firefox with document.write (which is no longer used as of
this commit).
Bug: T107399
Change-Id: Icba6d7a87b239bf127a221bc6bc432cfa71a4a72
To make unit testing easier, allow any module to be loaded within
the unit test suite. Regardless of the intended 'target'.
Targets are meant for restricting front-end scope in production.
Enforcing that in the test suite causes various test suites to
get de-registered at run time client-side.
Otherwise, in order to truly run all unit tests, Jenkins would
have to re-run the entire test suite in all known targets. This
wouldn't make sense because modules have to be globally uniquely
named (no conflicts) and unit tests are atomic. They can all run
in the same suite.
To prevent modules being comitted with incompatible target
dependencies, we already have a Structure test in the PHPUnit
suite to catch those issues in the module registry.
This makes the main 'qunit' build for MobileFrontend more useful,
where currently many modules aren't being tested due to them not
being in the 'desktop' target.
Bug: T103027
Change-Id: I69f735eb56c1362189298d9859d3add576faaadb
Most wikis only use user signatures on pages set aside from discussion,
(Talk namespaces and sometimes project/main namespaces depending on
the wiki), so having the button available everywhere is confusing.
The few wikis that need the button (especially non-content,
internal/corporate/planning wikis), can add relevant namespaces to the
$wgExtraSignatureNamespaces array in LocalSettings.
This would make it possible to solve bugs like T59727 or T53154.
Since this is a change to default behavior, a release note is added.
Bug: T7645
Change-Id: I7ccf1093b888c7b33721234349ca0ac054c3cd3f
The code is easier to maintain in an actual JavaScript file.
Especially with how variables were declared and concatenated in
a different order.
Change-Id: I758acb78de1cdf2128e81c86f992807ef0dbf444
- Removed space after casts
- Removed spaces in array index
- Added spaces around string concat
- Added space after words: switch, foreach
- else if -> elseif
- Removed parentheses around require_once, because it is not a function
- Added newline at end of file
- Removed double spaces
- Added spaces around operations
- Removed repeated newlines
Bug: T102609
Change-Id: Ib860222b24f8ad8e9062cd4dc42ec88dc63fb49e
By providing context as a parameter in getDependencies, we allow
modules to dyanamically determine dependencies based on context.
Note: To ease rollout, the parameter is optional in this patch. It is expected
that it will be made non-optional in the near future.
The use case is for CentralNotice campaigns to be able to add special
modules ahead of deciding which banner to show a user. The dynamically
chosen RL modules would replace ad-hoc JS currently sent with some banners.
A list of possible campaigns and banners is already sent as a PHP-
implemented RL module; that's the module that will dynamically choose other
modules as dependencies when appropriate. This approach will save a round
trip as compared to dynamically loading the modules client-side.
For compatibility, extensions that override
ResourceLoaderModule::getDependencies() should be updated with the new
method signature. Here are changes for extensions currently deployed on
Wikimedia wikis:
* CentralNotice: I816bffa3815e2eab7e88cb04d1b345070e6aa15f
* Gadgets: I0a10fb0cbf17d095ece493e744296caf13dcee02
* EventLogging: I67e957f74d6ca48cfb9a41fb5144bcc78f885e50
* PageTriage: Ica3ba32aa2fc76d11a44f391b6edfc871e7fbe0d
* UniversalLanguageSelector: Ic63e617f51702c27104e123d4bed91983a726b7f
* VisualEditor: I0ac775ca286e64825e31a9213b94648e41a5bc30
For more on the CentralNotice use case, please see I9f80edcbcacca2.
Bug: T98924
Change-Id: Iee61e5b527321d01287baa03ad9b4d4f526ff3ef
As of b1e4006b4, the tokens are different on every request.
Caching these is completely useless because the cache entry is
simply unreachable and is extra overhead on every request for
logged-in users to save content to Memcached.
Whether they should be minified at all and whether they perhaps
shouldn't change on every request is a separate matter.
Bug: T84960
Change-Id: I6016e4b01e44e0acbfd6d49f7d99555e2290c9cb
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
