The startup module varies by skin, so it should be possible for
consumers of the ResourceLoaderGetConfigVars hook to know what
skin it being requested.
Bug: T186062
Change-Id: I97d6b4bc245bc64ff148c3665b46116b8a6be409
Follows-up dc3fc6cf81, which documented the reasoning for the
specific json flags in StartupModule. In wanting to re-use them
in a different module it became apparant that perhaps it was a
bit too conservative in only allowing the flags to be used in
a script HTTP response.
Lax the contract a little bit (that is, do more escaping) to also
allow safe re-use in HTML context.
I considered making these separate methods (e.g. forScriptResponse,
and forInlineScript) but decided not to because the vast majority
of callers would have to be forInlineScript eventhough the code
in question had no responsiblity or knowledge of it becoming an
inline script, because ResourceLoader allows most modules to
become embedded if they support a private or preview mode.
Those modes are implemented by calling makeModuleResponse, and
wrapping it an inline script. It seems appealing and simplifying
to the API to require that script output is always safe for
embedding rather than complicating the API for winning back
a literal handful of bytes in the edge case that a user-generated
string contains a '<' and was not embedded. I estimate that with
gzip, it will literally save only a single byte, even if used
multiple times. Let's focus optimisation efforts elsewhere :)
Change-Id: I7742dabba6750deecf6fbf51cf9a77ee8cbfc727
This significantly simplifies the getVersionHash implementation for
StartupModule, and fixes a couple of bugs.
Previously, the startup module's E-Tag was determined by the
'getDefinitionSummary' method, which combined the E-Tag values
from all registered modules, plus what we thought is all information
used by 'getScript' (config vars, embedded script files, list
of base modules, ...)
However, this were various things part of the manifest that it
forgot about, including:
* Changes to the list of dependencies of a module.
* Changes to the name of module.
* Changes to the cache group of module.
* Adding or removing a foreign module source (mw.loader.addSource).
These are all quite rare, and when they do change, they usually
also involve a change that *was* tracked already. But, sometimes
they don't and that's when bugs happened.
Instead of the tracking array of getDefinitionSummary, we now
use the 'enableModuleContentVersion' option for StartupModule,
which simply calls the actual getScript() method and hashes that.
Of note: When an exception happens with the version computation of
any individual module, we catch it, log it, and continue with the
rest. Previously, the first time such error was discovered at
run-time would be in the getCombinedVersion() call from
StartupModule::getAllModuleHashes(). That public getCombinedVersion()
method of ResourceLoader had the benefit of also outputting details
of that exception in the HTTP response output. In order to keep that
behaviour, I made outputErrorAndLog() public so that StartupModule
can call it directly now. This is covered by
ResourceLoaderTest::testMakeModuleResponseStartupError.
Bug: T201686
Change-Id: I8e8d3a2cd2ccd68d2d78e988bcdd0d77fbcbf1d4
* Remove use of MediaWiki's FormatJson class. Ref T32956.
* FormatJson::decode() was a direct alias of json_decode.
* FormatJson::encode() is a wrapper for json_encode that
sets `JSON_UNESCAPED_SLASHES | JSON_HEX_TAG | JSON_HEX_AMP`
by default, plus a boolean parameter to add JSON_PRETTY_PRINT.
Instead, use json_encode() directly. By default json_encode() escapes
slashes and non-ASCII unicode.
* Audit our uses of JSON encoding, document their needs, and set flags
as needed.
* Remove $mapToJson abstraction from ResourceLoaderStartUpModule.
* Always pretty-print the list of module names in the error message,
regardless of debug mode.
* Remove hacky indentation-adder from ResourceLoaderStartUpModule.
This was relying on there not being line breaks in the json output
outside debug mode, and adding a level of indentation to roughly
match the code in the JSON is substituted. This didn't match and
just generally doesn't seem worth it.
Change-Id: I3e09ddeb4d53c8980195c1855303c0ee25bd4c20
When enabling $wgResourceLoaderEnableJSProfiler, mw.loader gets instrumented
with the following timing values for each of the modules loaded on the page:
* 'total' - This measures the time spent in mw.loader#execute(), and
represents the initialisation of the module's implementation, including
the registration of messages, templates, and the execution of the 'script'
closure received from load.php.
* 'script' – This measures only the subset of time spent in the internal
runScript() function, and represents just the execution of the module's
JavaScript code as received through mw.loader.implement() from load.php.
For user scripts and site scripts, this measures the call to domEval
(formerly known as "globalEval").
* 'execute' - This measures the self time of mw.loader#execute(), which is
effectively `total - script`.
To view the report, enable the feature, then run `mw.inspect( 'time' )` from
the browser console, which will render a table with the initialisation
overhead from each module used on the page.
Bug: T133646
Change-Id: I68d1193b62c93c97cf09b7d344c896afb437c5ac
This commit implements step 4 and step 5 of the plan outlined at T192623.
Before this task began, the typical JavaScript execution flow was:
* HTML triggers request for startup module (js req 1).
* Startup module contains registry, site config, and triggers
a request for the base modules (js req 2).
* After the base modules arrive (which define jQuery and mw.loader),
the startup module invokes a callback that processes RLQ,
which is what will request modules for this page (js req 3).
In past weeks, we have:
* Made mediawiki.js independent of jQuery.
* Spun off 'mediawiki.base' from mediawiki.js – for everything
that wasn't needed for defining `mw.loader`.
* Moved mediawiki.js from the base module request to being embedded
as part of startup.js.
The concept of dependencies is native to ResourceLoader, and thanks to the
use of closures in mw.loader.implement() responses, we can download any
number of interdependant modules in a single request (or parallel requests).
Then, when a response arrives, mw.loader takes care to pause or resume
execution as-needed. It is normal for ResourceLoader to batch several modules
together, including their dependencies.
As such, we can eliminate one of the two roundtrips required before a
page can request modules. Specifically, we can eliminate "js req 2" (above),
by making the two remaining base modules ("jquery" and "mediawiki.base") an
implied dependency for all other modules, which ResourceLoader will naturally
fetch and execute in the right order as part of the batch request.
Bug: T192623
Change-Id: I17cd13dffebd6ae476044d8d038dc3974a1fa176
Embed the essential files to define mw.loader directly as part of
the startup module.
* This means the internal 'mediawiki' module no longer exists.
This is safe to remove because:
1) While registered server-side for loading from startup.js, a PHPUnit
structure test disallowed being specified as a dependency.
2) Anything that attempted to load it client-side failed because the
module was marked in the registry as 'raw', thereby excluding it
from the data sent to the client-side. As such, it was seen as an
unknown module that the client refused to fetch from the server.
* Deprecate getStartupModules() and getLegacyModules().
These are no longer needed. There are no known callers anywhere in
Wikimedia Git or elsewhere indexed by Codesearch, but easy enough
to leave as no-op for one release.
* Remove ResourceLoaderRawFileModule class.
No longer needed. Was created as a hack specifically for the 'mediawiki'
module so that it would not leak global variables in debug mode.
It has no usage anywhere in Wikimedia Git, nor elsewhere in Codesearch.
Remove without deprecation given this was meant to be a 'private' class.
* Introduce (private) getBaseModules(). Previously, this list only existed
locally in getStartupModulesUrl() by merging getStartupModules() and
getLegacyModules(). This value was factored out into its own method.
* Make getStartupModulesUrl() private and rename to getBaseModulesUrl().
It is only used internally to export the 'baseModulesUri' value.
Its name was already confusing before, but it would've been even more
confusing now given it doesn't even call getStartupModules() any more.
Bug: T192623
Change-Id: I14ba282d7b65e99ca54b7c2f77ba6e1adaddd11c
This was overlooked as part of the whole shift from timestamps
to file hashes (T94074, f37cee996e, T104950, 28f6d7fbde).
The key name and method call here matches that of FileModule.
The impact of this still using mtimes was fairly low, given:
* The startup module is only cached for 5 minutes.
* The startup module's hash varies on "everything" which includes
wgVersion, as such, when a new branch is cut and cloned and this file's
timestamp is reset, there will still be other factors causing the
overall hash to vary. As such, this change is unlikely to improve
200-304 response ratio.
Change-Id: I6543fb75575e9a793a7fc93e15d2f3e0b5a04342
The startup.js template used '$CODE.registrations();', which makes
sense syntatically (and for the linter), but the substitution logic
was only replacing the '$CODE.registrations()' portion.
This made the generated output contain two consecutive semi-colons.
Change-Id: Ibef1a0d932b19037987a7effa12aa57f578a142d
This effectively applies safemode to the mw.loader client,
without the client itself needing specific knowledge of safemode.
Test Plan:
* Unchanged: When viewing a page in safemode, the 'user' and
'site' modules are still not queued by OutputPage.
* New: mw.loader.getState('site'), previously would yield
'registered', but will now yield null.
* New: mw.loader.load('site'), previously loaded the module,
it now logs a dependency warning for unknown module, like for
any other unknown module.
* New: mw.loader.using('site'), previously resolved, it is now
rejected.
Bug: T185303
Change-Id: I672e3891c8e1b3c2d13655fa134d0f1d031b8247
* Remove left-over mention of the .php5 entry points in docs.
* Remove dead logic in NoLocalSettings for php5 entry points.
* Remove dead match in WebRequest for php5 entry points (they'd
redirect since 1.25, and not seen by PHP).
Change-Id: Ia0ee8588591860b8fe34030c8503f38e9bce31f3
The siteinfo API response's 'writeapi' value is now hard-set to true,
as are the ResourceLoader variables wgEnableAPI and wgEnableWriteAPI,
to be deprecated later.
Bug: T115414
Change-Id: I54ff9428b247ba203d67aba079149393f323d5a9
For the same reasons as in ResourceLoaderContext. The only valid values
here are known strings like "desktop" and "mobile". No Language/UTF
normalisation required.
Ensures that Language instance won't get initialised just for this.
Change-Id: If219463d80a66132786710184329388080fbe713
For unfortunate historical reasons, browsers' native maxlength counts
the number of UTF-16 code units rather than Unicode codepoints [1],
which means that codepoints outside the Basic Multilingual Plane
(e.g. many emojis) count as 2 characters each. That could be good
enough, but we want our software to be excellent rather than merely
good enough.
1. Introduce a few new functions, added to the existing modules:
* mediawiki.String:
* codePointLength() counts the length of a string in Unicode
codepoints (characters).
* trimCodePointLength() trims a string to the desired length in
Unicode codepoints (characters).
* jquery.lengthLimit:
* $.fn.codePointLimit() enforces the specified maximum length in
codepoints of an input field.
* mediawiki.widgets.visibleLengthLimit:
* mw.widgets.visibleCodePointLimit() enforces the maximum length
in codepoints of an OOUI widget and displays the remaining
length in an inline label.
2. Add client-side mw.config variables:
* wgCommentByteLimit for the old byte limit, equal to 255.
* wgCommentCodePointLimit for the new codepoint limit, equal to 1000.
Only one of them may be set, depending on which limit should be applied.
3. Make use of both of these new features. For the sake of an example,
I updated the forms shown on action=edit (using visibleCodePointLimit)
and on action=protect (using codePointLimit). Many remain to be updated.
[1] https://www.w3.org/TR/html5/sec-forms.html#limiting-user-input-length-the-maxlength-attribute
Bug: T185948
Change-Id: Ia1269fd898dabbcf1582618eab46cef97e10a3b1
This allows browsers that support W3C Preload <https://www.w3.org/TR/preload/>,
to start downloading and parsing the second load.php JavaScript request
as soon as the headers of the startup module arrive.
Before:
- HTML request, response start.
|\- Discover <script src async> for startup.js, request, response start.
| \ - Startup response end, JS parsing.
| |
|__|- HTML parsing and rendering yields for async JS execution.
\- JavaScript inserts <script>, Base-modules request starts. <<<
After:
- HTML request, response start.
|\- Discover <script src async> for startup.js, request, response start
\ - Base-modules request starts. <<<
| |- Startup response end, JS parsing.
| |
|__|- HTML parsing and rendering yields for async JS execution.
\- JavaScript inserts <script>. (Base-modules req on-going, or done)
On local testing with "Slow 3G" network throttling in Chrome,
this reduced page load time (window.onload, time to interaction for JS)
by about 1 second (from 14s to 13s).
See also https://phabricator.wikimedia.org/T164299#3572231
Bug: T164299
Change-Id: I7047f4ab881947cf3392256087cc5a0cb177dd3a
Follows-up 0ac4f998 (restore "blocking" legacy modules).
After d790562, legacy modules in the top queue were no longer consistently
loaded before the bottom queue due to the top queue being async.
The implied dependency was made explicit by 0ac4f998 by forcing all modules
to wait for legacy modules before executing.
This had the negative side-effect of putting an extra HTTP request between
the startup module request, base modules request, and actual execution
of page modules.
(Indentation aligns with when a request is triggered.)
Before:
1. Request: Startup module.
2. Request: Base modules
3. Request: Legacy modules
4. Page module request (or local store hit) and execution
After:
1. Request: Startup module.
2. Request: Base+legacy modules
3. Page module request (or local store hit) and execution
This could alternatively be fixed by moving the top queue to be before
the embedded modules and enforcing the embed in a different way.
It could also be fixed by debouncing module load calls so they naturally
end up in the same request as page modules.
However for now I'm addressing this by adding legacy modules to the
list of modules in the initial load request from the startup module.
This was not possible before because the legacy wikibits had dependencies
and base modules cannot have dependencies. Fixed in I7f9f61ea81ad1ef.
Bug: T159911
Change-Id: I54f087655e1cde1b8ff1ca5fe56e82f7f7d80965
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.
Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
When getScript (or some other method used in a module response)
throws an error, only that module fails (by outputting mw.loader.state
instead of mw.loader.implement). Other modules will work.
This has always been the case and is working fine. For example,
"load.php?modules=foo|bar", where 'foo' throws, will return:
```js
/* exception message: .. */
mw.loader.implement('bar', ..)
mw.loader.state('foo', 'error')
```
The problem, however, is that during the generation of the startup
module, we iterate over all other modules. In 2011, the
getVersionHash method (then: getModifiedTime) was fairly simple
and unlikely to throw errors.
Nowadays, some modules use enableModuleContentVersion which will
involve the same code path as for regular module responses.
The try/catch in ResourceLoader::makeModuleResponse() suffices
for the case of loading modules other than startup. But when
loading the startup module, and an exception happens in getVersionHash,
then the entire startup response is replaced with an exception comment.
Example case:
* A file not existing for a FileModule subclass that uses
enableModuleContentVersion.
* A database error from a data module, like CiteDataModule or
CNChoiceData.
Changes:
* Ensure E-Tag is still useful while an error happens in production
because we respond with 200 OK and one error isn't the same as
another.
Fixed by try/catch in getCombinedVersion.
* Ensure start manifest isn't disrupted by one broken module.
Fixed by try/catch in StartupModule::getModuleRegistrations().
Tests:
* testMakeModuleResponseError: The case that already worked fined.
* testMakeModuleResponseStartupError: The case fixed in this commit.
* testGetCombinedVersion: The case fixed in this commit for E-Tag.
Bug: T152266
Change-Id: Ice4ede5ea594bf3fa591134bc9382bd9c24e2f39
makeVersionQuery and getCombinedVersion return the same string in
most cases, but there are exceptions, and it could diverge further
in the future. Use the semantically correct method.
Before 6fa1e56 it didn't matter how 'version' was computed as long
as it's deterministic and sufficiently unique. Now that we validate
the hashes it's important all methods use the same logic.
Rename method to makeVersionQuery since it's no longer used just
for comparison against the expected value.
Change-Id: I19f5818e27c8a0920d3d1374b40aeb0b6ba0b614
Avoid manually crafting a url. The packing logic and key sorting logic
is already in this method and keeps it central that way.
Test plan:
* Check output of /w/load.php?modules=startup&only=scripts.
This is used to construct the url to 'jquery%2Cmediawiki'.
No change in output.
Change-Id: I51ba51b6e74d78761108341c0ee318699ca7952f
mw.Title.newFromFileName() will now correctly use $wgIllegalFileChars,
rather than partially and incorrectly duplicating it.
$wgIllegalFileChars is currently defined to contain ':', '/' and '\'.
Previously we incorrectly did not filter out backslashes '\'. The hash
'#' is already disallowed by $wgLegalTitleChars.
Bug: T140222
Change-Id: I17e3ef61e04ae002ef576c9416c8f265ee000e89
$wgForceUIMsgAsContentMsg forces Message::inContentLanguage() to not
actually do anything, thus falling back to the session user's language
which we can't use in ResourceLoader. And 'mainpage' is added to that
array on various multi-lingual wikis.
Bug: T127233
Change-Id: I9dc1e57922641881b36c70658083c2c8c8a608a0
This was always false since introduction in 3c72b527 released in MediaWiki
1.19, and deprecated in 0ac4f998, released as part of MediaWiki 1.26. Any
code that still needs the mediawiki.util module can continue to use it via
expressing this dependency through their ResourceLoader manifest as usual.
Bug: T111077
Change-Id: Ic838af8727476c047f01ef0dbbeb952c85e263e1
We don't want it to accidentally end up in MediaWiki 1.27 release.
It can be restored again when needed.
This reverts commit d0e47d475c.
Bug: T120867
Change-Id: Ie1a90ad2d2ccdecf189313e18c3c5b24576363f4
It turns out that people click the checkbox affirming that they are
they author of the file and that they release it under CC BY-SA even
when neither of those is true. So we're trying some interfaces that
require a modicum of thought rather than just a click on "I agree".
Option 1: The form we have right now, with a single checkbox.
Option 2: Four checkboxes, each with a label explaining one facet of
the requirements (own work; no pictures of copyrighted work;
educational/useful; irrevocably released as CC BY-SA).
Option 3: Some Yes/No questions structured so that 'Yes' is not
always the right answer to continue uploading.
Option 4: Longer introduction before a single checkbox (as in option
1), with examples of good and unacceptable content.
As only logged in users are able to upload files, we're able to bucket
them into four groups by user ID number. When the user completes a
file upload, the bucket number is saved server-side in a change tag by
the companion patch I90cb12c505b2581f36113ec6b4f7bf732f0971b7 (we could
match the user IDs cross-wiki by username, but that sounds painful).
For testing and debugging, add '?uploadbucket=N' to the URL to force
given interface option to appear. Any completed upload won't count
towards the bucket.
Note that for expediency, the tested options all assume uploads to
'shared' repository (that is, Wikimedia Commons). The winner's
messages will be tweaked to work with 'local' and other targets too.
This patch DOES NOT ENABLE THE TEST yet, it just implements the options.
Enabling it on specific wikis can be done via config:
* $wgForeignUploadTestEnabled = true/false (defaults to 'false')
Whether the test is running.
* $wgForeignUploadTestDefault = 1/2/3/4 (defaults to '1')
Interface to use when the test is not running (and for anons).
Bug: T120867
Bug: T121021
Change-Id: I557056b867c6a55ef2c9af321eb48893312632a3
* Add logger to ResourceLoaderContext for convenient use within modules.
* Group related members within ResourceLoaderContext.
Change-Id: Ifbc3de1b6e0838386735f1573df328d9b331ac37
Follows-up e9b9e4df7a, which started using target=test as reserved for
Special:JavaScriptTest. However that conflicted with this.
Without this change, the unit tests fail if the MediaWiki install has
$wgEnableJavaScriptTest enabled.
Change-Id: I471017e7aedb74dddb4dc18ee4e89c1706ff09b5
* The '.php5' entrypoints were deprecated in I68b1ae842, $wgScriptExtension
in I3690f78bc.
* Drop the associated ResourceLoader configuration variable, too. `mwgrep`
shows no usage in the MediaWiki namespace.
* Keep the scriptExtension configuration parameter for FileRepo for people who
would like to interoperate with older MediaWiki installations that still use
'.php5'.
Change-Id: I17c8a15484b7e82cd5970d34e688109a2aae3840
This seems like a (slightly) cleaner system. We'll need to change
some documentation and add some configuration, but at least this will
be easier to handle.
Bug: T114765
Change-Id: If962fd3066e25e43e745efd29058eae82195bfb1
* Add support for a '/* @nomin */' annotation in ResourceLoader. If present in
JavaScript or CSS, the code will not be minified or cached. This allows
modules like the ResourceLoaderUserTokensModule to declare themselves unfit
for minification / caching without requiring a complicated refactor.
* Make ResourceLoader::filter() static, at the cost of not having minifier
errors in the ResourceLoader log bucket. (They will continue to be logged as
exceptions, however).
Change-Id: Ic1d802ee20565e61046bfbd8fd209bc56a4cbd6c
This ensures 'wikibits' and 'mediawiki.util' (if so configured)
finish loading before executing other modules.
This was previously implicitly provided for the bottom queue by
being in the top-queue, but since this is now asynchronous that
is no lower guaranteed. Fix this by making this explicit instead
of implicit.
Keep them in the top queue as-is to ensure consistency with cached
pages and it allows them to preload and batch in the same request
instead of being discovered later at run-time as a separate request.
Bug: T108124
Change-Id: I74e0cbe616404da927ea46d06308a7bae930eb69
