HTML formatting of the queue was distributed over several OutputPage methods.
Each method demanding a snippet of HTML by calling makeResourceLoaderLink()
with a limited amount of information. As such, makeResourceLoaderLink() was
unable to provide the client with the proper state information.
Centralising it also allows it to better reduce duplication in HTML output
and maintain a more accurate state.
Problems fixed by centralising:
1. The 'user' module is special (due to per-user 'version' and 'user' params).
It is manually requested via script-src. To avoid a separate (and wrong)
request from something that requires it, we set state=loading directly.
However, because the module is in the bottom, the old HTML formatter could
only put state=loading in the bottom also. This sometimes caused a wrong
request to be fired for modules=user if something in the top queue
triggered a requirement for it.
2. Since a464d1d4 (T87871) we track states of page-style modules, with purpose
of allowing dependencies on style modules without risking duplicate loading
on pages where the styles are loaded already. This didn't work, because the
state information about page-style modules is output near the stylesheet,
which is after the script tag with mw.loader.load(). That runs first, and
mw.loader would still make a duplicate request before it learns the state.
Changes:
* Document reasons for style/script tag order in getHeadHtml (per 09537e83).
* Pass $type from getModuleStyles() to getAllowedModules(). This wasn't needed
before since a duplicate check in makeResourceLoaderLink() verified the
origin a second time.
* Declare explicit position 'top' on 'user.options' and 'user.tokens' module.
Previously, OutputPage hardcoded them in the top. The new formatter doesn't.
* Remove getHeadScripts().
* Remove getInlineHeadScripts().
* Remove getExternalHeadScripts().
* Remove buildCssLinks().
* Remove getScriptsForBottomQueue().
* Change where Skin::setupSkinUserCss() is called. This methods lets the skin
add modules to the queue. Previously it was called from buildCssLinks(),
via headElement(), via prepareQuickTemplate(), via OutputPage::output().
It's now in OutputPage::output() directly (slightly earlier). This is needed
because prepareQuickTemplate() calls bottomScripts() before headElement().
And bottomScript() would lazy-initialise the queue and lock it before
setupSkinUserCss() is called from headElement().
This makes execution order more predictable instead of being dependent on
the arbitrary order of data extraction in prepareQuickTemplate (which varies
from one skin to another).
* Compute isUserModulePreview() and isKnownEmpty() for the 'user' module early
on so. This avoids wrongful loading and fixes problem 1.
Effective changes in output:
* mw.loader.state() is now before mw.loader.load(). This fixes problem 2.
* mw.loader.state() now sets 'user.options' and 'user.tokens' to "loading".
* mw.loader.state() now sets 'user' (as "loading" or "ready"). Fixes problem 1.
* The <script async src> tag for 'startup' changed position (slightly).
Previously it was after all inline scripts and stylesheets. It's still after
all inline scripts and after most stylesheets, but before any user styles.
Since the queue is now formatted outside OutputPage, it can't inject the
meta-ResourceLoaderDynamicStyles tag and user-stylesheet hack in the middle
of existing output. This shouldn't have any noticable impact.
Bug: T87871
Change-Id: I605b8cd1e1fc009b4662a0edbc54d09dd65ee1df
Follows-up ea862efa7, which enables use of Less_Cache as part
of the switch from leafo/lessphp to oyejorge/less.php.
Since then 12afb3607 introduced a better caching method in
FileModule::compileLessFile(), which:
* Uses APC where possible, with fallback to main objectcache.
* Verifies both the input file and the indirectly used files have
not changed before using the cache. The built-in cache system
only seems to check the root file.
As such, we're incurring the overhead of generating them and storing
on-disk, but raraly ever making use of them. (And even if we do
use it when the APC object first expires, it's risky due to the
lack of indirect file validation).
Bug: T121519
Change-Id: I6e3c66f86a536463488697efc166fb704be0715c
* Instead of having messy code to create a hidden HTML
comment of English strings at the bottom of the page,
expose the structured data of the parse information
to JS so tools can use it.
* Make makeConfigSetScript() use pretty output so these
variables are also easy to read in "view source".
* Remove ParserLimitReportFormat hook, since the data
is not formatted to HTML anymore.
Bug: T110763
Change-Id: I2783c46c6d80f828f9ecf5e71fc8f35910454582
SHA-1 is not secure enough to be used as a cryptographic hash function, and its
implementation in JavaScript is too long and too slow for it to be a good
general-purpose hash function. And we currently throw away most of the work:
SHA-1 produces 160-bit hash values, of which we keep 48.
Although the JavaScript implementation is not exported, SHA-1 is a well-known
hash function, and I'm willing to bet that sooner or later someone will move to
make it accessible to other modules, at which point usage will start to spread.
For ResourceLoader, the qualities we're looking for in a hash function are:
* Already implemented in PHP
* Easy to implement in JavaScript
* Fast
* Collision-resistant
The requirement that hashes be cheap to compute in JavaScript narrows the field
to 32-bit hash functions, because in JavaScript bitwise operators treat their
operands as 32 bits, and arithmetic uses double-precision floats, which have a
total precision of 53 bits. It's possible to work around these limitations, but
it's a lot of extra work.
The best match I found is the 32-bit variant of FNV-1, which is available in
PHP as of version 5.4 (as 'fnv1a32'). The fnv132 JavaScript function is
around ten times faster and eight times shorter than sha1.
Change-Id: I1e4fb08d17948538d96f241b2464d594fdc14578
* Send 'module' and 'require' parameters to module closures.
This depends on Ia925844cc22f143 being deployed one cycle earlier.
* Patch Moment and OOjs to ensure these libraries continue to expose
their module as globals as well. AMD/UMD-compatible libraries
only expose a global *OR* an export, not both. We need both
for back-compat.
* Update pluralRuleParser to make use of module export to allow
usage via require().
To test, check out the patch and run:
> mw.loader.load('moment');
> mw.loader.require('moment')()
> mw.loader.require('moment')('2011-04-01').fromNow()
Bug: T108655
Change-Id: Idbd054880ee70d659ec760aef8fcb38d0704a394
* Remove trailing space in self-closing tag.
Brings parserTest output of Parser and Parsoid closer together.
* Remove various line breaks at begin and end of script contents.
* Remove FILTER_NOMIN from makeConfigSetScript() output.
This isn't part of any user- or page-dependent module and not minified.
And Xml::encodeJsCall already ensures compact output for prod mode.
Bug: T127328
Change-Id: I85a5a59fd0955c1a112e8b24b933f0d9e983a156
* Fix errors spotted by new release
* Introduce "composer fix", which uses phpcbf to automatically fix some
errors spotted by phpcs.
* Drop $PHPCS_ARGS variable that didn't work on Windows, and add -s flag
* Remove rules from phpcs.xml that are now in MW-CS ruleset.
Change-Id: I13e2155695918c918b67497ac65b85a03897095e
startup.js:
* Don't assign to "window.RLQ" and read from "RLQ". Use the same
identifier in both places instead of relying on global scope.
* Create the global window.RLQ only once. The temporary [] fallback
doesn't need to be exposed globally, so use a local variable for
the loop that processes the pre-existing queue built up while
the startup module was loading.
OutputPage:
* Simplify the RLQ wrap to be more idiomatic and less repetitive.
Matches the pattern used in Google and Facebook open-source projects.
Change-Id: I9176377bc05432e51add841696a356428feec8ce
Cache Less variables in the instance instead of statically.
This allows tests to populate their own less variabless via ResourceLoaderLESSVars.
Make getLessCompiler() and getLessVars() regular public methods and
update callers.
Change-Id: I95506b8bb20a4b2b3f82014a7b0fcee5f28973e6
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
- Batch retrieval.
- Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
part of get() instead of within updateMessage().
ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
Timestamps are unreliable and roll over too frequently for message blob store
because there is no authoritative source. The timestamps were inferred based on
when a change is observed. Message overrides from the local wiki have an
explicit update event when the page is edited. All other messages, such as
from MediaWiki core and extensions using LocalisationCache, have a single
timestamp for all messages which rolls over every time the cache is rebuilt.
A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
Instead, detect whether a module's message list is empty at runtime.
ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.
cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.
Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
* Add logger to ResourceLoaderContext for convenient use within modules.
* Group related members within ResourceLoaderContext.
Change-Id: Ifbc3de1b6e0838386735f1573df328d9b331ac37
Follows-up e103ba265b.
This method doesn't need to be backward-compatible as this is already
normalised server-side.
When upgrading from before MediaWiki 1.24 the startup response will
change to the new format and also inline a request to the newer
version of mediawiki.js.
Also update method documentation to be up to date and remove
spurious return statements that aren't used.
Change-Id: I03d1b5666fcddedaa46ecf878ee7636042a5e3be
This table is not needed because module names and their messages
array are available to the runtime environment at very little cost.
The only purpose it was serving is reverse lookup from message
key to module name (e.g. when MessageCache receives update that
need to propagate to MessageBlobStore). However that is better
achieved by simply looping through modules in PHP. The overhead
of a database is not worth this minor convenience.
MessageBlobStore
* insertMessageBlob: Doesn't need to update msg_resource_links.
* updateModule: Doesn't need to update msg_resource_links.
* getUpdatesForMessage: Reimplement with list from memory
instead of msg_resource_links.
The database table will be removed in If009e2620e59002e1.
Bug: T113092
Change-Id: Ia9131f570001f00c9800b260ac4b3469d54d2784
I have not had the time and wherewithal to write the follow-up code that
actually uses this value. So just remove it for now.
This reverts commit 3d5aca4a40.
Change-Id: Ice9b65f39802b55b3d4ddf801c06d1be3227fd5b
My first pass at what became Ie3499d9f2 was to keep the exception for duplicate
registrations with conflicting parameters but downgrade to a warning if the
registration attempts are identically parametrized. I then removed the
exception entirely, but left the if-block, which does not make much sense, as
it means we pass over silently module registrations that set different
parameters, and throw a warning only when they are the same.
Change-Id: I13da33faf2f058ac75ef4b1615c4fafc18f14505
If an attempt is made to register a ResourceLoader module which has already
been registered, instead of bailing with an error, issue a warning instead, and
allow the lattermost set of module parameters to stick.
Because modules are registered from extensions, and because modules are often
migrated from one extension to another as extensions are refactored, and
because the developers of different extensions are not always aware of each
other's work, throwing an error runs a high risk of causing an outage.
Bug: T116628
Change-Id: Ie3499d9f2ff3bf071f1c2b0176e19440ff55211c
Use BagOStuff::makeKey() and BagOStuff::makeGlobalKey() instead of
wfMemcKey() and wfGlobalCacheKey().
Change-Id: Id4e58c0e616b74de220faf13ba7c9ea606fef8c1
Using the following command line, I have found doc comments (and
a wfDeprecated() call) mentioning "1.26" when they should mention
"1.27" instead, which I have fixed manually:
git diff -M REL1_26 | grep --color=always -C 10 -iP \
'^\+.*\D1\.26(\D|$)' | aha > oldver.html
Follows-up these commits:
* 047b60b96d
* 169b7b98b5
* 25a44aa3e4
* 2fb2a3f14b
* 3f1e9fa268 [1]
* 8c84af70b6
* c0cb80beac
* d04a92a551
* d3b85592ea
[1] Release notes moved in I195dd1cf.
Because a release note stating that the UserRights hook is deprecated
was added in 37062a0c0d (before the branch cut), this commit does not
change the Hooks::run call that had been changed in 21206c8fbe.
Change-Id: I5a427f003e7e3b4559fe377bcdfdca466a570708
This is a micro-optimization, but ResourceLoader is so well-optimized at this
point that the call to PCRE accounts for 1.25% of all load.php CPU time. So
might as well making it a tiny bit faster.
Change-Id: Iefab804a6ca6d54ce230958513a3bea44f4e7c62
* Add support for a '/* @nomin */' annotation in ResourceLoader. If present in
JavaScript or CSS, the code will not be minified or cached. This allows
modules like the ResourceLoaderUserTokensModule to declare themselves unfit
for minification / caching without requiring a complicated refactor.
* Make ResourceLoader::filter() static, at the cost of not having minifier
errors in the ResourceLoader log bucket. (They will continue to be logged as
exceptions, however).
Change-Id: Ic1d802ee20565e61046bfbd8fd209bc56a4cbd6c
* Decline to cache minified private modules, because they exist in as many
variants as there are users, and are unlikely to be cache hits now that we
use APC.
* Other modules are minified individually, to improve cache hit rate.
Bug: T107377
Change-Id: Id6f5142062d73b5701126724e0fe8264105f7813
When minifying JavaScript, never put each statement on a separate line, and
always set a target maximum line length of 1000. These behaviors were
previously configurable via $wgResourceLoaderMinifierStatementsOnOwnLine and
$wgResourceLoaderMinifierMaxLineLength, respectively.
Change-Id: I0b0eb632875b5e16f728fd0aa62f7f5ecd79ef62
Re-use the md_skin database field for now as to not need a
schema change, beause this table is going away soon (T113916).
Bug: T113868
Change-Id: I7c7546ec58fd9be0447604989b908dd2084b0fe3
* Remove ResourceLoaderFileModule::getLessCompiler(). There is no reason for a
module to need to get a compiler in a different manner than
ResourceLoader::getLessCompiler().
* Add ResourceLoaderModule::getLessVars(). This method provides a means for
subclasses to easily inject custom LESS variables. The default implementation
simply returns an empty array.
* Make the $context parameter for ResourceLoaderFileModule::readStyleFiles()
non-optional (via graceful deprecation). The only callers I found were either
already calling it with a ResourceLoader context, or had a perfectly usable
ResourceLoaderContext in local scope.
* Make ResourceLoaderFileModule::{readStyleFile,getLessCompiler} require a
context. These methods are protected, so I didn't bother with a deprecation.
* Call ksort() on the LESS variables array in the only place it matters -- when
hashing its serialized representation to construct a cache lookup key. This
relieves getLessVars() subclasses from having to remember to re-sort the
variables array if they modify it.
* These changes make it possible to substantially simplify
ResourceLoaderEditToolbarModule, because the only thing it needs to do now is
implement its own getLessVars() method.
* This also allows it to be versioned like any other ResourceLoaderFileModule,
rather than having to use enableModuleContentVersion().
Change-Id: Ic3eab71691e502bfe19bdf4eb6f82cc679a7782f
- Removed space after cast
- Removed spaces in array index
- Removed double spaces
- Added spaces around string concat
- Fixed mixed tabs and spaces at begin of line
Change-Id: I38e849723f055d2d4c05cba72f5c245a28e8d5da
This changes the current:
> Internal error.
To be:
> [89466ede] 2015-09-23 16:40: Fatal exception of type Less_Exception
This matches how exceptions are reported on HTML pages by MediaWiki
(via MWException::getHTML in OutputPage).
The IDs are okay to be public and exposed on HTML pages even when
$wgShowExceptionDetails is disabled.
Follows-up d478ffde10. Ref T110659.
Change-Id: I501eaa629161184ba89d44e53cdac20b39cb0903
We don't allow modifying existing modules or registering the same module twice.
We do have an established convention of registering plugins for optional
dependencies conditionally.
E.g.
- VisualEditor should only register 'uls.data' if ULS extension isn't installed.
- The Math extension should only register its VE plugin if VE is installed.
Typically the dependency being checked is registered unconditionally. The
conditional one is the supplemental one in case the canonical source within
the MediaWiki landscape is absent.
Currently this means code will check both isModuleRegistered() and the global
$wgResourceModules. This doesn't make sense. Reverse this so that the
unconditional registry is processed first, and the programmatic one after.
This also helps exceptions come from a more sensible place in case of a duplicate
registration.
Change-Id: Ib2d01c88511ade75a97b2509bccfabb1a4cc5c15
Mechanical stacktraces are mostly program identifiers (which are English) with some
glue text.
When stacktraces are disabled by the site admin, the replacement message doesn't
seem valuable to localise. And one could argue it should be in the user language
instead of the site language in that case, which complicates the output buffer
and caching layers for this.
It also reduces coupling with MediaWiki (Ref T32956).
Change-Id: If446f1b815add2be4baf1c333f18a4bab0cc9c37
wfExpandUrl() should only be used if the url needs to be absolute
and/or needs to point to the canonical, internal or default wgServer
for the wiki.
For meta-data or API responses this can make sense, but for resources
loaded by the browser from the current page, there is no added value
in expanding these to the default domain.
This is causing problems in custom setups such as at Wikimedia,
with "zero" and "m" sub domains for which $wgServer points to the
canonical domain.
Bug: T106966
Bug: T112646
Change-Id: Ie25d3a8f69564319a1b29b5c5904f3d352924a31
* Use implode() instead of foreach concat.
* Use two new lines instead of just one since the exception message
with stacktrace spans multiple lines (makes it easier to distinguish
when there are indeed exceptions from multiple sources).
* Output a single comment instead of one for each.
* If context is a JavaScript response, also include a console.error
call with the erro message.
To try out:
* Break a file module descriptor in Resources.php by e.g. making
a typo in one of the scripts arrays.
* View a page on-wiki that uses the module (e.g. jquery.accessKeyLabel
is loaded on most pages).
* Observe error in the console.
Bug: T110659
Change-Id: I4272795f1fab96a2effe2a6c068a56421adaa512
Deprecated in 1.24, for reasons explained in a0c41ae39d. I don't see any
usage in core or extensions.
Change-Id: I46f9e04ae633e7ff1ee112b652e1865731172f1f
This makes it consistent with how we handle the resourceloader_cache hit metric.
When I added the miss metric I figured it was nice to get both counts and timings
in a single metric, but I can't trust StatsD/Graphite not to screw this up so
let's untangle them to be sure.
Change-Id: I24f68e34a9fa491806fdb6023bdd757f6688c2ed
Follows-up r88883 (8dab43f703). The IEUrlExtension library only considers the
query string if the base path has no extension (or if the dot is urlencoded).
Neither of which is the case for load.php urls.
If the server doesn't expose undecoded request URIs (per IEUrlExtension)
then it will look in the query string always. However even then it is
only an issue if the last query parameter in the url contains a dot,
which is very unlikely for load.php (they typically end in 'skin' or
'version' which don't contain plain dots).
Ref T30840.
Change-Id: I4a4ef7ef5ad7d74ecb47a44342e6324fa9844173
Minify the JavaScript generated in ResourceLoader::makeConfigSetScript() unless
debug mode is set.
Change-Id: I45b5839803546c8e35e74b72c679777815294315
In the three years or so that I have been around, we at Wikimedia have not to
my knowledge used the cache keys to manage the cache of ResourceLoader modules.
Instead, if we want to force a module to rebuild, we simply touch one of its
files. (Though even that is not often required these days.)
Change-Id: Ib62a792e053572f268d4be19a10f99f84c07abc5
In IE8 there is a race condition. If the window.RLQ runs first (that is, the
startup module loads first) then 'var RLQ' does not get associated
properly to our custom window.RLQ object.
The other way around is fine, but never happens because <script async>
isn't supported in IE8.
Consistently use 'window.RLQ' (never 'var RLQ') to bypass this bug.
Also updating the startup.js file for consistency (though not strictly
needed since the implicit global reference without 'var' also works fine
in IE8.
Bug: T107954
Change-Id: I3f46fee7c4528abf806bb9c51fc767eceb795009
Update unit test to account for the internal 'html' prop now
being an array instead of string. And update expected values to
no longer have a trailing line break.
Bug: T27202
Change-Id: I105b6ef2e64ab8b891562e16940edb88592bd415
Follows-up 19a40cd3ad, Id599b6be4.
TODO: Change minification to happen per-module instead of per-request so that
these hacks aren't needed (T107377).
Change-Id: Iaa281ee117f2ae7a51884d256dfbb1807224fc52
Instead of having inline statements be plain statements wrapped
in an if-conditional block, convert them to inline functions
pushed into a queue.
The queue is kept in-memory until the startup module is loaded
at which point it transforms into a function that is immediately
invoked.
This is a prerequisite to making the top queue asynchronous.
Until then these functions will just run immediately as they
occur in the HTML after the startup module.
This is based on the previously reverted commit e86e5f8460.
Bug: T107399
Change-Id: Ifb38efca219c10ab973ad4c4ebb21c6a4239b005
Regression from 19a40cd3ad which made the 'site' module load asynchronously,
but the 'user' module was still loaded synchronously which meant it ran before
the site module finished.
Full test script at <https://gist.github.com/Krinkle/5db1d237da241b243485>.
Also:
* This changes the 'user' module to load asynchronously.
* Similar to 19a40cd3ad for site module, this makes the styles for the user
module load twice. Harmless but doesn't look pretty internally.
* Remove the obsolete XXX-comment from 0b5389d98d (r56770).
* Add comment documenting the fact that the 'excludepages' feature can cause
User/common.js and User/vector.js to be mis-ordered when the user previews
common.js edits. This has always been the case (since 2009) and is merely
being documented here.
Bug: T32358
Bug: T106736
Bug: T102077
Change-Id: Id599b6be42613529fb7f4dd3273f36ccadb3a09e
* No longer a dedicated <script> with only=scripts.
This means it creates no extra script request and becomes a versioned
request using data from the startup module.
* No longer in group=site.
This means it collapses into the existing bottom queue.
Not even one dedicated script request, but zero.
* No longer exclude from module storage. This can be cached like any other module.
It was previously excluded because it was already loaded separately.
* Change mw.loader#execute to special-case the 'site' module with $.globalEval.
* Add hack to ensure the styles of the 'site' module still load without
JavaScript, in the top, and after the ResourceLoaderDynamicStyles marker.
This unfortunately stays its own request. Not sure how to avoid that.
Bug: T32358
Bug: T106736
Bug: T102077
Change-Id: I291a8c3aae1a71760bec58161891c1bd77c9b724
To help distinguish cache hits from cache misses, send a MediaWiki-Timestamp
header on ResourceLoader responses, indicating the current time as fractional
seconds since epoch, with microsecond precision.
Change-Id: I954a96ced1435a47d458de12411b257fe6b4ea34
Follows-up 458e7cabbb. HHVM's APC cache doesn't have any
limitation or eviction logic. Thus entries without a TTL will
linger indefinitely.
At time of writing over 99% of APC entries were from "resourceloader:filter"
because there are far more unpopular resources than popular ones.
A fixed TTL is suboptimal as it will also cause popular resources
(e.g. startup module, or jquery) to be re-minified every day.
Beware this isn't one single minification. It's one minification,
for every JS/CSS resource, on every app server, for many different
lang/skin/config variations.
Bug: T104769
Change-Id: Ib62c320da0eeede68aad3f2294006ec7dcf1df55
The keys already contain a hash of the contents and the version of
the filter system. No need for these to be fragmented by wiki.
Previously wiki farms would minify the same module build for
every wiki. For e.g. Wikimedia this should bring down minification
runs for a new module version from ~800x to something lower.
It won't be 1 since modules may still vary by language, or config.
This should speed up load.php responses when a new module version
is deployed.
It will also reduce response time for page views of logged-in
users due to improved minification cache of embedded modules.
Change-Id: Iee884208c5c4ba40b46abd332271df698c6afb6f
Follow-up b88e88b, which converted the count stats to include
timing information but incidentally moved the gathering point
inside the method, which meant it now also applied to uncached
minifications from logged-in user html views (called from one
branch higher in the filter method).
This should reset the 'resourceloader_cache.minify-js.miss.count'
back to normal levels.
Change-Id: I7761242f016e9026f5012352ec2f7297c1753f25
- Removed space after casts
- Removed spaces in array index
- Added spaces around string concat
- Added space after words: switch, foreach
- else if -> elseif
- Removed parentheses around require_once, because it is not a function
- Added newline at end of file
- Removed double spaces
- Added spaces around operations
- Removed repeated newlines
Bug: T102609
Change-Id: Ib860222b24f8ad8e9062cd4dc42ec88dc63fb49e
Centralise the building of module packages into this method so
that it can be easily re-used.
This is in preparation for providing the option for modules to use
content-based hashing (instead of based on meta-data) in the version
string of modules. Having a getModuleContent method allows that to
be implemented in a way that is well-cached (as we'll be calling it
from multiple multiple code paths) and without duplicating this logic.
Bug: T98087
Change-Id: I376233caaabe44b6101565b70a50904abdf8ab4f
* Change "private module denied" level to 'debug'.
This is quite unimportant and occupies over 90% of the logstash
entries for resourceloader.
* Changes all failure signals to 'warning' and using context data.
Left out method in most case since the messages are unambiguous.
Change-Id: Ibb2faaa54020ab0fd9685551689d9a89161e1ce7
Also:
* Update wfHttpError() to use uppercase DOCTYPE, to match other code
such as Html.php, wfThumbError(), HttpError.php, etc.
Change-Id: I4027e7fe1a138b03f78797b6d1bfe7bd1064d360
Graphite expects name components to be dot-separated, so our habit of using
dashes doesn't really make sense. Change metric names to be more compatible
with Graphite, except the job queue's, since that will require a gdash
dashboard definition migration.
Change-Id: I77d0ff7606a8fc88434e4352d23415a9a8f4725a
On HHVM, (a) the cost of a ResourceLoader::filter cache miss is smaller, and
(b) APC is faster and more resilient than PHP's. These two things tilt the
scales in favor of using APC rather than Memcached.
Change-Id: I30b490e5bc815849d4b0bb06f2dbbb68825ed225
As of b1e4006b4, the tokens are different on every request.
Caching these is completely useless because the cache entry is
simply unreachable and is extra overhead on every request for
logged-in users to save content to Memcached.
Whether they should be minified at all and whether they perhaps
shouldn't change on every request is a separate matter.
Bug: T84960
Change-Id: I6016e4b01e44e0acbfd6d49f7d99555e2290c9cb
* Split off OOjs UI's module definitions to a separate file from
Resources.php.
* Extend ResourceLoaderImageModule to support per-skin images and
variants.
* Allow skins to specify their preferred OOjs UI theme using new
skin registry attribute 'SkinOOUIThemes'. The default remains the
'mediawiki' theme, 'apex' can also be chosen now.
* Implement custom ResourceLoaderOOUIImageModule which implements some
special sauce required to shoehorn OOjs UI image set definitions into
ResourceLoaderImageModule with skin-specific styles support.
Bug: T100895
Change-Id: I3fbacbce95126a4cc29221352f3ef846f41e7b1b
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
Follows-up a4379d8f9e which broke the warning output.
Previously it was appending warnings to a non-existent "$response"
variable and then re-assigning it with the actual response, thus
never outputting this anywhere.
Change-Id: I6c53e7ccb4ae4783d496c06c7174c9d8a1a88b55
Other code paths in this class already make use of this method,
but this was still embedding arbitrary strings directly, at the
risk of producting invalid CSS or invalid JavaScript.
Change-Id: Ied6c420536a3dd1ef8f3816a078705136e8475b1
Fatal error: [] operator not supported for strings in
includes/resourceloader/ResourceLoader.php on line 354
This would occur when a skin customized ResourceModuleSkinStyles using
the '+' syntax, which preserves default skinStyles, and the default
skinStyles was defined as a string rather than array.
Change-Id: I6cd12a638d562c75bc4704e098de84f3882248b6
There was a code path that sent plain 304s without Expires or any
other headers. Fix this by moving the call to sendResponseHeaders()
into tryLastModified().
Bug: 51283
Change-Id: I15d13c5d32102f53bf3e3aaac01d76967e968f78
No skin in gerrit uses the feature of per-module local or remote paths,
and not supporting it will make implementing extension/skin.json support
easier.
Change-Id: I22863ef29426dbacd65439a12aeae56d4d757951
Follows-up ebeb29723, 1f393b6da, 0e719ce23.
Also:
* Add tests for ResourceLoader::makeLoaderImplementScript().
* Apply ResourceLoader::trimArray to makeLoaderImplementScript (new in c0c221bf).
This commit changes the load.php response to omit empty parameters.
These parameters were required until recently. The client has been
updated (1f393b6da and 0e719ce23) to make these optional, thus supporting
both the old server format and the change this commit makes
Clients with a tab open from before 0e719ce23 are naturally not
compatible with load.php responses from after this commit. Ensure
this is deployed several days after 0e719ce23 to reduce race
conditions of this nature.
(This is a re-submitted version of 4ce0c0da4)
Bug: T88879
Change-Id: I9e998261ee9b0b745e3339bc3493755c0cb04b6a
The test for OutputPage::makeResourceLoaderLink was triggering database
queries through MessageBlobStore even though it doesn't use any
messages.
In bb03d1a8e0 I had made MessageBlobStore a singleton instead of static
functions, however there's no need for it to be one since the class is
stateless. Callers can just create a new MessageBlobStore instance and
call functions upon it. Using getInstance() is now deprecated.
ResourceLoader now has a setMessageBlobStore setter to allow overriding
which MessageBlobStore instance will be used. OutputPageTest uses this
to set a NullMessageBlobStore, which makes no database queries.
Change-Id: Ica7436fb6f1ea59bd445b02527829ab0742c0842
The patch did not improve performance. I'd like to think that the increased
control over when inline scripts are executed makes the patch worthwhile
regardless, but that is post hoc justification and possibly a bit of personal
ego. Krinkle agrees that we may use some of the ideas in this patch in the
future but he thinks we're better off not heading down this path before we
have a better sense of where we're going, and I trust his judgment.
This reverts commit e86e5f8460.
Change-Id: I151f74a41dd664b5a0aa5cfd99fcc95e2686a1e6
The current ordering of scripts and stylesheets in <head> causes all major
browsers to serialize and defer requests that could be performed in parallel.
The problem is that external stylesheets are loaded before inline scripts. As
Steven Souders explains, "all major browsers preserve the order of CSS and
JavaScript. The stylesheet has to be fully downloaded, parsed, and applied
before the inline script is executed. And the inline script must be executed
before the remaining resources can be downloaded. Therefore, resources that
follow a stylesheet and inline script are blocked from downloading."[1]
In other words: the browser could start loading body images, but it refuses to
do that until it has executed inline scripts in head. And it refuses to execute
those scripts until the external CSS is downloaded, parsed and applied. You can
see the effect of this in this image, showing the request waterfall for
[[en:Gothic Alphabet]]: [2]. Notice how no images were requested before the
browser had finished processing the three load.php requests at the top.
To fix this, we want to move the inline scripts above the external CSS. This is
a little bit tricky, because the inline scripts depend on mw.loader, which is
loaded via an external script. If we move the external script so that it too is
above the external stylesheet, we force the browser to serialize requests,
because the browser will not retrieve the external CSS until it has retrieved
and executed the external JS code. So what we want is to move the inline
scripts above the external stylesheet, but keep the external script (which the
inline scripts depend on) below the external stylesheet.
We can do this by wrapping the inline script code in a closure (which binds
'mw') and enqueuing the closure in a global array which will be processed by
the startup module at just the right time.
Net result: external CSS and JS is retrieved in parallel, retrieval of images
(and other external assets) is unblocked, but the order in which code is
evaluated remains the same.
[1]: <http://www.stevesouders.com/blog/2009/05/06/positioning-inline-scripts/>
[2]: <http://people.wikimedia.org/~ori/enwiki-waterfall.png> (excerpted from
<http://www.webpagetest.org/result/150316_0C_7MB/1/details/>.
Change-Id: I98d383a6299ffbd10210431544a505338ca8643f
Currently if code wants to check whether a module is registered it
has to call getModule() and see if the response !== null.
Change-Id: I4b470083ddaa5d8cd6be50d5c5b690d4b99b6c4a
Add new hook (ResourceLoaderGetLessVars) called in ResourceLoader::getLessVars to
allow context-based less variables. Cache the resulting array to avoid multiple runs
of this hook.
Change-Id: I5a73bbd0ab58f8fe34519931c4f26c90998e3451
Works as intended, but didn't account for the first implement() parameter
being omitted client-side. Revert until that is accounted for, then re-try after
that fix is rolled out for > 1 week.
This reverts commit 4ce0c0da42.
Change-Id: I36c1619991663c0303636d1d3f037b0021ac79bf
