Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
Similar to what getDefinitionMtime() does already. No need to
repeat the hash. No need for an array that needs serialising and
unserialising internally.
Change the hash key to avoid using old cache values.
Also moved the comment about hashes being included in the key to
this method. getDefinitionMtime() is a later method that performs
the same logic but doesn't need the entire story again.
Follows-up 044713c4, d3bdda32.
Change-Id: Idd83de5ac27138a2dbf2ec49d81ea9188bd6ad57
Xhprof generates this data now. Custom profiling of various
sub-function units are kept.
Calls to profiler represented about 3% of page execution
time on Special:BlankPage (1.5% in/out); after this change
it's down to about 0.98% of page execution time.
Change-Id: Id9a1dc9d8f80bbd52e42226b724a1e1213d07af7
* Use time() instead of:
- wfTimestamp()
- wfTimestamp( TS_UNIX )
- wfTimestamp( TS_UNIX, 0 )
- wfTimestamp( TS_UNIX, time() )
- intval( wfTimestamp( TS_UNIX, time() ) )
* Consistently use 1 as default instead of 0. Previously the
unwritten convention was that anything "final" max()'ed with 1,
and any internal method would use 0, but this wasn't applied
consistently made it fragile. There doesn't seem to be any
value in returning 0 only to have it maxed up to 1 (because if
the 0 would ever make it out alive, we'd be in trouble).
* wfTimestamp returns a string for TS_UNIX. In PHP this doesn't
matter much. In fact, max() takes number-like integers so
transparently, it even preserves it:
> max( 1, 3, '2' );
< 3
> max( 1, '3', 2 );
< "3"
Just cast it in one place at the very end (StartupModule)
instead of doing intval( wfTimestamp() ).
* Fix weird documentation claiming getModifiedTime can return
an array, or mixed.
* Remove 'version > 1 ? version : 1' logic in
ResourceLoader::makeLoaderRegisterScript. The client doesn't
have "0 means now" behaviour so this isn't needed. And the method
was only doing it for variadic argument calls.
Removal of quotes around timestamps reduced the size of the startup
module from 26.8KB to 25.9KB before gzip. After gzip the size was
and still is 5.7KB, though. (From 5456 bytes to 5415 bytes.)
Change-Id: If92ca3e7511e78fa779f2f2701e2ab24db78c8a8
A base ResourceLoaderModule::getTemplates() exists for subclasses
to override. An implementation is provided for ResourceLoaderFileModule.
For file modules, templates can be specified in the following manner:
'example' => array(
'templates' => array(
'bar' => 'templates/foo.html',
),
'scripts' => 'example.js',
),
The delivery system is template language agnostic, and currently
only supports "compiling" plain HTML templates.
This also adds template support to the following modules as a POC:
* mediawiki.feedback
* mediawiki.action.view.postEdit
* mediawiki.special.upload
Works with $wgResourceLoaderStorageEnabled
Change-Id: Ia0c5c8ec960aa6dff12c9626cee41ae9a3286b76
Not ready for merging, and Roan says that the +2 was
most likely accidental and meant to be a -1.
This reverts commit d146934f94.
Change-Id: I3926c9ae9e3c8026fceb3aeedd3b1f1d9b91667b
Preparation work for templating in core.
RL should allow us to ship HTML / template markup from server to client.
Use in Special:Upload and mediawiki.feedback as a proof of concept.
Separation of concerns etc...
See Also:
Ia63d6b6868f23a773e4a41daa0036d4bf2cd6724
Change-Id: I6ff38c12897e3164969a1090449e626001926c3b
- Swap "$variable type" to "type $variable"
- Added missing types
- Fixed spacing inside docs
- Makes beginning of @param/@return/@var/@throws in capital
- Changed some types to match the more common spelling
Change-Id: I8ebfbcea0e2ae2670553822acedde49c1aa7e98d
To do so, created ResourceLoader::createLoaderURL(), which takes a
ResourceLoaderContext object. ResourceLoader::makeLoaderURL() was
deprecated.
While reviewing usage of the old function, many of the callers only
differed by one or two parameters from their respective
ResourceLoaderContext object. To simplify that use case, I created
DerivativeResourceLoaderContext, based of off DerivativeContext for
IContextSource.
Change-Id: I961c641ab953153057be3e3b8cf6c07435e9a0b0
A module can be registered with a skip function. Such function,
if provided, will be invoked by the client when a module is
queued for loading. If the function returns true, the client will
bypass any further loading action and mark the module as 'ready'.
This can be used to implement a feature test for a module
providing a shim or polyfill.
* Change visibility of method ResourceLoader::filter to public.
So that it can be invoked by ResourceLoaderStartupModule.
* Add option to suppress the cache key report in ResourceLoader::filter.
We usually only call the minifier once on an entire request
reponse (because it's all concatenated javascript or embedded
javascript in various different closures, still valid as one
large script) and only add a little bottom line for the cache
key. When embedding the skip function we have to run the minifier
on them separately as they're output as strings (not actual
functions). These strings are typically quite small and blowing
up the response with loads of cache keys is not desirable in
production.
* Add method to clear the static cache of ResourceLoader::inDebugMode.
Global static state is evil but, as long as we have it, we at
least need to clear it after switching contexts in the test suite.
Also:
* Remove obsolete setting of 'debug=true' in the FauxRequest in
ResourceLoaderTestCase. It already sets global wgResourceLoaderDebug
in the setUp() method.
Bug: 66390
Change-Id: I87a0ea888d791ad39f114380c42e2daeca470961
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Change-Id: Ifbb1da2a6278b0bde2a6f6ce2e7bd383ee3fb28a
We were continuously invalidating the cache for the
ResourceLoaderLanguageDataModule every time a request for a langauge
other than the previous request came in as they all resulted in
the same cache key.
This could be resolved by adding $context->getLang() to the key
(which would be enough for langdata module, but not enough for
other uses of hashmtime), or by adding $context->getHash() (which
will fragment it for all possible context factor combinations,
slightly overkill for most cases, but would work).
Instead resolving it the same way we did in d3bdda3 for a similar
issue in #getDefinitionMtime: by adding the hash itself to the
key. That way it will solve the problem, including two nice things:
* Not fragment anymore than necessary (if it doesn't vary by
skin, it won't vary by skin, and same for language etc.).
* Always separate keys when the value is different (if something
can very based on other factors, e.g. LocalSettings globals or
whatever untracked method may exist for influencing anything,
it will all work as expected).
* Don't keep alternating and incrementing the timestamp while a
multi-server infrastructure is syncing code (e.g. old code and
new code for the same request context will not result in the
same key and each old/new server alternatingly increasing the
timestamp, instead the first new server will make the new key
and the other servers will gradually start finding the new key
with the timestamp of when it was first seen).
Change-Id: Ifa9088c11c388ca6c912e62efc43494e91702603
I added the profiling calls in PS13 of I00cf086c9 but forgot to add
wfProfileOuts for the early returns.
Change-Id: Ibcd22daba8da0e087376553bca271796e0d207e5
Currently we invalidate module caches based on timestamps from
various places (including message blob, file mtimes and more).
This meant that when a module changes such that the maximum
detectable timestamp is still the same, the cache would not
invalidate.
Module classes can now implement a method to build a summary.
In most cases this should be (a normalised version of) the
definition array originally given to ResourceLoader::register
(such as $wgResourceModules items).
The most common scenarios this addresses:
* File lists (scripts, styles) being re-ordered.
* Files being removed when more recently changed files remain
part of the module (e.g. removing an older file).
* Files or messages being added to a module while other more
recently changed things are already part of the module.
E.g. you create a message and use it in a js file, but forget
to add the message to the module. Then later you add the msg
key. This last update would be ignored, because the mtime of
the message is no newer than the total which already included
the (same) mtime of the js file added in the previous update.
* Change the concatenation order of pages in a Gadget definition.
Bug: 37812
Change-Id: I00cf086c981a84235623bf58fb83c9c23aa2d619
For now ResourceLoader will still internally deal with timestamps,
especially because they are easier to max() on the client side
(on the server we could make a hash of different hashes, on the
client that's a bit annoying).
However ResourceLoaderModule#getHashMtime will abstract the logic
we already use in ResourceLoaderLanguageDataModule and have
encouraged others to use, which is to put the timestamp and
hash in Memcached, and use that to observer the hash change and
update the timestamp when it does.
Updated ResourceLoaderLanguageDataModule to make use of this.
Change-Id: Ib051ef41ba239084671c30fd275b8d94099d5d52
A reworked version of script from I068686854ad79e2f63a08d81b1af02f373110613
Move leccs instantiation code to ResourceLoader because it doesn't depend on any
particular module's state.
Change-Id: I733b53171dca77f50a30e5bd0bd5f1b456e4c85d
Follows-up b67b9e1, lessphp now has a public method to add to the
list of files for compilation cache.
Change-Id: I62a6c7cdeb34ea742fa7547e3ca10e24ee484b97
'embeddedImages' is a custom property we set on lessc compiler instances to
track the set of files that were embedded in the generated source as data URIs.
This is so done so we know to regenerate the CSS if the source file for an
embedded asset changes. (If my pull request[1] is merged, we won't have to use
a custom property at all.)
I realized that the property name should be 'embeddedFiles' rather than
'embeddedImages', because it's not just images that can be usefully embedded in
CSS, but fonts as well.
[1]: https://github.com/leafo/lessphp/pull/472
Change-Id: Ief3afaa23e4532f4a536e0dfef943d4fa20db80d
This patch adds support for the LESS stylesheet language to ResourceLoader.
LESS is a stylesheet language that compiles into CSS. The patch includes
lessphp, a LESS compiler implemented in PHP. The rationale for choosing LESS is
explained in a MediaWiki RFC which accompanies this patch, available at
<https://www.mediawiki.org/wiki/Requests_for_comment/LESS>.
LESS support is provided for ResourceLoader file modules. It is triggered by
the presence of the '.less' extension in stylesheet filenames. LESS files are
compiled by lessc, and the resultant CSS is subjected to the standard set of
transformations (CSSJanus & CSSMin). The immediate result of LESS compilation
is encoded as an array, which includes the list of LESS files that were
compiled and their mtimes. This array is cached. Cache invalidation is
performed by comparing the cached mtimes with the mtimes of the files on disk.
If the compiler itself throws an exception, ResourceLoader constructs a
compilation result which consists of the error message encoded as a CSS
comment. Failed compilation results are cached too, but with an expiration time
of five minutes. The expiration time is required because the full list of
referenced files is not known.
Three configuration variables configure the global environment for LESS
modules: $wgResourceLoaderLESSVars, $wgResourceLoaderLESSFunctions, and
$wgResourceLoaderLESSImportPaths. $wgResourceLoaderLESSVars maps variable names
to CSS values, specified as strings. Variables declared in this array are
available in all LESS files. $wgResourceLoaderLESSFunctions is similar, except
it maps custom function names to PHP callables. These functions can be called
from within LESS to transform values. Read more about custom functions at
<http://leafo.net/lessphp/docs/#custom_functions>. Finally,
$wgResourceLoaderLESSImportPaths specifies file system paths in addition to the
current module's path where the LESS compiler should look up files referenced
in @import statements.
The issue of handling of /* @embed */ and /* @noflip */ annotations is left
unresolved. Earlier versions of this patch included an @embed analog
implemented as a LESS custom function, but there was enough ambiguity about
whether the strategy it took was optimal to merit discussing it in a separate,
follow-up patch.
Bug: 40964
Change-Id: Id052a04dd2f76a1f4aef39fbd454bd67f5fd282f
- A cast statement must not be followed by a space.
- The method parameter $context is never used.
- Avoid function calls in a FOR loop test part.
- Opening brace should be on the same line as closing parenthesis.
Change-Id: I0eba7fcc9ceab372003d1134857346690c525e87
* Ran spell-checker over code comments in /includes/
* A few spellchecking fixes for wfDebug() calls
Found one very strange (NOOP?) line in Linker.php - see "TODO: BUG?"
Change-Id: Ibb86b51073b980eda9ecce2cf0b8dd33f058adbf
Doxygen expects parameter types to come before the
parameter name in @param tags. Used a quick regex
to switch everything around where possible. This
only fixes cases where a primitve variable (or a
primitive followed by other types) is the variable
type. Other cases will need to be fixed manually.
Change-Id: Ic59fd20856eb0489d70f3469a56ebce0efb3db13
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments
Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
This is a useful method not just for inside and sub classes of
ResourceLoaderFileModule (i.e. it could've been useful in
VisualEditor's ResourceLoaderModule class as well)
Also moved up getTargets() to be in the right section (looking at
the file as a whole).
Change-Id: If696ffbdc5aa7f0a51603bcf9d52adab38b9c686
Goes along with MobileFrontend changes: https://gerrit.wikimedia.org/r/28434
Adds 'targets' option to module definitions, defaulting to 'desktop'.
Only a couple more modules are whitelisted into both desktop and mobile for now.
Startup module takes a 'target' parameter (defaults to 'desktop') to filter
the resource module registration list. Only modules matching the filter get
registered, and thus will be loadable from client-side RL.
Change-Id: Ifb772d4995b8e2ec4b63776fe0bb5b0214f82e04
Fixes:
* bug 31676: Work around IE stylesheet limit.
* bug 35562: @import styles broken in modules that combine
multiple stylesheets.
* bug 40498: Don't output empty "@media print { }" blocks.
* bug 40500: Don't ignore media-type for urls in debug mode.
Approach:
* Re-use the same <style> tag so that we stay under the 31
stylesheet limit in IE. Unless the to-be-added css text from
the being-loaded module contains @import, in which case we do
create a new <style> tag and then re-use that one from that
point on (bug 31676).
* Return stylesheets as arrays, instead of a concatenated string.
This fixes bug 35562, because @import only works when at the
top of a stylesheet. By not unconditionally concatenating files
within a module on the server side already, @import will work
in e.g. module 'site' that contains 2 wiki pages.
This is normalized in ResourceLoader::makeCombinedStyles(),
so far only ResourceLoaderWikiModule makes use of this.
Misc. clean up and bug fixes:
* Reducing usage of jQuery() and mw.html.element() where
native DOM would be very simple and faster. Aside from
simplicity and speed, this is also working towards a more
stand-alone ResourceLoader.
* Trim server output a little bit more
- Redundant new line after minify-css (it is now an array, so
no need to keep space afterwards)
- Redundant semi-colon after minify-js if it ends in a colon
* Allow space in styleTest.css.php
* Clean up and extend unit tests to cover for these features
and bug fixes.
* Don't set styleEl.rel = 'stylesheet'; that has no business
on a <style> tag.
* Fix bug in mw.loader's addStyleTag(). It turns out IE6
has an odd security measure that does not allow manipulation
of elements (at least style tags) that are created by a
different script (even if that script was served from the same
domain/origin etc.). We didn't ran into this before because
we only created new style tags, never appended to them. Now
that we do, this came up. Took a while to figure out because
it was created by mediawiki.js but it calls jQuery which did
the actual dom insertion. Odd thing is, we load jquery.js and
mediawiki.js in the same request even...
Without this all css-url related mw.loader tests would fail
in IE6.
* mediawiki.js and mediawiki.test.js now pass jshint again.
Tested (and passing qunit/?module=mediawiki; 123 of 123):
* Chrome 14, 21
* Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta
* IE 6, 7, 8, 9
* Safari 4.0, 5.0, 5.1
* Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta
* iPhone 3GS / iOS 3.0 / Mobile Safari 4.0
iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5
iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0
Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
When inserting XML elements inline <such as this one>, doxygen chokes
about it not being known. Simply enclosing the tag in double quotes
prevents doxygen from emitting a warning.
Also enclosed a few invalid functions calls such as \. and double quoted
the HTML entities such as &foobar;
Change-Id: I4019637145e683c2bec3d17b2fd98b0c50a932f1
This is needed for mobile scripts that are supposed to work on dumb
devices and as such can't rely on client-side RL, but still can take
advantage of server-side minification and concatenation.
Patchset 2: Allow loading raw modules with &raw=true appended to URL.
Change-Id: I9410ffbf6633075e07bd06b10a98a4d12d9b6106
Also adding a documentation comment in ResourceLoaderModule to clarify that it is the subclass's responsibility to make sure message blob timestamps are taken into account
