Follows-up f37cee996e which replaced the getHashMtime() and
getDefinitionMtime() methods with dummies that always return 1.
These getModifiedTime() implementations were only tracking the
definition summary or custom hash, which is already tracked
by getVersionHash().
Notes:
* SpecialCharacterDataModule: This one was odd as it was tracking
both the mtime *and* the file contents.
* UserCSSPrefsModule/UserOptionsModule: Remove redundant caching.
Already taken care of by getVersionHash() as of f37cee996e.
Bug: T94074
Change-Id: I6e37c3c2f85ef4599a8643b0efabc18de2f51ec4
Modules now track their version via getVersionHash() instead of getModifiedTime().
== Background ==
While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.
For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.
Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.
For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.
More scenarios can be found in the associated Phabricator tasks.
== Version hash ==
Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.
All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.
For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.
As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.
* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule
The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).
== ETag ==
Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).
Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).
Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.
While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).
== File cache ==
If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.
For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.
This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.
== Misc ==
* Remove redundant modifiedTime cache in ResourceLoaderFileModule.
* Change bugzilla references to Phabricator.
* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
logic was duplicated in each place the modified timestamp was used.
* It's easy to forget calling the parent class in getDefinitionSummary().
Previously this method only tracked 'class' by default. As such, various
extensions hardcoded that one value instead of calling the parent and extending
the array. To better prevent this in the future, getVersionHash() now asserts
that the '_cacheEpoch' property made it through.
* tests: Don't use getDefinitionSummary() as an API.
Fix ResourceLoaderWikiModuleTest to call getPages properly.
* In tests, the default timestamp used to be 1388534400000 (which is the unix time
of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
'{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'
* Add sha1.js library for client-side hash generation.
Compared various different implementations for code size (after minfication/gzip),
and speed (when used for short hexidecimal strings).
https://jsperf.com/sha1-implementations
- CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
Chrome: 45k, Firefox: 89k, Safari: 92k
- jsSHA <https://github.com/Caligatio/jsSHA>
https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
Chrome: 65k, Firefox: 53k, Safari: 69k
- phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
Chrome: 200k, Firefox: 280k, Safari: 78k
Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
needed in the module loader, we'd have to load the fallback regardless. Considering
this is not used in a critical path for performance, it's not worth shipping two
implementations for this optimisation.
May also resolve:
* T44094
* T90411
* T94810
Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
This makes it easier for subclasses to use ResourceLoaderWikiModule. Currently
many subclasses of this simply need to override the getPages() method.
UserModule and SiteModule keep their getPages override due to the set of pages
being dependent on context.
Change-Id: I388531398671afacfec36c6c5746d72267b5bdac
* Use time() instead of:
- wfTimestamp()
- wfTimestamp( TS_UNIX )
- wfTimestamp( TS_UNIX, 0 )
- wfTimestamp( TS_UNIX, time() )
- intval( wfTimestamp( TS_UNIX, time() ) )
* Consistently use 1 as default instead of 0. Previously the
unwritten convention was that anything "final" max()'ed with 1,
and any internal method would use 0, but this wasn't applied
consistently made it fragile. There doesn't seem to be any
value in returning 0 only to have it maxed up to 1 (because if
the 0 would ever make it out alive, we'd be in trouble).
* wfTimestamp returns a string for TS_UNIX. In PHP this doesn't
matter much. In fact, max() takes number-like integers so
transparently, it even preserves it:
> max( 1, 3, '2' );
< 3
> max( 1, '3', 2 );
< "3"
Just cast it in one place at the very end (StartupModule)
instead of doing intval( wfTimestamp() ).
* Fix weird documentation claiming getModifiedTime can return
an array, or mixed.
* Remove 'version > 1 ? version : 1' logic in
ResourceLoader::makeLoaderRegisterScript. The client doesn't
have "0 means now" behaviour so this isn't needed. And the method
was only doing it for variadic argument calls.
Removal of quotes around timestamps reduced the size of the startup
module from 26.8KB to 25.9KB before gzip. After gzip the size was
and still is 5.7KB, though. (From 5456 bytes to 5415 bytes.)
Change-Id: If92ca3e7511e78fa779f2f2701e2ab24db78c8a8
- Added newline at end of file
- Removed double spaces/newlines
- Added space after if/function and parentheses/brackets
- Removed space before comma/cast
- Fixed indent of some lines
Change-Id: I29867ffdffdfb7d2b56997e9393497c7dc12f7d3
This reverts most of commit 2d842f1425,
leaving only the test added in it, and reimplements the same
functionality better.
Instead of stripping /*@noflip*/ annotations in CSSJanus, which is
incompatible with other implementations that preserve it, extend
CSSMin to allow other CSS comments to be present before the
rule-global @embed annotation. (This required making the regex logic
in it even worse than it was, but it's actually slightly less terrible
than I expected it would be. Good thing we have tests!)
Bug: 69698
Change-Id: I58603ef64f7d7cdc6461b34721a4d6b15f15ad79
For Gadgets 2.0, we have custom content models that represent
JavaScript and CSS pages. Comparing the exact content model here
is too strict, when we really want to know if the page supports
the JavaScript or CSS formats.
Also fix a debug log comment that didn't work due to improper quoting.
Change-Id: I64e86c74173b1c88cac71483327cacdd8b9967b9
In most cases, we just check whether the pages exist before saying
the module is not empty to avoid generating cached HTML without
the appropriate <script> or <link> tags.
However, for modules in the 'user' group, normal users cannot
delete their personal JavaScript/CSS pages, causing needless
extra requests, even though we know the pages are empty.
ResourceLoader::isKnownEmpty() now checks the page_len field
for modules in the 'user' group to check that there is
some actual content.
Bug: 68488
Change-Id: I0570f62887fd4642fd60367ae0b51d7dc19488ca
To do it, just remove /*@noflip*/ annotations in CSSJanus after
we're done processing. They are not needed anymore and some obscure
interactions with CSSMin logic for preserving comments caused
`/*@noflip*/ /*@embed*/ background-image: url(…)` not to work
correctly (it would not be embedded).
This also requires us to always do CSSJanus processing, even when we
don't need flipping, to consistently handle the annotations.
I'm not entirely sure if this is worth it, but I still greatly prefer
doing it to documenting this stupid limitation. :)
Bug: 69698
Change-Id: I311b12b08b2dff9d45efb584db08cf4a11318f59
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Change-Id: Ifbb1da2a6278b0bde2a6f6ce2e7bd383ee3fb28a
It's a bit odd to simply decline to entitle wiki page modules that have '*/' in
their title when the title could simply be escaped.
Change-Id: I4456ae683cdae4a5a5e4a4a731046cadf1d8ee4d
Currently we invalidate module caches based on timestamps from
various places (including message blob, file mtimes and more).
This meant that when a module changes such that the maximum
detectable timestamp is still the same, the cache would not
invalidate.
Module classes can now implement a method to build a summary.
In most cases this should be (a normalised version of) the
definition array originally given to ResourceLoader::register
(such as $wgResourceModules items).
The most common scenarios this addresses:
* File lists (scripts, styles) being re-ordered.
* Files being removed when more recently changed files remain
part of the module (e.g. removing an older file).
* Files or messages being added to a module while other more
recently changed things are already part of the module.
E.g. you create a message and use it in a js file, but forget
to add the message to the module. Then later you add the msg
key. This last update would be ignored, because the mtime of
the message is no newer than the total which already included
the (same) mtime of the js file added in the previous update.
* Change the concatenation order of pages in a Gadget definition.
Bug: 37812
Change-Id: I00cf086c981a84235623bf58fb83c9c23aa2d619
Some were using generic wfDebug().
Removed obsolete new-line which is already added by wfDebugLog.
Bug: 44018
Change-Id: I9907b374fa868c04ff2ce40964238936b9084a4a
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments
Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
Before the introduction of the content handler, missing content was
signified by getText() returning null instead of a string. null will
work much like an empty string in most contexts, so in many places,
it was not checked explcitely whether the conent was null.
Now, when getContent() returns null, this often caused a fatal error,
because the code would access whatever getContent() returned as an object,
without checking whether it was null (because no such check was performed
previously, when the content was represented as a string).
This check introduces explicite checks for getContent() returning null
in the most essential core classes.
Change-Id: I551a90b0b67b8edc7570ca5d252ecc1de903f097
Fixed by no longer using MessageCache::get() in
ResourceLoaderWikiModule. The only reason I see for using it is to make
default values for .js pages in the MediaWiki: namespace work (by
specifying them in an .i18n.php file), but that's evil and no one was
doing that other than putting in descriptive comments like
/* Site scripts go here */. That, and the previous implementation of
MediaWiki namespace JS that RL replaced did it, so I kept it at the
time, assuming it must have been needed for some reason.
Change-Id: Ica7f35fefcc200f0b3daf1d06d98a0b8ea878f3a
Fixes:
* bug 31676: Work around IE stylesheet limit.
* bug 35562: @import styles broken in modules that combine
multiple stylesheets.
* bug 40498: Don't output empty "@media print { }" blocks.
* bug 40500: Don't ignore media-type for urls in debug mode.
Approach:
* Re-use the same <style> tag so that we stay under the 31
stylesheet limit in IE. Unless the to-be-added css text from
the being-loaded module contains @import, in which case we do
create a new <style> tag and then re-use that one from that
point on (bug 31676).
* Return stylesheets as arrays, instead of a concatenated string.
This fixes bug 35562, because @import only works when at the
top of a stylesheet. By not unconditionally concatenating files
within a module on the server side already, @import will work
in e.g. module 'site' that contains 2 wiki pages.
This is normalized in ResourceLoader::makeCombinedStyles(),
so far only ResourceLoaderWikiModule makes use of this.
Misc. clean up and bug fixes:
* Reducing usage of jQuery() and mw.html.element() where
native DOM would be very simple and faster. Aside from
simplicity and speed, this is also working towards a more
stand-alone ResourceLoader.
* Trim server output a little bit more
- Redundant new line after minify-css (it is now an array, so
no need to keep space afterwards)
- Redundant semi-colon after minify-js if it ends in a colon
* Allow space in styleTest.css.php
* Clean up and extend unit tests to cover for these features
and bug fixes.
* Don't set styleEl.rel = 'stylesheet'; that has no business
on a <style> tag.
* Fix bug in mw.loader's addStyleTag(). It turns out IE6
has an odd security measure that does not allow manipulation
of elements (at least style tags) that are created by a
different script (even if that script was served from the same
domain/origin etc.). We didn't ran into this before because
we only created new style tags, never appended to them. Now
that we do, this came up. Took a while to figure out because
it was created by mediawiki.js but it calls jQuery which did
the actual dom insertion. Odd thing is, we load jquery.js and
mediawiki.js in the same request even...
Without this all css-url related mw.loader tests would fail
in IE6.
* mediawiki.js and mediawiki.test.js now pass jshint again.
Tested (and passing qunit/?module=mediawiki; 123 of 123):
* Chrome 14, 21
* Firefox 3.0, 3.6, 4, 7, 14, 15, 16beta
* IE 6, 7, 8, 9
* Safari 4.0, 5.0, 5.1
* Opera 10.0, 11.1, 11.5, 11.6, 12.0, 12.5beta
* iPhone 3GS / iOS 3.0 / Mobile Safari 4.0
iPhone 4 / iOS 4.0.1 / Mobile Safari 4.0.5
iPhone 4S / iOS 6.0 Beta / Mobile Safari 6.0
Change-Id: I3e8227ddb87fd9441071ca935439fc6467751dab
Doxygen does recognize abstract methods when it is properly marked as
abstracted. That keyword is a left.
Change-Id: I7df790058edf0eefb07eb4c37a2586b5c16898e3
The $isFullKey parameter means the message key already contains the language subpage ("/de", etc.)
Change-Id: I9bcc93563a2a2f8c38c98e50e18067e70f6fb3d7
Also adding a documentation comment in ResourceLoaderModule to clarify that it is the subclass's responsibility to make sure message blob timestamps are taken into account
This is possibly not perfect but seems to serve for a start; follows up on r91591 that adds JSMin+ to use it in some unit tests. May want to adjust some related bits.
- $wgResourceLoaderValidateJs on by default (can be disabled)
- when loading a JS file through ResourceLoaderFileModule or ResourceLoaderWikiModule, parse it using JSMinPlus's JSParser class. If the parser throws an exception, the JS code of the offending file will be replaced by a JS exception throw listing the file or page name, line number (in original form), and description of the error from the parser.
- parsing results are cached based on md5 of content to avoid re-parsing identical text
- for JS pages loaded via direct load.php request, the parse error is thrown and visible in the JS console/error log
Issues:
- the primary use case for this is when a single load.php request implements multiple modules via mw.loader.implement() -- the loader catches the exception and skips on to the next module (good) but doesn't re-throw the exception for the JS console. It does log to console if present, but it'll only show up as a regular debug message, not an error. This can suppress visibility of errors in a module that's loaded together with other modules (such as a gadget).
- have not done performance testing on the JSParser
- have not done thorough unit testing with the JSParser
Done by adding isKnownEmpty() to ResourceLoaderModule and overriding it to check for page existence in ResourceLoaderWikiModule. Needed to rearrange some code in OutputPage::makeResourceLoaderLink() to have the emptiness check and dropping of modules work properly. Also factored the page_touched check in ResourceLoaderWikiModule::getModifiedTime() out to a separate method (getTitleMtimes()) and moved in-object caching there as well, so getModifiedTime() and isKnownEmpty() share code and caching for their timestamp/existence checks.
This does not account for the case where e.g. a user has user CSS but no user JS: I had implemented this by checking for $context->getOnly() in getTitleMtimes(), but then realized it's not safe to do this in a function called by getModifiedTime(): it causes the timestamp list in the startup module to only take scripts in account for wiki modules, because the startup module has &only=scripts set
