Commit graph

477 commits

Author SHA1 Message Date
Timo Tijhof
98911418c2 CSSMin: Remove file-existance filter in getLocalFileReferences()
Follows-up 8f5cd11d82.

The old getLocalFileReferences() method is no longer used anywhere.
Remove it and rename getAllLocalFileReferences back to it.

Change-Id: I864258aad128ba9b54464c7bc854543f2937f977
2016-03-03 00:16:13 +00:00
Timo Tijhof
bfb94f6f58 objectcache: Fix MultiWriteBagOStuff docs ('asyncHandler', not 'asyncCallback')
Follows-up 1171cc00cd.

Change-Id: I97189e916d0aba507273216e3e9ab28ce0d5ffec
2016-03-02 02:49:53 +00:00
Timo Tijhof
46b04ec7ae Use static::class instead of get_called_class()
Available as of PHP 5.5 and more idomatic. Foo::class (explicit),
self::class (defined), and static::class (late bound).

Change-Id: I66937f32095a4e4ecde94ca20a935a3c3efc9cee
2016-02-29 22:43:58 +00:00
Bartosz Dziewoński
c161c46d26 Improve code suffering from PHP 5.3's lack of support for foo()[]
I searched for /\$(\S+) = (.+?\(.*?\);)\n.*?\$\1\[/, ignored
everything involving isset(), unset() or array assigments, then
skimmed through the remaining results and changed things where they
made sense. These changes were not automated, so please review them.

Change-Id: Ib37b4c66fc57648470f151ad412210b3629c2538
2016-02-28 22:49:20 +01:00
Bryan Davis
2bcf34421a CachedBagOStuff: cache backend misses
Cache misses from the backend cache the same as hits.

Bug: T127772
Change-Id: If2fe1920411b24862acea888c627db13717da8bd
2016-02-22 18:45:20 -07:00
Brad Jorsch
405c2c8b6b Fix session store logging
Two things were wrong here:
1. SessionManager shouldn't override the configured logger of the
   underlying store.
2. CachedBagOStuff shouldn't override the configured logger of the
   backend store on construction.

Change-Id: I24ed5a81d5b238a42934a7890dd8d0a9f9bb3b6f
2016-02-22 17:13:03 +00:00
jenkins-bot
aeb7882171 Merge "CSSMin: Make isRemoteUrl and isLocalUrl really private, now that we can in PHP 5.5" 2016-02-18 08:21:35 +00:00
Bartosz Dziewoński
1decdbce53 CSSMin: Make isRemoteUrl and isLocalUrl really private, now that we can in PHP 5.5
Well, protected, because we want to have unit tests for them, apparently.

Change-Id: I734b88599e5860aa59a07a89cc5389eb73b48813
2016-02-17 15:50:08 +01:00
Kunal Mehta
b9668d6d80 build: Update mediawiki-codesniffer to 0.6.0, add "composer fix"
* Fix errors spotted by new release
* Introduce "composer fix", which uses phpcbf to automatically fix some
errors spotted by phpcs.
* Drop $PHPCS_ARGS variable that didn't work on Windows, and add -s flag
* Remove rules from phpcs.xml that are now in MW-CS ruleset.

Change-Id: I13e2155695918c918b67497ac65b85a03897095e
2016-02-17 02:54:42 -08:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
jenkins-bot
ca8aa5a6a6 Merge "Remove extra line breaks in memcached debug output" 2016-02-11 04:05:51 +00:00
Ricordisamoa
60be0296c3 Stop doing $that = $this in includes/libs
Closures support $this as of PHP 5.4

Change-Id: I1b5a5d7e619029684cb8d2a8d150fcc13051c2e0
2016-02-10 18:26:25 +01:00
Max Semenik
b41307acc0 Drop FastStringSearch support from ReplacementArray
It's not needed after a PHP 5.5 switch

Change-Id: I989f39d59878039d8cc320af957a90982e48d08d
2016-02-10 08:16:33 +00:00
Tim Starling
a57eb53907 Remove extra line breaks in memcached debug output
Possibly introduced by ed90263776

Change-Id: Ibbead449289eb1ffeec737c4c520925542b2fcfc
2016-02-08 09:01:30 +11:00
Brad Jorsch
25dbd91513 Clean up after Ie161e0f
Ie161e0f was done in a hurry, and so didn't do things in the best ways.
This introduces a new "CachedBagOStuff" that transparently handles all
the logic that had been copy-pasted all over in Ie161e0f.

The differences between CachedBagOStuff and MultiWriteBagOStuff are:
* CachedBagOStuff supports only one "backend".
* There's a flag for writes to only go to the in-memory cache.
* The in-memory cache is always updated.
* Locks go to the backend cache (with MultiWriteBagOStuff, it would wind
  up going to the HashBagOStuff used for the in-memory cache).

Change-Id: Iea494729bd2e8c6c5ab8facf4c241232e31e8215
2016-02-03 21:45:18 +00:00
Timo Tijhof
419db58c8b Centralise url handling for urls to static resources
Keep in CSSMin as-is for back-compat and to ensure library remains
independent of MediaWiki.

Moved down a few lines as there is no need to compute the md5 hash when we're
returning a data URI. Previously md5_file was called twice during module builds
(once for the fallback url, and another time when producing the embedded data uri).

Applied to logo in SkinModule as example. To be applied elsewhere as needed.
Without it, fallback is current behaviour (no cache invalidation).

Bug: T99096
Change-Id: I7f38bfc1bea5c241bc4f8ec4f4b640fd65f2c04f
2016-02-02 23:26:16 +00:00
Florian
9dec78d4c0 convertExtensionToRegistration: Detect if composer autoloader is needed
If an extension specifies a require section in its composer.json, which contains
some dependencies, we can assume that this extension needs to load the composer
autoloader.

The maintenance script convertExtensionToRegistration.php will now check if the
extension has such a composer.json and, if so, adds the load_composer_autoloader
property in extension.json, which will try to load the autoloader of composer
is it is present.

Also add a check for existence of a require section in ComposerJson library
(ComposerJson::getRequiredDependencies()). 

Bug: T119766
Change-Id: Icdbc37abc44e642afee2aab4c0e9298d3471124d
2016-01-07 17:23:34 +00:00
Aaron Schulz
9ea5cfe377 Make sure docs encourage __METHOD__ use for begin/commit
Change-Id: Id9c2d1b30f9dd758b418894b5751556daff20f0e
2015-12-23 17:23:15 -08:00
Aaron Schulz
5ffaa6ae0d Avoid "Unable to set value to APCBagOStuff" exceptions
* This can happen due to incr/add races. Use incrWithInit()
  instead to handle such cases.
* Also made BagOStuff:incrWithInit() return the new value like incr().

Change-Id: I0e3b02a4cff7c20544a9db2eaabd3f61e5a470b1
2015-12-21 20:41:43 -08:00
Roan Kattouw
e95721aae1 SECURITY: Work around CURL insanity breaking POST parameters that start with '@'
CURL has a "feature" where passing array( 'foo' => '@bar' )
in CURLOPT_POSTFIELDS results in the contents of the file named "bar"
being POSTed. This makes it impossible to POST the literal string "@bar",
because array( 'foo' => '%40bar' ) gets double-encoded to foo=%2540bar.

Disable this "feature" by setting CURLOPT_SAFE_UPLOAD to true,
if available. According to the PHP manual, this option became
available in 5.5 and started defaulting to true in 5.6.
However, we support versions as low as 5.3, and this option
doesn't exist at all in 5.6.99-hhvm, which we run in production.

For versions where this option is not available (pre-5.5 versions
and HHVM), serialize POSTFIELDS arrays to strings. This works
around the issue because the '@' "feature" only works
for arrays, not strings, as of PHP 5.2. (We don't support pre-5.2
versions, and I've verified 5.6.99-hhvm behaves this way as well.)

Bug: T118032
Signed-off-by: Chad Horohoe <chadh@wikimedia.org>
Change-Id: I3f996e2eb87c7bd3b94ca9d3cc14a3e12f34f241
2015-12-18 01:22:30 -08:00
Ori Livneh
ad3c2d0202 Work around APCu memory corruption bug
APCu's object serialization causes memory corruption when apc.serializer is set
to 'default' (see https://github.com/krakjoe/apcu/issues/38). We can work
around this bug by falling back to the pre-I4b2cf17155 behavior of
APCBagOStuff, which is not to trust apc_store() with anything other than
strings and integers, and instead serialize everything in user-space PHP code.

Bug: T120267
Change-Id: If34a1d959e2700792514b372af52919940222d83
2015-12-13 21:35:59 -08:00
Ori Livneh
54f9e1f159 Timing::measure(): handle missing marks better
Currently Timing::measure() does not check that the requested start and end
marks exist, causing it to return bogus values without any indication that
something has gone wrong. Fix this by logging and error and returning false
in case either the start or end markers do not exist. To make it possible to
log, make Timing implement the LoggerAware interface.

Change-Id: I75af5273e9a8a52b31d0af1de206b0d8a4c82fbc
2015-12-10 14:58:11 -08:00
Timo Tijhof
5d5b269e0e resourceloader: Migrate from msg_resource table to object cache
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
  - Batch retrieval.
  - Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
  part of get() instead of within updateMessage().

ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
  Timestamps are unreliable and roll over too frequently for message blob store
  because there is no authoritative source. The timestamps were inferred based on
  when a change is observed. Message overrides from the local wiki have an
  explicit update event when the page is edited. All other messages, such as
  from MediaWiki core and extensions using LocalisationCache, have a single
  timestamp for all messages which rolls over every time the cache is rebuilt.
  A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
  This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
  Instead, detect whether a module's message list is empty at runtime.

ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.

cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.

Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
2015-12-08 14:02:38 +00:00
Timo Tijhof
8065936ec3 objectcache: Make protected WANObjectCache::makePurgeValue non-static
It was already used everywhere as non-static via $this.
This is needed in order to allow MessageBlobStore unit tests
to disable the holdoff via a mock (mocks can't override static methods).

Change-Id: I3aad5b6e780addf1b6ce9de56c81b91f5ab358b2
2015-12-06 14:34:03 +00:00
Aaron Schulz
ba06ccb394 Fix get()/getMulti() check key race condition in WANObjectCache
If a set() happened around the exact same time as check key was
initialized via get(), the curTTL in future get() calls could
sometimes be 0 instead of a negative value, even though hold-off
should apply.

Change-Id: Ide1fd65112aff425a4798e2ec406d71f2a8e84a7
2015-12-04 14:46:18 -08:00
addshore
b796300aa6 Fix phpdoc of StatusValue::merge
Change-Id: I6fec36f9911d647fa7507c36b57f6b11ef04d63d
2015-12-04 21:50:22 +01:00
jenkins-bot
df629219aa Merge "Make getCacheSetOptions() and WAN cache handle broken replication" 2015-12-03 12:24:37 +00:00
Aaron Schulz
cb8842dbab Make getCacheSetOptions() and WAN cache handle broken replication
TTL_LAGGED now triggers if the slave or I/O threads stopped

Change-Id: I5e7bf2d33b8d3a60182ec53a93d65f7e55f02222
2015-12-02 15:38:23 -08:00
jenkins-bot
cd42bfbe24 Merge "Fix relayPurge() IDEA warnings" 2015-12-02 22:41:43 +00:00
Aaron Schulz
2f1f8d9bf2 Fix relayPurge() IDEA warnings
Change-Id: I6e6c0f611c344eab79350468552a2ccbf4cd1d18
2015-12-02 14:31:51 -08:00
Timo Tijhof
ff1bd769ef objectcache: Add $holdoff parameter to WANObjectCache::touchCheckKey()
Change-Id: I14b6d7660b34271826b77875c660c34343712648
2015-12-02 21:33:05 +00:00
Timo Tijhof
4b7f59bdcc objectcache: Move WANObjectCache holdoff from get() to purge value
Move the holdoff period into the purge value instead of deciding
it at runtime. This opens the way for touchCheckKey() to support
a custom $holdoff parameter, which will allow callers to invalidate
keys without a holdoff period. Similar to what we already support
in delete().

Right now the holdoff period is decided at run time.

Change-Id: Id10c036272e92ae4429effc823b75e08fb11a48b
2015-12-02 21:33:04 +00:00
addshore
40c8a710be Fix packet reduction in SamplingStatsdClient
Doc block has also been changed to show that
this implementation does not follow the defined
interface.
It also says why...

Bug: T116031
Change-Id: If8cca355ae77b751486b40e56a6b3a51b1d7fe0b
2015-11-27 20:49:46 +01:00
addshore
80022b638e Enhance SamplingStatsdClient::sampleData return doc
Change-Id: Idde67e9514fdf736881cb42b6413045f7bc6f3b3
2015-11-27 19:21:52 +01:00
jenkins-bot
8b0a2d80e0 Merge "StringUtils: Fixup Doxygen warnings" 2015-11-24 19:57:19 +00:00
Aaron Schulz
680f152715 Add 0 hold-off TTL support to WANObjectCache::delete()
This removes the peculiar >= 1 restriction

Change-Id: I255dc6ef9750ad2ed6234e3fcfed9ae22a038748
2015-11-23 21:46:36 -08:00
Gergő Tisza
10cf632683 Disallow associative arguments in ObjectFactory
There is no strong use case for associative "decoration"
of constructor arguments (the documentation benefits
mentioned in I43aa085 are outweighed by the confusion
caused by not failing loudly when someone passes an
associative argument by accident, e.g. by omitting an
array nesting level), so disallow them but make sure
they fail nicely, not with an invalid offset error.

Change-Id: I09e4af85ded6a1497b0db0265d2ee6707f91f5e3
2015-11-24 00:10:42 +00:00
Gergő Tisza
20b7696494 Support named arguments in ObjectFactory
Arguments are sometimes given as array( 'foo' => 1, 'bar' => 2 )
which makes the configuration self-documenting.

Change-Id: I43aa085090f1014ba841641867ebf9559d16e76d
2015-11-23 23:04:08 +00:00
Timo Tijhof
95d5448ad5 StringUtils: Fixup Doxygen warnings
* Code blocks must be indented by 4 spaces (so 5 in total from the star).

* Code blocks are only supported in descriptions, and for some reason the first
  line is separate from the description, so there must be an additional line of
  text between the summary and the code block since it can't be the first thing
  in a description.

* Fix regex confusion with comment syntax due to "/*\/".

To verify these findings, use:

    $ php maintenance/mwdocgen.php --file includes/libs/StringUtils.php

And check the output warnings and generated html pages.

Change-Id: I259b87c0bf3b48a95df0108eb6ee799c25b2853f
2015-11-23 21:11:21 +00:00
Timo Tijhof
a0cce5e4b6 objectcache: Implement check keys per cache key in WANObjectCache::getMulti()
To allow batch queries for multiple keys that themselves have different check
keys. Previously check keys always applied to all keys being retrieved.

Change-Id: I9e5ba198d79020ce05a802a510762e29fcfb2f1b
2015-11-17 04:15:33 +00:00
Timo Tijhof
304eaf2868 objectcache: Reduce MediaWiki-specificness from Memcached classes
Follows-up f36b73e96c, which moved these classes to libs/objectcache.

* Fix wrong @throws in MemcachedPeclBagOStuff.
* Fix wrong @returns in MemcachedBagOStuff::getClient().
* Rename MWMemcached to MemcachedClient.
* Remove mention of 'memcached.php', which doesn't exist anymore.

Change-Id: I34dbc859be4778cea489fd2344f233f30452605c
2015-11-10 03:32:58 +00:00
Aaron Schulz
209217da8a objectcache: Make HashBagOStuff LRU instead of least-recently-set
Refresh key in get() in addition to just set().

Change-Id: I7985b98b6a346eaed8bf0a7349b95fabea8e8614
2015-11-04 01:30:39 +00:00
Ori Livneh
8340400f27 Add Timing interface
* Add `Timing`, an interface which mimics the W3C User Timing API.
  It provides a canonical way to store and retrieve markers (timestamps)
  and measures (timestamps + duration).
* As the initial use-case, use it to record 'requestShutdown'.

Change-Id: I36b29162ffcc091406df025463b0e2797e52f19a
2015-11-03 15:05:37 -08:00
Aaron Schulz
5b3ad3a46b Merge "objectcache: Add a clear() method to HashBagOStuff" 2015-11-03 10:16:19 +00:00
jenkins-bot
fb5516524c Merge "objectcache: Refresh key in HashBagOStuff::set() for maxKeys eviction" 2015-11-03 10:13:08 +00:00
Timo Tijhof
caa3e2ea06 objectcache: Add a clear() method to HashBagOStuff
Including tests for delete() and clear().

Change-Id: If39d729838e312523f0df3ae8b235ebe939a17fd
2015-11-03 10:07:10 +00:00
Timo Tijhof
4ec550812b objectcache: Refresh key in HashBagOStuff::set() for maxKeys eviction
* Match behaviour of MapCacheLRU and ProcessCacheLRU.
* Add missing unit tests for TTL and maxCacheKeys eviction behaviour.

Change-Id: I559eae1cd336274b21728e86775cfbad7e2f2c6d
2015-11-03 01:59:30 -08:00
Timo Tijhof
03b281d55b objectcache: Add TTL_INDEFINITE to IExpiringStore
Especially useful when interacting with getWithSetCallback of
HashBagOStuff where a $exptime is required before the callback.

In set() it is typically left out and thus not as counter-intuitive.

Change-Id: Ic2f7adda3c00cefe701d77bea91a7e8e77ef1439
2015-11-03 06:36:06 +00:00
Aaron Schulz
d2d935483f Make makeKeyInternal() limit more conservative
This should avoid error log entries for long WAN cache keys

Change-Id: I401482d25dd5bf47052a3c6729c5f8bc9fd68770
2015-11-01 19:37:21 -08:00
Aaron Schulz
7cddc22fb8 objectcache: Allow bounded HashBagOStuff sizes and limit it in WANObjectCache
Change-Id: Icca2474b1ea6feb7134f8958aecf79aa51b7f71e
2015-10-31 02:04:01 +00:00