Commit graph

372 commits

Author SHA1 Message Date
Timo Tijhof
a4fa0229c1 Add unit tests for MessageBlobStore
* Also added getBlob() and getBlobs() methods. These don't
  require passing ResourceLoader as argument.
  In preparation for T113092.

Change-Id: I3239c45f0243eca1eaf85bc53fca736e4c9a209c
2015-11-21 00:09:47 +00:00
Timo Tijhof
53bfdc624d MessageCache: Remove $ucfirst and clean up getMessageFromFallbackChain()
The $ucfirst and $wgContentLang variables are not used by get().
Keep these inside getMessageFromFallbackChain() instead.

Also add mark getMsgFromNamespace() as public since it is used
in other places in core and by extensions.

Change-Id: I83bf77c17aa46f3929f89c4af97693611761629b
2015-11-17 18:41:10 +00:00
Timo Tijhof
4020bf947b resourceloader: Update MessageBlobStore documentation and code cleanup
* Update and improve class documentation.

* Remove comment that claims to return false if the module has no message.
  generateMessageBlob() returns '{}' in that case. And this is expected since
  we want to cache the absence of messages.

* Re-use the module objects passed to MessageBlobStore::get() in getFromDB()
  instead of using ResourceLoader::getModule(). The object is the same either
  way (since ResourceLoader re-uses the objects also) but reduces complexity
  of getFromDB() to not need the ResourceLoader object as parameter.

Change-Id: I89a14d7185877fae52791f6837883ed3a6749cd7
2015-11-13 00:41:57 +00:00
Timo Tijhof
976f43327d resourceloader: Remove deprecated MessageBlobStore::getInstance()
Deprecated since MediaWiki 1.25.
Not used anywhere in public Wikimedia Git.

Change-Id: Iaa31d0e83b95fb033c62430bc3fc4dd66af16906
2015-11-12 00:24:38 +00:00
Timo Tijhof
e53bf8bdcb resourceloader: Log if MessageBlobStore failed to fetch a message
Bug: T93800
Change-Id: I6f174251a6b9a6b31921037f672dbc197f3abb7e
2015-11-10 04:40:31 +00:00
Timo Tijhof
2d3ea7996f MessageCache: Remove redundant Language::ucfirst() optimisation
Language::ucfirst() already has this optimisation. The expensive
method is Language::uc(), which is only called when needed.

The ord() < 128 is one of the conditionals in Language::ucfirst().

Change-Id: Idd648b7b84eba1f92dda904438122fe38d3cf22b
2015-11-10 00:41:45 +00:00
Timo Tijhof
e6205019a4 BacklinkCache: Remove use of ProcessCacheLRU
Follows-up af89b09a06 and e40a90f0bc, and general phasing out
of MapCacheLRU and ProcessCacheLRU.

Use of ProcessCacheLRU seems redundant in this case as it is
essentially just a keyed singleton.

Change-Id: I7cc84cf6c76ecc422ea337dba2d216c3d7ebf281
2015-11-10 00:06:09 +00:00
Timo Tijhof
03a620004b LinkCache: Convert from MapCacheLRU to HashBagOStuff.
Change-Id: I28a9509ad4c6d724a12dc72bc50d3cf58642c651
2015-11-09 23:17:47 +00:00
Timo Tijhof
dcf8ec0283 LinkCache: Minor clean up of documentation and variable names
In preparation for conversion from MapCacheLRU to HashBagOStuff.

* Use early-return pattern for error and rejection cases.
* Add 'int' to @return for getGoodLinkFieldObj() per the
  values set in addGoodLinkObj().
* Add type hints for Title where possible.
* Avoid single-letter variables like $f and $s.
* Add visibility for public static methods.

Change-Id: Iab90539922f648c21cb147f749943a64f598e8de
2015-11-09 23:17:43 +00:00
jenkins-bot
0bb4b4312e Merge "Convert LCStoreDB to using startAtomic/endAtomic" 2015-11-09 20:08:02 +00:00
Aaron Schulz
ef9f0ed8a1 Convert LCStoreDB to using startAtomic/endAtomic
* This avoids throwing atomic section errors when recache()
  is triggered in the middle of transactions.
* Also made the read-only checks handle $wgReadOnly mode.

Change-Id: Ie00cf8454656a6dc3b1862475a959b2af3b472f0
2015-11-09 19:54:07 +00:00
Timo Tijhof
70f8a31afe resourceloader: Remove use of msg_resource_links table
This table is not needed because module names and their messages
array are available to the runtime environment at very little cost.

The only purpose it was serving is reverse lookup from message
key to module name (e.g. when MessageCache receives update that
need to propagate to MessageBlobStore). However that is better
achieved by simply looping through modules in PHP. The overhead
of a database is not worth this minor convenience.

MessageBlobStore
* insertMessageBlob: Doesn't need to update msg_resource_links.
* updateModule: Doesn't need to update msg_resource_links.
* getUpdatesForMessage: Reimplement with list from memory
  instead of msg_resource_links.

The database table will be removed in If009e2620e59002e1.

Bug: T113092
Change-Id: Ia9131f570001f00c9800b260ac4b3469d54d2784
2015-11-09 19:16:38 +00:00
Ori Livneh
36171312ef LocalisationCache: try harder to use LCStoreCDB
When the LocalisationCache class is set to 'detect', we prefer LCStoreCDB if
$wgCacheDirectory is set, but we default to LCStoreDB otherwise. Rather than
give up, we should try wfTempDir(), since any directory that meets its criteria
is also suitable for LCStoreCDB.

Change-Id: Id3e2d2b18ddb423647bf2e893bcf942722c0e097
2015-11-05 17:48:05 -08:00
Aaron Schulz
ac1325aa43 Improve BacklinkCache field docs
Change-Id: I8303d5577ac84b8882e5ad79ac3058d6bf49356f
2015-11-03 02:56:30 -08:00
Timo Tijhof
8b463047a9 LinkCache: Remove deprecated getBadLinks and getGoodLinks methods
Deprecated since MediaWiki 1.26 (32d1017e7d). These methods contain the
only remaining callers of MapCacheLRU::getAllKeys() which is the only odd
method of MapCacheLRU not supported by BagOStuff.

Change-Id: I64e054803595d49a5bd779accc94353924f7d87d
2015-11-03 06:08:36 +00:00
jenkins-bot
dad19d999c Merge "Convert SiteStatsUpdate to using getMainStashInstance()" 2015-11-02 22:47:10 +00:00
Aaron Schulz
17c91ad610 Replace newAccelerator() with getLocalServerInstance()
The name is clearer and more consistent, with simpler arguments.

Change-Id: I7205a99ce033e8b086a52cd02c8a721e99c84b1e
2015-11-02 21:39:08 +00:00
Aaron Schulz
6cc65335f3 Convert SiteStatsUpdate to using getMainStashInstance()
Also fixed various $wgMemc related comments

Change-Id: I20602b672f724c8df1e82bbe3c586cb899a54640
2015-10-31 13:05:03 -07:00
Kevin Israel
eb8d6eb754 Update description of LinkCache::forUpdate()
The $mForUpdate flag no longer causes the FOR UPDATE option to be added.
This was at first only true for some values of $wgAntiLockFlags (since
r9248 / ba8a00bcb833), though since 12757b50f8 (when the setting was
removed), is always the case.

The added text is based on the description for $wgAntiLockFlags.

Change-Id: I02a3e2df37ec40c7e36ae6210c8263f13d9e3e5b
2015-10-30 06:09:29 -04:00
jenkins-bot
f84ebddc13 Merge "Convert buildSidebar() to using getWithSetCallback()" 2015-10-29 08:04:32 +00:00
Aaron Schulz
141197dd27 Convert buildSidebar() to using getWithSetCallback()
Change-Id: Id9a27ba2bbd3aceee26bf35844d1c970dbb32d47
2015-10-28 18:24:31 -07:00
Timo Tijhof
afcfc3290c resourceloader: Consistently refer to the framework as ResourceLoader
Change-Id: Ia59e4eac9662723e80d62f7cfcb9e4292e3ee4de
2015-10-28 03:24:40 +00:00
Aaron Schulz
92bbfd5ea6 Migrate more callers away from $wgMemc
Callers should use more expliciy lazy-loaded
cache factory methods.

Change-Id: Ifa0bf389720a09a931ee6466b993f787d83a09a7
2015-10-26 19:07:12 +00:00
Timo Tijhof
445136b5bf resourceloader: Remove CacheEpoch from MessageBlobStore::getFromDB
CacheEpoch isn't meant for this (it's meant for page cache), I can't
imagine a scenario in which we'd want to bump that to invalidate
MessageBlobStore. It should be standalone and can be easily cleared
if needed by truncating the relevant table (it's automatically
repopulated).

This also removes wfTimestamp/DateTime overhead.

Change-Id: Iab06edbf71f20f3430207a80df90131c79dc03a7
2015-10-21 02:40:06 +00:00
Aaron Schulz
4f781c7bf9 Convert some users to WANObjectCache for consistency
These callers don't need to do purges, but can still perfectly
take advantage of this instance over a plain BagOStuff. Namely:
* Replication and snapshot lag awareness
* Preemptive regeneration
* Easy process cache support

The idea is for there to only be one caching class/factory
to use, instead of having rules for picking which one to use.

Change-Id: I8e362df451c0c28731fc853c044c4c4b8e097f01
2015-10-19 16:46:52 +00:00
Timo Tijhof
68a54d6d7a objectcache: Rename WANObjectCache TTL_NONE to TTL_INDEFINITE
"None" has a somewhat unobvious meaning, also ambiguous with "uncachable".

Change-Id: I799de385427aeb8f581d51800606552bdc998252
2015-10-14 17:08:59 -04:00
Kevin Israel
fd7a480402 LocalisationCache::recache(): Use array_fill_keys()
LocalisationCache was added in 1.16, so in order to retain PHP 5.1
compatibility, array_fill_keys() was avoided. In 1.17, support for
PHP 5.1 was dropped, so we can use that function now.

Change-Id: I435705639f1a470324a4ba46153351aadc0d40e2
2015-10-08 16:29:16 -04:00
jenkins-bot
1569779421 Merge "Update various @params from DatabaseBase to IDatabase" 2015-10-05 19:59:21 +00:00
Thiemo Mättig
56d2a644f7 Add null to @return tags if a method can return null
Change-Id: I420998351663d92c4a101f61842e40591eebcd5f
2015-10-05 17:53:13 +02:00
Aaron Schulz
0f7893f877 Update various @params from DatabaseBase to IDatabase
Change-Id: I98e44cdffb0fc0d729f69f702799139afb988c20
2015-10-05 05:24:29 +00:00
Siebrand Mazeland
4e582a507d Fix 3 PHPCS warnings in MessageCache.php
Change-Id: I0970b8844fa89d3bdc4fec200d912ecf80a52176
2015-09-28 13:18:54 +02:00
Kevin Israel
3f516b5139 Remove unused private property LinkCache::$mClassVer
This was added in r2084 / c280ef342b for a "persistent link cache"
feature, which became obsolete in 1.4 and was removed in 1.5.

https://www.mediawiki.org/wiki/Manual:$wgEnablePersistentLC

Follows-up r9276 / cb45389b9c.

Change-Id: I2f27d6acc1a1f9a9382a47d56921f570107a7be9
2015-09-25 07:54:02 -04:00
Niklas Laxström
4a3fd2e42a Use wikimedia/cldr-plural-rule-parser
Replaces the parser included in MediaWiki with same code in
a library.

Change-Id: I1d2675466a543269e17faf213aa68d2b7afaf78e
2015-09-24 21:41:50 +02:00
Aaron Schulz
4eea6b5d71 Avoid PHP warning if key is not set in getValidationHash()
* Follow up to 2be60e777

Change-Id: Ic5c27226bc3f5cc870728a925b504a0dcbedaefb
2015-09-04 00:43:55 +00:00
Kunal Mehta
4b7ba45d0c MessageCache: Don't try to log a ContentHandler object
Change-Id: I000859370a8588b93682221d1e9fb0a2b11a4670
2015-09-02 23:16:44 -07:00
Aaron Schulz
2be60e777a Avoid MessageCache rebuilds if replace() was called recently
* If replace() was called recently, then we know that this is the
  master data center and that the messages are up-to-date. With this
  change, replace() calls have nothing to contend with aside from other
  replace() calls. Even if there were timeouts due to such contention,
  caused by high MediaWiki: page edit rates, the replace() updates would
  pick up the prior changes in passing since they do load().
* This also avoids the following scenario:
  a) Someone edits a message page, triggering replace()
  b) Some page view causes load() to trigger loadFromDB()
     due to the hash being seen as volatile due to replace()
  c) The loadFromDB() loads stale slave data and undoes
     the message key update the replace() did

Change-Id: I9cdf7ad3d67f168fcba7f633af9e32a8d1fa928e
2015-09-03 01:06:22 +00:00
Aaron Schulz
89a167b500 More MessageCache locking/update cleanups
* Made the status key only act as a backoff key inside
  loadFromDBWithLock(), instead of also being a lock key. The
  getReentrantScopedLock() call is now non-blocking, which has
  a similar affect to the add() call on the status key but much
  better prioritizes replace(); that method already has a blocking
  getReentrantScopedLock() call, so once it gets that lock, there
  is no worry of load() failing due to contention.
* Bail out in loadFromDBWithLock() if the scope lock was not acquired.
  Normally the status lock often assured the lock could be obtained,
  unless a competing replace() came in. The old status lock would
  cause a bail if not acquired, and this carries over that behavior
  but also avoids clobbering updates when replace() contention happens.
* Avoid calling saveToCaches() in replace() if the lock was not
  acquired to avoid clobbering conflicting writes.
* Made the loadFromDB() call in replace() use DB_MASTER to avoid
  seeing stale data if the cache is volatile.
* Lowered WAIT_SEC, as the default PHP timeout is 30.
  We want this to be able to at least finish the other calls
  in replace() even if the lock times out.
* Avoid checking the status key an extra time in load().
* Made a few small code and doc cleanups.

Change-Id: Ibaf1f67618ec374c83c3135a71e90223dd2b1856
2015-09-02 22:41:18 +00:00
Aaron Schulz
3adf586c78 Removed READ_LATEST from revision load in getMsgFromNamespace()
* The key salting of the invidial keys via the WAN cache makes
  it OK to use slaves here. The salting happens via the delete()
  in the MessageCache::replace() call, which is hit when the
  message page is edited.

Bug: T92357
Change-Id: Ic191f6c3fc49c4c58461d26468dd0fa94a52051e
2015-08-31 20:20:45 +00:00
Aaron Schulz
846769fad1 Made MessageCache use the WAN cache for individual cache keys
* These keys are used for larger messages not in the main blob,
  and they still use explicit purges as the pages change

Change-Id: I254ddcc9ffe7c49654788d6aabcca2ab7ed4f03f
2015-08-28 23:25:48 +00:00
Aaron Schulz
6cdbf82be3 Removed useless $wgUseLocalMessageCache check in MessageCache
* The localCache member will use CACHE_NONE if this is off,
  so the set()/get() calls are already no-oped
* Also allow using the local cache if $hash is false
  in "stale mode", which handles the case where the hash
  key fell out of memory for some reason more gracefully

Change-Id: Ie12efcd4088a6dc4a4cdd2fd06646f2881df53d7
2015-08-27 18:31:35 -07:00
Aaron Schulz
5806931375 Cleaned up int|INF doc type to avoid IDEA errors
* INF is not a class (technically a float, which would look confusing here)

Change-Id: I9c437ecd0f2fe1a2a3e15197d50829a638f3073a
2015-08-27 18:18:59 +00:00
Aaron Schulz
a88e9ba007 Avoid self-deadlocks in MessageCache::replace()
* This makes use of the $rclass flag in BagOStuff

Bug: T109183
Change-Id: I305f51e744aac53876e5865f860c282aa2efbd8b
2015-08-25 20:04:19 +00:00
Timo Tijhof
79ce51d1c9 objectcache: Make first parameter of newAccelerator optional
Makes it more convenient to use.

Change-Id: I1e11f7a759bd2816e47d1c2453cbe39b8f44b2f0
2015-08-24 23:43:19 +02:00
Aaron Schulz
b8f231420c Use stock BagOStuff lock methods in MessageCache
Change-Id: Ic77b1b0f742590a10cddc7db067a4fcff246540f
2015-08-19 12:27:20 -07:00
Aaron Schulz
6d8dd29c59 Always call setValidationHash() in MessageCache::saveToCaches()
* This avoids constant churn when $wgUseLocalMessageCache is set to false
* Follow up to db464b8a84

Bug: T109183
Change-Id: I8da324c53527da32d09964be6c3a92176af4ee7b
2015-08-15 20:17:52 +00:00
Ori Livneh
6147de1182 MessageCache: derive the hash from the cache contents
Follow-up for I020617d, where I got this wrong. It is probable that multiple
application servers will try to build a local cache at the same time, in which
case they will cause the cache key to thrash. It is necessary that two
application servers building a local cache from the database generate the same
hash for a given set of messages.

Change-Id: Ieeefc2094a83be9401c466bec859c1588ddfbcdf
2015-08-14 19:49:07 +00:00
Aaron Schulz
db464b8a84 More multi-DC tweaks to MessageCache::load()
* Use hash key volatility to propagate invalidations over DCs
  in addition to memcached->APC instances.
* Make use of stale APC cache due to hash mismatch instead
  of waiting around in some cases. This is similar to the
  TTL expiry and volatility cases.
* Renamed $hashExpired and added some comments for clarity.

Change-Id: I8890fb174cae72c4ce8872df64f52f5cbd55183b
2015-08-14 15:49:53 +00:00
Ori Livneh
3a252efa12 MessageCache: use APC for local caching, rather than files
In addition to eliminating disk IO in a hot path, using APC spares us from
having to serialize and unserialize cache arrays. Since we're not serializing,
though, we don't have a string representation to hash, so use a random string
instead. (The code already treats the association of hash string to cache as
purely symbolic, so this is not problematic.)

Whereas the hash was previously stored as the first 32 bytes of each cache
file, we now store it as an array key instead (like VERSION and EXPIRY were
already). Because this changes the structure of cached data, we have to bump
MSG_CACHE_VERSION.

While we're here, make MessageCache::getLocalCache() and
MessageCache::saveToLocal() protected, make their signatures more
consistent with other methods in this class. While they were (implicitly)
public before, there are absolutely no external callers in Core or
extensions[0][1], so we can skip the standard deprecation process.

[0]: https://github.com/search?q=%40wikimedia+getlocalcache&type=Code&utf8=%E2%9C%93
[1]: https://github.com/search?utf8=%E2%9C%93&q=%40wikimedia+savetolocal&type=Code

Change-Id: I020617d2df2a8f0f243b85f3383dc7b16f15aaad
2015-08-10 12:39:11 -07:00
Kunal Mehta
32d1017e7d Don't let LinkCache grow indefinitely
Especially in long running maintenance scripts, this can be problematic.

LinkCache is now LRU-based, and will store a maximum of 10,000 good titles,
and 10,000 bad ones.

LinkCache::getGoodLinks() and getBadLinks() are deprecated since they
problematic to support in this implementation and are unused.

Bug: T106998
Change-Id: I1328149d65a5e75a5d6e10cb2686a099562a1847
2015-07-27 10:26:46 -07:00
Brian Wolff
cf501475fd Fix fatal when non-ascii message key is normalized
Missing global declaration when using $wgContLang in
MessageCache::normalizeKey

Change-Id: Ia1c1f41244fd5629527b99a5f2038f607cff42c4
Follow-up: 47e0f0c3
2015-07-19 10:01:55 +00:00