Commit graph

563 commits

Author SHA1 Message Date
jenkins-bot
0bb4b4312e Merge "Convert LCStoreDB to using startAtomic/endAtomic" 2015-11-09 20:08:02 +00:00
Aaron Schulz
ef9f0ed8a1 Convert LCStoreDB to using startAtomic/endAtomic
* This avoids throwing atomic section errors when recache()
  is triggered in the middle of transactions.
* Also made the read-only checks handle $wgReadOnly mode.

Change-Id: Ie00cf8454656a6dc3b1862475a959b2af3b472f0
2015-11-09 19:54:07 +00:00
Timo Tijhof
70f8a31afe resourceloader: Remove use of msg_resource_links table
This table is not needed because module names and their messages
array are available to the runtime environment at very little cost.

The only purpose it was serving is reverse lookup from message
key to module name (e.g. when MessageCache receives update that
need to propagate to MessageBlobStore). However that is better
achieved by simply looping through modules in PHP. The overhead
of a database is not worth this minor convenience.

MessageBlobStore
* insertMessageBlob: Doesn't need to update msg_resource_links.
* updateModule: Doesn't need to update msg_resource_links.
* getUpdatesForMessage: Reimplement with list from memory
  instead of msg_resource_links.

The database table will be removed in If009e2620e59002e1.

Bug: T113092
Change-Id: Ia9131f570001f00c9800b260ac4b3469d54d2784
2015-11-09 19:16:38 +00:00
Ori Livneh
36171312ef LocalisationCache: try harder to use LCStoreCDB
When the LocalisationCache class is set to 'detect', we prefer LCStoreCDB if
$wgCacheDirectory is set, but we default to LCStoreDB otherwise. Rather than
give up, we should try wfTempDir(), since any directory that meets its criteria
is also suitable for LCStoreCDB.

Change-Id: Id3e2d2b18ddb423647bf2e893bcf942722c0e097
2015-11-05 17:48:05 -08:00
Aaron Schulz
ac1325aa43 Improve BacklinkCache field docs
Change-Id: I8303d5577ac84b8882e5ad79ac3058d6bf49356f
2015-11-03 02:56:30 -08:00
Timo Tijhof
8b463047a9 LinkCache: Remove deprecated getBadLinks and getGoodLinks methods
Deprecated since MediaWiki 1.26 (32d1017e7d). These methods contain the
only remaining callers of MapCacheLRU::getAllKeys() which is the only odd
method of MapCacheLRU not supported by BagOStuff.

Change-Id: I64e054803595d49a5bd779accc94353924f7d87d
2015-11-03 06:08:36 +00:00
jenkins-bot
dad19d999c Merge "Convert SiteStatsUpdate to using getMainStashInstance()" 2015-11-02 22:47:10 +00:00
Aaron Schulz
17c91ad610 Replace newAccelerator() with getLocalServerInstance()
The name is clearer and more consistent, with simpler arguments.

Change-Id: I7205a99ce033e8b086a52cd02c8a721e99c84b1e
2015-11-02 21:39:08 +00:00
Aaron Schulz
6cc65335f3 Convert SiteStatsUpdate to using getMainStashInstance()
Also fixed various $wgMemc related comments

Change-Id: I20602b672f724c8df1e82bbe3c586cb899a54640
2015-10-31 13:05:03 -07:00
Kevin Israel
eb8d6eb754 Update description of LinkCache::forUpdate()
The $mForUpdate flag no longer causes the FOR UPDATE option to be added.
This was at first only true for some values of $wgAntiLockFlags (since
r9248 / ba8a00bcb833), though since 12757b50f8 (when the setting was
removed), is always the case.

The added text is based on the description for $wgAntiLockFlags.

Change-Id: I02a3e2df37ec40c7e36ae6210c8263f13d9e3e5b
2015-10-30 06:09:29 -04:00
jenkins-bot
f84ebddc13 Merge "Convert buildSidebar() to using getWithSetCallback()" 2015-10-29 08:04:32 +00:00
Aaron Schulz
141197dd27 Convert buildSidebar() to using getWithSetCallback()
Change-Id: Id9a27ba2bbd3aceee26bf35844d1c970dbb32d47
2015-10-28 18:24:31 -07:00
Timo Tijhof
afcfc3290c resourceloader: Consistently refer to the framework as ResourceLoader
Change-Id: Ia59e4eac9662723e80d62f7cfcb9e4292e3ee4de
2015-10-28 03:24:40 +00:00
Aaron Schulz
92bbfd5ea6 Migrate more callers away from $wgMemc
Callers should use more expliciy lazy-loaded
cache factory methods.

Change-Id: Ifa0bf389720a09a931ee6466b993f787d83a09a7
2015-10-26 19:07:12 +00:00
Timo Tijhof
445136b5bf resourceloader: Remove CacheEpoch from MessageBlobStore::getFromDB
CacheEpoch isn't meant for this (it's meant for page cache), I can't
imagine a scenario in which we'd want to bump that to invalidate
MessageBlobStore. It should be standalone and can be easily cleared
if needed by truncating the relevant table (it's automatically
repopulated).

This also removes wfTimestamp/DateTime overhead.

Change-Id: Iab06edbf71f20f3430207a80df90131c79dc03a7
2015-10-21 02:40:06 +00:00
Aaron Schulz
4f781c7bf9 Convert some users to WANObjectCache for consistency
These callers don't need to do purges, but can still perfectly
take advantage of this instance over a plain BagOStuff. Namely:
* Replication and snapshot lag awareness
* Preemptive regeneration
* Easy process cache support

The idea is for there to only be one caching class/factory
to use, instead of having rules for picking which one to use.

Change-Id: I8e362df451c0c28731fc853c044c4c4b8e097f01
2015-10-19 16:46:52 +00:00
Timo Tijhof
68a54d6d7a objectcache: Rename WANObjectCache TTL_NONE to TTL_INDEFINITE
"None" has a somewhat unobvious meaning, also ambiguous with "uncachable".

Change-Id: I799de385427aeb8f581d51800606552bdc998252
2015-10-14 17:08:59 -04:00
Kevin Israel
fd7a480402 LocalisationCache::recache(): Use array_fill_keys()
LocalisationCache was added in 1.16, so in order to retain PHP 5.1
compatibility, array_fill_keys() was avoided. In 1.17, support for
PHP 5.1 was dropped, so we can use that function now.

Change-Id: I435705639f1a470324a4ba46153351aadc0d40e2
2015-10-08 16:29:16 -04:00
jenkins-bot
1569779421 Merge "Update various @params from DatabaseBase to IDatabase" 2015-10-05 19:59:21 +00:00
Thiemo Mättig
56d2a644f7 Add null to @return tags if a method can return null
Change-Id: I420998351663d92c4a101f61842e40591eebcd5f
2015-10-05 17:53:13 +02:00
Aaron Schulz
0f7893f877 Update various @params from DatabaseBase to IDatabase
Change-Id: I98e44cdffb0fc0d729f69f702799139afb988c20
2015-10-05 05:24:29 +00:00
Siebrand Mazeland
4e582a507d Fix 3 PHPCS warnings in MessageCache.php
Change-Id: I0970b8844fa89d3bdc4fec200d912ecf80a52176
2015-09-28 13:18:54 +02:00
Kevin Israel
3f516b5139 Remove unused private property LinkCache::$mClassVer
This was added in r2084 / c280ef342b for a "persistent link cache"
feature, which became obsolete in 1.4 and was removed in 1.5.

https://www.mediawiki.org/wiki/Manual:$wgEnablePersistentLC

Follows-up r9276 / cb45389b9c.

Change-Id: I2f27d6acc1a1f9a9382a47d56921f570107a7be9
2015-09-25 07:54:02 -04:00
Niklas Laxström
4a3fd2e42a Use wikimedia/cldr-plural-rule-parser
Replaces the parser included in MediaWiki with same code in
a library.

Change-Id: I1d2675466a543269e17faf213aa68d2b7afaf78e
2015-09-24 21:41:50 +02:00
Aaron Schulz
4eea6b5d71 Avoid PHP warning if key is not set in getValidationHash()
* Follow up to 2be60e777

Change-Id: Ic5c27226bc3f5cc870728a925b504a0dcbedaefb
2015-09-04 00:43:55 +00:00
Kunal Mehta
4b7ba45d0c MessageCache: Don't try to log a ContentHandler object
Change-Id: I000859370a8588b93682221d1e9fb0a2b11a4670
2015-09-02 23:16:44 -07:00
Aaron Schulz
2be60e777a Avoid MessageCache rebuilds if replace() was called recently
* If replace() was called recently, then we know that this is the
  master data center and that the messages are up-to-date. With this
  change, replace() calls have nothing to contend with aside from other
  replace() calls. Even if there were timeouts due to such contention,
  caused by high MediaWiki: page edit rates, the replace() updates would
  pick up the prior changes in passing since they do load().
* This also avoids the following scenario:
  a) Someone edits a message page, triggering replace()
  b) Some page view causes load() to trigger loadFromDB()
     due to the hash being seen as volatile due to replace()
  c) The loadFromDB() loads stale slave data and undoes
     the message key update the replace() did

Change-Id: I9cdf7ad3d67f168fcba7f633af9e32a8d1fa928e
2015-09-03 01:06:22 +00:00
Aaron Schulz
89a167b500 More MessageCache locking/update cleanups
* Made the status key only act as a backoff key inside
  loadFromDBWithLock(), instead of also being a lock key. The
  getReentrantScopedLock() call is now non-blocking, which has
  a similar affect to the add() call on the status key but much
  better prioritizes replace(); that method already has a blocking
  getReentrantScopedLock() call, so once it gets that lock, there
  is no worry of load() failing due to contention.
* Bail out in loadFromDBWithLock() if the scope lock was not acquired.
  Normally the status lock often assured the lock could be obtained,
  unless a competing replace() came in. The old status lock would
  cause a bail if not acquired, and this carries over that behavior
  but also avoids clobbering updates when replace() contention happens.
* Avoid calling saveToCaches() in replace() if the lock was not
  acquired to avoid clobbering conflicting writes.
* Made the loadFromDB() call in replace() use DB_MASTER to avoid
  seeing stale data if the cache is volatile.
* Lowered WAIT_SEC, as the default PHP timeout is 30.
  We want this to be able to at least finish the other calls
  in replace() even if the lock times out.
* Avoid checking the status key an extra time in load().
* Made a few small code and doc cleanups.

Change-Id: Ibaf1f67618ec374c83c3135a71e90223dd2b1856
2015-09-02 22:41:18 +00:00
Aaron Schulz
3adf586c78 Removed READ_LATEST from revision load in getMsgFromNamespace()
* The key salting of the invidial keys via the WAN cache makes
  it OK to use slaves here. The salting happens via the delete()
  in the MessageCache::replace() call, which is hit when the
  message page is edited.

Bug: T92357
Change-Id: Ic191f6c3fc49c4c58461d26468dd0fa94a52051e
2015-08-31 20:20:45 +00:00
Aaron Schulz
846769fad1 Made MessageCache use the WAN cache for individual cache keys
* These keys are used for larger messages not in the main blob,
  and they still use explicit purges as the pages change

Change-Id: I254ddcc9ffe7c49654788d6aabcca2ab7ed4f03f
2015-08-28 23:25:48 +00:00
Aaron Schulz
6cdbf82be3 Removed useless $wgUseLocalMessageCache check in MessageCache
* The localCache member will use CACHE_NONE if this is off,
  so the set()/get() calls are already no-oped
* Also allow using the local cache if $hash is false
  in "stale mode", which handles the case where the hash
  key fell out of memory for some reason more gracefully

Change-Id: Ie12efcd4088a6dc4a4cdd2fd06646f2881df53d7
2015-08-27 18:31:35 -07:00
Aaron Schulz
5806931375 Cleaned up int|INF doc type to avoid IDEA errors
* INF is not a class (technically a float, which would look confusing here)

Change-Id: I9c437ecd0f2fe1a2a3e15197d50829a638f3073a
2015-08-27 18:18:59 +00:00
Aaron Schulz
a88e9ba007 Avoid self-deadlocks in MessageCache::replace()
* This makes use of the $rclass flag in BagOStuff

Bug: T109183
Change-Id: I305f51e744aac53876e5865f860c282aa2efbd8b
2015-08-25 20:04:19 +00:00
Timo Tijhof
79ce51d1c9 objectcache: Make first parameter of newAccelerator optional
Makes it more convenient to use.

Change-Id: I1e11f7a759bd2816e47d1c2453cbe39b8f44b2f0
2015-08-24 23:43:19 +02:00
Aaron Schulz
b8f231420c Use stock BagOStuff lock methods in MessageCache
Change-Id: Ic77b1b0f742590a10cddc7db067a4fcff246540f
2015-08-19 12:27:20 -07:00
Aaron Schulz
6d8dd29c59 Always call setValidationHash() in MessageCache::saveToCaches()
* This avoids constant churn when $wgUseLocalMessageCache is set to false
* Follow up to db464b8a84

Bug: T109183
Change-Id: I8da324c53527da32d09964be6c3a92176af4ee7b
2015-08-15 20:17:52 +00:00
Ori Livneh
6147de1182 MessageCache: derive the hash from the cache contents
Follow-up for I020617d, where I got this wrong. It is probable that multiple
application servers will try to build a local cache at the same time, in which
case they will cause the cache key to thrash. It is necessary that two
application servers building a local cache from the database generate the same
hash for a given set of messages.

Change-Id: Ieeefc2094a83be9401c466bec859c1588ddfbcdf
2015-08-14 19:49:07 +00:00
Aaron Schulz
db464b8a84 More multi-DC tweaks to MessageCache::load()
* Use hash key volatility to propagate invalidations over DCs
  in addition to memcached->APC instances.
* Make use of stale APC cache due to hash mismatch instead
  of waiting around in some cases. This is similar to the
  TTL expiry and volatility cases.
* Renamed $hashExpired and added some comments for clarity.

Change-Id: I8890fb174cae72c4ce8872df64f52f5cbd55183b
2015-08-14 15:49:53 +00:00
Ori Livneh
3a252efa12 MessageCache: use APC for local caching, rather than files
In addition to eliminating disk IO in a hot path, using APC spares us from
having to serialize and unserialize cache arrays. Since we're not serializing,
though, we don't have a string representation to hash, so use a random string
instead. (The code already treats the association of hash string to cache as
purely symbolic, so this is not problematic.)

Whereas the hash was previously stored as the first 32 bytes of each cache
file, we now store it as an array key instead (like VERSION and EXPIRY were
already). Because this changes the structure of cached data, we have to bump
MSG_CACHE_VERSION.

While we're here, make MessageCache::getLocalCache() and
MessageCache::saveToLocal() protected, make their signatures more
consistent with other methods in this class. While they were (implicitly)
public before, there are absolutely no external callers in Core or
extensions[0][1], so we can skip the standard deprecation process.

[0]: https://github.com/search?q=%40wikimedia+getlocalcache&type=Code&utf8=%E2%9C%93
[1]: https://github.com/search?utf8=%E2%9C%93&q=%40wikimedia+savetolocal&type=Code

Change-Id: I020617d2df2a8f0f243b85f3383dc7b16f15aaad
2015-08-10 12:39:11 -07:00
Kunal Mehta
32d1017e7d Don't let LinkCache grow indefinitely
Especially in long running maintenance scripts, this can be problematic.

LinkCache is now LRU-based, and will store a maximum of 10,000 good titles,
and 10,000 bad ones.

LinkCache::getGoodLinks() and getBadLinks() are deprecated since they
problematic to support in this implementation and are unused.

Bug: T106998
Change-Id: I1328149d65a5e75a5d6e10cb2686a099562a1847
2015-07-27 10:26:46 -07:00
Brian Wolff
cf501475fd Fix fatal when non-ascii message key is normalized
Missing global declaration when using $wgContLang in
MessageCache::normalizeKey

Change-Id: Ia1c1f41244fd5629527b99a5f2038f607cff42c4
Follow-up: 47e0f0c3
2015-07-19 10:01:55 +00:00
Geoffrey Mon
47e0f0c34d Normalize message name in Allmessages
* Normalize the message name returned by allmessages
* Separate message key normalization into
  MessageCache::normalizeKey()

Bug: T63894
Change-Id: I1d89fc73fea705243d390bab91255a635d8f9eee
2015-07-15 16:47:15 -04:00
jenkins-bot
08db2b721e Merge "Hygiene: Use strtr() instead of str_replace() for character swapping" 2015-07-15 09:39:45 +00:00
Ori Livneh
1020ac4edc Fix variable name (follows Ib2c5856d)
Change-Id: I1c673110e35daa6d5e382fad0fd99d6e4daa9f73
2015-07-13 07:27:39 +00:00
umherirrender
a0f83703cf Use correct variable in UserCache::doQuery
The variable $row is set to the last user of the first loop.
It is better to use the $name which is set in the foreach correctly.
This now adds all userpages to the LinkBatch and that avoids some extra
queries on at least Special:ListFiles.

Change-Id: Ied378b1596ec9d38eda41ce5ee413203c65eb21b
2015-07-11 10:22:32 +00:00
Timo Tijhof
c318e3a265 Hygiene: Use strtr() instead of str_replace() for character swapping
strtr() is marginally faster as it runs through the string only
once. A better fit for one-for-one character translation.

The strtr() function also supports an associative array as second
parameter for entire string replacements. This, too, has the same
performance and predictable behaviour (starts with the longest key).
Whereas str_replace is for more aggressive needs where you want
multiple passes until there are no further matches.

The associative array form is arguably also easier to understand
and harder to mess up since the needle/replacement pairs are
explicitly connected instead of two separate arrays.

Also:
* Use getFormattedNsText instead of strtr( getNsText, .. ) which
  reduces duplication of this fact through a more semantic intent.

Change-Id: Ie23e4210a5b6908dd79eebc8a2b931d12fe31af6
2015-07-06 20:25:19 +00:00
Jackmcbarn
c4f69827bc Fix some PHPCS issues
Change-Id: I6289ce362f8dfde7baf2b665f082820898844973
2015-06-26 01:32:28 -04:00
Seb35
ad0d70fd52 $wgUseGzip had no effect
Since MediaWiki 1.24 a constructor is defined in HTMLFileCache,
so the constructor of the parent class, FileCacheBase, is no more
called, and $wgUseGzip is no more used. This fix explicitely calls
the parent constructor.

Bug: T103237
Change-Id: I80c1871881049b9618c23aa76e6665867ecec2aa
2015-06-20 19:07:07 +02:00
umherirrender
b4068ae922 Remove double str_replace( ' ', '_', $ ) when using LinkBatch
LinkBatch::add already handle the underscore/space part, that means it
is not need to do it on the caller side when adding user names to
LinkBatch

Change-Id: I09e80712903a539164141cc0a88d321203114677
2015-06-19 22:04:25 +02:00
umherirrender
70f3afd548 Remove unneeded empty lines at begin of if/else/foreach body
An if body must not begin with an empty line

Change-Id: I62b058be337fcc85a120fcd3dadce564db59a271
2015-06-19 20:05:45 +02:00
umherirrender
d8821f2b0b Fixed spacing
- Removed space after casts
- Removed spaces in array index
- Added spaces around string concat
- Added space after words: switch, foreach
- else if -> elseif
- Removed parentheses around require_once, because it is not a function
- Added newline at end of file
- Removed double spaces
- Added spaces around operations
- Removed repeated newlines

Bug: T102609
Change-Id: Ib860222b24f8ad8e9062cd4dc42ec88dc63fb49e
2015-06-17 20:22:32 +00:00
Aaron Schulz
a4ff1c14bf Made MessageCache use the WAN cache
* This makes sure edits to MediaWiki: pages update
  the cache in all DCs

Bug: T99208
Change-Id: I177608729063b800fb97374f31f316779effce15
2015-06-16 17:09:12 +00:00
Kunal Mehta
f6e5079a69 Use mediawiki/at-ease library for suppressing warnings
wfSuppressWarnings() and wfRestoreWarnings() were split out into a
separate library. All usages in core were replaced with the new
functions, and the wf* global functions are marked as deprecated.

Additionally, some uses of @ were replaced due to composer's autoloader
being loaded even earlier.

Ie1234f8c12693408de9b94bf6f84480a90bd4f8e adds the library to
mediawiki/vendor.

Bug: T100923
Change-Id: I5c35079a0a656180852be0ae6b1262d40f6534c4
2015-06-11 18:49:29 +00:00
Fred Emmott
c403d4838d Add LCStore implementation that uses static arrays in PHP files
Implementation written by Fred Emmott of Facebook. Quoting Fred:

As well as array access being faster, the main advantage is actually
that this significantly reduces the use of unserialize(), which does a
lot of memcpys when making the strings.

Benchmarks compared to LCStoreCDB:
* HHVM (no repo-auth): ~7% improvement
* HHVM (with repo-auth): ~12% improvement
* PHP7: ~1% improvement

My (Legoktm) brief testing noted that the generated PHP files were
noticiably larger than the CDB ones:
* 1.5M	en.l10n.php
* 932K	l10n_cache-en.cdb

Bug: T99740
Change-Id: Ib2c5856d40cd928cab4a79cb935b3ce08c598300
2015-06-11 18:14:51 +00:00
Aaron Schulz
2a1b2ef334 Made MessageCache::replace() ignore messages with content language suffix
* This should not be used, and load() does not see them.
  If the content language is 'de', then message overrides
  will not include MediaWiki:<key>/de pages.

Change-Id: Ie4b6b356bd309814dd4a88040a29a7ebd509712a
2015-06-09 02:39:55 -07:00
jenkins-bot
b305d72af5 Merge "Fixed race condition in MessageCache::replace" 2015-06-09 09:25:55 +00:00
Aaron Schulz
4028bcb506 Fixed race condition in MessageCache::replace
* The cache has to reload and *after* locking to avoid
  losing any concurrent changes.
* Also fixed incorrect assumption in MessageCacheTest.
  Message overrides for the content language do not use
  the language suffix.

Change-Id: I98ff158a1575330bc59efe6badb27f8de8717951
2015-06-04 13:53:37 -07:00
Timo Tijhof
ef79d8361e cache: Add in-process caching to MessageBlobStore
This allows us to continue to use MessageBlobStore::get() as a way
to pre-fetch messages in batches for multiple modules while also
allowing individual modules to retrieve  their messages in a convenient
way.

This is in preparation for implementing ResourceLoaderModule::buildContents
and using it in ResourceLoaderModule::getVersionHash.

ResourceLoader::respond() already fetches message blobs in batches for the
actual response, but it also needs getVersionHash(), which would otherwise
fetch the same messages a second time.

Change-Id: I7e4c8b65765b54807123e85cfbb7eb2e5b2f39bd
2015-06-02 23:10:16 +01:00
Timo Tijhof
f37cee996e resourceloader: Replace timestamp system with version hashing
Modules now track their version via getVersionHash() instead of getModifiedTime().

== Background ==

While some resources have observeable timestamps (e.g. files stored on disk),
many other resources do not. E.g. config variables, and module definitions.

For static file modules, one can e.g. revert one of more files in a module to a
previous version and not affect the max timestamp.

Wiki modules include pages only if they exist. The user module supports common.js
and skin.js. By default neither exists. If a user has both, and then the
less-recently modified one is deleted, the max-timestamp remains unchanged.

For client-side caching, batch requests use "Math.max" on the relevant timestamps.
Again, if a module changes but another module is more recent (e.g. out-of-order
deployment, or out-of-order discovery), the change would not result in a cache miss.

More scenarios can be found in the associated Phabricator tasks.

== Version hash ==

Previously we virtually mapped these variables to a timestamp by storing the current
time alongside a hash of the value in ObjectCache. Considering the number of
possible request contexts (wikis * modules * users * skins * languages) this doesn't
work well. It results in needless cache invalidation when the first time observation
is purged due to LRU algorithms. It also has other minor bugs leading to fewer
cache hits.

All modules automatically get the benefits of version hashing with this change.
The old getDefinitionMtime() and getHashMtime() have been replaced with dummies
that return 1. These functions are often called from getModifiedTime() in subclasses.

For backward-compatibility, their respective values (definition summary and hash)
are now included in getVersionHash directly.

As examples, the following modules have been updated to use getVersionHash directly.
Other modules still work fine and can be updated later.

* ResourceLoaderFileModule
* ResourceLoaderEditToolbarModule
* ResourceLoaderStartUpModule
* ResourceLoaderWikiModule

The presence of hashes in place of timestamps increases the startup module size on
a default MediaWiki install from 4.4k to 5.8k (after gzip and minification).

== ETag ==

Since timestamps are no longer tracked, we need a different way to implement caching
for cache proxies (e.g. Varnish) and web browsers. Previously we used the
Last-Modified header (in combination with Cache-Control and Expires).

Instead of Last-Modified (and If-Modified-Since), we use ETag (and If-None-Match).

Entity tags (new in HTTP/1.1) are much stricter than Last-Modified by default.
They instruct browsers to allow usage of partial Range requests. Since our responses
are dynamically generated, we need to use the Weak version of ETag.

While this sounds bad, it's no different than Last-Modified. As reassured by
RFC 2616 <http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.3.3> the
specified behaviour behind Last-Modified follows the same "Weak" caching logic as
Entity tags. It's just that entity tags are capable of a stricter mode (whereas
Last-Modified is inherently weak).

== File cache ==

If $wgUseFileCache is enabled, ResourceLoader uses ResourceFileCache to cache
load.php responses. While the blind TTL handling (during the allowed expiry period)
is still maxage/timestamp based, tryRespondNotModified() now requires the caller to
know the expected ETag.

For this to work, the FileCache handling had to be moved from the top of
ResoureLoader::respond() to after the expected ETag is computed.

This also allows us to remove the duplicate tryRespondNotModified() handling since
that's is already handled by ResourceLoader::respond() meanwhile.

== Misc ==

* Remove redundant modifiedTime cache in ResourceLoaderFileModule.

* Change bugzilla references to Phabricator.

* Centralised inclusion of wgCacheEpoch using getDefinitionSummary. Previously this
  logic was duplicated in each place the modified timestamp was used.

* It's easy to forget calling the parent class in getDefinitionSummary().
  Previously this method only tracked 'class' by default. As such, various
  extensions hardcoded that one value instead of calling the parent and extending
  the array. To better prevent this in the future, getVersionHash() now asserts
  that the '_cacheEpoch' property made it through.

* tests: Don't use getDefinitionSummary() as an API.
  Fix ResourceLoaderWikiModuleTest to call getPages properly.

* In tests, the default timestamp used to be 1388534400000 (which is the unix time
  of 20140101000000; the unit tests' CacheEpoch). The new version hash of these
  modules is "XyCC+PSK", which is the base64 encoded prefix of the SHA1 digest of:
  '{"_class":"ResourceLoaderTestModule","_cacheEpoch":"20140101000000"}'

* Add sha1.js library for client-side hash generation.
  Compared various different implementations for code size (after minfication/gzip),
  and speed (when used for short hexidecimal strings).
  https://jsperf.com/sha1-implementations
  - CryptoJS <https://code.google.com/p/crypto-js/#SHA-1> (min+gzip: 2.5k)
    http://crypto-js.googlecode.com/svn/tags/3.1.2/build/rollups/sha1.js
    Chrome: 45k, Firefox: 89k, Safari: 92k
  - jsSHA <https://github.com/Caligatio/jsSHA>
    https://github.com/Caligatio/jsSHA/blob/3c1d4f2e/src/sha1.js (min+gzip: 1.8k)
    Chrome: 65k, Firefox: 53k, Safari: 69k
  - phpjs-sha1 <https://github.com/kvz/phpjs> (RL min+gzip: 0.8k)
    https://github.com/kvz/phpjs/blob/1eaab15d/functions/strings/sha1.js
    Chrome: 200k, Firefox: 280k, Safari: 78k

  Modern browsers implement the HTML5 Crypto API. However, this API is asynchronous,
  only enabled when on HTTPS in Chromium, and is quite low-level. It requires boilerplate
  code to actually use with TextEncoder, ArrayBuffer and Uint32Array. Due this being
  needed in the module loader, we'd have to load the fallback regardless. Considering
  this is not used in a critical path for performance, it's not worth shipping two
  implementations for this optimisation.

May also resolve:
* T44094
* T90411
* T94810

Bug: T94074
Change-Id: Ibb292d2416839327d1807a66c78fd96dac0637d0
2015-05-19 22:28:17 +00:00
Aaron Schulz
e9a609c7af Converted sidebar cache to the WAN cache
Bug: T93141
Change-Id: Id99f886c48501bbfef85cb0e7c5d2e2810a68581
2015-05-07 23:30:48 +00:00
Aaron Schulz
f6d1bbb8ee Moved MessageBlobStore to /cache
Change-Id: Ib628cc2f5d9079f4538561d585725fd79876d6f2
2015-04-15 20:36:03 -07:00
jenkins-bot
88bf68f5fb Merge "Kill Dwimmerlaik" 2015-04-07 20:10:21 +00:00
Max Semenik
536f98c760 Kill Dwimmerlaik
Seriously, the ops team spent some time trying to find that page during an outage,
while in fact it's an obscure Tolkien reference - better be clear.

Also, set the other dummy titles to something very clearly explaining what's
going on and where.

Change-Id: I6f33a2ea5030f22a258830a33f7bcefa7f0acd85
2015-04-03 14:05:37 -07:00
Aaron Schulz
4bdbfe6c1b Removed BloomFilter classes
* This ends up being more complex than its worth
  and even more so for multi-DC support

Bug: T93006
Change-Id: Iaa774fe69061e42955b11dc82d30dba93208e606
2015-04-03 09:10:04 +00:00
jenkins-bot
c864eab820 Merge "Don't trigger MessageBlobStore during tests" 2015-04-02 21:16:37 +00:00
Aaron Schulz
061904a913 Rely less on file stat cache in FileDependency
Change-Id: Ica16ddb7db00a56a16332c4dbb9a04e7b40a1844
2015-03-31 17:22:31 +00:00
Kunal Mehta
4fb5c877f6 Don't trigger MessageBlobStore during tests
The test for OutputPage::makeResourceLoaderLink was triggering database
queries through MessageBlobStore even though it doesn't use any
messages.

In bb03d1a8e0 I had made MessageBlobStore a singleton instead of static
functions, however there's no need for it to be one since the class is
stateless. Callers can just create a new MessageBlobStore instance and
call functions upon it. Using getInstance() is now deprecated.

ResourceLoader now has a setMessageBlobStore setter to allow overriding
which MessageBlobStore instance will be used. OutputPageTest uses this
to set a NullMessageBlobStore, which makes no database queries.

Change-Id: Ica7436fb6f1ea59bd445b02527829ab0742c0842
2015-03-28 21:25:25 -07:00
jenkins-bot
badc035712 Merge "Drop outdated "documentation reviewed" tags" 2015-03-18 16:56:58 +00:00
Thiemo Mättig
40342aba58 Drop outdated "documentation reviewed" tags
We review documentation all the time. Even if this was a big, notable
review, it was 5 years ago. It's probably outdated again, e.g. because
methods changed but the corresponting documentaion did not. In my
opinion the fact that a review happened 5 years ago is not useful any
more.

Change-Id: I6f4fb88ea790520bf2443aae4144cdde394b5e78
2015-03-18 17:43:01 +01:00
Aaron Schulz
df5ef8b5d7 Removed doCascadeProtectionUpdates method to avoid DB writes on page views
* Use special prioritized refreshLinksJobs instead, which triggers when
  transcluded pages are changed
* Also added a triggerOpportunisticLinksUpdate() method to handle
  dynamic transcludes

bug: T89389
Change-Id: Iea952d4d2e660b7957eafb5f73fc87fab347dbe7
2015-02-22 13:36:13 -08:00
Aaron Schulz
d7988e9447 Made BloomFilterTitleHasLogs refresh odds linear to be smoother
Change-Id: I25a4de8cf3537c9b3f27002e8bbb210322decec5
2015-02-17 01:37:33 -08:00
Aaron Schulz
6921770414 Updated some try-catch statements: MWException -> Exception
Change-Id: I76601a86e30f4984e3b1a8c8ec5ef5a0f652433a
2015-01-09 17:20:22 -08:00
Ricordisamoa
2ae155da52 Fix phpcs errors in includes/
Mostly Squiz.WhiteSpace.SuperfluousWhitespace.EmptyLines

Change-Id: I678b2f0902f11cd1dfa1611b9da24e7237df9122
2015-01-08 20:15:07 +01:00
Aaron Schulz
4ff8136807 Removed remaining profile calls
Change-Id: I31c81c78715048004fc8fca0f27d09c1fa71c118
2015-01-08 02:49:33 -08:00
Aaron Schulz
3226337f17 Converted some section profiling to scopedProfileIn
Change-Id: If69de34a046f4f33fc57b93beaf264f522ce7386
2015-01-07 14:19:06 -08:00
Chad Horohoe
aa21e125a3 Remove obvious function-level profiling
Xhprof generates this data now. Custom profiling of various
sub-function units are kept.

Calls to profiler represented about 3% of page execution
time on Special:BlankPage (1.5% in/out); after this change
it's down to about 0.98% of page execution time.

Change-Id: Id9a1dc9d8f80bbd52e42226b724a1e1213d07af7
2015-01-07 11:14:24 -08:00
Reedy
4d9143c7f5 Add lots of @throws
Change-Id: I09d0c13070f966fcf23d2638d8fc1328279a5995
2014-12-24 13:49:20 +00:00
Kunal Mehta
25c5bd41b9 MessageCache: Improve GlobalTitleFail log message
Same as 18d952dbcb

Change-Id: Iabb1eb7481026cfdf0610b0125d5b577febcc0f4
2014-12-15 18:34:32 +00:00
This, that and the other
c139126841 LocalisationCache: Use file_get_contents instead of DOMDocument::load
DOMDocument::load fails to load the plurals data during the import
process.

This is a work-around for https://bugs.php.net/bug.php?id=64938. This
bug only rears its head when using Special:Import, because that is
essentially the only place in MediaWiki where we fiddle with
libxml_disable_entity_loader.

Bug: T58439
Change-Id: Idcb4ab1cef2a7b080543e7cc1cee5464fc476456
2014-12-14 05:56:02 +00:00
Aaron Schulz
e369f66d00 Replace wfRunHooks calls with direct Hooks::run calls
* This avoids the overhead of an extra function call

Change-Id: I8ee996f237fd111873ab51965bded3d91e61e4dd
2014-12-10 12:26:59 -08:00
Bartosz Dziewoński
aa00a3e838 ResourceLoaderImageModule for icons
ResourceLoaderImageModule needs a set of SVG files and some data in
the module definition, and produces styles for a set of CSS classes,
one for each image, optionally with differently colored variants,
generated in SVG and PNG, data-URI-embedded if possible, compatible
with all browsers, and generally slick.

The intended usage is to ship icon libraries with MediaWiki that can
be used throughout the pages with no additional code.

* ResourceLoaderImageModule implements all of the logic for data
  parsing and CSS generation.
* ResourceLoaderImage implements the logic for SVG image colorization
  (for variants) and rasterization.
* ResourceLoader and ResourceLoaderContext were extended to serve a
  new kind of load.php request that delivers a single image file. This
  is used for fallback PNG images served to browsers that don't
  understand SVG.

See change Ic6a76bfb for a demo.

Bug: T76473
Co-Authored-By: Trevor Parscal <trevorparscal@gmail.com>
Co-Authored-By: Bartosz Dziewoński <matma.rex@gmail.com>
Change-Id: Idf6ff4eb8e94f45946f15d283d34108b881fae6e
2014-12-09 20:11:07 +01:00
umherirrender
489d793882 Fixed spacing
- Added/removed spaces around parenthesis
- Added newline in empty blocks
- Added space after switch/foreach/function
- Use tabs at begin of line
- Add newline at end of file

Change-Id: I244cdb2c333489e1020931bf4ac5266a87439f0d
2014-12-05 22:28:07 +01:00
Kunal Mehta
7fe5b87e8c Make LocalisationCache::getMessagesDirs() public
So it can be used inside LocalisationUpdate

Change-Id: I2399ddd7fd4462f2c6d1bf81036af451a67c58b1
2014-12-03 23:46:47 -08:00
Kunal Mehta
27d21e3117 Move core message dirs from $wgMessagesDirs to LocalisationCache::getMessagesDirs()
If $wgMessagesDirs is initially empty, we can optimize when batch loading
extensions:

    if ( !$wgMessagesDirs ) {
        $wgMessagesDirs = $cache['MessagesDirs'];
    }

With APC, this should be O(1) CPU time.

This was suggested by Tim in the code review of I7074b65d07c5.

Change-Id: I66fa907cdaafe18b74b5b9afaa8b6b1db069bea3
2014-12-03 14:32:18 -08:00
Aaron Schulz
140a4f9788 Fixed BloomCache handling of network partitions
* As documented, it should return true on error, so that DB is
  checked if the filter is down.

Change-Id: I883fafc9f5f3a84f85207de6e916f1630c78d1a4
2014-11-20 09:11:09 +00:00
Kunal Mehta
987d2e4c77 Use cdb library and provide a back-compat layer
The new cdb library is pulled in via composer. Since the
library uses namespaces, a backwards-compatability layer
is provided for the old class names:
* CdbReader
* CdbWriter
* CdbException

The PHP/DBA-specific classes should never have been used directly.

Depends on I98302bdf1 in mediawiki/vendor

Change-Id: I39549ac8540b262cf91f7d1830d36327afb3033d
2014-11-19 18:39:27 +00:00
Aaron Schulz
93d2167ca6 Moved MapCacheLRU to libs/
Change-Id: Icb820787afe7298dd3124f5168d3efff19e53ab6
2014-11-10 22:51:29 +00:00
jenkins-bot
ddda8de53f Merge "Improved/additional logging for $wgTitle abuses" 2014-11-10 19:12:14 +00:00
Chad Horohoe
95e6efc4d4 Improved/additional logging for $wgTitle abuses
Change-Id: I6599aa8abe30c1888ab5df023a0d4bf346315400
2014-11-10 11:04:43 -08:00
MZMcBride
627ccbcd7b Minor code comment tweaks for spelling and consistency
Change-Id: I51391f45d0f81e4245ccc0e435a71ccd5b0e3ca3
2014-11-08 14:07:19 -05:00
jenkins-bot
f36b0fbca2 Merge "Turn a wfDebug call in MessageCache::get into a wfDebugLog call" 2014-10-03 18:08:27 +00:00
Aaron Schulz
12757b50f8 Removed $wgAntiLockFlags to unify the code paths
Change-Id: Ia709ccdc9addc55c99cbff21a5ff3009b5fbb53c
2014-10-03 10:42:17 -07:00
Ori Livneh
494909453d Turn a wfDebug call in MessageCache::get into a wfDebugLog call
It's a very hot code path, so it makes sense to be able to dive into it in
detail, and having a discrete log bucket makes that easier to do.

Change-Id: I4b547e97711b9e45a99f7b747d785690c70883ec
2014-10-02 20:52:05 -07:00
umherirrender
df24b7209d Fixed spacing
- Added newline at end of file
- Removed double spaces/newlines
- Added space after if/function and parentheses/brackets
- Removed space before comma/cast
- Fixed indent of some lines

Change-Id: I29867ffdffdfb7d2b56997e9393497c7dc12f7d3
2014-09-29 20:46:19 +02:00
Aaron Schulz
6c45ff18bc Fixed bug in BloomCacheRedis::getConnection
* The array keys need to be re-indexed to avoid picking a deleted one

Change-Id: I3c22674a0194597598d66179bb5e3625da1dd38c
2014-09-23 17:26:00 -07:00
jenkins-bot
fca90adbd7 Merge "Deprecate HTMLFileCache::newFromTitle() in favor of constructor" 2014-09-19 00:24:16 +00:00
jenkins-bot
ab29cc0ec6 Merge "Add a hook to allow extensions to prevent HTML file caching" 2014-09-19 00:08:58 +00:00
Max Semenik
0fddd42ca4 Deprecate HTMLFileCache::newFromTitle() in favor of constructor
Change-Id: I903e68fc1f486501d790ca69146ecb835d90c9cc
2014-09-19 00:07:51 +00:00
Max Semenik
bbc92a1433 Add a hook to allow extensions to prevent HTML file caching
Needed for MobileFrontend on third-party sites that use this feature.
Ideally, there should be a way to cache mobile requests too, but that's
a different issue requiring much more effort to do properly.

Bug: 68106
Change-Id: I01a76c571d9186b325f19a00cec136459707c791
2014-09-18 17:00:39 -07:00
Aaron Schulz
ca62566ab9 Doc typo fixes
Change-Id: I41286754185dbd1d23f409dfb60eb0fe9d4aa5f5
2014-09-17 11:30:07 -07:00
Aaron Schulz
b1d215726e Removed LCStoreAccel class
* Since individual message keys can expire, this can cause broken
  messages when such a key is needed (the message is treated as if
  it did not exist). The base class clearly documents a need for
  more atomicity (only top level keys can fall it separately).

Change-Id: I992bba77a0afdeeeade8be013708277b79f22314
2014-09-05 13:33:10 -07:00
Brad Jorsch
b6fc9067b0 LocalisationCache: Process one fallback at a time
Currently LocalisationCache merges the core data for all languages in
the fallback chain, then the extension data, then merges those two, and
then gives extensions like LocalisationUpdate a chance to make final
overrides with the LocalisationCacheRecache hook.

But if LocalisationUpdate doesn't want to locally duplicate all the
messages for every language (e.g. r104041), LocalisationCacheRecache is
too late: the information as to whether a message came from the primary
language or a fallback has been lost, so when LU itself has an override
for a fallback language it can't know whether or not the existing
message should be overridden or not.

The solution is for LocalisationCache to gather the data for each
fallback language separately, call a new hook for LU to affect just that
language (LocalisationCacheRecacheFallback), and only then merge the
fallback languages together.

Bug: 68781
Change-Id: Iacfe96063fcc66c1f97ca5e5292a8fc70af988cf
2014-09-04 16:56:31 +02:00
Aaron Schulz
a9ba7d45f4 Added BloomCache classes
* Implemented a version of BloomCache using Redis
* Added a BloomCheckTitleHasLogs handler class for avoiding
  slow logging table queries when large amounts of 404 pages
  are viewed (by various web crawlers at the moment).

bug: 67439
Change-Id: I26e5034755e3a7208a45991b1cf2f12467679cc1
2014-09-03 17:43:21 +00:00
jenkins-bot
26bef0e5bf Merge "Turn MessageBlobStore into a singleton instead of static functions" 2014-08-25 11:39:09 +00:00
Kunal Mehta
bb03d1a8e0 Turn MessageBlobStore into a singleton instead of static functions
For easier testability and other things. There are no uses
of this class in any extensions in gerrit.

Change-Id: I606de4259239e128ed7e0477fc98b84c647430c4
2014-08-25 03:53:39 -07:00
umherirrender
7c6a25856c Add missing @return to function docs
Change-Id: I45b9d02f94ecc58372268ec5e6a0b572a0b7e2a9
2014-08-23 23:14:57 +02:00
Aaron Schulz
b728d6920b Made getCachedRevisionObject() use MapCacheLRU
* Previously the cache size was unbounded and leaky
* Also made MapCacheLRU handle null values properly

Change-Id: Ia02258cf051e1ccf11457c758741b8c7922afe89
2014-08-22 18:12:29 +00:00
Kevin Israel
dd5c1b7fb7 Title::getContentModel(): load from DB if necessary
Also don't cast $model to int in LinkCache::addGoodLinkObj(); content
model IDs are non-numeric strings, not integers, so that field was
always populated with the value 0. Because 0 is a falsy value, this
caused subsequent calls to Title::getContentModel() to return the
default model rather than the correct one.

Also (hopefully) fixed every single query that could cause a
LinkCache entry to be added without the content model.

Bug: 69789
Change-Id: I94f06baf406afa538cd2b10139598442f9fc6759
2014-08-20 19:44:17 -04:00
jenkins-bot
088a8bd2b4 Merge "Revert "Made LCStoreDB try to use a separate DB connection"" 2014-08-13 21:01:36 +00:00
Tom Arrow
8db655e94b Correct variable types in phpdoc
Change-Id: I1a57024868fa47d9537585f1bb738d6f7c725c7a
in: LocalisationCache.php and ImageQueryPage.php
2014-08-09 13:31:02 +01:00
Chad Horohoe
405abdb61a Move MW_LC_VERSION to class constant instead of global define
Change-Id: I184ec13081a5cf60d593a1c25c091b1db7b344a9
2014-08-06 13:36:46 +01:00
C. Scott Ananian
9c0fff5a4b Revert "Made LCStoreDB try to use a separate DB connection"
This commit broke `php tests/parserTests.php`.

This reverts commit 4c0871fa0e.

Bug: 69082
Change-Id: If0781f8f51681717a848ec359c7b8c70932ed72d
2014-08-03 17:40:03 -04:00
umherirrender
02dc9da399 Cleanup some docs (includes/[a-d])
- Swap "$variable type" to "type $variable"
- Added missing types
- Fixed spacing inside docs
- Makes beginning of @param/@return/@var/@throws in capital
- Changed some types to match the more common spelling

Change-Id: I7b65fe04db431342cc58b469dc48f41a50c4e891
2014-07-24 19:42:45 +02:00
umherirrender
abfa0d7854 Standardize @todo
Always use lowercase @todo
Changed some "@FIXME" to "@todo FIXME: "

Change-Id: I016d1f9511e6f3bbd22427f0fffeccd77ab5e0db
2014-07-23 22:27:20 +02:00
umherirrender
cabbc6eaa4 Docs: {function} -> callable
Change-Id: Iab0760ce5c518854a127330d3e0c0ee7ed0c526c
2014-07-21 21:38:56 +02:00
withoutaname
d1e0959eef Move CacheHelper to includes/cache/ folder
Change-Id: I13c9cf1be7fe0d436ea1a5d23f162abac7d97a29
2014-07-18 19:45:05 +00:00
Aaron Schulz
4c0871fa0e Made LCStoreDB try to use a separate DB connection
* This avoids breaking the main transaction in finishWrite().
* Also removed an unused variable in SqlBagOStuff.

Change-Id: Ia330ac362b080c1338616d95b793032c4752dc23
2014-07-15 19:20:01 -07:00
umherirrender
de39f3e019 Use some callable hints on @param docs
Callbacks can be given as a string or array, so the hint 'callable' is
used.

Change-Id: I3842606f74c8c3705dffc70bf13e31f44a37fa65
2014-07-03 21:20:35 +02:00
Aaron Schulz
8d6c724ade Improved DB handling in LCStoreDB
* Only do the DB inserts in finishWrite(). Previously this would
  COMMIT there but would spread out queries throughout the set()
  calls. This pushes all the contention down to the last step.

Change-Id: Ia8afb79a8328c18a7d8a386bcd15a5074007d52e
2014-06-30 17:16:42 -07:00
withoutaname
ad1d07f34f Rename Parser_DiffTest class to ParserDiffTest
Change-Id: I5fca44a3fffbc60a66be32fad9ed6d1713056d81
2014-06-22 13:14:01 -07:00
Siebrand Mazeland
2c04b0163b Remove incorrect return type for getGoodLinkID()
Change-Id: Ib9c0dcf5104fdc5f364e0b49d539b63bf2eb046c
2014-05-11 18:19:00 +02:00
Jeff Janes
0582e3339d PostgreSQL: Make l10n_cache.lc_value binary
Change-Id I427c6de5a0a29b43cff755db0eb8a750db620173 increases the
probability that a null byte will attempt to be stored in the
lc_value column.  PostgreSQL does not allow that byte in a text
column, so convert the column to bytea.

If the column already contains corrupted data, the upgrade routine
might fail.  To prevent this, delete the contents of the table before
changing the type.

Bug: 62098
Change-Id: Ie8368bde398b2ae4d3cfc9ee7bf35874bd2ded68
2014-05-05 11:20:52 -07:00
Siebrand Mazeland
f994817f6b Pass phpcs-strict on various files in includes/
These files have all had treatment before, and these occurrences have either
been missed or have been introduced after.

Change-Id: I06cdab4616b5bff47c85152df28f18c861730a23
2014-04-24 21:50:01 +02:00
umherirrender
fd66273c94 Fixed some @params documentation (includes/[cache|objectcache])
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Also added some missing @param.

Change-Id: I86fd10e3f2d4bb80e7432533038d124693acfb3c
2014-04-19 01:19:46 +02:00
Kevin Israel
e0043257c0 Skip loading PHP shims for JSON i18n files
For now, non-message data (e.g. special page aliases, magic words) belong
in separate files having their own keys in $wgExtensionMessagesFiles. It
is unnecessary to read any PHP files listed under keys that exist in
$wgMessagesDirs; they are merely compatibility shims.

Bug: 63926
Change-Id: I92013d0a45e83ad0f5ac483a3db867806eb71f8a
2014-04-14 20:37:51 -04:00
Alexandre Emsenhuber
6ea0417cef Fix language statistics after messages have been moved to JSON
Statistics are now showing always no messages since languages.inc
does not manage to load the JSON messages.

Had to make LocalisationCache::readJSONFile() public so that it can
be used in  languages.inc; since all methods from LocalisationCache
return localisations merged with fallbacks, which is not what we
want here.

Fix for I918cfdc46c (0dd91d5).

Change-Id: Ib52287db618b9d072e847130070d165a3e7ae44b
2014-04-02 20:05:51 +02:00
Siebrand Mazeland
8e0c0a9fc9 Preparations for migrating core to use JSON based i18n
LocalisationCache and Language have to take the JSON files into account
in deciding if a language is present or not.

Standardizing language validity checking with isSupportedLanguage
and isValidBuiltInCode.

Co-Authored-By: Niklas Laxström <niklas.laxstrom@gmail.com>
Co-Authored-By: Siebrand Mazeland <siebrand@kitano.nl>
Change-Id: I35bbb3a7a145fc48d14fff620407dff5ecfdd4fc
2014-04-01 14:22:03 -07:00
jenkins-bot
ed8668b925 Merge "Add missing line breaks to wfDebug() calls" 2014-03-31 11:50:41 +00:00
Alexandre Emsenhuber
19a9618589 Send the localisation store debug message to the "caches" log group
Rationale is the same as for Ifb6dc2666f (96b04ce): ability to
separate it form the main log group, as it always has a predicitible
value.

Change-Id: I0ac5baf5ab8fbe4d2025ccf6439b883dc82e9d5b
2014-03-29 17:42:08 +01:00
Alexandre Emsenhuber
449ee32451 Add missing line breaks to wfDebug() calls
Also removed true as second parameter to it from CloneDatabase.php
since it is the default value of that parameter.

Change-Id: I727ebae2bd4df0e26019985ce8c7ce73381c5642
2014-03-29 11:52:07 +01:00
jenkins-bot
2eefaf1705 Merge "Moved ProcessCacheLRU to /libs" 2014-03-18 17:22:34 +00:00
Alexandre Emsenhuber
c4ff5e4b6a Fix profiling error in LocalisationCache::readJSONFile()
Follow-up I8d137e15e1 (6380e81).

Also added more blank lines for better readability.

Change-Id: Iae7921b017f81d5512e71384d7999502154c034c
2014-03-18 09:41:45 +01:00
Aaron Schulz
400e4848ec Moved ProcessCacheLRU to /libs
Change-Id: I7052d04d9847f0310c1e62bd66365c813fddeab5
2014-03-16 16:32:34 -07:00
Niklas Laxström
a064f78053 New hook MessageCache::get
Example usage:

$wgHooks['MessageCache::get'][] = function( &$key ) {
	static $keys = null;
	if ( $keys === null ) {
		global $wgExtensionMessagesFiles;
		require( $wgExtensionMessagesFiles['OverrideMessages'] );
		$keys = array_flip( array_keys( $messages['en'] ) );
	}

	if ( isset( $keys["myprefix-$key"] ) ) {
		$key = "myprefix-$key";
	}

	return true;
}

Pros:
* Easy way to override standard core and extension messages without
  any changes to them
* Messages can be stored in a standard i18n file
* Messages can be translated easily with Translate
* Messages can be shared accross multiple wikis easily
* Takes advantage of the normal message cache behavior unlike the
  MessagePreLoad hook
* Missing translations fallback to the override, not to the
  uncustomized standard translation
* Do not need to handle conflicting message keys at translatewiki.net
  if adopted by WMF

Cons:
* This method is called often, so there will be small performance
  impact if no hooks are registered. Impact can be big if the
  implementation of hook subscriber is inefficient.

This can help with bugs like 36149. It doesn't remove the manual work
needed to detect those messages and adding them to the i18n file.

I have been using this patch in a wiki farm for months.

Change-Id: Ib39937a440e71ae7292cf992ab37a569189741e4
2014-02-10 15:51:52 +00:00
Siebrand Mazeland
f077c4b1d6 Update formatting
Change-Id: I18aff576262479c9bb1c56eb8e1d1aaae200e4b1
2014-02-06 09:27:05 +01:00
Alexandre Emsenhuber
8d701eeeea Remove trailing line breaks from wfDebugLog() messages
This is useless since the message is passed through trim() and a
line break is added afterwards.

Change-Id: I1a26b30a07f7c9c749fce5bb6b2b4b3d79901b7c
2014-02-04 22:16:13 +01:00
Alexandre Emsenhuber
a7a0883019 Enhance the destination control parameter of wfDebug() and wfDebugLog()
- The parameter is now a string, making is more understandable than
  boolean values
- It takes the same values in both wfDebug() and wfDebugLog() (except
  for 'private' which is only used in the latter)
- This adds a new possibility to wfDebugLog() to log the message either
  on the specific log or the general one, but not to the debug toolbar
- Old boolean values are still recognised for backward compatibility
- Also send the messages passed to wfDebugLog() to the debug toolbar
  when they are written to a specific log and not restricted to logs
- Updated the calls of and wfDebug() and wfDebugLog() with the last
  parameter to change it into a string
- Renamed MWDebug::sendWarning() to MWDebug::sendMessage() and added
  $group parameter to it; will not break anything since that method
  is marked as private
- Changed the call to wfDebug() from MWDebug::sendMessage() to use
  wfDebugLog() with 'log' as thrid parameter, so that those messages
  can be logged separately from the main log and they don't show up
  a second time on the "debug log" tab of the debug toolbar

Change-Id: I1be09d4c1d3408ed5b26a5db02691c17c0ec0926
2014-02-04 19:56:24 +00:00
Jacob Clark
cced9d13f9 Added wfSuppressWarnings and wfRestoreWarnings
Added wfSuppressWarnings and wfRestoreWarnings before and after the
ini_set call to apc cache by default to suppress errors if ini_set is
disabled.

Change-Id: If91e99c30fce3af14e69acd35112af188e62b6ac
2014-02-02 19:23:24 +00:00
MaxSem
0998c41943 Revert "Include short descriptions for extensions bundled in the release"
Contains var_dump(), self-merged non-trivial code.

This reverts commit 96b2c66e24.

Change-Id: Ie630466ef50e6bcdabf01daff28c283c764aae35
2014-01-12 23:59:44 +00:00
Mark A. Hershberger
96b2c66e24 Include short descriptions for extensions bundled in the release
Also fix styling for the list of extensions.

Bug: 43817
Change-Id: I5335225683ec8f1c163bb67f478787ebc52ee3a9
2014-01-12 18:21:55 -05:00
Siebrand Mazeland
8a096e3444 Random documentation tweaks
Change-Id: I57c3bde6551c14324cf34e4db003a319d6cebc47
2014-01-02 23:45:14 +01:00
Aaron Schulz
7bd3a8c576 Added $purgeBlobs flag to LocalisationCacheRecache hook
* This helps replace the WMF live hack to skip calling clear()

Change-Id: I37bd6cf87e72b6d32e4c6c261abe2a0ef43d067e
2013-12-30 21:45:59 +00:00
jenkins-bot
1728c1a698 Merge "Made Title cache use MapCacheLRU" 2013-12-26 22:24:14 +00:00
jenkins-bot
c9eaaf7093 Merge "Plural rules: updates for UTS #35 Rev 33" 2013-12-23 14:00:34 +00:00
Siebrand Mazeland
2f7b68ffde Rename $wgExtensionMessagesDirs to $wgMessagesDirs
This is a better name, as we'll also be adding core and the installer i18n
to this.

Change-Id: Ic4c5849654aef54f3e5aea01a2d68e47d148b9bd
2013-12-20 17:02:18 +01:00
jenkins-bot
b3970523a3 Merge "Add support for JSON i18n files" 2013-12-20 13:41:42 +00:00
Roan Kattouw
6380e81cd0 Add support for JSON i18n files
Implementation for https://www.mediawiki.org/wiki/Requests_for_comment/Localisation_format

Add $wgExtensionMessagesDirs, which tracks the directory
(or directories) where each extension stores it's JSON i18n files.
In this commit only support for messages is implemented, but adding
support for other i18n variables (e.g. magic words) is easy to do later.

To be backwards compatible, an extension can specify both
$wgExtensionMessagesFiles and $wgExtensionMessagesDirs. Older versions
of MediaWiki will just work, and newer versions will use the JSON files
while ignoring the PHP file (except if the PHP file contains non-message
data like magic words).

Misc changes:
* Updated mergeMessageFileList.php to output both
  $wgExtensionMessagesFiles and $wgExtensionMessagesDirs

Change-Id: I8d137e15e1670880a9847263e6ce796c62a4670d
2013-12-20 14:34:06 +01:00
Tim Starling
e571717e06 Plural rules: updates for UTS #35 Rev 33
* New operands i, v, w, f, t
* New operators =, !=, %
* Ignore "samples", which are basically unit tests embedded in rule
  specifications
* Ignore the new "other" rules, which have an empty condition. It
  doesn't really makes sense to parse them, since the empty condition
  means special handling should be done in the caller, it is not
  equivalent to an unconditional true or false.
* Trailing zero support requires that the input number be a string.
  Documented this.
* Fixed some comments
* Added test cases for new features

Bug: 56931
Change-Id: I96986c0c664f785e75b0a4ced2ec9e37b72681c1
2013-12-13 11:53:29 +11:00
Aaron Schulz
5ac27d708b Made Title cache use MapCacheLRU
Change-Id: I913238adc2b8471e247df77150cfd772da1ac3f4
2013-12-04 00:27:23 -08:00
Aaron Schulz
a0368b3762 Made RepoGroup use ProcessCacheLRU
Change-Id: I322a6cf15566bde4fc5ee5bfa1be5fdc680cd763
2013-12-03 12:24:50 -08:00
Aaron Schulz
77f7e9ba65 Added MapCacheLRU class, a simpler cousin to ProcessCacheLRU
Change-Id: Ibe1b44c8e168a086afb3481b4bb7660dc7d31ae7
2013-12-03 19:47:02 +00:00
jenkins-bot
bc5951e743 Merge "Update some documentation" 2013-12-03 17:10:21 +00:00
Siebrand Mazeland
db9ebea5ac Remove unused local variable
Change-Id: I5f13b3bf7e769d0b0960b44e15b9c2454d9eb165
2013-12-03 15:01:07 +01:00
Siebrand Mazeland
cbb6648205 Update some documentation
Change-Id: I6a75964d2048de451e9b01a08f87cfe1ebd842ba
2013-12-03 14:59:45 +01:00
jenkins-bot
01596b2bbf Merge "Fixed spacing" 2013-12-02 06:32:10 +00:00
umherirrender
5ca5672aac Fixed spacing
- Place commas correct
- Moved comments
- Add space after if/foreach/catch
- Reformat some conditions
- Removed trailing spaces/tabs

Change-Id: I40ccda72c418c4a33fcd675773cb08d971510cdb
2013-12-01 20:58:51 +01:00
umherirrender
e0b8d17b79 Simplify switch in BacklinkCache::getConditions
Change-Id: I9f1ddad705d1697dd81f74eb735dc386bab9d200
2013-12-01 11:55:44 +01:00
Chad Horohoe
13d64f3cb1 Improved CdbException handling in LocalisationCache
Change-Id: If1ad9a982c2a4a7a1e9ca7739b3c215bb821211a
2013-11-26 13:54:24 -08:00
jenkins-bot
9e0c2a0658 Merge "Fixed "Undefined property: stdClass::$page_namespace" error" 2013-11-20 06:36:30 +00:00
Aaron Schulz
ddaffed317 Fixed "Undefined property: stdClass::$page_namespace" error
* BacklinkCache::fullResultCache should always have all the columns
* Fixes bug from a9dc4d943e

Change-Id: Ic0042ef6b1ce655bd97814362d95d8b0fe53affa
2013-11-19 15:37:06 -08:00
jenkins-bot
b6dd4a28f5 Merge "Declare visibility on class properties of LinkBatch" 2013-11-19 19:49:57 +00:00
Siebrand Mazeland
7f56ab1d7c Declare visibility on class properties of LinkBatch
Set to public. Usage in for example api/ApiPageSet.php.

Change-Id: I8d6b8a0728a465544b09832b2916235628a21b8e
2013-11-19 08:32:31 +00:00
Siebrand Mazeland
a0c583e408 Remove unused classes Title[List]Dependency
Classes TitleListDependency and TitleDependency that are currently not
used. Searched core and extensions in Gerrit.

Change-Id: I51404ce36c4ef55393588817d1673fd17c81b4a2
2013-11-18 21:46:23 +01:00
Siebrand Mazeland
4a4fc11008 Update LinkCache::addGoodLinkObj param defaults to int
Update the defaults for $revision and $model from bool to int, making it
consistent with the documented input. Also update docs and line length.
Change "inval( $x )" to preferred "(int) $x" while changing the method.

Change-Id: Ic19a408aa7c50fb03e2c3aca8df3fa7cedc2420b
2013-11-18 21:09:24 +01:00
jenkins-bot
4bed5565db Merge "Declare visibility on class properties of LCStore* classes" 2013-11-18 18:58:22 +00:00
jenkins-bot
618a686ae6 Merge "Declare visibility on class properties of LocalisationCache" 2013-11-18 18:56:27 +00:00
jenkins-bot
10b41a1fb0 Merge "Remove underscore from classes LCStore_*" 2013-11-18 18:54:25 +00:00
jenkins-bot
3f8d11fe84 Merge "Set visibility on class properties for *Dependency classes" 2013-11-18 18:46:48 +00:00
jenkins-bot
4910f271f1 Merge "Declare visibility for class properties of LocalisationCacheBulkLoad" 2013-11-18 18:45:10 +00:00
jenkins-bot
f678aaa3a8 Merge "Remove underscore from class LocalisationCache_BulkLoad" 2013-11-18 18:43:16 +00:00
jenkins-bot
4792e870bd Merge "Break long lines in includes/cache/" 2013-11-18 18:41:27 +00:00
jenkins-bot
5bd4512e18 Merge "Update formatting" 2013-11-18 18:39:29 +00:00
Siebrand Mazeland
0ec4a01507 Set visibility on class properties for *Dependency classes
Set visibility to private for properties that are not referenced outside
of the class itself, and when the class is not currently being subclassed.

Change-Id: I427c6de5a0a29b43cff755db0eb8a750db620173
2013-11-18 04:53:13 +00:00
Siebrand Mazeland
e9c996a1b8 Set visibility for class properties of DependencyWrapper
Set to private. No subclassing and no uses outside of the class in core or
extensions.

Change-Id: If195d02b8ce853242312de7fb4bc7dc1cbe94365
2013-11-18 04:52:49 +00:00
Siebrand Mazeland
7d11e65dcc Declare visibility on class properties of LCStore* classes
No direct references found in core or extensions outside of the classes
themselves. None of these are subclassed, so set all of them to private.

Change-Id: Id599c0830c1e6d3679c7ae55147f4444f8252987
2013-11-17 23:18:01 +01:00
Siebrand Mazeland
540d8dafeb Declare visibility on class properties of LocalisationCache
Mark all private, except the one that's being used in subclass. Checked
core and extension for direct uses. None found.

Change-Id: If73d65ca094ff56deb19d3b6f3ef99892654726d
2013-11-17 22:49:03 +01:00
Siebrand Mazeland
e711503e7a Remove underscore from classes LCStore_*
iPart of program to remove underscores from class names. Checked core and
600+ extensions for occurrences. All uses are in core, and are updated in
this change.

Change-Id: I432dc249d22053728013ae7d0d56c3c398021c5e
2013-11-17 22:09:31 +01:00
Siebrand Mazeland
1e07d64050 Declare visibility for class properties of LocalisationCacheBulkLoad
Mark all as private. Not referenced elsewhere in core or extensions.

Change-Id: Ibe0e854f39261053e5147138309c3a4b1944e5fe
2013-11-17 22:03:53 +01:00
Siebrand Mazeland
04021a8e81 Remove underscore from class LocalisationCache_BulkLoad
Part of program to remove underscores from class names. Checked core and
600+ extensions for occurrences. All uses are in core, and are updated in
this change.

Change-Id: Ib157c3cf209677718ff184552cf872b1d3f4e969
2013-11-17 22:00:34 +01:00
Siebrand Mazeland
ddb4e02634 Break long lines in includes/cache/
Change-Id: I23a6f46f89b3cdcaf2a3b982dc63e779eed5840d
2013-11-17 21:49:59 +01:00
Siebrand Mazeland
8251479984 Update formatting
Change-Id: I9aa56cbe0989d6f302ee0f47c4a6154101f73e90
2013-11-17 21:46:55 +01:00
Aaron Schulz
a9dc4d943e BacklinkCache performance tweaks
* In partition(), avoid doing the JOIN since it is not needed and prevents
  index-only queries.
* Made numLinks() wrap partition when no $max is specified.
* Also fixed some docs

Change-Id: I05a83f71efb4c6e99e40883d7fa53da59184b13c
2013-11-09 12:35:07 -08:00
Tim Starling
64d0dc06a0 Move the DeferredUpdates hierarchy to its own directory
The grouping makes at least as much sense as job/, and certainly makes
more sense than cache/. With directories named after base classes, it is
fairly easy to tell what should go where. The grouping of
DeferredUpdates, DataUpdate and CallableUpdate would surely be
uncontroversial.

The move of SearchUpdate out of search/ demonstrates the conflict between
arrangement by module versus arrangement by type, which is the most
difficult design question here. I think arrangement by type is more
consistent with e.g. the arrangement of the core root, i.e. tests/,
resources/, maintenance/, etc. where a given feature will have its files
split up into a mostly type-based hierarchy.

I also tidied up AutoLoader.php by moving includes/content to the correct
location, sorted alphabetically by subdirectory.

Verified with AutoLoaderTest.

Change-Id: Ib369411d0caca38e72978084aa57348f1b892ed0
2013-11-04 10:32:40 +11:00
Siebrand Mazeland
e61cb8218b Update docs and declare visibility on class props
Change-Id: Ib0f02202d075d4a56dc4e37b08d7ac9399e8c86c
2013-11-01 12:35:27 +01:00
Brad Jorsch
db9c4cb3cf Remove linkprefix message, add $linkPrefixCharset
The existing "linkprefix" message is unlikely to be accurately
customized by message translators (as shown by the fact that, of the 10
distinct customizations prior to Iaa7eaa44 (which made them even more
complicated), 3 were broken or entirely ineffective, 1 was half
ineffective, and 2 more seem to have included the Latin-1 Supplement by
accident) or by local wiki admins. So, like linktrail before it, let's
move it out of the system messages and into a separate language
variable.

At the same time, let's make it a simple character set (like
$wgLegalTitleChars) rather than a complicated regular expression. The
complicated regex now lives in the parser.

This also adjusts the output of the API's action=query&meta=siteinfo and
adds an accessor parallel to the linkTrail accessor to Language.

Note the following changes that are not simply extracting the existing
charset from the linkprefix message for $linkPrefixCharset:
* The En message matched all non-ASCII UTF-8 characters by matching the
  component bytes (\\x80-\\xff). The new character set is equivalent.
* Various languages were identical to En and so have no $linkPrefixCharset
  set. These are: Ary Az Ce Ga Id Ka Kiu Km Ltg Mk Ms Ne Nn Ro Roa_tara Sc Si
  Sr_ec Sr_el Tl Tt_cyrl Tt_latn Ug_arab War
* Cu, Uk, and Udm are changed to match any number of „ or « in the prefix.
* Cv tried to include "«" that was redundant to the range \\x80-\\xff
  (see En comment). This was removed.
* Diq was entirely bogus, and so was removed.
* Gu included many additional UTF-8 characters that are redundant to the
  range \\x80-\\xff (see En comment). These were removed, and the
  resulting character set is equivalent to En.
* Mt has been broken since it was introduced in r37242. The charset used is
  equivalent to the broken regex.

Bug: 56031
Change-Id: I3369851b33113fc118a1bace38f3ac310cdd9725
2013-10-29 19:06:15 -04:00
Tim Starling
d1bc243f65 Remove all instances of the word "iff"
It's elitist mathematical jargon. In all cases dealt with here, it adds
no additional meaning compared to "if", beyond what was already obvious
from context. Thus, its only purpose is to smugly demonstrate that the
author attended their second-year mathematics classes, at the expense of
causing confusion for everyone who doesn't have such a background.

If you really think you need to convey extra information beyond what
"if" gives you, the English language contains plenty of devices for doing
so, without resorting to neologisms.

Change-Id: Iae21095d02ec2935c10e94f532235c2671c115b1
2013-10-23 10:56:54 -07:00
Bryan Davis
8b6be1c1ab Modernize SquidUpdate php code
- Update to PHP5 object coding style
- Fix @param comments using wrong argument order
- Add some phpdoc comments

Change-Id: Ia3190a6276e807ebef4f4aaeedc0e39819dc992a
2013-10-11 16:54:58 -06:00
MatmaRex
df8ec1e216 No spaces after (casts)
Also removed some unnecessary ones. I think I've caught them all.

The spaceless version already appears in core ~300 times (after
accounting for false positives when grepping). Some consistency would
be nice.

Change-Id: I607655b5f4366e66dc78730d5fd2f57ed8776cae
2013-09-04 20:05:43 +02:00
umherirrender
24bfde2710 Fix spacing and break some lines
Change-Id: Ia57685d8858e02e399ad5c75ce64d12609d340ac
2013-08-24 17:06:25 +02:00
Antoine Musso
e137df71af $wgHTCPMulticastRouting rename + multi hosts support
This patch does two things:

A) rename $wgHTCPMulticastRouting to $wgHTCPRouting since you can have
   MediaWiki send purge packets over unicast.
B) add support for multi hosts purge in the few cases one want to split
   the multicast groups per cache role but still want to purge more than
   one cache group.

A) rename

The rename adds deprecation notices in the comments and adds a back
compatibility layer in the case someone is already using this feature.
Given Wikimedia Foundation is not even using it, it is very unlikely,
but yet: better safe than sorry.

My logic is flawed sometime, so that needs a bit of review :) There is
two levels of deprecations to watch for:
- pre mw1.20 which useds $wgHTCPMulticastAddress and
  $wgHTCPMulticastPort
- pre mw1.22 (aka before this patch) that used $wgHTCPMulticastRouting

The resulting configuration should be properly loaded in $wgHTCPRouting.

B) multi hosts

The HTCP routing let you split purges to different hosts according to
the URL.  In some case, we want to be able to purge an URL from more
than one multicast group.  A Wikimedia example would be to send text
related purges to the text and mobiles caches while upload purges are
sent to the upload caches.

An abstracted example would be:

$wgHTCPRouting = array(

	// upload URLs to upload caches
	'/(upload|thumbs)/' => array(
		'host' => '<ip for upload caches>'
	),

	// Everything else to text & mobile
	'' => array(
		array( 'host' => '<ip for text caches>' ),
		array( 'host' => '<ip for mobile caches>' ),
	),

);

Change-Id: Ie87f6b81007f47f9cb439371743f3ad9a4b0c774
2013-07-31 16:07:05 +02:00
Antoine Musso
9cf0a3d81f HTCPPurge() early exit on socket error
I tend to dislike code style such as:

	if( $ok ) {
		// lot of code
	} else {
		return false;
	}

The SquidUpdate::HTCPPurge() used that pattern to handle a
socket_create() issue.  I have simply moved the block above and inverted
the condition.  Ie:

	if( ! $ok ) {
		return false;
	}
	// lot of code

That saves a level of indentation and makes the code a bit easier to
follow IMHO.

Change-Id: Ic3c6e22bbb637352fef740a0c1663a4b6f5a2b6a
2013-07-16 14:41:27 +00:00
jenkins-bot
403b4fed47 Merge "Clear shallowFallbacks in LocalisationCache::unload" 2013-07-02 02:18:24 +00:00
Brad Jorsch
316bbe97ca Clear shallowFallbacks in LocalisationCache::unload
When a language code is unloaded, it also needs to clear the
shallowFallbacks entry. Otherwise the next time that language is used it
will incorrectly think that the other setup done by initShallowFallbacks
is still set up, leading to warnings.

Change-Id: Idf6a78f56a3bb864cc921427ac82972594610047
2013-07-01 16:53:29 -04:00
Antoine Musso
cf99ae3f28 SquidUpdate debug message are now in 'squid' group
To debug purging issues in beta, it would be useful to have all squid
related messages grouped at the same place.  wfDebug() is logged on beta
but it has a good amount of spam which is not that useful.  Regrouping
all squid operations under the debug log group 'squid' will make it
easier to investigate.

Change-Id: I11112585f38307ae56ec3e05e7a8d6d9dab31bfa
2013-07-01 15:16:43 +02:00
Kevin Israel
876bddf637 Change @since and @deprecated notes to 1.22
Using the following command line, I have found doc comments mentioning
"1.21" when they should mention "1.22" instead, which I have fixed
manually:

git diff REL1_21 | grep --color=always -C 10 -iE \
'^\+.*(since|deprecated).*1\.21(\D|$)' | aha > oldver.html

I also moved the release notes for I1987190f ("Combine JavaScript and
JSON encoding logic") from RELEASE-NOTES-1.21 to RELEASE-NOTES-1.22
because I had reverted the commit on REL1_21 only (see Id3b88102 and
bug 47431 for the rationale).

Change-Id: I11b917a371e07267dfa98b8449776d0c1cb29b15
Follows-Up: I25cf5a94f6e47f85a9d0b80cc1c9c9f957288478
Follows-Up: I3d72e4105f6244b0695116940e62a2ddef66eb66
Follows-Up: I3faa9c3e8107c6e46cdf21f8c18adda1f42890d7
Follows-Up: I6aab19c8d68bf47beddad42632b0360a7b12f251
Follows-Up: I86368821fc2cd0729df5342b8572eb470c0f77a0
Follows-Up: Id3b88102e768318e3605a19e9952121091a40915
Follows-Up: Ie667088010e24eb6cb569f9e8e8e2553005223eb
2013-06-21 05:33:22 +00:00
Brian Wolff
485daef140 UserCache::getUserName: Docs said @since 1.21, but not in REL1_21
Change-Id: I4ad85ee36ed348e8b082b5a4dba11d7cb92d8ad0
Follow-up: ef915e37e3
2013-06-19 16:17:52 -03:00
Tyler Anthony Romeo
126f66d955 Fix return type of MessageCache::getMsgFromNamespace for existing
Functions expect the message cache to return a string if a message
exists, even empty, and false if it does not exist. This adds casting
to the substr() function, which would return false for existing
messages that were just blank.

Bug: 14176
Change-Id: Id91914a3701fe53f1e2e894824512489392c628b
2013-06-12 08:48:05 +10:00
Antoine Musso
ca934127d7 LinkCache singleton can now be altered
This patch adds the ability to destroy the LinkCache singleton or to
replace the instance with a different object.  Note the $instance static
variable is now at the class level instead of at the method level.

Change-Id: Ib110ad061868834a52a6bac651976b3ffab4a6ce
2013-06-01 09:20:02 +02:00
jenkins-bot
a492cf4c34 Merge "doc: various updates" 2013-05-30 08:46:35 +00:00
Tyler Romeo
86b54375de Enable fallback languages when retrieving messages
The core function behind wfMessage() (MessageCache->get()) did not
apply the language fallback chain to on-wiki messages.

This patch has changed the behavior to iterate over all possible
languages, first checking on-wiki and then checking the CDB cache,
until it finds the message. Note that fallback languages never
take precedence over the actual requested language.

This patch was taken from the following changes and then
adjusted to fix issues that caused bug 46579.

* Change-Id: Iaaf6ccebd8c40c9602748c58c3a5c73c29e7aa4d
- Author: Matthew Walker <mwalker@wikimedia.org>
- (cherry picked from commit d434bfcf3b)
* Change-Id: Ib607a446d3499a3c042dce408db5cbaf12fa9e3d
- Author: Mormegil <mormegil@centrum.cz>
- (cherry picked from commit 1b8cb8dc31)

Bug: 1495
Bug: 46579
Change-Id: I420457863eeb79824698d06abc7784032b267af2
2013-05-26 15:46:35 +02:00
Antoine Musso
43ae2fb6f9 doc: various updates
[includes/cache/MessageCache.php]
- internal constants
- constructor
- a few missing @var

[includes/clientpool/RedisConnectionPool.php]
- group internal settings applying to the pool
- misc protected members updates

[includes/debug/Debug.php]
- missing parameter name in @var statements

Change-Id: I6ff0a68d659529d128f40f32b0fd1c1d39af952f
2013-05-20 12:02:37 +02:00
umherirrender
15ff79312d Fixed spacing and removed unneeded parenthesis
Added spaces after/before parenthesis
Removed unneeded parenthesis around some statements
Broke a long line

Change-Id: I7fbe129f7bbf524dd0598ece2a9708643f08453b
2013-05-17 16:12:08 +00:00
Timo Tijhof
4bd5471ca3 docs: Remove odd colons after @todo
Most were this way already:
https://doc.wikimedia.org/mediawiki-core/master/php/html/todo.html

Ran a find/replace on the odd ones. Also made them all
lower case.

Change-Id: I70c6a69344ddebc603e9a1c1d87e3cc4f4f4c560
2013-05-15 06:23:40 +00:00
Timo Tijhof
50e7985d4d phpcs: Fix WhiteSpace.LanguageConstructSpacing warnings
Squiz.WhiteSpace.LanguageConstructSpacing:
   Language constructs must be followed by a single space;
   expected "require_once expression" but found
   "require_once(expression)"

It is a keyword (e.g. like `new`, `return` and `print`). As
such the parentheses don't make sense.

Per our code conventions, we use a space after keywords like
these. We appeared to have an unwritten exception for `require`
that doesn't make sense. About 60% of require/include usage
was missing the space and/or had superfluous parentheses.

It is as silly as print("foo") or return("foo"), it works
because keywords have no significance for whitespace between
it and the expression that follows, and since experessions can
be wrapped in parentheses for clarity (e.g. when doing string
concatenation or mathematical operations) the parenthesis
before and after basiclaly just ignored.

Change-Id: I2df2f80b8123714bea7e0771bf94b51ad5bb4b87
2013-05-09 05:56:26 +02:00
Reedy
d560eb0b3e Revert "Commit of various live hacks"
This reverts commit c6af934ae6.

Change-Id: I3e863bb632531d7cd4ec0614325a7c4a7ba9aa3d
2013-04-29 19:19:54 +01:00
Reedy
c6af934ae6 Commit of various live hacks
Disable of MessageBlobStore clear

Reset $wgAutopromote (should be moved to a config change!!)

Disable setting of wgStyleSheetPath

Throttle page_touched

Add apc htcp packet numbers to SquidUpdate

Disable set names binary/utf8

Commment out searchindex table indexes

Was c532e81d583d3d0439fe76eea4d105d675461b56

Original revision Change-Id I42c4f859e55eb198f6c6841e582b3552aad7b31f
https://gerrit.wikimedia.org/r/#/c/7606

Change-Id: I5ec8dd53188e9e4128f99ceaff38ebf9dcf570bb
2013-04-29 15:46:38 +00:00
umherirrender
6f79eef473 Fixed spacing around parenthesis in includes
Change-Id: Ie8adc00f4ee8ecec4554e584c18d5d2073415397
2013-04-28 15:50:07 +00:00
umherirrender
1bfc8feb25 Fixed spacing in actions/cache/filebackend/filerepo/job folder
Added spaces before if, foreach
Added some braces for one line statements

Change-Id: Idb93d34e314e5f314223b79208968d6bcd30c40e
2013-04-20 19:18:13 +02:00
Brian Wolff
de1aefd55c Make UserCache only look up a user once.
I noticed on special:listfiles/username, UserCache was doing queries
like select user_whatever from user where user_id in ( '1', '1', ...)
with the same user id 50 times (one for each result returned in the
special page). That seemed a little insane, so put the list of
users to query through a array_unique. (Quite likely the db would
optimize that query to not literally look up the same user 50 times,
but nonetheless it seems better to filter the list before then)

Change-Id: I80c8a359d0f7a53b2420ebcda641e594dd3c56e9
2013-04-17 18:28:05 -03:00
Aaron Schulz
9906460241 Use onTransactionIdle() for page invalidations when not using the job queue.
* This will do the page_touched UPDATE in autocommit mode to avoid
  locking a bunch of titles for some unknown amount of time that
  depends on how big the transaction is.

Change-Id: I6ec16a73c11fbf19c24d2b7fe7d01f91b9dcf33a
2013-04-16 10:16:10 -07:00
jenkins-bot
562acce759 Merge "Cleanup and performance tweaks for BacklinkCache." 2013-04-16 06:50:19 +00:00
umherirrender
15abcf71ca Added/Removed spaces around string concatenation
And added/removed spaces around some other tokens,
like +, -, *, /, <, >, =, !

Fixed windows newline style

Change-Id: I0b9c8c408f3f6bfc0d685a074d7ec468fb848fc8
2013-04-13 13:36:24 +02:00
Aaron Schulz
135ae433d2 Made $wgMaxBacklinksInvalidate actually work.
* The check never worked before due to the 200 limit passed to getNumLinks().

Change-Id: I009c757c6437f22544a8d7d09d3534950c32c887
2013-04-12 10:48:10 -07:00
Siebrand Mazeland
57f7bfc662 Update documentation and styling
Remove some trailing whitespace, break long lines, make PHPDoc consistent
and update it in places.

Change-Id: I340b43a47e121a44ebd4aaa2a45dff12e945e645
2013-04-11 17:01:18 +02:00
Tim Starling
3c3ba5e03e Fix message cache expiry semantics
* Use the stale message cache while the new one is being generated
* Revert I811755d4 (make message cache load failure fatal). This
  escalated several very plausible temporary site issues from barely
  noticeable to complete downtime -- for example, memcached being down
  on a site with only one memcached server.
* Remove $wgLocalMessageCacheSerialized, it's always been pointless
* Clarify a couple of comments.
* Increased lock wait timeout to 30s
* Make lock() fail immediately on memcached connection refused

Tests done:
* With local cache enabled: normal cold refill; refill local from
  global cache; use stale local cache during remote refill; use stale
  global cache during remote refill; cold cache wait for remote refill;
  saveToCaches() failure; memcached connection refused.
* With local cache disabled: saveToCaches() failure; cache disabled due
  to "error" status key; memcached connection refused.

Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is
not the best idea since it inevitably leads to a cache stampede. Dealing
with the stampede by waiting for a lock is not ideal, even if it were
implemented properly, since it's not necessary to deliver perfectly
fresh message cache data to all clients.

This is especially obvious when you note that barring bugs, expiry and
regeneration always gives you back the exact same data, because we have
incremental updates (MessageCache::replace()). Keeping all clients
waiting for 10s just to give them the data they have already is pretty
pointless.

So, continue to serve the site from the stale message cache while the
new one is being generated.

One caveat: if local caching enabled, when the message cache becomes
stale, a sudden spike in network bandwidth may result due to the full
array (also typically stale) being fetched from the shared cache.

Bug: 43516
Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-09 20:40:20 -07:00
umherirrender
978bb31c5e Add missing wfProfileOut before throwing an exception
Change-Id: I1d830da0597f19efd0b2ae48642389975e736e23
2013-04-08 18:37:24 +00:00
umherirrender
007c1dc1d6 Add missing wfProfileOut to LocalisationCache
Follow up Ib94a8c18c4e270a7bd46faa17eb27a22ff950f75

Moved some empty lines to match the places of other profile calls in the
file.

Change-Id: I6616b18027cdb9e70b43948aaff999a2e25b07e0
2013-04-08 19:37:17 +02:00
Max Semenik
da5c29cdd4 LocalisationCache: add profiling, fix doxygen warnings
Change-Id: Ib94a8c18c4e270a7bd46faa17eb27a22ff950f75
2013-04-06 01:18:40 +04:00
Aaron Schulz
4eb056d549 Cleanup and performance tweaks for BacklinkCache.
* Batch the queries to get the partitions.
* Improved caching of getNumLinks() with $max.
* Added a TTL to the ProcessCacheLRU use for sanity.
* Some small logic cleanups in partitionResult().
* Also cleaned up some code duplication.

Bug: 43452
Change-Id: I0b9710fe222b3d2cb4dc9ab2eeb0758873a8066c
2013-04-04 14:54:53 -07:00
Mwalker
757fb8071a Merge "Use MessageCache::getMsgFromNamespace() to fetch conversion tables." 2013-03-28 21:42:13 +00:00
Siebrand Mazeland
9d7e9c3141 Revert changes in fallback behavior
This patch set reverts the following:
* Iaaf6ccebd8c40c9602748c58c3a5c73c29e7aa4d
* Ib607a446d3499a3c042dce408db5cbaf12fa9e3d
* Ic59fd20856eb0489d70f3469a56ebce0efb3db13 (partially)

Bug 46579 comment 17 describes a desired solution. In
If88923119179924a5ec091394ccab000ade16b3e we are working on a fix, but it is
taking longer than we anticipated. There was a deployment window planned
about now, but we didn't make it. It makes sense to revert for now, and commit
a proper solution somewhere next week.

Bug: 46579
Bug: 1495
Change-Id: Iac7ac4357dd80e8cdb238a6a207393f0712b3fe5
2013-03-28 10:32:02 +00:00
Liangent
080a017fd9 Use MessageCache::getMsgFromNamespace() to fetch conversion tables.
After Iaaf6cceb, MessageCache::get() goes through the fallback chain,
which is unwanted for conversion tables (for example, we don't want
zh-hans table for zh, where zh means "no conversion"). Since the only
needed feature is to fetch text from MediaWiki namespace (conversion
tables in PHP are stored somewhere else), it is now changed to use
MessageCache::getMsgFromNamespace() instead to avoid fallbacks.

Change-Id: I46e0be31c9c0fe0a6e4923fc1aff0fbbadbf1d67
2013-03-28 13:47:56 +08:00
jenkins-bot
dc07116c22 Merge "Added UserCache::getUserName() convenience function." 2013-03-27 11:39:36 +00:00
umherirrender
6c278b6d7e fix some spacing
* Removed spaces around array index
* Removed double spaces or added spaces to begin or end of function
  calls, method signature, conditions or foreachs
* Added braces to one-line ifs
* Changed multi line conditions to one line conditions
* Realigned some arrays

Change-Id: Ia04d2a99d663b07101013c2d53b3b2e872fd9cc3
2013-03-25 22:22:46 +00:00
Kaldari
06b0967caa Allow the retrieval of the plural rule type for a given number
For example, find out which rule type should be applied for 5 items
in Arabic. The result would be 'few'.

This implementation should be non-disruptive and completely backwards
compatible (which is the main reason it isn't a lot simpler).

Change-Id: I3d72e4105f6244b0695116940e62a2ddef66eb66
2013-03-20 14:34:12 +05:30
Aaron Schulz
ef915e37e3 Added UserCache::getUserName() convenience function.
Change-Id: Ice6b5e8608927db588edb1e8458f7d74e53f1214
2013-03-18 15:03:49 -07:00
ASchulz
5bafa9545a Tweaked MessageCache locks to handle exceptions better.
Change-Id: I3066d8dbebc97abcc0567d71625f995d62549b4c
2013-03-15 05:47:44 +00:00
jenkins-bot
6114f05fc1 Merge "Correct case of Title::getPrefixedDBkey() calls" 2013-03-12 15:13:21 +00:00
Tyler Anthony Romeo
4dcc7961df Fixed @param tags to conform with Doxygen format.
Doxygen expects parameter types to come before the
parameter name in @param tags. Used a quick regex
to switch everything around where possible. This
only fixes cases where a primitve variable (or a
primitive followed by other types) is the variable
type. Other cases will need to be fixed manually.

Change-Id: Ic59fd20856eb0489d70f3469a56ebce0efb3db13
2013-03-11 13:15:01 -04:00
Alexandre Emsenhuber
cd4dd7ca85 Correct case of Title::getPrefixedDBkey() calls
Change-Id: Ic9ef0cc6dbb0a8eb0ee93432299ef59759223e2c
2013-03-09 21:14:22 +01:00
umherirrender
d63121016d fix some spacing
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments

Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
2013-03-07 17:53:21 +01:00
Matthew Walker
d434bfcf3b (bug 1495) Enable on-wiki message language fallbacks
The core function behind wfMessage() (MessageCache->get()) did not
apply the language fallback chain to on-wiki messages.

This patch has changed the behavior to iterate over all possible
on-wiki fallbacks (starting with the user's language) before
using the built-in language cache (CDB files). Previously we only looked
for the existence of an on-wiki message in the users's language.

Performance wise, using the 'ab' language ('ru', 'en' fallbacks)
MessageCache::get (Averaged over runs and calls)
New Code: ~8.5% TET (110us/call)
Old Code: ~6.5% TET ( 90us/call)

TET: Total Execution Time

Change-Id: Iaaf6ccebd8c40c9602748c58c3a5c73c29e7aa4d
2013-03-06 19:54:02 +00:00
umherirrender
909d61056a fix hardcoded quote in BacklinkCache
Also make the IS NULL in upper case, looks better and sql reservered
words often written in upper case

Change-Id: I844ec37bbf3fb00d95a43dfd6f58db1e67724d53
2013-02-18 08:35:26 +00:00
IAlex
3f92a5672e Merge "remove space before semicolon" 2013-02-10 12:51:02 +00:00
umherirrender
ca31ebad3f fix some spacing
Added/removed spaces after opening/before closing parentheses

Added a space after a comma

Removed unneeded parentheses in condition

Change-Id: I306091347ccaaf11dee0cdfda3019cb0c12be51b
2013-02-09 23:03:53 +01:00
umherirrender
f3cf109e62 remove space before semicolon
Change-Id: Ic0d890f5e27cec017c7f7910a67d53b2edf82079
2013-02-09 22:44:24 +01:00
umherirrender
be90ce07b3 fix some spacing
Change-Id: I93f0a87ba7129bc336083e7289247d3150feb606
2013-02-04 19:54:07 +00:00
umherirrender
2e8da558ba fix some spacing
Change-Id: Id7eda67a43f9040117edd79fdbeb678f1c3a6da2
2013-01-26 22:11:09 +01:00
jenkins-bot
3d2cedf096 Merge "Use ResultWrapper::numRows() instead of DatabaseBase::numRows()" 2013-01-11 23:10:05 +00:00
Aaron Schulz
6d2ed516be Fixed prefixed title text call.
Change-Id: I796a2cc1687a7d50c8874b002839133f5f5326c7
2013-01-10 21:30:48 -08:00
Aaron Schulz
b23188b8ad Added $wgMaxBacklinksInvalidate to avoid massive html cache invalidation.
Change-Id: I4fadded0e351883a907d307d64af31cdc6164240
2013-01-07 01:50:47 +00:00
Alexandre Emsenhuber
d85239a186 Use ResultWrapper::numRows() instead of DatabaseBase::numRows()
Change-Id: I87f059aaff72bf06ebc72ad3a7904cca8e11fab9
2013-01-06 11:52:40 +01:00
Aaron Schulz
c9e8a7012f [MessageCache] Cleaned message cache load() error handling.
Change-Id: Ic8d62fa4162b54c776d934bd7106978fb36a6d32
2012-12-19 12:50:53 -08:00
Aaron Schulz
940b57f1d3 Moved LocalisationCache under /cache.
Change-Id: I54b230376ed682d6a6b7d6fa4fd5ab133c8c8b1b
2012-12-16 07:14:48 +00:00
Chad Horohoe
551d81df41 Revert "Commit of various live hacks"
This reverts commit acddbc862a.

Change-Id: Iabf5f917938c347ca6ddc7a373c731f44f23363b
2012-12-10 20:56:45 -05:00
Reedy
acddbc862a Commit of various live hacks
Disable of MessageBlobStore clear

Reset $wgAutopromote (should be moved to a config change!!)

Disable setting of wgStyleSheetPath

Disable squid updates in HTMLCacaheUpdate invalidateTitles

Throttle page_touched

Add apc htcp packet numbers to SquidUpdate

Disable set names binary/utf8

Commment out searchindex table indexes

Was c532e81d583d3d0439fe76eea4d105d675461b56

Original revision Change-Id I42c4f859e55eb198f6c6841e582b3552aad7b31f
https://gerrit.wikimedia.org/r/#/c/7606

Change-Id: I5ec8dd53188e9e4128f99ceaff38ebf9dcf570bb
2012-12-10 13:33:12 -05:00
Reedy
d537d96868 Add numerous missing @throws to method documentation
Change-Id: Iba868e82a75fef7c7d011bc5be192bf059d037c0
2012-12-09 03:09:48 +00:00
Niklas Laxström
3117541201 Treat message cache loading as fatal
I suspect this is the primary cause we occasionally have
* Sidebar stuck in default values
* No gadgets at all
* Main page full of <int:custom-message> occurances

Change-Id: I811755d4c250800177c4ea9e509bf6e1d13443a0
2012-12-04 12:35:01 +00:00
Aaron Schulz
5ef62175bf [JobQueue] Improved refreshLinks/htmlCacheUpdate job de-duplication.
* Added JobQueue::deduplicateRootJob() function which uses cache
  records of the last time a "root" job was initiated for a task
  in order to invalidate prior jobs for that task. For refreshLinks,
  the "task" is basically "enqueueing the refresh jobs for title X".
* (bug 27914) Also added new Job::getDeduplicationFields() function
  and made use of it with refreshLinks to exclude things like 'masterPos'
  from duplicate job check comparisons for refreshLinks.
* (bug 27914) Always resolve refreshLinks2 jobs down to refreshLinks jobs.
  For each affected pages, one of them will get their job popped
  first, which will remove the duplicates for that page unless
  one page is in a refreshLinks2 jobs and the other in refreshLinks.
* (bug 37731) Made LinksUpdate/HTMLCacheUpdate defer the large
  backlinks query by doing it in an outer job.
* (bug 42065) HTMLCacheUpdate will no longer purge pages that were
  already purged since the job was added.

Change-Id: I71b743e0a38e60a874ca856e80cb761bea06b689
2012-11-28 09:29:41 +00:00
Aaron Schulz
12b76fcabb Made BackLinkCache use object caching better.
* This way, we can actually get persistent cache hits for
the HTMLCacheUpdate queries rather than always hitting a DB.
* Also moved BacklinkCache under the /cache directory.

Change-Id: I0666ee575fb42675f1a7dd9cb52665f0a12a66a9
2012-11-27 09:41:50 -08:00