wiki.techinc.nl/includes/cache/MessageCache.php

977 lines
28 KiB
PHP
Raw Normal View History

<?php
/**
* Localisation messages cache.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License along
* with this program; if not, write to the Free Software Foundation, Inc.,
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
* http://www.gnu.org/copyleft/gpl.html
*
* @file
* @ingroup Cache
*/
/**
*
*/
define( 'MSG_LOAD_TIMEOUT', 60 );
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
define( 'MSG_LOCK_TIMEOUT', 30 );
define( 'MSG_WAIT_TIMEOUT', 30 );
define( 'MSG_CACHE_VERSION', 1 );
/**
* Message cache
* Performs various MediaWiki namespace-related functions
* @ingroup Cache
*/
2005-12-04 18:27:59 +00:00
class MessageCache {
/**
* Process local cache of loaded messages that are defined in
* MediaWiki namespace. First array level is a language code,
* second level is message key and the values are either message
* content prefixed with space, or !NONEXISTENT for negative
* caching.
*/
protected $mCache;
// Should mean that database cannot be used, but check
protected $mDisable;
/// Lifetime for cache, used by object caching
protected $mExpiry;
/**
* Message cache has it's own parser which it uses to transform
* messages.
*/
protected $mParserOptions, $mParser;
/// Variable for tracking which variables are already loaded
protected $mLoadedLanguages = array();
/**
* Singleton instance
2011-04-25 22:41:54 +00:00
*
* @var MessageCache
*/
private static $instance;
/**
* @var bool
*/
protected $mInParser = false;
/**
* Get the signleton instance of this class
*
* @since 1.18
* @return MessageCache object
*/
public static function singleton() {
if ( is_null( self::$instance ) ) {
global $wgUseDatabaseMessages, $wgMsgCacheExpiry;
self::$instance = new self( wfGetMessageCacheStorage(), $wgUseDatabaseMessages, $wgMsgCacheExpiry );
}
return self::$instance;
}
/**
* Destroy the singleton instance
*
* @since 1.18
*/
public static function destroyInstance() {
self::$instance = null;
}
function __construct( $memCached, $useDB, $expiry ) {
if ( !$memCached ) {
$memCached = wfGetCache( CACHE_NONE );
}
$this->mMemc = $memCached;
$this->mDisable = !$useDB;
$this->mExpiry = $expiry;
}
/**
* ParserOptions is lazy initialised.
*
* @return ParserOptions
*/
Merged localisation-work branch: * Made lines from initialiseMessages() appear as list items during installation * Moved the bulk of the localisation data from the Language*.php files to the Messages*.php files. Deleted most of the Languages*.php files. * Introduced "stub global" framework to provide deferred initialisation of core modules. * Removed placeholder values for $wgTitle and $wgArticle, these variables will now be null during the initialisation process, until they are set by index.php or another entry point. * Added DBA cache type, for BDB-style caches. * Removed custom date format functions, replacing them with a format string in the style of PHP's date(). Used string identifiers instead of integer identifiers, in both the language files and user preferences. Migration should be transparent in most cases. * Simplified the initialisation API for LoadBalancer objects. * Removed the broken altencoding feature. * Moved default user options and toggles from Language to User. Language objects are still able to define default preference overrides and extra user toggles, via a slightly different interface. * Don't include the date option in the parser cache rendering hash unless $wgUseDynamicDates is enabled. * Merged LanguageUtf8 with Language. Removed LanguageUtf8.php. * Removed inclusion of language files from the bottom of Language.php. This is now consistently done from Language::factory(). * Add the name of the executing maintenance script to the debug log. Start the profiler during maintenance scripts. * Added "serialized" directory, for storing precompiled data in serialized form.
2006-07-26 07:15:39 +00:00
function getParserOptions() {
if ( !$this->mParserOptions ) {
$this->mParserOptions = new ParserOptions;
$this->mParserOptions->setEditSection( false );
Merged localisation-work branch: * Made lines from initialiseMessages() appear as list items during installation * Moved the bulk of the localisation data from the Language*.php files to the Messages*.php files. Deleted most of the Languages*.php files. * Introduced "stub global" framework to provide deferred initialisation of core modules. * Removed placeholder values for $wgTitle and $wgArticle, these variables will now be null during the initialisation process, until they are set by index.php or another entry point. * Added DBA cache type, for BDB-style caches. * Removed custom date format functions, replacing them with a format string in the style of PHP's date(). Used string identifiers instead of integer identifiers, in both the language files and user preferences. Migration should be transparent in most cases. * Simplified the initialisation API for LoadBalancer objects. * Removed the broken altencoding feature. * Moved default user options and toggles from Language to User. Language objects are still able to define default preference overrides and extra user toggles, via a slightly different interface. * Don't include the date option in the parser cache rendering hash unless $wgUseDynamicDates is enabled. * Merged LanguageUtf8 with Language. Removed LanguageUtf8.php. * Removed inclusion of language files from the bottom of Language.php. This is now consistently done from Language::factory(). * Add the name of the executing maintenance script to the debug log. Start the profiler during maintenance scripts. * Added "serialized" directory, for storing precompiled data in serialized form.
2006-07-26 07:15:39 +00:00
}
return $this->mParserOptions;
}
2006-01-07 13:09:30 +00:00
/**
* Try to load the cache from a local file.
*
* @param string $hash the hash of contents, to check validity.
* @param $code Mixed: Optional language code, see documenation of load().
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
* @return The cache array
*/
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
function getLocalCache( $hash, $code ) {
global $wgCacheDirectory;
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
$filename = "$wgCacheDirectory/messages-" . wfWikiID() . "-$code";
# Check file existence
wfSuppressWarnings();
$file = fopen( $filename, 'r' );
wfRestoreWarnings();
if ( !$file ) {
return false; // No cache file
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
// Check to see if the file has the hash specified
$localHash = fread( $file, 32 );
if ( $hash === $localHash ) {
// All good, get the rest of it
$serialized = '';
while ( !feof( $file ) ) {
$serialized .= fread( $file, 100000 );
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
fclose( $file );
return unserialize( $serialized );
} else {
fclose( $file );
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
return false; // Wrong hash
}
}
/**
* Save the cache to a local file.
*/
function saveToLocal( $serialized, $hash, $code ) {
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
global $wgCacheDirectory;
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
$filename = "$wgCacheDirectory/messages-" . wfWikiID() . "-$code";
wfMkdirParents( $wgCacheDirectory, null, __METHOD__ ); // might fail
wfSuppressWarnings();
$file = fopen( $filename, 'w' );
wfRestoreWarnings();
if ( !$file ) {
wfDebug( "Unable to open local cache file for writing\n" );
return;
}
fwrite( $file, $hash . $serialized );
fclose( $file );
wfSuppressWarnings();
chmod( $filename, 0666 );
wfRestoreWarnings();
}
/**
* Loads messages from caches or from database in this order:
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
* (1) local message cache (if $wgUseLocalMessageCache is enabled)
* (2) memcached
* (3) from the database.
*
* When succesfully loading from (2) or (3), all higher level caches are
* updated for the newest version.
*
2010-08-07 23:41:03 +00:00
* Nothing is loaded if member variable mDisable is true, either manually
* set by calling code or if message loading fails (is this possible?).
*
* Returns true if cache is already populated or it was succesfully populated,
* or false if populating empty cache fails. Also returns true if MessageCache
* is disabled.
*
* @param bool|String $code String: language to which load messages
* @throws MWException
2012-02-09 21:33:27 +00:00
* @return bool
*/
function load( $code = false ) {
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
global $wgUseLocalMessageCache;
if( !is_string( $code ) ) {
# This isn't really nice, so at least make a note about it and try to
# fall back
wfDebug( __METHOD__ . " called without providing a language code\n" );
$code = 'en';
}
# Don't do double loading...
if ( isset( $this->mLoadedLanguages[$code] ) ) {
return true;
}
# 8 lines of code just to say (once) that message cache is disabled
if ( $this->mDisable ) {
2005-05-28 11:07:55 +00:00
static $shownDisabled = false;
if ( !$shownDisabled ) {
wfDebug( __METHOD__ . ": disabled\n" );
2005-05-28 11:07:55 +00:00
$shownDisabled = true;
}
return true;
}
# Loading code starts
wfProfileIn( __METHOD__ );
$success = false; # Keep track of success
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$staleCache = false; # a cache array with expired data, or false if none has been loaded
$where = array(); # Debug info, delayed to avoid spamming debug log too much
$cacheKey = wfMemcKey( 'messages', $code ); # Key in memc for messages
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
# Local cache
# Hash of the contents is stored in memcache, to detect if local cache goes
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
# out of date (e.g. due to replace() on some other server)
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
if ( $wgUseLocalMessageCache ) {
wfProfileIn( __METHOD__ . '-fromlocal' );
$hash = $this->mMemc->get( wfMemcKey( 'messages', $code, 'hash' ) );
if ( $hash ) {
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$cache = $this->getLocalCache( $hash, $code );
if ( !$cache ) {
$where[] = 'local cache is empty or has the wrong hash';
} elseif ( $this->isCacheExpired( $cache ) ) {
$where[] = 'local cache is expired';
$staleCache = $cache;
} else {
$where[] = 'got from local cache';
$success = true;
$this->mCache[$code] = $cache;
}
}
wfProfileOut( __METHOD__ . '-fromlocal' );
}
if ( !$success ) {
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
# Try the global cache. If it is empty, try to acquire a lock. If
# the lock can't be acquired, wait for the other thread to finish
# and then try the global cache a second time.
for ( $failedAttempts = 0; $failedAttempts < 2; $failedAttempts++ ) {
wfProfileIn( __METHOD__ . '-fromcache' );
$cache = $this->mMemc->get( $cacheKey );
if ( !$cache ) {
$where[] = 'global cache is empty';
} elseif ( $this->isCacheExpired( $cache ) ) {
$where[] = 'global cache is expired';
$staleCache = $cache;
} else {
$where[] = 'got from global cache';
$this->mCache[$code] = $cache;
$this->saveToCaches( $cache, 'local-only', $code );
$success = true;
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
wfProfileOut( __METHOD__ . '-fromcache' );
if ( $success ) {
# Done, no need to retry
break;
}
# We need to call loadFromDB. Limit the concurrency to a single
# process. This prevents the site from going down when the cache
# expires.
$statusKey = wfMemcKey( 'messages', $code, 'status' );
$acquired = $this->mMemc->add( $statusKey, 'loading', MSG_LOAD_TIMEOUT );
if ( $acquired ) {
# Unlock the status key if there is an exception
$that = $this;
$statusUnlocker = new ScopedCallback( function() use ( $that, $statusKey ) {
$that->mMemc->delete( $statusKey );
} );
# Now let's regenerate
$where[] = 'loading from database';
# Lock the cache to prevent conflicting writes
# If this lock fails, it doesn't really matter, it just means the
# write is potentially non-atomic, e.g. the results of a replace()
# may be discarded.
if ( $this->lock( $cacheKey ) ) {
$mainUnlocker = new ScopedCallback( function() use ( $that, $cacheKey ) {
$that->unlock( $cacheKey );
} );
} else {
$mainUnlocker = null;
$where[] = 'could not acquire main lock';
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$cache = $this->loadFromDB( $code );
$this->mCache[$code] = $cache;
$success = true;
$saveSuccess = $this->saveToCaches( $cache, 'all', $code );
# Unlock
ScopedCallback::consume( $mainUnlocker );
ScopedCallback::consume( $statusUnlocker );
if ( !$saveSuccess ) {
# Cache save has failed.
# There are two main scenarios where this could be a problem:
#
# - The cache is more than the maximum size (typically
# 1MB compressed).
#
# - Memcached has no space remaining in the relevant slab
# class. This is unlikely with recent versions of
# memcached.
#
# Either way, if there is a local cache, nothing bad will
# happen. If there is no local cache, disabling the message
# cache for all requests avoids incurring a loadFromDB()
# overhead on every request, and thus saves the wiki from
# complete downtime under moderate traffic conditions.
if ( !$wgUseLocalMessageCache ) {
$this->mMemc->set( $statusKey, 'error', 60 * 5 );
$where[] = 'could not save cache, disabled globally for 5 minutes';
} else {
$where[] = "could not save global cache";
}
}
# Load from DB complete, no need to retry
break;
} elseif ( $staleCache ) {
# Use the stale cache while some other thread constructs the new one
$where[] = 'using stale cache';
$this->mCache[$code] = $staleCache;
$success = true;
break;
} elseif ( $failedAttempts > 0 ) {
# Already retried once, still failed, so don't do another lock/unlock cycle
# This case will typically be hit if memcached is down, or if
# loadFromDB() takes longer than MSG_WAIT_TIMEOUT
$where[] = "could not acquire status key.";
break;
} else {
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$status = $this->mMemc->get( $statusKey );
if ( $status === 'error' ) {
# Disable cache
break;
} else {
# Wait for the other thread to finish, then retry
$where[] = 'waited for other thread to complete';
$this->lock( $cacheKey );
$this->unlock( $cacheKey );
}
}
}
}
if ( !$success ) {
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$where[] = 'loading FAILED - cache is disabled';
$this->mDisable = true;
$this->mCache = false;
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
# This used to throw an exception, but that led to nasty side effects like
# the whole wiki being instantly down if the memcached server died
} else {
# All good, just record the success
$this->mLoadedLanguages[$code] = true;
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$info = implode( ', ', $where );
wfDebug( __METHOD__ . ": Loading $code... $info\n" );
wfProfileOut( __METHOD__ );
return $success;
}
/**
* Loads cacheable messages from the database. Messages bigger than
* $wgMaxMsgCacheEntrySize are assigned a special value, and are loaded
* on-demand from the database later.
*
* @param string $code language code.
* @return Array: loaded messages for storing in caches.
*/
function loadFromDB( $code ) {
wfProfileIn( __METHOD__ );
global $wgMaxMsgCacheEntrySize, $wgLanguageCode, $wgAdaptiveMessageCache;
$dbr = wfGetDB( DB_SLAVE );
$cache = array();
# Common conditions
$conds = array(
'page_is_redirect' => 0,
'page_namespace' => NS_MEDIAWIKI,
);
$mostused = array();
if ( $wgAdaptiveMessageCache && $code !== $wgLanguageCode ) {
if ( !isset( $this->mCache[$wgLanguageCode] ) ) {
$this->load( $wgLanguageCode );
}
$mostused = array_keys( $this->mCache[$wgLanguageCode] );
foreach ( $mostused as $key => $value ) {
$mostused[$key] = "$value/$code";
}
}
if ( count( $mostused ) ) {
$conds['page_title'] = $mostused;
} elseif ( $code !== $wgLanguageCode ) {
$conds[] = 'page_title' . $dbr->buildLike( $dbr->anyString(), '/', $code );
} else {
# Effectively disallows use of '/' character in NS_MEDIAWIKI for uses
# other than language code.
$conds[] = 'page_title NOT' . $dbr->buildLike( $dbr->anyString(), '/', $dbr->anyString() );
}
# Conditions to fetch oversized pages to ignore them
$bigConds = $conds;
$bigConds[] = 'page_len > ' . intval( $wgMaxMsgCacheEntrySize );
# Load titles for all oversized pages in the MediaWiki namespace
$res = $dbr->select( 'page', 'page_title', $bigConds, __METHOD__ . "($code)-big" );
foreach ( $res as $row ) {
$cache[$row->page_title] = '!TOO BIG';
}
# Conditions to load the remaining pages with their contents
$smallConds = $conds;
$smallConds[] = 'page_latest=rev_id';
$smallConds[] = 'rev_text_id=old_id';
$smallConds[] = 'page_len <= ' . intval( $wgMaxMsgCacheEntrySize );
$res = $dbr->select(
array( 'page', 'revision', 'text' ),
array( 'page_title', 'old_text', 'old_flags' ),
$smallConds,
__METHOD__ . "($code)-small"
);
foreach ( $res as $row ) {
$text = Revision::getRevisionText( $row );
if( $text === false ) {
// Failed to fetch data; possible ES errors?
// Store a marker to fetch on-demand as a workaround...
$entry = '!TOO BIG';
wfDebugLog( 'MessageCache', __METHOD__ . ": failed to load message page text for {$row->page_title} ($code)" );
} else {
$entry = ' ' . $text;
}
$cache[$row->page_title] = $entry;
2006-01-07 13:31:29 +00:00
}
$cache['VERSION'] = MSG_CACHE_VERSION;
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$cache['EXPIRY'] = wfTimestamp( TS_MW, time() + $this->mExpiry );
wfProfileOut( __METHOD__ );
return $cache;
}
/**
* Updates cache as necessary when message page is changed
*
* @param string $title name of the page changed.
* @param $text Mixed: new contents of the page.
*/
public function replace( $title, $text ) {
global $wgMaxMsgCacheEntrySize;
wfProfileIn( __METHOD__ );
if ( $this->mDisable ) {
2011-02-10 16:04:19 +00:00
wfProfileOut( __METHOD__ );
return;
}
list( $msg, $code ) = $this->figureMessage( $title );
$cacheKey = wfMemcKey( 'messages', $code );
$this->load( $code );
$this->lock( $cacheKey );
$titleKey = wfMemcKey( 'messages', 'individual', $title );
if ( $text === false ) {
# Article was deleted
$this->mCache[$code][$title] = '!NONEXISTENT';
$this->mMemc->delete( $titleKey );
} elseif ( strlen( $text ) > $wgMaxMsgCacheEntrySize ) {
# Check for size
$this->mCache[$code][$title] = '!TOO BIG';
$this->mMemc->set( $titleKey, ' ' . $text, $this->mExpiry );
} else {
$this->mCache[$code][$title] = ' ' . $text;
$this->mMemc->delete( $titleKey );
}
# Update caches
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$this->saveToCaches( $this->mCache[$code], 'all', $code );
$this->unlock( $cacheKey );
// Also delete cached sidebar... just in case it is affected
2010-02-01 04:57:42 +00:00
$codes = array( $code );
if ( $code === 'en' ) {
2010-02-01 04:57:42 +00:00
// Delete all sidebars, like for example on action=purge on the
// sidebar messages
$codes = array_keys( Language::fetchLanguageNames() );
2010-02-01 04:57:42 +00:00
}
global $wgMemc;
2010-02-01 04:57:42 +00:00
foreach ( $codes as $code ) {
$sidebarKey = wfMemcKey( 'sidebar', $code );
$wgMemc->delete( $sidebarKey );
}
// Update the message in the message blob store
global $wgContLang;
MessageBlobStore::updateMessage( $wgContLang->lcfirst( $msg ) );
wfRunHooks( 'MessageCacheReplace', array( $title, $text ) );
wfProfileOut( __METHOD__ );
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
/**
* Is the given cache array expired due to time passing or a version change?
*/
protected function isCacheExpired( $cache ) {
if ( !isset( $cache['VERSION'] ) || !isset( $cache['EXPIRY'] ) ) {
return true;
}
if ( $cache['VERSION'] != MSG_CACHE_VERSION ) {
return true;
}
if ( wfTimestampNow() >= $cache['EXPIRY'] ) {
return true;
}
return false;
}
/**
* Shortcut to update caches.
*
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
* @param $cache array: cached messages with a version.
* @param $dest string: Either "local-only" to save to local caches only
* or "all" to save to all caches.
* @param $code string: Language code.
* @return bool on somekind of error.
*/
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
protected function saveToCaches( $cache, $dest, $code = false ) {
wfProfileIn( __METHOD__ );
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
global $wgUseLocalMessageCache;
$cacheKey = wfMemcKey( 'messages', $code );
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
if ( $dest === 'all' ) {
$success = $this->mMemc->set( $cacheKey, $cache );
} else {
$success = true;
}
# Save to local cache
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
if ( $wgUseLocalMessageCache ) {
$serialized = serialize( $cache );
$hash = md5( $serialized );
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$this->mMemc->set( wfMemcKey( 'messages', $code, 'hash' ), $hash );
$this->saveToLocal( $serialized, $hash, $code );
}
wfProfileOut( __METHOD__ );
return $success;
}
/**
* Represents a write lock on the messages key
2010-04-09 21:27:23 +00:00
*
2011-09-14 15:07:20 +00:00
* @param $key string
*
2010-04-09 21:27:23 +00:00
* @return Boolean: success
*/
function lock( $key ) {
$lockKey = $key . ':lock';
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
$acquired = false;
$testDone = false;
for ( $i = 0; $i < MSG_WAIT_TIMEOUT && !$acquired; $i++ ) {
$acquired = $this->mMemc->add( $lockKey, 1, MSG_LOCK_TIMEOUT );
if ( $acquired ) {
break;
}
# Fail fast if memcached is totally down
if ( !$testDone ) {
$testDone = true;
if ( !$this->mMemc->set( wfMemcKey( 'test' ), 'test', 1 ) ) {
break;
}
}
sleep( 1 );
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
return $acquired;
}
function unlock( $key ) {
$lockKey = $key . ':lock';
$this->mMemc->delete( $lockKey );
}
/**
* Get a message from either the content language or the user language.
*
* @param $key String: the message cache key
* @param $useDB Boolean: get the message from the DB, false to use only
* the localisation
* @param bool|string $langcode Code of the language to get the message for, if
* it is a valid code create a language for that language,
* if it is a string but not a valid code then make a basic
* language object, if it is a false boolean then use the
* current users language (as a fallback for the old
* parameter functionality), or if it is a true boolean
* then use the wikis content language (also as a
* fallback).
2010-04-09 21:27:23 +00:00
* @param $isFullKey Boolean: specifies whether $key is a two part key
* "msg/lang".
2011-09-14 15:07:20 +00:00
*
* @throws MWException
* @return string|bool
*/
function get( $key, $useDB = true, $langcode = true, $isFullKey = false ) {
global $wgLanguageCode, $wgContLang;
if ( is_int( $key ) ) {
// "Non-string key given" exception sometimes happens for numerical strings that become ints somewhere on their way here
$key = strval( $key );
}
if ( !is_string( $key ) ) {
throw new MWException( 'Non-string key given' );
}
if ( strval( $key ) === '' ) {
# Shortcut: the empty key is always missing
return false;
}
$lang = wfGetLangObj( $langcode );
if ( !$lang ) {
throw new MWException( "Bad lang code $langcode given" );
}
$langcode = $lang->getCode();
$message = false;
2011-02-16 16:07:58 +00:00
# Normalise title-case input (with some inlining)
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
$lckey = str_replace( ' ', '_', $key );
if ( ord( $key ) < 128 ) {
$lckey[0] = strtolower( $lckey[0] );
$uckey = ucfirst( $lckey );
} else {
$lckey = $wgContLang->lcfirst( $lckey );
$uckey = $wgContLang->ucfirst( $lckey );
}
# Try the MediaWiki namespace
if( !$this->mDisable && $useDB ) {
$title = $uckey;
if( !$isFullKey && ( $langcode != $wgLanguageCode ) ) {
$title .= '/' . $langcode;
2004-11-21 13:56:04 +00:00
}
$message = $this->getMsgFromNamespace( $title, $langcode );
}
# Try the array in the language object
if ( $message === false ) {
$message = $lang->getMessage( $lckey );
if ( is_null( $message ) ) {
$message = false;
}
}
# Try the array of another language
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
if( $message === false ) {
$parts = explode( '/', $lckey );
# We may get calls for things that are http-urls from sidebar
# Let's not load nonexistent languages for those
# They usually have more than one slash.
if ( count( $parts ) == 2 && $parts[1] !== '' ) {
$message = Language::getMessageFor( $parts[0], $parts[1] );
if ( is_null( $message ) ) {
$message = false;
2006-08-10 09:10:06 +00:00
}
}
}
# Is this a custom message? Try the default language in the db...
if( ( $message === false || $message === '-' ) &&
!$this->mDisable && $useDB &&
!$isFullKey && ( $langcode != $wgLanguageCode ) ) {
$message = $this->getMsgFromNamespace( $uckey, $wgLanguageCode );
}
# Final fallback
if( $message === false ) {
return false;
}
# Fix whitespace
2010-02-05 04:25:30 +00:00
$message = strtr( $message,
array(
# Fix for trailing whitespace, removed by textarea
'&#32;' => ' ',
# Fix for NBSP, converted to space by firefox
'&nbsp;' => "\xc2\xa0",
Remove most named character references from output Recommit of r66254 to trunk. This was just find extensions phase3 -iname '*.php' \! -iname '*.i18n.php' \! -iname 'Messages*.php' \! -iname '*_Messages.php' -exec sed -i 's/&nbsp;/\&#160;/g;s/&mdash;/―/g;s/&bull;/•/g;s/&aacute;/á/g;s/&acute;/´/g;s/&agrave;/à/g;s/&alpha;/α/g;s/&auml;/ä/g;s/&ccedil;/ç/g;s/&copy;/©/g;s/&darr;/↓/g;s/&deg;/°/g;s/&eacute;/é/g;s/&ecirc;/ê/g;s/&euml;/ë/g;s/&egrave;/è/g;s/&euro;/€/g;s/&harr;//g;s/&hellip;/…/g;s/&iacute;/í/g;s/&igrave;/ì/g;s/&larr;/←/g;s/&ldquo;/“/g;s/&middot;/·/g;s/&minus;/−/g;s/&ndash;/–/g;s/&oacute;/ó/g;s/&ocirc;/ô/g;s/&oelig;/œ/g;s/&ograve;/ò/g;s/&otilde;/õ/g;s/&ouml;/ö/g;s/&pound;/£/g;s/&prime;/′/g;s/&Prime;/″/g;s/&raquo;/»/g;s/&rarr;/→/g;s/&rdquo;/”/g;s/&Sigma;/Σ/g;s/&times;/×/g;s/&uacute;/ú/g;s/&uarr;/↑/g;s/&uuml;/ü/g;s/&yen;/¥/g' {} + followed by reading over every single line of the resulting diff and fixing a whole bunch of false positives. The reason for this change is given in <http://lists.wikimedia.org/pipermail/wikitech-l/2010-April/047617.html>. I cleared it with Tim and Brion on IRC before committing. It might cause a few problems, but I tried to be careful; please report any issues. I skipped all messages files. I plan to make a follow-up commit that alters wfMsgExt() with 'escapenoentities' to sanitize all the entities. That way, the only messages that will be problems will be ones that output raw HTML, and we want to get rid of those anyway. This should get rid of all named entities everywhere except messages. I skipped a few things like &nbsp that I noticed in manual inspection, because they weren't well-formed XML anyway. Also, to everyone who uses non-breaking spaces when they could use a normal space, or nothing at all, or CSS padding: I still hate you. Die.
2010-05-30 17:33:59 +00:00
'&#160;' => "\xc2\xa0",
) );
return $message;
}
/**
* Get a message from the MediaWiki namespace, with caching. The key must
* first be converted to two-part lang/msg form if necessary.
*
* Unlike self::get(), this function doesn't resolve fallback chains, and
* some callers require this behavior. LanguageConverter::parseCachedTable()
* and self::get() are some examples in core.
*
* @param string $title Message cache key with initial uppercase letter.
* @param string $code code denoting the language to try.
2011-09-14 15:07:20 +00:00
*
2012-02-10 15:37:33 +00:00
* @return string|bool False on failure
*/
function getMsgFromNamespace( $title, $code ) {
$this->load( $code );
if ( isset( $this->mCache[$code][$title] ) ) {
$entry = $this->mCache[$code][$title];
if ( substr( $entry, 0, 1 ) === ' ' ) {
return substr( $entry, 1 );
} elseif ( $entry === '!NONEXISTENT' ) {
return false;
} elseif( $entry === '!TOO BIG' ) {
// Fall through and try invididual message cache below
}
} else {
// XXX: This is not cached in process cache, should it?
$message = false;
wfRunHooks( 'MessagesPreLoad', array( $title, &$message ) );
if ( $message !== false ) {
return $message;
}
return false;
}
# Try the individual message cache
$titleKey = wfMemcKey( 'messages', 'individual', $title );
$entry = $this->mMemc->get( $titleKey );
if ( $entry ) {
if ( substr( $entry, 0, 1 ) === ' ' ) {
$this->mCache[$code][$title] = $entry;
return substr( $entry, 1 );
} elseif ( $entry === '!NONEXISTENT' ) {
$this->mCache[$code][$title] = '!NONEXISTENT';
return false;
} else {
# Corrupt/obsolete entry, delete it
$this->mMemc->delete( $titleKey );
}
}
# Try loading it from the database
$revision = Revision::newFromTitle(
Title::makeTitle( NS_MEDIAWIKI, $title ), false, Revision::READ_LATEST
);
if ( $revision ) {
$content = $revision->getContent();
if ( !$content ) {
// A possibly temporary loading failure.
2012-02-16 05:04:17 +00:00
wfDebugLog( 'MessageCache', __METHOD__ . ": failed to load message page text for {$title} ($code)" );
$message = null; // no negative caching
} else {
// XXX: Is this the right way to turn a Content object into a message?
// NOTE: $content is typically either WikitextContent, JavaScriptContent or CssContent.
// MessageContent is *not* used for storing messages, it's only used for wrapping them when needed.
$message = $content->getWikitextForTransclusion();
if ( $message === false || $message === null ) {
wfDebugLog( 'MessageCache', __METHOD__ . ": message content doesn't provide wikitext "
. "(content model: " . $content->getContentHandler() . ")" );
$message = false; // negative caching
} else {
$this->mCache[$code][$title] = ' ' . $message;
$this->mMemc->set( $titleKey, ' ' . $message, $this->mExpiry );
}
}
} else {
$message = false; // negative caching
}
if ( $message === false ) { // negative caching
$this->mCache[$code][$title] = '!NONEXISTENT';
$this->mMemc->set( $titleKey, '!NONEXISTENT', $this->mExpiry );
}
return $message;
}
2004-03-01 05:51:55 +00:00
/**
* @param $message string
* @param $interface bool
* @param $language
* @param $title Title
* @return string
*/
function transform( $message, $interface = false, $language = null, $title = null ) {
* Introduced a new system for localisation caching. The system is based around fast fetches of individual messages, minimising memory overhead and startup time in the typical case. It handles both core messages (formerly in Language.php) and extension messages (formerly in MessageCache.php). Profiling indicates a significant win for average throughput. * The serialized message cache, which would have been redundant, has been removed. Similar performance characteristics can be achieved with $wgLocalisationCacheConf['manualRecache'] = true; * Added a maintenance script rebuildLocalisationCache.php for offline rebuilding of the localisation cache. * Extension i18n files can now contain any of the variables which can be set in Messages*.php. It is possible, and recommended, to use this feature instead of the hooks for special page aliases and magic words. * $wgExtensionAliasesFiles, LanguageGetMagic and LanguageGetSpecialPageAliases are retained for backwards compatibility. $wgMessageCache->addMessages() and related functions have been removed. wfLoadExtensionMessages() is a no-op and can continue to be called for b/c. * Introduced $wgCacheDirectory as a default location for the various local caches that have accumulated. Suggested $IP/cache as a good place for it in the default LocalSettings.php and created this directory with a deny-all .htaccess. * Patched Exception.php to avoid using the message cache when an exception is thrown from within LocalisationCache, since this tends to fail horribly. * Removed Language::getLocalisationArray(), Language::loadLocalisation(), Language::load() * Fixed FileDependency::__sleep() * In Cdb.php, fixed newlines in debug messages In MessageCache::get(): * Replaced calls to $wgContLang capitalisation functions with plain PHP functions, reducing the typical case from 99us to 93us. Message cache keys are already documented as being restricted to ASCII. * Implemented a more efficient way to filter out bogus language codes, reducing the "foo/en" case from 430us to 101us * Optimised wfRunHooks() in the typical do-nothing case, from ~30us to ~3us. This reduced MessageCache::get() typical case time from 93us to 38us. * Removed hook MessageNotInMwNs to save an extra 3us per cache hit. Reimplemented the only user (LocalisationUpdate) using the new hook LocalisationCacheRecache.
2009-06-28 07:11:43 +00:00
// Avoid creating parser if nothing to transform
if( strpos( $message, '{{' ) === false ) {
return $message;
}
if ( $this->mInParser ) {
return $message;
}
$parser = $this->getParser();
if ( $parser ) {
$popts = $this->getParserOptions();
$popts->setInterfaceMessage( $interface );
$popts->setTargetLanguage( $language );
$userlang = $popts->setUserLang( $language );
$this->mInParser = true;
$message = $parser->transformMsg( $message, $popts, $title );
$this->mInParser = false;
$popts->setUserLang( $userlang );
}
return $message;
}
/**
* @return Parser
*/
function getParser() {
global $wgParser, $wgParserConf;
if ( !$this->mParser && isset( $wgParser ) ) {
# Do some initialisation so that we don't have to do it twice
$wgParser->firstCallInit();
# Clone it and store it
$class = $wgParserConf['class'];
if ( $class == 'Parser_DiffTest' ) {
# Uncloneable
$this->mParser = new $class( $wgParserConf );
} else {
$this->mParser = clone $wgParser;
}
}
return $this->mParser;
}
/**
* @param $text string
* @param $title Title
* @param $linestart bool
2011-09-14 15:07:20 +00:00
* @param $interface bool
* @param $language
* @return ParserOutput|string
*/
public function parse( $text, $title = null, $linestart = true, $interface = false, $language = null ) {
if ( $this->mInParser ) {
return htmlspecialchars( $text );
}
$parser = $this->getParser();
$popts = $this->getParserOptions();
$popts->setInterfaceMessage( $interface );
$popts->setTargetLanguage( $language );
wfProfileIn( __METHOD__ );
if ( !$title || !$title instanceof Title ) {
global $wgTitle;
$title = $wgTitle;
}
// Sometimes $wgTitle isn't set either...
if ( !$title ) {
# It's not uncommon having a null $wgTitle in scripts. See r80898
# Create a ghost title in such case
$title = Title::newFromText( 'Dwimmerlaik' );
}
$this->mInParser = true;
$res = $parser->parse( $text, $title, $popts, $linestart );
$this->mInParser = false;
wfProfileOut( __METHOD__ );
return $res;
}
function disable() {
$this->mDisable = true;
}
function enable() {
$this->mDisable = false;
}
2010-02-05 04:25:30 +00:00
/**
* Clear all stored messages. Mainly used after a mass rebuild.
*/
function clear() {
$langs = Language::fetchLanguageNames( null, 'mw' );
foreach ( array_keys( $langs ) as $code ) {
# Global cache
$this->mMemc->delete( wfMemcKey( 'messages', $code ) );
# Invalidate all local caches
$this->mMemc->delete( wfMemcKey( 'messages', $code, 'hash' ) );
}
$this->mLoadedLanguages = array();
}
2011-09-14 15:07:20 +00:00
/**
* @param $key
* @return array
*/
public function figureMessage( $key ) {
global $wgLanguageCode;
$pieces = explode( '/', $key );
if( count( $pieces ) < 2 ) {
return array( $key, $wgLanguageCode );
}
$lang = array_pop( $pieces );
if( !Language::fetchLanguageName( $lang, null, 'mw' ) ) {
return array( $key, $wgLanguageCode );
}
$message = implode( '/', $pieces );
return array( $message, $lang );
}
/**
* Get all message keys stored in the message cache for a given language.
* If $code is the content language code, this will return all message keys
* for which MediaWiki:msgkey exists. If $code is another language code, this
* will ONLY return message keys for which MediaWiki:msgkey/$code exists.
* @param $code string
* @return array of message keys (strings)
*/
public function getAllMessageKeys( $code ) {
global $wgContLang;
$this->load( $code );
if ( !isset( $this->mCache[$code] ) ) {
// Apparently load() failed
return null;
}
Fix message cache expiry semantics * Use the stale message cache while the new one is being generated * Revert I811755d4 (make message cache load failure fatal). This escalated several very plausible temporary site issues from barely noticeable to complete downtime -- for example, memcached being down on a site with only one memcached server. * Remove $wgLocalMessageCacheSerialized, it's always been pointless * Clarify a couple of comments. * Increased lock wait timeout to 30s * Make lock() fail immediately on memcached connection refused Tests done: * With local cache enabled: normal cold refill; refill local from global cache; use stale local cache during remote refill; use stale global cache during remote refill; cold cache wait for remote refill; saveToCaches() failure; memcached connection refused. * With local cache disabled: saveToCaches() failure; cache disabled due to "error" status key; memcached connection refused. Setting a 1-day expiry in memcached, with a ~10s CPU cost to replace, is not the best idea since it inevitably leads to a cache stampede. Dealing with the stampede by waiting for a lock is not ideal, even if it were implemented properly, since it's not necessary to deliver perfectly fresh message cache data to all clients. This is especially obvious when you note that barring bugs, expiry and regeneration always gives you back the exact same data, because we have incremental updates (MessageCache::replace()). Keeping all clients waiting for 10s just to give them the data they have already is pretty pointless. So, continue to serve the site from the stale message cache while the new one is being generated. One caveat: if local caching enabled, when the message cache becomes stale, a sudden spike in network bandwidth may result due to the full array (also typically stale) being fetched from the shared cache. Bug: 43516 Change-Id: Ia145fd90da33956d8aac127634606aaecfaa176b
2013-04-03 10:54:34 +00:00
// Remove administrative keys
$cache = $this->mCache[$code];
unset( $cache['VERSION'] );
unset( $cache['EXPIRY'] );
// Remove any !NONEXISTENT keys
$cache = array_diff( $cache, array( '!NONEXISTENT' ) );
// Keys may appear with a capital first letter. lcfirst them.
return array_map( array( $wgContLang, 'lcfirst' ), array_keys( $cache ) );
}
}