This also removes all the in-core calls that had been kept for the
benefit of extensions, and causes them to not have any effect since
anything that had been calling them was already either a no-op or will
probably be broken now that nothing in core is setting or checking the
flags.
Change-Id: Id22c1a5a6d6a249debb14063ae3f8838d105b634
And deprecate passing false for ParserOptions::setWrapOutputClass().
There are three cases for the Parser wrapper: the default
mw-parser-output, a custom wrapper, or no wrapper. As things currently
stand, we have to fragment the parser cache on each of these options,
which uses a nontrival amount of storage space (T167784).
Ideally we'd do all the wrapping as a post-cache transform, but
TemplateStyles needs to know the wrapper in use in order to properly
prefix its CSS rules (that's why we added the wrapper in the first
place). So, second best option is to make *un*wrapping be a post-cache
transform and make "custom wrapper" be uncacheable.
This patch does the first bit (unwrapping as a post-cache transform),
and a followup will do the second part once the deprecation process is
satisfied.
Bug: T181846
Change-Id: Iba16e78c41be992467101e7d83e9c3134765b101
These comments do not add anything. I argue they are worse than having
no comments, because I have to read them first to understand they
actually don't explain anything. Removing them makes room for actual
improvements in the future (if needed).
Change-Id: Iee70aad681b3385e9af282d5581c10addbb91ac4
This is a re-submission of I4f24e7fbb68.
As a first major step towards Multi-Content-Revisions (MCR),
this patch turns the Revision class into a legacy proxy for
the new RevisionRecord and RevisionStore classes.
Backwards compatibility is maintained for all but some
rare edge cases, like constructing a completely empty
Revision object.
For more information on MCR, see
<https://www.mediawiki.org/wiki/Requests_for_comment/Multi-Content_Revisions>.
NOTE: once this is merged, verify create/delete/restore cycle on beta,
ideally with emulated replication lag.
Bug: T174025
Change-Id: Ia4c20a91e98df0b9b14b138eb4825c55e5200384
This reverts commit 9dcc56b3c9.
With this patch applied, newly created revisions are sometimes not found
just after submitting an edit, until replicas have caught up.
Our best theory is that it somehow interfere with ChronologyProtector,
but we don't have a good idea how.
Also, as legoktm mentioned, the commit message is terrible and needs fixing.
Change-Id: Idf3404f3fa8f8d08a7fb2ab8268726e2c1edecfe
And auto-fix all errors.
The `<exclude-pattern>` stanzas are now included in the default ruleset
and don't need to be repeated.
Change-Id: I928af549dc88ac2c6cb82058f64c7c7f3111598a
On WMF wikis, the partitioning of the revision table on the
'contributions' replicas makes the query here perform really poorly.
Specify 'api' as a hack for now to avoid those replicas.
This query happens once per day per wiki, plus when someone edits the
MediaWiki namespace, so it shouldn't be much additional load.
Bug: T164666
Change-Id: I5ae74d1702144f6475e9cfb13effc43389d66233
This will allow CSS to target just the parser output, without also
accidentally targeting the edit form, diff tables, and so on.
Bug: T37247
Change-Id: If4eb5bf71f94fa366ec4eddb6964e8f4df6b824a
Depends-On: I330c6aa4aaee045614b1801ed34bc9e03be69650
Depends-On: I52a518fa44e017841fe78474012cd69823e0a41d
System messages may take parameters from untrusted sources. This
may include taking parameters from urls given by unauthenticated
users even if the wiki is a read-only wiki. Allowing <html> tags
in such a context seems like an accident waiting to happen.
Bug: T156184
Change-Id: I661f482986d319cf41da1d3e7b20a0f028a42e90
Only 1 message typically changed per run, so for wikis using
external storage and with many customized messages, this can
make rebuilds considerably faster.
Bug: T158084
Change-Id: Ib668e69a207e3fbeb7871f2f6a102ff1af567368
The member variable is needed in the next lines, which previously
just used the array with "LATEST" set and would be seen as invalid
and discarded next time.
Bug: T157033
Change-Id: I5b84b1ae4a9c7b710ee452c61d7d9d6076ec9e6a
Do the process cache update immediately (as before) but push
the shared cache updates to a deferred update. This update
will thus start with a clear transaction snapshot, so it can
acquire the lock before the first SELECT as is proper.
Also added some missing method visibilities.
Bug: T144952
Change-Id: I462554b300d4688b09ab80cd1bb8a4340ffaa786
So that extensions like MessageCommons can try to use this (and
$wgLanguageCode) instead of $wgLang->getCode()/$wgContLang->getCode(), as
the latter ones cause fatals and recursion, at least with the Gadgets
extension also enabled at the same time.
Change-Id: If71fe1ded26c7a1c771128397783783ad5715b00
Re-submission of 9339a08b7.
* Increase time range for getValidationHash() using "latest" values.
The lower value ran the risk of regenerating from slaves and ending
up with *older* data than what was there.
* Avoid cache set() calls in replace() unless the lock was acquired.
Use delete() instead in that case, which invalidates the cache.
* Remember if the cache is volatile in process memory instead of doing
check key lookups for each "big" message to determine this. Use the
message hash in the big message keys so purges to the former chain
down to the latter. An "EXCESSIVE" key/revision map is now used in
the main cache for big messages. This means that editing an existing
big message will result in a different hash value. This is needed so
purges propage correctly.
* Add logging when replace() fails to acquire the lock.
* Factored message cache update code duplication into a new method.
* Use makeKey() in more places, replacing deprecated wfMemcKey().
Change-Id: I82c5baa8137d1ffaaec6adace82ccb0181441342
This reverts commit 9339a08b72
(Change-Id: Idc337a787171949c4f70186b13d7b65304c9b57f).
This is a temporary revert to prevent the change's inclusion in
wmf/1.29.0-wmf.4. That branch is scheduled to be deployed to WMF wikis
during the first week of the WMF's 2016 year-end fundraiser. Since
CentralNotice relies on MessageCache to fetch fundraising banners,
it is preferable not to deploy changes of any significant complexity in
that system at this time.
The original change should be re-applied at a later date. Sincere
apologies to the change's authors! :)
Change-Id: I8330838bbe03ce6ed38fa2e755b44519211d9d43
* Increase time range for getValidationHash() using "latest" values.
The lower value ran the risk of regenerating from slaves and ending
up with *older* data than what was there.
* Avoid cache set() calls in replace() unless the lock was acquired.
Use delete() instead in that case, which invalidates the cache.
* Remember if the cache is volatile in process memory instead of doing
check key lookups for each "big" message to determine this. Use the
message hash in the big message keys so purges to the former chain
down to the latter. An "EXCESSIVE" key/revision map is now used in
the main cache for big messages. This means that editing an existing
big message will result in a different hash value. This is needed so
purges propage correctly.
* Add logging when replace() fails to acquire the lock.
* Factored message cache update code duplication into a new method.
* Use makeKey() in more places, replacing deprecated wfMemcKey().
Change-Id: Idc337a787171949c4f70186b13d7b65304c9b57f
Also make use of the cache set options and use
Revision::newKnownCurrent() to avoid excessive
revision table queries during miss periods.
Bug: T144952
Change-Id: Ic1c649478b0f87420052d8c99b2962920f8b5c96
* Use services container in more places.
* Undeprecated getLocalServerInstance() since $fallback is not
handled elsewhere.
Change-Id: Id1fcd1c465d2d92653357523f4225f1c4d1ace2f
htmlspecialchars/Message->__toString/Message->toString/Message->parseText/MessageCache->parse
is useless without the caller of htmlspecialchars().
Change-Id: I2dad5a487225a11b98149610f1c775057f8baad2
This is more consistent with LoadBalancer, modern, and inclusive
of master/master mysql, NDB cluster, and MariaDB galera cluster.
The old constant is an alias now.
Change-Id: I0b37299ecb439cc446ffbe8c341365d1eef45849
* Timeouts happen when memcached is cleared or on DC-switchover likewise
* Disable the sidebar cache when the MessageCache is disabled
* Also lower any output expiry for CDN and set a custom header
when the MessageCache is disabled
* Log when this happens to a dedicated "MessageCacheError" channel
Bug: T133069
Bug: T7092
Change-Id: I1b80e250532033c52680246279e1e8f56350ae83
This also corrects a bug in MessageCache::parse() where it takes a
language code string but the Parser expects a Language object in
ParserOptions.
Bug: T128809
Change-Id: I3508c9e515e505890b3c62bc4a0982aa2a313782
The plan here is to take it out of 1.27.0-wmf.12 and put it back in
1.27.0-wmf.13.
Since BotPasswords depends on SessionManager, that's getting temporarily
removed too.
This reverts the following commits:
* 6acd424e0d SessionManager: Notify AuthPlugin before calling hooks
* 4d1ad32d8a Close a loophole in CookieSessionProvider
* fcdd643a46 SessionManager: Don't save non-persisted sessions to backend storage
* 058aec4c76 MessageCache: Don't get a ParserOptions for $wgUser before the end of Setup.php
* b5c0c03bb7 SessionManager: Save user name to metadata even if the user doesn't exist locally
* 13f2f09a19 SECURITY: Fix User::setToken() call on User::newSystemUser
* 305bc75b27 SessionManager: Don't generate user tokens when checking the tokens
* 7c4bd85d21 RequestContext::exportSession() should only export persisted session IDs
* 296ccfd4a9 SessionManager: Save 'persisted' flag in session metadata
* 94ba53f677 Move CSRF token handling into MediaWiki\Session\Session
* 46a565d6b0 Avoid false "added in both Session and $_SESSION" when value is null
* c00d0b5d94 Log backtrace for "User::loadFromSession called before the end of Setup.php"
* 4eeff5b559 Use $wgSecureCookie to decide whether to actually mark secure cookies as 'secure'
* 7491b52f70 Call session_cache_limiter() before starting a session
* 2c34aeea72 SessionManager: Abstract forceHTTPS cookie setting
* 9aa53627a5 Ignore auth cookies with value 'deleted'
* 43f904b51a SessionManager: Kill getPersistedSessionId()
* 50c5256352 SessionManager: Add SessionBackend::setProviderMetadata()
* f640d40315 SessionManager: Notify AuthPlugin when auto-creating accounts
* 70b05d1ac1 Add checks of $wgEnableBotPasswords in more places
* bfed32eb78 Do not raise a PHP warning when session write fails
* 722a7331ad Only check LoggedOut timestamp on the user loaded from session
* 4f5057b84b SessionManager: Change behavior of getSessionById()
* 66e82e614e Fix typo in [[MediaWiki:Botpasswords-editexisting/en]]
* f9fd9516d9 Add "bot passwords"
* d7716f1df0 Add missing argument for wfDebugLog
* a73c5b7395 Add SessionManager
Change-Id: I2389a8133e25ab929e9f27f41fa9a05df8147a50
Until Setup.php completes, we can't access the "current" user and trying
to do so logs a warning. So don't try.
Bug: T124367
Change-Id: If31230407829c04f34d9cfefcbb97edacb949b6d
MessageBlobStore class:
* Make logger aware.
* Log an error if json encoding fails.
* Stop using the DB table. WANObjectCache supports everything we need:
- Batch retrieval.
- Invalidate keys with wildcard selects or cascading check keys.
* Update tests slightly since the actual update now happens on-demand as
part of get() instead of within updateMessage().
ResourceLoader class:
* Remove all interaction with the msg_resource table. Remove db table later.
* Refactor code to use a hash of the blob instead of a timestamp.
Timestamps are unreliable and roll over too frequently for message blob store
because there is no authoritative source. The timestamps were inferred based on
when a change is observed. Message overrides from the local wiki have an
explicit update event when the page is edited. All other messages, such as
from MediaWiki core and extensions using LocalisationCache, have a single
timestamp for all messages which rolls over every time the cache is rebuilt.
A hash is deterministic, and won't cause needless invalidation (T102578).
* Remove redundant pre-fetching in makeModuleResponse().
This is already done by preloadModuleInfo() in respond().
* Don't bother storing and retreiving empty "{}" objects.
Instead, detect whether a module's message list is empty at runtime.
ResourceLoaderModule class:
* Make logger aware.
* Log if a module's message blob was not preloaded.
cleanupRemovedModules:
* Now that blobs have a TTL, there's no need to prune old entries.
Bug: T113092
Bug: T92357
Change-Id: Id8c26f41a82597e34013f95294cdc3971a4f52ae
The $ucfirst and $wgContentLang variables are not used by get().
Keep these inside getMessageFromFallbackChain() instead.
Also add mark getMsgFromNamespace() as public since it is used
in other places in core and by extensions.
Change-Id: I83bf77c17aa46f3929f89c4af97693611761629b
Language::ucfirst() already has this optimisation. The expensive
method is Language::uc(), which is only called when needed.
The ord() < 128 is one of the conditionals in Language::ucfirst().
Change-Id: Idd648b7b84eba1f92dda904438122fe38d3cf22b
This table is not needed because module names and their messages
array are available to the runtime environment at very little cost.
The only purpose it was serving is reverse lookup from message
key to module name (e.g. when MessageCache receives update that
need to propagate to MessageBlobStore). However that is better
achieved by simply looping through modules in PHP. The overhead
of a database is not worth this minor convenience.
MessageBlobStore
* insertMessageBlob: Doesn't need to update msg_resource_links.
* updateModule: Doesn't need to update msg_resource_links.
* getUpdatesForMessage: Reimplement with list from memory
instead of msg_resource_links.
The database table will be removed in If009e2620e59002e1.
Bug: T113092
Change-Id: Ia9131f570001f00c9800b260ac4b3469d54d2784
* If replace() was called recently, then we know that this is the
master data center and that the messages are up-to-date. With this
change, replace() calls have nothing to contend with aside from other
replace() calls. Even if there were timeouts due to such contention,
caused by high MediaWiki: page edit rates, the replace() updates would
pick up the prior changes in passing since they do load().
* This also avoids the following scenario:
a) Someone edits a message page, triggering replace()
b) Some page view causes load() to trigger loadFromDB()
due to the hash being seen as volatile due to replace()
c) The loadFromDB() loads stale slave data and undoes
the message key update the replace() did
Change-Id: I9cdf7ad3d67f168fcba7f633af9e32a8d1fa928e
* Made the status key only act as a backoff key inside
loadFromDBWithLock(), instead of also being a lock key. The
getReentrantScopedLock() call is now non-blocking, which has
a similar affect to the add() call on the status key but much
better prioritizes replace(); that method already has a blocking
getReentrantScopedLock() call, so once it gets that lock, there
is no worry of load() failing due to contention.
* Bail out in loadFromDBWithLock() if the scope lock was not acquired.
Normally the status lock often assured the lock could be obtained,
unless a competing replace() came in. The old status lock would
cause a bail if not acquired, and this carries over that behavior
but also avoids clobbering updates when replace() contention happens.
* Avoid calling saveToCaches() in replace() if the lock was not
acquired to avoid clobbering conflicting writes.
* Made the loadFromDB() call in replace() use DB_MASTER to avoid
seeing stale data if the cache is volatile.
* Lowered WAIT_SEC, as the default PHP timeout is 30.
We want this to be able to at least finish the other calls
in replace() even if the lock times out.
* Avoid checking the status key an extra time in load().
* Made a few small code and doc cleanups.
Change-Id: Ibaf1f67618ec374c83c3135a71e90223dd2b1856
* The key salting of the invidial keys via the WAN cache makes
it OK to use slaves here. The salting happens via the delete()
in the MessageCache::replace() call, which is hit when the
message page is edited.
Bug: T92357
Change-Id: Ic191f6c3fc49c4c58461d26468dd0fa94a52051e
* These keys are used for larger messages not in the main blob,
and they still use explicit purges as the pages change
Change-Id: I254ddcc9ffe7c49654788d6aabcca2ab7ed4f03f
* The localCache member will use CACHE_NONE if this is off,
so the set()/get() calls are already no-oped
* Also allow using the local cache if $hash is false
in "stale mode", which handles the case where the hash
key fell out of memory for some reason more gracefully
Change-Id: Ie12efcd4088a6dc4a4cdd2fd06646f2881df53d7
* This avoids constant churn when $wgUseLocalMessageCache is set to false
* Follow up to db464b8a84
Bug: T109183
Change-Id: I8da324c53527da32d09964be6c3a92176af4ee7b
Follow-up for I020617d, where I got this wrong. It is probable that multiple
application servers will try to build a local cache at the same time, in which
case they will cause the cache key to thrash. It is necessary that two
application servers building a local cache from the database generate the same
hash for a given set of messages.
Change-Id: Ieeefc2094a83be9401c466bec859c1588ddfbcdf
* Use hash key volatility to propagate invalidations over DCs
in addition to memcached->APC instances.
* Make use of stale APC cache due to hash mismatch instead
of waiting around in some cases. This is similar to the
TTL expiry and volatility cases.
* Renamed $hashExpired and added some comments for clarity.
Change-Id: I8890fb174cae72c4ce8872df64f52f5cbd55183b
In addition to eliminating disk IO in a hot path, using APC spares us from
having to serialize and unserialize cache arrays. Since we're not serializing,
though, we don't have a string representation to hash, so use a random string
instead. (The code already treats the association of hash string to cache as
purely symbolic, so this is not problematic.)
Whereas the hash was previously stored as the first 32 bytes of each cache
file, we now store it as an array key instead (like VERSION and EXPIRY were
already). Because this changes the structure of cached data, we have to bump
MSG_CACHE_VERSION.
While we're here, make MessageCache::getLocalCache() and
MessageCache::saveToLocal() protected, make their signatures more
consistent with other methods in this class. While they were (implicitly)
public before, there are absolutely no external callers in Core or
extensions[0][1], so we can skip the standard deprecation process.
[0]: https://github.com/search?q=%40wikimedia+getlocalcache&type=Code&utf8=%E2%9C%93
[1]: https://github.com/search?utf8=%E2%9C%93&q=%40wikimedia+savetolocal&type=Code
Change-Id: I020617d2df2a8f0f243b85f3383dc7b16f15aaad
Missing global declaration when using $wgContLang in
MessageCache::normalizeKey
Change-Id: Ia1c1f41244fd5629527b99a5f2038f607cff42c4
Follow-up: 47e0f0c3
* Normalize the message name returned by allmessages
* Separate message key normalization into
MessageCache::normalizeKey()
Bug: T63894
Change-Id: I1d89fc73fea705243d390bab91255a635d8f9eee
wfSuppressWarnings() and wfRestoreWarnings() were split out into a
separate library. All usages in core were replaced with the new
functions, and the wf* global functions are marked as deprecated.
Additionally, some uses of @ were replaced due to composer's autoloader
being loaded even earlier.
Ie1234f8c12693408de9b94bf6f84480a90bd4f8e adds the library to
mediawiki/vendor.
Bug: T100923
Change-Id: I5c35079a0a656180852be0ae6b1262d40f6534c4
* This should not be used, and load() does not see them.
If the content language is 'de', then message overrides
will not include MediaWiki:<key>/de pages.
Change-Id: Ie4b6b356bd309814dd4a88040a29a7ebd509712a
* The cache has to reload and *after* locking to avoid
losing any concurrent changes.
* Also fixed incorrect assumption in MessageCacheTest.
Message overrides for the content language do not use
the language suffix.
Change-Id: I98ff158a1575330bc59efe6badb27f8de8717951
Seriously, the ops team spent some time trying to find that page during an outage,
while in fact it's an obscure Tolkien reference - better be clear.
Also, set the other dummy titles to something very clearly explaining what's
going on and where.
Change-Id: I6f33a2ea5030f22a258830a33f7bcefa7f0acd85
The test for OutputPage::makeResourceLoaderLink was triggering database
queries through MessageBlobStore even though it doesn't use any
messages.
In bb03d1a8e0 I had made MessageBlobStore a singleton instead of static
functions, however there's no need for it to be one since the class is
stateless. Callers can just create a new MessageBlobStore instance and
call functions upon it. Using getInstance() is now deprecated.
ResourceLoader now has a setMessageBlobStore setter to allow overriding
which MessageBlobStore instance will be used. OutputPageTest uses this
to set a NullMessageBlobStore, which makes no database queries.
Change-Id: Ica7436fb6f1ea59bd445b02527829ab0742c0842
Xhprof generates this data now. Custom profiling of various
sub-function units are kept.
Calls to profiler represented about 3% of page execution
time on Special:BlankPage (1.5% in/out); after this change
it's down to about 0.98% of page execution time.
Change-Id: Id9a1dc9d8f80bbd52e42226b724a1e1213d07af7
It's a very hot code path, so it makes sense to be able to dive into it in
detail, and having a discrete log bucket makes that easier to do.
Change-Id: I4b547e97711b9e45a99f7b747d785690c70883ec
For easier testability and other things. There are no uses
of this class in any extensions in gerrit.
Change-Id: I606de4259239e128ed7e0477fc98b84c647430c4
- Swap "$variable type" to "type $variable"
- Added missing types
- Fixed spacing inside docs
- Makes beginning of @param/@return/@var/@throws in capital
- Changed some types to match the more common spelling
Change-Id: I7b65fe04db431342cc58b469dc48f41a50c4e891
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Also added some missing @param.
Change-Id: I86fd10e3f2d4bb80e7432533038d124693acfb3c
Example usage:
$wgHooks['MessageCache::get'][] = function( &$key ) {
static $keys = null;
if ( $keys === null ) {
global $wgExtensionMessagesFiles;
require( $wgExtensionMessagesFiles['OverrideMessages'] );
$keys = array_flip( array_keys( $messages['en'] ) );
}
if ( isset( $keys["myprefix-$key"] ) ) {
$key = "myprefix-$key";
}
return true;
}
Pros:
* Easy way to override standard core and extension messages without
any changes to them
* Messages can be stored in a standard i18n file
* Messages can be translated easily with Translate
* Messages can be shared accross multiple wikis easily
* Takes advantage of the normal message cache behavior unlike the
MessagePreLoad hook
* Missing translations fallback to the override, not to the
uncustomized standard translation
* Do not need to handle conflicting message keys at translatewiki.net
if adopted by WMF
Cons:
* This method is called often, so there will be small performance
impact if no hooks are registered. Impact can be big if the
implementation of hook subscriber is inefficient.
This can help with bugs like 36149. It doesn't remove the manual work
needed to detect those messages and adding them to the i18n file.
I have been using this patch in a wiki farm for months.
Change-Id: Ib39937a440e71ae7292cf992ab37a569189741e4
It's elitist mathematical jargon. In all cases dealt with here, it adds
no additional meaning compared to "if", beyond what was already obvious
from context. Thus, its only purpose is to smugly demonstrate that the
author attended their second-year mathematics classes, at the expense of
causing confusion for everyone who doesn't have such a background.
If you really think you need to convey extra information beyond what
"if" gives you, the English language contains plenty of devices for doing
so, without resorting to neologisms.
Change-Id: Iae21095d02ec2935c10e94f532235c2671c115b1
