Commit graph

218 commits

Author SHA1 Message Date
Aaron Schulz
c30b049580 Add $wgMaxUserDBWriteDuration to limit user-generated transactions
This is a low level catch-all net for huge updates that still slip
through. Features that let users add/remove arbitrarily many rows
to lists of arbitrary size can easily result in high lag due to
strange usage patterns or deliberate attacks.

Also removed duplicate 'autochange-username' JSON key.

Bug: T95501
Change-Id: I58a91ca23cae528ef1954d2d78c8f0a90681983e
2015-12-04 13:08:52 -08:00
Aaron Schulz
99610822d8 Add mergeable update support DeferredUpdates
* DeferrableUpdate classes can implement MergeableUpdate.
  Duplicate updates will be merged via the merge() method.
* Make SquidUpdate support merge() so that duplicate URL
  purges are now caught accross the entire pre-send request
  execution.

Change-Id: Idffdd3e71d89e4a0f28281e65a881113caae497c
2015-12-04 19:08:51 +00:00
Aaron Schulz
34dd248e28 Add pre-send update support to DeferredUpdates
* PRESEND/POSTSEND constants can now be used in addUpdate()
  and addCallableUpdate() to control when the update runs.
  This is useful for updates that may report errors the client
  should see or to just get a head start on queued or pubsub
  based updates like CDN purges. The OutputPage::output() method
  can easily take a few 100ms.
* Removed some argument b/c code from doUpdates().
* Also moved DeferrableUpdate to a separate file.

Change-Id: I9831fe890f9f68f9ad8c4f4bba6921a8f29ba666
2015-12-04 19:08:27 +00:00
jenkins-bot
323a2909ba Merge "Extracted some code from PrefixSearch for re-usability by other search engines." 2015-12-04 16:56:39 +00:00
Aaron Schulz
6dedffc2d7 Move category membership RC updates to CategoryMembershipChangeJob
* Recursive link updates no longer mention an category changes.
  It's hard to avoid either duplicate mentioning of changes or
  confusing explicit and automatic category changes.
* LinksUpdate no longer handles this logic, but rather WikiPage
  decides to spawn this update when needed in doEditUpdates().
* Fix race conditions with calculating category deltas. Do not
  rely on the link tables for the read used to determine these
  writes, as they may be out-of-date due to slave lag. Using the
  master would still not be good enough since that would assume
  FIFO and serialized job execution, which is not garaunteed.
  Use the parser output of the relevant revisions to determine
  the RC rows. If 3 users quickly edit a page's categories, the
  old way could misattribute who actually changed what.
* Make sure RC rows are inserted in an order that matches that
  of the corresponding revisions.
* Better avoid mentioning time-based (parser functions) category
  changes so they don't get attributed to the next editor.
* Also wait for slaves between RC row insertions if there where
  many category changes (it theory it could well over 10K rows).
* Using a separate job better separates concerns as LinksUpdate
  should not have to care about recent changes updates.
* Added more docs to $wgRCWatchCategoryMembership.

Bug: T95501
Change-Id: I5863e7d7483a4fd1fa633597af66a0088ace4c68
2015-12-03 11:28:05 +00:00
Brad Jorsch
d032bb52cd Add a central ID lookup service
Anything that wants to be "central" right now has to depend on
CentralAuth, and then either can't work without CentralAuth or has to
branch all over the place based on whether CentralAuth is present. Most
of the time all it really needs is a mapping from local users to central
user IDs and back or the ability to query whether the local user is
attached on some other wiki, so let's make an interface for that in
core.

See I52aa0460 for an example implementation (CentralAuth), and Ibd192e29
for an example use (OAuth).

Bug: T111302
Change-Id: I49568358ec35fdfd0b9e53e441adabded5c7b80f
2015-12-02 14:50:36 -07:00
dcausse
353ae71963 Extracted some code from PrefixSearch for re-usability by other search engines.
(This patch is needed for Ida9b9f8)

Bug: T112028
Change-Id: I35aece88333a65f6b1f55f7a87e2d14de4f5bea7
2015-12-02 17:18:08 +00:00
Reedy
5d032c0f65 Rename wfBaseConvert.php maintenance script
Change-Id: I3174ceb040e9029ad8049a3e49c119bfac4aa386
2015-11-24 23:34:21 +00:00
jenkins-bot
8732532d47 Merge "Implement HTMLComboboxField" 2015-11-24 19:20:07 +00:00
jenkins-bot
899bd46cbb Merge "Add script to fix content model of JSON pages" 2015-11-24 18:53:43 +00:00
Kunal Mehta
2986d47c90 Add script to fix content model of JSON pages
MediaWiki:Foo.json and User:Foo/bar.json pages now have a default
content model of JSON, but existing pages using those names will be set
to defaults of wikitext.

The content models of those pages are now set to "json", unless it has
invalid syntax, in which case it will be set to "wikitext".

For convenience, the script is automatically run as part of update.php.

Bug: T108663
Change-Id: I1412937ccea8e65dba58580beec79cbf2286ae01
2015-11-24 10:08:44 -08:00
jenkins-bot
aa5e579028 Merge "Move details of import logs into log params" 2015-11-24 08:52:12 +00:00
Brad Jorsch
42e257a1a2 API: Remove dbg, txt, and yaml formats
Per RFC: https://www.mediawiki.org/wiki/Requests_for_comment/Ditch_crappy_API_formats

Bug: T95715
Change-Id: I8e4e9ad3c96504cf4587cb4632504eff21b5ff75
2015-11-18 09:19:51 -05:00
Bartosz Dziewoński
77d759aab4 Implement HTMLComboboxField
It's a dropdown select with the ability to add custom options, or a
text field with input suggestions, whichever you prefer.

This is meant to be a replacement for HTMLSelectOrOtherField and
HTMLAutoCompleteSelectField.

In regular HTML mode, it uses HTML5 `<datalist>` element with no
custom JavaScript. This is supported by a wide range of browsers
(IE 10+, modern Firefox and Chrome, Opera 12+).

In OOUI mode, it uses a ComboBoxInputWidget.

Depends on: I14b40884f185fb4e5

Bug: T118119
Change-Id: I954d3d24ed4efe90be9596a1bd9586ba3aee1e23
2015-11-12 09:48:27 +01:00
umherirrender
dc78d4d12c Move details of import logs into log params
The detail information about the count of revision and the interwiki
title get moved into log params. To show the new information an
ImportLogFormatter was written, which changes the key for new log items.

This allows to show the detail information in the user language.
It also decouple the user supplied data from detail information

Change-Id: Iaa57da0fc3e20c33c94fd850cd02db96fdb32dac
2015-11-11 18:34:18 +01:00
Timo Tijhof
304eaf2868 objectcache: Reduce MediaWiki-specificness from Memcached classes
Follows-up f36b73e96c, which moved these classes to libs/objectcache.

* Fix wrong @throws in MemcachedPeclBagOStuff.
* Fix wrong @returns in MemcachedBagOStuff::getClient().
* Rename MWMemcached to MemcachedClient.
* Remove mention of 'memcached.php', which doesn't exist anymore.

Change-Id: I34dbc859be4778cea489fd2344f233f30452605c
2015-11-10 03:32:58 +00:00
aude
b0c81a3e09 Remove deprecated SiteObject and SiteArray classes
SiteObject and SiteArray were deprecated in 1.21 and
can't find any use of them anywhere else in gerrit.

Change-Id: Iff3ba8a60ac9566998cce828c4034066fdefe804
2015-11-04 13:43:32 +01:00
Ori Livneh
8340400f27 Add Timing interface
* Add `Timing`, an interface which mimics the W3C User Timing API.
  It provides a canonical way to store and retrieve markers (timestamps)
  and measures (timestamps + duration).
* As the initial use-case, use it to record 'requestShutdown'.

Change-Id: I36b29162ffcc091406df025463b0e2797e52f19a
2015-11-03 15:05:37 -08:00
jenkins-bot
41f9c91545 Merge "Add script to find orphaned LocalRepo files" 2015-11-03 10:48:31 +00:00
Reedy
d9d22e8f0b Remove ORM code from core
Bug: T114538
Change-Id: Ic4196ca9da927fc5c85b01cfff65f0636e3202ae
2015-10-29 13:06:36 +00:00
Aaron Schulz
828b598911 Add script to find orphaned LocalRepo files
Bug: T111838
Change-Id: I60ac7f3451c8240a0716933dcfff413669458206
2015-10-29 01:59:10 -07:00
Timo Tijhof
e8275758fe objectcache: Introduce IExpiringStore for convenient TTL constants
Also consistently use self:: instead of BagOStuff:: for constants
referenced within the BagOStuff class.

Change-Id: I20fde9fa5cddcc9e92fa6a02b05dc7effa846742
2015-10-28 04:07:25 +00:00
Ori Livneh
e9d64a69bc Replace local RunningStat with wikimedia/running-stat library
* mediawiki/vendor change: I66b419c8168df.
* Add back-compat shim in includes/compat/RunningStatCompat.php.

Change-Id: Ibad4dc0ce5aa8697791865ed8ea61a25bae7b3e0
2015-10-27 19:16:58 +00:00
Perside Rosalie
6ba942258a Changed paths to package dependency file
Paths to ./composer.lock is changed to ./vendor/composer/installed.json
and dependency information now is read from the installed.json file.
A new ComposerInstalled class has been declared and uses the class now
to access the getInstalledDependencies() method to read data

Bug: T106247
Change-Id: Ic216577bb19b4fc5832ba003fcbbe9195d707b41
2015-10-26 15:58:03 +00:00
Aaron Schulz
f36b73e96c Move some memcached classes to /libs
Also removed a few MW dependencies from MemcachedPeclBagOStuff.
It still uses an IP class method, so it has to stay for now.

Change-Id: I8c5c83046c58fb58091d6ce11b2385208262460f
2015-10-24 15:27:42 -07:00
Aaron Schulz
cce813a922 Move MultiWriteBagOStuff to /libs
Also moved related tests files to /libs.

Change-Id: I806eeaa30205733d497adde933baf0c4157f7aae
2015-10-24 12:15:26 -07:00
Aaron Schulz
660a775931 Remove old ProfilerOutputUdp class
This uses a non-standard output and requires a custom
collector that wmf does not maintain nor use anymore.

Change-Id: I41a68f7061465417fbdc5ca41f8eb6e1f99f1111
2015-10-21 20:59:47 -07:00
Reedy
8c8383a53d Remove unused ApiQueryORM
Bug: T114538
Change-Id: I4673977cef336ab3b4e375a3deab4d4fc67830eb
2015-10-15 17:01:38 +00:00
jenkins-bot
5b1d551953 Merge "Improve error message for missing content model on import" 2015-10-12 13:55:12 +00:00
This, that and the other
cdc36570ea Improve error message for missing content model on import
I was initially going to refactor out the error message into Import.php,
but it quickly became apparent that WikiImporter's error handling needs a
LOT of work. In particular, to localise the error message into the user's
language is sadly non-trivial.

Although not used currently, the MWUnknownContentModelException subclass
will help with error handling improvements in the future.

Bug: T49270
Change-Id: I9f53c9d6a8a2ea842cb2ba94d4131e10a8b08f5d
2015-10-10 19:51:20 +11:00
Aaron Schulz
a8ec2f8daf Move SquidPurgeClient under /clientpool
Each class has its own file now too.

Change-Id: I11593d6efbfce8e3981895e84edb4e0dea3998e4
2015-10-08 00:37:49 -07:00
jenkins-bot
f0b51b4200 Merge "API: Add ApiQueryAllRevisions" 2015-10-07 16:38:35 +00:00
Timo Tijhof
6d412cf028 Move WebRequestUpload class to its own file
Moving out of WebRequest.php. It's not even a subclass.

Change-Id: I87e5c01afe215a084e39b442c65d22d9d0cc1015
2015-10-06 19:24:58 -07:00
jenkins-bot
12ce3c8e8d Merge "Enforce lagged-slave read-only mode on the DB layer" 2015-10-07 01:40:34 +00:00
Ori Livneh
da3443bfa2 Add MemoizedCallable for APC-backed function memoization
Add a simple class to `libs/` for memoizing functions by caching return values
in APC. I decided not to make this an external library just yet because I see
this as potentially a part of a larger functional programming library.

Doesn't use APCBagOStuff for two reasons: (1) avoid dependency on MediaWiki
code; (2) ability to pass third &$success parameter to apc_store, to
distinguish between cache misses and cached false values.

Use this in ResourceLoaderFileModule to cache CSSMin::remap.

Change-Id: I00a47983a2583655d4631ecc9c6ba17597e36b5f
2015-10-06 23:35:03 +00:00
Brad Jorsch
2516ca03ed API: Add ApiQueryAllRevisions
Bug: T113885
Change-Id: I43bdc1e33945dab27466fc047d78af5e65df1740
2015-10-06 12:32:12 -04:00
jenkins-bot
c5ac11e2eb Merge "Start rename of DatabaseBase => Database" 2015-10-05 23:45:36 +00:00
Aaron Schulz
a88df43d3f Database debug log cleanup (remove wgDebugDumpSqlLength/wgDebugDBTransactions)
* Simplify the debug log call and use queries group
* Remove $wgDebugDumpSqlLength, as profiler output
  already has shortened query strings (one can use
  profiling without DBO_DEBUG)
* Removed $wgDebugDBTransactions as BEGIN/COMMIT already show
* Removed PostgresTransactionState as it was only used for
  $wgDebugDBTransactions handling
* This cuts down on lots of global variable usage

Change-Id: I185adb1694441d074dea965960429b4910727620
2015-10-05 12:47:09 -07:00
Aaron Schulz
db170ebe78 Enforce lagged-slave read-only mode on the DB layer
* Most callers gracefully check wfReadOnly(),
  but fail in case they dont. This also catches
  foreign DBs which might slip through the cracks.
* Also remove useless wfDebug() call around
  mDoneWrites check as write queries show in
  the logs anyway.

Change-Id: I560ebd19c4eb2b3a040d4331702346440617cfaa
2015-10-05 05:41:19 +00:00
Aaron Schulz
1b67a929ab Start rename of DatabaseBase => Database
This gives static method callers the option
to use methods like Database::factory() instead
of having to use the uglier DatabaseBase::factory().

Change-Id: I61800626b71ad2803a897df060059dbaf8778679
2015-10-04 12:36:56 -07:00
Brad Jorsch
77d62edef3 Refactor hashing utility functions from MWCryptRand and make public
MWCryptRand already has some useful utility functions wrapping PHP's
hash() and hash_hmac(). Let's make them public so we can use them from
other code.

But since "MWCryptRand" isn't really a good place for hashing functions,
let's move them to "MWCryptHash" instead.

Change-Id: I7542c719ac72beba7b0f6aa170bdb4c69fa6beab
2015-10-01 01:18:23 +00:00
jenkins-bot
2cb893c085 Merge "Introduce CategoryMembershipChange" 2015-09-30 12:44:37 +00:00
Aaron Schulz
c0cb80beac Added DeleteLinksJob to support purging backlinks via job runners
* This jobs should only be constructed via relevant Content object,
  e.g. the result of enqueueUpdate() being called on a DataUpdate
  returned by Content::getSecondaryUpdates().
* Also modified LinksDeletionUpdate to support a $pageId parameter.
* LinksDeletionUpdate can now be enqueued to a DeleteLinksJob.

Change-Id: I650dcf0bd172ede0d61357ec158a4704ae1f2033
2015-09-29 17:40:19 -07:00
addshore
2fb2a3f14b Introduce CategoryMembershipChange
This is split from:
I03516bb34144d95e5f25c46ae98ab70ce699b31b

Change-Id: I27539d25ef3e81cf991657dffc0a62b9719d21d8
2015-09-29 23:59:34 +01:00
Kunal Mehta
9a3c7b43ea Move FileContentsHasher into includes/utils/
The class only contains two dependencies upon MediaWiki (ObjectCache &
wfGlobalCacheKey) which are suitable for inclusion in the utils
directory.

Change-Id: I85b4c763be2670c40f26d93e75cedcb68eaa7987
2015-09-24 18:27:28 -07:00
jenkins-bot
827e10dc40 Merge "Use wikimedia/cldr-plural-rule-parser" 2015-09-24 23:15:56 +00:00
Aaron Schulz
9f01cee8fa Moved all load balancer classes to /loadbalancer
* Also split out LBFactory classes to their own files

Change-Id: I9e649878afcffdb736c09c0574e5f99b1e72382f
2015-09-24 22:56:05 +00:00
Niklas Laxström
4a3fd2e42a Use wikimedia/cldr-plural-rule-parser
Replaces the parser included in MediaWiki with same code in
a library.

Change-Id: I1d2675466a543269e17faf213aa68d2b7afaf78e
2015-09-24 21:41:50 +02:00
jenkins-bot
173493f0ce Merge "resourceloader: Improve caching for LESS file compilation" 2015-09-24 00:44:39 +00:00
Ori Livneh
12afb3607d resourceloader: Improve caching for LESS file compilation
Caching the output of a LESS compiler is tricky, because a LESS file may
include additional LESS files via @imports, in which case the cache needs
to vary as the contents of those files vary (and not just the contents of
the primary LESS file).

To solve this, we first introduce a utility class, FileContentsHasher. This
class is essentially a smart version of md5_file() -- given one or more file
names, it computes a hash digest of their contents. It tries to avoid
re-reading files by caching the hash digest in APC and re-using it as long as
the files' mtimes have not changed. This is the same approach I used in
I5ceb8537c.

Next, we use this class in ResourceLoaderFileModule in the following way:
whenever we compile a LESS file, we cache the result as an associative array
with the following keys:

* `files` : the list of files whose contents influenced the compiled CSS.
* `hash`  : a hash digest of the combined contents of those files.
* `css`   : the CSS output of the compiler itself.

Before using a cached value, we verify that it is still current by asking
FileContentHasher for a hash of the combined contents of all referenced files,
and we compare that against the value of the `hash` key of the cached entry.

Bug: T112035
Change-Id: I1ff61153ddb95ed17e543bd4af7dd13fa3352861
2015-09-24 00:34:17 +00:00