Commit graph

298 commits

Author SHA1 Message Date
jenkins-bot
d716155c8b Merge "Reset scoped session for upload jobs after deferred updates" 2016-03-01 11:30:55 +00:00
Bartosz Dziewoński
c161c46d26 Improve code suffering from PHP 5.3's lack of support for foo()[]
I searched for /\$(\S+) = (.+?\(.*?\);)\n.*?\$\1\[/, ignored
everything involving isset(), unset() or array assigments, then
skimmed through the remaining results and changed things where they
made sense. These changes were not automated, so please review them.

Change-Id: Ib37b4c66fc57648470f151ad412210b3629c2538
2016-02-28 22:49:20 +01:00
Aaron Schulz
1ac63f1767 Reset scoped session for upload jobs after deferred updates
Bug: T126450
Change-Id: I8a6588209647252a4509078aaa7bf0cb1d9d299a
2016-02-24 10:54:31 -08:00
Niklas Laxström
daf2abc4f5 Document memory-limit as a possible exit reason
Change-Id: I96748cd8397969447bd2315f0e6b6e7989ddfc4f
2016-02-20 21:33:41 +01:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
Bartosz Dziewoński
05c2c65ab9 Allow users to tag file uploads
Using either action=upload API or Special:Upload. (No user interface
is provided for the latter, this is meant to be used by on-wiki
scripts/gadgets enhancing the upload process.)

Modelled after how ae3ab9eef0
implemented tagging of regular edits.

Bug: T121876
Change-Id: Ia3e0dbd895b2f8bc66985b24db35f112b6f9a22d
2016-02-16 03:51:29 +01:00
Ricordisamoa
7a97368c35 Stop doing $that = $this in includes/jobqueue
Closures support $this as of PHP 5.4

Change-Id: Icf4eb9ffeab410c01c68a031ec6704cd83c2681e
2016-02-10 17:51:47 +00:00
Aaron Schulz
39ef7941b0 Style and commenting tweaks to CategoryMembershipChangeJob
Also renamed safeWaitForPos() => safeWaitForMasterPos()

Change-Id: Ic89e2a0b17cd6e6ef09cf703bbbcea1988a5bde9
2016-01-30 10:27:28 -08:00
addshore
368cbdecdd Use slave for selects in CategoryMembershipJob
This patch switches to using a slave but imediatly
waits for the slave to catch up with master
(so as not to miss things).

This may result in more delay between an edit and
category changes being inserted.

It may be possible to instead wait for the timestamp
that is passed in $this->params['revTimestamp']
which could result in slightly less delay.

I can't see any uses of waitForReplication in quite
this way but see no imediate reason this would not work.

Bug: T125147
Change-Id: Ia0aa722c97f41a3959bcd3cb4210b39db0c3bc45
2016-01-30 18:22:05 +01:00
Aaron Schulz
60013aa328 Add IDatabase::getScopedLockAndFlush() method
This method is less manual and avoids the usual pitfalls of
not unlocking for a return statement or not flushing out any
prior transaction.

Change-Id: Ib1681244767de860105a68210e181e2f024ee525
2016-01-29 20:32:05 +00:00
Aaron Schulz
006d2d8976 Use InvalidArgumentException in ActivityUpdateJob
Change-Id: Ib73c02868327d5e8f286bce79cdf44ebb1f47ff3
2016-01-29 11:56:04 -08:00
Aaron Schulz
3a15ecebea Migrate callers to waitForReplication()
Change-Id: I7b2b13b9315891561d2d8cc04a12ecad2dc73d70
2016-01-28 03:00:08 +00:00
Bartosz Dziewoński
1ebc9128d0 Remove putrid remains of $wgAllowAsyncCopyUploads
None of this works and it's been long begging for a mercy kill.
All it does is waste contributor time on updating deprecations
in the dead code. I imagine we wouldn't reuse much of this
code if we're ever going to reimplement it.

Bug: T119336
Change-Id: Ibd26a4bea621857aac77823017e9be9b7dc52cca
2016-01-22 00:18:25 +00:00
Brad Jorsch
4f5057b84b SessionManager: Change behavior of getSessionById()
It's easily possible for SessionManager::getSessionById() to not be
able to load the specified session and to not be able to create an empty
one by that ID, for example if the user's token changed. So change this
from an exceptional condition to an expected one, and adjust callers to
deal with it appropriately.

Let's also make the checks for invalid data structure when loading the
session from the store delete the bogus data entirely.

At the same time, let's change the silly "$noEmpty" parameter to
"$create" and make the default behavior be not to create an empty
session.

Bug: T124126
Change-Id: I085d2026d1b366b1af9fd0e8ca3d815fd8288030
2016-01-20 12:45:26 -05:00
jenkins-bot
a1ac862ba9 Merge "More complete, more straightforward JobQueueMemoryTest" 2016-01-16 01:02:09 +00:00
Thiemo Mättig
d22a146393 Fix mistakes in JobQueue type documentations
The missing "bool" should be obvious.

I'm also changing type hints from the implementation to the interface.
All public methods from the JobSpecification class are also in the
interface, except for two: toSerializableArray and newFromArray.
These two are not used here.

Change-Id: I36867cdfdf012a4f3233ac4730ab46dac1edc0ab
2016-01-16 00:46:58 +00:00
Thiemo Mättig
793dbbb870 More complete, more straightforward JobQueueMemoryTest
I created a basic test yesterday to cover two bugs. Now the test covers
all public methods. I was also able to get rid of the test double.

Change-Id: I53110280e3ef7b7a72d175b11b7fc4ccf1d648b3
2016-01-15 10:28:30 +01:00
Thiemo Mättig
1910da184e Do not return null reference in JobQueueMemory
See Ia5b7a96 and the unrelated error raised there.
https://integration.wikimedia.org/ci/job/mwext-testextension-zend/19682/consoleFull

By writing this simple test I not only found one but two issues.

Bug: T123539
Change-Id: I17ed5b69992aa98ab2384b7a6aafc96b0fcba1ce
2016-01-14 19:19:55 +01:00
Aaron Schulz
3592f8cea3 Add simple JobQueueMemory class for testing
Change-Id: I1acce43dbab3b12393d6dc4675850695c619d88a
2016-01-14 00:23:05 +00:00
Brad Jorsch
a73c5b7395 Add SessionManager
SessionManager is a general-purpose session management framework, rather
than the cookie-based sessions that PHP wants to provide us.

While fallback is provided for using $_SESSION and other PHP session
management functions, they should be avoided in favor of using
SessionManager directly.

For proof-of-concept extensions, see OAuth change Ib40b221 and
CentralAuth change I27ccabdb.

Bug: T111296
Change-Id: Ic1ffea74f3ccc8f93c8a23b795ecab6f06abca72
2016-01-12 21:57:01 +00:00
Aaron Schulz
159e71f6d2 Track memory RSS increases in JobRunner
Bug: T123284
Change-Id: Ic6b76a722cc8d1e18a42b9779f776850ae4700f4
2016-01-12 20:41:45 +00:00
Aaron Schulz
bb6f51f79f Refactor JobRunner::run() by breaking it up a bit
Change-Id: I3abb9520b8479ae9f3c8ff502253625c93a62e92
2016-01-11 12:42:15 -08:00
Aaron Schulz
58cffd04b8 Pass __METHOD__ to load balancer commit/rollback methods
Change-Id: I3fd87908c2a493fae49de6a29efe79f8d433c112
2015-12-22 18:30:20 -08:00
jenkins-bot
12e6fd1ec0 Merge "RefreshLinksJob: Restore LinksUpdate::setTriggeringUser() call" 2015-12-22 00:38:55 +00:00
jenkins-bot
b59ca1b44c Merge "No-op the notify*/purge methods in JobQueueAggregatorRedis" 2015-12-18 20:14:41 +00:00
Aaron Schulz
8677213249 Avoid negative JobRunner time stats
This can happen in sub-second cases with skew. It makes
graphana tend to see -1 as the min for some time values.

Change-Id: I4e39d8ac29f515fd76548f1a7b64d71a03064407
2015-12-18 02:54:13 +00:00
Aaron Schulz
2f2641dbda No-op the notify*/purge methods in JobQueueAggregatorRedis
This should be managed by the background service now

Change-Id: I5d905760b509f66e10019ff8a4146db8a6c4c9d7
2015-12-17 16:43:43 -08:00
Kunal Mehta
829c4a8503 RefreshLinksJob: Restore LinksUpdate::setTriggeringUser() call
This partially reverts 22476baa85, as the setTriggeringUser()
call that was removed was being used by Echo to be able to determine
which user caused a LinksUpdate to be triggered.

Bug: T121780
Change-Id: I62732032a6b74f17b5ae6a2497fa519f9ff38d4f
2015-12-17 10:50:53 -08:00
Aaron Schulz
2878c6443a Make redis JobQueueConnectionError mention the host
Change-Id: I00a43265a65de9c9fa58b7046770d754b7d27815
2015-12-11 21:39:02 +00:00
Aaron Schulz
d54591216e Remove wfForeignMemcKey() usage from JobQueueRedis::getQueueKey()
This class should manage the escaping it uses, rather than use some
random BagOStuff that has nothing to do with the job queue.

Change-Id: Ie716dc4a3429754a99c5f0670555e5e049b61aa1
2015-12-10 17:34:22 -08:00
Aaron Schulz
ec351986e5 Remove JobQueue::setTestingPrefix() hack
The tests are only run on dev install and only touch the
null queue anyway.

Change-Id: I441a2a4605a9e2984142485b022dd524ff819360
2015-12-10 17:20:10 -08:00
Aaron Schulz
afb86872c0 Fix bad regex in 6fe2f48df
Change-Id: I51d136a491111b7c980ddb13fff74bf4a63b15ad
2015-12-10 16:35:02 -08:00
Aaron Schulz
6fe2f48df7 Add per-partition JobQueueRedis aggregation
* Track queues with non-abandoned jobs per partition server.
  The s-queuesWithJobs key can easily be queried to see which
  queues need to have periodic tasks run (or for debugging).
* This is requirement for the redis jobchron service to be able to
  avoid hitting N=(no. types X no. wikis) queues for periodic tasks
  when only a tiny fraction of those actually have any jobs. For WMF,
  there are over 30K queues, most of them empty, so doing that can help
  lower redis-server CPU (or at least make jobchron more responsive).
* This also allows for jobchron to manage the aggregator by taking the
  per-server aggregator sets and merging them. This scales much better
  as there are only a modest number of these daemons (18 for WMF) but
  vastly more web thread pushing jobs. This cuts down on the connections
  to the active aggregator server (the one with the hash table).
* Use Lua unpack() more for stylistic consistency.

Change-Id: I1549f0edc78cc4004dd887b475dec4c0ebd306c6
2015-12-10 23:59:59 +00:00
Aaron Schulz
01c2b0a425 Add $wgCdnReboundPurgeDelay for more consistent CDN purges
Bug: T113192
Change-Id: I89deb4f8143c1cc6154cdd05bcee1f49d3e3a75a
2015-12-10 22:52:30 +00:00
Aaron Schulz
6af3c39c07 Replace "squid" with "CDN" in various comments
Change-Id: Idcc528daf28e119349155d36e30a9bcf61b2e7d5
2015-12-09 17:35:37 -08:00
Aaron Schulz
282c5fa9f3 Rename SquidUpdate => CdnCacheUpdate
Squid is not the only possible CDN

Change-Id: Ie2a2955847c5706e630322bbbab71c9d063b378f
2015-12-09 16:31:17 -08:00
Krinkle
a2c30ecb02 Merge "Remove obsolete category links code" 2015-12-06 18:42:39 +00:00
jenkins-bot
916318a0f6 Merge "Make RefreshLinksJob de-duplication more robust" 2015-12-06 18:41:48 +00:00
Aaron Schulz
22476baa85 Remove obsolete category links code
* These calls and methods should no longer be needed
* Follow-up to 6dedffc2d7

Change-Id: Iff121263610117112c84edb5e575f039456d1ac8
2015-12-05 16:07:59 -08:00
Aaron Schulz
25a39d255c Make RefreshLinksJob de-duplication more robust
* Do not de-duplicate jobs with "masterPos". It either does not
  catch anything or is not correct. Previously, it was the later,
  by making getDuplicationInfo() ignore the position. That made the
  oldest DB position win among "duplicate" jobs, which is unsafe.
* From graphite, deduplication only applies .5-2% of the time for
  "refreshLinks", so there should not be much more duplicated
  effort. Dynamic and Prioritized refreshLinks jobs remain
  de-duplicated on push() and root job de-duplication still applies
  as it did before. Also, getLinksTimestamp() is still checked to
  avoid excess work.
* Document the class constants.

Change-Id: Ie9a10aa58f14fa76917501065dfe65083afb985c
2015-12-04 12:40:10 -08:00
jenkins-bot
4d20257029 Merge "Add more BacklinkJobUtils comments" 2015-12-04 19:24:54 +00:00
jenkins-bot
eb735233fe Merge "Fix some IDEA errors in BacklinkJobUtils" 2015-12-04 19:19:51 +00:00
jenkins-bot
d57d989f73 Merge "Make CDN purge calls use DeferredUpdates" 2015-12-04 19:19:09 +00:00
Aaron Schulz
afbff42aca Make CDN purge calls use DeferredUpdates
* Using addUpdate() makes sure purges are coalesced and
  de-duplicated.
* Also removed incosistent $wgUseSquid checks. If CDN caching
  is not used, then $wgSquidServers will just be empty anyway.

Bug: T119016
Change-Id: I8b448366f037f668385d252f9d68289b71d1a707
2015-12-04 19:09:03 +00:00
jenkins-bot
10e1c76a97 Merge "Make HTMLCacheUpdate always use the job queue" 2015-12-04 19:07:58 +00:00
addshore
10703fae49 Fix categories with a namespace prefix in the cat name for CatWatch
Title::newFromText will use the given namespace as default, but when
parsing a title with a namespace at the begin the method will not use
the default, instead used the namespace from the given text.
Use Title::makeTitle to create a title with always belongs to the given
namespace.

Bug: T119763
Change-Id: Ic96671e1c33c6572b07f0f859d97c85a7a15bd39
2015-12-04 12:03:44 +01:00
addshore
84568051da Fix CatMemberChanges for numeric category names
Change-Id: I5ff3ff2d26d8e2d0b4250843f028eabc96020ccc
2015-12-03 14:19:44 +01:00
Aaron Schulz
6dedffc2d7 Move category membership RC updates to CategoryMembershipChangeJob
* Recursive link updates no longer mention an category changes.
  It's hard to avoid either duplicate mentioning of changes or
  confusing explicit and automatic category changes.
* LinksUpdate no longer handles this logic, but rather WikiPage
  decides to spawn this update when needed in doEditUpdates().
* Fix race conditions with calculating category deltas. Do not
  rely on the link tables for the read used to determine these
  writes, as they may be out-of-date due to slave lag. Using the
  master would still not be good enough since that would assume
  FIFO and serialized job execution, which is not garaunteed.
  Use the parser output of the relevant revisions to determine
  the RC rows. If 3 users quickly edit a page's categories, the
  old way could misattribute who actually changed what.
* Make sure RC rows are inserted in an order that matches that
  of the corresponding revisions.
* Better avoid mentioning time-based (parser functions) category
  changes so they don't get attributed to the next editor.
* Also wait for slaves between RC row insertions if there where
  many category changes (it theory it could well over 10K rows).
* Using a separate job better separates concerns as LinksUpdate
  should not have to care about recent changes updates.
* Added more docs to $wgRCWatchCategoryMembership.

Bug: T95501
Change-Id: I5863e7d7483a4fd1fa633597af66a0088ace4c68
2015-12-03 11:28:05 +00:00
Aaron Schulz
5820f735ae Add more BacklinkJobUtils comments
Change-Id: Ic1b9f730c3876214dfebf7623a2d4dec5c9c08af
2015-12-01 15:17:42 -08:00
Aaron Schulz
912975bfe1 Fix some IDEA errors in BacklinkJobUtils
Change-Id: I0e61f93bc19c183442933dbb8192264838c1c9a1
2015-12-01 13:54:34 -08:00