Commit graph

402 commits

Author SHA1 Message Date
jenkins-bot
38a2a5661e Merge "Add $wgMaxJobDBWriteDuration setting for avoiding replication lag" 2017-06-12 18:15:57 +00:00
Aaron Schulz
95fdff36c2 Make DeferredUpdates detect LBFactory transaction rounds
Previously, tryOpportunisticExecute() tried to nest transaction rounds,
which would fail. Added LBFactory::hasTransactionRound() as needed.

Also cleaned up some unqualified class names in callbacks and set the
PRESEND flag for the JobQueueDB AutoCommitUpdate callback. Use the
proper getMasterDB() method while at it. These follow up 24842cfac.

Bug: T154425
Change-Id: Ib1d38f68bd217903d1a7d46fb15b7d7d9620daa6
2017-06-10 15:22:32 +00:00
Seb35
24842cfac0 Use AutoCommitUpdate instead of Database->onTransactionIdle
This is needed for deferred updates LinksDeletionUpdate and LinksUpdate, else
callbacks registered with onTransactionIdle prevent other transactions from
being executed, at least in this case.

Bug: T154425
Bug: T154438
Bug: T157679
Change-Id: Iecd396d584a62ac936cd963915339159467b44cd
2017-06-06 14:23:37 +02:00
Seb35
d80fca05e1 Better handling of jobs execution in post-connection shutdown
In the postprocessing, some jobs can be executed but given the deferred
updates were already "closed", any new DeferredUpdate were directly called
(as explained by Krinkle on T165714), and the transactions opened by
classical jobs are badly mixed with transactions (directly) executed by
DeferredUpdates jobs, issuing a DBError, avoiding the job, which stays
in a 'claimed' status even if failed.

Quite similarly, some DeferredUpdates callables use JobQueueGroup::lazyPush
so it is needed to really push the generated jobs.

This change removes the run-immediately-deferred-updates behaviour even
in the post-connection shutdown, and given there is a call to
DeferredUpdates::doUpdates in JobRunner::execute it is not necessary to
add another one and hence execution of Web jobs is more similar to execution
of CLI jobs. In the same spirit to reconcile Web jobs and CLI jobs, the
call to JobQueueGroup::pushLazyJobs is done in JobRunner::execute.

Bug: T165714
Bug: T100085
Change-Id: I721e7167eca5b0b6227234fe516005243ab22388
2017-06-01 13:16:08 +02:00
Aaron Schulz
ac202927d4 Add $wgMaxJobDBWriteDuration setting for avoiding replication lag
This is similar to $wgMaxUserDBWriteDuration except for jobs.

Also use the Config class in JobRunner instead of globals.

Bug: T95501
Change-Id: I4949bb99c26451429c7acf82ecc4444bf9fb835f
2017-05-25 19:43:27 +00:00
Aaron Schulz
1455cbd8a3 Add type hint to ThumbnailRenderJob::hitThumbUrl
Change-Id: Ifab9d74ae9147349b0a3417f35088b96ee47b6fa
2017-05-19 12:17:27 -07:00
jenkins-bot
58e1e472ba Merge "Allow callback functions for creating jobs" 2017-05-09 18:38:57 +00:00
Lucas Werkmeister
a57252a1f5 Allow callback functions for creating jobs
$wgJobClasses can now specify a factory function for creating a job,
instead of a class to be instantiated directly. This makes it possible
to inject services in a job constructor, and register a factory function
that calls the constructor with default services.

This follows Ieb85493a7765 and Ia2107dc5af78, which introduced factory
functions for API modules and special pages.

Change-Id: I0461e59da2a8fa6681e3b1fcdfc38bfed7f3ac32
2017-05-09 19:26:13 +02:00
Paladox
54c56da85a Fix php code style
Preparation change for updating mediawiki code sniffer to 0.8.0

Change-Id: Ib0b3fe4afea9096ffa3a1347b4f7e07d3398b0b2
2017-05-05 12:03:54 +00:00
Aaron Schulz
fcc2895cad Fix bogus variable use in RefreshLinksJob::run()
Also removed two unused loop variables.

Change-Id: I9a9e0a83bdaa13c031857bc20f977161cf85baff
2017-04-20 11:29:25 -07:00
Aaron Schulz
0b58cdc7f2 Lower the amount of jobs pushed into redis at once
This further limits how long the server can be tied up by push().

Change-Id: I02d242578dadc19912c9fccfdcf5e15c5eb78e9e
2017-04-19 15:38:32 -07:00
Aaron Schulz
dd359741cc Move DB errors to Rdbms namespace
Change-Id: I463bd86123501abc68fdb78b4cda6110f7af2549
2017-04-15 10:47:41 -07:00
Seb35
e00f7180b3 Prevent PHPDBG from issuing notices in unit tests
Although issuing PHP notices in unit tests is only a nice-to-have, the cause
is PHPDBG has no special treatment in JobQueueGroup at the contrary of the
PHP SAPI 'cli', and it may be desirable both 'command line SAPIs' have the
same behaviour.

This is the most apparent difference between cli and phpdbg but there could
be a dozen of other occurrences (see bug), potentially creating differences
in unit tests depending on the SAPI.

Bug: T162591
Change-Id: Idf9c14db72f1f768c5a17b49ed689a05922c57d3
2017-04-10 13:32:15 +02:00
Aaron Schulz
3c2a172c3f Move DBConnRef/MaintainableDBConnRef to Rdbms namespace
Change-Id: I7ca6b40f973b2aae83747de92e280ded222c87e3
2017-03-29 11:04:47 -07:00
Aaron Schulz
488a647831 Move IDatabase/IMaintainableDatabase to Rdbms namespace
Change-Id: If7e8a8ff574661fd827de8bcec11d2c39a687300
2017-03-28 15:32:38 -07:00
WMDE-Fisch
7b5f08e703 Replace deprecated Context::getStats() with MWServices::getStatsdDataFactory()
Change-Id: I1756f69ca2ebd301a5049bf758d1a87c37771fe6
2017-03-17 12:07:03 +01:00
Timo Tijhof
3a2a707546 Clean up remaining get_class() uses
* get_class()        -> __CLASS__ (same as self::class)
* get_called_class() -> static::class
* get_class($this)   -> static::class

Change-Id: I1888a1897ecf4548a2e5a67a942e5c080dd7e3d3
2017-03-07 22:03:47 +00:00
jenkins-bot
aa3319c4c0 Merge "Miscellaneous indentation tweaks" 2017-02-28 18:38:36 +00:00
Bartosz Dziewoński
ecdef925bb Miscellaneous indentation tweaks
I was bored. What? Don't look at me that way.

I mostly targetted mixed tabs and spaces, but others were not spared.
Note that some of the whitespace changes are inside HTML output,
extended regexps or SQL snippets.

Change-Id: Ie206cc946459f6befcfc2d520e35ad3ea3c0f1e0
2017-02-27 19:23:54 +01:00
James D. Forrester
9635dda73a includes: Replace implicit Bugzilla bug numbers with Phab ones
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.

Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
2017-02-21 18:13:24 +00:00
jenkins-bot
0b94a8e3b9 Merge "Make RecentChangesUpdateJob::updateActiveUsers more robust" 2017-02-16 16:18:06 +00:00
WMDE-Fisch
ae3419f532 Replace some of the deprecated wfGetLBFactory() calls
Change-Id: I87aa0ebe273706438f1634167de8c60f51432878
2017-02-07 14:41:09 +01:00
Aaron Schulz
4a177b34ef Move LBFactory to Rdbms namespace
Change-Id: I5ae10783228d0252284807c9562bc8e328d4becb
2017-02-03 17:24:03 -08:00
Marius Hoch
bf8c90538d Make RecentChangesUpdateJob::updateActiveUsers more robust
Do work we can outside of the lock (purge no longer active users)
and fail more gracefully if the lock is taken (if this is the case,
we're probably on a high traffic wiki, so this job is going to
run very often anyway, so no need to warn).

Given this is an asynchronous job people can't expect this to be
always fully consistent with the actual state of the RC table anyway.

Note: This will still sometimes log the warning (if we have a race
between Database::lockIsFree and Database::lock), but that should be
negligible.

Bug: T156638
Change-Id: I2e49a8e12bc64156d64a32ccf54911a76087346a
2017-01-31 11:06:59 +01:00
Aaron Schulz
2d4ed16bd8 Make RefreshLinksJob handle LinksUpdateConstructed hooks doing DB writes
Bug: T153618
Change-Id: Iae52e9225fe132f2aa99e161611bf8258736d38d
2017-01-07 17:40:11 +00:00
jenkins-bot
9ac29c74ed Merge "Cleanup some incorrect return annotations" 2016-12-16 07:22:24 +00:00
Erik Bernhardson
d67197fa11 Cleanup some incorrect return annotations
Most of these are simply changing annotations to reflect
reality. If a function can return false to indicate failure
the @return should indicate it.

Some are fixing preg_match calls, preg match returns 1, 0 or false,
but the functions all claim to return booleans.

This is far from all the incorrect return types in mediawiki, there
are around 250 detected by phan, but have to start somewhere.

Change-Id: I1bbdfee6190747bde460f8a7084212ccafe169ef
2016-12-12 10:15:05 -08:00
Aaron Schulz
6477026675 Back off from job types longer for DB read-only errors
Such error are likely to persist longer than other random
exceptions. In that case, it is better to avoid burning
through the job retry count.

Change-Id: I6785bd608856f98d21e0b0b05d3899a7081c38e2
2016-12-09 23:26:34 -08:00
jenkins-bot
69ae945e8d Merge "Update weblinks in comments from HTTP to HTTPS" 2016-11-08 21:32:00 +00:00
Fomafix
202f695f67 Update weblinks in comments from HTTP to HTTPS
Use HTTPS instead of HTTP where the HTTP link is a redirect to the HTTPS link.

Also update some defect links.

Change-Id: Ic3a5eac910d098ed5c2a21e9f47c9b6ee06b2643
2016-11-07 15:24:46 +01:00
Aaron Schulz
1eb86e0e4d Rename JobQueueDB::getSlaveDB() to getReplicaDB()
Change-Id: I7a88c4876e76836c92cc0b45b4e72124a087121c
2016-10-29 00:04:39 -07:00
Kunal Mehta
61adc1e146 Use namespaced ScopedCallback
The un-namespaced \ScopedCallback is deprecated.

Change-Id: Ie014d5a775ead66335a24acac9d339915884d1a4
2016-10-17 15:46:05 -07:00
Gergő Tisza
d304f5e394 Pass Job success status to teardown callbacks
Change-Id: Icf2e03efcfd9232fe4ead776096b61cef1c06141
2016-10-05 02:55:45 +00:00
Aaron Schulz
3740e4eed2 Use @lang tags for Lua scripts
Change-Id: Ifdbff687539078b375330ebe3a1493b281e5c518
2016-09-25 11:50:16 -07:00
Aaron Schulz
10593ffaab Move RedisConnectionPool to /libs/redis
Change-Id: Ied4a85d7172ab76b90f6d9ce4d47a83c3fd7d111
2016-09-23 00:20:19 -07:00
Aaron Schulz
83e8d972fe Avoid triggering SiteConfiguration lookup in JobQueueGroup::push()
Just use $wgJobTypesExcludedFromDefaultQueue directly

Bug: T145819
Change-Id: I0d978cdbef40ced98880c1a7d7524fd2b7438bfc
2016-09-16 11:16:19 -07:00
Aaron Schulz
a2790b1b80 Remove wfGetCaller() dependency from DatabaseBase
Change-Id: I3e240b2eb5c1f6a21f1bc974c3d28f5755c7451a
2016-09-15 14:40:00 -07:00
Aaron Schulz
e4cbf4b9d4 Reduce CategoryMembershipChangeJob lock timeout
Bug: T133801
Change-Id: Ic5db3983a8b67b9138b602188a52d60dd95fcb59
2016-09-14 10:28:02 +00:00
jenkins-bot
47d437a6c9 Merge "Remove pointless double exception logging from JobRunner" 2016-09-13 22:26:26 +00:00
jenkins-bot
ec7d855d42 Merge "Reorganize RefreshLinksJob code slightly and avoid deprecated functions" 2016-09-13 04:39:50 +00:00
Aaron Schulz
1cb13cff08 Remove pointless double exception logging from JobRunner
Change-Id: I12a2e6db326af25a3a276a477fbff505feac87b6
2016-09-13 04:38:36 +00:00
jenkins-bot
05870e1c9f Merge "Avoid using deprecated methods in JobQueueDB" 2016-09-13 04:27:03 +00:00
Aaron Schulz
f8a9490f88 Reorganize RefreshLinksJob code slightly and avoid deprecated functions
Change-Id: I6ff4bec61b37bfbffc1e96eac61d692dd7feb31a
2016-09-12 21:11:11 -07:00
Aaron Schulz
835795a6ad Simplify LBFactory ticket code in CategoryMembershipChangeJob
Change-Id: I5800bbf6fa718604ffa12d8cde1aa3675fced6fd
2016-09-12 20:44:19 -07:00
Aaron Schulz
fca4beb302 Avoid using deprecated methods in JobQueueDB
Change-Id: Ib35b8792e3e4902b52c3e708d8c6e756f35986bd
2016-09-12 19:56:37 -07:00
jenkins-bot
854a931db3 Merge "Make JobQueueGroup::push() update the queuesHaveJobs() cache" 2016-09-12 22:20:20 +00:00
Aaron Schulz
d077eb9daa Make PurgeJobUtils avoid creating DB replication lag
Large affected rows counts were being reported in DBPerformance logs.

Change-Id: Ia5504aa4fbd27473771c65688f0b9e78e3a5caae
2016-09-11 16:59:12 -07:00
Aaron Schulz
703b0691ca Use ESTIMATE_DB_APPLY for total transaction time estimate
Individual write queries already do this, but the COMMIT step
still used the old accounting.

Change-Id: I416a524d6652f933cbc49033b49745db732c8b92
2016-09-11 16:04:21 -07:00
Aaron Schulz
6f9a246d25 Make JobQueueGroup::push() update the queuesHaveJobs() cache
This makes it so jobs can run sooner with $wgJobRunRate

Bug: T145206
Change-Id: Ia7673f51312f90a696fa056bd3daf7f3f440782b
2016-09-11 04:57:12 -07:00
jenkins-bot
8e22988441 Merge "Make sure the lock in JobRunner::commitMasterChanges() releases" 2016-09-09 03:04:59 +00:00