The mediawiki.runJobs errors are collected in Logstash but the whole job
description and errors are a single field message. That is challenging
to split logs per job type, get the longest running jobs ...
That can be worked around on the log receiving side by parsing
MediaWiki messages eg https://gerrit.wikimedia.org/r/#/c/312504/
Bryan Davis suggested a better long term solution is to use the PSR3
logger with structured log messages.
Culprit: 'type' is a reverved word. Hence prefix all context variables
with 'job_'.
Bug: T146469
Change-Id: Ib6a771c7d3f83bd75b2994bfab9bbebfd1f5aa6c
As a part of creating the kafka-based JobQueue implementation,
we need to transition to the new queue smoothly and be able to
first debug the event production side, then try executing a no-op
job in parallel with each of the real jobs (to be able to check
deduplication and all the jpb management code) and only then
try switching jobs one-by-one to the new backend queue.
In order to achieve that at first we need to temporary produce
both to the old job pipeline and to the new one - thus creating
a queue 'splitter' - it delegates all the methods to the main
queue while the produce requests also go to the EventBus.
This class it temporary, so it's marked deprecated right away.
It will not be needed once the queue transition is over.
Bug: T163380
Change-Id: I483b8f00f1c593ff3af6ba6826868b553339525f
And auto-fix all errors.
The `<exclude-pattern>` stanzas are now included in the default ruleset
and don't need to be repeated.
Change-Id: I928af549dc88ac2c6cb82058f64c7c7f3111598a
Adds RecentChangesPurgeRows hook which gets called in the same
transaction where old recentchanges rows are purged, and gets the
list of deleted rows. Extensions which store data for joining to
recentchanges can use this to purge their own data.
Bug: T159753
Change-Id: I03f1d485a1a3004412e0859d9d878b7895c95b40
Previously, tryOpportunisticExecute() tried to nest transaction rounds,
which would fail. Added LBFactory::hasTransactionRound() as needed.
Also cleaned up some unqualified class names in callbacks and set the
PRESEND flag for the JobQueueDB AutoCommitUpdate callback. Use the
proper getMasterDB() method while at it. These follow up 24842cfac.
Bug: T154425
Change-Id: Ib1d38f68bd217903d1a7d46fb15b7d7d9620daa6
This is needed for deferred updates LinksDeletionUpdate and LinksUpdate, else
callbacks registered with onTransactionIdle prevent other transactions from
being executed, at least in this case.
Bug: T154425
Bug: T154438
Bug: T157679
Change-Id: Iecd396d584a62ac936cd963915339159467b44cd
In the postprocessing, some jobs can be executed but given the deferred
updates were already "closed", any new DeferredUpdate were directly called
(as explained by Krinkle on T165714), and the transactions opened by
classical jobs are badly mixed with transactions (directly) executed by
DeferredUpdates jobs, issuing a DBError, avoiding the job, which stays
in a 'claimed' status even if failed.
Quite similarly, some DeferredUpdates callables use JobQueueGroup::lazyPush
so it is needed to really push the generated jobs.
This change removes the run-immediately-deferred-updates behaviour even
in the post-connection shutdown, and given there is a call to
DeferredUpdates::doUpdates in JobRunner::execute it is not necessary to
add another one and hence execution of Web jobs is more similar to execution
of CLI jobs. In the same spirit to reconcile Web jobs and CLI jobs, the
call to JobQueueGroup::pushLazyJobs is done in JobRunner::execute.
Bug: T165714
Bug: T100085
Change-Id: I721e7167eca5b0b6227234fe516005243ab22388
This is similar to $wgMaxUserDBWriteDuration except for jobs.
Also use the Config class in JobRunner instead of globals.
Bug: T95501
Change-Id: I4949bb99c26451429c7acf82ecc4444bf9fb835f
$wgJobClasses can now specify a factory function for creating a job,
instead of a class to be instantiated directly. This makes it possible
to inject services in a job constructor, and register a factory function
that calls the constructor with default services.
This follows Ieb85493a7765 and Ia2107dc5af78, which introduced factory
functions for API modules and special pages.
Change-Id: I0461e59da2a8fa6681e3b1fcdfc38bfed7f3ac32
Although issuing PHP notices in unit tests is only a nice-to-have, the cause
is PHPDBG has no special treatment in JobQueueGroup at the contrary of the
PHP SAPI 'cli', and it may be desirable both 'command line SAPIs' have the
same behaviour.
This is the most apparent difference between cli and phpdbg but there could
be a dozen of other occurrences (see bug), potentially creating differences
in unit tests depending on the SAPI.
Bug: T162591
Change-Id: Idf9c14db72f1f768c5a17b49ed689a05922c57d3
I was bored. What? Don't look at me that way.
I mostly targetted mixed tabs and spaces, but others were not spared.
Note that some of the whitespace changes are inside HTML output,
extended regexps or SQL snippets.
Change-Id: Ie206cc946459f6befcfc2d520e35ad3ea3c0f1e0
It's unreasonable to expect newbies to know that "bug 12345" means "Task T14345"
except where it doesn't, so let's just standardise on the real numbers.
Change-Id: I6f59febaf8fc96e80f8cfc11f4356283f461142a
Do work we can outside of the lock (purge no longer active users)
and fail more gracefully if the lock is taken (if this is the case,
we're probably on a high traffic wiki, so this job is going to
run very often anyway, so no need to warn).
Given this is an asynchronous job people can't expect this to be
always fully consistent with the actual state of the RC table anyway.
Note: This will still sometimes log the warning (if we have a race
between Database::lockIsFree and Database::lock), but that should be
negligible.
Bug: T156638
Change-Id: I2e49a8e12bc64156d64a32ccf54911a76087346a
Most of these are simply changing annotations to reflect
reality. If a function can return false to indicate failure
the @return should indicate it.
Some are fixing preg_match calls, preg match returns 1, 0 or false,
but the functions all claim to return booleans.
This is far from all the incorrect return types in mediawiki, there
are around 250 detected by phan, but have to start somewhere.
Change-Id: I1bbdfee6190747bde460f8a7084212ccafe169ef
Such error are likely to persist longer than other random
exceptions. In that case, it is better to avoid burning
through the job retry count.
Change-Id: I6785bd608856f98d21e0b0b05d3899a7081c38e2
Use HTTPS instead of HTTP where the HTTP link is a redirect to the HTTPS link.
Also update some defect links.
Change-Id: Ic3a5eac910d098ed5c2a21e9f47c9b6ee06b2643
