* combine `JobQueueFederated` and `JobQueueRedis` into a single
`JobQueue` channel.
* Remove duplicate descriptions from file blocks in favour of class
doc blocks. This reduces needless duplication and was often
incorrect or outdated, and helps (ironically) to make the file header
more consistently visually ignorable. Various files in this patch
contained bogus copy-pasta descriptions from unrelated classes,
and re-defined `defgroup JobQueue` many times, showing exactly
how this is defacto ignored and counter-productive to maintain
in two places.
Remove `ingroup` from file blocks in class files as otherwise
the file is indexed twice (e.g. in Doxygen) which makes navigation
on doc.wikimedia.org rather messy for classes in this group.
Ref <https://gerrit.wikimedia.org/r/q/message:ingroup+is:merged>
Change-Id: I926a3aec2bc98fefa1075c4a794c46108579ae3f
Currently, we have two services: ReadOnlyMode and
ConfiguredReadOnlyMode. The former takes the latter and can easily
support every functionality of CRO. I added methods to support that
functionality (the current class is quite small so it's not adding a lot
more to a monster class) and basically fully deprecate the service
itself to improve developer experience.
It is only called in two extensions in production with straightforward
fix so I directly went with hard-deprecation.
Bug: T343917
Depends-On: Icec0ad3f30c471b428efe80dfb9129f55a351194
Change-Id: I421fd5c8fee7af41c059419c2bbc85b8dccc04ed
=== Change ===
Follows-up Ia61e00d6dc98c20650 which moved injection of $wgJobClasses
from happening lazily in methods that are never called in any
production code, to happen unconditionally when any JobQueueGroup
is constructed and injected, which happens a lot nowadays given that
most code is now dependency-injected.
For example, when logging in, CentralAuth can call into services
for getUserGroupManager( $wikiID )->getUserGroupMemberships() which
merely performs a simple database select query. But, because this
service class does contain other methods that can queue a job, this
config now needs to be computed across wikis via a potentially slow
shell execution even in simple GET requests.
One way to solve this, which brings back the previous state, is to
keep this logically dependency-injected but deferred by wrapping it
in a closure. We could inject `callable $getJobClassesFn` as param
from JobQueueGroupFactory to JobQueueGroup.
However, given that $wgConf->getConfig() was broken in production
for two weeks and there is not 1 log entry in Logstash during this
time, I think that means these methods are actually never called.
Hence, I'm instead going in the opposite direction of extending the
restriction of pop() to these other methods as well.
Thus, we reduce JobQueueGroup support for cross-wiki method calls
to get(), push() and lazyPush().
=== History ===
The natural question is, why was this added, and do we know that
this reason no longer applies? The getConfig() call was introduced
in 2013 with commit 04e0d75f86 (I7e6904ead1). This mentions as
reasoning to support maintenance/nextJobDB.php, which was was removed
in commit ce2ae144e6 (Ia74386c650) a year later.
It looks like this script supported only JobQueueDB, as used by
a WMF-specific jobrunner known as "jobs-loop.sh", which predates
WMF's JobQueue migration from MySQL to Redis, and later to Kafka,
as per
<https://wikitech.wikimedia.org/wiki/History_of_job_queue_runners_at_WMF>
Bug: T344223
Bug: T343291
Change-Id: Ic2293c4b4be10a698a2f891eaa63c1de7383f982
WANObjectCache has already been injected, just use it instead via the
member variable. Also, make use of a proper config system like
GlobalVarConfig for accessing global variables and MainConfigNames
to pass in JobClasses setting.
Centralize checking for job beings added to the queue if they can't
be run because they're not on the correct wiki. Let the factory
deal with this and avoid the JQG to know about it. Also, centralize
logic for getting cached job classes in the factory.
NOTE: private constant CACHE_VERSION is not used within the class,
so removed it. It's private so easy to remove directly.
Change-Id: Ia61e00d6dc98c206509724413cf641a225649dbd
The db/ directory does not have an owner and it's a mess in general.
These classes don't depend on anything in core except the rdbms library.
Let's simply move it there. In other words, Krinkle made me do it.
Since the class was moved in I6202e52ba73 merged less than a week ago,
no need to alias anything.
Bug: T321882
Change-Id: I24ceeb8bf765a50f441270136acd612359d50aa2
They are not suitable to go to rdbms library as they depend on mediawiki
pieces, the second best place is the db/ directory.
Bug: T321882
Change-Id: I6202e52ba7306d74261206c2ba7930c5f1a0a18e
This method should not exist and JobQueue::deduplicateRootJob() should
only be called internally, as it already is when actually needed.
Change-Id: I0dff7870fe8ec6d9a72c8bcd116d5e931bdbc79d
JobQueueGroup::get( $type ) returns a new object for each job type even
if it's the same class (default class). Also it builds different queues
and push them separelty a couple lines lower limiting the ability to
batch push them in the default job queue object.
Given that JobQueue cares about job types being the same as the class
type (which is important for JobQueueDB implementation), adding a config
option called typeAgnostic to ignore type mismatch in JobQueue
implementations that don't care about the type (like JobQueueEventBus)
Bug: T292048
Change-Id: I151f067ca94a985c816446b545921c387b083911
This replaces JobQueueGroup::singleton and ::destroySingletons.
A JobQueueGroup service has been added to serve as convenience
getter.
Bug: T277648
Change-Id: Ic8709119773ab788a07531193dc65d418536eda2
This is micro-optimization of closure code to avoid binding the closure
to $this where it is not needed.
Created by I25a17fb22b6b669e817317a0f45051ae9c608208
Change-Id: I0ffc6200f6c6693d78a3151cb8cea7dce7c21653
This includes fixing some mistakes, as well as removing
redundant text that doesn't add new information, either because
it literally repeats what the code already says, or is actually
duplicated.
Change-Id: I3a8dd8ce57192deda8916cc444c87d7ab1a36515
If the root job timestamp keys are lost or otherwise unknown, they
will now be deductively recached with the best known values as jobs
are popped and executed. This means the running any of many child
jobs of a root job can restore the root timestamp if it was lost.
This does not need to use the main stash given this fact.
Bug: T227376
Change-Id: Iae0f3af15803af048ff49f3bf281b2bde18c87f2
JobQueueGroup is giving RunnableJob on pop(), so it should take the same
type for ack() and deduplicateRootJob()
JobQueue::ack alsready accept the interface
Also change to RunnableJob in JobRunner to work with the type from the
job queue
Change-Id: I7b09586cff8affabe807ee16e80d04f5137dce45
Simplify the code of jobs that do not care about titles and removes
the direct Title dependency from JobQueue. Remove getTitle() from
IJobSpecification itself. Move all the Job::factory calls into a
single JobQueue::factoryJob() method.
Depends-on: Iee78f4baeca0c0b4d6db073f2fbcc56855114ab0
Change-Id: I9c9d0726d4066bb0aa937665847ad6042ade13ec
Also moved some WikiMap/$wgJobClasses checks to JobQueueGroup::pop
which is the method callers are supposed to use.
Change-Id: I2ab82d8adc4ae1f54697d2935afa2053539cf2db
This was originally a global search and replace. I manually checked all
replacements and reverted them if (due to the lack of type hints) either
null (that would be 0 when counted) or a Countable object can end in the
variable or property in question.
Now this patch only touches places where I'm sure nothing can break.
For the sanity of the honorable reviewers this patch is exclusively touching
negated counts. You should not find a single `!== []` in this patch, that
would be a mistake.
Change-Id: I5eafd4d8fccdb53a668be8e6f25a566f9c3a0a95
Using domains means thats JobQueueDB has the right value to use for calls
like LoadBalancer::getConnection(). The full domain includes the schema in
the case of Postgres. This makes calls to getConnection() less awkward by
not relying on the fallback logic in reallyOpenConnection() for null schemas.
Make getWikiIdFromDomain/isCurrentWikiDomain account for the schema if it
is both defined and is not simply the generic "mediawiki" schema MediaWiki
uses by default. If all wikis use the default schema, the wiki IDs can get
by with DB/prefix alone, which various config and methods may be built around.
Otherwise, the config callbacks must account for schema and the config must
include it in various wiki domain ID lists to properly disambiguate wikis.
Also, clean up SiteConfiguration::siteFromDB() since it is not meant
to handle schemas unless the callback method was taylored to do so.
Finally, add more comments to DefaultSettings.php about already existing
limitations of wiki domain IDs and their components.
Change-Id: I8d94a650e5c99a19ee50551c5be9544318eb05b1
Use these in place of various wfWikiID() calls.
Also cleanup UserRightsProxy wiki ID variable names and removed unused
and poorly named getDBname() method.
Change-Id: Ib28889663989382d845511f8d34712b08317f60e
This assures that MergeableUpdate tasks that lazy push job will actually
have those jobs run instead of being added after the lone callback update
to call JobQueueGroup::pushLazyJobs() already ran.
This also makes it more obvious that push will happen, since a mergeable
update is added each time lazyPush() is called and a job is buffered,
rather than rely on some magic callback enqueued into DeferredUpdates at
just the right point in multiple entry points.
Bug: T207809
Change-Id: I13382ef4a17a9ba0fd3f9964b8c62f564e47e42d
We use $wgReadOnly for various reasons, one of which is to disallow
writes in the currently-non-active DC. However, we should allow the
readOnlyReason configuration variable to be available per JobQueue type
and have it the code respect that.
In our current set-up, we use JobQueueEventBus which ever only uses the
enqueue execution path and does not depend upon which DC it is executed
in, so this will allow us to enqueue jobs in both DCs.
Note that this is an alternative approach to the one outlined in
Ibbad6063b6b154d7f7d172c79f7be324bf80eb7e
Bug: T199594
Change-Id: I8f1a57a81ea11c1c587c0057fa8bb3454b0e0b56
