The "AUTO" means AUTOCOMMIT, not "automatic transactions"/DBO_TRX,
which is basically the opposite concept. The new name does not
suffer from that ambiguity.
Keep the old constant as an alias for backwards compatibility.
Also remove LoadBalancer comment about non-existing field
Change-Id: I63beeb061fc9be73f320308e4d6393b58628b8c8
When LoadBalancer opens new local domain connections, it currently
assumes that the domain specified by the server info array is the
same. For sanity, make sure that the handle is set to the local
domain.
The main LBFactory/LoadBalancer use $wgDBname/$wgDBprefix as the
local domain, corresponding with wfWikiId(). This relation is set
automatically in MWLBFactory. If $wgLBFactoryConf/$wgDBservers is
manually configured in a way breaking this correspondance, then it
is misconfigured.
Fixes made to avoid test failure:
* Make sure LoadBalancer::setDomainPrefix() updates the local
domain alias member. Also do not bother changing the domain of
foreign connections.
* Use the right domain ID for the connection array key names in
LoadBalancer::openForeignConnection().
* Now that JobQueueTest no longer mistakenly uses the non-test
tables, force it to use the main DB_MASTER handle so that it can
see the unit test tables even if they are TEMPORARY; such tables
are tied to the TCP connection, so separate handles see different
temporary tables.
Change-Id: I56f8b32fe957f984b8c9753e6db3b20abe96b038
* Track queues with non-abandoned jobs per partition server.
The s-queuesWithJobs key can easily be queried to see which
queues need to have periodic tasks run (or for debugging).
* This is requirement for the redis jobchron service to be able to
avoid hitting N=(no. types X no. wikis) queues for periodic tasks
when only a tiny fraction of those actually have any jobs. For WMF,
there are over 30K queues, most of them empty, so doing that can help
lower redis-server CPU (or at least make jobchron more responsive).
* This also allows for jobchron to manage the aggregator by taking the
per-server aggregator sets and merging them. This scales much better
as there are only a modest number of these daemons (18 for WMF) but
vastly more web thread pushing jobs. This cuts down on the connections
to the active aggregator server (the one with the hash table).
* Use Lua unpack() more for stylistic consistency.
Change-Id: I1549f0edc78cc4004dd887b475dec4c0ebd306c6
If we really need this we can do it in MediaWikiTestCase, next
to the setting of wgMainCacheType. But from what I can see the
code being tested here already doesn't use the old $wgMemc.
Change-Id: I9e4b2109b2f3c18d8d5551bbadae5711c1d4c0a6
* Remove some getAcquiredCount() assertions when claimTTL=0
as this is not well defined enough (queues may take a few
minutes to garbage collect the failed jobs).
* Added some tests to make sure push() only de-duplicates
among unclaimed jobs.
Change-Id: Ie0a5e539095c245dfcc8c160417e12824eb7ab83
I noticed JobQueueTest::testRootDeduplication takes ~ 6.5 seconds, which
is due to the test method using sleep(1) and being passed the provider
provider_queueLists which yields six items.
The reason is to have the array returned by Job::newRootJobParams() to
have an incread value for 'rootJobTimestamp'. Instead, just copy the
previous array of parameters and increment the UNIX timestamp and
converting back to TS_MW format.
Change-Id: I75066df73f9f92e56b89eb6d928c41e949a2d6a9
Also update some previous inconsistencies pointed out by Krinkle in change IDs:
* Ide20743a2e84ff68549286120e6cff9d9f396f54
* I811ca957b6588085d67606ebc0cd4033a1e53839
Change-Id: Ife33b931870d0d7e04fcb40974997436d27f528f
Change some tests to use setMwGlobals to have restoring of globals after
the test.
This also removes some save/restore code, which is not needed, due to
the automatically restoring on tearDown with setMwGlobals.
Change-Id: I8d2ac9f6cc14f0bd4ee8eb851c09f2e71babc6e0
* Cleaned up some data structures into hashes, which get better
compression and play well with the KEYS parameter in Lua scripts.
The claim list is now a sorted set with O(logN) removal in ack()
and O(log(N)+M) searching in recycleAndDeleteStaleJobs().
* Made the class itself control object serialization, so that lua
scripts have an easy time. Only the job data itself needs to be
serialized, where as other things just get bloated.
* Used Lua scripts to get push(), pop() and ack() down to 1 RTT.
* Likewise rewrote recycleAndDeleteStaleJobs() to use a script.
* Fixed bug where claimed duplicate jobs removed the data on ack(),
which meant that claimed duplicated jobs could no-op newer ones.
De-duplication should only apply to unclaimed jobs like for the
JobQueueDB class, so that unfinished jobs don't no-op new ones.
* Removed locking in recycleAndDeleteStaleJobs(), which would not do
much since the exclusive set request would serialize on the lua
script anyway. The lua script will finish quickly the next times
if done more than once in a row due to sorted set usage.
Also made recycleAndDeleteStaleJobs() run randomly to reduce the
chance of a single calling tying up the server.
* Removed useless hDel() call in getJobFromUidInternal().
* Changed unit tests to handle the different supported orders better.
Added tests for the 'timestamp' ordering.
Change-Id: Ib2d7aff18753195248ab856afd4a46e18b301db9
* Cleaned up 'server' option to not fragment the pool.
Also made it actually match the documentation.
* Made it use doGetPeriodicTasks() for job recycling.
* Made it so that other job queue classes can be tested.
* Renamed "redisConf" => "redisConfig".
* Tweaked comments about the "random" order option.
Change-Id: I7823d90010e6bc9d581435c3be92830c5ba68480
