I noticed JobQueueTest::testRootDeduplication takes ~ 6.5 seconds, which
is due to the test method using sleep(1) and being passed the provider
provider_queueLists which yields six items.
The reason is to have the array returned by Job::newRootJobParams() to
have an incread value for 'rootJobTimestamp'. Instead, just copy the
previous array of parameters and increment the UNIX timestamp and
converting back to TS_MW format.
Change-Id: I75066df73f9f92e56b89eb6d928c41e949a2d6a9
- Place commas correct
- Moved comments
- Add space after if/foreach/catch
- Reformat some conditions
- Removed trailing spaces/tabs
Change-Id: I40ccda72c418c4a33fcd675773cb08d971510cdb
* The changes refreshLinks to handle both per-title (leaf) and backlink jobs.
The base job now splits into some leaf jobs and a remaining partition job.
The partition job does the same until there are only a small number
of backlinks in the remaining range (so only leaf jobs are added).
Since the leaf jobs are pushed first, this works well for FIFO queues
to avoid bloating the queue. This also improves per-title job
de-duplication, which isQueueDeprioritized() pretty much killed.
* The refreshLinks2 class is no longer used for new jobs.
* Fix process cache bug with JobQueueGroup::push with empty arrays.
* This adds a BacklinksJobUtils with helper functions for partitioning.
* RefreshLinksJob jobs now have a simple version parameter.
* Also moved refreshLinks2Job to its own file.
Change-Id: Id378d47df17248ae02938d5a54ef7ecd29efadbd
Also update some previous inconsistencies pointed out by Krinkle in change IDs:
* Ide20743a2e84ff68549286120e6cff9d9f396f54
* I811ca957b6588085d67606ebc0cd4033a1e53839
Change-Id: Ife33b931870d0d7e04fcb40974997436d27f528f
Change some tests to use setMwGlobals to have restoring of globals after
the test.
This also removes some save/restore code, which is not needed, due to
the automatically restoring on tearDown with setMwGlobals.
Change-Id: I8d2ac9f6cc14f0bd4ee8eb851c09f2e71babc6e0
* Cleaned up some data structures into hashes, which get better
compression and play well with the KEYS parameter in Lua scripts.
The claim list is now a sorted set with O(logN) removal in ack()
and O(log(N)+M) searching in recycleAndDeleteStaleJobs().
* Made the class itself control object serialization, so that lua
scripts have an easy time. Only the job data itself needs to be
serialized, where as other things just get bloated.
* Used Lua scripts to get push(), pop() and ack() down to 1 RTT.
* Likewise rewrote recycleAndDeleteStaleJobs() to use a script.
* Fixed bug where claimed duplicate jobs removed the data on ack(),
which meant that claimed duplicated jobs could no-op newer ones.
De-duplication should only apply to unclaimed jobs like for the
JobQueueDB class, so that unfinished jobs don't no-op new ones.
* Removed locking in recycleAndDeleteStaleJobs(), which would not do
much since the exclusive set request would serialize on the lua
script anyway. The lua script will finish quickly the next times
if done more than once in a row due to sorted set usage.
Also made recycleAndDeleteStaleJobs() run randomly to reduce the
chance of a single calling tying up the server.
* Removed useless hDel() call in getJobFromUidInternal().
* Changed unit tests to handle the different supported orders better.
Added tests for the 'timestamp' ordering.
Change-Id: Ib2d7aff18753195248ab856afd4a46e18b301db9
* Cleaned up 'server' option to not fragment the pool.
Also made it actually match the documentation.
* Made it use doGetPeriodicTasks() for job recycling.
* Made it so that other job queue classes can be tested.
* Renamed "redisConf" => "redisConfig".
* Tweaked comments about the "random" order option.
Change-Id: I7823d90010e6bc9d581435c3be92830c5ba68480
