With this change, when async uploads are enabled, upload-by-url
will spawn a job and a form with a button to check the status of the
process is shown to the user.
In the process, add processing of warnings in the remote jobs spawned by
the API or the Special page. This is done by adding checks to
UploadJobTrait::verifyUpload. In order to manage warnings serialized in
the job status, a method to unserialize the result of
UploadBase::makeWarningsSerializable.
Things that we might want to fix:
* The form's UI is abysmal, we should probably use Codex
* While it's not a huge deal, I'd like to figure out why I need to
purge the page cache if I want the file to show up. And more
interestingly, why this doesn't happen when uploading via the API
Bug: T295007
Bug: T118887
Change-Id: I49181d93901f064815808380285fc4abae755341
* combine `JobQueueFederated` and `JobQueueRedis` into a single
`JobQueue` channel.
* Remove duplicate descriptions from file blocks in favour of class
doc blocks. This reduces needless duplication and was often
incorrect or outdated, and helps (ironically) to make the file header
more consistently visually ignorable. Various files in this patch
contained bogus copy-pasta descriptions from unrelated classes,
and re-defined `defgroup JobQueue` many times, showing exactly
how this is defacto ignored and counter-productive to maintain
in two places.
Remove `ingroup` from file blocks in class files as otherwise
the file is indexed twice (e.g. in Doxygen) which makes navigation
on doc.wikimedia.org rather messy for classes in this group.
Ref <https://gerrit.wikimedia.org/r/q/message:ingroup+is:merged>
Change-Id: I926a3aec2bc98fefa1075c4a794c46108579ae3f
To this end if 'async' is passed with 'url' to the api:
* Avoid downloading the file synchronously, but verify early
if the upload is allowed by adding a canFetchFile to UploadBase
overridden in UploadFromUrl
* Spawn an UploadFromUrlJob
* When checking for the status of the job, do it fetching the data in
the main stash.
Bug: T295007
Change-Id: If95ccf376cfa9fbe9b3cd058e8e334a6bdd2eb44
Seen in CI for php8 (see T273769 for more information on that test):
PHP Fatal error: Declaration of
PublishStashedFileJob::logJobParams($status) must be compatible with
UploadJobTrait::logJobParams($status): array in
/workspace/src/includes/jobqueue/jobs/PublishStashedFileJob.php on line
52
Follow-Up: I28d0be0239f3b25efc6ea8bc3d400b9d0f5bc7bf
Change-Id: I5933ad025e29cf289e36d53e20254935641311e5
Seen in CI for php8
PHP Fatal error: Declaration of PublishStashedFileJob::getUpload() must
be compatible with UploadJobTrait::getUpload(): UploadBase in
/workspace/src/includes/jobqueue/jobs/PublishStashedFileJob.php on line
68
Follow-Up: I28d0be0239f3b25efc6ea8bc3d400b9d0f5bc7bf
Change-Id: Icb516ac1cfffd9589aa047aa8450c49bc1ab2379
Was getting Fatal error: Declaration of Job::setLastError($error) must be
compatible with UploadJobTrait::setLastError($error): void in
includes/jobqueue/jobs/PublishStashedFileJob.php on line 434
Follow-up 61ed857f9e
Change-Id: I455c7c3f1a1e8ac879d517633be0e347c6db9133
When using WMF style change-propagation, the code processing
allowRetries() may not execute during a timeout resulting in the
job being retried even if it should not be. This is confusing
so add a code comment about it.
Bug: T358939
Change-Id: Iecf7fe452f16413199fee75131274d65a5ccd5e0
To this end, we create a trait to coalesce together all upload methods
common to all upload mechanisms.
Change-Id: I28d0be0239f3b25efc6ea8bc3d400b9d0f5bc7bf
The ChunkedUpload feature stores information about in progress uploads
in the uploadstash table. The AssembleUploadChunksJob reads this data
from the primary DB to prevent race conditions. However the job is
inserted into the job queue before the current transaction commits,
so the race is still present. The job sees the old version, and as
a result drops the final chunk, corrupting the file (Often noticable
by files that have a size which are a multiple of 5mb).
This changes the job push from ->push to ->lazyPush which will make
the job insert happen in a DeferredUpdate after the primary transaction
has commited. Additionally, pass the expected size to the job queue
and assert that UploadStash class has the same filesize as expected.
This will ensure that if this issue happens again, the upload will
abort and it will show up in the logs instead of silently corrupting
files.
Bug: T350917
Change-Id: I37c5c8ffc882026a9fab252b20a10cf4c43d48bc
One possible theory for why chunked upload is unreliable is that
there are multiple jobs racing each other. Add some logging warnings
that should detect such a situation, and abort job if it appears the
job is already in progress
Bug: T200820
Change-Id: Ifaf217bc288dfaa1f7f4ca7e58f8210df232db1b
If we are serious about getting rid of wfGetDB(), we should start
replcing callers with proper replacement.
These classes don't have any injection so it doesn't make sense to
introduce it for this specific usecase.
Bug: T330641
Change-Id: I645f67324d441288c63787c7e42390b59106c585
* Switch out raw Exceptions, mostly for InvalidArgumentExceptions.
* Fake exceptions triggered to give Monolog a backtrace are for
some reason "traditionally" RuntimeExceptions, instead, so we
continue to use that pattern in remaining locations.
* Just entirely give up on PostgresResultWrapper's resource vs. object mess.
* Drop now-unneeded false positive hits.
Change-Id: Id183ab60994cd9c6dc80401d4ce4de0ddf2b3da0
Some less trivial cases. Also update variable names.
This doesn't really change anything, but I hope it helps promote
getConnectionProvider() as the common way to do this.
Follow-up to 8604c384f6.
Change-Id: I6657d783375fac5c7fa856b884ff1fb09285e94c
Found via (?<!IDBAccessObject)::READ_
We are planning to deprecate and remove implementing IDBAccessObject
interface just to use the constants.
Bug: T354194
Change-Id: I89d442fa493b8e5332ce118e5bf13f13b8dd3477
The TitleFactory should really be where TitleArrayFromResult should
live as it's a place where Title objects are created based on certain
specifications.
This patch tries to consolidate the creation of TitleArrayFromResult
via the TitleFactory services and in the future, we will consolidate
more of the logic and kill TitleArrayFromResult.
Follow up on: a99ec1b and e54665d.
Change-Id: I98215af3252d756de4435b77d1280dacda8e85dc
Update cases where one of the IConnectionProvider methods is called
immediately.
This doesn't really change anything, but I hope it helps promote
getConnectionProvider() as the common way to do this.
Follow-up to 8604c384f6.
Change-Id: Id0e7d02bab0c570343c2b1f03c70b44ee39db112
Follows-up I8518e0488 (9c02258a04).
Instead of documenting how to compute the number to manually verify
it, use the expression directly. This should make it significantly
easier to understand, verify, and modify.
Noteworthy:
* Language.php, I kept 31_556_952 as-is because the
calculation would otherwise involve a float. It also has the benefit
of allowing the long durations to build upon that as a given number.
* SqlBlobStore.php, remove this irrelevant default value as it is
unreachable. The only call to new SqlBlobStore is BlobStoreFactory,
which always calls setCacheExpiry. For back-compat and to keep
tests as-is, move to re-used constant between class and config.
Change-Id: I86b034883bd7efdf93b8365b43178af826f1c703
Also, simplify parameter handling when RefreshSecondaryDataUpdate
generates a RefreshLinksJob specification as there is no way for
the user/revision to be null.
Change-Id: I9ede49123e253dbf190b1a08a3ebb317dbabd17d
API modules are high level request handler, lower level code should not
depend on them.
This patch solves the problem only partially, since it leaves references
to ApiUpload in AssembleUploadChunksJob and PublishStashedFileJob. These
jobs were already accessing ApiMain, so while this does not fully resolve
the problem, it reduces it.
Change-Id: I39c9e30cfb2860c573eed8a791f1a292a83cbd76
For readability. Allowed since PHP 7.4.
I searched for integer literals of 6 or more digits, and also changed
some nearby smaller numbers for consistency.
Bug: T353205
Change-Id: I8518e04889ba8fd52e0f9476a74f8e3e1454b678
Make JobSpecification inherit the request ID of the queuing
request just like Job does. This makes it easier to find log
events for actions that are triggered by a web request but happen
via jobs. Especially useful for enqueueable updates which might
or might not happen via jobs depending on the load of the system
at the moment.
Bug: T351729
Change-Id: I965ee211b00b2b33970651368930b5c59704a827
Depends-On: Iccf0f3bf666b9a77fbbd5874fe2f56bfffc9bd4c
Depends-On: I9613c8c293ac3fb4b1ac25c229bb4dc83e2f34fa
This config has been set to false in production since 2018 (Ie4ea1dc0d3927).
This doesn't provide much benefit and its impact yet to be proven and
it's blocking removal of ILoadBalancer::getAnyOpenConnection()
That removal makes $dbwSerial unconditionally false which turned the
whole method into a one-liner and since it was used only once,
we just replaced that.
Bug: T325389
Depends-On: I24f33a15d214d49bfb6c6013e7fd64c7d0eb0086
Change-Id: I19043e775fce9f7ddded128cb2c9395b46dd19a3
