UploadBase::performUpload() now takes a $tags parameter and passes it
to LocalFile::upload() and LocalFile::recordUpload2(), which
eventually adds the requested tags to the log_id, rev_id and rc_id
that are created for the file upload.
Previously you'd have to query the database for the latest rev_id and
log_id for the page title under which the title is being uploaded, as
performUpload() is unable to return them to you because it's all
deferred in funny ways.
Bug: T121874
Change-Id: I99a8fd67c84219d2715d3d88cc21500614431179
For file patrolling, we rely on the timestamp of image revision
(`image` table) and log entry (`logging` table) being the same, but we
weren't actually making sure it is the same.
Note that the timestamp of page revision can still differ, but that
shouldn't matter and it's a more involved fix.
Change-Id: I8e1b75d744c4a96cb0bb636b153d6e8ea2ee1fcf
The implementation from 0160b410bd only
stored the associated rev id in the `recentchanges` table, which was
sufficient for its purpose of allowing patrol of these log entries and
provided some change tagging improvements, but ephemeral.
Now we also store it in `logging` and `log_search` tables, which makes
it possible to find a revision for the log entry and vice versa.
* For `log_search` we just add a row with 'associated_rev_id' as key,
it was designed for exactly this purpose.
* For `logging` we store it in log_params field in ManualLogEntry, and
transparently remove it from there when building a DatabaseLogEntry.
Bug: T122089
Change-Id: I559587ff77b3c94b31a6c5951287b6f9c1e167ce
This allows to patrol file uploads, both new files and new file
versions, from the description page, provided $wgUseFilePatrol
is set to true. Special:NewFiles can be filtered to hide patrolled
files.
Bug: T11501
Change-Id: If71af58719a4461f12d125455b7bef07164525ca
* Using addUpdate() makes sure purges are coalesced and
de-duplicated.
* Also removed incosistent $wgUseSquid checks. If CDN caching
is not used, then $wgSquidServers will just be empty anyway.
Bug: T119016
Change-Id: I8b448366f037f668385d252f9d68289b71d1a707
Use deferred updates and transaction hooks
to get the cache updates after the publish()
DB updates instead
Change-Id: I0a922182f7dd5f22c1e661171b0df1a49a637aef
* This gets lag information that is useful when
the calling code is about to run queries that
will have their results cached.
* This is now used in place of trxTimestamp() for
WANObjectCache set() and getWithSetCallback().
* The WAN cache will use a low TTL if the lag is
too high to avoid caching stale data for weeks.
* Bumped MAX_COMMIT_DELAY as nothing enforces it.
Bug: T113204
Change-Id: I2a95b4088cff42d6e980351555f81a4b13519e24
* Use "bypassReadOnly" as other file backend maintenance
scripts do. One may want to run this while site traffic
is off via $wgReadOnly.
* Fixed handling for when img_sha1 is not set.
* Fixed some IDEA errors.
Change-Id: I95c426c5f2082576fc9ea40282d2869750a9f3d8
* This can avoid some stale write race conditions
* Made use of this option in a few key places
* HOLDOFF_TTL was also bumped
Change-Id: I83505a59cce0119e456191c3100f7d97bc86bdbf
* They now issue the delete() write before COMMIT of
the relevant DB (or immediately if no trx is active)
* This can avoid some stale write race conditions
* Updated the WAN cache delete() docs
Change-Id: Id54887976051120b76528070d5f2ceb357d57897
* Added a "storageLayout" flag to LocalRepo config (supports "sha1")
* Added a simple migration script to copy files the SHA1 paths
* Currently works with img_auth.php + thumb_handler.php for URLs
* Added visibility to some LocalFile methods
* Simple tests for the wrapper class.
Co-Authored-By: Gilles Dubuc <gdubuc@wikimedia.org>
Change-Id: Iad46ad669c8ae3c02d10da10c3f7a16fe161663f
Bug: T1210
Graphite expects name components to be dot-separated, so our habit of using
dashes doesn't really make sense. Change metric names to be more compatible
with Graphite, except the job queue's, since that will require a gdash
dashboard definition migration.
Change-Id: I77d0ff7606a8fc88434e4352d23415a9a8f4725a
Most are already, but some had differnet names. In PHP 5.4+ this is
redundant as they inherit automatically, but we need these for PHP 5.3
compatibility. Settle on "$that" to make these easier to find.
Change-Id: I0b68b2c550fde9c2da53d844421e34b8df0c42ed
Mysqli is returning SELECTed ints as strings rather than as ints, I'm
guessing to avoid problems with 64-bit int types on 32-bit systems. PHP
mostly doesn't care, but it causes API JSON output to have strings
instead of ints all over the place.
This also fixes ForeignAPIFile::getUser( 'id' ) returning the user
*name*.
Bug: T98276
Change-Id: Ie6591d72b3ac40172f8176a8ca8b6fad8e9275a5
* This just purges an unused key as OldLocalFile::getCacheKey
simply returns false (rather than a key with "oldfile" in it)
Change-Id: Ic7dda9bf192803a75eaa9bd3bb5a145ad1409dd2
This used to happen automatically via markVolatile() but that was
removed in I808c3a6, which caused cache updates after move/delete
to retain outdated information from slaves.
Bug: T93009
Bug: T88506
Change-Id: Ie4a6c5fbecec9b38ad3ff45fbc22c5d74d66601c
Fetch page_touched timestamp of file description page via a DB query
(possibly to a foreign DB).
Not sure about the performance implications; could add a memcached
layer on top, but the same DB lookup was already done for local
files.
Bug: T88648
Change-Id: I891c806aab235ff2c0e73c98b06b64fbe78e1517
Xhprof generates this data now. Custom profiling of various
sub-function units are kept.
Calls to profiler represented about 3% of page execution
time on Special:BlankPage (1.5% in/out); after this change
it's down to about 0.98% of page execution time.
Change-Id: Id9a1dc9d8f80bbd52e42226b724a1e1213d07af7
The edit summary for the null revision on upload overwrite was in the
user language and not in content language
Possible a regression from Idf281898d8a5a023a0b9ce3bc90b3ca55c1a6376
Bug: T72293
Change-Id: Ie70a3874e170b3849e91265e0c9c61da838f2d20
* Often users restore a batch of many files with similar names.
This fails due to gap locking on non-existing rows which tangles up
unrelated actions. Avoid that locking for the common case.
* This changes the return value of LocalFile::lock() from a useless
one (due to the exception) to one that conveys some useful information.
The method is fairly internal in any case.
Change-Id: Idb86367cf45b731d4b2a67b1813a660bc0e7a84f
* Previously, the file existence checks would not distinguish an answer
in the negative from a non-answer. This was a long-standing problem.
This avoids moving DB entries without moving the files (unless the
partition happens in the middle of the moves of course).
* Optimized fileExistsBatch() to do concurrent stats if possible.
bug: 40927
bug: 69312
Change-Id: I9132fd5591bb7a3d5852f17514dcf51a3d8b7812
- Swap "$variable type" to "type $variable"
- Added missing types
- Fixed spacing inside docs
- Makes beginning of @param/@return/@var/@throws in capital
- Changed some types to match the more common spelling
Change-Id: I41a84e8e1dec39170aa655250325ffc485eaeaef
* This avoids excess contention where inserts of rows for
similarly named files get blocked. This often would effect
users of UploadWizard or any bot doing mass uploads.
Change-Id: Ie7a328f7d4f03aa249770804417347a50356ea42
* If a move is blocked by a re-upload and the TRX started outside of
LocalFile::lock, then the move did not see the new row in the oldimage
table and so it will not move that version of the file in storage. Use
the latest DB data to avoid that.
Change-Id: I7ec46941170293c96b99a449cb396517108e8b3d
The img_metadata field was not being decoded from bytea into text.
This change to core fixes the reported problem with PdfHandler against
1.22.6, origin/REL1_23 and git head.
Bug: 59147
Change-Id: I285a317a57a29713af1f7f630b340278bd5400da
This avoids the use of $wgUser in Revision constructor and makes the
dependency on the global visible.
Change-Id: Ib67bd706a3c4ef081f475406e9aa1094c42222ef
These files have all had treatment before, and these occurrences have either
been missed or have been introduced after.
Change-Id: I06cdab4616b5bff47c85152df28f18c861730a23
* Previously LocalFile cache misses would hit the master DB
(even for shared repos like Commons).
* Since lock() is always called before file changes, this has
been modified to call a new markVolatile() method to track
changing (or recently changed) files in memcached. On cache
miss this tracking is used to decide whether to use the
slave or the master DB.
Change-Id: I942f5e28dd9e0953b6382cc2247ca480494d5718
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Also added some missing @param.
Change-Id: I8804ebe0922d3a414863b162a2110e0b9e49b80f
* If the publish() call did not really do anything, then there is
no use in saving the page and file DB records. Previously the
user would get an error but the description page would be made.
bug: 64071
Change-Id: I1959f8c1ef33dbd1b238f6020117d2862d4b0fd3
Previously if an exception was thrown when creating the description
page, the old (negative) memcached entry would not be purged, and
the file would appear not to exist until 24 hours later.
This way, as soon as we have the image row committed to db, we
change the memcache entry, so memcache of image row always looks
like actual image row.
Bug: 32551
Change-Id: I9c260e9a4a1e0d6ade1771542dd834f969904cef
* Lowered the lock() timeout
* Wrap the destination DB write with memcached locks for move
ops since it is easier to control that lock timeout than the
DB lock timeouts.
* Cleanup locks in the destructor (e.g. on exceptions). Other
process can pile up more if locks have to expire.
bug: 63058
Change-Id: I38b28d81ec96daa80ece2354db284a614289ba0b
Due to bug 17259 (which is fixed in a better way in Id44d566a),
imagelinks only records the redirect and not the actual image used when
an image redirect is referenced. This causes various problems, such as
cascading protection not working through image redirects.
It makes more sense for imagelinks to work like tempaltelinks, recording
both so that things like cascading protection don't have to care about
image redirects explicitly.
Comparing imagelinks to templatelinks also reveals a few places
(WikiPage::doDeleteUpdates, WikiPage::doCascadeProtectionUpdates) that
should be triggering a LinksUpdate if the image links changed.
Bug: 23002
Bug: 23542
Bug: 26503
Change-Id: I64fe7d25646cae2c8213211893c6f821f3504dbf
- Removed double spaces
- Added space after if/switch/foreach
- Removed space on elseif
- Added space around parentheses
- Added newline at end of file
- Removed space before semicolon at end of line
Change-Id: Id40b87e04786c6111e6686d7f7eea1e588bdf37d
Uploading a file with backslashes in the metadata
causes database exception on PostgreSQL, because
backslashes are sent to the 'bytea' datatype unencoded.
Bug: 52017
Change-Id: Ife56cffe3df83ec1a7bd39cdc4a0489f23354494
I want to get the file description in a language other then
the user language in an extension (I5e6bc45f9751).
Change-Id: Ifcae821a51f4207e7816e710d3b3857c7ed438b6
MediaHandler::getHandler() can return false when a handler cannot be
determined for the given file's derived mime type. This change adds
guards to invocations that I could find that did not properly check for
this potential return result.
Bug: 53820
Change-Id: I8c0165311cc75f9920ac30ce2b38ccd207439198
Inform upstream caches when a file asset is deleted by calling
SquidUpdate::purge(). This helps prevent unintentional information leakage in
the event that the asset is being deleted for reasons of license or content.
Bug: 51064
Change-Id: Ibf57e8f32d7dd04b8508a69706be5a40b49e5abd
This tends to become extremely expensive as the number of
oldversions of a file increase. It is also not generally
needed since ?action=purge is usually targeting the
current version (additionally old versions of the file
have fixed urls, so they're less likely to get out
of sync). If an old version does need to be purged, one
can revdel and un-revdel it.
It should be noted that this extra purging was added
for bug 30192. However, since that bug was fixed,
most of the places requiring purging of old thumbnails
have now done it directly instead of by relying on
$file->purgeCache. The exception being revision
deletion, which still assumes $file->purgeCache clears
the thumbnails of old versions of the file.
There's no real bug for this, but I kind of hijacked
bug 49362
Change-Id: Ib399132cabe79fd2b4b23bad5708bfa50b282074
getPageDimensions returning false for failure wasn't being handled
properly, causing ugly output. The doc comment on that method was
wrong as well.
Most notably causing random whitespace output:
https://commons.wikimedia.org/wiki/?curid=22151015
(screenshot: http://i.imgur.com/c21EpVx.png).
Bug: 41281
Change-Id: I1a49474309e15808928f877dfc29ae366d028928
Primary motivation here is to make Special:Recentchangelinked work
correctly for uploads where the description page doesn't exist yet.
(Previously it was mostly only showing reupload log entries)
Mild downside to this approach is adding the recentchanges row is
now in a different commit...
Change-Id: I74c68b972c47c03066d3065e3748769a27a654fb
When someone uploads a new file, we should refresh all the pages
that use that image, so that [[Category:Pages with broken file links]]
gets removed.
Future fixme would to run this again on file deletes, but that's
less critical as they will eventually get added on next reparse
and people don't expect that to happen instantly like when fixing
the issue. Ditto for someone creating a new file redirect.
Bug: 46749
Change-Id: I7889598eee54a1c07c64f750b8499e922ba59704
I came across people complaining that it was hard to associate
upload log events to actual images since img_timestamp could
be different from log_timestamp, and generally no unique id.
Well I was there I made uploads use the new logging system.
Change-Id: Icd8662ecb9eb0f6c0ff9841bdbd5736d6dd0d015
And added/removed spaces around some other tokens,
like +, -, *, /, <, >, =, !
Fixed windows newline style
Change-Id: I0b9c8c408f3f6bfc0d685a074d7ec468fb848fc8
Doxygen expects parameter types to come before the
parameter name in @param tags. Used a quick regex
to switch everything around where possible. This
only fixes cases where a primitve variable (or a
primitive followed by other types) is the variable
type. Other cases will need to be fixed manually.
Change-Id: Ic59fd20856eb0489d70f3469a56ebce0efb3db13
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments
Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
Moves the giving of a user one level up, because File::recordUpload2
already needs this (and fallback to $wgUser).
Can also use the user for the watching of the file, which was using
$wgUser before.
Change-Id: I697ae0df65b07ea59ab11b62804853cdc03cb172
* Only BEGIN on lock() if no trx was in progress.
Likewise, only COMMIT in unlock() if the lock() call started a trx.
* This can avoid problems with commiting page update transactions
prematurely, which could leave broken page stub rows. (bug 40178)
Change-Id: I9a0adb25ee107df9a6bf70c6103ddfb7f034be25
* Previously, large values that could not fit into memcached would not be cached
and the master DB would be hit for any file loading (djvu files hit this often).
Change-Id: I3ed8ad4a85d6e3084330b56c3b48ee76103bd2b8
While we don't want to re-encode videos on purge, they should still
be removed from squid; compute the list of all urls before extensions
can filter it to remove files that should not be deleted from the
file backend.
Change-Id: Id837843b5a203e49a8594439ce3620da9b3c7b6a
reverting a file a virtual url is passed to upload without props,
get props from backend, virutal urls are not local files.
Change-Id: I1d56a6237e1d2af9941f29e4ba3f0262817b0e8d
* For backends that support it, custom HTTP headers can be set on files.
* Added a getStreamHeaders() function to MediaHandler to let subclasses
recommend header name/value pairs to be used for responses to GET/HEAD
requests. For example, an OGG handler could set "X-Content-Duration".
* Made LocalFile use this function to set HTTP headers of new uploads.
Change-Id: I1b017e1342513f0097fe6d142aae18e819403293
This allows sha1 searches with the api in miser mode for deleted files
Added script to populate the rows
Adding new field to selects and handle it in all places, where needed
Using a 10 byte index for the new field per
http://lists.wikimedia.org/pipermail/wikitech-l/2012-September/063429.html
Change-Id: Ie54a513fe361202e63df44be44a0fdd91926c974
The transaction bracket in LocalFile::recordUpload2 used to span a call
to WikiPage::doEdit, which in turn opens a transaction. Nesting transactions
this way does not work: the first transaction is committed prematurely when
the second one starts. This may cause serious database corruption and
generally exciting behavior.
This change commits LocalFile's own transaction before any interaction
with WikiPage. There may be a race condition here, but that case seems to
be already handled in the code. Also, in the previous "broken" state, all
transactional protection got lost anyway - so this should at least be no
worse than what we had.
This may be changed back if and when we have support for nested
transactions.
Change-Id: I20d90fedb2d19c64ccf0f3942ddda756fe511c12
This change updates the log_page in a second step after performing
the edit, which is not terribly elegant, but is very robust.
Change-Id: I9015230df772d049b2879fb34914c68183f1da05
* Replaced WikiPage::DATA_FROM_* constants with IDBAccessObject ones.
* Renamed IDBAccessObject constants a bit for visual consistency.
* Removed AVOID_MASTER parameter and replaced calling instances with READ_NORMAL.
Instead of getting page_latest from the master and the revision from a
slave, just get it all from the master in one RTT. Most callers used
AVOID_MASTER (and now READ_NORMAL), so this case is barely hit anymore.
Change-Id: Ifbefdcd4490094b38e49bbb46c95fdb71b5c9e1a
