Commit graph

187 commits

Author SHA1 Message Date
Aaron Schulz
75f06d5294 Added $opts to WANObjectCache::set() to detect snapshot lag
* This can avoid some stale write race conditions
* Made use of this option in a few key places
* HOLDOFF_TTL was also bumped

Change-Id: I83505a59cce0119e456191c3100f7d97bc86bdbf
2015-09-28 23:03:36 +00:00
Siebrand Mazeland
cc1285a708 Fix 3 PHPCS warnings in includes/filerepo/
Change-Id: I00ea32f3a0666d225096d6c45710d0de67e2fe14
2015-09-28 13:29:33 +02:00
Aaron Schulz
d02b98b8f3 Updated key WANObjectCache::delete() callers to avoid races
* They now issue the delete() write before COMMIT of
  the relevant DB (or immediately if no trx is active)
* This can avoid some stale write race conditions
* Updated the WAN cache delete() docs

Change-Id: Id54887976051120b76528070d5f2ceb357d57897
2015-09-26 18:02:00 +00:00
jeroendedauw
eb347efa1f Fix type hint in unprefixRow
Change-Id: Iba80cadfdb06b3f9b3f5a6e1d367260405bea1d0
2015-09-24 19:16:27 +02:00
Tim Starling
b4b64c82c8 In LocalFile normalize integer fields to integers
File::getWidth() etc. should return an integer not a numeric string.

Bug: T111089
Change-Id: Ic767fb6cf3db0be9810b7bfa369c209ec05a6e58
2015-09-07 06:50:35 +00:00
Aaron Schulz
f1e1313a16 Support for storing files under SHA-1 names
* Added a "storageLayout" flag to LocalRepo config (supports "sha1")
* Added a simple migration script to copy files the SHA1 paths
* Currently works with img_auth.php + thumb_handler.php for URLs
* Added visibility to some LocalFile methods
* Simple tests for the wrapper class.

Co-Authored-By: Gilles Dubuc <gdubuc@wikimedia.org>
Change-Id: Iad46ad669c8ae3c02d10da10c3f7a16fe161663f
Bug: T1210
2015-07-21 16:19:22 +02:00
Ori Livneh
b31e567b78 hierarchicalize(!) stat names
Graphite expects name components to be dot-separated, so our habit of using
dashes doesn't really make sense. Change metric names to be more compatible
with Graphite, except the job queue's, since that will require a gdash
dashboard definition migration.

Change-Id: I77d0ff7606a8fc88434e4352d23415a9a8f4725a
2015-06-03 16:27:13 -07:00
Aaron Schulz
1404c6ca2d Avoid calling load() in LocalFile::invalidateCache; not needed
Change-Id: I1e2d61722a18e4046bdf7c375ea9c1dbbea4a157
2015-05-21 18:49:02 +00:00
Timo Tijhof
19bff2537e Consistently name the $this callback variable "$that"
Most are already, but some had differnet names. In PHP 5.4+ this is
redundant as they inherit automatically, but we need these for PHP 5.3
compatibility. Settle on "$that" to make these easier to find.

Change-Id: I0b68b2c550fde9c2da53d844421e34b8df0c42ed
2015-05-13 23:05:07 +01:00
Aaron Schulz
1f954a10cd Fixed variable collision in LocalFile::saveToCache()
Bug: T98754
Change-Id: I656b0c415e1b7ec64ee6c15d9a91ebc427663009
2015-05-11 10:20:25 -07:00
jenkins-bot
fac57b4b96 Merge "Converted LocalFile to using the WAN cache" 2015-05-11 16:21:20 +00:00
Aaron Schulz
429a22cd08 Converted LocalFile to using the WAN cache
Bug: T93141
Change-Id: Iecb182428ae0d8e7511bc28340b57f6e7042b723
2015-05-08 11:52:33 -07:00
Brad Jorsch
a90cbb48f8 Cast various things that are supposed to be ints
Mysqli is returning SELECTed ints as strings rather than as ints, I'm
guessing to avoid problems with 64-bit int types on 32-bit systems. PHP
mostly doesn't care, but it causes API JSON output to have strings
instead of ints all over the place.

This also fixes ForeignAPIFile::getUser( 'id' ) returning the user
*name*.

Bug: T98276
Change-Id: Ie6591d72b3ac40172f8176a8ca8b6fad8e9275a5
2015-05-06 11:40:41 -04:00
Aaron Schulz
c518eaf0e0 Removed LocalFile::purgeHistory method
* This just purges an unused key as OldLocalFile::getCacheKey
  simply returns false (rather than a key with "oldfile" in it)

Change-Id: Ic7dda9bf192803a75eaa9bd3bb5a145ad1409dd2
2015-04-27 17:36:02 -07:00
Gergő Tisza
00e925fe92 Force LocalFile::purgeMetadataCache use the master DB
This used to happen automatically via markVolatile() but that was
removed in I808c3a6, which caused cache updates after move/delete
to retain outdated information from slaves.

Bug: T93009
Bug: T88506
Change-Id: Ie4a6c5fbecec9b38ad3ff45fbc22c5d74d66601c
2015-03-18 17:56:11 +00:00
Kunal Mehta
b1d1a7f51a Fix phpcs errors and warnings in includes/filerepo
Change-Id: I8a2b437f9f393baf4ee6546d654d56845fcdd1b6
2015-03-15 02:34:41 +00:00
Aaron Schulz
7782dee83b Remove isVolatile/markVolatile logic from LocalFile
* This is obsoleted by the "latest" flag
* Follows-up 5085a4b

Bug: T88506 
Change-Id: I808c3a6f0edb3466cb600c06c5e742ab030baf14
2015-03-06 21:12:39 +00:00
Aaron Schulz
7508b86ad3 Made the "bypassCache" flag for file locator methods use the master DB
* Added a File::load() stub method
* Cleaned up the File loading flags bitfield a bit

bug: T89184
Change-Id: I1aa4b096c0cad5f5ca34321cc897019005c53a76
2015-03-04 17:02:05 -08:00
Gergő Tisza
bf40d6267c Add File::getDescriptionTouched() method
Fetch page_touched timestamp of file description page via a DB query
(possibly to a foreign DB).

Not sure about the performance implications; could add a memcached
layer on top, but the same DB lookup was already done for local
files.

Bug: T88648
Change-Id: I891c806aab235ff2c0e73c98b06b64fbe78e1517
2015-02-14 00:49:26 +00:00
Ricordisamoa
2ae155da52 Fix phpcs errors in includes/
Mostly Squiz.WhiteSpace.SuperfluousWhitespace.EmptyLines

Change-Id: I678b2f0902f11cd1dfa1611b9da24e7237df9122
2015-01-08 20:15:07 +01:00
Aaron Schulz
4ff8136807 Removed remaining profile calls
Change-Id: I31c81c78715048004fc8fca0f27d09c1fa71c118
2015-01-08 02:49:33 -08:00
Chad Horohoe
aa21e125a3 Remove obvious function-level profiling
Xhprof generates this data now. Custom profiling of various
sub-function units are kept.

Calls to profiler represented about 3% of page execution
time on Special:BlankPage (1.5% in/out); after this change
it's down to about 0.98% of page execution time.

Change-Id: Id9a1dc9d8f80bbd52e42226b724a1e1213d07af7
2015-01-07 11:14:24 -08:00
umherirrender
d2a0367914 Use content language for edit summary on upload overwrite
The edit summary for the null revision on upload overwrite was in the
user language and not in content language

Possible a regression from Idf281898d8a5a023a0b9ce3bc90b3ca55c1a6376

Bug: T72293
Change-Id: Ie70a3874e170b3849e91265e0c9c61da838f2d20
2014-12-28 14:42:14 +01:00
jenkins-bot
8eb1344fba Merge "Removed some unnecessary code in LocalFileDeleteBatch" 2014-12-12 21:18:30 +00:00
Ricordisamoa
fc5fd5c37a Typo fixes and non-code tweaks
Skipped replacements:
* prefered → preferred
* prolly → probably

Skipped files:
* resources/lib/jquery.ui/jquery.ui.mouse.js
* resources/lib/jquery/jquery.form.js

Change-Id: Ib7923f362ddfca1b892bf5d601785d6b5aa5d44c
2014-12-12 18:31:15 +00:00
Aaron Schulz
e369f66d00 Replace wfRunHooks calls with direct Hooks::run calls
* This avoids the overhead of an extra function call

Change-Id: I8ee996f237fd111873ab51965bded3d91e61e4dd
2014-12-10 12:26:59 -08:00
Aaron Schulz
4269e65ff9 Removed some unnecessary code in LocalFileDeleteBatch
Change-Id: I4d03d03fada5c299779b7fb7e998285568672a6c
2014-11-27 12:50:41 -08:00
Aaron Schulz
17cc78767a Switch to LOCK IN SHARE MODE in recordUpload2()
* We just need the latest data; the advisory locking in lock()
  already manages mutual exclusion.

Change-Id: I9af7a892b2dcafd30149332b0084f61d117c9cad
2014-10-29 06:59:51 +00:00
Aaron Schulz
745637c3da Reduced LocalFileRestoreBatch::execute deadlocks when doing batch restores
* Often users restore a batch of many files with similar names.
  This fails due to gap locking on non-existing rows which tangles up
  unrelated actions. Avoid that locking for the common case.
* This changes the return value of LocalFile::lock() from a useless
  one (due to the exception) to one that conveys some useful information.
  The method is fairly internal in any case.

Change-Id: Idb86367cf45b731d4b2a67b1813a660bc0e7a84f
2014-10-17 04:07:54 +00:00
Aaron Schulz
a578bd195b Made LocalFileMoveBatch::addOlds use SH locks instead of EX ones
Change-Id: I928f98deded6e71b4b1e63167ca8b565e39c6394
2014-10-09 22:50:26 +00:00
jenkins-bot
396927c0b6 Merge "Made LocalFile move/delete/restore handle network partitions better" 2014-08-21 15:38:10 +00:00
umherirrender
3b2b6a2773 Add missing @param to function docs
Change-Id: I47fa96a976f55a1a93cb75397285edb8c7f4cd8a
2014-08-14 20:22:52 +02:00
Aaron Schulz
8dca7488ec Made LocalFile move/delete/restore handle network partitions better
* Previously, the file existence checks would not distinguish an answer
  in the negative from a non-answer. This was a long-standing problem.
  This avoids moving DB entries without moving the files (unless the
  partition happens in the middle of the moves of course).
* Optimized fileExistsBatch() to do concurrent stats if possible.

bug: 40927
bug: 69312
Change-Id: I9132fd5591bb7a3d5852f17514dcf51a3d8b7812
2014-08-14 17:26:57 +00:00
rillke
c31fbf073e Unify the spelling of MIME in documentation
Writing MIME as written in Wikipedia and some documentation clean up.

Change-Id: I9dfc36d2bf55d72d9374c4075bd6d45eef0415a4
2014-08-07 23:38:45 +02:00
umherirrender
b883e8c7c0 Cleanup some docs (includes/[e-l])
- Swap "$variable type" to "type $variable"
- Added missing types
- Fixed spacing inside docs
- Makes beginning of @param/@return/@var/@throws in capital
- Changed some types to match the more common spelling

Change-Id: I41a84e8e1dec39170aa655250325ffc485eaeaef
2014-07-24 19:43:03 +02:00
umherirrender
d5d37cd034 Add missing rollback to LocalFile::recordUpload2
When begin was called, all returns should have rollback called to
close the transaction.

Change-Id: I8197f73fb45c2e2900db6d5e588623f9b0c46ea5
2014-07-17 10:41:18 +02:00
Aaron Schulz
c87c9c14da Added missing unlock() calls in LocalFile
Change-Id: Id427fb424bfb46670f2074664aca750014c70225
2014-07-16 17:14:23 -07:00
jenkins-bot
6f38f78cdd Merge "Fixed possible race condition in LocalFileMoveBatch" 2014-05-22 04:56:48 +00:00
Aaron Schulz
fc6d99cb20 Tweaked timestamp kludge logic in recordUpload2
* This now only does the SELECT FOR UPDATE for the re-upload case

Change-Id: I21b1b0328b6dbfb30f4f0293212515b9ee081778
2014-05-20 14:51:16 -07:00
Aaron Schulz
399198c28e Replace FOR UPDATE with LockManager use in LocalFile::lock()
* This avoids excess contention where inserts of rows for
  similarly named files get blocked. This often would effect
  users of UploadWizard or any bot doing mass uploads.

Change-Id: Ie7a328f7d4f03aa249770804417347a50356ea42
2014-05-20 14:22:52 -07:00
Aaron Schulz
51c173912a Fixed possible race condition in LocalFileMoveBatch
* If a move is blocked by a re-upload and the TRX started outside of
  LocalFile::lock, then the move did not see the new row in the oldimage
  table and so it will not move that version of the file in storage. Use
  the latest DB data to avoid that.

Change-Id: I7ec46941170293c96b99a449cb396517108e8b3d
2014-05-19 15:18:05 -07:00
Aaron Schulz
582fd02fb9 Use FOR UPDATE in prior LocalFile timestamp check
* This covers the case with a snapshotted transaction outside of lock()/unlock()

Change-Id: I8639c2e51cb918f452dc1576681d6224fb4fb2c0
2014-05-14 09:43:02 -07:00
Aaron Schulz
605329976f Made LocalFile avoid duplicate (name,timestamp) pairs
* Various bits of code seem to assume uniqueness already

bug: 65264
Change-Id: Ib00fdbe87a79296c640cd69d74928236ce7a4aee
2014-05-13 15:03:32 -07:00
Siebrand Mazeland
4b8523198f Make phpcs-strict pass on includes/ (4/~10)
Change-Id: I2767205a7059adfb8fa090a82abd332e518f2590
2014-05-11 19:35:09 +00:00
umherirrender
2d11016ecd Pass $user to LocalFileDeleteBatch
This avoids the use of $wgUser

Change-Id: I60cab27e0708a818c77791cc89194b6dd726da0b
2014-05-06 21:31:29 +02:00
Jeff
8c62609dc8 PostgreSQL: decode image.img_metadata
The img_metadata field was not being decoded from bytea into text.

This change to core fixes the reported problem with PdfHandler against
1.22.6, origin/REL1_23 and git head.

Bug: 59147
Change-Id: I285a317a57a29713af1f7f630b340278bd5400da
2014-05-05 16:02:55 -07:00
umherirrender
3cca294d75 Avoid raw sql in LocalFileDeleteBatch::doDBInserts
Use Database::conditional instead of raw sql.

Change-Id: Ic00146e9ec582a263b9dec117b12532fdc24789e
2014-05-05 21:28:32 +02:00
Aaron Schulz
250b23aa01 Removed commented-out migrateThumbFile() code
Change-Id: Ic52d3a8d93808b365c3f06fef9e51d65efcd1d57
2014-04-30 00:30:37 -07:00
umherirrender
26e705f808 Add $user param to Revision::newNullRevision
This avoids the use of $wgUser in Revision constructor and makes the
dependency on the global visible.

Change-Id: Ib67bd706a3c4ef081f475406e9aa1094c42222ef
2014-04-28 18:59:20 +02:00
Ori Livneh
776d0900b9 Simplify LocalFile::isVolatile
Make the logic a bit easier to follow. Follow-up to I99a39c3d2.

Change-Id: I9b815d7f0d229ef25db5b7f48f6e49ca4dac046e
2014-04-24 15:57:09 -07:00