Commit graph

61 commits

Author SHA1 Message Date
Aaron Schulz
c2acd8f642 Simplify and clean up FileBackend exceptions
Use standard exceptions for unexpected errors and remove
FileBackendException class, leaving FileBackendError. The
later is actually intended to be caught in some cases.

Change-Id: I735a525e0b14e518b2da5f18762e0f293064dfc2
2016-09-22 15:28:08 -07:00
Aaron Schulz
c64c3ef9a8 Make Status extend StatusValue and start FileBackend update
* This avoids code duplication.
* Callers can safely start type-hinting StatusValue as well.
* Also moved the wrap() logic out of Status::__construct(), which
  only wrap() was (and should have been) using. Use "static" as
  well, so subclass behave properly.
* The docs and type-hints in /filebackend are updated as an example.
* A migration pattern is to inject a StatusValue wrapper
  into the backend and use it on all returned statuses, so MediaWiki
  will still get Status for message methods.

Change-Id: Iff9255f34870ea6b0c4b91f6ddc69eea95186aba
2016-09-17 15:45:40 -07:00
Aaron Schulz
7ed633595d Log when resyncFiles() fails in FileBackendMultiWrite
This distinguishes cases that recover from sync errors
from those that do not.

Change-Id: Ibcdea18041e7344077c8308359a1a6087139f2e0
2016-04-29 12:16:13 -07:00
Aaron Schulz
48fc04e656 Optimize consistencyCheck() in FileBackendMultiwrite
doOperationsInternal() already set "preserveCache" but never actually
filled it in to begin with. This should lower round trips if sync checks
are enabled.

Change-Id: Ica67b8d66e7602faed842408365edbd466688f61
2016-04-27 05:18:25 +00:00
Aaron Schulz
8cec086be0 filebackend: Make resyncFiles() easier to use
Pass in the resync mode as a flag instead of using
the object instance.

Change-Id: I2da661a65afd712403db936b6e31fba8e8069096
2016-04-12 21:17:51 +00:00
Aaron Schulz
a14ef91251 Add FileOperationReplication log to improve multiwrite backend logging
Change-Id: I541eac740f4a07182067294c9597cdd49330e86c
2016-03-07 13:55:27 -08:00
Aaron Schulz
a5d903860a Allow FSFile objects for src in FileBackend::do*Operations
Convenience aside, this lets multiwrite backends do async replication for
"store" operations safely, buy keeping a handle to the source file that
prevents it from getting prematurely deleted before the post-send writes
to the secondary backends can happen.

Bug: T91869
Change-Id: I1254de527c47835c35fed6e526b42953c1b2b2ca
2016-03-02 18:49:56 +00:00
Aaron Schulz
c0446571d5 Use sync writes for replication of 'store' operations
The local temporary file might not live long enough to be
copied into storage during post-send.

Bug: T128124
Change-Id: Ifb5260958db008ec8b9f4db90f95a42e5ecaeadc
2016-02-27 00:20:13 +00:00
Kunal Mehta
6e9b4f0e9c Convert all array() syntax to []
Per wikitech-l consensus:
 https://lists.wikimedia.org/pipermail/wikitech-l/2016-February/084821.html

Notes:
* Disabled CallTimePassByReference due to false positives (T127163)

Change-Id: I2c8ce713ce6600a0bb7bf67537c87044c7a45c4b
2016-02-17 01:33:00 -08:00
Aaron Schulz
169b7b98b5 Added replication=async option to FileBackendMultiWrite
* This will defer writes to non-master backends till the
  end up the web request. This is useful for multi-DC setups.

Bug: T112708
Change-Id: I118c07764dd4a4f4f2590d4548238df12860e750
2015-10-01 23:30:41 +00:00
jenkins-bot
b3fdef18a1 Merge "Added readAffinity flag to FileBackendMultiWrite" 2015-09-21 12:20:45 +00:00
jenkins-bot
d9859b32cd Merge "Removed old "noPush" config hacks from FileBackendMultiWrite" 2015-09-21 09:40:43 +00:00
jenkins-bot
de82ab7c24 Merge "Prevent IDEA error in doOperationsInternal()" 2015-09-21 09:31:01 +00:00
jenkins-bot
8d13ffae12 Merge "Made some FileBackendMultiWrite docs more accurate" 2015-09-21 09:29:24 +00:00
Aaron Schulz
fb0994d5c5 Prevent IDEA error in doOperationsInternal()
Change-Id: I2f0a1885803fff9dc0eb8ca4a595a83736ce421b
2015-09-20 23:04:29 -07:00
Aaron Schulz
bd6ec3bc78 Added readAffinity flag to FileBackendMultiWrite
* A backend with this set (normally 1) will be used for
  non-latest reads. This can be used to prefer a local,
  replicated, backend instead of one farther away (for
  multi-DC setups)
* Note that listings still come from the master always.

Bug: T112708
Change-Id: Ic4bf4ba5a2c9ef78abd11dbd0d4b48c73cad6923
2015-09-20 19:25:43 +00:00
Aaron Schulz
49e9910d0f Made some FileBackendMultiWrite docs more accurate
Change-Id: I14c152601fef3d3c75e7ea7aac72ff87c9f5fd52
2015-09-19 21:11:53 -07:00
Aaron Schulz
92b9ee98a1 Removed old "noPush" config hacks from FileBackendMultiWrite
Change-Id: I80e688cc61c0a656524a0db6aeabb3d3f6e667bd
2015-09-18 17:36:09 -07:00
Aaron Schulz
5cea7769d3 Cleaned up getScopedFileLocks() return value
* Just return the ScopedLock itself without wrappig it in an array.
  This also makes getScopedLocksForOps() actually return the correct
  type according to the docs

Change-Id: Ic2d01d56c3fcb22af3adb5e5d4d1586d62316552
2015-09-18 17:58:11 +00:00
Bartosz Dziewoński
1021f20100 Fix instances of preg_quote() without second parameter
This might work in specific cases, depending on the string we're
quoting, but is never correct. The instance in SpecialAllMessages.php
was resulting in incorrect behavior (and log spam, T103879).

Searched for this regex: preg_quote\((?:\([^)]+?\)|[^,()])+?\)

Change-Id: Icc88775970f4927ddf953e5ad7c5c19d59491573
2015-06-25 19:43:56 +02:00
umherirrender
6b24d10b6d Fix some line indent of string concat
Change-Id: I5883c9cd57d106168648b4502051b4b49046fea2
2015-06-19 20:52:43 +02:00
Ricordisamoa
fc5fd5c37a Typo fixes and non-code tweaks
Skipped replacements:
* prefered → preferred
* prolly → probably

Skipped files:
* resources/lib/jquery.ui/jquery.ui.mouse.js
* resources/lib/jquery/jquery.form.js

Change-Id: Ib7923f362ddfca1b892bf5d601785d6b5aa5d44c
2014-12-12 18:31:15 +00:00
Gilles Dubuc
a6f2310cc7 filebackend: Log more details about backend-fail-internal errors
Bug: 73094
Change-Id: I9e45284e3cfd55bc34d87009387a2c1766e2db4a
Mingle: https://wikimedia.mingle.thoughtworks.com/projects/multimedia/cards/991
2014-11-17 23:36:47 +00:00
Aaron Schulz
2cfce9fd8d Bail out on FileBackend operations if the initial stat calls failed
* The preloadFileStat() call acts as a good canary for whether all
  affected servers are up. If anything failed in it, then it's not
  worth bothering to send the actual write requests.
* Also made FileBackend::preloadFileStat abstract while at it since
  the two subclasses implemented it and so should any others.
* Fixed a silly comment typo.

Change-Id: I5bd427f654aa4a9d6bfe4ed7566276e8ac520b30
2014-04-18 12:41:23 -07:00
Aaron Schulz
8aaab01e1d Made preloadFileStat/preloadCache actually work for FileBackendMultiWrite
* Previously the path substitution did not take place

Change-Id: I58ad3e2aa44a2b08f4c6bad1a783fb92ea79dbf4
2014-04-12 15:42:39 -07:00
Aaron Schulz
6b1a60a1ca Added concurrent HEAD request support for Swift
* This adds a new preloadFileStat() method to FileBackend.
* FileBackend::doOperations() makes use of this instead of primeFileCache().
  The later was mostly useless due to the 'latest' flag, which stat entries
  rarely had (and since they use add(), 'latest' ones do not override the
  non-'latest' ones).
* Simplified primeContainerCache/primeFileCache to no longer accept
  FileOp objects and made sure nothing passes those in anymore.

Change-Id: Iee3878cadd69c3a594c701c0aff81aab5f473de5
2014-02-20 08:50:08 +00:00
Siebrand Mazeland
f077c4b1d6 Update formatting
Change-Id: I18aff576262479c9bb1c56eb8e1d1aaae200e4b1
2014-02-06 09:27:05 +01:00
Aaron Schulz
ac14c1c8a6 filebackend: Added supported for retrieving file metadata/headers
* This can be useful for carrying over metadata when copying files around
* Also fixed a bug in sanitizeHdrs() for Swift (broken content-disposition)

Change-Id: I4534e9acac2b306086797b3677f85c05b98e39fc
2014-01-08 13:22:11 -08:00
jenkins-bot
76dc198651 Merge "filebackend: cleaned up the FileBackend constructor" 2013-12-10 22:09:21 +00:00
Aaron Schulz
a0878fe2b4 filebackend: exception handling cleanups
* Split out FileBackendException class and reduced direct use of MWException

Change-Id: I325c1798b6d90972c12a5dccc37989af34d857f3
2013-12-07 23:22:07 -07:00
Aaron Schulz
2aa84651ca filebackend: cleaned up the FileBackend constructor
* Added some b/c code with deprecation warnings

Change-Id: Ifceffbc0a37a223bcd7cd3dc60181fc85765bc46
2013-12-03 13:57:01 -08:00
Aaron Schulz
2f29c7551b Revert "filebackend: cleaned up the FileBackend constructor"
Actually this messes with the implicit backend made for things like Math (when unconfigured), which uses the "new" operator.

This reverts commit 1f129a22cb.

Change-Id: I4c72c4f7c8b82e38df5496cf2b90fc9e19c40334
2013-12-03 20:40:24 +00:00
Aaron Schulz
1f129a22cb filebackend: cleaned up the FileBackend constructor
* Moved some of the graph construction work to FileBackendGroup.
  This helps the code in not depending on the rest of MW so much.
* Updated tests and FileBackendMultiwrite, which are the only things
  directly constructing FileBackend objects.

Change-Id: I188a053c70ce088ce34613d5db40e6708e3ea9b7
2013-11-23 20:21:53 -08:00
Siebrand Mazeland
4b985bd667 Add missing type hint for $autoSync
Change-Id: I63deed0b2b9b6fc29f67c0b522987df156f685fa
2013-11-23 20:30:39 +01:00
Siebrand Mazeland
34a4bcbd48 Update documentation of file backend classes
Change-Id: I2113aecc53f1b361d9c276113bc0a9f7076cd21c
2013-11-23 19:24:43 +01:00
Siebrand Mazeland
a5c35215e2 Update formatting of file backend classes
Change-Id: Ie69c324e2f1ea0b8a654adee06ebda494cebd10c
2013-11-22 22:17:15 +01:00
Aaron Schulz
ef2f2610ab filebackend: try to combine SH and EX lock acquisition
* getPathsToLockForOpsInternal() now returns an array in the format LockManager can use
* Also fixed getScopedLocksForOps() for multiwrite backend (it should accept proxy backend paths)
* Updated getScopedFileLocks() docs

Change-Id: Id0dc50c159c5fbc8fca10a9e46c5be23afcb3e9a
2013-10-18 22:57:38 +00:00
Aaron Schulz
7a5902e613 Removed redundant doc comments.
Change-Id: Iced16c6d32ef0e4d58376ed3c092a500ec917446
2013-06-15 22:36:23 -07:00
Aaron Schulz
7242d938f3 Cleaned up variable/type order in docs.
* A few other minor doc cleanups and obvious type hints.

Change-Id: I454fad0c1942068b4f3632ebd1b605bfbb9dfe7c
2013-06-13 11:18:52 -07:00
Aaron Schulz
3bc080c946 [FileBackend] Cleaned up resyncFiles() and added conservative resync option to FileBackendMultiWrite.
Change-Id: I4fd96fda71fc1aba44104241a8329e95ba5cdda7
2013-05-09 12:46:38 -07:00
umherirrender
6c278b6d7e fix some spacing
* Removed spaces around array index
* Removed double spaces or added spaces to begin or end of function
  calls, method signature, conditions or foreachs
* Added braces to one-line ifs
* Changed multi line conditions to one line conditions
* Realigned some arrays

Change-Id: Ia04d2a99d663b07101013c2d53b3b2e872fd9cc3
2013-03-25 22:22:46 +00:00
Aaron Schulz
1045622590 [FileBackend] Fixed "ignoreMissingSource" bug in multiwrite backend.
* Beefed up unit tests for this case and other "ignoreMissingSource" cases
  such as when the source container or parent directory do not exist.

Change-Id: Iea6dae2424edfd0c4367e5cff606c09a4e8a865b
2013-03-12 13:39:40 -07:00
Tyler Anthony Romeo
4dcc7961df Fixed @param tags to conform with Doxygen format.
Doxygen expects parameter types to come before the
parameter name in @param tags. Used a quick regex
to switch everything around where possible. This
only fixes cases where a primitve variable (or a
primitive followed by other types) is the variable
type. Other cases will need to be fixed manually.

Change-Id: Ic59fd20856eb0489d70f3469a56ebce0efb3db13
2013-03-11 13:15:01 -04:00
Reedy
c3e4057e06 Kill off numerous unused variables
Change-Id: I7039f1328f37ee669b694f73ee282602186bffd1
2013-03-08 02:36:24 +00:00
umherirrender
d63121016d fix some spacing
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments

Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
2013-03-07 17:53:21 +01:00
Aaron Schulz
aa7fbbb0cd [FileBackend] Documentation tweaks.
Change-Id: I63b03207a18c7523675a4929031dc560a2d8b5f8
2013-02-04 23:20:18 -08:00
Aaron Schulz
9f95bddda7 [FileBackend] Added getFileHttpUrl() function.
* This can speed up certain video file operations for scripts that support
  specifying source files via URLs and support HTTP Range headers.
* Updated unit tests.

Change-Id: I60cb95c2e3dd9f7df1f740e9182be7c79af69d6e
2012-11-08 23:01:19 +00:00
Siebrand Mazeland
d4b046a893 Update docs for return and exception info
* Removed some inline tabs in the process.
* IDE fixed some incorrect leading spaces, too.

Change-Id: Ic9303eff6db4424ac3f1fa2816839692b43e6190
2012-10-09 09:41:58 +00:00
Aaron Schulz
812487e902 [FileBackend] Tweaked FileBackendMultiWrite behavior to fix test leakage.
Change-Id: I0a967780b64d1883cfeb1a7a183bbb82dc1ae432
2012-10-05 15:07:30 -07:00
Brion VIBBER
a570a73510 Merge "[FileBackend] Added getFileContentsMulti() and improved it for Swift." 2012-10-03 20:01:46 +00:00