Commit graph

38 commits

Author SHA1 Message Date
Aaron Schulz
2cfce9fd8d Bail out on FileBackend operations if the initial stat calls failed
* The preloadFileStat() call acts as a good canary for whether all
  affected servers are up. If anything failed in it, then it's not
  worth bothering to send the actual write requests.
* Also made FileBackend::preloadFileStat abstract while at it since
  the two subclasses implemented it and so should any others.
* Fixed a silly comment typo.

Change-Id: I5bd427f654aa4a9d6bfe4ed7566276e8ac520b30
2014-04-18 12:41:23 -07:00
Aaron Schulz
8aaab01e1d Made preloadFileStat/preloadCache actually work for FileBackendMultiWrite
* Previously the path substitution did not take place

Change-Id: I58ad3e2aa44a2b08f4c6bad1a783fb92ea79dbf4
2014-04-12 15:42:39 -07:00
Aaron Schulz
6b1a60a1ca Added concurrent HEAD request support for Swift
* This adds a new preloadFileStat() method to FileBackend.
* FileBackend::doOperations() makes use of this instead of primeFileCache().
  The later was mostly useless due to the 'latest' flag, which stat entries
  rarely had (and since they use add(), 'latest' ones do not override the
  non-'latest' ones).
* Simplified primeContainerCache/primeFileCache to no longer accept
  FileOp objects and made sure nothing passes those in anymore.

Change-Id: Iee3878cadd69c3a594c701c0aff81aab5f473de5
2014-02-20 08:50:08 +00:00
Siebrand Mazeland
f077c4b1d6 Update formatting
Change-Id: I18aff576262479c9bb1c56eb8e1d1aaae200e4b1
2014-02-06 09:27:05 +01:00
Aaron Schulz
ac14c1c8a6 filebackend: Added supported for retrieving file metadata/headers
* This can be useful for carrying over metadata when copying files around
* Also fixed a bug in sanitizeHdrs() for Swift (broken content-disposition)

Change-Id: I4534e9acac2b306086797b3677f85c05b98e39fc
2014-01-08 13:22:11 -08:00
jenkins-bot
76dc198651 Merge "filebackend: cleaned up the FileBackend constructor" 2013-12-10 22:09:21 +00:00
Aaron Schulz
a0878fe2b4 filebackend: exception handling cleanups
* Split out FileBackendException class and reduced direct use of MWException

Change-Id: I325c1798b6d90972c12a5dccc37989af34d857f3
2013-12-07 23:22:07 -07:00
Aaron Schulz
2aa84651ca filebackend: cleaned up the FileBackend constructor
* Added some b/c code with deprecation warnings

Change-Id: Ifceffbc0a37a223bcd7cd3dc60181fc85765bc46
2013-12-03 13:57:01 -08:00
Aaron Schulz
2f29c7551b Revert "filebackend: cleaned up the FileBackend constructor"
Actually this messes with the implicit backend made for things like Math (when unconfigured), which uses the "new" operator.

This reverts commit 1f129a22cb.

Change-Id: I4c72c4f7c8b82e38df5496cf2b90fc9e19c40334
2013-12-03 20:40:24 +00:00
Aaron Schulz
1f129a22cb filebackend: cleaned up the FileBackend constructor
* Moved some of the graph construction work to FileBackendGroup.
  This helps the code in not depending on the rest of MW so much.
* Updated tests and FileBackendMultiwrite, which are the only things
  directly constructing FileBackend objects.

Change-Id: I188a053c70ce088ce34613d5db40e6708e3ea9b7
2013-11-23 20:21:53 -08:00
Siebrand Mazeland
4b985bd667 Add missing type hint for $autoSync
Change-Id: I63deed0b2b9b6fc29f67c0b522987df156f685fa
2013-11-23 20:30:39 +01:00
Siebrand Mazeland
34a4bcbd48 Update documentation of file backend classes
Change-Id: I2113aecc53f1b361d9c276113bc0a9f7076cd21c
2013-11-23 19:24:43 +01:00
Siebrand Mazeland
a5c35215e2 Update formatting of file backend classes
Change-Id: Ie69c324e2f1ea0b8a654adee06ebda494cebd10c
2013-11-22 22:17:15 +01:00
Aaron Schulz
ef2f2610ab filebackend: try to combine SH and EX lock acquisition
* getPathsToLockForOpsInternal() now returns an array in the format LockManager can use
* Also fixed getScopedLocksForOps() for multiwrite backend (it should accept proxy backend paths)
* Updated getScopedFileLocks() docs

Change-Id: Id0dc50c159c5fbc8fca10a9e46c5be23afcb3e9a
2013-10-18 22:57:38 +00:00
Aaron Schulz
7a5902e613 Removed redundant doc comments.
Change-Id: Iced16c6d32ef0e4d58376ed3c092a500ec917446
2013-06-15 22:36:23 -07:00
Aaron Schulz
7242d938f3 Cleaned up variable/type order in docs.
* A few other minor doc cleanups and obvious type hints.

Change-Id: I454fad0c1942068b4f3632ebd1b605bfbb9dfe7c
2013-06-13 11:18:52 -07:00
Aaron Schulz
3bc080c946 [FileBackend] Cleaned up resyncFiles() and added conservative resync option to FileBackendMultiWrite.
Change-Id: I4fd96fda71fc1aba44104241a8329e95ba5cdda7
2013-05-09 12:46:38 -07:00
umherirrender
6c278b6d7e fix some spacing
* Removed spaces around array index
* Removed double spaces or added spaces to begin or end of function
  calls, method signature, conditions or foreachs
* Added braces to one-line ifs
* Changed multi line conditions to one line conditions
* Realigned some arrays

Change-Id: Ia04d2a99d663b07101013c2d53b3b2e872fd9cc3
2013-03-25 22:22:46 +00:00
Aaron Schulz
1045622590 [FileBackend] Fixed "ignoreMissingSource" bug in multiwrite backend.
* Beefed up unit tests for this case and other "ignoreMissingSource" cases
  such as when the source container or parent directory do not exist.

Change-Id: Iea6dae2424edfd0c4367e5cff606c09a4e8a865b
2013-03-12 13:39:40 -07:00
Tyler Anthony Romeo
4dcc7961df Fixed @param tags to conform with Doxygen format.
Doxygen expects parameter types to come before the
parameter name in @param tags. Used a quick regex
to switch everything around where possible. This
only fixes cases where a primitve variable (or a
primitive followed by other types) is the variable
type. Other cases will need to be fixed manually.

Change-Id: Ic59fd20856eb0489d70f3469a56ebce0efb3db13
2013-03-11 13:15:01 -04:00
Reedy
c3e4057e06 Kill off numerous unused variables
Change-Id: I7039f1328f37ee669b694f73ee282602186bffd1
2013-03-08 02:36:24 +00:00
umherirrender
d63121016d fix some spacing
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments

Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
2013-03-07 17:53:21 +01:00
Aaron Schulz
aa7fbbb0cd [FileBackend] Documentation tweaks.
Change-Id: I63b03207a18c7523675a4929031dc560a2d8b5f8
2013-02-04 23:20:18 -08:00
Aaron Schulz
9f95bddda7 [FileBackend] Added getFileHttpUrl() function.
* This can speed up certain video file operations for scripts that support
  specifying source files via URLs and support HTTP Range headers.
* Updated unit tests.

Change-Id: I60cb95c2e3dd9f7df1f740e9182be7c79af69d6e
2012-11-08 23:01:19 +00:00
Siebrand Mazeland
d4b046a893 Update docs for return and exception info
* Removed some inline tabs in the process.
* IDE fixed some incorrect leading spaces, too.

Change-Id: Ic9303eff6db4424ac3f1fa2816839692b43e6190
2012-10-09 09:41:58 +00:00
Aaron Schulz
812487e902 [FileBackend] Tweaked FileBackendMultiWrite behavior to fix test leakage.
Change-Id: I0a967780b64d1883cfeb1a7a183bbb82dc1ae432
2012-10-05 15:07:30 -07:00
Brion VIBBER
a570a73510 Merge "[FileBackend] Added getFileContentsMulti() and improved it for Swift." 2012-10-03 20:01:46 +00:00
Brion VIBBER
ba652c59fa Merge "[FileBackend] Added getLocalCopyMulti() and getLocalReferenceMulti()." 2012-10-03 19:54:34 +00:00
Aaron Schulz
613c5949a6 Reorder keys for fileStoragePathsForOps() function since its a list.
Change-Id: I83cf6f6db92c9a0b39c3229d688463c5cda61d69
2012-10-02 10:40:35 -07:00
Aaron Schulz
8641c9af9d [FileBackend] Added getFileContentsMulti() and improved it for Swift.
Change-Id: I6a2173eccda8fe7d4e2e779421e6edf05c8201b4
2012-09-22 11:01:58 -07:00
Aaron
ae65453e84 [FileBackend] Added getLocalCopyMulti() and getLocalReferenceMulti().
* Optimized these in Swift to use pipelined GETs.
* This can also be used to improve concatenate().

Change-Id: Ibeb5df7532f9f5c16736b20c28b7c0d9ddfb412f
2012-09-21 21:10:54 -07:00
Aaron Schulz
5a4046e80c [FileBackend] Added automatic recovery option for backend sync errors.
* Separated consistency checks from permission checks.
* Made fileStoragePathsForOps() exclude non-storage paths.

Change-Id: If15c5fb890f7506ac9c9bfb78c46ab9228a55a8f
2012-09-12 21:33:16 +00:00
Aaron
0da6eaea05 [FileBackend] Reduce backend sync errors a bit by aborting operations more.
Change-Id: Iedb8d7e1f95c971dbb9f9baed67de047753ab40d
2012-08-30 10:10:50 -07:00
Aaron Schulz
feb7a3e10e [FileBackend] Reduced stat calls when using the multiwrite backend.
Change-Id: I46805637a9dcaaea597bf6eeba204a6889a36a51
2012-08-26 02:11:31 -07:00
Aaron
36c0c1edb6 [FileBackend] MultiWrite options to avoid pushing certain changes to all backends.
* This can be used to no-op dir and file operations on the clone backends
  for the 'thumb' and 'temp' zones to reduce load or avoid hitting certain mounts.

Change-Id: I2c57904b8264b7479b17736333b0b9228b469bbc
2012-08-22 16:02:12 -07:00
Aaron
e6a1ba3fd7 [FileBackend] Actually respend the "syncChecks" config option
Change-Id: Ibb6a243b49970da7c22ff14231b04cb0ac5c64e7
2012-08-15 17:43:36 -07:00
Aaron
c51a9a288b [FileBackend] Added resyncFiles() function to multiwrite backend.
* Also fixed the the SHA1 function calls in consistencyCheck().

Change-Id: I29758ff79102929aa0ffb756526d7918bf74ca39
2012-08-15 17:24:20 -07:00
Aaron Schulz
a47892abef [FileBackend] Moved filerepo/backend/ up to filebackend
Change-Id: I62ba23bd22e0f177fb48acaf6bcc38cbffacd3e7
2012-08-08 18:20:57 -07:00
Renamed from includes/filerepo/backend/FileBackendMultiWrite.php (Browse further)