Commit graph

79 commits

Author SHA1 Message Date
Aaron Schulz
0fa9a391e3 filebackend: Avoid using popen() when "parallelize" is disabled
Bug: 66467
Change-Id: Iaf8eb2ecfad166e26e265a98933399dc4bc255a3
2014-06-18 20:12:03 +00:00
Aaron Schulz
1d59d6152e Improved FileBackend failure logging a bit
* This partly follows up on 2cfce9f

Change-Id: I9d15f44d8000f52fd7b307d358c895452ee96a8d
2014-05-22 17:37:33 -07:00
jenkins-bot
a21d59cc6e Merge "Make room for preloadFileStat() call in FileBackend::doOperationsInternal" 2014-04-21 17:13:35 +00:00
jenkins-bot
e40b19ed4e Merge "Bail out on FileBackend operations if the initial stat calls failed" 2014-04-21 17:13:33 +00:00
umherirrender
957adbef22 Fixed some @params documentation (includes/[file...|upload])
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Also added some missing @param.

Change-Id: I8804ebe0922d3a414863b162a2110e0b9e49b80f
2014-04-19 17:19:17 +02:00
umherirrender
b9cd789fce docs: closure -> Closure; callback -> callable
Changed closure to capital word Closure in doc and type hint,
also changed callback in docs to callable

Change-Id: I52c8e8f13d38a837052101c38b9986be780ca057
2014-04-19 08:43:31 +02:00
Aaron Schulz
d52a559fba Make room for preloadFileStat() call in FileBackend::doOperationsInternal
Change-Id: I60f71155abcf7f69423639b10ec301ac192df728
2014-04-18 17:02:19 -07:00
Aaron Schulz
2cfce9fd8d Bail out on FileBackend operations if the initial stat calls failed
* The preloadFileStat() call acts as a good canary for whether all
  affected servers are up. If anything failed in it, then it's not
  worth bothering to send the actual write requests.
* Also made FileBackend::preloadFileStat abstract while at it since
  the two subclasses implemented it and so should any others.
* Fixed a silly comment typo.

Change-Id: I5bd427f654aa4a9d6bfe4ed7566276e8ac520b30
2014-04-18 12:41:23 -07:00
Aaron Schulz
02e0d115a1 Tweaked "latest" handling of filebackend stat entries
* Let "latest" stat entries override non "latest" so
  that future getFileStat() calls with the "latest" flag
  can actually have a cache hit.

Change-Id: I1e9391039537d608b89773b4d51575e3b364a751
2014-03-24 06:55:01 +00:00
Aaron Schulz
6112acac29 Make use of strong consistency with Ceph RGW
Change-Id: Ic3b764363e91c3279d3dc017ddce630dae26bec5
2014-03-19 17:27:48 +00:00
Aaron Schulz
f5f2233daa Avoid key fragmentation in FileBackend
* Make sure that global backends do not prefix the cache keys
  with local wiki IDs for container and file stat entries.
  This causes fragmentation and breaks invalidation.

Change-Id: Ic02d6e41f828dc82cc07c7d24c8af29c46392a9c
2014-02-24 13:22:15 -08:00
Aaron Schulz
6b1a60a1ca Added concurrent HEAD request support for Swift
* This adds a new preloadFileStat() method to FileBackend.
* FileBackend::doOperations() makes use of this instead of primeFileCache().
  The later was mostly useless due to the 'latest' flag, which stat entries
  rarely had (and since they use add(), 'latest' ones do not override the
  non-'latest' ones).
* Simplified primeContainerCache/primeFileCache to no longer accept
  FileOp objects and made sure nothing passes those in anymore.

Change-Id: Iee3878cadd69c3a594c701c0aff81aab5f473de5
2014-02-20 08:50:08 +00:00
Siebrand Mazeland
f077c4b1d6 Update formatting
Change-Id: I18aff576262479c9bb1c56eb8e1d1aaae200e4b1
2014-02-06 09:27:05 +01:00
Aaron Schulz
c745ff9a58 Fixed bogus var usage in SwiftFileBackend
Change-Id: Ia0dec914208cb3100b7f5bbd2291985eac05dc02
2014-01-09 12:39:13 -08:00
Aaron Schulz
ac14c1c8a6 filebackend: Added supported for retrieving file metadata/headers
* This can be useful for carrying over metadata when copying files around
* Also fixed a bug in sanitizeHdrs() for Swift (broken content-disposition)

Change-Id: I4534e9acac2b306086797b3677f85c05b98e39fc
2014-01-08 13:22:11 -08:00
Aaron Schulz
a0878fe2b4 filebackend: exception handling cleanups
* Split out FileBackendException class and reduced direct use of MWException

Change-Id: I325c1798b6d90972c12a5dccc37989af34d857f3
2013-12-07 23:22:07 -07:00
Siebrand Mazeland
8c0cbe6f83 Update ternary notation
Change-Id: I553dd1d42477326848e6e40b824058b6790e2eb5
2013-12-04 09:52:51 +01:00
Siebrand Mazeland
c232652303 Remove unused local variable
Just count if there are array members

Change-Id: I4f9cc8ba554404796822317fdb285e2f560d01dd
2013-11-25 06:48:23 +01:00
Siebrand Mazeland
34a4bcbd48 Update documentation of file backend classes
Change-Id: I2113aecc53f1b361d9c276113bc0a9f7076cd21c
2013-11-23 19:24:43 +01:00
Siebrand Mazeland
a5c35215e2 Update formatting of file backend classes
Change-Id: Ie69c324e2f1ea0b8a654adee06ebda494cebd10c
2013-11-22 22:17:15 +01:00
Aaron Schulz
beb7359243 Removed excess profile calls
Change-Id: Idd57e8d7fff56c62a66c7fcaf4627b9096f0c363
2013-11-16 23:44:41 -08:00
Aaron Schulz
2afdc066f5 filebackend: improved "adviseStat" performance
* Use the normal page size instead of limiting it way down to the cache
  size. Track the stat information in the pages and load into into the
  stat cache as entries are accessed. This should also be less prone to
  evictions causing HEAD requests (or memcached hits).
* Also bumped CACHE_CHEAP_SIZE up to 500.
* Fix a few doc bits

Change-Id: I8d44a072e7bcc56c83d8d9c8c9ac9864530bccf8
2013-11-08 04:35:57 +00:00
Aaron Schulz
584de11eb1 Fixed broken quickDescribe() method
Change-Id: Ic68f483c4008ff6ae8c4accef2f9d1f180a5d377
2013-10-24 16:34:07 -07:00
Aaron Schulz
ef2f2610ab filebackend: try to combine SH and EX lock acquisition
* getPathsToLockForOpsInternal() now returns an array in the format LockManager can use
* Also fixed getScopedLocksForOps() for multiwrite backend (it should accept proxy backend paths)
* Updated getScopedFileLocks() docs

Change-Id: Id0dc50c159c5fbc8fca10a9e46c5be23afcb3e9a
2013-10-18 22:57:38 +00:00
Aaron Schulz
91cab51cb5 Factored out a small filebackend getContentType() function
Change-Id: I221483ab4a0ed36ea34c19db1740dfaecf223677
2013-09-25 09:57:07 -07:00
Aaron Schulz
ad372af149 Switched FileBackendStore to use ProfileSection when possible.
* Also removed outer profiling layer.

Change-Id: Ice3b921fe0c9981ac16f005d66a78bf4f4a5c601
2013-06-19 00:43:54 +00:00
Aaron Schulz
7242d938f3 Cleaned up variable/type order in docs.
* A few other minor doc cleanups and obvious type hints.

Change-Id: I454fad0c1942068b4f3632ebd1b605bfbb9dfe7c
2013-06-13 11:18:52 -07:00
Aaron Schulz
a379dbf9db Removed redundant documentation for some methods.
Change-Id: Iad422716f6796cc4b69405182f4a74daa0c93663
2013-06-12 19:58:15 +00:00
Aaron Schulz
40078d1c69 [FileBackend] Avoid concurrency for the 1 op case in doQuickOperationsInternal().
Change-Id: I6526d3b77fc078d80a0fb537d504a7fabcbb4c41
2013-05-15 10:35:51 -07:00
Aaron Schulz
b14212992d [FileBackend] Moved short-circuit check from 6b1194b9 up to FileBackend.
Change-Id: I57566a1779f353c18d96c59c47112c753aea6b90
2013-05-10 13:55:39 -07:00
Aaron Schulz
6b1194b9da [FileBackend] Optimize the case were no operations are provided.
Change-Id: Iaf858ce2e8b21e031d27b984029f0424518201af
2013-05-10 13:45:34 -07:00
Aaron Schulz
d1940caa1f [FileBackend] Added "adviseStat" option for the "listing followed by stat" case.
* Used this parameter to speed up copyFileBackend.php.
* Also added mtime checks to copyFileBackend.php and a few cleanups.
* Also fixed some incorrect getDirListPageInternal/getFileListPageInternal docs.

Change-Id: I424ef238f7adf4cf1f33b74e3a4e187dcb328a99
2013-05-09 16:19:40 +00:00
Aaron Schulz
dfd2277cd1 [FileBackend] Process cache negatives for SHA1 on file stat.
Change-Id: Idc9f82309d52818acf57adc9c4eb04370a7b422a
2013-05-07 14:15:11 -07:00
umherirrender
1bfc8feb25 Fixed spacing in actions/cache/filebackend/filerepo/job folder
Added spaces before if, foreach
Added some braces for one line statements

Change-Id: Idb93d34e314e5f314223b79208968d6bcd30c40e
2013-04-20 19:18:13 +02:00
Aaron Schulz
79910f36d4 [FileBackend] Made stat cache factor in last-modified time.
* This should lesson the chance of consistency problems
  if there is any memcached flapping.
* Also cleaned up some documentation.

Change-Id: I89401892c7e848fcff0b093eaa861dfe02387a5c
2013-04-19 03:32:57 +00:00
Aaron Schulz
14fc5f5a93 [FileBackend] Disallow files ending in / and clarified some docs.
Change-Id: I162ca34fe9f92d862345004bd184013531da9e8d
2013-04-19 03:30:14 +00:00
Aaron Schulz
dcc69671e0 [FileBackend] Simplified the shard list iterator to use AppendIterator and FilterIterator.
Change-Id: I22f79447b1edec4fa6d7fc06d67f3f831a484f16
2013-04-17 17:06:52 +00:00
umherirrender
15abcf71ca Added/Removed spaces around string concatenation
And added/removed spaces around some other tokens,
like +, -, *, /, <, >, =, !

Fixed windows newline style

Change-Id: I0b9c8c408f3f6bfc0d685a074d7ec468fb848fc8
2013-04-13 13:36:24 +02:00
umherirrender
978bb31c5e Add missing wfProfileOut before throwing an exception
Change-Id: I1d830da0597f19efd0b2ae48642389975e736e23
2013-04-08 18:37:24 +00:00
Platonides
be7ec69a90 Use canonical class name.
Change-Id: I7dc9980d334bc5ad6421135d713d890a8588281c
2013-04-02 22:58:01 +02:00
ASchulz
c328cf6401 [FileBackend] Cleanup behavior for coping/moving a file over itself.
* Previously, for doQuickOperations(), the default move function
  would just delete the file if it was moved over itself. In fact,
  the php-cloudfiles bindings have this same bug in its move function.
* For both copy and move, when the source and destination are the
  same path, make sure that the headers get updated as specified.
  This only applies for the 'overwrite' case (not 'overwriteSame').
* Fixed bad status for doQuickOperations() when copying a file over
  itself using FSFileBackend.
* Clarified the documentation for 'overwriteSame' option.
* Renamed destSameAsSource to overwriteSameCase in FileOp for clarity.

Change-Id: I3f609d9413795c654de27958b1e807b1fc785f31
2013-03-25 17:06:30 -07:00
Aaron Schulz
139292dadd [FileBackend] Replaced redundant "disposition" param with "headers".
* Deprecated the "disposition" parameter.
* This also adds "headers" to to copy/move operations.

Change-Id: I63faa1dbbadb42c401b6ed4ac58776dc336ba3ab
2013-03-12 14:16:36 -07:00
Aaron Schulz
797e52dd05 Reformatted some array parameter docs.
Change-Id: Ie9eefd3693e82f483a9ef0da3f972452ae7bf49e
2013-03-11 11:00:35 -07:00
Tyler Anthony Romeo
4dcc7961df Fixed @param tags to conform with Doxygen format.
Doxygen expects parameter types to come before the
parameter name in @param tags. Used a quick regex
to switch everything around where possible. This
only fixes cases where a primitve variable (or a
primitive followed by other types) is the variable
type. Other cases will need to be fixed manually.

Change-Id: Ic59fd20856eb0489d70f3469a56ebce0efb3db13
2013-03-11 13:15:01 -04:00
Reedy
c3e4057e06 Kill off numerous unused variables
Change-Id: I7039f1328f37ee669b694f73ee282602186bffd1
2013-03-08 02:36:24 +00:00
umherirrender
d63121016d fix some spacing
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments

Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
2013-03-07 17:53:21 +01:00
Aaron Schulz
f44c768fba [FileBackend] Process cache negatives for file SHA1 function.
Change-Id: I2a6b801cd97e8ee1a678a43d5f6d01d3e9603637
2013-02-26 11:09:36 -08:00
Aaron Schulz
ac56107ba3 Merge "Revert "[FileBackend] Added optional callback parameter to concatenate()."" 2012-11-25 01:05:56 +00:00
Aaron Schulz
65612b4290 Revert "[FileBackend] Added optional callback parameter to concatenate()."
This gives updates for the faster part (local FS file changes) and not
the slow GETs that really need this. This should be re-done to use a
byte based read progress callback (that handles the parallel GETs).

This reverts commit 6c9f13b547

Change-Id: Id8d739e1d5048e7f38c68eda4f9a008682707cba
2012-11-24 17:01:12 -08:00
Aaron Schulz
ab0a45cc06 [FileBackend] More stat caching improvements.
* Extended negative caching to handle the "latest" parameter.
* Added a new "dstExists" parameter to some write functions to avoid
  salting the cache when a file is created at an unused path. The ability
  to do this was already mentioned in the setFileCache() doc comments.

Change-Id: Ib64e4c128e16f4d284033fff70b88686fa0593ab
2012-11-21 11:24:38 -08:00