Commit graph

104 commits

Author SHA1 Message Date
Aaron Schulz
c699000ae0 Fixed fatal in SwiftFileBackend when a bogus HTTP status is given
* Error was "Fatal error: Call to a member function getSize() on a non-object"

Change-Id: Id1a1b1add9daee4402b2ed017b95c9a9f0e08170
2014-05-30 11:48:14 -07:00
Aaron Schulz
0d54e6e0bf Added unicode encoding support flags to FileBackend
* Fixed bug where even using Swift/Azure on Windows
  would disallow non-ASCII file names.

bug: 1780
Change-Id: I19ed72da0b099d35cae74fb08eeb22c113da1065
2014-05-13 14:11:36 -07:00
Aaron Schulz
45c1d73bdb Made partial file download errors in Swift more useful
Change-Id: I97a5cd1ed38508468122e33968b88adf2570a9a3
2014-05-01 22:44:24 +00:00
umherirrender
957adbef22 Fixed some @params documentation (includes/[file...|upload])
Swapped some "$var type" to "type $var" or added missing types
before the $var. Changed some other types to match the more common
spelling. Makes beginning of some text in captial.
Also added some missing @param.

Change-Id: I8804ebe0922d3a414863b162a2110e0b9e49b80f
2014-04-19 17:19:17 +02:00
umherirrender
2000672ac3 Fixed spacing
- Added spaces after if/foreach/catch
- Added new line before end of file
- Added or removed spaces before/after parenthesis, comma
- Added spaces around string concat

Change-Id: I0590070f1b3542108e242730e8d9a3ba9831e94f
2014-03-20 20:37:30 +00:00
Aaron Schulz
6112acac29 Make use of strong consistency with Ceph RGW
Change-Id: Ic3b764363e91c3279d3dc017ddce630dae26bec5
2014-03-19 17:27:48 +00:00
Aaron Schulz
4f7a786fbe Remove useless "src" param when wrapping doGetFileStatMulti()
Change-Id: I6008384ea872d71dcd2d183186a4aee5048b4dd4
2014-03-03 10:02:31 -08:00
Aaron Schulz
96ae9c07e3 Removed pointless cache bypass in Swift backend
* This was added by mistake in 2af7ad8 apparently

Change-Id: Ic4004428095953ce07e1e97da43748eba04f6639
2014-02-25 17:44:24 -08:00
Aaron Schulz
6b1a60a1ca Added concurrent HEAD request support for Swift
* This adds a new preloadFileStat() method to FileBackend.
* FileBackend::doOperations() makes use of this instead of primeFileCache().
  The later was mostly useless due to the 'latest' flag, which stat entries
  rarely had (and since they use add(), 'latest' ones do not override the
  non-'latest' ones).
* Simplified primeContainerCache/primeFileCache to no longer accept
  FileOp objects and made sure nothing passes those in anymore.

Change-Id: Iee3878cadd69c3a594c701c0aff81aab5f473de5
2014-02-20 08:50:08 +00:00
Aaron Schulz
6d377ae0e3 Added some profiling calls to Swift backend
Change-Id: Ic692c897493fe4388568930c2f4f861b89f2ff78
2014-02-18 11:32:51 -08:00
Aaron Schulz
0259fccfe3 Actually write-back the container cache in SwiftFileBackend
Change-Id: Ia072dd09c1fa5b4dec20a80e69425f7364488362
2014-02-16 07:33:13 +00:00
Aaron Schulz
3d670f1719 Reduce the amount of Swift auth cache keys used on wiki farms
Change-Id: Ic0a2f744d5d94bfb22286a19be3de0c37ecdf519
2014-02-12 07:07:20 +00:00
Aaron Schulz
288941aa53 Fixed broken auth caching for Swift
* A set() call was missing since 2af7ad8

Change-Id: I62b277e886c7ce6142a4d20f705b8337c3e4be58
2014-02-12 06:29:44 +00:00
Aaron Schulz
914fb04afa Simplified Swift backend to use maxConns flag in MultiHttpClient
Change-Id: Iee3756a490243511be86676c01d0a47940529e0b
2014-02-12 00:08:48 +00:00
Siebrand Mazeland
f077c4b1d6 Update formatting
Change-Id: I18aff576262479c9bb1c56eb8e1d1aaae200e4b1
2014-02-06 09:27:05 +01:00
Siebrand Mazeland
0cef781886 Fix CodeSniffer errors and warnings
Change-Id: Idc74e34634d88625773fb8f73315f61edfa67e28
2014-02-05 11:20:17 +01:00
Aaron Schulz
da802de2f2 Use "rb" instead of "rb+" for Swift store operation handles
bug: 60318
Change-Id: I411b2505287d4e4addff3231f9374775353b347b
2014-02-03 23:17:19 +00:00
jenkins-bot
0db4c8d514 Merge "Always return an array from SwiftFileBackend::addMissingMetadata" 2014-01-27 17:59:48 +00:00
Bryan Davis
03efc39055 Always return an array from SwiftFileBackend::addMissingMetadata
Correct the behavior of SwiftFileBackend::addMissingMetadata to match
the documentation and calling code expectations by always returning an
array.

Bug: 60477
Change-Id: I8d12888a6ebfa23502f8cf3a28866dcb00a18ec8
2014-01-27 09:54:25 -08:00
addshore
6e57736499 Fix phpdoc in SwiftFileBackend
Docs for $status and $code were swapped.

Change-Id: I88bd81b671f0e9a4176483fb744e45f2477672b5
2014-01-27 16:07:55 +00:00
Aaron Schulz
688200c607 Fixed use of X-Newest in Swift backend
Change-Id: I9c9572269cd5366272070ccba954c3bd417aa696
2014-01-15 16:48:59 -08:00
Aaron Schulz
f46cd2a593 Fixed wrong usage of $tmpFile in Swift backend
Change-Id: I21293be38c53b4dca3030c26a201f70d8ed243f3
2014-01-15 13:05:01 -08:00
Aaron Schulz
c00ab2af44 Fixed temp URL breakage in Swift
* The API behavior is not well-documented in that it wants the HMAC
  to use the unencoded object name. CloudFiles had this same issue.

bug: 59894
Change-Id: I3f0c80eed55991bb85d0c12f98927efa19af6760
2014-01-13 19:50:41 +00:00
Aaron Schulz
12d88861c5 Fixed silly doc typos
Change-Id: I53792affed74aa5985f82994b886b13603e23c65
2014-01-09 20:25:25 -08:00
jenkins-bot
00e5290c56 Merge "filebackend: Added supported for retrieving file metadata/headers" 2014-01-09 18:36:43 +00:00
Aaron Schulz
6fe98000b5 Made use of proper backend-fail-store message
Change-Id: I89d275926530b16f5660863608b89541d72c9f33
2014-01-08 15:43:56 -08:00
Aaron Schulz
10c0a19195 Added sanity check for doGetLocalCopyMulti() in Swift
* This better handles the case of /tmp being full or something

Change-Id: Ibf4fed0fd053f82ef5ede46b6c2ec72feee95699
2014-01-08 15:02:32 -08:00
Aaron Schulz
ac14c1c8a6 filebackend: Added supported for retrieving file metadata/headers
* This can be useful for carrying over metadata when copying files around
* Also fixed a bug in sanitizeHdrs() for Swift (broken content-disposition)

Change-Id: I4534e9acac2b306086797b3677f85c05b98e39fc
2014-01-08 13:22:11 -08:00
Aaron Schulz
2af7ad827d Removed cloudfiles dependency in SwiftFileBackend
* Added a new curl multi wrapper under libs/.
* Used the new MultiHttpClient class instead of CloudFiles
* Removed CDN related config and code as it does not work.
  well due to mutating files and tiny purge rate limits by CDNs.
* Also removed the unused and esoteric 'swiftAnonUser' parameter.
* Support concurrent 'describe' operations.
* Set ACLs on containers on intial PUT instead of in two steps.
  Previously if the second step failed, prepare() would never
  set the ACLs correctly since it aborts if the container exists.
* Improve token deletion to only trigger on 401s rather than other
  random errors like 50X timeouts.
* Removed duplicative getResponse* code by just using one closure
  called for both the concurrent and serial cases. Both cases are
  now funneled through doExecuteOpHandlesInternal().
* Add the MD5 to stat info, as it might be useful for speeding up
  the copy scripts in some cases (when both backends have md5 set).
* Avoid use of trigger_error() since there is a swift log group.
* A few other small code cleanups.

Change-Id: I1adb3e2df6df8cf01d3ad74158de96ea9a79da2e
2014-01-07 23:56:34 -08:00
Aaron Schulz
a0878fe2b4 filebackend: exception handling cleanups
* Split out FileBackendException class and reduced direct use of MWException

Change-Id: I325c1798b6d90972c12a5dccc37989af34d857f3
2013-12-07 23:22:07 -07:00
jenkins-bot
ba7c6b4594 Merge "Remove leading underscore from methods _getResponse*" 2013-11-24 03:20:23 +00:00
Siebrand Mazeland
26d203e2e5 Remove leading underscore from methods _getResponse*
Resolves CodeSniffer warning PSR2.Methods.MethodDeclaration.Underscore

Change-Id: I6cf0fccb47a7e1070999ee510b4d3e0112c41cea
2013-11-24 03:11:57 +00:00
Siebrand Mazeland
8f90b1cfa8 Fix various type and reference issues
Change-Id: Iced174d9f94faf80c5fa21e15fa6c81b6d8fa21c
2013-11-23 20:50:02 +01:00
Siebrand Mazeland
34a4bcbd48 Update documentation of file backend classes
Change-Id: I2113aecc53f1b361d9c276113bc0a9f7076cd21c
2013-11-23 19:24:43 +01:00
Siebrand Mazeland
a5c35215e2 Update formatting of file backend classes
Change-Id: Ie69c324e2f1ea0b8a654adee06ebda494cebd10c
2013-11-22 22:17:15 +01:00
Aaron Schulz
519883bd67 Fixed loading of stat cache entries into the wrong key
* Silly bug from 2afdc066f5 which caused HEAD
  requests. Probably missed in local testing due to memcached.

Change-Id: I3b82ffe964e252eafa5254f7752e5ba45b49ba0e
2013-11-08 21:24:24 -08:00
Aaron Schulz
2afdc066f5 filebackend: improved "adviseStat" performance
* Use the normal page size instead of limiting it way down to the cache
  size. Track the stat information in the pages and load into into the
  stat cache as entries are accessed. This should also be less prone to
  evictions causing HEAD requests (or memcached hits).
* Also bumped CACHE_CHEAP_SIZE up to 500.
* Fix a few doc bits

Change-Id: I8d44a072e7bcc56c83d8d9c8c9ac9864530bccf8
2013-11-08 04:35:57 +00:00
Aaron Schulz
eaa917adad Handle Swift JSON listings with unqualified timestamps
* Throw errors next time invalid timestamps are encountered

Change-Id: Ie5d6c63863cacbf7dc3bc3f408b395bdce859db9
2013-11-06 17:12:41 -08:00
Aaron Schulz
cf03ff3366 Added ceph comment
Change-Id: Ice9efd7ada10e1896afd0185a6305f6852e9605d
2013-10-21 10:20:34 -07:00
Aaron Schulz
91cab51cb5 Factored out a small filebackend getContentType() function
Change-Id: I221483ab4a0ed36ea34c19db1740dfaecf223677
2013-09-25 09:57:07 -07:00
Aaron Schulz
754b8babca filebackend: throw exceptions during file iteration.
* This lets calling code be far more robust rather than just
  silently ignoring entries due to some temporary problem.

Change-Id: I3ce2ae34f6cff5e40a80b8da5688503a387ce2a6
2013-06-15 23:01:47 -07:00
Aaron Schulz
7242d938f3 Cleaned up variable/type order in docs.
* A few other minor doc cleanups and obvious type hints.

Change-Id: I454fad0c1942068b4f3632ebd1b605bfbb9dfe7c
2013-06-13 11:18:52 -07:00
Aaron Schulz
a379dbf9db Removed redundant documentation for some methods.
Change-Id: Iad422716f6796cc4b69405182f4a74daa0c93663
2013-06-12 19:58:15 +00:00
Aaron Schulz
01b564403e Made SwiftFileBackend::loadObjectListing() populate stat entries in reverse order.
Change-Id: I07bf9c852729d65cee72364ca3249e31936d0562
2013-05-19 13:15:50 -07:00
Aaron Schulz
d1940caa1f [FileBackend] Added "adviseStat" option for the "listing followed by stat" case.
* Used this parameter to speed up copyFileBackend.php.
* Also added mtime checks to copyFileBackend.php and a few cleanups.
* Also fixed some incorrect getDirListPageInternal/getFileListPageInternal docs.

Change-Id: I424ef238f7adf4cf1f33b74e3a4e187dcb328a99
2013-05-09 16:19:40 +00:00
Tim Starling
1fe9340bb3 Remove hphpc support and deprecate related functions
hphpc has been superseded by hhvm, so support for hphpc is no longer
needed.

* Continue to use Preprocessor_Hash under HipHop since it is still
  faster under hhvm
* Keep $wgCompiledFiles for now, so that wikihiero doesn't give an error
  before Ic9d1e795 is merged
* Migrate the run-server script and associated configuration file to
  hhvm. Enable EnableStaticContentFromDisk since it doesn't seem
  ridiculously inefficient at first glance. Run from $IP rather than
  $IP/.. since hhvm is apparently not picky about sourcing files from
  outside of the current directory.

Change-Id: Ic3e769f1fbad4f7ad26dd819406796fee48c6b45
2013-05-09 08:28:05 +10:00
umherirrender
1bfc8feb25 Fixed spacing in actions/cache/filebackend/filerepo/job folder
Added spaces before if, foreach
Added some braces for one line statements

Change-Id: Idb93d34e314e5f314223b79208968d6bcd30c40e
2013-04-20 19:18:13 +02:00
umherirrender
15abcf71ca Added/Removed spaces around string concatenation
And added/removed spaces around some other tokens,
like +, -, *, /, <, >, =, !

Fixed windows newline style

Change-Id: I0b9c8c408f3f6bfc0d685a074d7ec468fb848fc8
2013-04-13 13:36:24 +02:00
Aaron Schulz
cf46d892b9 Fixed small doc typo.
Change-Id: Ia2a86e1c8126bde48c17f0c8f56905ed325d8be1
2013-03-31 01:53:35 -07:00
Aaron Schulz
139292dadd [FileBackend] Replaced redundant "disposition" param with "headers".
* Deprecated the "disposition" parameter.
* This also adds "headers" to to copy/move operations.

Change-Id: I63faa1dbbadb42c401b6ed4ac58776dc336ba3ab
2013-03-12 14:16:36 -07:00