Commit graph

77 commits

Author SHA1 Message Date
Aaron Schulz
ac14c1c8a6 filebackend: Added supported for retrieving file metadata/headers
* This can be useful for carrying over metadata when copying files around
* Also fixed a bug in sanitizeHdrs() for Swift (broken content-disposition)

Change-Id: I4534e9acac2b306086797b3677f85c05b98e39fc
2014-01-08 13:22:11 -08:00
Aaron Schulz
2af7ad827d Removed cloudfiles dependency in SwiftFileBackend
* Added a new curl multi wrapper under libs/.
* Used the new MultiHttpClient class instead of CloudFiles
* Removed CDN related config and code as it does not work.
  well due to mutating files and tiny purge rate limits by CDNs.
* Also removed the unused and esoteric 'swiftAnonUser' parameter.
* Support concurrent 'describe' operations.
* Set ACLs on containers on intial PUT instead of in two steps.
  Previously if the second step failed, prepare() would never
  set the ACLs correctly since it aborts if the container exists.
* Improve token deletion to only trigger on 401s rather than other
  random errors like 50X timeouts.
* Removed duplicative getResponse* code by just using one closure
  called for both the concurrent and serial cases. Both cases are
  now funneled through doExecuteOpHandlesInternal().
* Add the MD5 to stat info, as it might be useful for speeding up
  the copy scripts in some cases (when both backends have md5 set).
* Avoid use of trigger_error() since there is a swift log group.
* A few other small code cleanups.

Change-Id: I1adb3e2df6df8cf01d3ad74158de96ea9a79da2e
2014-01-07 23:56:34 -08:00
Aaron Schulz
a0878fe2b4 filebackend: exception handling cleanups
* Split out FileBackendException class and reduced direct use of MWException

Change-Id: I325c1798b6d90972c12a5dccc37989af34d857f3
2013-12-07 23:22:07 -07:00
jenkins-bot
ba7c6b4594 Merge "Remove leading underscore from methods _getResponse*" 2013-11-24 03:20:23 +00:00
Siebrand Mazeland
26d203e2e5 Remove leading underscore from methods _getResponse*
Resolves CodeSniffer warning PSR2.Methods.MethodDeclaration.Underscore

Change-Id: I6cf0fccb47a7e1070999ee510b4d3e0112c41cea
2013-11-24 03:11:57 +00:00
Siebrand Mazeland
8f90b1cfa8 Fix various type and reference issues
Change-Id: Iced174d9f94faf80c5fa21e15fa6c81b6d8fa21c
2013-11-23 20:50:02 +01:00
Siebrand Mazeland
34a4bcbd48 Update documentation of file backend classes
Change-Id: I2113aecc53f1b361d9c276113bc0a9f7076cd21c
2013-11-23 19:24:43 +01:00
Siebrand Mazeland
a5c35215e2 Update formatting of file backend classes
Change-Id: Ie69c324e2f1ea0b8a654adee06ebda494cebd10c
2013-11-22 22:17:15 +01:00
Aaron Schulz
519883bd67 Fixed loading of stat cache entries into the wrong key
* Silly bug from 2afdc066f5 which caused HEAD
  requests. Probably missed in local testing due to memcached.

Change-Id: I3b82ffe964e252eafa5254f7752e5ba45b49ba0e
2013-11-08 21:24:24 -08:00
Aaron Schulz
2afdc066f5 filebackend: improved "adviseStat" performance
* Use the normal page size instead of limiting it way down to the cache
  size. Track the stat information in the pages and load into into the
  stat cache as entries are accessed. This should also be less prone to
  evictions causing HEAD requests (or memcached hits).
* Also bumped CACHE_CHEAP_SIZE up to 500.
* Fix a few doc bits

Change-Id: I8d44a072e7bcc56c83d8d9c8c9ac9864530bccf8
2013-11-08 04:35:57 +00:00
Aaron Schulz
eaa917adad Handle Swift JSON listings with unqualified timestamps
* Throw errors next time invalid timestamps are encountered

Change-Id: Ie5d6c63863cacbf7dc3bc3f408b395bdce859db9
2013-11-06 17:12:41 -08:00
Aaron Schulz
cf03ff3366 Added ceph comment
Change-Id: Ice9efd7ada10e1896afd0185a6305f6852e9605d
2013-10-21 10:20:34 -07:00
Aaron Schulz
91cab51cb5 Factored out a small filebackend getContentType() function
Change-Id: I221483ab4a0ed36ea34c19db1740dfaecf223677
2013-09-25 09:57:07 -07:00
Aaron Schulz
754b8babca filebackend: throw exceptions during file iteration.
* This lets calling code be far more robust rather than just
  silently ignoring entries due to some temporary problem.

Change-Id: I3ce2ae34f6cff5e40a80b8da5688503a387ce2a6
2013-06-15 23:01:47 -07:00
Aaron Schulz
7242d938f3 Cleaned up variable/type order in docs.
* A few other minor doc cleanups and obvious type hints.

Change-Id: I454fad0c1942068b4f3632ebd1b605bfbb9dfe7c
2013-06-13 11:18:52 -07:00
Aaron Schulz
a379dbf9db Removed redundant documentation for some methods.
Change-Id: Iad422716f6796cc4b69405182f4a74daa0c93663
2013-06-12 19:58:15 +00:00
Aaron Schulz
01b564403e Made SwiftFileBackend::loadObjectListing() populate stat entries in reverse order.
Change-Id: I07bf9c852729d65cee72364ca3249e31936d0562
2013-05-19 13:15:50 -07:00
Aaron Schulz
d1940caa1f [FileBackend] Added "adviseStat" option for the "listing followed by stat" case.
* Used this parameter to speed up copyFileBackend.php.
* Also added mtime checks to copyFileBackend.php and a few cleanups.
* Also fixed some incorrect getDirListPageInternal/getFileListPageInternal docs.

Change-Id: I424ef238f7adf4cf1f33b74e3a4e187dcb328a99
2013-05-09 16:19:40 +00:00
Tim Starling
1fe9340bb3 Remove hphpc support and deprecate related functions
hphpc has been superseded by hhvm, so support for hphpc is no longer
needed.

* Continue to use Preprocessor_Hash under HipHop since it is still
  faster under hhvm
* Keep $wgCompiledFiles for now, so that wikihiero doesn't give an error
  before Ic9d1e795 is merged
* Migrate the run-server script and associated configuration file to
  hhvm. Enable EnableStaticContentFromDisk since it doesn't seem
  ridiculously inefficient at first glance. Run from $IP rather than
  $IP/.. since hhvm is apparently not picky about sourcing files from
  outside of the current directory.

Change-Id: Ic3e769f1fbad4f7ad26dd819406796fee48c6b45
2013-05-09 08:28:05 +10:00
umherirrender
1bfc8feb25 Fixed spacing in actions/cache/filebackend/filerepo/job folder
Added spaces before if, foreach
Added some braces for one line statements

Change-Id: Idb93d34e314e5f314223b79208968d6bcd30c40e
2013-04-20 19:18:13 +02:00
umherirrender
15abcf71ca Added/Removed spaces around string concatenation
And added/removed spaces around some other tokens,
like +, -, *, /, <, >, =, !

Fixed windows newline style

Change-Id: I0b9c8c408f3f6bfc0d685a074d7ec468fb848fc8
2013-04-13 13:36:24 +02:00
Aaron Schulz
cf46d892b9 Fixed small doc typo.
Change-Id: Ia2a86e1c8126bde48c17f0c8f56905ed325d8be1
2013-03-31 01:53:35 -07:00
Aaron Schulz
139292dadd [FileBackend] Replaced redundant "disposition" param with "headers".
* Deprecated the "disposition" parameter.
* This also adds "headers" to to copy/move operations.

Change-Id: I63faa1dbbadb42c401b6ed4ac58776dc336ba3ab
2013-03-12 14:16:36 -07:00
Aaron Schulz
70eabe29c5 [FileBackend] Suppress missing container errors with "ignoreMissingSource".
Change-Id: I2b9f4764c4938dcee61f17c31ed4d4b84fbfbcb2
2013-03-12 00:50:16 -07:00
Aaron Schulz
797e52dd05 Reformatted some array parameter docs.
Change-Id: Ie9eefd3693e82f483a9ef0da3f972452ae7bf49e
2013-03-11 11:00:35 -07:00
Tyler Anthony Romeo
4dcc7961df Fixed @param tags to conform with Doxygen format.
Doxygen expects parameter types to come before the
parameter name in @param tags. Used a quick regex
to switch everything around where possible. This
only fixes cases where a primitve variable (or a
primitive followed by other types) is the variable
type. Other cases will need to be fixed manually.

Change-Id: Ic59fd20856eb0489d70f3469a56ebce0efb3db13
2013-03-11 13:15:01 -04:00
Reedy
c3e4057e06 Kill off numerous unused variables
Change-Id: I7039f1328f37ee669b694f73ee282602186bffd1
2013-03-08 02:36:24 +00:00
Reedy
3dd34fee32 Merge "[FileBackend] Added a "ttl" option to getFileHttpUrl()." 2013-03-07 17:59:48 +00:00
umherirrender
d63121016d fix some spacing
Added/removed spaces around logical/arithmetic operator
Reduced multiple empty lines to one empty line
Removed wrong tabs before comments at end of line
Removed too many spaces in assigments

Change-Id: I2bba4e72f9b5f88c53324d7b70e6042f1aad8f6b
2013-03-07 17:53:21 +01:00
Aaron Schulz
66e73f8583 [FileBackend] Added a "ttl" option to getFileHttpUrl().
Change-Id: Ifc7b9c012276ed1f1e436ffafbd5615689680054
2013-03-06 20:55:43 -08:00
Reedy
a4be20a386 Bug 40167 - Filename cannot be empty in SwiftFileBackend
As we're evaluating the output, we can just suppress these
warnings

Change-Id: I5b0af085d163824a7406b675eb0decc617ef4b4c
2013-02-12 15:42:36 +00:00
Ori Livneh
6c163ad265 (Bug 37957) Replace php_sapi_name() with PHP_SAPI
The PHP_SAPI constant has been available since PHP 4.2.0. It's more
concise to use the constant and has less overhead than a function call.

Furthermore, PHP_SAPI rhymes with "happy", whereas "php_sapi_name"
rhymes with "lame". QED, etc.

Change-Id: Ie8c121cb8fcef50536af8d3f66723b458f0bf9af
2013-02-04 12:50:45 -08:00
Aaron Schulz
1063008826 [FileBackend] Fixed bug where "false" could be included in getDirectoryList().
Change-Id: Ib14192d4f5c5143b07af2191b62ea3919483445a
2013-01-23 12:50:21 -08:00
Aaron Schulz
297a110e6d Align comment whitespace a bit.
Change-Id: I0e52a610586aab95021424eac28175b1a3b6d861
2013-01-18 12:31:04 -08:00
Aaron Schulz
4304146501 Moved $function definition up to make IDE happy.
Change-Id: Ie42ba484c8c2f65a2ae233f0176a286578555cb2
2013-01-17 16:32:19 -08:00
Aaron Schulz
f4c25df570 [FileBackend] Added work-around for lack of temp url support in rgw.
* The S3 side of rgw can be configured for use with just temp URLs.
  This works around http://tracker.newdream.net/issues/3454.

Change-Id: Id0ee68da10b38e1d0a0ba7ebba26fccc8d32e2d4
2013-01-17 23:51:59 +00:00
Reedy
d537d96868 Add numerous missing @throws to method documentation
Change-Id: Iba868e82a75fef7c7d011bc5be192bf059d037c0
2012-12-09 03:09:48 +00:00
Aaron Schulz
22bc6df81c [FileBackend] Actually unset the "conn" field in closeConnection().
* This means that handleException() will actually trigger a new auth
  attempt for the next operation if there was an unexpected error.

Change-Id: I8c651cc57d0967549ec67257078c8e126674235d
2012-11-27 14:52:11 -08:00
Aaron Schulz
611ae607a0 Fixed metadata-loss bug from b80bd6159b.
* Contrary to Swift docs, doing a POST seems to kill all the old metadata
  headers. I noticed this in testing due to addMissingMetadata() notices.

Change-Id: I988d7a8d38569823c9d66605b02705ee15a8bbe7
2012-11-20 18:17:19 -08:00
Jan Gerber
b80bd6159b [FileBackend] Added support for changing headers on existing objects.
* Added a 'describe' file operation type to doOperations()/doQuickOperations().
  This can be used by scripts to fill in headers like X-Content-Duration for
  files that already exists.
* Removed wrong comments about removing headers (they don't get removed with null).
* Added some quick unit tests.

Change-Id: I43c5907b59421beaa9487eefac0cdbf8bc6c6d85
2012-11-20 16:07:28 -08:00
Aaron Schulz
81ebc15b05 [FileRepo] [FileBackend] Added support for custom file headers.
* For backends that support it, custom HTTP headers can be set on files.
* Added a getStreamHeaders() function to MediaHandler to let subclasses
  recommend header name/value pairs to be used for responses to GET/HEAD
  requests. For example, an OGG handler could set "X-Content-Duration".
* Made LocalFile use this function to set HTTP headers of new uploads.

Change-Id: I1b017e1342513f0097fe6d142aae18e819403293
2012-11-20 14:07:02 -08:00
Aaron Schulz
9f95bddda7 [FileBackend] Added getFileHttpUrl() function.
* This can speed up certain video file operations for scripts that support
  specifying source files via URLs and support HTTP Range headers.
* Updated unit tests.

Change-Id: I60cb95c2e3dd9f7df1f740e9182be7c79af69d6e
2012-11-08 23:01:19 +00:00
Aaron Schulz
24a6e8eab6 [FileBackend] Support "ignoreMissingSource" for copy and move operations.
* This lets callers use "copy if exist" semantics more easily and avoids extra stat
  queries to the backend (since the cache is cleared before doOperations()).
* Tweaked FileOp::fileSha1() to reduce backend stat requests as 404s are not cached.

Change-Id: Icb5ca14b3316f273d53593f48979d14e113990e1
2012-10-31 04:24:05 +00:00
Demon
ed131b3213 Merge "[FileBackend] Use the new CloudFiles metadata functions." 2012-10-30 18:30:24 +00:00
Aaron Schulz
81e38d3d39 [FileBackend] Use the new CloudFiles metadata functions.
* This helps proof against annoying case sensitivity and case mangling issues.

Change-Id: I580f609847260ee8c5bd99926a615f88bb7395d6
2012-10-30 02:28:37 -07:00
Aaron Schulz
9fd48446a1 [FileBackend] Simplified code for handling "overwrite" parameter.
* Since doQuickOperations() implicitely sets "overwrite", and doOperations() handles it
  via FileOp, there is no reason to also have each backend double checking this parameter
  to handle it. The parameter is no implicit for all the *Internal() functions. This does
  not affect callers. It does reduce the amount of HEAD requests since 404s are not cached.

Change-Id: I7d827e16bc55fe5c7b9aa51ec0c6b2f7c0bb629e
2012-10-29 17:13:02 -07:00
Aaron Schulz
406bb1ef32 [FileBackend] Give warnings when metadata is missing from swift.
Change-Id: I2ec2429c5fca02fbd8b5130b3f0db31faf6e365e
2012-10-22 10:41:43 -07:00
Antoine Musso
9362bb6c56 miscellaneous doxygen warnings
* @licence -> @license
* Protects inline HTML by using double quotes, our inline comments uses
  elements such as <h1> or <firstnameLastname@gmail.com>
* Commands in lowercase (@TODO -> @todo, @NOTE -> @note)
* removes @abstract and @static since doxygen detects them from PHP
  code.
* various undocumented function parameters
* typos in parameters declarations

Change-Id: I62ad6fc124c355bf31acc780b9614a59cf79a421
2012-10-22 14:00:08 +02:00
Aaron Schulz
a2207038e2 [FileBackend] Removed pointless container process cache invalidation for Swift.
Change-Id: I0c0dab602474d27ac744be1ebfb5605022c97be7
2012-10-06 20:07:57 -07:00
Aaron Schulz
19aa631522 Fixed documentation typo.
Change-Id: I31c7959d6feb2241984883e4a02882903a8475d4
2012-10-03 13:32:16 -07:00