Commit graph

39 commits

Author SHA1 Message Date
Aaron Schulz
611ae607a0 Fixed metadata-loss bug from b80bd6159b.
* Contrary to Swift docs, doing a POST seems to kill all the old metadata
  headers. I noticed this in testing due to addMissingMetadata() notices.

Change-Id: I988d7a8d38569823c9d66605b02705ee15a8bbe7
2012-11-20 18:17:19 -08:00
Jan Gerber
b80bd6159b [FileBackend] Added support for changing headers on existing objects.
* Added a 'describe' file operation type to doOperations()/doQuickOperations().
  This can be used by scripts to fill in headers like X-Content-Duration for
  files that already exists.
* Removed wrong comments about removing headers (they don't get removed with null).
* Added some quick unit tests.

Change-Id: I43c5907b59421beaa9487eefac0cdbf8bc6c6d85
2012-11-20 16:07:28 -08:00
Aaron Schulz
81ebc15b05 [FileRepo] [FileBackend] Added support for custom file headers.
* For backends that support it, custom HTTP headers can be set on files.
* Added a getStreamHeaders() function to MediaHandler to let subclasses
  recommend header name/value pairs to be used for responses to GET/HEAD
  requests. For example, an OGG handler could set "X-Content-Duration".
* Made LocalFile use this function to set HTTP headers of new uploads.

Change-Id: I1b017e1342513f0097fe6d142aae18e819403293
2012-11-20 14:07:02 -08:00
Aaron Schulz
9f95bddda7 [FileBackend] Added getFileHttpUrl() function.
* This can speed up certain video file operations for scripts that support
  specifying source files via URLs and support HTTP Range headers.
* Updated unit tests.

Change-Id: I60cb95c2e3dd9f7df1f740e9182be7c79af69d6e
2012-11-08 23:01:19 +00:00
Aaron Schulz
24a6e8eab6 [FileBackend] Support "ignoreMissingSource" for copy and move operations.
* This lets callers use "copy if exist" semantics more easily and avoids extra stat
  queries to the backend (since the cache is cleared before doOperations()).
* Tweaked FileOp::fileSha1() to reduce backend stat requests as 404s are not cached.

Change-Id: Icb5ca14b3316f273d53593f48979d14e113990e1
2012-10-31 04:24:05 +00:00
Demon
ed131b3213 Merge "[FileBackend] Use the new CloudFiles metadata functions." 2012-10-30 18:30:24 +00:00
Aaron Schulz
81e38d3d39 [FileBackend] Use the new CloudFiles metadata functions.
* This helps proof against annoying case sensitivity and case mangling issues.

Change-Id: I580f609847260ee8c5bd99926a615f88bb7395d6
2012-10-30 02:28:37 -07:00
Aaron Schulz
9fd48446a1 [FileBackend] Simplified code for handling "overwrite" parameter.
* Since doQuickOperations() implicitely sets "overwrite", and doOperations() handles it
  via FileOp, there is no reason to also have each backend double checking this parameter
  to handle it. The parameter is no implicit for all the *Internal() functions. This does
  not affect callers. It does reduce the amount of HEAD requests since 404s are not cached.

Change-Id: I7d827e16bc55fe5c7b9aa51ec0c6b2f7c0bb629e
2012-10-29 17:13:02 -07:00
Aaron Schulz
406bb1ef32 [FileBackend] Give warnings when metadata is missing from swift.
Change-Id: I2ec2429c5fca02fbd8b5130b3f0db31faf6e365e
2012-10-22 10:41:43 -07:00
Antoine Musso
9362bb6c56 miscellaneous doxygen warnings
* @licence -> @license
* Protects inline HTML by using double quotes, our inline comments uses
  elements such as <h1> or <firstnameLastname@gmail.com>
* Commands in lowercase (@TODO -> @todo, @NOTE -> @note)
* removes @abstract and @static since doxygen detects them from PHP
  code.
* various undocumented function parameters
* typos in parameters declarations

Change-Id: I62ad6fc124c355bf31acc780b9614a59cf79a421
2012-10-22 14:00:08 +02:00
Aaron Schulz
a2207038e2 [FileBackend] Removed pointless container process cache invalidation for Swift.
Change-Id: I0c0dab602474d27ac744be1ebfb5605022c97be7
2012-10-06 20:07:57 -07:00
Aaron Schulz
19aa631522 Fixed documentation typo.
Change-Id: I31c7959d6feb2241984883e4a02882903a8475d4
2012-10-03 13:32:16 -07:00
Brion VIBBER
a570a73510 Merge "[FileBackend] Added getFileContentsMulti() and improved it for Swift." 2012-10-03 20:01:46 +00:00
Brion VIBBER
ba652c59fa Merge "[FileBackend] Added getLocalCopyMulti() and getLocalReferenceMulti()." 2012-10-03 19:54:34 +00:00
Aaron Schulz
8641c9af9d [FileBackend] Added getFileContentsMulti() and improved it for Swift.
Change-Id: I6a2173eccda8fe7d4e2e779421e6edf05c8201b4
2012-09-22 11:01:58 -07:00
Aaron
ae65453e84 [FileBackend] Added getLocalCopyMulti() and getLocalReferenceMulti().
* Optimized these in Swift to use pipelined GETs.
* This can also be used to improve concatenate().

Change-Id: Ibeb5df7532f9f5c16736b20c28b7c0d9ddfb412f
2012-09-21 21:10:54 -07:00
Aaron
c5e336c9ad [FileBackend] Fallback to $wgMemc for swift auth caching in cli mode.
Change-Id: I4338c68a18f1424b0d0ec4fe3fcf77c79ba774c8
2012-09-20 12:24:14 -07:00
Aaron Schulz
239e168818 [FileBackend] Avoid stat call in getFileContents() for swift.
* Also removed unused "nostat" flag in addMissingMetadata().

Change-Id: I5f4d827d916b9ca97b9ffd6943c73b6b2faae724
2012-09-17 22:32:58 -07:00
Aaron Schulz
354882095b [FileBackend] Treat NoSuchObjectException as a normal error in streamFile().
Change-Id: I6b64e524a7f3cba6cabe5282f5915cd05dbf8ad0
2012-09-09 10:54:16 -07:00
Aaron
da05f24bf2 [FileBackend] Work-around low header value limits in Swift.
* Also added sanity check to makeContentDisposition() and made the file name optional

Change-Id: Ie4bfef743d11227631606498f026e693dd8d21f3
2012-09-06 10:27:43 -07:00
Demon
b6fcfa633d Merge "[FileBackend] Clear swift connection on invalid HTTP responses." 2012-09-05 19:14:09 +00:00
Aaron
45b9073b61 [FileBackend] Added Content-Disposition support to Swift.
* Renamed $handle => $op to avoid line breaks in a few spots.

Change-Id: I4598e685cc27552425a47f9d97eaeddaaf1a64a1
2012-08-31 16:00:47 -07:00
Aaron
11e9a00af2 [FileBackend] Clear swift connection on invalid HTTP responses.
Change-Id: I1a42b1a4d5fea9ea9fa7b80f97f9b98c29041efb
2012-08-31 10:22:02 -07:00
Demon
cf0cb43263 Merge "[FileBackend] Reduced CDN purging when CDN is enabled." 2012-08-31 13:40:17 +00:00
Aaron
d01f1874c9 [FileBackend] Reduced CDN purging when CDN is enabled.
Change-Id: I63fc9271ca997b19dad957b544503e9ec3b54fa1
2012-08-30 14:23:41 -07:00
Aaron
4b29fa376f [FileBackend] Improved getLocalCopy() performance for Swift.
* This eliminates HEAD requests to speed up concatenate() operations (by ~9 seconds for 300 chunks for wmf).

Change-Id: I43456bd59eeb7943e66cae9c1e42dc410c2a8daf
2012-08-30 12:17:28 -07:00
Aaron Schulz
e74ba4b4fd [FileBackend] Made getLocalCopy() handle long file names better.
Change-Id: I54d7c610dba8cb3e7bc0ef4d4bfbb2327e63db07
2012-08-29 22:06:16 -07:00
Catrope
a1607bf74e Merge "[FileBackend] Made Swift clear the auth cache on certain HTTP errors." 2012-08-28 21:13:05 +00:00
Catrope
3fe42d7504 Merge "[FileBackend] Bumped Swift auth caching TTL." 2012-08-28 21:12:46 +00:00
Aaron
338e84e122 [FileBackend] Made Swift clear the auth cache on certain HTTP errors.
* This can handle when the auth token is expired sooner than expected.

Change-Id: I2e5ecc0adbde032b987af37dfe05a4aadb12b319
2012-08-27 12:36:47 -07:00
Aaron
c5466ba03e [FileBackend] Bumped Swift auth caching TTL.
Change-Id: I5f119ff640cc7af604191450f4577a4564307499
2012-08-27 11:56:05 -07:00
Aaron Schulz
e17b467d40 [FileBackend] Use strcmp() for listing comparisons.
Change-Id: I60da57bd6cdb77a21763bb34afb51a02d905eb3a
2012-08-25 12:07:35 -07:00
Catrope
eb82a6f94e Merge "[FileBackend] Added profiling calls to addMissingMetadata()." 2012-08-17 20:09:55 +00:00
Aaron
bd91a7fd6e [FileBackend] Added auth token/url cache to SwiftFileBackend.
Change-Id: I5a053dec59730f642ac51817a1af57a1e257d3fe
2012-08-17 10:51:39 -07:00
Aaron Schulz
6b0f0b3ffb [FileBackend] Added profiling calls to addMissingMetadata().
Change-Id: I1bc624516542d07b2c4c428532d653e33d3c758a
2012-08-16 19:46:43 -07:00
Catrope
95561afde9 Merge "[FileBackend] Refactored Swift backend to use ProcessCacheLRU." 2012-08-17 00:12:33 +00:00
Aaron
9803fd6b2e [FileBackend] Added UTF-8 restriction to file names to avoid CF breakage.
Change-Id: Id0f246c80f2595fba96e99b3bbe778e89dbc6e84
2012-08-10 14:25:59 -07:00
Aaron
73f11bd45a [FileBackend] Refactored Swift backend to use ProcessCacheLRU.
Change-Id: I1d74c751a98584ca8b1c9ca4bc58d64d521fb679
2012-08-10 12:16:27 -07:00
Aaron Schulz
a47892abef [FileBackend] Moved filerepo/backend/ up to filebackend
Change-Id: I62ba23bd22e0f177fb48acaf6bcc38cbffacd3e7
2012-08-08 18:20:57 -07:00
Renamed from includes/filerepo/backend/SwiftFileBackend.php (Browse further)