Commit graph

55 commits

Author SHA1 Message Date
Hashar
8dde0cf4f5 Merge "[FileBackend] Process cache fixes and cleanups." 2012-06-15 21:09:34 +00:00
awjrichards
c29fd59775 Big oops - merged to wrong branch.
Revert "Revert to arbitrarily old point before initial remote branch creation to help clean up"

This reverts commit ee0d3d330f
2012-06-05 22:58:54 +00:00
awjrichards
ee0d3d330f Revert to arbitrarily old point before initial remote branch creation to help clean up
Change-Id: I41a3d1e55d3ea9dffa42451237fe065f9334361d
2012-06-02 08:43:04 -07:00
Demon
0593465614 Merge "[FileBackend] Purge Swift process cache before container delete for sanity." 2012-06-04 20:46:32 +00:00
Aaron Schulz
a998d3ff09 [FileBackend] Process cache fixes and cleanups.
* Made getLocalReference() and getFileSha1Base36() respect latest wrt to cache.
* Made Made getLocalReference() and getFileSha1Base36() normalize the paths for cache.
* Avoid misleading (suggesting cache misses) getFileSha1Base36() profile calls for Swift.

Change-Id: I192a4f00c123029cfa338ec74cd08a355e29896b
2012-05-26 13:06:40 -07:00
Aaron Schulz
6435a6e737 [FileBackend] Purge Swift process cache before container delete for sanity.
* Also tweaked some documentation.

Change-Id: I89369d6211de216193ae1d895356137cacded5d7
2012-05-25 03:41:12 -07:00
Aaron Schulz
ec002cdf8a [FileBackend] Worked around Akamai CDN purge limits.
* Added swiftCDNTTL to let the cache TTL be configurable
* Also added swiftCDNPurgable option to disable purges

Change-Id: Ic8e779c20f7501a5a55f4a49fc612923d0dc268c
2012-05-23 15:29:22 -07:00
Aaron Schulz
723b886429 [FileBackend] Avoid an extra RTT on Swift listing operations.
* Also tweaked timestamp tolerance in tests, intended for Swift
* Also made a few tweaks to speed up tests a bit

Change-Id: Ibdee36d3bf86089b027dc74bb2582acc1ab4b96b
2012-05-22 10:53:48 -07:00
Aaron Schulz
a350d68e09 [FileBackend] Added some Swift listing profiling.
Change-Id: I61d1ec5725be2ebed4bb08e713dfcab7e85160fc
2012-05-20 12:44:38 -07:00
Aaron
24cad34372 [FileBackend] Moved closeResources() call up to parent class function.
Change-Id: I903b40c17e8f757bc3642e24552ddd518c0ef39d
2012-05-17 17:21:39 -07:00
Tim Starling
4024de4bd8 Merge "[FileBackend] Improved connection error handling and logging a bit for Swift." 2012-05-17 07:52:06 +00:00
Tim Starling
ea973a8a00 Merge "[FileBackend] Added CDN support to Swift for third-party sites." 2012-05-17 07:16:09 +00:00
Aaron Schulz
1abcd5e30e [FileBackend] Improved connection error handling and logging a bit for Swift.
Change-Id: I9a128bc5027cba85b1cf4992434d697215b3277f
2012-05-16 18:51:38 -07:00
Aaron
634c3be2bb [FileBackend] Avoid infinite loops when populating missing metadata in Swift.
Change-Id: Ief13048256ee11caf3494d934eb5267f5eb71e8b
2012-05-15 13:05:46 -07:00
Aaron
908b4d4340 [FileBackend] Added CDN support to Swift for third-party sites.
Change-Id: I220ab8cf8fd97d3a080b8dcc8f010d97fcc09671
2012-05-14 16:10:11 -07:00
Aaron
23a0490b78 [FileBackend] Added support for concurrent file write operations.
* FS backends work via popen() and basic shell commands.
* Swift backends use the custom SwiftCloudFiles async features.
* Refactored storagePathsRead()/storagePathsChanged() to normalize the paths
  for correct dependency detection in FileOpBatch.
* Cleaned up SwiftFileBackend exception handling to make debugging easier.
* Added a quick and dirty performance testing script.
* Updated unit tests to include a run with parallelize=implicit.
* Improved file test failure output a bit.

Change-Id: I6a5ed743c30c598e0dd7301dbdb3631c460332fd
2012-05-09 05:37:41 -07:00
Alexandre Emsenhuber
d5381d0ea0 Added missing GPLv2 headers in some places.
Also made file/class documentation more consistent.

Change-Id: Iad15427368d13a6cc11c8512b7265e8c7b8878a5
2012-05-07 09:11:33 +02:00
Brion VIBBER
c9480d2eee Merge "[FileBackend] Some documentation and minor cleanups." 2012-04-26 23:40:12 +00:00
Brion VIBBER
0efd8c5acc Merge "[FileBackend] Tweaked various cache parameters and bumped tiny Swift auth TTL a bit." 2012-04-26 23:32:32 +00:00
Brion VIBBER
e8eddd20de Merge "[FileBackend] Added 'recursive' flag to directory clean() function." 2012-04-26 23:30:05 +00:00
Brion VIBBER
4ef894d9e1 Merge "[FileBackend] Added container stat caching to reduce RTTs to high latency backends." 2012-04-26 22:51:54 +00:00
Aaron
104e8bb0d7 [FileBackend] Added 'recursive' flag to directory clean() function.
Change-Id: I8fadd25c05840e2324bf8433dc0a4846daf8e2c3
2012-04-26 12:41:33 -07:00
Aaron
d6d4fb1219 [FileBackend] Tweaked various cache parameters and bumped tiny Swift auth TTL a bit.
Change-Id: Ibb1a43624a94ff737e9f5c73a0bedec75aba37ff
2012-04-26 10:38:32 -07:00
Aaron
23e2fbcd2f [FileBackend] Added container stat caching to reduce RTTs to high latency backends.
Change-Id: I73575fd65ca06a238803b30f8de6873801a224b3
2012-04-25 18:07:11 -07:00
Aaron
a2b9285634 [FileBackend] Some documentation and minor cleanups.
Change-Id: Ia5f01679c928a9053cfd05d26fe1a03b67dac433
2012-04-25 15:53:22 -07:00
Aaron Schulz
dd5326ccf1 [FileBackend] Added some basic directory functions.
* Added directoryExists() function to check if a dir exists.
* Added getDirectoryList() and getTopDirectoryList() functions to list out subdirs.
* Added getTopFileList() function to list out immediate files in a dir. This wraps getFileList(), which was modified.

Change-Id: I087835f40d76c41f5d6af55b9a34a51e0f44a675
2012-04-21 21:33:41 -07:00
Aaron
b93bbaeeba Removed a useless exception catch statement in SwiftFileBackend::getFileListPageInternal() and fixed visibility of SwiftFileBackend::doGetFileSha1base36().
Change-Id: I3efd0a4cd3fe9ee7d3871382caea70c4a79d8b12
2012-04-17 10:32:43 -07:00
Aaron Schulz
3ce1792d97 [FileBackend]
* Various documentation improvements.
* Moved a few protected FileBackendStoreShardListIterator functions down. Same with normalizeContainerPath().
2012-03-03 19:14:50 +00:00
Aaron Schulz
2687c2e368 r112839: missing file change 2012-03-01 22:01:57 +00:00
Aaron Schulz
736b978ede Improve error log formatting and added error message info for InvalidResponseException 2012-02-16 18:22:09 +00:00
Sam Reed
c052fc3b7b Documentation followups from lastnight 2012-02-10 15:37:33 +00:00
Sam Reed
76246b9bf5 More return documentation 2012-02-09 21:33:27 +00:00
Sam Reed
e1d83d5721 Fixing some of the "@return true" or "@return false", need to be "@return bool" and then the metadata can say true if foo, false if bar
Other documentation improvements
2012-02-09 18:01:54 +00:00
Aaron Schulz
3476b2f30c Bumped $maxContCacheSize to handle container sharding a bit better. However, wikis large enough to need sharding would probably have sane 404 handling anyway, which avoids the need to hit a bunch of containers in one request anyway (like rendering thumbnails on parse). 2012-02-08 20:34:24 +00:00
Tim Starling
391b351f12 * Documentation fix
* Binary mode on php://output, for consistency.
2012-02-01 03:39:38 +00:00
Aaron Schulz
d84684ae30 Expanded 'shardViaHashLevels' config var in FileBackendStore to be able to recognize FileRepo-style deleted zone hash paths (which are different than for the other zones). 2012-01-31 21:52:58 +00:00
Aaron Schulz
92408acbfb Renamed FileBackend to FileBackendStore and ugly FileBackendBase to FileBackend. Callers doing things like FileBackend::extensionFromPath() are now more proper. 2012-01-29 22:22:28 +00:00
Aaron Schulz
6554e9877d Added addMissingMetadata() function to SwiftFileBackend to retroactively add metadata (sha1) on the fly. This can be useful if files were imported from some script that doesn't set the metadata. 2012-01-29 11:33:47 +00:00
Aaron Schulz
b3a3f86a5a Further reduced RTTs in SwiftFileBackend by making "file already exists" checks use the stat cache, typically already set in FileOp::doPrecheck(). FileBackendBase::doOperationsInternal() already clears the cache after locking (and before FileOp::attemptBatch) for consistency. 2012-01-26 20:17:18 +00:00
Aaron Schulz
d1b159ef63 In FileBackend:
* Use 'b' param in some fopen() calls as needed for Windows and newline handling.
* Removed some useless padding code in FileBackend::getContainerShard(). Initialized $m to make IDE happy.
* Updated some code comments.
In SwiftFileBackend:
* Manually set the ETag when using php-cloudfiles for creating files to avoid https://github.com/rackspace/php-cloudfiles/issues/59.
* Manually set the content type based on how StreamFile::getType(). This makes it safe to read files directly out of the proxy to end-users. The streamFile() backend functions already uses a similar content-type check.
2012-01-25 01:57:28 +00:00
Aaron Schulz
a0bce8f10e * Added some wfProfileIn() calls to file backend code.
* Made FileBackend::getFileProps() final.
* Added exception needed in SwiftFileBackend::getConnection().
* Various FileBackendTests fixes and additions. Optimized it a bit by keeping the backend instance in memory.
2012-01-23 08:33:31 +00:00
Aaron Schulz
76fc292ac8 Simplified last_modified timestamp conversion in SwiftFileBackend to just use wfTimestamp() 2012-01-22 03:25:19 +00:00
Aaron Schulz
ac536340e5 * Another fix for SwiftFileBackend file listings
* Fixed bogus array key reference in SwiftFileBackend constructor
2012-01-22 00:33:20 +00:00
Aaron Schulz
2d1d7f644b * Fixed bugs in SwiftFileBackend file listing code
* Clear container cache in SwiftFileBackend when clearCache() is called
* Updated a comment and fixed a comment typo
2012-01-22 00:06:18 +00:00
Aaron Schulz
e2e0c0d9b7 * Follow-up r109009: Check that paths are usable in FileOp::doPrecheck(). Also lock parent directories to avoid prepare()/clean() race conditions for FS backends.
* Fixed bogus $params var in logException() call in SwiftFileBackend.
* Added 'latest' param to FileBackendMultliWrite::consistencyCheck().
* Dummy-proof FileBackend::getFileStat() w.r.t the 'latest' param and removed related FileOp::allowStaleReads() comment.
* Tweaked backend-fail-batchsize message from r109469.
2012-01-19 23:18:03 +00:00
Aaron Schulz
d0fc7d5d89 Renamed 'overwriteDest' FileBackend operation parameter to just 'overwrite', which is shorter and more consistent with 'overwriteSame' 2012-01-19 02:24:49 +00:00
Aaron Schulz
2c6336063c In SwiftFileBackend:
* r108944: doCleanInternal() should delete empty containers only if the container dir was given.
* Made doSecureInternal() set container permissions. Also renamed swiftProxyUser -> swiftAnonUser.
* Made doGetFileStat() respect the 'latest' parameter (using r109235).
* Fixed connTTL default and renamed it to authTTL. Also added explicit close() call to getConnection().
* Reduced RTTs in doPrepareInternal() by checking getContainer() first (which is process cached).
* Killed an RTT in doStoreInternal(), doCreateInternal(), and doCopyInternal() by using create_object(). Also cleaned up logic with regards to the destination CF_Object object fields getting preloaded before write().
* Cleanups to getLocalCopy(); only create the tmp file if get_object() succeeds to short-circuits things.
* Made getContainer() limit the container cache size for sanity.
* Simplified doDeleteInternal() code a bit.
* Renamed $destRel => $dstRel for consistency.
2012-01-18 19:57:32 +00:00
Aaron Schulz
0fbf6e6288 * Killed an RTT in SwiftFileBackend::doStreamFile(). We already check existence in FileBackend::streamFile()
* A few minor code cleanups and comment tweaks.
2012-01-15 22:21:51 +00:00
Aaron Schulz
dbfcb63fe5 r108944: fixed bogus field...php-cloudfiles documentation misspelled this, it's 'object_count' not 'count' 2012-01-14 23:15:32 +00:00
Aaron Schulz
90525eacc4 In SwiftFileBackend:
* Added getContainer(), createContainer(), and deleteContainer() helper functions, which cache container objects to avoid RTTs
* Made doCleanInternal() delete empty containers
* Bumped authentication TTL to 120 seconds
* Some comment tweaks
2012-01-14 23:11:21 +00:00