* Also tweaked timestamp tolerance in tests, intended for Swift
* Also made a few tweaks to speed up tests a bit
Change-Id: Ibdee36d3bf86089b027dc74bb2582acc1ab4b96b
* doQuickOperations() lets us do things like purge thumbnails as fast as possible.
Stat calls, SHA1 checks, and RTTs in general are avoided. It also avoids the
slowness of lazy population of thumbnail SHA1s in Swift (for those made by rewrite.py).
* Removed supportedOperations() to avoid the extra complexity.
* Made a few variable type and exception documentation cleanups.
* Cleaned up unit test file removal a bit and made some functions private.
Change-Id: I6922368c6af7752a6927d96402519132203108a1
* FS backends work via popen() and basic shell commands.
* Swift backends use the custom SwiftCloudFiles async features.
* Refactored storagePathsRead()/storagePathsChanged() to normalize the paths
for correct dependency detection in FileOpBatch.
* Cleaned up SwiftFileBackend exception handling to make debugging easier.
* Added a quick and dirty performance testing script.
* Updated unit tests to include a run with parallelize=implicit.
* Improved file test failure output a bit.
Change-Id: I6a5ed743c30c598e0dd7301dbdb3631c460332fd
* Fixed unlocking logic in FSLockManager for case when an EX lock was made, then an SH one, and then the EX one was "unlocked"
* Avoid hiding useful unlink() warnings in FSLockManager
* Reduced locking use in test cleanup code
* Added a simple testLockCalls() test function
* Made a few cleanups & fixes to backend tests
Change-Id: I1110d9b537c450d9feca5a2fb35519c22435e81d
* Added directoryExists() function to check if a dir exists.
* Added getDirectoryList() and getTopDirectoryList() functions to list out subdirs.
* Added getTopFileList() function to list out immediate files in a dir. This wraps getFileList(), which was modified.
Change-Id: I087835f40d76c41f5d6af55b9a34a51e0f44a675
* Moved some public static functions from FileBackend to FileBackendBase as the later defines the public API.
* Made splitStoragePath() return null if the backend or container name is empty.
* Made normalizeContainerPath() kill leading directory separators.
* Added more unit tests and made some documentation tweaks.
In FSFileBackend:
* Added resolveContainerName() to disallow '.' a container name, since this would cause a traversal.
* Use 'b' param in some fopen() calls as needed for Windows and newline handling.
* Removed some useless padding code in FileBackend::getContainerShard(). Initialized $m to make IDE happy.
* Updated some code comments.
In SwiftFileBackend:
* Manually set the ETag when using php-cloudfiles for creating files to avoid https://github.com/rackspace/php-cloudfiles/issues/59.
* Manually set the content type based on how StreamFile::getType(). This makes it safe to read files directly out of the proxy to end-users. The streamFile() backend functions already uses a similar content-type check.
new FileRepo() requires an array of parameters having at least the
'name' and 'backend' key setup.
TODO: 'backend' keyword should probably default to FileBackend.
* Made FileBackend::getFileProps() final.
* Added exception needed in SwiftFileBackend::getConnection().
* Various FileBackendTests fixes and additions. Optimized it a bit by keeping the backend instance in memory.
* In StoreBatchTest: sed proper clean() function to remove temp dirs (follows up r109641). Also removed some commented out code.
* Fixed temp dir leakage in FileBackendTest.
* Added SHA-1 to FileOp::attemptBatch $predicates (since concatenate was removed from FileOp). This lets us get rid of temp file backups, as the remaining failure case is just the backend medium going down (the ops would break in that case anyway). Doing so reduces RTTs and backup file I/O overhead. This also simplifies expiry object support by avoiding having to stash the expiry values of temp backup objects somewhere.
* Improved precheck() and attempt() status logic in FileOp::attemptBatch() a bit. Use separate $subStatus var to check if each op is OK.
* A few minor code cleanups and comment tweaks.
* Fixed MoveFileOp bug found in unit tests when the source/dest are the same and an overwriteDest/overwriteSame param is given (the file would just get deleted). Improved CopyFileOp in this case too.
Other:
* Added more unit tests.
* Made secure() call doPrepare() for caller dummy proofing (especially those that don't check the status).
In FSFileBackend:
* Removed redundant wfMkdirParents() calls in FSFileBackend, we already have prepare() for this purpose. This also keeps it's behavior more consistent with the other backends.
* Made use of 'backend-fail-store' message in doStoreInternal().
Other:
* Updated unit tests and renamed $src => $source in some functions for consistency.
* Added some documentation comments and @since tags.
* Made 'success', 'successCount', and 'failCount' fields reflect the overall operation in FileBackendMultiWrite::doOperationsInternal(). This makes it match up with single-write backends.
* Made FileBackend::clearCache() part of the public API.
* Added getFileSize()/getFileStat() functions. Refactored some functions to use the stat function for better reuse and caching/consistency.
* Refactored streamFile() to allow for subclasses to avoid local file copying with less duplication. Also make last-modified check actually work since we always get the timestamp of the original file.
* Renamed 'ignoreErrors' parameter to 'force'.
In FileBackendMultiWrite:
* Simplified how read ops are done (use 'master' backend for consistency).
* Added consistency check to doOperationsInternal() to check if the files are synced.
* Various fixes after testing.
In StreamFile:
* Split out prepareForStream() function from stream() in StreamFile for code reuse.
In FileBackendTest:
* Properly cover FileBackendMultiWrite in tests.
* Various test improvements.
* Make sure FSFileIterator::current() directly returns the string path and that it is relative to the directory being searched.
* Fixed silly bug in testGetFileList that masked any bugs.
* Added a few code comments.
* Changed concatenate to store to a specified temp FS file rather than a final storage destination. This makes it better fit the use case (chunked upload), so we can avoid extra copying around. Subclasses no longer have to implement this function now as well.
* Added extensionFromPath() helper function.
* Moved createInternal() up a bit and fixed @see comments pointing to the wrong functions.
In FSFileBackend:
* Use parent implementation of doConcatenateInternal().
In FileRepo/File:
* Added FileRepo::ALLOW_STALE and made thumbnail transforms use it.
