Actually the download works fine for restoring stuff uploaded by s3sync in the first place.  I understand this isn't the use case that you want, but still it is misleading to imply the download is not useful at all.

I am leaving this thread in the bugs section to remind me to add the feature to create local folders on the fly even when the folder nodes don't exist on the S3 side.  I *do* want to help but the issue is not simple to handle correctly.

As for your other thread about why it keeps trying to get more copies even after transferring once.. that seems like a bug and I will investigate it.

It's not perfect but:
I keep one s3sync target copy named for each day of the week and sync to that copy on its respective day.  Initial upload overhead is 7x total, but after that it's just updating the Tuesday prefix every Tuesday (the sync would only transfer items that changed since the previous Tuesday).

The Better solution would be to build a block device abstraction layer on top of S3 (which I intend to do eventually)

Agreed with your analysis.

I know it's confusing bit this isn't a bug, just a design/usage limitation.  There are already lots of posts about this if you want to learn more.  Try Search.

I want to use s3sync to backup the AWS files locally.

s3sync seems to be downloading the files locally every time even when they exist already.  It should check to see if the file exists and skip it if it does.  Or at least it should have this option.

Thanks.

If the data was stored with s3sync to local once, then I agree that subsequent syncs should not re-copy the files unless they changed.  If you create a detailed set of steps including things like dir/file names and the order of your steps, I may be able to help more.

In the S3 paradigm it is not possible to copy a node without re-transferring it.

Excellent work, it works for me.
But there seems to be a small typo in README.txt: The environment variable to choose the calling format -needed for european buckets- seems to be AWS_CALLING_FORMAT instead of S3SYNC_CALLING_FORMAT.

Greets,
Clou

Fixed thanks!

yet me again - sorry!

I guess I found the problem (although I don't understand it) - creating the bucket using S3Sync somehow helps. After that it works.. Obiously the error only comes up when creating the bucket with e.g. the s3fox firefox plugin. Then it gets listed when doing a listbucket but produces the error mentioned above when trying to put files on.

Regards - maxi

Tell me the command line you're using and the rest of the error message and I might be able to provide insight.

This topic has been moved to Closed Bugs.

http://s3sync.net/forum/index.php?topic=105.0

Fixed @1.2.1

This topic has been moved to Report Bugs.

http://s3sync.net/forum/index.php?topic=102.0

This happens when calculating the md5 (but the same kind of failure could happen during a transfer)
The stream says it has data left, but then when we try to read it.. failure.

Best guess is the item it's failing on is either a non-regular file (a device, socket, term, pipe, etc.) or it is being modified by some external program while the sync is trying to read it.

Via excludes and other precautions you need to be responsible for making sure the things you ask it to do are well-defined and possible.

It is certainly the case that the software could have some softer padding around these problems.  I will try to improve the error message for this case.

Not promiscuously creating directories locally when there are no dir nodes stored @S3 for them is a design decision.  Dow would it know what permissions and owner to use, etc. 

Fixed @1.2.1

I need testers, especially EU-bucket users! 

AWS changed their Ruby API a bunch, and in order to support EU buckets I had to update my code to use it (as well as fix a couple bugs in it).

I have done smoke-testing of s3cmd (now it can create EU buckets finally) and of s3sync to make sure they at least nominally function.  But seriously I'm ultra-busy so please help me by putting it through its paces!

http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz