2007-11-20
Version 1.2.3
Fix SSL verification settings that broke in new S3 API.
----------

I'll check this out.

From your post I am confused about "s3 bucket on an instance".  If you tell me the commands you used and what failed I might be able to offer advice.

Wow that's news to me!  I can say that my software does not contain any special code or limits here, it just prints to stderr and stdout a line at a time as you'd expect.

Done in 1.2.2

I still can't do 100-continue (I'd have to ditch the ruby http lib) but it shouldn't matter in practice.  For s3sync the first operation is always a list, so any redirect will be found out there.

For s3cmd the lack of 100-continue support means there is a time window on brand new EU buckets (several minutes after creation in my tests) where a PUT will have to transfer the data twice.  Once to the default host, then again after the 307 redirect.

Phew!  I'm done for a while.

Think all the issues in here are addressed now so, moving.

Thanks to everyone with feedback about the beta.  This release ties up some EU loose ends.

Also addressed a long standing gripe about local directories.  Now you can force local dir/subdir creation even when the S3 items were not stored with s3sync (ergo lack the appropriate dir nodes).  To preserve sanity and decorum, the default behavior is unchanged.

2007-11-20
Version 1.2.2
Handle EU bucket 307 redirects (in s3try.rb)
--make-dirs added
----------

Then moving this issue to closed.

Augh! The ruby S3 lib totally doesn't do the stuff I need to make this work =/

It's not just the 307's, I have to figure out how to make the PUTs do 100-continue responses.

Just to be as complete as possible, a european bucket created with s3fox returns this error:

Code:

S3 command failed:
list_bucket max-keys 200 prefix pre/test delimiter /
With result 307 Temporary Redirect
...

It's all in DNS... :-/

I guess this 307 case needs to be handled; I'll try to figure it out.

aws_calling_format: subdomain

Case sensitive value, SUBDOMAIN!  I don't use or test the yaml.  If there's a bug someone can point at I'll have a go at fixing it.

ferrix:  I appreciate your reply.  The folder nodes DO exist on the S3 side.  Just not on the local side.  I want to duplicate the layout I have on S3 locally.

If you didn't use s3sync to store it (which we already know you didn't) then no there aren't any *s3sync* folder nodes.  In S3 there's no actual concept of folders, just nodes.   So, every tool uses its own proprietary way of storing this info (my scheme being the best naturally) and in general the methods are not compatible.

Don't worry though because by popular demand I am adding --make-dirs to the next version.

maelcum my guess is the frustration must be not with you but with timdau's overall S3 tools experience.  I can relate: when I wanted some way to sync local and S3 content I was very frustrated.  I ended up making my own! 

And now that s3sync exists and is popular, there is a whole new target for people to express frustrations at

I don't see the same behavior.  Although I used small files, the below results plainly show it does not re-get:

Code:

C:\Users\Gregory\s3sync>s3sync.rb -r -v --progress ServEdgeTest: ./restore
Create node 100/ClearSigning.pfx
Create node 100/chain.cer

C:\Users\Gregory\s3sync>s3sync.rb -r -v --progress ServEdgeTest: ./restore

C:\Users\Gregory\s3sync>

Please try yours again but use -d so I get some idea what the generators and comparator are thinking.

My guess is that the connection might be interfered with before all bytes were sent somehow so S3 replies 400 (a HTTP protocol error)

Tried using -s in case some entity is trying to proxy your http and inserting problems accidentally?

Other than that, I can only really help if you send me a link to a wireshark trace of the session.