Yep, terse is a good descriptor for me.  Perhaps "too busy to spend lots of time with it", but also just terse.

To store info about a directory we store a node.  This is not to imply that a "file" is stored for it, S3 doesn't have concept of files and dirs, only nodes and URIs.

S3sync never "follows" symlinks as this would make the restore operation incorrect.  It saves the symlink in a way that will restore as a symlink.

Now let's talk about "trac": Is it a symlink too?  If not, can you verify with s3cmd list that in fact nothing else is stored but the directory node?  That sounds like either a bug or some non-understood behavior.

There is a wiki here (see main site).  I would be thrilled for anyone to add more documentation or clarifications.

Hi guys.  I like regularexpressions.info for a good reference.  There is also software that can test a regex against a particular string to see what it will do.

For example online: http://makedatamakesense.com/regex_debug/

Next I will try to look at your regex to see what it does.

Wow man, I didn't see this thread for some reason until your update bumped it in RSS.

It is in fact a logical and reasonable thing.  To delete directories in the same order as the nodes are processed (depth first) wouldn't work.  It would be trying to delete dirs that still contain unprocessed items, which doesn't work (at the programmatic level there's no rm -R).

Instead of adding complicated logic to do recursive removes, it just pushes the list of dirs that need deleting onto a stack, and reads them out in reverse order at the end for actual deletion.  This ensures that /foo/bar/etc is deleted before /foo/bar.

Hope this answers your question.

I'm a devoted follower of s3sync, but not it's rather unarticulate programmer.

By the way I'm happy to help, but calling me names isn't a good way to make your case.  Also, I happen to be extremely articulate.  Perhaps you meant "unresponsive" or something?

Cheers.

If you can create a self-contained test case that demonstrates the bug I can take a look at it.  Otherwise it's too much trouble to figure out what you're doing to get the result 

In the first test, you neglected to say that you deleted user1 from local, but I assume that's what you meant.  And yes, that's how it should work.  You can think of --delete as a synonym for "keep it the same, even if it's destructive to some things on the target side"

the shell script might not work right on mac?  Try the simpler one-cert method noted in the readme, might work better?

I'm guessing one of two Classic mistakes.  Just because you have stuff stored/with/slashes/in/key doesn't mean that upon restore all those levels will be created as folders.  s3 has no concept of folders and files, only nodes.  So s3sync saves special nodes as placeholders for folders. 

So if you store in some tool other than s3sync, restore will fail.  Or if you use s3sync to store a bunch of leaf folders but not the intermediate folder nodes, then when you try to restore them all in one go it will fail.

This is because on the local file system those intermediate folders aren't there, and s3sync sees none of its "folder nodes", so folders won't be created during the restore operation.

That's good.  I spent a little more time on this today and came to the following conclusion..

The slash accounting code is so incredibly spaghetti between trying to emulate rsync, and trying to make S3 list look like directory list..

Fixing it is no longer an option, I have to clean it up and make better sense of it next time.  And this is going to take more energy+time than I have to devote to it at the moment, alas.

This is just YASB
(yet another slash bug).

I hate that I chose the rsync-like interface, slash accounting ALWAYS leads to bugs.  Too tired to figure this out tonight, but at least I know what's going on now.

By the way, if you use recursive, this issue goes away.

This is incredible.. I wonder how long it's been broken.  S3 isn't returning MD5's in the list operation any more it looks like.  I suppose there's some change that I neglected to notice.

I will investigate.  Clearly this has incredibly far reaching effects.

ETA: OK I'm wrong, something else is going on.  I can get the etags ok, but not for this example.  Some problem with prefix interpretation.

I'll try to repro this and see what's going on.  I know it at least *mostly* works because I do daily backups and it certainly doesn't keep transferring all files.  Might be some weird edge case I missed.  I'll post again when I have had time to check.

Documented?  It doesn't take arguments.  It means delete stuff at the target which is not on the source.

As before I can't really tell what's going on from the short results.. need -d at a minimum.

You are correct about the 100% thing; there's no special case to print when it's done.
MD5 is not stored locally, it is recalculated each time.  S3 uses md5 as the etag so it comes down during list operations.