New subcommand that allows exports from git-annex?

[dmcardle]

Hi folks, is there any interest in a new rclone subcommand that acts as a git-annex remote? The term "remote" is unfortunately overloaded by git-annex and rclone; I'm describing an rclone subcommand that speaks the git-annex external special remote protocol, enabling git-annex to store and retrieve content from rclone remotes.

A similar project called git-annex-remote-rclone already exists, but my primary complaint is that it does not support git-annex-export, aka "tree exports". As a result, the content exported from git-annex to an rclone remote is opaque to humans -- just a bunch of files with names like "GPGHMACSHA512--${DIGEST}", not a browseable file tree.

I think there's also some potential performance improvements if we move the client into rclone. For instance, it could support git-annex's ASYNC protocol extension rather than transferring files one at a time. I assume there's also some overhead to exec-ing rclone once per file that could be eliminated with a more persistent client.

This is admittedly pretty niche, but I'd be pleasantly surprised if there's any community interest!

---

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/new-subcommand-for-git-annex-remote/44546

What is your current rclone version (output from rclone version)?

N/a.

What problem are you are trying to solve?

Export content from git-annex.

How do you think rclone should be changed to solve that?

With a new subcommand.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ncw]

This certainly sounds interesting.

I don't know how the git annex protocol works but if it involves calling rclone many times then it would be most efficient to run an rclone server.

Is this something you are interested in writing?

[dmcardle]

Glad to see there's some interest!

Is this something you are interested in writing?

Yep, I'm already working on it!

@ncw Would you accept this as a new subcommand, or would you prefer it to be a separate binary?

I find motivation/time in bursts, so it's hard to estimate when I'll have something worth sharing. Regardless, here's a rough sequence of milestones I'd like to hit:

1. ✔️ : Minimal support for the external special remote protocol. I believe this would achieve feature parity with git-annex-remote-rclone.
   • At minimum, needs to pass git annex testremote for some subset of rclone remotes.
   • Would also be nice to have unit tests for the new subcommand written in Go.
   • Done in cmd/gitannex: Add the gitannex command #7654
2. Support user migrations from git-annex-remote-rclone and improve end-to-end testing.
   1. ✔️ Improve the end-to-end test script to not make assumptions about the HOME directory.
      • Done in cmd/gitannex: Replace e2e test script with Go command #7716
   2. ✔️ Support git-annex-remote-rclone's repository layout options to enable user migration, e.g. frankencase.
      • Done in cmd/gitannex: Add layout modes for compatibility #7754
   3. Consider supporting aliases of configs for compatibility with git-annex-remote-rclone. This should make transitioning remotes less painful.
   4. Support PROGRESS messages.
3. Add support for the ASYNC protocol extension. This should save us the overhead of starting up rclone N times over N transfers.
   • Maybe develop a benchmark to measure the performance improvement.
4. Add support git-annex's simple export interface. This will enable exporting human-browseable file trees to rclone remotes.