A suggestion about product design

[myselfghost]

First of all, thank you very much for developing such a good tool with rich functions.
I have an idea for a product design.

1. A client only needs to configure the connection address of the server, and the client will be registered in the server.
2. On the server side, the registered client can be tagged.
3. On the server side, multiple file synchronization policies can be created.
4. On the server side, file synchronization policies can be associated with one or more tagged clients.
5. On the client side,The client will receive the file synchronization policy from the server and execute the actions of this policy

[mstmdev]

1. A client only needs to configure the connection address of the server, and the client will be registered in the server.
2. On the server side, the registered client can be tagged.

Thanks for your suggestion and that sounds great. @myselfghost
At present, there is a simple client registration implementation, see the online field in the Report API.
In order to achieve policy delivery, we need to further enhance security, stability, and reliability.

3. On the server side, multiple file synchronization policies can be created.
4. On the server side, file synchronization policies can be associated with one or more tagged clients.
5. On the client side,The client will receive the file synchronization policy from the server and execute the actions of this policy

Which scenarios require support for policy delivery: pulling files from the server, pushing files to the server, or both?
What policies do we need to support? All existing flags, or some other policies?
Do you have any user cases for specific scenarios involving multiple policies, and how to resolve or detect policy conflicts?

[myselfghost]

One of the scenarios is: building a backup center，

1. Machine A, path /data/p1, backed up once a day
2. Machine B, path /data/p2, backed up once a week
3. Machine C, path /data/p3, backed up once a week
4. Machine C, path /data/p4, backed up once a month

maybe need so many policy ,It would be better for us to manage these policies uniformly on the server side.

[mstmdev]

So, how to assign the policy 1 to machine A, and the policy 2 to machine B?
Do we need a back-end management platform to assign the policies by administrator?

[myselfghost]

So, how to assign the policy 1 to machine A, and the policy 2 to machine B? Do we need a back-end management platform to assign the policies by administrator?

Looks like a web backend is needed. We can use this web to create a new policy, tag machine, and then bind the policy to one or more tags

[mstmdev]

+----------+                           +----------+                           +----------+                          +----------+
|          |                           |          |<---(C)---connect&auth ----|          |                          |          |
|  Server  |                           |  Server  |                           |  Client  |                          |  Client  |
|  Disk    |                           |          |----(D)--deliver policy--->|          |                          |  Disk    |
|          |                           |          |                           |          |                          |          |
|          |<---(H)--- read file   ----|          |<---(G)-- pull file    ----|          |                          |          |
|          |                           |          |                           |          |                          |          |
|          |----(I)--- return file --->|          |----(J)--- send file   --->|          |----(K)--- write file --->|          |
|          |                           |          |                           |          |                          |          |
|          |                           |          |<---(M)-- report result----|          |<---(L)-- write result----|          |
+----------+                           +----------+                           +--|-------+                          +----------+
                                          ^    ^                                 |    ^ 
                        (A) create policy |    |           (E) add to task queue |    |   
                                          |    |                                 |    |
                                          |    | (B) assign policy               v    | (F) execute task  
                                       +--+-—--|--+                           +-------|--+
                                       |          |                           |          |
                                       |  Admin   |                           |  Task    |
                                       |          |                           |  Queue   |
                                       |          |                           |          |
                                       |          |                           |          |
                                       |          |                           |          |
                                       +----------+                           +----------+

The above is a simplified flow diagram, and there's a lot of work to do.

[myselfghost]

+----------+                           +----------+                           +----------+                          +----------+
|          |                           |          |<---(C)---connect&auth ----|          |                          |          |
|  Server  |                           |  Server  |                           |  Client  |                          |  Client  |
|  Disk    |                           |          |----(D)--deliver policy--->|          |                          |  Disk    |
|          |                           |          |                           |          |                          |          |
|          |<---(H)--- read file   ----|          |<---(G)-- pull file    ----|          |                          |          |
|          |                           |          |                           |          |                          |          |
|          |----(I)--- return file --->|          |----(J)--- send file   --->|          |----(K)--- write file --->|          |
|          |                           |          |                           |          |                          |          |
|          |                           |          |<---(M)-- report result----|          |<---(L)-- write result----|          |
+----------+                           +----------+                           +--|-------+                          +----------+
                                          ^    ^                                 |    ^ 
                        (A) create policy |    |           (E) add to task queue |    |   
                                          |    |                                 |    |
                                          |    | (B) assign policy               v    | (F) execute task  
                                       +--+-—--|--+                           +-------|--+
                                       |          |                           |          |
                                       |  Admin   |                           |  Task    |
                                       |          |                           |  Queue   |
                                       |          |                           |          |
                                       |          |                           |          |
                                       |          |                           |          |
                                       +----------+                           +----------+

The above is a simplified flow diagram, and there's a lot of work to do.

It is indeed a lot of work to develop into a mature product, and there is no rush. We can think carefully about the actual usage scenarios, slowly design the product architecture, and then develop it.

[mstmdev]

Yes, in order to better carry out this work, we can do some review and refactoring before starting.

[myselfghost]

+----------+                           +----------+                           +----------+                          +----------+
|          |                           |          |<---(C)---connect&auth ----|          |                          |          |
|  Server  |                           |  Server  |                           |  Client  |                          |  Client  |
|  Disk    |                           |          |----(D)--deliver policy--->|          |                          |  Disk    |
|          |                           |          |                           |          |                          |          |
|          |<---(H)--- read file   ----|          |<---(G)-- pull file    ----|          |                          |          |
|          |                           |          |                           |          |                          |          |
|          |----(I)--- return file --->|          |----(J)--- send file   --->|          |----(K)--- write file --->|          |
|          |                           |          |                           |          |                          |          |
|          |                           |          |<---(M)-- report result----|          |<---(L)-- write result----|          |
+----------+                           +----------+                           +--|-------+                          +----------+
                                          ^    ^                                 |    ^ 
                        (A) create policy |    |           (E) add to task queue |    |   
                                          |    |                                 |    |
                                          |    | (B) assign policy               v    | (F) execute task  
                                       +--+-—--|--+                           +-------|--+
                                       |          |                           |          |
                                       |  Admin   |                           |  Task    |
                                       |          |                           |  Queue   |
                                       |          |                           |          |
                                       |          |                           |          |
                                       |          |                           |          |
                                       +----------+                           +----------+

The above is a simplified flow diagram, and there's a lot of work to do.

Let me add one more thought. The client receives a policy from the server, and the actions performed by this policy should have multiple possibilities. for example:

1. Pull files from server
2. Push files to the server;
3. Pull files from other addresses such as minio or sftp;
4. Push files to other addresses such as minio or sftp;

[mstmdev]

Should the client support multiple tasks at the same time? We need an isolated context to handle multiple tasks.

[myselfghost]

Should the client support multiple tasks at the same time? We need an isolated context to handle multiple tasks.

maybe the client should support multiple tasks

One of the scenarios:

Machine C, path /data/p3, backed up once a week, push files to server
Machine C, path /data/p4, backed up once a month,push files to minio

[mstmdev]

+----------+                           +----------+                           +----------+                           +----------+                          +----------+
|          |                           |          |<---(C)---connect&auth ----|  Client  |                           |          |                          |          |
|  Server  |                           |  Server  |                           |  Manager |  (process or goroutine)   |  Client  |                          |  Client  |
|  Disk    |                           |          |----(D)--deliver policy--->|          |----(E)-- start worker --->|  Worker  |                          |  Disk    |
|          |                           |          |                           +----------+                           |          |                          |          |
|          |                           |          |<-----------------(H)--- connect&auth ----------------------------|          |                          |          |
|          |<---(J)--- read file   ----|          |<-----------------(I)--- pull file -------------------------------|          |                          |          |
|          |                           |          |                                                                  |          |                          |          |
|          |----(K)--- return file --->|          |------------------(L)--- send file ------------------------------>|          |----(M)--- write file --->|          |
|          |                           |          |                                                                  |          |                          |          |
|          |                           |          |<-----------------(O)--- report result ---------------------------|          |<---(N)-- write result----|          |
+----------+                           +----------+                                                                  +----------+                          +----------+
                                          ^    ^                                                                        |    ^ 
                        (A) create policy |    |                                                  (F) add to task queue |    |   
                                          |    |                                                                        |    |
                                          |    | (B) assign policy                                                      v    | (G) execute task  
                                       +--+-—--|--+                                                                  +-------|--+
                                       |          |                                                                  |          |
                                       |  Admin   |                                                                  |  Task    |
                                       |          |                                                                  |  Queue   |
                                       |          |                                                                  |          |
                                       |          |                                                                  |          |
                                       |          |                                                                  |          |
                                       +----------+                                                                  +----------+

For better isolation, consider creating a process for each task, something like the Daemon Mode.

[mstmdev]

WIP #157

[mstmdev]

See Task Mode