-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs/Compositing: composite cluster usecases. #278
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hsanjuan Overall this is great. I see the value of composite clusters more clearly already. I don't have any answers right now regarding the many open questions you've posed but they are now on my mind. Not sure if it belongs in this document but an important next step is figuring out exactly what to do next with these ideas. I am thinking we will need to nail down enough questions in "The IPFS-like cluster API" to know what we are are building and we will need to get a sense of the answers to your useability questions under "Administrative overhead" in order to select a use case that composite clusters can significantly impact so that we can build towards something concrete. Of course this is just my first take away.
docs/composite-clusters.md
Outdated
|
||
## Introduction | ||
|
||
The idea of composite ipfs-clusters is based on the capacity of a cluster node to provide an ipfs-proxy endpoint in which HTTP requests are partially hijacked and used to trigger cluster operations or retrieve cluster information. This means that ipfs-clusters offers a partial ipfs-compatible API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would consider saying "partial IPFS API", it's not about cluster implementing an api compatible with ipfs use, it's about cluster implementing the same api as ipfs.
|
||
#### Uses | ||
|
||
##### Destroy the world |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docs/composite-clusters.md
Outdated
|
||
* Overcluster: the cluster at the top of a cluster composition topology | ||
* Undercluster(s): the clusters at the bottom of a cluster composition topology | ||
* "Peer attached to a peer": A cluster peer that uses the ipfs-proxy-endpoint provided by another peer as its IPFS daemon address. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the over and under terminology. It feels a bit off to call the nodes in the undercluster the "peers" of the nodes in the overcluster so I would go with something like "provided by another cluster daemon" and "peer attached to a cluster". Though perhaps I have the wrong idea about the relationship of cluster nodes in over and underclusters.
* Undercluster(s): the clusters at the bottom of a cluster composition topology | ||
* "Peer attached to a peer": A cluster peer that uses the ipfs-proxy-endpoint provided by another peer as its IPFS daemon address. | ||
|
||
## Topologies |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The document is well organized for understanding.
docs/composite-clusters.md
Outdated
|
||
The most straightforward cluster composition topology consists of a regular ipfs-cluster where the IPFS-endpoints for each cluster peer point to underlying, separate ipfs-clusters. | ||
|
||
Todo: add drawing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm thinking if the ipfs daemons should be represented too?
docs/composite-clusters.md
Outdated
|
||
In this topology, each cluster peers uses as ipfs daemon another cluster peer from a different cluster, but all underlying peers belong to the same cluster. | ||
|
||
Todo: drawing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docs/composite-clusters.md
Outdated
|
||
##### Data partitioning | ||
|
||
A group of parties may which to share a common cluster. With an inverse multi-ring setup, each party will mantain their own pinset in their own cluster, but it will use a common infrastructure (the undercluster), which is shared with others. Each overcluster may administer how cluster APIs are exposed and what authentication to use. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am having a hard time understanding why this is something people would want to do. In what situation would this be desirable over everyone banding together into a single cluster?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Basically, maintaining a separate pinset, not worrying about ipfs setup, and being able to take down a cluster without affecting other people/clusters. Of course, most of it is just delegating problems to the person maintaining the under cluster.
|
||
### Administration overhead | ||
|
||
* How hard is it to manage a composite cluster vs. managing a single one? Is it worth it to the use-cases? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 This is going to be key to figure out when planning which features to tackle with composite approaches.
|
||
* How hard is it to manage a composite cluster vs. managing a single one? Is it worth it to the use-cases? | ||
* What happens when there are errors, how are they handled in the different levels? Is the handling appropiate? | ||
* Are there alternative ways of handling usecases, like pubsub? Are they better? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of these usecases could potentially be addressed by building features into simple clusters, of course doing so might be a lot harder.
This is a specific case of **inverse multi-ring topology** described below, with a single overcluster. | ||
|
||
### Multiple clusters have their cluster peers attached to peers from a single cluster (inverse multi-ring) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding some thoughts about how composite clusters can be organized and what to do with that. License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
License: MIT Signed-off-by: Hector Sanjuan <code@hector.link>
40f515c
to
fca83cc
Compare
There were the following issues with your Pull Request
Code contribution guidelines for IPFS Cluster are available at https://cluster.ipfs.io/developer/contribute/#code-contribution-guidelines This message was auto-generated by https://gitcop.com |
We need to move this to the website docs |
Adding some thoughts about how composite clusters
can be organized and what to do with that.
License: MIT
Signed-off-by: Hector Sanjuan code@hector.link