[WIP] IPFS Cluster Integration #831

sanderpick · 2019-06-16T00:14:03Z

Adds the ability to sidecar IPFS Cluster on Textile.

Changelog

coming

Closes

Closes Cafe peers should optionally use IPFS Cluster #734

Fixes

The --repo flag was no longer parsing ~ (missed in the kingpin refactor)

TODO

Implement IPFSConnector (ipfs/cluster.go)

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cmd/main.go

Signed-off-by: Sander Pick <sanderpick@gmail.com>

sanderpick · 2019-06-17T00:46:25Z

Ok, @hsanjuan - I've got all the pieces in place here. However, running into an authorization error when calling SyncAll (https://github.com/textileio/go-textile/blob/sander/cluster/cluster/main_test.go#L188):

17:31:31.842 ERROR  p2p-gorpc: error handling RPC: client does not have permissions to this method, service name: Cluster, method name: SyncAllLocal server.go:176
17:31:31.859 DEBUG    cluster: rpc auth error client does not have permissions to this method, service name: Cluster, method name: SyncAllLocal cluster.go:1597

If I understand correctly, the RPC auth is based solely on the cluster secret. They are matching in the test-generated config files.

I'm putting this down for now but let me know if you have any quick ideas...

Signed-off-by: Sander Pick <sanderpick@gmail.com>

sanderpick · 2019-06-17T07:22:14Z

Something occurred to me at dinner... I think the intention here was to not use a secret since the host needs to be on the main IPFS network. Instead, use the new trusted peers mechanism. That seems to work, but so far I haven't been able to get the pin state of the local daemon to reflect in the cluster consensus. More later...

hsanjuan · 2019-06-19T13:16:38Z

Something occurred to me at dinner... I think the intention here was to not use a secret since the host needs to be on the main IPFS network. Instead, use the new trusted peers mechanism. That seems to work, but so far I haven't been able to get the pin state of the local daemon to reflect in the cluster consensus. More later...

Yes, correct. You will need to set TrustedPeers to the peer IDs of your other cafes (suggestions to improve this for your usecase accepted). Cluster secret is not used at all since you already have a host without a network protector.

Can I get write access here, I will have a look to the code and it may be useful now or in the future to commit some changes to the branch (simpler than a PR to the PR :P)

hsanjuan · 2019-06-19T13:23:02Z

cluster/connector.go

+}
+
+func (c *Connector) SetClient(client *rpc.Client) {
+	// noop


Here and in Shutdown you will need to copy from https://github.com/ipfs/ipfs-cluster/blob/master/ipfsconn/ipfshttp/ipfshttp.go#L191

The main Cluster component will SetClient with an RPC client that allows this component to talk to any component in any peer in the Cluster. It is also the signal that the component can move forward with any internal tasks (the normal IPFSConnector implementation will wait for a bit and trigger an automatic ipfs swarm connect to every other daemon attached to every other known cluster peer).

Actually, since you don't use rpc at all, it's actually ok like this!

hsanjuan · 2019-06-19T13:34:04Z

cluster/connector.go

+	return statusMap, nil
+}
+
+func (c *Connector) ConnectSwarms(ctx context.Context) error {


For information: this will get triggered when a new cafe "bootstraps" (uses Cluster.Join(<peer maddr>)). It will be called in the cluster peer it bootstraps to. If textile never calls Join() to introduce new peers to the Cluster*, then it can be a noop.

*Join() is not strictly necessary with CRDTs, but it is a way of getting two libp2p hosts in the same cluster connected (and this should 1) allow pubsub to start working (otherwise the peer may never receive pubsub messages because it doesn't know any of the other peers subscribed to them), 2) Potentially allow dht-discovery of other cluster peers and better connectivity.

hsanjuan · 2019-06-19T13:36:35Z

cluster/connector.go

+	return nil
+}
+
+func (c *Connector) SwarmPeers(ctx context.Context) ([]peer.ID, error) {


Only used by ConnectGraph (which generates a .dot file with cluster and ipfs daemons connections).

Ah, nice that's cool

hsanjuan · 2019-06-19T13:38:14Z

cluster/connector.go

+}
+
+func (c *Connector) RepoStat(ctx context.Context) (*api.IPFSRepoStat, error) {
+	stat, err := corerepo.RepoStat(ctx, c.node)


To check: we call ipfs with size-only=true. https://github.com/ipfs/ipfs-cluster/blob/master/ipfsconn/ipfshttp/ipfshttp.go#L600

This makes the RepoStat call WAY faster (no full node count).

Here should be calling RepoSize: https://github.com/ipsn/go-ipfs/blob/master/core/corerepo/stat.go#L66

hsanjuan · 2019-06-19T13:42:54Z

cluster/main.go

+		return err
+	}
+
+	if listenAddr != "" {


Interesting, I think we don't use it. This is another of the options (like secret) which are only used to configure the libp2p host and not in Cluster per-se.

However the config will fail to validate if unset. I think it makes sense to set it to the real listen endpoint of the cafe (like here).

Ok, cool. Should be easy enough to just copy the value used in the IPFS config.

hsanjuan · 2019-06-19T13:48:31Z

cluster/main.go

+	numpinInfCfg *numpin.Config,
+) (ipfscluster.Informer, ipfscluster.PinAllocator, error) {
+	switch name {
+	case "disk", "disk-freespace":


Probably you want to default to this one. The others are not so useful, but maybe a good placeholder to have this for the future.

hsanjuan · 2019-06-19T13:51:12Z

cluster/main.go

+	peerName string,
+) (ipfscluster.PinTracker, error) {
+	switch name {
+	case "map":


for the moment defaulting to this one makes most sense. We want to level up the stateless one.

hsanjuan · 2019-06-19T13:54:30Z

core/cluster.go

+		return err
+	}
+
+	ipfscluster.ReadyTimeout = raft.DefaultWaitForLeaderTimeout + 5*time.Second


I think you can ignore this (like set to 5 seconds or leave default). Raft is not involved anyways.

hsanjuan · 2019-06-19T13:54:49Z

cmd/main.go

@@ -436,6 +436,9 @@ Stacks may include:
 	initCafeOpen := initCmd.Flag("cafe-open", "Open the p2p cafe service for other peers").Bool()
 	initCafeURL := initCmd.Flag("cafe-url", "Specify a custom URL of this cafe, e.g., https://mycafe.com").Envar("CAFE_HOST_URL").String()
 	initCafeNeighborURL := initCmd.Flag("cafe-neighbor-url", "Specify the URL of a secondary cafe. Must return cafe info, e.g., via a Gateway: https://my-gateway.yolo.com/cafe, or a cafe API: https://my-cafe.yolo.com").Envar("CAFE_HOST_NEIGHBOR_URL").String()
+	initIpfsCluster := initCmd.Flag("cluster", "Treat the node as an IPFS Cluster peer").Bool()
+	initIpfsClusterBindMultiaddr := initCmd.Flag("cluster-bind-maddr", "Set the IPFS Cluster multiaddrs").Default("/ip4/0.0.0.0/tcp/9096").String()


technically cluster won't bind to anything as it re-uses your already existing ipfs peer.

Ah, ok. That makes sense. I can remove this flag then.

hsanjuan · 2019-06-19T13:58:38Z

core/cluster.go

+
+	// noop if no bootstraps
+	// if bootstrapping fails, consensus will never be ready
+	// and timeout. So this can happen in background and we


this is not true for CRDT consensus. it will work regardless of bootstrap. Old comment in cluster, we will fix it.

sanderpick · 2019-06-19T22:28:56Z

Thanks for the comments @hsanjuan! You should have write access now.

hsanjuan

I have added some more comments. I don't know if you want to keep the functions like SetupConsensus etc. since the user cannot really choose, so I think you can just create the right component directly simplifies and reduces the amount of code.

hsanjuan · 2019-07-11T11:43:14Z

cluster/connector.go

+
+// @todo handle maxDepth
+func (c *Connector) Pin(ctx context.Context, cid icid.Cid, maxDepth int) error {
+	return c.api.Pin().Add(ctx, path.New(cid.String()))


For record. The original ipfs-cluster connector, by default, calls refs -r <cid> and then pin. The reason is many refs -r can happen in parallel, while only one pin does. For the general case, this way works better.

On the other hand, refs -r uses a non-parallel dag walking approach while pin uses the async, faster version.

The original cluster-pin also uses the RPC to request an update of the freespace metrics every 10th pin.

hsanjuan · 2019-07-11T11:45:45Z

cluster/connector.go

+}
+
+func (c *Connector) RepoStat(ctx context.Context) (*api.IPFSRepoStat, error) {
+	stat, err := corerepo.RepoStat(ctx, c.node)


Here should be calling RepoSize: https://github.com/ipsn/go-ipfs/blob/master/core/corerepo/stat.go#L66

hsanjuan · 2019-07-11T11:48:44Z

cluster/main.go

+		cfgs.ClusterCfg.ListenAddr = addr
+	}
+
+	return cfgMgr.SaveJSON(ConfigPath(repoPath))


I think you don't want your users to modify cluster configs, rather you want to give them something that "just works" and let them adjust some paramemeters via go-textile flags? how's the approach with the embedded ipfs daemon now, can they manually edit its configuration?

hsanjuan · 2019-07-11T12:25:51Z

cluster/main.go

+			log.Errorf("bootstrap to %s failed: %s", bstrap, err)
+		} else {
+			for _, p := range cluster.Peers(ctx) {
+				err = cons.Trust(ctx, p.ID)


Thinking about this. The bootstrap concept in cluster is part of the daemon binary (not of the cluster library itself), so while we can fix this there (ipfs-cluster/ipfs-cluster#834), it won't help you.

hsanjuan · 2019-07-11T12:26:50Z

cluster/main.go

+	clusterCfg := &ipfscluster.Config{}
+	crdtCfg := &crdt.Config{}
+	maptrackerCfg := &maptracker.Config{}
+	statelessCfg := &stateless.Config{}


I think not worth registering if not using, same for numpinInformer.

sanderpick added 4 commits June 15, 2019 13:32

cluster: initial cluster integration

d7394fa

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: add config and lifecycle management

a117256

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: add flags to init

8bb596e

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: clean up flags

8a865c6

Signed-off-by: Sander Pick <sanderpick@gmail.com>

sanderpick requested review from andrewxhill, asutula, balupton and carsonfarmer as code owners June 16, 2019 00:14

sanderpick self-assigned this Jun 16, 2019

This comment has been minimized.

Sign in to view

sanderpick commented Jun 16, 2019

View reviewed changes

cmd/main.go Show resolved Hide resolved

This comment has been minimized.

Sign in to view

sanderpick added 6 commits June 16, 2019 12:36

cluster: implement IPFSConnector

bdb6b69

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: adds two node test

12c2ae0

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: move to own package

4e5a3fc

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: fix ugorji/go module issue

d0a476d

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: cleanups two node test

492644d

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: adds bind multiaddr setting

df42c22

Signed-off-by: Sander Pick <sanderpick@gmail.com>

sanderpick force-pushed the sander/cluster branch from ef2871c to df42c22 Compare June 17, 2019 00:38

sanderpick added 2 commits June 17, 2019 00:01

cluster: remove secret, use trusted peers

e9e6400

Signed-off-by: Sander Pick <sanderpick@gmail.com>

cluster: move connector to cluster package

839b267

Signed-off-by: Sander Pick <sanderpick@gmail.com>

hsanjuan reviewed Jun 19, 2019

View reviewed changes

hsanjuan reviewed Jul 11, 2019

View reviewed changes

sanderpick added the on-hold waiting for an event to happen label Nov 21, 2019

sanderpick removed their assignment Jan 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] IPFS Cluster Integration #831

[WIP] IPFS Cluster Integration #831

sanderpick commented Jun 16, 2019 •

edited

This comment has been minimized.

This comment has been minimized.

sanderpick commented Jun 17, 2019 •

edited

sanderpick commented Jun 17, 2019

hsanjuan commented Jun 19, 2019

hsanjuan Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jul 11, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

hsanjuan Jun 19, 2019

sanderpick Jun 19, 2019

sanderpick commented Jun 19, 2019

hsanjuan left a comment

hsanjuan Jul 11, 2019

hsanjuan Jul 11, 2019

hsanjuan Jul 11, 2019

hsanjuan Jul 11, 2019

hsanjuan Jul 11, 2019

[WIP] IPFS Cluster Integration #831

Are you sure you want to change the base?

[WIP] IPFS Cluster Integration #831

Conversation

sanderpick commented Jun 16, 2019 • edited

Changelog

Closes

Fixes

This comment has been minimized.

This comment has been minimized.

sanderpick commented Jun 17, 2019 • edited

sanderpick commented Jun 17, 2019

hsanjuan commented Jun 19, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderpick commented Jun 19, 2019

hsanjuan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanderpick commented Jun 16, 2019 •

edited

sanderpick commented Jun 17, 2019 •

edited