Log Entries created on remote/other Peers are not Pinned Locally

[justin0mcateer]

In the previous implementation of OpLog.joinEntry, Log entries retrieved from other nodes were 'put' in the IPFSBlockStorage and optionally (via configuration) pinned. In the rewrite of OpLog.joinEntry _entries.put is no longer called and so the remotely created Log entries are no longer stored locally.

This results in the behavior that all Peers which have previously written to the log MUST be online or traversal of the Log will fail and unhandled exception in IPFSBlockStorage.get. This does not seem to be desirable behavior.

This code will crash with an unhandled promise rejection if any block fails to be fetched, which currently appears to occur if any previous log writer is offline:
https://github.com/orbitdb/orbitdb/blob/main/src/storage/ipfs-block.js#L61

In older versions of the code (prior to the OpLog.joinEntry rewrite), fetched OpLog entries were persisted locally:
https://github.com/orbitdb/orbitdb/blob/v1.0.0/src/oplog/log.js#L269

[haydenyoung]

Is the peer using an in-memory blockstore for Helia? What happens if you configure the node with a persistent blockstore?

E.g.

import { LevelBlockstore } from 'blockstore-level' 

const blockstore = new LevelBlockstore('./some/path')
const libp2p = await createLibp2p()
const helia = await createHelia({ libp2p, blockstore })

My understanding is that the replicated block should also be pinned on the remote peer (isPinned should == "true" on the remote peer also).

[haydenyoung]

Just to add that a persistent datastore may also be needed as I believe pins are stored to datastore not blockstore.

import { LevelBlockstore } from 'blockstore-level' 
import { LevelDatastore } from 'datastore-level' 

const blockstore = new LevelBlockstore('./some/path/helia/blocks')
const datastore = new LevelDatastore('./some/path/helia/data')
const libp2p = await createLibp2p()
const helia = await createHelia({ libp2p, blockstore, datastore })

(sorry haven't been able to actually confirm this, just going on a hunch re: datastore and pins).

[justin0mcateer]

We are using a persistent store (IDB) for both. What I am saying is that the new logic doesn't call Helia.put for remote originated blocks at all, at least from what I can see. There was an explicit call to _entries.put prior to the joinEntry rewrite. I've looked and I can't see it being done anywhere else. We are seeing block not found errors when not all the nodes are online.

…

________________________________ From: Hayden Young ***@***.***> Sent: Monday, February 26, 2024 12:22:01 PM To: orbitdb/orbitdb ***@***.***> Cc: justin0mcateer ***@***.***>; Author ***@***.***> Subject: Re: [orbitdb/orbitdb] Log Entries created on remote/other Peers are not Pinned Locally (Issue #1163) Just to add that a persistent datastore may also be needed as I believe pins are stored to datastore not blockstore. import { LevelBlockstore } from 'blockstore-level' import { LevelDatastore } from 'datastore-level' const blockstore = new LevelBlockstore('./some/path/helia/blocks') const datastore = new LevelDatastore('./some/path/heliadata') const libp2p = await createLibp2p() const helia = await createHelia({ libp2p, blockstore, datastore }) (sorry haven't been able to actually confirm this, just going on a hunch re: datastore and pins). — Reply to this email directly, view it on GitHub<#1163 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAT4Q7NJDXIGQIMQZIPKGH3YVTHETAVCNFSM6AAAAABDVIFX3KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRUHAZTENJZGI>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[justin0mcateer]

My understanding is that the replicated block should also be pinned on the remote peer (isPinned should == "true" on the remote peer also).

I am not really sure what you are referring to here. The only way to pin anything in Helia is either locally, or via the 'Pinning API'. There isn't any way to compel another node to Pin your data.

[haydenyoung]

Does IPFSBlockStorage fail immediately (I.e. as soon as the remote node is offline) or does it happen over time (I.e. the remote node has been offline for some time)? What would be the steps for me to reproduce this issue?