Unknown Recover Status

[gus4rs]

After trying to recover channels using the static backup file, with

/usr/local/bin/lncli --macaroonpath /home/bob/.lnd/data/chain/bitcoin/testnet/admin.macaroon --tlscertpath /home/bob/.lnd/tls.cert    restorechanbackup --multi_file /home/bob/channel.backup 

The command just returns without any feedback at all

Looking at pendingchannels:

 {
            "channel":  {
                "remote_node_pub":  "03781444afe27babf724304c25370f195d9d955065d7894539ed7db5fe596badb8",
                "channel_point":  "aa85d75273793fe5e613a19db199d1aa05eb6d8327e1ae89d9cd5a509b7ed8a5:0",
                "capacity":  "9994278",
                "local_balance":  "0",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "0",
                "remote_chan_reserve_sat":  "0",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusLocalDataLoss|ChanStatusRestored",
                "private":  true,
                "memo":  ""
            },
            "limbo_balance":  "0",
            "commitments":  {
                "local_txid":  "",
                "remote_txid":  "",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "0",
                "remote_commit_fee_sat":  "0",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  ""
        }

Nothing useful is presented

Surprising that after so many years the recover UX is that bad. Any clues what is happening?

[guggero]

It looks like the channel never made it on-chain. At least the transaction shown as "channel point" isn't confirmed and not in my mempool. So without more information I would assume there is nothing to recover.

[gus4rs]

I purposely deleted all files to test recovery, except for the SCB, and kept seed/passphrase. There is no way to avoid data loss or corruption, as hardware fail and power source can go down

The channels were opened only a few days ago, so the fees were already high enough. Both peers are online, I tried to use chantools to manually trigger a force close, there was some communication, but the force close never happened. This is a bit worrying, I am trying to get some confidence to use mainnet

[guggero]

The problem is that with the current mempool situation on testnet, any transaction below 16 sat/vByte cannot even make it into the mempool, which is very likely what's happening (the default max fee rate for the anchor channel commitment transaction fee rate is 10 sat/vByte, you can change that with --max-commit-fee-rate-anchors=, but both parties of a channel need to do that). So for your scenario testnet is actually less reliable than mainnet in this regard.

[gus4rs]

My understanding is mainchain is even worse, since ordinals started, fees are pushing above 2000 sat/vByte sometimes. So, assuming all my peers are lnd, on mainchain I'd be waiting for fees to go lower than 10 to have a force close?

[guggero]

No. I'm talking about the mempool minimum fee (also called "purge rate", meaning: what's the lowest fee rate a transaction needs to have to be accepted into a default mempool of 300 MB). Once your transaction is in the mempool, you can do CPFP, which is the whole point of anchor channels.
The current mempool purge rate of mainnet is 4.5 sat/vByte while on testnet it is 15 sat/vByte.

[gus4rs]

Right, got it

[gus4rs]

One of the channels was closed, as testnet is now only purging < 6 sats/vByte

The other channel is still unclosed, although the peer is alive and I can connect to it:

$ chantools triggerforceclose -t --peer 03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62@34.224.22.127:9735 --channel_point e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0
2024-05-15 06:30:18.955 [INF] CHAN: No spends found yet, waiting 5 seconds...
...
...

[guggero]

Are you using the latest version of chantools? The triggerforceclose only works for non-CLN peers in the very last released version (v0.13.1). What do you see in the logs for that channel? I mean on the node where you used the SCB file.

[gus4rs]

Yes, using 0.13.1

Here are the logs:

[INF] PEER: Peer(03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62): unable to read message from peer: read next header: EOF
[INF] PEER: Peer(03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62): disconnecting 03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62@34.224.22.127:9735, reason: read handler closed
[INF] NTFN: Cancelling epoch notification, epoch_id=14
[INF] DISC: Removing GossipSyncer for peer=03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62
[INF] SRVR: Established connection to: 03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62@34.224.22.127:9735
[INF] SRVR: Finalizing connection to 03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62@34.224.22.127:9735, inbound=false
[INF] PEER: Peer(03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62): Loading ChannelPoint(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0), isPending=false
[WRN] PEER: Peer(03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62): Unable to find our forwarding policy for channel e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0, using default values
[INF] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): starting
[INF] HSWC: Trimming open circuits for chan_id=2814744:1898:0, start_htlc_id=0
[INF] HSWC: Adding live link chan_id=8a160f5139fd065311d8394e1d2eedee9f2161660f801bd446f44265111758e4, short_chan_id=2814744:1898:0
[INF] CNCT: Attempting to update ContractSignals for ChannelPoint(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0)
[INF] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): HTLC manager started, bandwidth=0 mSAT
[INF] PEER: Peer(03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62): Negotiated chan series queries
[INF] NTFN: New block epoch subscription
[INF] DISC: Creating new GossipSyncer for peer=03345d4d00a5a65535513b739e78566c24d1f34a3f853814daf4fbb37f413fbf62
[INF] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): Attempting to re-synchronize channel: SCID=2814744:1898:0, status=ChanStatusLocalDataLoss|ChanStatusRestored, initiator=true, pending=false, local commitment has height=0, local_htlc_index=0, local_log_index=0, remote_htlc_index=0, remote_log_index=0, remote commitment has height=0, local_htlc_index=0, local_log_index=0, remote_htlc_index=0, remote_log_index=0
[INF] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): received re-establishment message from remote side
[ERR] LNWL: ChannelPoint(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): sync failed with local data loss: remote believes our tail height is 129, while we have 0!
[WRN] LNWL: ChannelPoint(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): detected restored triggering DLP
[WRN] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): error when syncing channel states: ChannelPoint(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0) with CommitPoint(036a56be8c5e26964271700c0b2bee73f7080edb4452e886cfa0d54c12bdc6b6df) had possible local commitment state data loss
[ERR] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): failing link: unable to synchronize channel states: ChannelPoint(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0) with CommitPoint(036a56be8c5e26964271700c0b2bee73f7080edb4452e886cfa0d54c12bdc6b6df) had possible local commitment state data loss with error: unable to resume channel, recovery required
[INF] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): exited
[INF] HSWC: ChannelLink(e45817116542f446d41b800f6661219feeed2e1d4e39d8115306fd39510f168a:0): stopping
[INF] HSWC: Removing channel link with ChannelID(8a160f5139fd065311d8394e1d2eedee9f2161660f801bd446f44265111758e4)

[guggero]

That looks like the expected entries, the node responds with the correct error code. So my guess is they are trying to publish the transaction but can't. Perhaps the last negotiated fee rate is still below the current mempool purge rate.

[gus4rs]

UPDATE: as testnet3 is now purging < 1.66 sats/vB, chantools was able to see the closing transaction:

[INF] CHAN: Message sent, waiting for force close transaction to appear in mempool
[INF] CHAN: Found force close transaction 488ceb90119e243abc1983e866a0b13bef69d70a0b57a200093babd6f946972b
[INF] CHAN: You can now use the sweepremoteclosed command to sweep the funds from the channel