[bug]: new version of CLN and maybe eclair won't force close channel from SCB request

[btweenthebars]

Background

cdecker has it that there's a change in the bolt spec on the work flow of SCB that hasn't been implemented on LND

Please take a look here .

Your environment

v0.15.5-beta

Steps to reproduce

I mentioned my incident in here as I wasn't sure what's going on

[Roasbeef]

So we investigated a bit, and concluded that we were already sending the Error once the SCB flow was triggered. So instead it seems some other behavior changed here that is preventing some other implementations from closing either once we send that Error, or just from receiving the purposefully incorrect sync message.

[Crypt-iQ]

If you are experiencing this issue on v0.15.5-beta, the following steps should resolve the issue:

• Ensure that you have already tried connecting to the CLN node and received Warning followed by a disconnect.
• Checkout https://github.com/Crypt-iQ/lnd/tree/scb_cln_issue_7301_v0.15.5-beta
• Build lnd with the dev build tag.
• Boot up the newer version of lnd with the flag --protocol.custom-message=17.
• If you have logs, try to fetch the 32-byte (64 hexadecimal characters) chan_id/ChannelID/channel_id value. This may be appear with ChannelLink logs in the HSWC category. If you can't fetch it, feel free to post a comment asking for help.
• Once you have the 32-byte chan_id/ChannelID/channel_id value, fetch the peer's hexadecimal pubkey. This should appear in listpeers if you are connected to the peer or in some of your logs.
• Finally, use the sendcustom API like so:

lncli sendcustom --peer=<peer pubkey in hex> --type=17 --data=`chan_id`0000

• Note that the data field in the sendcustom call is the chan_id followed by 4 zeros.
• Once you see that your channel has been force closed via the lnd API, continue using the original v0.15.5-beta. This patch is only meant to be used temporarily.

If you are uncomfortable with using my personal branch and would instead prefer a temporary release, please let us know. Or if you don't know how to perform any of the steps above, leave a comment and I can help out.

[Crypt-iQ]

@btweenthebars were you able to resolve your issue using the steps above?

[btweenthebars]

@btweenthebars were you able to resolve your issue using the steps above?

Hi sorry, I'm on CLN ( the latest version) not LND

[Crypt-iQ]

Can you contact your peer to see if he wants to use the steps to above?

[btweenthebars]

Can you contact your peer to see if he wants to use the steps to above?

Well, that's a while ago and I already manually closed the channels. Next time I see LND peers with the issue, I'll tell them to try the steps above.

[ardevd]

I'm struggling with this bug. I've build @Crypt-iQ 's lnd version, but I can't seem to find the channel ID. Nothing in the logs with the HSWC tag or any mention of CahnnelLink anywhere.

[Crypt-iQ]

I'm struggling with this bug. I've build @Crypt-iQ 's lnd version, but I can't seem to find the channel ID. Nothing in the logs with the HSWC tag or any mention of CahnnelLink anywhere.

What's the funding point of the channel?

[ardevd]

I'm struggling with this bug. I've build @Crypt-iQ 's lnd version, but I can't seem to find the channel ID. Nothing in the logs with the HSWC tag or any mention of CahnnelLink anywhere.

What's the funding point of the channel?

d51881f36667cc318d5309bb4699254e26d7360a6c9603003c3c649e1b6ef76d:1

[Crypt-iQ]

d51881f36667cc318d5309bb4699254e26d7360a6c9603003c3c649e1b6ef76d:1

your channel id to use in the command should be d51881f36667cc318d5309bb4699254e26d7360a6c9603003c3c649e1b6ef76c

[ardevd]

Thanks. Care to share how it's calculated? why was f76d changed to f76c?

[guggero]

Now that I saw the example, I remember... I think it's XOR of the last byte (0x6d) and the outpoint index (0x01) in this case. So if the outpoint index (the part after the colon) is 0, then the channel ID is equal to the TXID (the part before the colon), if it's something else, you need to XOR.

[ardevd]

Btw, @Crypt-iQ your scb_cln_issue_7301_v0.15.5-beta branch doesnt seem to contain support for protocol.custom-message. I had to use the ``scb_cln_issue_7301` branch.

[ardevd]

@Crypt-iQ is this the expected log output after sending the custom message?

2023-02-23 18:53:33.140 [DBG] PEER: Peer(0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754): Sending Error(type=17) to 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735
2023-02-23 18:53:33.301 [DBG] PEER: Peer(0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754): Received Error(chan_id=89b72329e1c276633de91fed43ffc2a233f7a6bf503bb0b60ae65929dc797d93, err=Unknown channel for WIRE_ERROR) from 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735
2023-02-23 18:53:33.301 [INF] PEER: Peer(0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754): unable to read message from peer: EOF
2023-02-23 18:53:33.301 [INF] PEER: Peer(0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754): disconnecting 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735, reason: read handler closed
2023-02-23 18:53:33.301 [INF] NTFN: Cancelling epoch notification, epoch_id=10
2023-02-23 18:53:33.401 [DBG] SRVR: Peer 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735 has been disconnected
2023-02-23 18:53:33.402 [DBG] FNDG: Cancelling all reservations for peer 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754
2023-02-23 18:53:33.402 [DBG] FNDG: No active reservations for node: 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754
2023-02-23 18:53:33.402 [INF] DISC: Removing GossipSyncer for peer=0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754
2023-02-23 18:53:33.402 [DBG] SRVR: Notifying that peer 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735 is offline
2023-02-23 18:53:33.402 [DBG] SRVR: removing peer 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735
2023-02-23 18:53:33.402 [DBG] PRNF: PeerNotifier notifying peer: 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754 offline
2023-02-23 18:53:33.402 [DBG] BTCN: Disconnected from 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735 (reqid 4)
2023-02-23 18:53:33.402 [DBG] DISC: Stopping GossipSyncer(0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754)

[Crypt-iQ]

Btw, @Crypt-iQ your scb_cln_issue_7301_v0.15.5-beta branch doesnt seem to contain support for protocol.custom-message. I had to use the ``scb_cln_issue_7301` branch.

Ah that's right, were you on v0.15.5-beta before?

2023-02-23 18:53:33.301 [DBG] PEER: Peer(0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754): Received Error(chan_id=89b72329e1c276633de91fed43ffc2a233f7a6bf503bb0b60ae65929dc797d93, err=Unknown channel for WIRE_ERROR) from 0317983322379d859c0d43a90c8dcd3e7239b8e0671b00a657ce3924d4498f3754@198.58.118.99:9735

That is not the expected output. That means CLN doesn't recognize the channel. This chan_id is also wrong. Can you paste the entire command of sendcustom?

[ardevd]

Yeah, that was for another channel. Apologies for the confusion.

Here is the one I discussed above:
./lncli sendcustom --peer=0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f --type=17 --data=d51881f36667cc318d5309bb4699254e26d7360a6c9603003c3c649e1b6ef76c0000

And output:

2023-02-23 19:01:48.008 [DBG] PEER: Peer(0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f): Sending Error(type=17) to 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f@73.114.40.246:9735
2023-02-23 19:01:48.029 [DBG] BTCN: Attempting to connect to 34.73.79.114:8333 (reqid 12)
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.100 [DBG] SRVR: Broadcasting 1 messages
2023-02-23 19:01:48.132 [DBG] PEER: Peer(0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f): Received Error(chan_id=d51881f36667cc318d5309bb4699254e26d7360a6c9603003c3c649e1b6ef76c, err=Unknown channel for WIRE_ERROR) from 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f@73.114.40.246:9735
2023-02-23 19:01:48.132 [INF] PEER: Peer(0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f): unable to read message from peer: EOF
2023-02-23 19:01:48.133 [INF] PEER: Peer(0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f): disconnecting 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f@73.114.40.246:9735, reason: read handler closed
2023-02-23 19:01:48.133 [INF] NTFN: Cancelling epoch notification, epoch_id=15
2023-02-23 19:01:48.143 [DBG] BTCN: Connected to 34.73.79.114:8333 (reqid 12)
2023-02-23 19:01:48.233 [DBG] SRVR: Peer 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f@73.114.40.246:9735 has been disconnected
2023-02-23 19:01:48.233 [DBG] FNDG: Cancelling all reservations for peer 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f
2023-02-23 19:01:48.233 [DBG] FNDG: No active reservations for node: 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f
2023-02-23 19:01:48.233 [INF] DISC: Removing GossipSyncer for peer=0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f
2023-02-23 19:01:48.233 [DBG] SRVR: Notifying that peer 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f@73.114.40.246:9735 is offline
2023-02-23 19:01:48.233 [DBG] SRVR: removing peer 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f@73.114.40.246:9735
2023-02-23 19:01:48.233 [DBG] PRNF: PeerNotifier notifying peer: 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f offline
2023-02-23 19:01:48.233 [DBG] DISC: Stopping GossipSyncer(0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f)
2023-02-23 19:01:48.233 [DBG] BTCN: Disconnected from 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f@73.114.40.246:9735 (reqid 11)
2023-02-23 19:01:48.233 [DBG] SRVR: Scheduling connection re-establishment to persistent peer 0298f6074a454a1f5345cb2a7c6f9fce206cd0bf675d177cdbf0ca7508dd28852f in 2.1s

[ardevd]

Btw, @Crypt-iQ your scb_cln_issue_7301_v0.15.5-beta branch doesnt seem to contain support for protocol.custom-message. I had to use the ``scb_cln_issue_7301` branch.

Ah that's right, were you on v0.15.5-beta before?

You wrote > Checkout https://github.com/Crypt-iQ/lnd/tree/scb_cln_issue_7301_v0.15.5-beta

so I tried to build that tree first and found that it didnt have support for the protocol.custom-message argument. The scb_cln_issue_7301 branch worked.

[ardevd]

@Crypt-iQ any idea why it's not working?

[guggero]

@ardevd I pushed up a new branch that builds on top of the just-tagged v0.16.0-beta.rc1 and adds a command to make it easier to use: https://github.com/lightningnetwork/lnd/tree/scb_cln_issue_7301_v0.16.0-beta.rc1

Here's the updated HOWTO:

• Checkout https://github.com/lightningnetwork/lnd/tree/scb_cln_issue_7301_v0.16.0-beta.rc1 (git clone https://github.com/lightningnetwork/lnd && cd lnd && git checkout scb_cln_issue_7301_v0.16.0-beta.rc1)
• Build lnd with the dev build tag (make install tags=dev)
• Boot up the newer version of lnd with the flag --protocol.custom-message=17.
• Finally, use the sendchannelerror command like so:

lncli sendchannelerror --peer=<peer pubkey in hex> --chan_point <channel_point_with_colon_notation>

• Once you see that your channel has been force closed via the lnd API, continue using the original v0.16.0-beta.rc1. This patch is only meant to be used temporarily.

Could you please try with that one again, just to make sure the issue isn't the channel ID?

[ardevd]

Worked like a dream. Force close tx was broadcast to the mempool shortly after running the command.

[guggero]

Given that the solution above might be a problem for users running Umbrel and other node packaging solutions where running a custom version of lnd is not easy, I created a specific new command in chantools to solve this.

Steps (alternative to running the custom version of lnd mentioned above):

• Download the latest version of chantools (v0.10.7) (can be any machine, use the darwin release for MacOS).
• Run the following command (replacing the peer address with the full address of the remote peer and the channel point with the channel in question):

chantools triggerforceclose --peer 039d9c531877266e190af2d51795b6104a1e91a188dad4b9ccf3ae4154075dd8e3@localhost:9747 --channel_point 8e3efb6f5537d964c1c02c0587f75fc6995888675083400d4b10c02f47ad3592:0

• The tool will ask you for your seed, which is required to initiate the encrypted p2p communication with the peer.
• If the channel is observed by lnd (e.g. showing up in the list of lncli pendingchannels), then just wait until the force close transaction is confirmed and the channel should automatically update the status there.
• If the channel isn't showing up in the list of pending channels, you can use the chantools sweepremoteclosed command to sweep the funds.

[guggero]

Ah, right, I forgot that you restored from SCB, sorry. Then you cannot force close yourself. Unfortunately there is nothing else you can do other than wait and hope the peer eventually force closes and the transaction makes it into the mempool somehow.

[wealthcollector]

Damn :(. Ok one last question for you. Since I could be waiting indefinitely for them to close this channel, is it ok to transfer the entire balance of my onchain wallet to cold storage or do I need to keep some amount on here in the event of a channel closure? I've read some people saying don't withdraw the funds until all channels are closed because of anchor something. Can't recall the wording. Would I be able to transfer most of it and leave just a few hundred thousands sats or something in the wallet?

[guggero]

Yes, depending on whether it is an anchor channel type, you might need some sats to pull the funds. But a few hundred thousand sats should be enough for that, you can move the rest to another wallet.

[Cmac0519]

I am trying to run the tool but keep getting cannot execute binary file: Exec format error

[guggero]

Download it for the right architecture/operating system.

[caiomsbr]

The new command works flawlessly!!! Thank you so much @guggero!! The might be a horde of nodes with the same problem out there! Now they'll find the solution here!! I wont be opening channels with CLN for the time being!

[Crypt-iQ]

Can we close this issue to remove it from the issue tracker and convert it to a discussion?