Forward was compeleted successfully, but the HTLC is still pending

[SunnySarahNode]

I'm seeing a very weird situation that I haven't encountered before, and which seems like it could lead to loss of funds.
I hope I just misunderstand something and it's not a bug :)

Me: LND 0.17.3, bitcoind 25 (full indexed blockchain), Ubuntu (amd64), clearnet only
My peer: unknown

I have the following channel with the HTLC in it ( lncli listchannels ) :

{
            "active": false,
            "remote_pubkey": "0383a52db8e4063714c767e90628a5caee4d278c308cfe0fae82e21aebc313ccbe",
            "channel_point": "db8486422565a803e8dc82b4424e3ecadb4aa5cc942284079ede4a91a453fe98:1",
            "chan_id": "877767620541415425",
            "capacity": "4000000",
            "local_balance": "684572",
            "remote_balance": "532248",
            "commit_fee": "3574",
            "commit_weight": "1288",
            "fee_per_kw": "2757",
            "unsettled_balance": "2778946",
			...
            "pending_htlcs": [
                {
                    "incoming": true,
                    "amount": "2778946",
                    "hash_lock": "4ac9b86355ffed94847b01017d1f5976658883c75bb35a0a72fbb22cdee7ebdb",
                    "expiration_height": 832737,
                    "htlc_index": "551934",
                    "forwarding_channel": "887966690141143041",
                    "forwarding_htlc_index": "10105"
                }
            ],
            "csv_delay": 480,
            "chan_status_flags": "ChanStatusDefault",
            "local_chan_reserve_sat": "40000",
            "remote_chan_reserve_sat": "40000",
            "static_remote_key": false,
            "commitment_type": "ANCHORS",
            "local_constraints": {
                "csv_delay": 480,
                "chan_reserve_sat": "40000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "3960000000",
                "min_htlc_msat": "1",
                "max_accepted_htlcs": 483
            },
            "remote_constraints": {
                "csv_delay": 480,
                "chan_reserve_sat": "40000",
                "dust_limit_sat": "354",
                "max_pending_amt_msat": "3960000000",
                "min_htlc_msat": "100000",
                "max_accepted_htlcs": 483
            },
            "peer_alias": "blitzbold",
        }

This forward has the following route :

blitzbold --> (me) --> LOOP

This forward was successfully completed ( lncli fwdinghistory ) :

        {
            "timestamp": "1709147676",
            "chan_id_in": "877767620541415425",
            "chan_id_out": "887966690141143041",
            "amt_in": "2778946",
            "amt_out": "2773546",
            "fee": "5400",
            "fee_msat": "5400094",
            "amt_in_msat": "2778946094",
            "amt_out_msat": "2773546000",
            "timestamp_ns": "1709147676006760972",
            "peer_alias_in": "blitzbold",
            "peer_alias_out": "LOOP"
        }

...but as you can see, the HTLC in the channel 877767620541415425 is still "pending".

My peer is unable to receive connections correctly, so this channel is "disabled" :

2024-02-28 23:33:38.435 [ERR] RPCS: [/lnrpc.Lightning/DisconnectPeer]: unable to disconnect peer: peer 0383a52db8e4063714c767e90628a5caee4d278c308cfe0fae82e21aebc313ccbe is not connected
2024-02-28 23:33:53.877 [INF] SRVR: Established connection to: 0383a52db8e4063714c767e90628a5caee4d278c308cfe0fae82e21aebc313ccbe@46.101.122.181:49621
2024-02-28 23:33:53.877 [INF] SRVR: Finalizing connection to 0383a52db8e4063714c767e90628a5caee4d278c308cfe0fae82e21aebc313ccbe@46.101.122.181:49621, inbound=false
2024-02-28 23:33:53.967 [INF] PEER: Peer(0383a52db8e4063714c767e90628a5caee4d278c308cfe0fae82e21aebc313ccbe): disconnecting 0383a52db8e4063714c767e90628a5caee4d278c308cfe0fae82e21aebc313ccbe@46.101.122.181:49621, reason: unable to start peer: unable to read init msg: read next header: read tcp [ip]:34816->46.101.122.181:49621: read: connection reset by peer

So, for now :

1. My local liquidity in the channel with LOOP was decreased.
2. My local liquidity in the channel with blitzbold was NOT increased.
3. HTLC in the channel with blitzbold is "pending", channel is "disabled".

The main question : what exactly will happen at block 832737, when this channel is auto-FCed due to expiration of this HTLC? Will my peer receive funds (this is a large amount!) that should belong to me, because this is an outgoing HTLC for him?

[bitromortac]

This seems to be a normal case of an HTLC that can't be settled because the peer isn't reachable. That's covered in this test:

lnd/itest/lnd_multi-hop_test.go

Lines 338 to 547 in 72764b1

	// testMultiHopReceiverChainClaim tests that in the multi-hop setting, if the
	// receiver of an HTLC knows the preimage, but wasn't able to settle the HTLC
	// off-chain, then it goes on chain to claim the HTLC uing the HTLC success
	// transaction. In this scenario, the node that sent the outgoing HTLC should
	// extract the preimage from the sweep transaction, and finish settling the
	// HTLC backwards into the route.
	func testMultiHopReceiverChainClaim(ht *lntest.HarnessTest) {
	runMultiHopHtlcClaimTest(ht, runMultiHopReceiverChainClaim)
	}
	func runMultiHopReceiverChainClaim(ht *lntest.HarnessTest,
	alice, bob *node.HarnessNode, c lnrpc.CommitmentType, zeroConf bool) {
	// First, we'll create a three hop network: Alice -> Bob -> Carol, with
	// Carol refusing to actually settle or directly cancel any HTLC's
	// self.
	aliceChanPoint, bobChanPoint, carol := createThreeHopNetwork(
	ht, alice, bob, false, c, zeroConf,
	)
	// If this is a taproot channel, then we'll need to make some manual
	// route hints so Alice can actually find a route.
	var routeHints []*lnrpc.RouteHint
	if c == lnrpc.CommitmentType_SIMPLE_TAPROOT {
	routeHints = makeRouteHints(bob, carol, zeroConf)
	}
	// With the network active, we'll now add a new hodl invoice at Carol's
	// end. Make sure the cltv expiry delta is large enough, otherwise Bob
	// won't send out the outgoing htlc.
	const invoiceAmt = 100000
	var preimage lntypes.Preimage
	copy(preimage[:], ht.Random32Bytes())
	payHash := preimage.Hash()
	invoiceReq := &invoicesrpc.AddHoldInvoiceRequest{
	Value: invoiceAmt,
	CltvExpiry: finalCltvDelta,
	Hash: payHash[:],
	RouteHints: routeHints,
	}
	carolInvoice := carol.RPC.AddHoldInvoice(invoiceReq)
	// Subscribe the invoice.
	stream := carol.RPC.SubscribeSingleInvoice(payHash[:])
	// Now that we've created the invoice, we'll send a single payment from
	// Alice to Carol. We won't wait for the response however, as Carol
	// will not immediately settle the payment.
	req := &routerrpc.SendPaymentRequest{
	PaymentRequest: carolInvoice.PaymentRequest,
	TimeoutSeconds: 60,
	FeeLimitMsat: noFeeLimitMsat,
	}
	alice.RPC.SendPayment(req)
	// At this point, all 3 nodes should now have an active channel with
	// the created HTLC pending on all of them.
	ht.AssertActiveHtlcs(alice, payHash[:])
	ht.AssertActiveHtlcs(bob, payHash[:])
	ht.AssertActiveHtlcs(carol, payHash[:])
	// Wait for carol to mark invoice as accepted. There is a small gap to
	// bridge between adding the htlc to the channel and executing the exit
	// hop logic.
	ht.AssertInvoiceState(stream, lnrpc.Invoice_ACCEPTED)
	restartBob := ht.SuspendNode(bob)
	// Settle invoice. This will just mark the invoice as settled, as there
	// is no link anymore to remove the htlc from the commitment tx. For
	// this test, it is important to actually settle and not leave the
	// invoice in the accepted state, because without a known preimage, the
	// channel arbitrator won't go to chain.
	carol.RPC.SettleInvoice(preimage[:])
	// Increase the fee estimate so that the following force close tx will
	// be cpfp'ed.
	ht.SetFeeEstimate(30000)
	// We now advance the block height to the point where Carol will force
	// close her channel with Bob, broadcast the closing tx but keep it
	// unconfirmed.
	numBlocks := padCLTV(uint32(
	invoiceReq.CltvExpiry - lncfg.DefaultIncomingBroadcastDelta,
	))
	// Now we'll mine enough blocks to prompt carol to actually go to the
	// chain in order to sweep her HTLC since the value is high enough.
	ht.MineBlocks(numBlocks)
	// At this point, Carol should broadcast her active commitment
	// transaction in order to go to the chain and sweep her HTLC. If there
	// are anchors, Carol also sweeps hers.
	expectedTxes := 1
	hasAnchors := lntest.CommitTypeHasAnchors(c)
	if hasAnchors {
	expectedTxes = 2
	}
	ht.Miner.AssertNumTxsInMempool(expectedTxes)
	closingTx := ht.Miner.AssertOutpointInMempool(
	ht.OutPointFromChannelPoint(bobChanPoint),
	)
	closingTxid := closingTx.TxHash()
	// Confirm the commitment.
	ht.MineBlocksAndAssertNumTxes(1, expectedTxes)
	// Restart bob again.
	require.NoError(ht, restartBob())
	// After the force close transaction is mined, a series of transactions
	// should be broadcast by Bob and Carol. When Bob notices Carol's second
	// level transaction in the mempool, he will extract the preimage and
	// settle the HTLC back off-chain.
	switch c {
	// Carol should broadcast her second level HTLC transaction and Bob
	// should broadcast a sweep tx to sweep his output in the channel with
	// Carol.
	case lnrpc.CommitmentType_LEGACY:
	expectedTxes = 2
	// Carol should broadcast her second level HTLC transaction and Bob
	// should broadcast a sweep tx to sweep his output in the channel with
	// Carol, and another sweep tx to sweep his anchor output.
	case lnrpc.CommitmentType_ANCHORS, lnrpc.CommitmentType_SIMPLE_TAPROOT:
	expectedTxes = 3
	// Carol should broadcast her second level HTLC transaction and Bob
	// should broadcast a sweep tx to sweep his anchor output. Bob's commit
	// output can't be swept yet as he's incurring an additional CLTV from
	// being the channel initiator of a script-enforced leased channel.
	case lnrpc.CommitmentType_SCRIPT_ENFORCED_LEASE:
	expectedTxes = 2
	default:
	ht.Fatalf("unhandled commitment type %v", c)
	}
	// All transactions should be spending from the commitment transaction.
	txes := ht.Miner.GetNumTxsFromMempool(expectedTxes)
	ht.AssertAllTxesSpendFrom(txes, closingTxid)
	// We'll now mine an additional block which should confirm both the
	// second layer transactions.
	ht.MineBlocksAndAssertNumTxes(1, expectedTxes)
	// Carol's pending channel report should now show two outputs under
	// limbo: her commitment output, as well as the second-layer claim
	// output, and the pending HTLC should also now be in stage 2.
	ht.AssertNumHTLCsAndStage(carol, bobChanPoint, 1, 2)
	// Once the second-level transaction confirmed, Bob should have
	// extracted the preimage from the chain, and sent it back to Alice,
	// clearing the HTLC off-chain.
	ht.AssertNumActiveHtlcs(alice, 0)
	// If we mine 4 additional blocks, then Carol can sweep the second
	// level HTLC output once the CSV expires.
	ht.MineEmptyBlocks(defaultCSV)
	// We should have a new transaction in the mempool.
	ht.Miner.AssertNumTxsInMempool(1)
	// Finally, if we mine an additional block to confirm Carol's second
	// level success transaction. Carol should not show a pending channel
	// in her report afterwards.
	ht.MineBlocks(1)
	ht.AssertNumPendingForceClose(carol, 0)
	// The invoice should show as settled for Carol, indicating that it was
	// swept on-chain.
	ht.AssertInvoiceSettled(carol, carolInvoice.PaymentAddr)
	// Finally, check that the Alice's payment is correctly marked
	// succeeded.
	ht.AssertPaymentStatus(alice, preimage, lnrpc.Payment_SUCCEEDED)
	if c == lnrpc.CommitmentType_SCRIPT_ENFORCED_LEASE {
	// Bob still has his commit output to sweep to since he incurred
	// an additional CLTV from being the channel initiator of a
	// script-enforced leased channel, regardless of whether he
	// forced closed the channel or not.
	pendingChanResp := bob.RPC.PendingChannels()
	require.Len(ht, pendingChanResp.PendingForceClosingChannels, 1)
	forceCloseChan := pendingChanResp.PendingForceClosingChannels[0]
	require.Positive(ht, forceCloseChan.LimboBalance)
	require.Positive(ht, forceCloseChan.BlocksTilMaturity)
	// TODO: Bob still shows a pending HTLC at this point when he
	// shouldn't, as he already extracted the preimage from Carol's
	// claim.
	// require.Len(t.t, forceCloseChan.PendingHtlcs, 0)
	// Mine enough blocks for Bob's commit output's CLTV to expire
	// and sweep it.
	numBlocks := uint32(forceCloseChan.BlocksTilMaturity)
	ht.MineBlocks(numBlocks)
	commitOutpoint := wire.OutPoint{Hash: closingTxid, Index: 3}
	ht.Miner.AssertOutpointInMempool(commitOutpoint)
	ht.MineBlocks(1)
	}
	ht.AssertNumPendingForceClose(bob, 0)
	// We'll close out the channel between Alice and Bob, then shutdown
	// carol to conclude the test.
	ht.CloseChannel(alice, aliceChanPoint)
	}

so I think your node should go to chain to claim the HTLC 10 blocks before timeout, see

lnd/lncfg/config.go

Lines 20 to 27 in 72764b1

	// DefaultIncomingBroadcastDelta defines the number of blocks before the
	// expiry of an incoming htlc at which we force close the channel. We
	// only go to chain if we also have the preimage to actually pull in the
	// htlc. BOLT #2 suggests 7 blocks. We use a few more for extra safety.
	// Within this window we need to get our sweep or 2nd level success tx
	// confirmed, because after that the remote party is also able to claim
	// the htlc using the timeout path.
	DefaultIncomingBroadcastDelta = 10

[SunnySarahNode]

bitromortac Thank you for your answer, it's really helpful!

One more short question: can I FC this channel manually earlier with the same result?

[bitromortac]

I think that lnd should handle it, but there's a chance that the peer comes back online? It may also be that you provoke an edge case, it's maybe safer to let lnd go through standard procedures, but not sure.