Superfluous query cancelled exception

[cangyin]

There are superfluous similar exception messages saying "Query was cancelled or a client has unexpectedly dropped the connection".
These messages are only printed for secondary query (is_initial_query = 0) of distributed query.

2024.05.10 15:36:33.902900 [ 9401 ] {228489da-10f1-4ce3-b610-a82178a555b5} <Information> TCPHandler: Query was cancelled.
2024.05.10 15:36:33.932125 [ 9401 ] {228489da-10f1-4ce3-b610-a82178a555b5} <Information> executeQuery: Code: 394. DB::Exception: Query was cancelled or a client has unexpectedly dropped the connection. (QUERY_WAS_CANCELLED) (version 23.2.4.7) (from 10.244.166.28:36286) (in query: SELECT `recordTime`, `col1`, `col2`, `col3`, `...` FROM `business`.`network_security_log_local` WHERE foo = bar ORDER BY `recordTimestamp` DESC LIMIT 1, 100)

No stacktraces in log. I think it's because the initial server closed connection on enough data for LIMIT 100 and sent cancel packet just at the time when secondary server finished execution and found that query was canceled, here:
https://github.com/ClickHouse/ClickHouse/blob/v23.2.4.12-stable/src/Server/TCPHandler.cpp#L422

[cangyin]

This causes the metric FailedQuery to surge.

[Algunenano]

It might be possible to improve this, but it's expected in order to not waste resources.

[filimonov]

We also observed the same problem recently. After the upgrade of one of the clusters from 22.3 to 23.8 the number of QUERY_WAS_CANCELLED for non-initial queries jumped high (the same query always).

The real reason was not the LIMIT, but the INNER JOIN where the right side returned the empty set. (after that the left subquery was cancelled, which is good, but it was not obvious - required deeper look to debug / understand).

Initiator-side message was also not transparent enough

StorageDistributed (...): (10.166.2.140:9000) Cancelling query because enough data has been read