Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Broken moving to S3 by TTL data parts that created in 24.1 to S3 after upgrade to 24.2 #63700

Open
k-morozov opened this issue May 13, 2024 · 4 comments
Open

Broken moving to S3 by TTL data parts that created in 24.1 to S3 after upgrade to 24.2 #63700

k-morozov opened this issue May 13, 2024 · 4 comments
Assignees
@alesapin
Labels
potential bug To be reviewed by developers and confirmed/rejected.

Comments

@k-morozov
Copy link
Contributor

k-morozov commented May 13, 2024 •
edited

Please make sure that the version you're using is still supported (you can find the list here).

You have to provide the following information whenever possible.

I use Clickhouse 24.1
I've created table and filled with data with TTL (move to S3).

Update Clickhouse to 24.2.
Old parts are not moved to S3 after the TTL expires. There are no problem with new parts.

Example:

DROP TABLE test_ttl_table;

CREATE TABLE test_ttl_table (
	generation UInt64, 
	date_key DateTime, 
	number UInt64, 
	text String, 
	expired DateTime DEFAULT now()
) 
ENGINE=MergeTree 
ORDER BY (generation, date_key) 
PARTITION BY toMonth(date_key) 
TTL expired + INTERVAL 10 MINUTES TO DISK 'object_storage';


INSERT INTO test_ttl_table (
	generation,
	date_key,
	number,
	text
)
	SELECT
		1,
		toDateTime('2000-01-01 00:00:00') + rand(number) % 365 * 86400,
		number,
		toString(number)

	FROM numbers(10000000);

# update cluster to 24.2

SELECT
    partition,
    name,
    uuid,
    disk_name,
    move_ttl_info.min,
    move_ttl_info.max,
    bytes_on_disk
FROM system.parts
WHERE table = 'test_ttl_table'
ORDER BY toInt8(partition) ASC

Query id: e4bab693-3865-44a3-a2ac-664c0d82103c

┌─partition─┬─name─────────┬─uuid─────────────────────────────────┬─disk_name─┬─move_ttl_info.min───────┬─move_ttl_info.max───────┬─bytes_on_disk─┐
│ 1         │ 1_74_74_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        928770 │
│ 1         │ 1_93_93_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        930934 │
│ 1         │ 1_97_97_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        929958 │
│ 1         │ 1_115_115_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        492933 │
│ 1         │ 1_10_69_1    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5759285 │
│ 2         │ 2_7_62_1     │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5394180 │
│ 2         │ 2_110_110_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        460464 │
│ 2         │ 2_105_105_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        876439 │
│ 2         │ 2_92_92_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        869973 │
│ 2         │ 2_82_82_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        870436 │
│ 3         │ 3_117_117_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        488874 │
│ 3         │ 3_101_101_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        931489 │
│ 3         │ 3_94_94_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        930016 │
│ 3         │ 3_81_81_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        929802 │
│ 3         │ 3_11_67_1    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5770230 │
│ 4         │ 4_6_64_1     │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5580195 │
│ 4         │ 4_111_111_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        475575 │
│ 4         │ 4_106_106_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        901198 │
│ 4         │ 4_91_91_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        900482 │
│ 4         │ 4_79_79_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        900967 │
│ 5         │ 5_116_116_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        490023 │
│ 5         │ 5_104_104_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        926871 │
│ 5         │ 5_87_87_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        928796 │
│ 5         │ 5_84_84_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        932116 │
│ 5         │ 5_8_72_1     │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5771681 │
│ 6         │ 6_107_107_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        900318 │
│ 6         │ 6_120_120_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        470141 │
│ 6         │ 6_89_89_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        900147 │
│ 6         │ 6_80_80_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        899309 │
│ 6         │ 6_9_70_1     │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5579481 │
│ 7         │ 7_12_63_1    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5752767 │
│ 7         │ 7_76_76_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        925182 │
│ 7         │ 7_96_96_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        930183 │
│ 7         │ 7_103_103_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        925495 │
│ 7         │ 7_109_109_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        487546 │
│ 8         │ 8_4_68_1     │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5766840 │
│ 8         │ 8_83_83_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        926365 │
│ 8         │ 8_90_90_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        925285 │
│ 8         │ 8_98_98_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        930100 │
│ 8         │ 8_112_112_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        495023 │
│ 9         │ 9_114_114_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        476192 │
│ 9         │ 9_2_66_1     │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5583211 │
│ 9         │ 9_77_77_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        898147 │
│ 9         │ 9_88_88_0    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        902041 │
│ 9         │ 9_102_102_0  │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        903643 │
│ 10        │ 10_1_65_1    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5764723 │
│ 10        │ 10_75_75_0   │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        933274 │
│ 10        │ 10_86_86_0   │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        927844 │
│ 10        │ 10_99_99_0   │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        928094 │
│ 10        │ 10_118_118_0 │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        487836 │
│ 11        │ 11_5_71_1    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5568108 │
│ 11        │ 11_73_73_0   │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        899391 │
│ 11        │ 11_95_95_0   │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        900161 │
│ 11        │ 11_108_108_0 │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        896461 │
│ 11        │ 11_113_113_0 │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        471726 │
│ 12        │ 12_3_61_1    │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │       5570900 │
│ 12        │ 12_119_119_0 │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        471695 │
│ 12        │ 12_100_100_0 │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        901110 │
│ 12        │ 12_85_85_0   │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        900259 │
│ 12        │ 12_78_78_0   │ 00000000-0000-0000-0000-000000000000 │ default   │ ['2024-05-09 15:08:29'] │ ['2024-05-09 15:08:29'] │        898663 │
└───────────┴──────────────┴──────────────────────────────────────┴───────────┴─────────────────────────┴─────────────────────────┴───────────────┘

60 rows in set. Elapsed: 0.005 sec. 

rc1a-g123qt0h5lfckic3.mdb.yandexcloud.net :) select now()

SELECT now()

Query id: 6208cfb0-6062-4761-95d8-fce51e2e80d7

┌───────────────now()─┐
│ 2024-05-09 15:44:57 │
└─────────────────────┘

1 row in set. Elapsed: 0.002 sec.

Expected behavior

Expected that parts will move to S3.
@k-morozov k-morozov added the potential bug To be reviewed by developers and confirmed/rejected. label May 13, 2024
@den-crane
Copy link
Contributor

┌───────────────now()─┐
│ 2024-05-09 15:44:57 │
└─────────────────────┘
2024-05-09 15:08:29

It proofs nothing.
TTL works eventually.
Why those parts are not moved in 24.1 ? Did you stop Clickhouse immediately after insertion?

@k-morozov
Copy link
Contributor Author

k-morozov commented May 13, 2024 •
edited

Why those parts are not moved in 24.1 ? Did you stop Clickhouse immediately after insertion?

After inserting the data I immediately updated to 24.2. After update TTL for parts that created in 24.1 doesn't work.

@den-crane
Copy link
Contributor

Why those parts are not moved in 24.1 ? Did you stop Clickhouse immediately after insertion?

After inserting the data I immediately updated to 24.2. After update TTL for parts that created in 24.1 doesn't work.

OK, what if you just restart 24.1 ? I guess it will be the same. And it's expected behaviour, and I can explain it.

@k-morozov
Copy link
Contributor Author

Why those parts are not moved in 24.1 ? Did you stop Clickhouse immediately after insertion?

After inserting the data I immediately updated to 24.2. After update TTL for parts that created in 24.1 doesn't work.

OK, what if you just restart 24.1 ? I guess it will be the same. And it's expected behaviour, and I can explain it.

Ive done as you say: inserted data in CH 24.1 and immediately restart. TTL finished correctly. This behavior is diffrent if instead of restarting I do an update to 24.2.

@alesapin alesapin self-assigned this May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alesapin alesapin

Labels
potential bug To be reviewed by developers and confirmed/rejected.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@alesapin @den-crane @k-morozov