Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add defragment support for HFE #13229

Merged
merged 87 commits into from
May 14, 2024
Merged
Show file tree
Hide file tree
Changes from 40 commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
5ee153e
Hash Field Expiration Defragmentation
sundb Apr 19, 2024
53dbff1
Optimize ebScanDefrag
sundb Apr 20, 2024
3162c25
Spell
sundb Apr 20, 2024
9117b42
Add notification tests for HFE
sundb Apr 22, 2024
4986870
New way to skip defraging the fields with TTL
sundb Apr 23, 2024
f7bf686
Remove activeDefragHfieldDict
sundb Apr 23, 2024
99b3bad
Use dictUseStoredKeyApi to get hash from hfield
sundb Apr 23, 2024
09562c4
Revert the code for defraging hash obj with TTL
sundb Apr 23, 2024
761b3a4
Add commands.def
sundb Apr 23, 2024
5a86b46
Add activeDefragHfieldSkipTTL
sundb Apr 23, 2024
6942c4a
add new ebDefragItem
sundb Apr 25, 2024
fdbc69e
Revert some code
sundb Apr 25, 2024
af9eadc
Simplify code
sundb Apr 25, 2024
871ec6f
Fix compile warning
sundb Apr 25, 2024
b19f6f5
Fix crash due to forget skip defraging hfield with TTL in scanLaterHash
sundb Apr 25, 2024
5e8dfd1
Unify the defragment of hfield in the callback of dictScanDefrag
sundb Apr 25, 2024
2f988f0
Add comment
sundb Apr 25, 2024
f8bbf7f
Cleanup
sundb Apr 25, 2024
acfc293
Use zmalloc_usable_size instead of zmalloc_size
sundb Apr 25, 2024
4a9f8f6
Remove unused code
sundb Apr 25, 2024
d701bf8
Remove unused code
sundb Apr 25, 2024
dcc5da0
Add missing raxStop
sundb Apr 26, 2024
35ec1a3
Merge branch 'hfe' into hfe-defrag
sundb Apr 26, 2024
1f62c55
Add individual test for HFE defragment
sundb Apr 26, 2024
c0ec94c
For CI test
sundb Apr 26, 2024
c270ca4
Fix defrag HFE test failed
sundb Apr 26, 2024
5fe4b02
add listpack support (todo: fix rebase issues)
tezc Apr 26, 2024
1318272
test fix
tezc Apr 26, 2024
6177070
Make HFE defrag test more stable
sundb Apr 26, 2024
06d13a3
Remove test code
sundb Apr 26, 2024
cd7396c
add hgetf
tezc Apr 26, 2024
3b5b33b
silence compiler warning
tezc Apr 26, 2024
eada402
fix
tezc Apr 26, 2024
c952c1e
Reflect ebDefragItem() about rax and other CRs
sundb Apr 27, 2024
08c4276
Merge the tests about item defragmentation
sundb Apr 27, 2024
b8f81a0
add hashTypeIsDictWithMetaHFE to check if hash obj is dict with HFE
sundb Apr 28, 2024
78cebfc
cleanup
sundb Apr 28, 2024
ea61eb4
Try to fix the failure of expiration test
sundb Apr 28, 2024
2d40d61
hsetf
tezc Apr 28, 2024
b90ab61
Make dictExpireMetadata opaque
sundb Apr 29, 2024
66c8785
Fix the reply schema for HFE
sundb Apr 29, 2024
f2124a5
Add .def
sundb Apr 29, 2024
4f64806
Revert the reply schema for hexpire
sundb Apr 29, 2024
2ea37aa
Fix reply schema for hpersist
sundb Apr 29, 2024
f9c0106
Remvoe reply schema def
sundb Apr 30, 2024
293b127
Add test for hpexpireat for schema validate CI
sundb Apr 30, 2024
765ebc4
Defrag listpack for HFE
sundb Apr 30, 2024
b59dbb7
Fix missing update ob->ptr after defraging listpack
sundb Apr 30, 2024
7e8cd04
simply hsetf
tezc May 1, 2024
aa096d1
comment
tezc May 1, 2024
2296d05
expose struct listpackTTL
tezc May 1, 2024
920d784
minor
tezc May 1, 2024
d0b523c
minor
tezc May 1, 2024
6c6c6a4
rename listpackttl to listpackex
tezc May 2, 2024
7712b00
Merge branch 'hfe-listpack-v1' into hfe-listpack-defrag
sundb May 6, 2024
d20743e
Fix complaint
sundb May 6, 2024
ac1130f
Merge branch 'hash-field-expiry-integ' into hfe-listpack-defrag
sundb May 6, 2024
f0c74fa
comments
tezc May 6, 2024
07dad91
merge
tezc May 6, 2024
36bd618
schema fix
tezc May 6, 2024
2c0b560
comment
tezc May 6, 2024
e8ea643
comment
tezc May 6, 2024
d1ce106
minor
tezc May 6, 2024
5775ba0
minor
tezc May 6, 2024
f0ec7e3
minor
tezc May 6, 2024
9e323bb
Merge branch 'hfe-listpack-v1' into hfe-listpack-defrag
sundb May 6, 2024
01563e6
Revert code
sundb May 7, 2024
9a2fb77
test fix
tezc May 7, 2024
af548c8
Merge branch 'hfe-listpack-v1' into hfe-listpack-defrag
sundb May 7, 2024
661fc72
Merge branch 'hash-field-expiry-integ' into hfe-defrag
sundb May 8, 2024
b187702
Missing .def
sundb May 8, 2024
e6b8b6b
Add ebValidate() to check if the data is corrupted in the unittest
sundb May 8, 2024
1662f40
Merge branch 'hash-field-expiry-integ' into hfe-defrag
sundb May 9, 2024
1452444
Merge branch 'hfe-listpack-defrag' into hfe-defrag
sundb May 9, 2024
496e242
Revert some code
sundb May 9, 2024
b18cb26
Add comments
sundb May 9, 2024
2d86fc8
Revert some code
sundb May 9, 2024
6fdbfb3
Fix comment
sundb May 9, 2024
5bf9ddb
Fix CR
sundb May 9, 2024
0936517
Format
sundb May 9, 2024
1bea6c9
Update src/defrag.c
sundb May 10, 2024
b619ea5
Add a assertion for ExpireMeta's trash in ebDefragItem()
sundb May 10, 2024
5023c03
Fix a `else if` mistake
sundb May 11, 2024
ebbb291
Merge branch 'hash-field-expiry-integ' into hfe-defrag
sundb May 13, 2024
8db938b
Improve the HFE deframent test to cover listpackex and eblist
sundb May 14, 2024
8363b2c
Increase the threshold to make test more stable
sundb May 14, 2024
79cdbb0
Improve test
sundb May 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/commands.def
Original file line number Diff line number Diff line change
Expand Up @@ -3625,7 +3625,7 @@ struct COMMAND_ARG HMSET_Args[] = {
#ifndef SKIP_CMD_KEY_SPECS_TABLE
/* HPERSIST key specs */
keySpec HPERSIST_Keyspecs[1] = {
{NULL,CMD_KEY_RO|CMD_KEY_ACCESS,KSPEC_BS_INDEX,.bs.index={1},KSPEC_FK_RANGE,.fk.range={0,1,0}}
{NULL,CMD_KEY_RW|CMD_KEY_UPDATE,KSPEC_BS_INDEX,.bs.index={1},KSPEC_FK_RANGE,.fk.range={0,1,0}}
};
#endif

Expand Down Expand Up @@ -10995,7 +10995,7 @@ struct COMMAND_STRUCT redisCommandTable[] = {
{MAKE_CMD("hlen","Returns the number of fields in a hash.","O(1)","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HLEN_History,0,HLEN_Tips,0,hlenCommand,2,CMD_READONLY|CMD_FAST,ACL_CATEGORY_HASH,HLEN_Keyspecs,1,NULL,1),.args=HLEN_Args},
{MAKE_CMD("hmget","Returns the values of all fields in a hash.","O(N) where N is the number of fields being requested.","2.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HMGET_History,0,HMGET_Tips,0,hmgetCommand,-3,CMD_READONLY|CMD_FAST,ACL_CATEGORY_HASH,HMGET_Keyspecs,1,NULL,2),.args=HMGET_Args},
{MAKE_CMD("hmset","Sets the values of multiple fields.","O(N) where N is the number of fields being set.","2.0.0",CMD_DOC_DEPRECATED,"`HSET` with multiple field-value pairs","4.0.0","hash",COMMAND_GROUP_HASH,HMSET_History,0,HMSET_Tips,0,hsetCommand,-4,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_HASH,HMSET_Keyspecs,1,NULL,2),.args=HMSET_Args},
{MAKE_CMD("hpersist","Removes the expiration time for each specified field","O(N) where N is the number of arguments to the command","8.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPERSIST_History,0,HPERSIST_Tips,0,hpersistCommand,-4,CMD_READONLY|CMD_FAST,ACL_CATEGORY_HASH,HPERSIST_Keyspecs,1,NULL,3),.args=HPERSIST_Args},
{MAKE_CMD("hpersist","Removes the expiration time for each specified field","O(N) where N is the number of arguments to the command","8.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPERSIST_History,0,HPERSIST_Tips,0,hpersistCommand,-4,CMD_WRITE|CMD_FAST,ACL_CATEGORY_HASH,HPERSIST_Keyspecs,1,NULL,3),.args=HPERSIST_Args},
{MAKE_CMD("hpexpire","Set expiry for hash field using relative time to expire (milliseconds)","O(N) where N is the number of arguments to the command","8.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIRE_History,0,HPEXPIRE_Tips,0,hpexpireCommand,-5,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_HASH,HPEXPIRE_Keyspecs,1,NULL,5),.args=HPEXPIRE_Args},
{MAKE_CMD("hpexpireat","Set expiry for hash field using an absolute Unix timestamp (milliseconds)","O(N) where N is the number of arguments to the command","8.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIREAT_History,0,HPEXPIREAT_Tips,0,hpexpireatCommand,-5,CMD_WRITE|CMD_DENYOOM|CMD_FAST,ACL_CATEGORY_HASH,HPEXPIREAT_Keyspecs,1,NULL,5),.args=HPEXPIREAT_Args},
{MAKE_CMD("hpexpiretime","Returns the expiration time of a hash field as a Unix timestamp, in msec.","O(N) where N is the number of arguments to the command","8.0.0",CMD_DOC_NONE,NULL,NULL,"hash",COMMAND_GROUP_HASH,HPEXPIRETIME_History,0,HPEXPIRETIME_Tips,0,hpexpiretimeCommand,-4,CMD_READONLY|CMD_FAST,ACL_CATEGORY_HASH,HPEXPIRETIME_Keyspecs,1,NULL,3),.args=HPEXPIRETIME_Args},
Expand Down
38 changes: 20 additions & 18 deletions src/commands/hexpire.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,24 +46,26 @@
"type": "array",
"minItems": 1,
"maxItems": 4294967295,
"items": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "Specified NX | XX | GT | LT condition not met",
"const": 0
},
{
"description": "Expiration time was set or updated.",
"const": 1
},
{
"description": "Field deleted because the specified expiration time is in the past.",
"const": 2
}
]
"items": {
"oneOf": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "Specified NX | XX | GT | LT condition not met",
"const": 0
},
{
"description": "Expiration time was set or updated.",
"const": 1
},
{
"description": "Field deleted because the specified expiration time is in the past.",
"const": 2
}
]
}
}
]
},
Expand Down
38 changes: 20 additions & 18 deletions src/commands/hexpireat.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,24 +46,26 @@
"type": "array",
"minItems": 1,
"maxItems": 4294967295,
"items": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "Specified NX | XX | GT | LT condition not met",
"const": 0
},
{
"description": "Expiration time was set or updated.",
"const": 1
},
{
"description": "Field deleted because the specified expiration time is in the past.",
"const": 2
}
]
"items": {
"oneOf": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "Specified NX | XX | GT | LT condition not met",
"const": 0
},
{
"description": "Expiration time was set or updated.",
"const": 1
},
{
"description": "Field deleted because the specified expiration time is in the past.",
"const": 2
}
]
}
}
]
},
Expand Down
154 changes: 78 additions & 76 deletions src/commands/hexpiretime.json
Original file line number Diff line number Diff line change
@@ -1,83 +1,85 @@
{
"HEXPIRETIME": {
"summary": "Returns the expiration time of a hash field as a Unix timestamp, in seconds.",
"complexity": "O(N) where N is the number of arguments to the command",
"group": "hash",
"since": "8.0.0",
"arity": -4,
"function": "hexpiretimeCommand",
"history": [],
"command_flags": [
"READONLY",
"FAST"
],
"acl_categories": [
"HASH"
],
"key_specs": [
{
"flags": [
"RO",
"ACCESS"
],
"begin_search": {
"index": {
"pos": 1
"summary": "Returns the expiration time of a hash field as a Unix timestamp, in seconds.",
"complexity": "O(N) where N is the number of arguments to the command",
"group": "hash",
"since": "8.0.0",
"arity": -4,
"function": "hexpiretimeCommand",
"history": [],
"command_flags": [
"READONLY",
"FAST"
],
"acl_categories": [
"HASH"
],
"key_specs": [
{
"flags": [
"RO",
"ACCESS"
],
"begin_search": {
"index": {
"pos": 1
}
},
"find_keys": {
"range": {
"lastkey": 0,
"step": 1,
"limit": 0
}
}
}
},
"find_keys": {
"range": {
"lastkey": 0,
"step": 1,
"limit": 0
}
}
}
],
"reply_schema": {
"oneOf": [
{
"description": "Key does not exist.",
"type": "null"
},
{
"description": "Array of results",
"type": "array",
"minItems": 1,
"maxItems": 4294967295,
"items": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "The field exists but has no associated expire.",
"const": -1
},
{
"description": "Expiration Unix timestamp in seconds.",
"type": "integer",
"minimum": 1
}
],
"reply_schema": {
"oneOf": [
{
"description": "Key does not exist.",
"type": "null"
},
{
"description": "Array of results",
"type": "array",
"minItems": 1,
"maxItems": 4294967295,
"items": {
"oneOf": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "The field exists but has no associated expire.",
"const": -1
},
{
"description": "Expiration Unix timestamp in seconds.",
"type": "integer",
"minimum": 1
}
]
}
}
]
}
},
"arguments": [
{
"name": "key",
"type": "key",
"key_spec_index": 0
},
{
"name": "numfields",
"type": "integer"
},
{
"name": "field",
"type": "string",
"multiple": true
}
]
},
"arguments": [
{
"name": "key",
"type": "key",
"key_spec_index": 0
},
{
"name": "numfields",
"type": "integer"
},
{
"name": "field",
"type": "string",
"multiple": true
}
]
}
}
36 changes: 19 additions & 17 deletions src/commands/hpersist.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"function": "hpersistCommand",
"history": [],
"command_flags": [
"READONLY",
"WRITE",
"FAST"
],
"acl_categories": [
Expand All @@ -17,8 +17,8 @@
"key_specs": [
{
"flags": [
"RO",
"ACCESS"
"RW",
"UPDATE"
],
"begin_search": {
"index": {
Expand All @@ -45,20 +45,22 @@
"type": "array",
"minItems": 1,
"maxItems": 4294967295,
"items": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "The field exists but has no associated expire.",
"const": -1
},
{
"description": "Expiration time was removed",
"const": 1
}
]
"items": {
"oneOf": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "The field exists but has no associated expire.",
"const": -1
},
{
"description": "Expiration time was removed",
"const": 1
}
]
}
}
]
},
Expand Down
38 changes: 20 additions & 18 deletions src/commands/hpexpire.json
Original file line number Diff line number Diff line change
Expand Up @@ -46,24 +46,26 @@
"type": "array",
"minItems": 1,
"maxItems": 4294967295,
"items": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "Specified NX | XX | GT | LT condition not met",
"const": 0
},
{
"description": "Expiration time was set or updated.",
"const": 1
},
{
"description": "Field deleted because the specified expiration time is in the past.",
"const": 2
}
]
"items": {
"oneOf": [
{
"description": "The field does not exist.",
"const": -2
},
{
"description": "Specified NX | XX | GT | LT condition not met",
"const": 0
},
{
"description": "Expiration time was set or updated.",
"const": 1
},
{
"description": "Field deleted because the specified expiration time is in the past.",
"const": 2
}
]
}
}
]
},
Expand Down