ignore (workaround for lack of itest runability) #11987
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ribasushi
force-pushed
the
master
branch
5 times, most recently
from
9fbbc41
to
795c1ce
Compare
* remove and replace some linters * remove some exclusions * make all exclusions more explicit matches
chore: remove unmaintained bootstrappers
This will re-execute tipsets to forcibly re-compute and store events when they're missing. This is effectively lazy backfilling of events. NOTE: This _won't_ backfill the index itself, it'll just give us the events. fixes filecoin-project#11335
…project#11747) The gas limit is proportional to the number of blocks. fixes filecoin-project#11721
Eth subscribe tipsets API should only return tipsets that have been executed. We do this by only returning the parent tipset of the latest tipset received by ETH Subscribe from it's TipSetFilter subscription. Closes filecoin-project#11807 Subsumes filecoin-project#11816
…ilecoin-project#11748) This can happen if, e.g., we run a splitstore garbage collection. fixes filecoin-project#11117
Upstream has merged our code to expose a `SignatureBytes` function, so we can now switch back to using it directly. This also brings bug fixes.
This was used to run an internal IPFS node...
1. Demote a noisy blocksync request error to info. All this warning means is that someone is requesting a tipset we don't have. 3. Fix a TODO and return a single CompactedMessages rather than 4 separate values.
Use native go implementation to sort and adjust imports. Compared to the previous bash version, this is at least 2X faster.
* add claim-extend cli * fix arg usage * add missing question * fix client addr, datacap prompt * replace waitGrp with errGrp * use promptUI * replace fmt.ErrorF with xerror * apply var name suggestion * GST rc3, update types * add itest * make gen
…ilecoin-project#11764) * fix Datacap TermMax * fix itest * add batching * make batch size variable * docsgen-cli * reduce batch size, fix batch dc * apply suggestion from review * put back TODO --------- Co-authored-by: LexLuthr <lexluthr@protocol.ai>
There are no perf gains to be had here anyway
By a wide margin the top issue with large blockstores is the size of their index. Transparently switch the badger lookup key to the leading 128 bits of the hash, regardless of type/codec. In the proccess forbid AllKeysChan and rewrite ForEachKey. Also unconditionally enable HashOnRead (only safe way to do partial keys) This is safe (and has been tested) to run on an existing lotus install, with splitstore enabled and compacting properly. When converting an existing setup simply set the envvar LOTUS_CHAIN_BADGERSTORE_QUERY_LEGACY_KEYS to true. When sufficient amount of FullGC cycles have passed and all the keys in your blockstore are "new" - the envvar is no longer needed. Reasoning: The filecoin blockstore in April 2024 sits at about 26 billion blocks. The current scheme of /blocks/base32-enc-of-entire-mh puts a key at 66 bytes, for a total raw index of about 1.6TiB. Reduction of every key to mere 16 bytes reduces this to just 390GiB, potentially even fitting in memory.
Prefix every bin-key value with a single uint64 varint containing:
(3bit store-type) // 0 for as-is, 1 for basic zstd compression, remainder reserved
+
(one reserved bit) << 3 // always 0 for now
+
(3bit compressability) << 4 // 0 if store-type is 0, otherwise `c := (origLen-compLen) * 8 / origLen`
// (using integer math, `origLen > compLen > 0` holds for any non-0 store-type)
+
(IPLD block data size) << 7 // 0 if store-type is 0
Include a rudimentary, dictionary-less zstd compressor as the first
non-verbatim storage type 1
|
OK, but just be aware that the cost of running these tests is coming off a certain budget; it's good to be supporting the ecosystem, but bills will add up if this is going to be a regular thing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE
DO NOT LOOK
KEEP WALKING
☨ this PR is never meant to leave
draftstatus: it is merely temporary CI for https://github.com/ribasushi/ltsh#readme, as runningitestslocally is currently quite challenging