{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":429907564,"defaultBranch":"main","name":"mcap","ownerLogin":"foxglove","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2021-11-19T19:09:38.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/78454435?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1717094329.0","currentOid":""},"activityList":{"items":[{"before":null,"after":"cf6ff39de5918d6bd1f7afe6e9c25437a79d9943","ref":"refs/heads/jtbandes-patch-1","pushedAt":"2024-05-30T18:38:49.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"jtbandes","name":"Jacob Bandes-Storch","path":"/jtbandes","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14237?s=80&v=4"},"commit":{"message":"Remove close() call from protobuf writer example\n\nCalling close() after terminate() doesn't make sense for a couple reasons. First, close() itself calls terminate(). Second, close() attempts to write additional data to the file, and we are doing it here after a write already failed, so it might not be expected to succeed.\r\n\r\nOn the other hand, it looks like the possible error codes from write() are NotOpen, InvalidChannelId, or InvalidSchemaId. In all of these cases it might actually be fine to call close(), as long as we didn't already call terminate()?\r\n\r\nOn the third hand, ~McapWriter also calls close() so maybe we completely remove this? 🤷","shortMessageHtmlLink":"Remove close() call from protobuf writer example"}},{"before":"5c1f9149dae8396635a236d12bd4e414c9f7200e","after":"5b805219a9c8888d621ddf9bef2ffdafb4c5e1f3","ref":"refs/heads/main","pushedAt":"2024-05-30T17:51:42.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jtbandes","name":"Jacob Bandes-Storch","path":"/jtbandes","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14237?s=80&v=4"},"commit":{"message":"Stop example protobuf code from leaking (#1175)\n\n### Changelog\r\nprotobufs allocated with `New()` must be `delete`d later.\r\n\r\n### Docs\r\nNone\r\n\r\n### Description\r\n\r\n
BeforeAfter
\r\n\r\nLeaks\r\n\r\n\r\n\r\nDoesn't leak (leaks less?)\r\n\r\n
\r\n\r\n\r\n---------\r\n\r\nCo-authored-by: Jacob Bandes-Storch \r\nCo-authored-by: Hans-Joachim Krauch ","shortMessageHtmlLink":"Stop example protobuf code from leaking (#1175)"}},{"before":"ef1daa3685b63a0ec39e4af698111094af068a6f","after":"5c1f9149dae8396635a236d12bd4e414c9f7200e","ref":"refs/heads/main","pushedAt":"2024-05-30T01:39:55.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"jtbandes","name":"Jacob Bandes-Storch","path":"/jtbandes","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14237?s=80&v=4"},"commit":{"message":"Fix capitalization error on protobuf wiki page (#1174)\n\n### Changelog\r\n`Protobuf`->`protobuf`\r\n\r\n### Docs\r\nNone\r\n\r\n### Description\r\nOne of these versions compiles, the other does not.","shortMessageHtmlLink":"Fix capitalization error on protobuf wiki page (#1174)"}},{"before":"581634231b3cfedefcc755f12fd29e9d11df74b4","after":null,"ref":"refs/heads/james/fg-4427-conformance-test-for-log-time-0","pushedAt":"2024-05-28T23:10:47.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"jtbandes","name":"Jacob Bandes-Storch","path":"/jtbandes","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/14237?s=80&v=4"}},{"before":"7be806aee09f4c965a51e0f566bf1893f3b645c5","after":null,"ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-27T04:10:51.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"}},{"before":"c5db32fae1c9e7c4349f758a97cc50f8cb6c4979","after":"ef1daa3685b63a0ec39e4af698111094af068a6f","ref":"refs/heads/main","pushedAt":"2024-05-27T04:10:50.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: warn on duplicate matching schemas and channels (#1172)\n\n### Changelog\r\n\r\n- Added: `mcap doctor` now warns if a schema or channel record is\r\nduplicated in the data section. Some writers do this and it wastes\r\nspace, though reading still works.\r\n- Added: `mcap doctor` now prints an error if an MCAP contains messages\r\nin indexed chunks where their channel and schema information are not\r\nduplicated in the summary section. This is required by the spec:\r\nhttps://mcap.dev/spec#chunk-index-op0x08\r\n\r\n\r\n### Docs\r\n\r\nSpec updated to clarify what channels and schemas must be repeated in\r\nthe summary section.\r\n\r\n### Description\r\n\r\n\r\n\r\n\r\n\r\n\r\n
BeforeAfter
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
\r\n\r\n","shortMessageHtmlLink":"doctor: warn on duplicate matching schemas and channels (#1172)"}},{"before":"1e43775bbb5083b77532db7cb175df7876424f2a","after":"7be806aee09f4c965a51e0f566bf1893f3b645c5","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-27T04:03:22.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"don't use die","shortMessageHtmlLink":"don't use die"}},{"before":"31fafcb67f633b79b0ff456cb453e6e006d3c843","after":"1e43775bbb5083b77532db7cb175df7876424f2a","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-27T03:59:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"assert on error content","shortMessageHtmlLink":"assert on error content"}},{"before":"6ce864378a49aaa9233aad62b895cff3d135fe90","after":"31fafcb67f633b79b0ff456cb453e6e006d3c843","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-27T03:43:01.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"use quotes","shortMessageHtmlLink":"use quotes"}},{"before":"03a4235fb24a47970239540e33316eb284b9e701","after":"6ce864378a49aaa9233aad62b895cff3d135fe90","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-27T03:33:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"examine, capitalization","shortMessageHtmlLink":"examine, capitalization"}},{"before":"11f5351822116e8ddd8962999f57c970d195f3b5","after":"03a4235fb24a47970239540e33316eb284b9e701","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-23T11:32:49.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"amend spec for clarity","shortMessageHtmlLink":"amend spec for clarity"}},{"before":"7b428166c61ad3baf365e867a90f540f47a93160","after":"11f5351822116e8ddd8962999f57c970d195f3b5","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-23T11:24:33.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: check for duplicates correctly","shortMessageHtmlLink":"doctor: check for duplicates correctly"}},{"before":"ca653c8825ecae980d3601acd06f96c071b84151","after":"7b428166c61ad3baf365e867a90f540f47a93160","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-23T11:14:20.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: check for duplicates correctly","shortMessageHtmlLink":"doctor: check for duplicates correctly"}},{"before":"c67dfcf7185cb5839bd8a20b2a29c44779c88f1d","after":"ca653c8825ecae980d3601acd06f96c071b84151","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-23T11:08:08.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: check for duplicates correctly","shortMessageHtmlLink":"doctor: check for duplicates correctly"}},{"before":"ee02b35e76d69fefb9f28e8fb92072b208f9c496","after":"c67dfcf7185cb5839bd8a20b2a29c44779c88f1d","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-20T23:42:16.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: warn on duplicate matching schemas and channels","shortMessageHtmlLink":"doctor: warn on duplicate matching schemas and channels"}},{"before":"4ca3f7c6c6ede0427a1567e88a4914b1e9515605","after":null,"ref":"refs/heads/james/fg-7429-mcap-sort-doubles-of-chunks-increases-pre-and-post","pushedAt":"2024-05-20T23:31:07.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"}},{"before":"24737468344872cff47f1b5e2aa79cb31c09435a","after":"c5db32fae1c9e7c4349f758a97cc50f8cb6c4979","ref":"refs/heads/main","pushedAt":"2024-05-20T23:31:06.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"cli: sort does not duplicate channels and schemas on every message (#1171)\n\n### Changelog\r\n\r\n- fixed: `mcap sort` would rewrite the schema and channel for every\r\nmessage in the output file. This is fixed.\r\n\r\n### Docs\r\n\r\n\r\n\r\n### Description\r\n\r\n\r\n\r\n\r\n\r\n
BeforeAfter
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
\r\n\r\n","shortMessageHtmlLink":"cli: sort does not duplicate channels and schemas on every message (#…"}},{"before":"28b96e501c62ac81fd56ca033e94a95265622cd5","after":"ee02b35e76d69fefb9f28e8fb92072b208f9c496","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-20T23:29:05.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: warn on duplicate matching schemas and channels","shortMessageHtmlLink":"doctor: warn on duplicate matching schemas and channels"}},{"before":null,"after":"28b96e501c62ac81fd56ca033e94a95265622cd5","ref":"refs/heads/jrms/mcap-doctor-dupes","pushedAt":"2024-05-20T23:27:44.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: warn on duplicate matching schemas and channels","shortMessageHtmlLink":"doctor: warn on duplicate matching schemas and channels"}},{"before":"92053df1cc871d0c7b889cfaab5d8b1adb0a16df","after":"4ca3f7c6c6ede0427a1567e88a4914b1e9515605","ref":"refs/heads/james/fg-7429-mcap-sort-doubles-of-chunks-increases-pre-and-post","pushedAt":"2024-05-20T23:26:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"test sort","shortMessageHtmlLink":"test sort"}},{"before":"7678463edb95784ab92854e36e856e4ea9b317df","after":"92053df1cc871d0c7b889cfaab5d8b1adb0a16df","ref":"refs/heads/james/fg-7429-mcap-sort-doubles-of-chunks-increases-pre-and-post","pushedAt":"2024-05-20T23:25:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"doctor: warn on duplicate matching schemas and channels","shortMessageHtmlLink":"doctor: warn on duplicate matching schemas and channels"}},{"before":null,"after":"7678463edb95784ab92854e36e856e4ea9b317df","ref":"refs/heads/james/fg-7429-mcap-sort-doubles-of-chunks-increases-pre-and-post","pushedAt":"2024-05-20T22:52:33.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"cli: sort does not duplicate channels and schemas on every message","shortMessageHtmlLink":"cli: sort does not duplicate channels and schemas on every message"}},{"before":null,"after":"7678463edb95784ab92854e36e856e4ea9b317df","ref":"refs/heads/james-rust","pushedAt":"2024-05-20T22:49:52.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"cli: sort does not duplicate channels and schemas on every message","shortMessageHtmlLink":"cli: sort does not duplicate channels and schemas on every message"}},{"before":"2d926804f86e6494b240d77b8c19e404f2d334af","after":null,"ref":"refs/heads/jrms/faster-message-iterator","pushedAt":"2024-05-14T23:18:26.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"}},{"before":"fbcc2d31f68c25e79206c132fb4eb9dc52bc8331","after":"24737468344872cff47f1b5e2aa79cb31c09435a","ref":"refs/heads/main","pushedAt":"2024-05-14T23:18:25.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"go: reader: make ordered iteration faster (#1168)\n\n### Changelog\r\n- Added `Message.PopulateFrom([]byte, bool) error` method to Message, which\r\nallows readers to avoid reallocating Message structs between messages.\r\n- Added `MessageIterator.NextInto(*Message) (*Schema, *Channel, *Message,\r\nerror)` method to the MessageIterator interface. This method allows the\r\ncaller to re-use memory between message reads, and to avoid allocating a\r\nnew Message on the heap for every message read.\r\n- optimized indexed reading to read much faster and use much less\r\nmemory.\r\n\r\n### Docs\r\n\r\n- [x] Update docstrings\r\n- [x] Update examples\r\n\r\n### Description\r\n\r\nThis PR makes message iteration much faster. The included benchmark\r\nshows a pretty significant speedup:\r\n\r\n#### Before\r\n\r\nNote: the comparison benchmark based on `main` uses\r\nMessageIterator.Next() to read messages.\r\n\r\n```\r\n$ go test . -run=^$ -bench=BenchmarkReader -benchmem -benchtime=10x \r\ngoos: darwin\r\ngoarch: arm64\r\npkg: github.com/foxglove/mcap/go/mcap\r\nBenchmarkReader/inorder/no_index-8 10 532682221 ns/op 238.83 MB/s 7826155 msg/s 533306349 B/op 8054975 allocs/op\r\nBenchmarkReader/inorder/index_file_order-8 10 1886590288 ns/op 67.29 MB/s 2204813 msg/s 909884028 B/op 12114339 allocs/op\r\nBenchmarkReader/inorder/index_time_order-8 10 2248067917 ns/op 54.39 MB/s 1782145 msg/s 909889379 B/op 12114382 allocs/op\r\nBenchmarkReader/inorder/index_rev_order-8 10 2324738488 ns/op 48.90 MB/s 1602346 msg/s 910261216 B/op 12114355 allocs/op\r\nBenchmarkReader/inorder/bare_lexer-8 10 196806788 ns/op 660.42 MB/s 21640757 msg/s 17005039 B/op 4672 allocs/op\r\nBenchmarkReader/minor/no_index-8 10 509497992 ns/op 241.92 MB/s 7927082 msg/s 531637254 B/op 8054932 allocs/op\r\nBenchmarkReader/minor/index_file_order-8 10 1837735883 ns/op 66.97 MB/s 2194637 msg/s 909846889 B/op 12114373 allocs/op\r\nBenchmarkReader/minor/index_time_order-8 10 2250390946 ns/op 54.82 MB/s 1796497 msg/s 909844632 B/op 12114332 allocs/op\r\nBenchmarkReader/minor/index_rev_order-8 10 2360883250 ns/op 53.23 MB/s 1744292 msg/s 910212621 B/op 12114308 allocs/op\r\nBenchmarkReader/minor/bare_lexer-8 10 195830417 ns/op 638.20 MB/s 20912477 msg/s 15341232 B/op 4655 allocs/op\r\nBenchmarkReader/major/no_index-8 10 510946658 ns/op 241.74 MB/s 7921189 msg/s 532934768 B/op 8054945 allocs/op\r\nBenchmarkReader/major/index_file_order-8 10 1841807000 ns/op 66.35 MB/s 2174050 msg/s 909833931 B/op 12114348 allocs/op\r\nBenchmarkReader/major/index_time_order-8 10 2247866758 ns/op 54.53 MB/s 1786987 msg/s 909836941 B/op 12114379 allocs/op\r\nBenchmarkReader/major/index_rev_order-8 10 2328824133 ns/op 51.12 MB/s 1675101 msg/s 910215724 B/op 12114370 allocs/op\r\nBenchmarkReader/major/bare_lexer-8 10 198086167 ns/op 632.44 MB/s 20724011 msg/s 16635893 B/op 4661 allocs/op\r\nPASS\r\nok github.com/foxglove/mcap/go/mcap 248.060s\r\n```\r\n\r\n#### After\r\n\r\n```\r\n% go test . -run=^$ -bench=BenchmarkReader -benchmem -benchtime=10x \r\ngoos: darwin\r\ngoarch: arm64\r\npkg: github.com/foxglove/mcap/go/mcap\r\nBenchmarkReader/inorder/no_index-8 10 209814421 ns/op 596.62 MB/s 19550071 msg/s 17491784 B/op 6310 allocs/op\r\nBenchmarkReader/inorder/index_file_order-8 10 340508775 ns/op 360.08 MB/s 11799088 msg/s 10446040 B/op 16981 allocs/op\r\nBenchmarkReader/inorder/index_time_order-8 10 341343088 ns/op 359.00 MB/s 11763672 msg/s 10443932 B/op 16955 allocs/op\r\nBenchmarkReader/inorder/index_rev_order-8 10 348526088 ns/op 356.74 MB/s 11689775 msg/s 9996309 B/op 16964 allocs/op\r\nBenchmarkReader/inorder/bare_lexer-8 10 187405846 ns/op 664.57 MB/s 21776806 msg/s 17439823 B/op 4674 allocs/op\r\nBenchmarkReader/minor/no_index-8 10 211110267 ns/op 587.06 MB/s 19236916 msg/s 16522652 B/op 6284 allocs/op\r\nBenchmarkReader/minor/index_file_order-8 10 356903517 ns/op 336.77 MB/s 11035283 msg/s 10419253 B/op 17006 allocs/op\r\nBenchmarkReader/minor/index_time_order-8 10 552369746 ns/op 218.44 MB/s 7157955 msg/s 10444996 B/op 17744 allocs/op\r\nBenchmarkReader/minor/index_rev_order-8 10 555191658 ns/op 220.27 MB/s 7217936 msg/s 9971279 B/op 17665 allocs/op\r\nBenchmarkReader/minor/bare_lexer-8 10 194812112 ns/op 554.57 MB/s 18172347 msg/s 16473271 B/op 4670 allocs/op\r\nBenchmarkReader/major/no_index-8 10 211406192 ns/op 579.88 MB/s 19001450 msg/s 17365727 B/op 6291 allocs/op\r\nBenchmarkReader/major/index_file_order-8 10 354124750 ns/op 347.12 MB/s 11374355 msg/s 10418725 B/op 16979 allocs/op\r\nBenchmarkReader/major/index_time_order-8 10 566783688 ns/op 215.38 MB/s 7057431 msg/s 16452847 B/op 17690 allocs/op\r\nBenchmarkReader/major/index_rev_order-8 10 563155871 ns/op 218.15 MB/s 7148236 msg/s 15986112 B/op 17699 allocs/op\r\nBenchmarkReader/major/bare_lexer-8 10 195610721 ns/op 633.25 MB/s 20750327 msg/s 17316992 B/op 4672 allocs/op\r\nPASS\r\nok github.com/foxglove/mcap/go/mcap 68.716s\r\n```\r\n\r\nFor the unindexed message iterator, all of the difference comes from:\r\n- being able to re-use a Message struct between calls to Next()\r\n- switching from storing channels and schemas in maps to slices.\r\n- using and re-using an internal buffer for lexing MCAP records.\r\n\r\nFor the indexed message iterator, we do all of the same things plus:\r\n- We maintain a pool of decompressed chunk buffers, which are re-used\r\nafter all of the messages from a given chunk are read out.\r\n- We no longer read message indexes from the file, choosing instead to\r\nread the chunk content and build our own message index in memory. This\r\nallows us to read files written without message indexes in order, and\r\nalso reduces I/O, which in some cases is *probably faster* (slow network\r\nconnections with large message index overheads)\r\n- we no longer use a heap to maintain order, instead we maintain a\r\nsorted array of unread message indexes. Every time we encounter a new\r\nchunk:\r\n- if this chunk does not overlap with the last, clear the message index\r\narray and write the new chunk's messages in. If they are already in\r\norder, do not sort. This makes ordered iteration over an in-order MCAP\r\nas fast as unordered iteration.\r\n- if the new chunk's messages are not in order, sort the new chunk's\r\nmessages.\r\n- if the new chunk overlaps with the last, append the new chunk's\r\nmessages, and sort all unread messages.\r\n\r\n#### New API justification\r\nThe issues with `Next(p []byte) (*Schema, *Channel, *Message, error)`\r\nthat caused me to explore alternatives are:\r\n- The buffer `p` is used to store the message data, but if it isn't big\r\nenough, a new buffer is allocated for every message. A common idiom is\r\nto return the newly allocated buffer so that the caller can take\r\nownership of it and re-use it, but the current API doesn't allow that.\r\n- A new Message struct is allocated on the heap for every iteration.\r\nEven if the message goes out of scope on the next loop, this still\r\ncauses significant work for the garbage collector to do.\r\n\r\nThis new function signature re-uses the message passed in, if one is\r\npassed in. If `nil` is used, it creates a new Message.","shortMessageHtmlLink":"go: reader: make ordered iteration faster (#1168)"}},{"before":"36608cc3f36234ac0dc8da3afb732b8bf9e71792","after":"2d926804f86e6494b240d77b8c19e404f2d334af","ref":"refs/heads/jrms/faster-message-iterator","pushedAt":"2024-05-14T23:13:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"better names for benchmarks","shortMessageHtmlLink":"better names for benchmarks"}},{"before":"1e6e16bbb1313a28c6fbbc78ee54c11b2141d0b3","after":"36608cc3f36234ac0dc8da3afb732b8bf9e71792","ref":"refs/heads/jrms/faster-message-iterator","pushedAt":"2024-05-14T03:52:30.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"go back to nextinto","shortMessageHtmlLink":"go back to nextinto"}},{"before":"7903962ee79d8e222ac2488b5f56038b79728299","after":"1e6e16bbb1313a28c6fbbc78ee54c11b2141d0b3","ref":"refs/heads/jrms/faster-message-iterator","pushedAt":"2024-05-14T03:51:33.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"test slicemap","shortMessageHtmlLink":"test slicemap"}},{"before":"5e31a58314d36b325f3f14e196fae8e98ff29202","after":"7903962ee79d8e222ac2488b5f56038b79728299","ref":"refs/heads/jrms/faster-message-iterator","pushedAt":"2024-05-14T03:47:56.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"test slicemap","shortMessageHtmlLink":"test slicemap"}},{"before":"6f8c534488154cf554bec7fa8a857572f7e0f25e","after":"5e31a58314d36b325f3f14e196fae8e98ff29202","ref":"refs/heads/jrms/faster-message-iterator","pushedAt":"2024-05-14T03:42:42.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"james-rms","name":null,"path":"/james-rms","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/18162835?s=80&v=4"},"commit":{"message":"test slicemap","shortMessageHtmlLink":"test slicemap"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEWFKOqAA","startCursor":null,"endCursor":null}},"title":"Activity · foxglove/mcap"}