Skip to content

Commit

Permalink
Updates to the way meta indexing is handled for filestore. (#4450)
Browse files Browse the repository at this point in the history
Historically we kept indexing information, either by sequence or by
subject, as a per msg block operation. These were the "*.idx" and
"*.fss" indexing files. When streams became very large this could have
an impact on recovery time. Also, for encryption the fast path for
determining if the indexing was current would require loading and
decrypting the complete block.

This design moves to a more traditional WAL and snapshot approach. The
snapshots for the complete stream, including summary information, global
per subject information maps (PSIM) and per msg block details including
summary and dmap, are processed asynchronously. The snapshot includes
the msg block and has for the last record hash that was considered in
the snapshot. On recovery the snapshot is read and processed and any
additional records past the point of the snapshot itself are processed.
To this end, any non-system removal of a message has to be expressed as
a delete tombstone that is always added the the fs.lmb file. These are
processed on recovery and our indexing layer knows to skip them.

Changing to this method drastically improves startup and recovery times,
and has simplified the code. Some normal performance benefits have been
seen as well.

Signed-off-by: Derek Collison <derek@nats.io>
  • Loading branch information
derekcollison committed Aug 30, 2023
2 parents ce08d45 + adef828 commit b9b284d
Show file tree
Hide file tree
Showing 5 changed files with 1,758 additions and 1,558 deletions.

2 comments on commit b9b284d

@scosol
Copy link

@scosol scosol commented on b9b284d Sep 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the introduction of a WAL not decrease streaming throughput capacity?

@derekcollison
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just treat the lmb as a WAL and introduce delete tombstones that need processing on restart. Performance went up slightly for general streaming throughput and of course restart was greatly improved.

Please sign in to comment.