Skip to content
/ bigfile Public

Viewer for very large logfiles and JSON/CSV/XML dumps

Notifications You must be signed in to change notification settings

pweerd/bigfile

Repository files navigation

BigFile (viewer for very large logfiles)

Bigfile is designed to view and search in very large text files like logfiles, json-dumps, xml-dumps, etc.

I needed something like 'less' on unix systems, but easier and for Windows. Stumbled across the logviewer from Mark Woan, which did a lot of good things, but not everything. So I took his approach as an example and created Bigfile.

Main Bigfile features:

  • Loading of compressed (.gz or .zip) files without unpacking.
  • Background loading of files
  • Different encodings
  • Memory compression to support loading huge files in memory
  • Multi-threaded search
  • Search via boolean expressions and regex (grep).
  • Mimics some of less's navigation shortcuts
  • Viewing line-content as text/json/xml/csv
  • Supports multiple encodings

Installer can be downloaded from https://bitmanager.nl/distrib/

Maximum performance

Although Bigfile works without additional tools, it will use Bitmanager's core components if installed. What you get with the core-components is:

  • Much faster gzip decompression (bypassing sharpziplib)
  • Much smaller memory pressure because the needed memory will be compressed
  • Logging

The core components can be downloaded from https://bitmanager.nl/distrib/ as well.

Grid

The grid is interesting. Earlier versions of Bigfile were using ObjectListView (http://objectlistview.sourceforge.net/cs/index.html). A grid-library based on a ListView. Although it was working nice, I recently encountered errors in an application eating a textfile of 35GB and 350M lines. Viewing this file with Bigfile wasn't possible: it showed only 100M lines.

It turned out that this is a limitation in the ListView component. And it makes sense: all offsets are calculated as int's, limiting the #rows to int.MaxValue / RowHeight... Apparently the ListView team decided to introduce a limit of 100M rows to prevent integer overflows.

While searching for a replacement grid, I stumbled across Tomasz Rewak's DynamicGrid. It was too limited for me, and it didn't support enough rows, but it was nice. I took it as a starting point.

So now Bigfile's grid supports 2 billion rows and it is much, much faster than the virtual ObjectListView.

Btw: scrollbars suffer from the same limitation: all values are integer-based, and share the same limitation...

Screenshot

Credits