VisiData 4Ever

[ajkerrigan]

Following some discussion with @saulpw and @anjakefala - this is a follow-up to #595 since that one has gotten quite long. (That's a good thing!)

[saulpw]

There were many responses in favor of VisiData from this Twitter thread, in which @simonw asks:

If someone gives you a CSV file with 100,000 rows in it, what tools do you use to start exploring and understanding that data?

which is exactly what VisiData was built for :)

[saulpw]

Up to 1M rows, visidata is my go to tool. Quick custom filtering and histograms give a good sense of the data within a minute or so.

• Yasharth

[frosencrantz]

I like it when I see these comments in which people reference VisiData on Hacker News and it is not the main point of their comment. It is just a part of how they performed the task, like you might mention nailing piece of wood together as part of assembly. This is a tool in their tool belt, that is a high compliment.

pasabagi writes on Bank of England expects UK to fall into longest ever recession

Are we looking at the same graph? Aside from 2012-14, it consistently shows the UK trailing the pack in terms of growth, often by multiple percentage points. Bear in mind, even a single percentage point is quite a big deal: the chinese economy was 'roaring' at an average of 4% growth per year.
I loaded up the csv in visidata, and the graph you sent me shows the UK trails the EU growth average by half a percentage point since 2008. Against a relatively well-performing economy like germany, it trails by 0.94% over the same period.

OR

farresito writes on: Integrating variants in 42k autism cases identifies mutations in new genes

Getting your whole genome sequenced is around 200-300$. Nebula or Dante labs offer that. Once you receive the data, which is going to be a VCF file, you have to annotate it with something like Ensembl VEP, which adds useful information to analyze it. After that, you can download a list of genes associated with autism from SFARI and extract the variants that you have in those genes. You remove the common variants (say those that have more than 5% frequency) and focus on the coding variants (the ones that affect the exome). From there on, the interesting part begins. Depending on the kind of variant, you can get certain conclusions or certain others. There are tools that predict the pathogenicity of certain variants; other variants you know almost certainly that they render the resulting protein non-functional, so you have to take a closer look at the gene and see if having a single copy of the gene is enough. And so on.
It's not complicated at all and a lot of it can be done with basic Linux tools, like grep, and visualized with CSV viewers like Visidata. That's just a basic explanation. If you end up getting your genome sequenced and want to analyze it, let me know in a comment.

[saulpw]

Thanks, @frosencrantz ! How do you find these? I have a watch on 'visidata' and I don't think these turned up.

[frosencrantz]

This is what I check every so often:
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=visidata&sort=byDate&type=all

[aborruso]

Thanks, @frosencrantz ! How do you find these? I have a watch on 'visidata' and I don't think these turned up.

You have also RSS feed:

• https://hnrss.org/newest?q=visidata
• https://hnrss.org/newcomments?q=visidata

[frosencrantz]

Now I want RSS loading support in VisiData beyond what the XML loader does.

[saulpw]

David Thiel wrote up a Mastodon thread about their ucluster VisiData plugin for grouping similar text to detect spam:

The plugin implementation is due to the fact that while our “prod” is generally BigQuery with various frontends, I do much of my initial analysis (and sometimes all of it) using API-emitted JSONL in VisiData (because it is awesome), running it through a local enrichment pipeline. Part of this pipeline can be ucluster — here, I’ve run the fuzzy-cluster command on the “text” column, which returns the numeric cluster to which the tweet text belongs, or -1 if it couldn’t decide.

[saulpw]

From https://twitter.com/LovelyDataCZ/status/1600599361620217856:

[translation] What do you think, is there a tool that is equal to Excel or even better? A tool that can spreadsheet, is easy to use, is fast and does not cost a penny? https://www.lovelydata.cz/blog/visidata-lepsi-nez-excel/

[frosencrantz]

From https://chaos.social/@scy/110396676007788276

How could I miss out on #VisiData for so long? This might become my new favorite #CLI tool.

If you do anything with data and enjoy working in the terminal, check it out. It can

• provide a #TUI for viewing and editing data in #CSV, #Excel, #SQLite, #JSON, #YAML & #XML files and quite a few more
• sort, filter, join and edit that data, across files and across formats
• convert between the formats (interactively or not)
• record & play macros
• be scripted in #Python

[anjakefala]

https://discord.com/channels/880915750007750737/880915984481939466/1120227410588225548

just discovered that vd can draw shapefiles and i cannot stop smiling. omg thank you
very, very slightly life-changing lol

I'm doing some work with TIGER data, nothing super exciting but being able to look at stuff in vd to spot-check what I am doing is very useful (usually I'd load it into qgis and that takes a while)

[anjakefala]

Can confirm, I consider visidata to be the gold standard for data exploration.

I have largely given up on spreadsheets, replacing them with relational databases. And on top of being a world class tool for understanding the data for import, visidata makes for a pretty great query pager.

• somat https://news.ycombinator.com/item?id=38894276

[larstiq]

Originally posted a slightly different version on the #visidata irc channel but decided to go for a wider audience on Mastodon (https://mathstodon.xyz/@larstiq/112284974755111300)

One little #visidata trick that brought me joy yesterday: sort a dataset by one column (e.g. modified date), select a range of interesting rows, sort by another column (e.g name or description) and then use {/} to jump to the previous/next selected rows and inspect them in the context of their neighboring rows. One of the most delightful pieces of software I use on a regular basis.