Switch to gengo

[spenserblack]

See #1152 for previous conversation. With the drastic API changes to gengo it started to feel like more work to modify the existing PR instead of starting over.

To Do

• Gengo needs to support the remaining languages (or at least have reasons to not support them) (Onefetch languages checklist spenserblack/gengo#34)
• Change pretty much everything that previously matched on Language and tokei::LanguageType enums.
• Drop the "blob size" stat (Switch to gengo #1305 (comment))

Resolves #26
Closes #1152

[spenserblack]

@o2sh Before this moves any further we need to determine if we want to move forward with analyzing a git ref (HEAD) or a folder (I think preferably we don't support both). I've started listing reasons for each here: #1303

[o2sh]

It would be nice to have language type and glob filtering for the V1

[o2sh]

or is this going to be handled via .gitattributes?

[spenserblack]

That depends on the "file source." Right now the Git file source is highly opinionated and would use .gitattributes for ignoring files, overrides, etc.

---

I plan on the Directory file source being much less opinionated (can't get much more generic than just reading a folder 🙂). The functionality isn't there yet, but I was considering adding ways to configure excluding/including files. Right now .gitignore and .ignore files would ignore files via the ignore crate's default behavior. I was hoping to discuss the usage sometime, but it would probably either be like this:

let file_source = Directory::with_config("./", Config { ignored_files: Some(vec![]) });

or like this:

let file_source = Directory::new("./").extend_ignored_files(&[]);

[o2sh]

Very clear, and it also aplies to the language type filter ?

[spenserblack]

By default Gengo marks Programming, Markup, and Query as detectable, and Prose and Data as not detectable. detectable is basically a boolean that answers "should this factor in to the stats?" Also, documentation, generated files, and vendored files are not detectable by default. Assuming we use the Git file source with .gitattributes, this is example usage:

# Markdown is Prose, but we still want it in the stats
*.md gengo-detectable

# JavaScript is Programming, but we want to exclude it
*.js -gengo-detectable

# The contents of dist/ are not generated, so they will be detectable
dist/* -gengo-generated
# OR
# Even though dist/* is generated, it should still be included in the stats
dist/* gengo-detectable

By default any file that is not detectable is excluded from the Summary.

---

Again, the Git file source is highly opinionated and tries to behave a lot like github-linguist. The usage with the Directory file source is still undecided, and right now it doesn't implement any overrides. Or we could write our own file source (#1303 (reply in thread)) if we want very specific behavior that wouldn't be suitable for the gengo crate.

But, however we implement it, IMO the best way would be to inform gengo what files are and aren't detectable, and then make use of either the Analysis (detailed results) or Summary (simplified results).

---

tl;dr Yes, gengo would be the one handling included/excluded types, and we'd just pass configuration to it somehow.

[o2sh]

Are the pending changes taken into count for the language analysis?

[spenserblack]

#1303 (comment)

[spenserblack]

A bit of refactoring is inevitable to make things make more sense when using gengo over tokei. Stumbled on this. Don't mean to be harsh, but is a function for the sole purpose of indexing into [0] really necessary? Especially now that newer versions of Rust have .first() and .get(0) to safely index into collections.

[o2sh]

Sure, go ahead.

But for an easier review, we can keep the refactoring/cleanup in a separate PR, so we can focus on the current subject and don't lose track of the important changes.

[spenserblack]

Refactored it to handle a OOB panic that was introduced (see 82da923 for more info).

[spenserblack]

So I think that covers pretty much everything. All that's left is to add more language support (spenserblack/gengo#34) and make a new release to add back the supported language variants and then we're good AFAIK.