Determining Repetition

[Chaddeus]

How might I use WinkNLP to determine if a writer's content contains too much repetition?

I was thinking maybe something along the lines of a calculation of a the relative frequency a "proper" word (not a, the, an, etc) appears in the content. But I think it would also be important to consider pairs of words, or even three-word sets.

[sanjayaksaxena]

1. For word repetition, remove stopwords and use its.lemma & as.freqTable to build a frequency table and shortlist outliers or tokens > 75%tile and highlight them.
2. If you want to make to more sophisticated by looking at repetitions at short (token) distance using index() method.
3. For sentences, use a similarity method to flag near by sentences with very high similarity values.

[Chaddeus]

Fantastic. Thanks for the advice. Awesome project btw. I see all kinds of potential here.

[sanjayaksaxena]

You may like to look at one of winkjs repo: https://github.com/winkjs/showcase-writing-assistant.