should we refactor for better performance

[asimkt]

Since Intl.Segmenter (link) is available for most of the users, I think it's better to mention what are the extra features [sentence-splitter](https://github.com/azu/sentence-splitter) is providing. And it will be better to tell to use one over another with some scenarios so that user can take an informed decision.

[azu]

Intl.Segmenter is tokenizer, it split text into words(tokens).
So, it does not split text into sentences.

https://excalidraw.com/#json=PKbVfkc-JwDScvZHN8Ysu,jgqaHTMS03q6BsvQRYO1dg

[danielweck]

Intl.Segmenter is tokenizer, it split text into words(tokens).

This statement is correct ... but paints an incomplete picture :)

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter/Segmenter#granularity

Option granularity accepts value sentence (not just word).

---

const text = "吾輩は猫である。名前はたぬき。";
const japaneseWordSegmenter = new Intl.Segmenter("ja-JP", { granularity: "word" });
[...japaneseWordSegmenter.segment(text)].forEach((s)=>console.log(JSON.stringify(s, null, 4)));

==>

{
    "segment": "吾輩",
    "index": 0,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "は",
    "index": 2,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "猫",
    "index": 3,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "で",
    "index": 4,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "ある",
    "index": 5,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "。",
    "index": 7,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": false
}
{
    "segment": "名前",
    "index": 8,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "は",
    "index": 10,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "たぬき",
    "index": 11,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": true
}
{
    "segment": "。",
    "index": 14,
    "input": "吾輩は猫である。名前はたぬき。",
    "isWordLike": false
}

---

const text = "吾輩は猫である。名前はたぬき。";
const japaneseSentenceSegmenter = new Intl.Segmenter("ja-JP", { granularity: "sentence" });
[...japaneseSentenceSegmenter.segment(text)].forEach((s)=>console.log(JSON.stringify(s, null, 4)));

==>

{
    "segment": "吾輩は猫である。",
    "index": 0,
    "input": "吾輩は猫である。名前はたぬき。"
}
{
    "segment": "名前はたぬき。",
    "index": 8,
    "input": "吾輩は猫である。名前はたぬき。"
}