Recognizing abbreviations - custom entities

[jonzim]

The sentence parser is not handling certain strings well for me... such as

1 tbs. milk.
3 tbs unsweetened bakers chocolate.
2 tbs. of sugar.
1/4 tsp. almond extract.
1 c. milk.
2 tbs. whipped cream (for garnish).

i tried (roughly):

  var patterns = [
                { name: 'measurement', patterns: ['cup', 'c', 'teaspoon', 'tsp', 'tablespoon', 'tbs'] }
            ];
  wink = winkNLP(patterns);
  var d = wink.readDoc(input);
  var sentences = d.sentences();
  console.log(sentences.out());

output is breaking up the abbreviations as though it were the last word in the sentence... like this.

[
  '1 tbs.',
  'milk.',
  '3 tbs unsweetened bakers chocolate.',
  '2 tbs.',
  'of sugar.',
  '1/4 tsp.',
  'almond extract.',
  '1 c. milk.',
  '2 tbs.',
  'whipped cream (for garnish).',
  ''
]

because it's breaking apart the sentence at the punctuation for the abbreviation, it's making it impossible for me to define another pattern to recognize the full line as a phrase.

[sanjayaksaxena]

Hello @jonzim

The current model is not trained on recipe/cooking related corpus/data. As a result it is breaking the sentences incorrectly.

With you help we can possibly train the model to handle such text and also add QUANTITY entity detection as well. We will require a set of representative texts that will include such terms/quantities to begin with.

Best,
Sanjaya

[sanjayaksaxena]

Hello @jonzim

While we enhance the model, following should solve your problem:

const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );
const its = nlp.its;
const patterns = [
  { name: 'quantity', patterns: ['[CARDINAL] [c|c.|tbsp|tsp|tbs] [|.]'] },
  { name: 'quantity', patterns: [ 'CARDINAL cup', 'CARDINAL tea spoon', 'CARDINAL table spoon'] }
];
nlp.learnCustomEntities(patterns);
const text = `1 tbs. milk.
3 tbs unsweetened bakers chocolate
2 tbs. of sugar.
1/4 tsp. almond extract.
1 c. milk.
2 tbs. whipped cream (for garnish).`
const doc = nlp.readDoc( text );
console.log( doc.customEntities().out() );
// -> ["1 tbs.", "3 tbs", "2 tbs.", "1/4 tsp.", "1 c.", "2 tbs."]

Hope it helps.

Best,
Sanjaya

[jonzim]

Thank you - Yes, this helps. Even though the sentence parser still breaks on the abbreviation when I use sentences(), it's detecting the quantities when I modify the patterns as you defined. I was only going to detect one line at a time anyway, so it will be able to fulfill its purpose. Besides, I can probably get past the abbreviation issue by replacing abbreviations with the full value before running it through the NLP.

If I were to define a bunch of quantity patterns, would it be possible to include my own previously defined patterns (in this case quantity) later on in another pattern, like this where I try to include quantity in an ingredient phrase?

const patterns = [
  { name: 'quantity', patterns: ['[CARDINAL] [c|c.|tbsp|tsp|tbs] [|.]'] },
  { name: 'quantity', patterns: [ 'CARDINAL cup', 'CARDINAL tea spoon', 'CARDINAL table spoon'] }
  { name: 'ingredientPhrase', patterns: ['[|quantity] [|of] [|ADJ] [NOUN]'] }
];

And to borrow some regex terminology... are the entity pattern matches lazy or greedy? (as in match the larger phrase [ingredientPhrase] vs the shorter phrase [quantity]), or would both matches be returned?

this of course isn't working for me, but hopefully it gives an idea of what I'm trying to accomplish.

[sanjayaksaxena]

Hello @jonzim

We have made some improvements in wink-eng-lite-web-model — it now able to handle standard abbreviations such as tsp. and tbsp. during SBD. This is version 1.4.3.

Custom entities performs a greedy match and in case of multiple matches, the longest one is given preference.

The construct given in your example is not supported. However we are planing to add detection of more entities including QUANTITY.

We may be able to help you further if you share more details of your use case. We are giving below the revised code below.

Best,
Sanjaya

const winkNLP = require( 'wink-nlp' );
const model = require( 'wink-eng-lite-web-model' );
const nlp = winkNLP( model );
const patterns = [
  { name: 'quantity', patterns: ['[CARDINAL] [c|c.|tbsp|tbsp.|tsp|tsp.]'] }
];
nlp.learnCustomEntities(patterns);
const text = `1 tbsp. milk.
3 tbsp unsweetened bakers chocolate.
2 tbsp. of sugar.
1/4 tsp. almond extract.
1 c. milk.
2 tbsp. whipped cream (for garnish).`
const doc = nlp.readDoc( text );
console.log( doc.customEntities().out() );
// -> ["1 tbs.", "3 tbs", "2 tbs.", "1/4 tsp.", "1 c.", "2 tbs."]
doc.sentences().out();
// -> 1 tbsp. milk.
// -> 3 tbsp unsweetened bakers chocolate.
// -> 2 tbsp. of sugar.
// -> 1/4 tsp. almond extract.
// -> 1 c. milk.
// -> 2 tbsp. whipped cream (for garnish).