Skip to content

Programming for NLP Project - Implement a basic n-gram language model and generate sentence using beam search

Notifications You must be signed in to change notification settings

touhi99/N-gram-Language-model

Repository files navigation

N-gram Language Model

Data

  • train_shakespeare.txt (train file)
  • dev_shakespeare.txt (test file)
  • new_shakespeare.txt (generated file, based on bigram, beam size 30)

Files

  • main.py
  • corpus.py
  • lm.py
  • beam.py
  • test_ngram.py

Model

  • trained_model_ngram.pkl (Saved after training)

Requirement

`pip install docopt`
`pip install click`

How-to

python main.py train [--n <n>] [--path <path>] main.py generate [--lines <n>] main.py perplexity [--path <path>] main.py common [--number <n>] main.py testcase main.py --help

Options: --n <n> Number of n-gram --path <path> Train/Test file path --lines <n> No. of lines to be generated --number <n> No. of n-gram to show --help Show this screen

python test_ngram.py

About

Programming for NLP Project - Implement a basic n-gram language model and generate sentence using beam search

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages