Skip to content

This package consists of functionalities for dynamic topic modelling and its visualization

Notifications You must be signed in to change notification settings

GSukr/dtmvisual

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

dtmvisual

Package for Dynamic Topic Modeling training and visualization

Requirements

Currently, this package requires Python 3.4+ and the following dependencies:

  • 'gensim==3.4.0',
  • 'seaborn==0.8.1',
  • 'matplotlib==2.0.2'

We highly recommend you to clean and preprocess the collection before using it. We run the Dynamic topic modelling on the collection of abstracts from the SIOE conference from 1998 to 2018. We have 3225 documents. We use the following code for preprocessing code.

The following steps help install and use dtmvisual:

  1. Download dtmvisual repository
  2. Run in the terminal
$ cd Downloads
$ cd dtmvisual-master
$ python setup.py install
  1. Import dtmvisual package into a Python environment:
import dtmvisual

You can use the package for the following purposes:

  • Training the model:
  1. Please download the dtm binaries for your OS version and use it in the code as:
dtm_path = "/path/to/dtm/binary"
  1. Divide your collection of texts on time slices with desired granularity. For instance, the distribution of 3225 documents per 21 years (1998-2018) in our dataset equals the following list:
time_seq = [50, 87, 90, 106, 93, 71, 176, 157, 175, 176, 111, 217, 114, 175, 152, 238, 158, 195, 254, 240, 191]
  1. Transform your collection of documents to list of tokens. Then convert the list to the bag-of-words format:
sentences = docs_to_list(self.sentences)
corpus = corpus_dtm(sentences)
  1. Train the model:
model = dtm_model(dtm_path, corpus, time_seq, num_topics=8,
                 id2word=corpus.dictionary, alpha=0.01) 

Words evolution

  1. Create a dataframe with the columns ['topicId', 'period', 'word', 'weight']:
df = visualize_words.DF(number_of_timespans, number_of_topics, model)
  1. Ensure the words partition based on TopicID:
d = visualize_words.partitioning(df)
  1. Visualize the words evolution:
visualize_words.visualize_words(df, d)

With our dataset we got the following words evolution:

See more in: the folder

Topics evolution

  1. Create the dataframe with the topical distribution in a document:
df = VisualizeTopics.topic_distribution(number_of_topics, model, time_seq)
  1. Visualize the topics evolution:
VisualizeTopics.visualize_topics(df)

With our dataset we got the following words evolution: topics

Please note that we smoothed the lines with moving average for topic distribution and added the topic titles based on the expert knowledge after analysis of the top words for each topic. If you run the code your topics titles will correspond to the topic ID (e.g. Topic 1). You can adjust your Dataframe before running 'VisualizeTopics.visualize_topics(Dataframe)' to change the topic names and apply moving averdage.

Moreover, you save and then download your DTM model using:

save_model(model, output_path, output_name)
model = load_saved_model(output_path, output_name)

To print the words for a timeslice for a topic use:

print_all_topics(model, topicid = None, time = None, num_words=10)

The results with the SIOE data has been presented at the SIOE 2019 conference: abstract

About

This package consists of functionalities for dynamic topic modelling and its visualization

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages