Exploring Topic Modeling

Overview

In this assignment, you'll be exploring topic modeling. You can use whatever topic modeling package you'd like; some popular ones include MALLET, gensim, and scikit-learn; also see here for various pointers with a leaning toward using the R programming language. The intent is not for you to implement a sampler for LDA yourselves.

See here for a quick example of how to run LDA and build a topic model using MALLET. It assumes there is a directory called convote_train containing .txt files one per document, where each .txt file is already whitespace-tokenized. (You can download that here). For example:

mr. chairman , i thank the gentleman from texas ( mr. barton )  for yielding me this time .
the gentleman has done a magnificent job leading the committee on this new bill .
i would just say , in america we face some great challenges with regard to formulation of our energy policy .
the oil demand growth keeps rising due to the industrialization of the emerging world .
china consumes 7 million barrels per day ; and if china 's rise in world prominence is similar to that of korea and japan , china will consume 20 million barrels per day in less than 10 years .
the last big oil discovery was 30 years ago in the north sea .
china is trying to buy oil companies in canada ; india is trying to buy oil companies in russia ; the present world production capacity is 83 million barrels a day ; and we are running an estimated 81.5 million today , which means we are in the red zone .
the 14 largest oil fields in the world are 40 years old .
once they are taken out to 50 percent , water and fluids need to be pumped to keep production at existing levels .
we have some significant challenges .
support this bill .
See MALLET documentation for an alternative input format where instead of a directory, there's a single input file with one document per line. FYI, MALLET's --output-topic-keys argument specifies a file showing the top words for each topic (i.e. the words at the top of the sorted topic-word probability distribution); the --output-doc-topics argument specifies a file where it will write the inferred document-topic distribution; and the --output-state file contains the final sampled assignment of topics to words.

What to do

You can use any of the following datasets, or feel free to suggest some other dataset that you'd like to use instead.

Exploration 1: Trying out different numbers of topics.

As we discussed, LDA is an unsupervised learning algorithm. It is, more specifically, a parametric unsupervised learner, which here signifies that the key modeling parameter, the number of topics, is fixed in advance. In that respect it is very much like the unsupervised HMM you may have looked at in a previous assignment: you're going to get different behavior depending on the value you choose for the parameter.

As Steyvers and Griffiths (2007) discuss, there are a variety of ways to choose the number of topics. One of the simplest is simply to try out different numbers -- what happens when you use 5 topics? 10? 25? 30? 50? 100? 300? Using whichever topic modeling package you decide to work with, try a range from small to larger numbers of topics, and report on what you find. How does the collection of topics change? If the package you use gives you a quality measure for the model (for example (e.g. many provide the likelihood), how well does that value correspond with your subjective impression of the topics in terms of their quality, coherence, interpretability?

For those who are interested in learning more about evaluating topic models, a really groundbreaking paper was Chang et al. (2009)'s "Reading Tea Leaves" paper, and an important follow-on, providing an automated approximation to Chang et al.'s procedure, is Lau et al. (2014); there's code for doing that automatic evaluation here.

Exploration 2: The influence of phrases

Although LDA is a "bag of words" model and is often implemented using unigrams, there's good reason to think that this could potentially be suboptimal. For example in a document with a phrase like white house, you probably don't want the words white and house to be treated the same as if they occurred in the sentence John painted his house white.

Recently Brendan O'Connor's group at UMass has been investigating the role of phrase detection for NLP tasks; their work can be found in Handler et al. (2016a), A Little Bit of NLP Goes A Long Way: Adding Phrases to the Term-Document Matrix using Finite-State Shallow Parsing and Handler (2016b), Bag of What? Simple Noun Phrase Extraction for Text Analysis, which includes an open-source implementation of their approach linked here. A key element of their argument is that shallow noun phrase detection is a good way to group phrases into tokens as a preprocessing step for many NLP tasks, including topic modeling.

Now, I have a conjecture: I think they may be oversimplifying by not distinguishing among two very different kinds of phrases. Some phrases, like white house, new york, or vice president, are partially or mostly non-compositional. In those cases it feels to me like it makes sense to preprocess them into single tokens, because the "whiteness" or "houseness" of the White House is probably not relevant in terms of the topics it's mentioned in, any more than the "newness" of New York is relevant to it as a geographic location. (For example, White House can't be paraphrased "house that is white", vice president is not the same as "president who is associated with vice", etc.)

On the other hand, if you have a compositionally interpreted noun phrase like hardware store, the very fact of its compositionality means that the meanings of the individual words are still relevant: a hardware store is a store that sells hardware, and therefore store and hardware are independently relevant parts of its meaning.

Operationally, my conjecture suggests the following concrete question: instead of tokenizing phrases in preprocessing, is it perhaps better only to tokenize non-compositional phrases?

So, for this exploration, I'd like you to explore running topic models with different forms of preprocessing. Specifically, I'd like you to try running topic models with the following forms of preprocessing:

At least two of the following variations:

In terms of tools, you might find spacy useful since it identifies part of speech tags, noun phrases, and named entities (and even gives you dependency parses if you like). There are various tutorial introductions that you might find useful. Generally, I am open to people sending around pointers and helping each other out; I'm also open to people collaborating on this assignment.

Extra credit possibilities (up to 10%)

What to turn in

The main product for this assignment should be a thoughtful written discussion that includes what you did, why you did it, and what you found. Please also turn in a .tar.gz or .zip file of your code.

This assignment is due at the start of class on Wednesday.