If you have not already done so, please make sure you have done the readings for the previous lecture.
See here for a quick example of how to run LDA and build a topic model using MALLET. It assumes there is a directory called convote_train containing .txt files one per document, where each .txt file is already whitespace-tokenized. (You can download that here). For example:
mr. chairman , i thank the gentleman from texas ( mr. barton ) for yielding me this time .
the gentleman has done a magnificent job leading the committee on this new bill .
i would just say , in america we face some great challenges with regard to formulation of our energy policy .
the oil demand growth keeps rising due to the industrialization of the emerging world .
china consumes 7 million barrels per day ; and if china 's rise in world prominence is similar to that of korea and japan , china will consume 20 million barrels per day in less than 10 years .
the last big oil discovery was 30 years ago in the north sea .
china is trying to buy oil companies in canada ; india is trying to buy oil companies in russia ; the present world production capacity is 83 million barrels a day ; and we are running an estimated 81.5 million today , which means we are in the red zone .
the 14 largest oil fields in the world are 40 years old .
once they are taken out to 50 percent , water and fluids need to be pumped to keep production at existing levels .
we have some significant challenges .
support this bill .
See MALLET documentation for an alternative input format where instead of a directory, there's a single input file with one document per line. FYI, MALLET's --output-topic-keys argument specifies a file showing the top words for each topic (i.e. the words at the top of the sorted topic-word probability distribution); the --output-doc-topics argument specifies a file where it will write the inferred document-topic distribution; and the --output-state file contains the final sampled assignment of topics to words.
Do both of the two explorations below.
As Steyvers and Griffiths (2007) discuss, there are a variety of ways to choose the number of topics. One of the simplest is simply to try out different numbers -- what happens when you use 5 topics? 10? 25? 30? 50? 100? 300? Using whichever topic modeling package you decide to work with, try a range from small to larger numbers of topics, and report on what you find. How does the collection of topics change? If the package you use gives you a quality measure for the model (e.g. many provide the likelihood), how well does that value correspond with your subjective impression of the topics in terms of their quality, coherence, interpretability?
For those who are interested in learning more about evaluating topic models, a really groundbreaking paper was Chang et al. (2009)'s "Reading Tea Leaves" paper, and an important follow-on, providing an automated approximation to Chang et al.'s procedure, is Lau et al. (2014); there's code for doing that automatic evaluation here.
Recently Brendan O'Connor's group at UMass has been investigating the role of phrase detection for NLP tasks; their work can be found in Handler et al. (2016a), A Little Bit of NLP Goes A Long Way: Adding Phrases to the Term-Document Matrix using Finite-State Shallow Parsing and Handler (2016b), Bag of What? Simple Noun Phrase Extraction for Text Analysis, which includes an open-source implementation of their approach linked here. A key element of their argument is that shallow noun phrase detection is a good way to group phrases into tokens as a preprocessing step for many NLP tasks, including topic modeling.
Now, I have a conjecture: I think they may be oversimplifying by not distinguishing among two very different kinds of phrases. Some phrases, like white house, new york, or vice president, are partially or mostly non-compositional. In those cases it feels to me like it makes sense to preprocess them into single tokens, because the "whiteness" or "houseness" of the White House is probably not relevant in terms of the topics it's mentioned in, any more than the "newness" of New York is relevant to it as a geographic location. (For example, White House can't be paraphrased "house that is white", vice president is not the same as "president who is associated with vice", etc.)
On the other hand, if you have a compositionally interpreted noun phrase like hardware store, the very fact of its compositionality means that the meanings of the individual words are still relevant: a hardware store is a store that sells hardware, and therefore store and hardware are independently relevant parts of its meaning.
Operationally, my conjecture suggests the following concrete question: instead of tokenizing phrases in preprocessing, is it perhaps better only to tokenize non-compositional phrases?
So, for this exploration, I'd like you to explore running topic models with different forms of preprocessing. Specifically, I'd like you to try running topic models with the following forms of preprocessing:
At least two of the following variations:
In terms of tools, you might find spacy useful since it identifies part of speech tags, noun phrases, and named entities (and even gives you dependency parses if you like). There are various tutorial introductions that you might find useful, as well as good documentation on its processing pipelines. Generally, I am open to people sending around pointers and helping each other out. I'm also open to people collaborating on this assignment; if you're interested in doing that contact me for guidelines.
Researchers at CMU have developed non-negative sparse embeddings, which are similar to other kinds of embeddings you've seen (e.g. word2vec) except that the individual dimensions are more separable and interpretable (see Fyshe et al. (2015)). What if instead of using the words in a document themselves, we were to treat words as
WORD D1 D2 D3 apple 0.5 0.1 0.4 dessert 0.8 0.2 0.0 kitchen 0.0 0.8 0.2 table 0.0 0.6 0.4 windows 0.0 0.4 0.6the LDA vocabulary would be the semantic properties {D1,D2,D3}, and using the threshold of 0.3 the original document At my kitchen table I had an apple for dessert would be transformed into the document D2 D2 D1 D3 D1 (with semantic properties now being the "words" in the transformed document). The first and second D2 come from kitchen and table, respectively, then D1 and D3 from apple, and finally D1 from dessert. Notice that if we do this, we've used the information from the embeddings to provide LDA with evidence about the co-occurrence of semantic property D1 (things you eat) with semantic property D2 (domestic locations). One could try building an LDA topic model using these transformed representations of documents, and then explore what comes out. For example, do the topics that you get, as characterized by their top "words" (=properties), capture any interesting generalities? Are they capturing something that word-based LDA isn't? Can they capture equally good generalizations with fewer documents?