Assignment 2


In this assignment, you will get some hands-on experience using association scores to identify interesting collocations in a corpus.

The corpus

In this assignment, we'll work with the congressional speech corpus, created by Matt Thomas, Bo Pang, and Lillian Lee. It contains speeches made by politicians in the U.S. House of Representatives during debates over legislation. Although the work in this assignment is relatively straightforward from an NLP point of view, it's pretty close to what real political scientists are doing in text analysis; for example, see Monroe et al., Fightin’ Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict, Political Analysis, 2008 16(4):372-403.

You should begin by downloading the convote dataset v1.1 (9.8 Mb, tar.gz format). There are several different versions of the data in this archive; we are concerned only with the data_stage_three directory. For those with limited disk space, a way to extract just the directory of interest is: gunzip < convote_v1.1.tar.gz | tar xvf - convote_v1.1/data_stage_three

Each file in the data_stage_three directory contains a speech by a legislator. The filenames contain information about each speech. In the filename template, ###_@@@@@@_%%%%$$$_PMV:

The task

The idea here is to use what we've learned in class (and what you've read about in M&S Chapter 5) to answer some interesting questions about this corpus.
  1. What are some strong bigram collocations in this corpus, according to frequency plus at least two measures of association? (Measures you might use could include, for example, log-likelihood value, chisquare test, mutual information...)

  2. Same question, but limit your attention to speeches by Democrats.

  3. Same question, but limit your attention to speeches by Republicans.

  4. Discuss the advantages and limitations of the measures you used, illustrating using the sorted lists of collocations you obtained.

  5. Are there any interesting differences between what you found for Democrats and what you found for Republicans?

    Here's an example. When I used the Democratic speeches as a corpus, and sorted all non-stopword bigrams by log-likelihood value, and did the same separately for Republicans, the phrase middle class ranked 16th for Democrats and 115th for Republicans; conversely, the phrase law enforcement ranked 28th for Republicans and 141st for Democrats. One could argue that this empirical observation is consistent with at least some characterizations of the priorities of the two political parties -- e.g. see discussions of the role that the phrase middle class had during the 2008 presidential debates, like this one.

    Even if you're not particularly familiar with American politics, identify similar contrasts and offer your thoughts on why they might or might not be meaningful.

  6. Finding some interesting contrasts by inspection is nice, but why should I trust your intuitions? For several examples of differences between Democrats and Republicans, make a statistical argument that the difference you're pointing out is a genuine difference supported by the data.

    To continue my example, having discovered that middle class might be an interesting phrase, I could use a chi-square test to show that the difference in the frequency of middle class among Democrats and Republicans is statistically significant. (Note that doing a chi-square test in this way, looking at a contingency table of {Democrat,Republican} x {middle_class, NOT middle_class}, is different from using chi-square to find an association between the word middle and the word class. Both cases involve a 2x2 contingency table, i.e. looking for evidence of an association between some binary variable A and a second variable B, but one case is looking for a lexical association between two words, and the other is looking for an association between a bigram and the party of the speaker using that bigram.)

    For full credit, be very explicit about the statistical hypothesis testing you're doing. For example, an explicit argument would (at least) explicitly describe null hypothesis H0, identify the test statistic, identify your choice for alpha (the cutoff for statistical significance), give the p-value, and present an argument for why we should draw the conclusions you want us to draw. (It's fine to use some of the same language I used in class.)

    You are welcome to use an on-line calculator or off-the-shelf code to do the statistical calculation if you'd like. Though you'll learn more if you do the calculation by hand at least once. :-) If you're thinking of using the chi-square test, note that there are problems using it when counts are small (see Problems in the chi-square test Wikipedia page). Depending on the counts, the likelihood ratio test (G-test) may be a better choice.

Extra credit (10 points)

Ways to go about it

Conveniently, the corpus is already tokenized and lowercased for you, and the filename conventions make it very easy to identify particular subsets of the corpus.

In terms of implementation,

You are welcome to work together. If you do so, turn in separate assignments (since the discussion parts will be different), even if you're turning in the same code, but identify who you worked with.

What to turn in

As specified by the TAs.

Again, I strongly encourage you to talk to each other on the class mailing list. You guys are your own best resource.