In this assignment, we will get some hands-on experience using association scores for corpus analysis. Although the work in this assignment is relatively straightforward from an NLP point of view, it's pretty close to what some real political scientists have done recently in text analysis; for example, see Monroe et al., Fightin’ Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict, Political Analysis, 2008 16(4):372-403, and also this analysis of State of the Union addresses and references therein.
Congressional speeches. The congressional speech corpus was created by Matt Thomas, Bo Pang, and Lillian Lee. It contains speeches made by politicians in the U.S. House of Representatives during debates over legislation and has been used in a variety of research papers. Conveniently, the corpus is already tokenized and lowercased for you, and the filename conventions make it very easy to identify particular subsets of the corpus. When you download the convote dataset v1.1 (9.8 Mb, tar.gz format) you will see that there are several different versions of the data in this archive; we are concerned only with the data_stage_three directory. For those with limited disk space, a way to extract just the directory of interest is:
gunzip < convote_v1.1.tar.gz | tar xvf - convote_v1.1/data_stage_three
Each file in the data_stage_three directory contains a speech by a legislator. The filenames contain information about each speech. In the filename template,
###_@@@@@@_%%%%$$$_PMV:
State of the Union. Another interesting dataset is the set of State of the Union(SOTU) addresses of U.S. presidents. (Recall that in a Jan 15, 2016 blog post, Michael Heilman at Civis Analytics presented a simple analysis of State of the Union addresses primarily based on word frequencies. That posting contains a pointer to some other text-analytics research on the topic.) For our purposes here, let's restrict ourselves to the post-Reagan presidencies, i.e. Presidents Bush 1 (Republican), Clinton (Democrat), Bush 2 (Republican), and Obama (Democrat).
Here's an example. When I used the Democratic congressional speeches as a corpus, and sorted all non-stopword bigrams by log-likelihood value, and did the same separately for Republicans, the phrase middle class ranked 16th for Democrats and 115th for Republicans; conversely, the phrase law enforcement ranked 28th for Republicans and 141st for Democrats. One could argue that this empirical observation is consistent with at least some characterizations of the priorities of the two political parties -- e.g. see discussions of the role that the phrase middle class had during the 2008 presidential debates, like this one.
Even if you're not particularly familiar with American politics, identify similar contrasts and offer your thoughts on why they might or might not be meaningful.
For example, continuing the example above: having discovered that middle class might be an interesting phrase, I could use a chi-square test to show that the difference in the frequency of middle class among Democrats and Republicans is statistically significant. (Note that doing a chi-square test in this way, looking at a contingency table of {Democrat,Republican} x {middle_class, NOT middle_class}, is different from using chi-square to find an association between the word middle and the word class. Both cases involve a 2x2 contingency table, i.e. looking for evidence of an association between some binary variable A and a second variable B, but one case is looking for a lexical association between two words, and the other is looking for an association between a bigram and the party of the speaker using that bigram.)
For full credit, be very explicit about the statistical hypothesis testing you're doing. For example, an explicit argument would (at least) explicitly describe null hypothesis H0, identify the test statistic, identify your choice for alpha (the cutoff for statistical significance), give the p-value, and present an argument for why we should draw the conclusions you want us to draw. (It's fine to use some of the same language I used in class.)
You are welcome to use an on-line calculator or off-the-shelf code to do the statistical calculation if you'd like. Though you'll learn more if you do the calculation by hand at least once. :-) If you're thinking of using the chi-square test, note that there are problems using it when counts are small (see Problems in the chi-square test Wikipedia page). Depending on the counts, the likelihood ratio test may be a better choice. The classic formulation of that test for computational linguists is by Dunning (1993), but see also Moore (2004) for some more intuitive formulations (Figure 2) and an argument for using Fisher's Exact Test (typically viewed as intractable for large samples but Moore gives a number of optimizations that make it practical).
In terms of implementation,
You are welcome to work together. If you do so, turn in separate assignments (since the discussion parts will be different), even if you're turning in the same code, but identify who you worked with and who contributed what.
Recall from the course home page that the address for turning in assignments is ling773.sp2016@gmail.com, with "Assignment N" at the start of the subject line. Turn in assignments in PDF format, with assignment_N_firstname.lastname.zip (or .tar.gz) as the file name. (In this case N=2.) We will deduct 5% for not following these instructions. If you have any questions let us know.
Include:
Again, I strongly encourage you to talk to each other in person and/or on the class discussion board.