First, a couple of notes to follow up on things I mentioned in class, as promised, although they're not needed for this assignment.
For the mathematically inclined and/or those who liked the idea of deriving an information measure from a set of properties such a measure should have, here are several derivations of the definition of entropy based on axioms about such properties. The one I mentioned in class is from A.I. Khinchin, Mathematical Foundations of Information Theory, Dover, New York, 1957.
Also, in case you're curious, here's the pointer to an algorithm for creating optimal codes using Huffman coding (see also this interesting discussion by Huffman's nephew).
You can do this for the simple case of just bigram probabilities (i.e. context C=c means "the previous word is c"), although it would certainly also be interesting to see what things look like for higher-order ngrams. If you're doing that, I recommend using an existing language modeling module (e.g. in NLTK) or toolkit (e.g. KenLM and SRILM). As usual, just make sure to clearly identify any software that you didn't write yourself in your PDF writeup.
You should do your experiment for English and at least one other language. As one possible source of data to work with, see the NLTK corpora. The Brown corpus would be a smallish but decent source for English (just remove POS tags and lowercase everything; it's already tokenized); you could also augment that with other English corpora, although be careful to make sure that your sources don't overlap, which would throw off your counts. For other languages, one source would be the NLTK samples from the Europarl corpora; if you were feeling more ambitious you could look into the full Europarl corpora, which I believe are quite a bit larger. Or feel free to identify other sources of text. The corpus processing (acquisition, cleanup, etc.) are not the point of this assignment, so I encourage people to share pointers, cleanup code, and even corpus data using Piazza. (Recall that constructive participation on Piazza contributes to your class participation grade.)
You are not required to actually succeed in replicating the research results in the paper -- that would take a lot of work, since they used fairly sizeable corpora that you probably don't have easy access to. In fact, you're not even required to compute correlations (though that would be nice); we'll be happy if you produce charts that look roughly like their Figure 2, no error bars required (though that would be nice also).
What you do need to do involves (a) briefly summarizing what the study is about in your own words to show that you understand it, (b) clearly describing for us what you did in your own experiment (along with any simplifying choices you made and why/how you made them), and (c) clearly showing and telling us what you you found. If your results are very different from what you would have expected, then (d) suggest some explanations why (and ideally support your explanation with data). The more thoughtful and clearly written your writeup the better.
Remember that the goal is not to "get it right", but to show you understand the ideas and that you're taking a thoughtful approach. The writeup matters.