How do I analyze word use in a document or book?

I've got a bet with my husband and you need to help us settle it. I was reading Jane Austen's "Pride and Prejudice" and he said "I bet you $20 that "pride" shows up more than "prejudice" in that book." I think he's wrong, but have no idea how to figure it out. Can you help?

Dave's Answer:

Holy cow, that's really what you two bet on? Word frequency in classic literature? Wow. I mean, wow.

Ah, well, be that as it may, you're in luck because the book you're talking about appears as part of the brilliant Project Gutenberg library of out-of-copyright literature, so we can download a digital copy of the book for analysis. In fact, you can too, just click on download "Pride and Prejudice" and you'll be looking at the text of the book in its entirety.

Analyzing it is a bit more tricky and I'll do the heavy lifting for you because it's darn hard to do on a Windows computer but quite straightforward if you can pop up a command window a la Linux, which I can do with the Terminal.app program included with Mac OS X (or any Linux system, of course).

Armed with our command line, the first thing I need to do is break down the document into separate words and convert upper case to lower case. This is surprisingly easy to do on the command line:

cat pride-prejudice.txt | tr '[:upper:]' '[:lower:]' | tr ' ' '\ > '

The "cat" command displays the contents of a file, then "tr" translates uppercase to lowercase, then, the second time it's called, converts all spaces to end-of-line returns. What the above command would do if you actually typed it in is to display the entire book on your screen, one word at a time. ZOOOM!

To do something with that output, lets feed it to the oddly-named "grep" command to extract just the word "pride" from the book source:

cat pride-prejudice.txt | tr '[:upper:]' '[:lower:]' | tr ' ' '\ > ' | grep pride

You could now count the matches by hand, but since we're on the command line, we might as well make our lives easier and use "wc" to count the number of lines output. Put it all together and it looks like this:

$ cat pride-prejudice.txt | tr '[:upper:]' '[:lower:]' | tr ' ' '\ ' | grep pride | wc -l 49

So there you go. The word "pride" appears in the book Pride and Prejudice 49 times. How about "prejudice"?

Okay, there's the answer to your bet. "Pride" occurs 49 times in the book, while "prejudice" only appears 9 times. That means, sorry to report, that he's right and you're wrong.

Just for fun, since we have the book accessible, let's pop over to my readability site and find out some more stats on Jane's book...

readability grades:
        Kincaid: 10.6
        ARI: 12.2
        Coleman-Liau: 10.0
        Flesch Index: 65.4
        Fog Index: 13.9
        Lix: 45.3 = school year 8
        SMOG-Grading: 11.3
sentence info:
        547423 characters
        124991 words, average length 4.38 characters = 1.36 syllables
        4805 sentences, average length 26.0 words
        54% (2602) short sentences (at most 21 words)
        23% (1150) long sentences (at least 36 words)
        881 paragraphs, average length 5.5 sentences
        4% (200) questions
        66% (3173) passive sentences
        longest sent 179 wds at sent 2299; shortest sent 1 wds at sent 107
word usage:
        verb types:
        to be (5550) auxiliary (2913) 
        types as % of total:
        conjunctions 6(7985) pronouns 15(18711) prepositions 12(15203)
        nominalizations 2(2220)
sentence beginnings:
        pronoun (1961) interrogative pronoun (208) article (282)
        subordinating conjunction (194) conjunction (385) preposition (275)

Wow, way more than you probably wanted to know, but notice that it's rated as approximately 10th grade reading, with 4805 sentences that average 26 words per sentence.

Good luck on your future gambles with your husband too!

Categorized: Shell Script Programming (Article 8640, Written by Dave Taylor)
Tagged: english, grammar, jane austen, readability
Previous: Help! Someone Stole My Apple iPod / Sony PSP!
Next: How to check - and fix - your date and time on your Windows PC?

Subscribe!

Never miss another Q&A article! Click to subscribe:

Comments

Dave, this is a great example of the power of the command line, but mere mortals can achieve the same result without having to learn unix-fu.

Just open up the document in a word processor and find/replace "pride" and "prejudice" with other words. The find/replace dialog box in most word processing applications will give you the option to ignore case and will tell you how many instances have been changed.

Posted by: Glen Turpin at December 5, 2008 9:52 AM

Dave, I think you are easily duped into doing programming homework when it is presented to you as a "bet." :-)

I tried to follow along on my own command line and had some difficulty. On my MacOSX the character for carriage return in the tr command is '\r' instead of '\ > '. It took me several minutes of squinting at the man files to figure this out.

I am wondering why little things like this are different from one computer to the next.

Posted by: Dan at December 7, 2008 10:52 AM

Hi Dan. I bet you'll read this answer! :-)

It is true that each major release of Mac OS X has included a different version of NetBSD / Darwin / whatever-you-want-to-call-it, the underlying Linux operating system.

You need to be careful with what you type in too. Yes '\r' will work, but '\ >' won't work: you need a single character and you're showing two, one that's a protected space ("\ ") and one that's the right angle bracket. However, this will work:

tr 'x' '
'

where you literally open the quoted passage then press the Return key on your keyboard, then close the quote.

Posted by: Dave Taylor at December 9, 2008 4:06 PM

Hi Dave

I'm an applied linguist as well as a photographer.

If you want to find all the words in a text (usually pure txt) then you can use a condordancer. Press the button and you'll get a list of all the words and lots of other info.

Here's a chunk from Wikipedia. Lots of concordancers are free like AntConc

Hope this helps

Cheers John

Concordancers are also used in corpus linguistics to retrieve alphabetically or otherwise sorted lists of linguistic data from the corpus in question, which the corpus linguist then analyzes. Some concordancers used in corpus linguistics are AntConc (Freeware), ApSIC Xbench, WordSmith, MonoConc, GlossaNet, and CorpusEye.

Posted by: John Rocha at December 19, 2008 12:59 AM

Hi Dave,

Here's a slight optimization. Instead of running the commands twice, you can use the grep -e option and the uniq command and get both results in a one-liner.

$ cat pride-prejudice.txt | tr '[:upper:]' '[:lower:]' | tr ' ' '\n' | grep -e pride -e prejudice | uniq -c

Posted by: Howard Hong at October 30, 2009 3:40 PM

Oops sorry, that previous command should be

Posted by: Howard Hong at October 31, 2009 1:32 PM

Hi Dave,

Hope this will be an easy way.

$ cat pride | tr ' ' '\n' | grep -i -c pride

Posted by: Sujay Kumar at December 3, 2009 8:54 AM

Another way could be:
$ grep -o -i pride pride-prejudice.txt | wc -l

Also, I think it might be better to pass '-w' to grep too, just in case other words contain the string "pride".

Posted by: Antriksh Pany at March 5, 2010 8:00 PM

I have something to say, now that you mention it, but ...

I do have a lot to say, and questions of my own for that matter, but first I'd like to say thank you for all your efforts on this Web site by buying you a cup of coffee!

I do have a comment, now that you mention it!