NLP Analysis in Python using Modal Verbs

Modal verbs are auxiliary verbs which indicate semantic information about an action, i.e. likelihood (will, should) , permission (could, may), obligation (shall/must). One interesting concept to explore is whether the presence of these verbs varies over different types of text, and whether that means anything.

“Natural Language Processing with Python” (read my review) has an example of how to start this process, comparing verb frequencies across various genres of text using the Brown corpus, a well-known collection of texts assembled in the 60’s for language research.

I extended the example to include an additional corpus of court cases, and extra helper verbs- This includes the contents of ~15,000 legal documents.

We first define a function to retrieve genres of literature, and a second to retrieve words from the genre. For the legal documents, I am reading from an index I previously built of n-grams (i.e. word/phrase counts).

import nltk
import os

def get_genres():
yield 'legal'
for genre in brown.categories():
yield genre

modals = ['can', 'could', 'may', 'might', 'must', 'will', 'would', 'should']

def get_words(genre):
  if (genre == 'legal'):
    grams = open('1gram', 'rU')
    for line in grams:
      vals = line.split(' ')
      word = vals[0]
      count = int(vals[1])
      if (word in modals):
        for index in range(0, count):
          yield word
        else:
          yield word
    grams.close()
  else:
    for word in brown.words(categories=genre):
      yield word

The Natural Language Toolkit provides a class for tracking frequencies of “experiment” results – here we track the use of different verb tenses.

cfd = nltk.ConditionalFreqDist(
  (genre, word)
  for genre in get_genres()
  for word in get_words(genre)
)

genres = [g for g in get_genres()]
cfd.tabulate(conditions=genres, samples=modals)

cfd.tabulate(conditions=genres, samples=modals)

The tabulate method is provided by NTLK, and makes a nicely formatted chart (in a command line it makes everything line up neatly)

	can	could	may	might	must	will	would	should
legal	13059	7849	26968	1762	15974	20757	19931	13916
adventure	46	151	5	58	27	50	191	15
belles_lettres	246	213	207	113	170	236	392	102
editorial	121	56	74	39	53	233	180	88
fiction	37	166	8	44	55	52	287	35
government	117	38	153	13	102	244	120	112
hobbies	268	58	131	22	83	264	78	73
humor	16	30	8	8	9	13	56	7
learned	365	159	324	128	202	340	319	171
lore	170	141	165	49	96	175	186	76
mystery	42	141	13	57	30	20	186	29
news	93	86	66	38	50	389	244	59
religion	82	59	78	12	54	71	68	45
reviews	45	40	45	26	19	58	47	18
romance	74	193	11	51	45	43	244	32
science_fiction	16	49	4	12	8	16	79	3

Looking at these numbers, it is clear that we need to add a concept of normalization. My added corpus has a lot more tokens than the Brown corpus, which makes it hard to compare across.

The frequency distribution class exists to count things, and I didn’t see a good way to normalize the rows. I re-wrote the tabulate function to do this – it simply finds the max for each row, divides by that, and multiplies by 100.

def tabulate(cfd, conditions, samples):
  max_len = max(len(w) for w in conditions)
  sys.stdout.write(" " * (max_len + 1))
  for c in samples:
    sys.stdout.write("%-s\t" % c)
    sys.stdout.write("\n")
  for c in conditions:
    sys.stdout.write(" " * (max_len - len(c)))
    sys.stdout.write("%-s" % c)
    sys.stdout.write(" ")
    dist = cfd[c]
    norm = sum([dist[w] for w in modals])
    for s in samples:
      value = 100 * dist[s] / norm
      sys.stdout.write("%-d\t" % value)
      sys.stdout.write("\n")

tabulate(cfd, genres, modals)

This makes it easier to scan up and down the chart-

	can	could	may	might	must	will	would	should
legal	10	6	22	1	13	17	16	11
adventure	8	27	0	10	4	9	35	2
belles_lettres	14	12	12	6	10	14	23	6
editorial	14	6	8	4	6	27	21	10
fiction	5	24	1	6	8	7	41	5
government	13	4	17	1	11	27	13	12
hobbies	27	5	13	2	8	27	7	7
humor	10	20	5	5	6	8	38	4
learned	18	7	16	6	10	16	15	8
lore	16	13	15	4	9	16	17	7
mystery	8	27	2	11	5	3	35	5
news	9	8	6	3	4	37	23	5
religion	17	12	16	2	11	15	14	9
reviews	15	13	15	8	6	19	15	6
romance	10	27	1	7	6	6	35	4
science_fiction	8	26	2	6	4	8	42	1

One thing this makes clear is most genres have numerous references to ‘would’ and few have ‘should’.

It might be nice to see these on a scale of 1-10 – seeing the columns of numbers communicates something in the lengths.

def tabulate(cfd, conditions, samples):
  max_len = max(len(w) for w in conditions)
  sys.stdout.write(" " * (max_len + 1))
  for c in samples:
    sys.stdout.write("%-s\t" % c)
    sys.stdout.write("\n")
    for c in conditions:
      sys.stdout.write(" " * (max_len - len(c)))
      sys.stdout.write("%-s" % c)
      sys.stdout.write(" ")
      dist = cfd[c]
      norm = sum([dist[w] for w in modals])
  for s in samples:
    value = 10 * float(dist[s]) / norm
    sys.stdout.write("%.1f\t" % value)
    sys.stdout.write("\n")

tabulate(cfd, genres, modals)

	can	could	may	might	must	will	would	should
legal	1.1	0.7	2.2	0.1	1.3	1.7	1.7	1.2
adventure	0.8	2.8	0.1	1.1	0.5	0.9	3.5	0.3
belles_lettres	1.5	1.3	1.2	0.7	1.0	1.4	2.3	0.6
editorial	1.4	0.7	0.9	0.5	0.6	2.8	2.1	1.0
fiction	0.5	2.4	0.1	0.6	0.8	0.8	4.2	0.5
government	1.3	0.4	1.7	0.1	1.1	2.7	1.3	1.2
hobbies	2.7	0.6	1.3	0.2	0.8	2.7	0.8	0.7
humor	1.1	2.0	0.5	0.5	0.6	0.9	3.8	0.5
learned	1.8	0.8	1.6	0.6	1.0	1.7	1.6	0.9
lore	1.6	1.3	1.6	0.5	0.9	1.7	1.8	0.7
mystery	0.8	2.7	0.3	1.1	0.6	0.4	3.6	0.6
news	0.9	0.8	0.6	0.4	0.5	3.8	2.4	0.6
religion	1.7	1.3	1.7	0.3	1.2	1.5	1.4	1.0
reviews	1.5	1.3	1.5	0.9	0.6	1.9	1.6	0.6
romance	1.1	2.8	0.2	0.7	0.6	0.6	3.5	0.5
science_fiction	0.9	2.6	0.2	0.6	0.4	0.9	4.2	0.2

It would be nice to see how similar these genres are – we can compute that by imagining the counts of modals as describing vectors. The angle between vectors approximates “similarity”. The nice thing about this measure is that it removes other words (words which may only exist in one text – some of this will be due to how well the data is cleaned, which does not reflect on the genre of literature).

import math

def distance(cfd, conditions, samples, base):
  base_cond = cfd[base]
  base_vector = [base_cond[w] for w in samples]
  base_length = math.sqrt(sum(a * a for a in base_vector))
  for c in conditions:
    cond = cfd[c]
    cond_vector = [cond[w] for w in samples]
    dotp = sum(a * b for (a,b) in zip(base_vector, cond_vector))
    cond_length = math.sqrt(sum(a * a for a in cond_vector))
    angle = math.acos(dotp / (cond_length * base_length))
    percent = (math.pi / 2 - angle) / (math.pi / 2) * 100
    print "%-s similarity to %-s: %-.1f" % (c, base, percent)

The result are interesting – the genres showing closes to legal in this case are government and religion.

As an interesting side-note, belles_lettres means “fine writing”, i.e. poems, drama, fiction.

legal similarity to legal: 100.0
adventure similarity to legal: 41.6
belles_lettres similarity to legal: 72.4
editorial similarity to legal: 68.8
fiction similarity to legal: 42.9
government similarity to legal: 80.6
hobbies similarity to legal: 63.5
humor similarity to legal: 50.1
learned similarity to legal: 80.6
lore similarity to legal: 78.6
mystery similarity to legal: 41.3
news similarity to legal: 58.1
religion similarity to legal: 81.2
reviews similarity to legal: 73.5
romance similarity to legal: 42.9
science_fiction similarity to legal: 41.8

Some genres appear similar to legal documents – it is possible, however, that some verbs are not independent. For instance, you might see “may” and “might” with equal similarity. One way to test this might be to flip what we track for distance (make a vector for each modal, rather than genre)

The following code tracks the distance between each modal and the mean, using the different genres as dimensions. Since each of them contributes to the mean somewhat, there is guaranteed to be some similarity, but note that some are closer than others. Note also that these have to be normalized, like the last example, or the answer will be defined by the ‘legal’ genre.

def distance(cfd, conditions, samples):
  base_vector = [0.0 for w in conditions]
  norm = {}
  for c_i in range(0, len(conditions)):
    cond_name = conditions[c_i]
    cond = cfd[cond_name]
    norm[cond_name] = float(sum(cond[s] for s in samples))
    for s in samples:
      base_vector[c_i] = base_vector[c_i] + float(cond[s]) / norm[cond_name]
      base_length = math.sqrt(sum(a * a for a in base_vector))
  for s in samples: # compute each vector - which, might, etc
    sample_vector = []
    for c in conditions: # find condition for each vector
      sample_vector.append(cfd[c][s] / norm[c])
      dotp = sum(a * b for (a,b) in zip(base_vector, sample_vector))
      sample_length = math.sqrt(sum(a * a for a in sample_vector))
      angle = math.acos(dotp / (sample_length * base_length))
      percent = (math.pi / 2 - angle) / (math.pi / 2) * 100
      print "%-s similarity to mean: %-.1f" % (s, percent)

distance(cfd, genres, modals)

What I’d infer from this is that the least helpful verb for distinguishing genres is “must,” and the most helpful is “may.”

can similarity to mean: 76.0
could similarity to mean: 67.6
may similarity to mean: 61.5
might similarity to mean: 70.0
must similarity to mean: 79.7
will similarity to mean: 67.7
would similarity to mean: 73.6
should similarity to mean: 74.2