Song Lyrics: Modality Score

As we considered the lyrical content of The Beatles' songs, we started to wonder: Which Beatles songs are most typical, most quintessentially "Beatles", in their vocabulary? Calculating this turned out to be possible, but very tricky, and it required us to break the problem into several smaller steps, generating three separate XQuery document and two ancillary XML documents along the way. Here are our steps.

  1. First, we needed to write an XQuery to string together the lyrics from all the songs, and then to "tokenize" the text (divide the lyrics into individual words). It would then count the frequency of each word across the entire corpus of Beatles lyrics. The first version of this created this page, for visual display; the second version instead created a new XML file with the same data. This assigned each word an "overall frequency score" indicating how common it was across the whole discography.
  2. Second, we needed to run a similar XQuery to count the frequency of each word on a song-by-song basis. This generated another new XML file in which each song had a list of the words used in it, and the number of times each one was used.
  3. Third, bringing these together, was a new XQuery to generate the HTML page you are reading right now, and especially the table below. On a song-by-song basis, it cross-references the two new XML files. As it loops over each song (excluding instrumentals, which have no lyrics), it looks at the frequency of each word in that song, and then multiplies that by the frequency of that word's use in the whole discography. We then took the average of those multiples across the song.

What does this mean in practice?

The word "you" is used 2073 times across the discography. It appears 11 times in the song "All I've Got to Do". That means that, in that song, we multiply 2073 times 11 to get 22803. This high number means that the song has used a very common word a large number of times. This contributes to a higher score for the song, meaning that is closer to the "modal" -- that is, the most typical -- Beatles song, as measured by word use frequency.

At the same time, looking at the same song, the word "whisper" appears only once, and "whisper" is an uncommon word in the discography overall, appearing just 6 times. Its appearance in the song "All I've Got to Do" is therefore only worth 6 points.

When we average these calculations for all the words in the song, we arrive at a final score for that song, which can be compared to all the other songs. Songs, then, that use more words like "you", and fewer words like "whisper", will have a higher modality score.

Graphs and Patterns

Further down on this page, you will see a table ranking all 251 songs with lyrics. To get a visual summary, though, you will want to look at some graphs produced with these data.

First, take a look at a page showing how cover songs (those written by others, but recorded by The Beatles) stack up in terms of how quintessentially "Beatles" their word-use patterns are. (See here for the XQuery that generated that page.)

Then check out this page to see how song modality scores relate to song popularity, as measured by which songs made the Billboard Top 100 Charts in the United States. (See here for the XQuery that generated that page.)

And finally, looking at trends over time, yet another XQuery created an HTML page to compare language use over time.

And the winner is...

...a song you may never have heard of: "I Want You (She's So Heavy)", with an exceedingly high score of nearly 8200. (For context, only a handful of songs have scores exceeding 2000, and only four other songs exceed 3000.) What could possibly account for a song being both the most linguistically typical of all Beatles songs, and yet also an obscure deep cut from the Abbey Road album?

The answer is simple: there are very few different words (14) used in this song; almost all of them are very common words; and they are repeated over and over. (Lennon said that he had "made a deliberate choice to keep the lyrics simple and concise.") If you wanted to write a song to rack up the maximum score, this is exactly how you would do it.

In a sense, then, "I Want You (She's So Heavy)" really is an outlier that "wins" top score by a kind of linguistic accident. It is interesting that several other deep cuts from Abbey Road scored near or at the opposite end of the spectrum, such as "Sun King" (the very lowest score, at 168) and "Mean Mr. Mustard" (343). This simple calculation did not take into account the many other factors that a proper linguistic analysis would include: it is not stylometry. Nonetheless, while this method does not really tell us what a truly typical Beatles song lyric looks like, it certainly does help us to identify outliers. "Sun King" scores so low in large measure because it includes a more or less random collection of Spanish and other Romance-language words, using most of them only once; and the second-lowest scoring song, "Komm gib mir deine Hand" (score 189), was recorded in German. Certainly these are thus true linguistic outliers.

Another way to calculate what might be a linguistically typical Beatles song, then, would be to look instead for a medium score. Perhaps a more typical Beatles song lyric contains a certain proportion of common words tapering off to a certain number of unique or unusual words. The average score across all songs is 1144. The songs closest to that are "It's All Too Much" (1148) and "Blackbird"(1138).

Finally, one could look for the median. There are 251 non-instrumental songs in our collection, which would make the song in position #126, "Don't Ever Change" (score 981), the most middle-of-the-road Beatles song lyric.

Song Modality Score Table

Rank Year Song Title Modality Score
1 1969 I Want You (She's So Heavy) 8197
2 1967 Hello, GoodBye 6142
3 1963 I Wanna Be Your Man 4890
4 1967 You Know My Name (Look Up the Number) 4085
5 1969 The End 3413
6 1968 Why Don't We Do It in the Road? 2886
7 1962 P.S. I Love You 2777
8 1962 Love Me Do 2776
9 1969 Oh! Darling 2634
10 1962 Like Dreamers Do 2495
11 1963 From Me to You 2491
12 1963 She Loves You 2308
13 1967 All You Need Is Love 2288
14 1967 Magical Mystery Tour 2226
15 1969 Dig a Pony 2157
16 1963 I Want to Hold Your Hand 2057
17 1958 In Spite of All the Danger 2018
18 1964 Leave My Kitten Alone 1977
19 1964 You Know What to Do 1970
20 1965 You Like Me Too Much 1940
21 1965 The Night Before 1933
22 1964 I Should Have Known Better 1914
23 1964 You Can't Do That 1867
24 1963 Lonesome Tears in My Eyes 1859
25 1970 I Me Mine 1854
26 1962 Hello Little Girl 1835
27 1968 Helter Skelter 1835
28 1963 A Shot of Rhythm and Blues 1819
29 1964 A Hard Day's Night 1816
30 1964 I Feel Fine 1787
31 1964 Sie liebt dich 1755
32 1962 Love of the Loved 1720
33 1965 The Word 1653
34 1963 Bad to Me 1652
35 1963 I Just Don't Understand 1616
36 1963 All I've Got to Do 1606
37 1963 I'll Get You 1598
38 1964 Tell Me Why 1585
39 1963 Thank You Girl 1580
40 1964 One and One Is Two 1575
41 1965 Michelle 1573
42 1963 Anna (Go to Him) 1552
43 1968 Revolution 1 1543
44 1968 Everybody's Got Something to Hide Except Me and My Monkey 1537
45 1965 You Won't See Me 1523
46 1966 She Said She Said 1520
47 1965 You're Going to Lose That Girl 1516
48 1968 Child of Nature 1514
49 1963 Do You Want to Know a Secret? 1513
50 1965 That Means a Lot 1497
51 1965 Tell Me What You See 1473
52 1969 Because I Know You Love Me So 1460
53 1962 Please Please Me 1457
54 1963 Ask Me Why 1451
55 1969 For You Blue 1451
56 1969 Come and Get It 1450
57 1961 Ain't She Sweet 1440
58 1965 Ticket to Ride 1429
59 1965 I'm Down 1418
60 1968 Birthday 1397
61 1964 She's a Woman 1376
62 1968 Long, Long, Long 1372
63 1965 Run for Your Life 1369
64 1964 I'll Be Back 1361
65 1965 Wait 1357
66 1967 Your Mother Should Know 1351
67 1964 Mr. Moonlight 1342
68 1965 Drive My Car 1326
69 1964 Baby's in Black 1321
70 1968 What's The New Mary Jane 1320
71 1969 Don't Let Me Down 1312
72 1965 If You've Got Trouble 1307
73 1963 Tip of My Tongue 1287
74 1967 With a Little Help from My Friends 1280
75 1965 I'm Looking Through You 1267
76 1964 What You're Doing 1258
77 1968 Revolution 1245
78 1964 I'm Happy Just to Dance with You 1243
79 1963 Hold Me Tight 1242
80 1969 One After 909 1239
81 1964 Eight Days a Week 1219
82 1965 What Goes On 1219
83 1969 Carry That Weight 1218
84 1963 I Got a Woman 1211
85 1969 Something 1210
86 1965 Act Naturally 1203
87 1964 I Call Your Name 1203
88 1964 Can't Buy Me Love 1191
89 1965 Help! 1179
90 1964 Everybody's Trying to Be My Baby 1178
91 1969 Here Comes the Sun 1165
92 1965 It's Only Love 1160
93 1964 Things We Said Today 1152
94 1963 Devil in Her Heart 1149
95 1963 I'm In Love 1149
96 1967 It's All Too Much 1148
97 1968 Blackbird 1138
98 1968 Don't Pass Me By 1138
99 1964 No Reply 1128
100 1968 Back in the U.S.S.R. 1126
101 1963 It Won't Be Long 1123
102 1963 Keep Your Hands Off My Baby 1122
103 1965 Think for Yourself 1119
104 1967 Getting Better 1113
105 1964 I'll Follow the Sun 1110
106 1968 Circles 1107
107 1963 There's a Place 1102
108 1968 Sexy Sadie 1097
109 1968 I'm So Tired 1093
110 1963 Baby It's You 1072
111 1967 Baby, You're a Rich Man 1054
112 1966 Strawberry Fields Forever 1048
113 1965 Yes It Is 1034
114 1968 I Will 1032
115 1966 And Your Bird Can Sing 1027
116 1964 I'm a Loser 1027
117 1964 I Don't Want to Spoil the Party 1026
118 1964 And I Love Her 1012
119 1964 Every Little Thing 1008
120 1960 You'll Be Mine 1008
121 1966 Rain 1005
122 1965 Day Tripper 1000
123 1965 I Need You 996
124 1964 If I Fell 987
125 1968 While My Guitar Gently Weeps 983
126 1963 Don't Ever Change 981
127 1969 Taking a Trip to Carolina 969
128 1966 I Want to Tell You 948
129 1965 We Can Work It Out 937
130 1968 Wild Honey Pie 916
131 1963 Chains 908
132 1968 Martha My Dear 906
133 1963 I Saw Her Standing There 904
134 1968 Ob-La-Di, Ob-La-Da 895
135 1968 The Inner Light 887
136 1968 Hey Jude 886
137 1980 Real Love 882
138 1968 Step Inside Love/Los Paranoias 882
139 1963 I'll Be on My Way 881
140 1964 When I Get Home 881
141 1969 Get Back 873
142 1966 Tomorrow Never Knows 869
143 1963 Carol 867
144 1963 I Got to Find My Baby 864
145 1964 Any Time at All 861
146 1966 For No One 860
147 1969 Golden Slumbers 859
148 1963 All My Loving 856
149 1969 Old Brown Shoe 855
150 1969 The Ballad of John and Yoko 849
151 1968 Hey Bulldog 848
152 1966 Taxman 847
153 1963 Lucille 836
154 1968 Not Guilty 836
155 1968 Yer Blues 834
156 1966 Paperback Writer 817
157 1963 From Us to You 814
158 1969 Let It Be 813
159 1968 Honey Pie 811
160 1969 Dig It 810
161 1968 Dear Prudence 806
162 1968 Cry Baby Cry 789
163 1969 Teddy Boy 789
164 1966 Here, There and Everywhere 788
165 1965 In My Life 780
166 1969 Octopus's Garden 774
167 1968 Glass Onion 772
168 1960 Hallelujah, I Love Her So 770
169 1969 Because 769
170 1964 I'll Cry Instead 764
171 1968 The Continuing Story of Bungalow Bill 763
172 1965 Girl 761
173 1965 Another Girl 746
174 1964 Matchbox 746
175 1963 Lend Me Your Comb 742
176 1965 Yesterday 742
177 1963 Crying, Waiting, Hoping 740
178 1969 The Long and Winding Road 736
179 1963 Little Child 733
180 1968 Julia 729
181 1968 Happiness Is a Warm Gun 719
182 1967 Blue Jay Way 718
183 1967 I Am the Walrus 718
184 1967 Fixing a Hole 716
185 1967 Good Morning Good Morning 716
186 1968 Good Night 713
187 1969 You Never Give Me Your Money 712
188 1968 Savoy Truffle 705
189 1969 Watching Rainbows 703
190 1969 Two of Us 699
191 1966 Yellow Submarine 697
192 1977 Free as a Bird 688
193 1964 I Forgot to Remember to Forget 683
194 1966 I'm Only Sleeping 679
195 1967 Christmas Time (Is Here Again) 671
196 1969 Her Majesty 670
197 1963 A Taste of Honey 667
198 1967 Lucy in the Sky with Diamonds 656
199 1967 Sgt. Pepper's Lonely Hearts Club Band 655
200 1966 Eleanor Rigby 646
201 1966 Penny Lane 644
202 1967 All Together Now 643
203 1963 Glad All Over 635
204 1966 When I'm Sixty-Four 627
205 1963 This Boy 624
206 1965 Norwegian Wood (This Bird Has Flown) 621
207 1965 You've Got to Hide Your Love Away 620
208 1963 Don't Bother Me 614
209 1967 Within You Without You 608
210 1969 Polythene Pam 606
211 1965 If I Needed Someone 602
212 1963 Misery 598
213 1966 Doctor Robert 589
214 1965 Nowhere Man 587
215 1967 A Day in the Life 574
216 1969 I've Got a Feeling 573
217 1967 The Fool on the Hill 568
218 1963 Clarabella 567
219 1963 Boys 564
220 1962 I Lost My Little Girl 559
221 1966 Love You To 559
222 1969 She Came in Through the Bathroom Window 556
223 1966 Good Day Sunshine 552
224 1968 Junk 542
225 1963 Not a Second Time 536
226 1968 Sour Milk Sea 535
227 1969 Come Together 524
228 1965 I've Just Seen a Face 516
229 1964 Johnny B. Goode 516
230 1968 Rocky Raccoon 514
231 1967 Lovely Rita 512
232 1968 Heather 498
233 1967 Only a Northern Song 495
234 1969 All Things Must Pass 465
235 1968 Piggies 459
236 1968 Lady Madonna 448
237 1965 Bad Boy 433
238 1968 Across the Universe 422
239 1968 Revolution 9 420
240 1967 Being for the Benefit of Mr. Kite! 399
241 1969 Mailman, Bring Me No More Blues 384
242 1969 Maxwell's Silver Hammer 369
243 1967 Sgt. Pepper's Lonely Hearts Club Band (Reprise) 350
244 1964 Long Tall Sally 345
245 1969 Mean Mr. Mustard 343
246 1963 Moonlight Bay 338
247 1967 She's Leaving Home 323
248 1969 Maggie Mae 281
249 1968 Mother Nature's Son 281
250 1964 Komm gib mir deine Hand 189
251 1969 Sun King 168