from lec_utils import *
Announcements 📣¶
- Homework 5 is due on Thursday. It includes a required Pre-Midterm Survey.
Homework 6 will be released later this week, but won't be due until after Fall Break. - The Midterm Exam is on Wednesday, October 9th from 7-9PM.
- Lectures 1-12 and Homeworks 1-6 are in scope.
- The lecture before the exam will be review, and the TAs will run a review session on Monday from 6-8PM in FXB 1109 too.
- You can bring one double-sided 8.5"x11" notes sheet that you handwrite yourself (no printing, no using an iPad, etc.).
- Work through old exam problems here.
- Looking for sources of data, or other supplemental resources? Look at our updated Resources page!
Aside: Spreadsheets¶
- We recorded a walkthrough video, linked here, on the spreadsheets part of Lecture 10 that we didn't get to cover last Thursday.
We won't have time to cover it now, either.
from IPython.display import IFrame
IFrame(src='https://www.loom.com/embed/eb06b185428542c391f21e55480a0d2d?sid=3891cb7f-a4c9-4a34-8211-0347a283d413',
width=400, height=300)
- The spreadsheet I created in the video can be found here.
- Spreadsheets won't be in an assignment or on an exam, but they will be useful at some point in your life, so now is as good a time as any to follow along and learn!
Let me know if you have any feedback or comments on it.
Agenda¶
Today's lecture will mostly be about regular expressions. Good resources:
- regex101.com, a helpful site to have open while writing regular expressions.
- Python
re
library documentation and how-to.
The "how-to" is great, read it! - regex "cheat sheet".
- These are all on the resources tab of the course website as well.
Motivation¶
email = '''
Thank you for buying our expensive product!
If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309.
If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you!
Due to high demand, please allow one-hundred (100) business days for a response.
'''
Who called? 📞¶
- Goal: Extract all phone numbers from a piece of text, assuming they are of the form
'(###) ###-####'
.
print(email)
Thank you for buying our expensive product! If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309. If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you! Due to high demand, please allow one-hundred (100) business days for a response.
- We can do this using the same string methods we've come to know and love.
- Strategy:
- Split by spaces.
- Check if there are any consecutive "words" where:
- the first "word" looks like an area code, like
'(678)'
. - the second "word" looks like the last 7 digits of a phone number, like
'999-8212'
.
- the first "word" looks like an area code, like
Checking formatting¶
- Let's first implement a function that takes in a string and returns whether it looks like an area code.
def is_possibly_area_code(s):
'''Does `s` look like (678)?'''
return (len(s) == 5 and
s.startswith('(') and
s.endswith(')') and
s[1:4].isnumeric())
is_possibly_area_code('(123)')
True
is_possibly_area_code('(99)')
False
- Let's also implement a function that takes in a string and returns whether it looks like the last 7 digits of a phone number.
def is_last_7_phone_number(s):
'''Does `s` look like 999-8212?'''
return len(s) == 8 and s[0:3].isnumeric() and s[3] == '-' and s[4:].isnumeric()
is_last_7_phone_number('999-8212')
True
is_last_7_phone_number('534 1100')
False
- Finally, let's split the entire text by spaces, and check whether there are any instances where
pieces[i]
looks like an area code andpieces[i+1]
looks like the last 7 digits of a phone number.
print(email)
Thank you for buying our expensive product! If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309. If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you! Due to high demand, please allow one-hundred (100) business days for a response.
# Removes punctuation from the end of each string.
pieces = [s.rstrip('.,?;"\'') for s in email.split()]
for i in range(len(pieces) - 1):
if is_possibly_area_code(pieces[i]):
if is_last_7_phone_number(pieces[i+1]):
print(pieces[i], pieces[i+1])
(800) 867-5309 (800) 123-4567
Is there a better way?¶
- This was an example of pattern matching.
- Pattern matching can be done with string methods, but there is often a better approach: regular expressions.
print(email)
Thank you for buying our expensive product! If you have a complaint, please send it to complaints@compuserve.com or call (800) 867-5309. If you are happy with your purchase, please call us at (800) 123-4567; we'd love to hear from you! Due to high demand, please allow one-hundred (100) business days for a response.
import re
re.findall(r'\(\d{3}\) \d{3}-\d{4}', email)
['(800) 867-5309', '(800) 123-4567']
🤯
Basic regular expressions¶
Regular expressions¶
- A regular expression, or regex for short, is a sequence of characters used to match patterns in strings.
- For example,
\(\d{3}\) \d{3}-\d{4}
describes a pattern that matches US phone numbers of the form'(XXX) XXX-XXXX'
.
- Think of regex as a "mini-language".
Formally, they are a grammar for describing a language.
- Pros ✅: They are very powerful and are widely used – virtually every programming language has a module for working with them.
- Cons ❌: They can be hard to read and have many different "dialects."
Writing regular expressions¶
- You will ultimately write most of your regular expressions in Python, using the
re
module. We will see how to do so shortly.
- However, a useful tool for designing regular expressions is regex101.com.
- We will use it heavily during lecture; you should have it open as we work through examples. If you're trying to revisit this lecture in the future, you'll likely want to watch the recording; just looking at the notebook won't give you enough context.
Literals¶
A literal is a character that has no special meaning.
Letters, numbers, and some symbols are all literals.
Some symbols, like
.
,*
,(
, and)
, are special characters.*Example*: The regex
hey
matches the string'hey'
. The regexhe.
also matches the string'hey'
.
Regex building blocks 🧱¶
The four main building blocks for all regexes are shown below.
table source, inspiration.
operation | order of op. | example | matches ✅ | does not match ❌ |
---|---|---|---|---|
concatenation | 3 | AABAAB |
'AABAAB' |
every other string |
or | 4 | AA|BAAB |
'AA' , 'BAAB' |
every other string |
closure (zero or more) |
2 | AB*A |
'AA' , 'ABBBBBBA' |
'AB' , 'ABABA' |
parentheses | 1 | A(A|B)AAB (AB)*A |
'AAAAB' , 'ABAAB' 'A' , 'ABABABABA' |
every other string'AA' , 'ABBA' |
Note that |
, (
, )
, and *
are special characters, not literals. They manipulate the characters around them.
*Example (or, parentheses)*:
- What does
EECS 280|398
match? - What does
EECS (280|398)
match?
*Example (closure, parentheses)*:
- What does
eecs*
match? - What does
(eecs)*
match?
Activity
Write a regular expression that matches 'billy'
, 'billlly'
, 'billlllly'
, etc.
- First, think about how to match strings with any even number of
'l'
s, including zero'l'
s (i.e.'biy'
). - Then, think about how to match only strings with a positive even number of
'l'
s.
✅ Click here to see the answer after you've tried it yourself at regex101.com.
bi(ll)*y
will match any even number of 'l'
s, including 0.
To match only a positive even number of 'l'
s, we'd need to first "fix into place" two 'l'
s, and then follow that up with zero or more pairs of 'l'
s. This specifies the regular expression bill(ll)*y
.
Activity
Write a regular expression that matches 'billy'
, 'billlly'
, 'biggy'
, 'biggggy'
, etc.
Specifically, it should match any string with a positive even number of 'l'
s in the middle, or a positive even number of 'g'
s in the middle.
✅ Click here to see the answer after you've tried it yourself at regex101.com.
Possible answers: bi(ll(ll)*|gg(gg)*)y
or bill(ll)*y|bigg(gg)*y
.
Note, bill(ll)*|gg(gg)*y
is not a valid answer! This is because "concatenation" comes before "or" in the order of operations. This regular expression would match strings that match bill(ll)*
, like 'billll'
, OR strings that match gg(gg)*y
, like 'ggy'
.
Intermediate regex¶
More regex syntax¶
operation | example | matches ✅ | does not match ❌ |
---|---|---|---|
wildcard | .U.U.U. |
'CUMULUS' 'JUGULUM' |
'SUCCUBUS' 'TUMULTUOUS' |
character class | [A-Za-z][a-z]* |
'word' 'Capitalized' |
'camelCase' '4illegal' |
at least one | bi(ll)+y |
'billy' 'billlllly' |
'biy' 'bily' |
between $i$ and $j$ occurrences | m[aeiou]{1,2}m |
'mem' 'maam' 'miem' |
'mm' 'mooom' 'meme' |
.
, [
, ]
, +
, {
, and }
are also special characters, in addition to |
, (
, )
, and *
.
*Example (character classes, at least one): [A-E]+
is just shortform for `(A|B|C|D|E)(A|B|C|D|E)`.
*Example (wildcard)*:
- What does
.
match? - What does
he.
match? - What does
...
match?
*Example (at least one, closure)*:
- What does
123+
match? - What does
123*
match?
*Example (number of occurrences)*: What does tri{3, 5}
match? Does it match 'triiiii'
?
*Example (character classes, number of occurrences)*:
What does [1-6a-f]{3}-[7-9E-S]{2}
match?
Activity
Write a regular expression that matches any lowercase string has a repeated vowel, such as 'noon'
, 'peel'
, 'festoon'
, or 'zeebraa'
.
✅ Click here to see the answer after you've tried it yourself at regex101.com.
One answer: [a-z]*(aa|ee|ii|oo|uu)[a-z]*
This regular expression matches strings of lowercase characters that have 'aa'
, 'ee'
, 'ii'
, 'oo'
, or 'uu'
in them anywhere. [a-z]*
means "zero or more of any lowercase characters"; essentially we are saying it doesn't matter what letters come before or after the double vowels, as long as the double vowels exist somewhere.
Activity
Write a regular expression that matches any string that contains both a lowercase letter and a number, in any order. Examples include 'billy398'
, '398!!billy'
, and 'bil3ly98'
.
✅ Click here to see the answer after you've tried it yourself at regex101.com.
One answer: (.*[a-z].*[0-9].*)|(.*[0-9].*[a-z].*)
We can break the above regex into two parts – everything before the |
, and everything after the |
.
The first part, .*[a-z].*[0-9].*
, matches strings in which there is at least one lowercase character and at least one digit, with the lowercase character coming first.
The second part, .*[0-9].*[a-z].*
, matches strings in which there is at least one lowercase character and at least one digit, with the digit coming first.
Note, the .*
between the digit and letter classes is needed in the event the string has non-digit and non-letter characters.
This is the kind of task that would be easier to accomplish with regular Python string methods.
Even more regex syntax¶
operation | example | matches ✅ | does not match ❌ |
---|---|---|---|
escape character | umich\.edu |
'umich.edu' |
'umich!edu' |
beginning of line | ^ark |
'ark two' 'ark o ark' |
'dark' |
end of line | ark$ |
'dark' 'ark o ark' |
'ark two' |
zero or one | cat? |
'ca' 'cat' |
'cart' (matches 'ca' only) |
built-in character classes* | \w+ \d+ |
'billy' '231231' |
'this person' '858 people' |
character class negation | [^a-z]+ |
'WOLVERINE551' '1721$$' |
'porch' 'billy.edu' |
**Note*: in Python's implementation of regex,
\d
refers to digits.\w
refers to alphanumeric characters ([A-Z][a-z][0-9]_
). Whenever we say "alphanumeric" in an assignment, we're referring to\w
!\s
refers to whitespace.\b
is a word boundary.
*Example (escaping)*:
- What does
he.
match? - What does
he\.
match? - What does
(734)
match? - What does
\(734\)
match?
*Example (anchors)*:
- What does
734-764
match? - What does
^734-764
match? - What does
734-764$
match?
*Example (built-in character classes)*:
- What does
\d{3} \d{3}-\d{4}
match? - What does
\bcat\b
match? Does it find a match in'my cat is hungry'
? What about'concatenate'
,'kitty cat'
, or'in-the-cat-hat'
?
Remember, in Python's implementation of regex,
\d
refers to digits.\w
refers to alphanumeric characters ([A-Z][a-z][0-9]_
). Whenever we say "alphanumeric" in an assignment, we're referring to\w
!\s
refers to whitespace.\b
is a word boundary.
Activity
Write a regular expression that matches any string that:
- is between 5 and 10 characters long, and
- is made up of only vowels (either uppercase or lowercase, including
'Y'
and'y'
), periods, and spaces.
Examples include 'yoo.ee.IOU'
and 'AI.I oey'
.
✅ Click here to see the answer after you've tried it yourself at regex101.com.
One answer: ^[aeiouyAEIOUY. ]{5,10}$
Key idea: Within a character class (i.e. [...]
), special characters do not generally need to be escaped.
Activity
This is an old exam question!
^\w{2,5}.\d*\/[^A-Z5]{1,}
Select all strings below that contain any match with the regular expression above.
"billy4/Za"
"billy4/za"
"DAI_s2154/pacific"
"daisy/ZZZZZ"
"bi_/_lly98"
"!@__!14/atlantic"
Regex in Python¶
re
in Python¶
- The
re
module is built into Python. It allows us to use regular expressions to find, extract, and replace strings.
import re
re.findall
takes in a stringregex
and a stringtext
and returns a list of all matches ofregex
intext
. You'll use this most often.
re.findall('AB*A',
'here is a string for you: ABBBA. here is another: ABBBBBBBA')
['ABBBA', 'ABBBBBBBA']
re.sub
takes in a stringregex
, a stringrepl
, and a stringtext
, and replaces all matches ofregex
intext
withrepl
.
re.sub('AB*A',
'billy',
'here is a string for you: ABBBA. here is another: ABBBBBBBA')
'here is a string for you: billy. here is another: billy'
Raw strings¶
When using regular expressions in Python, it's a good idea to use raw strings, denoted by an r
before the quotes, e.g. r'exp'
.
re.findall('\bcat\b', 'my cat is hungry')
[]
re.findall(r'\bcat\b', 'my cat is hungry')
['cat']
# Huh?
print('\bcat\b')
cat
Capture groups¶
- Surround a regex with
(
and)
to define a capture group within a pattern. Capture groups are useful for extracting relevant parts of a string.
re.findall(r'\w+@(\w+)\.edu',
'my old email was billy@notumich.edu, my new email is notbilly@umich.edu')
['notumich', 'umich']
- Notice what happens if we remove the
(
and)
!
re.findall(r'\w+@\w+\.edu',
'my old email was billy@notumich.edu, my new email is notbilly@umich.edu')
['billy@notumich.edu', 'notbilly@umich.edu']
- Earlier, we also saw that parentheses can be used to group parts of a regex together. When using
re.findall
, all groups are treated as capturing groups.
# A regex that matches strings with two of the same vowel followed by 3 digits.
# We only want to capture the digits, but...
re.findall(r'(aa|ee|ii|oo|uu)(\d{3})', 'eeoo124')
[('oo', '124')]
Example: Extracting hashtags¶
- The dataset
'data/ira.csv'
contains tweets tagged by Twitter as likely being posted by the Internet Research Agency, the tweet factory facing allegations for attempting to influence US political elections.
For more context, read this Wikipedia article.
tweets = pd.read_csv('data/ira.csv', names=['id', 'user', 'time', 'text'])
tweets.head()
id | user | time | text | |
---|---|---|---|---|
0 | 3906258 | ea85ac8be1e8ab479064ca4c0fe3ac6587f76b1ef97452... | 2016-11-16 09:04 | The Best Exercise To Lose Belly Fat In 2 weeks... |
1 | 1051443 | 8e58ab0f46d273103d9e71aa92cdaffb6e330ec7d15ae5... | 2016-12-24 04:31 | RT @Philanthropy: Dozens of ‘hate groups’ have... |
2 | 2823399 | Room Of Rumor | 2016-08-18 20:26 | Artificial intelligence can find, map poverty,... |
3 | 272878 | San Francisco Daily | 2016-03-18 19:28 | Uber balks at rules proposed by world’s busies... |
4 | 7697802 | 41bb9ae5991f53996752a0ab8dd36b543821abca8d5aed... | 2016-07-30 15:44 | RT @dirtroaddiva1: #IHatePokemonGoBecause he ... |
tweets.shape
(90000, 4)
- Question: What are the most common hashtags among all 9000 tweets?
A hashtag is any alphanumeric string beginning with'#'
, e.g.'#GoBlue'
.
Extracting hashtags¶
- We can use
re.findall
to find all of the hashtags in a particular string.
example_tweet = tweets['text'].iloc[0]
example_tweet
'The Best Exercise To Lose Belly Fat In 2 weeks https://t.co/oHFToG7rh6 #Exercise #LoseBellyFat #CatTV #TeenWolf… https://t.co/b4pr9gEx38'
re.findall(r'#(\w+)', example_tweet)
['Exercise', 'LoseBellyFat', 'CatTV', 'TeenWolf']
re.findall(r'#(\w+)', 'hey there, no hashtags here')
[]
- We can use the Series
str.findall
method, with the regular expression above, to extract hashtags out of each tweet intweets['text']
.
tags = tweets['text'].str.findall(r'#(\w+)')
tags.head()
0 [Exercise, LoseBellyFat, CatTV, TeenWolf] 1 [] 2 [tech] 3 [news] 4 [IHatePokemonGoBecause, PokesAreJokes] Name: text, dtype: object
- We can use the
sum()
method on the above Series to concatenate all of these lists into a large list!
(
pd.Series(tags.sum())
.value_counts()
.head(15)
.sort_values()
.plot(kind='barh', title='Most Common Hashtags in IRA Tweets')
)