16. Advanced Regular Expressions

By Bernd Klein. Last modified: 08 Mar 2024.

Introduction

In our introduction to regular expressions of our tutorial we have covered the basic principles of regular expressions. We have shown, how the simplest regular expression looks like. We have also learnt, how to use regular expressions in Python by using the search() and the match() methods of the re module. The concept of formulating and using character classes should be well known by now, as well as the predefined character classes like \d, \D, \s, \S, and so on. You must have learnt how to match the beginning and the end of a string with a regular expression. You must know the special meaning of the question mark to make items optional. We have also introduced the quantifiers to repeat characters and groups arbitrarily or in certain ranges.

You must also be familiar with the use of grouping and the syntax and usage of back references.

Furthermore, we had explained the match objects of the re module and the information they contain and how to retrieve this information by using the methods span(), start(), end(), and group().

The introduction ended with a comprehensive example in Python.

In this chapter we will continue with our explanations about the syntax of the regular expressions. We will also explain further methods of the Python module re. E.g. how to find all the matched substrings of a regular expression. A task which needs programming in other programming languages like Perl or Java, but can be dealt with the call of one method of the re module of Python. So far, we only know how to define a choice of characters with a character class. We will demonstrate how to formulate alternations of substrings in this chapter of our tutorial,

Live Python training

Enjoying this page? We offer live Python training courses covering the content of this site.

See our Python training courses

See our Machine Learning with Python training courses

Finding all Matched Substrings

The Python module re provides another great method, which other languages like Perl and Java don't provide. If you want to find all the substrings in a string, which match a regular expression, you have to use a loop in Perl and other languages, as can be seen in the following Perl snippet:

while ($string =~ m/regex/g) {
  print "Found '$&'.  Next attempt at character " . pos($string)+1 . "\n";
}

It's a lot easier in Python. No need to loop. We can just use the findall method of the re module:

 re.findall(pattern, string[, flags]) 

findall returns all non-overlapping matches of pattern in string, as a list of strings. The string is scanned left-to-right, and matches are returned in the order in which they are found.

t="A fat cat doesn't eat oat but a rat eats bats."
mo = re.findall("[force]at", t)
print(mo)

['fat', 'cat', 'eat', 'oat', 'rat', 'eat']

If one or more groups are present in the pattern, findall returns a list of groups. This will be a list of tuples if the pattern has more than one group. We demonstrate this in our next example. We have a long string with various Python training courses and their dates. With the first call to findall, we don't use any grouping and receive the complete string as a result. In the next call, we use grouping and findall returns a list of 2-tuples, each having the course name as the first component and the dates as the second component:

import re
courses = "Python Training Course for Beginners: 15/Aug/2011 - 19/Aug/2011;Python Training Course Intermediate: 12/Dec/2011 - 16/Dec/2011;Python Text Processing Course:31/Oct/2011 - 4/Nov/2011"
items = re.findall("[^:]*:[^;]*;?", courses)
items

['Python Training Course for Beginners: 15/Aug/2011 - 19/Aug/2011;',
 'Python Training Course Intermediate: 12/Dec/2011 - 16/Dec/2011;',
 'Python Text Processing Course:31/Oct/2011 - 4/Nov/2011']

items = re.findall("([^:]*):([^;]*;?)", courses)
items

[('Python Training Course for Beginners', ' 15/Aug/2011 - 19/Aug/2011;'),
 ('Python Training Course Intermediate', ' 12/Dec/2011 - 16/Dec/2011;'),
 ('Python Text Processing Course', '31/Oct/2011 - 4/Nov/2011')]

Alternations

In our introduction to regular expressions we had introduced character classes. Character classes offer a choice out of a set of characters. Sometimes we need a choice between several regular expressions. It's a logical "or" and that's why the symbol for this construct is the "|" symbol. In the following example, we check, if one of the cities London, Paris, Zurich, Konstanz Bern or Strasbourg appear in a string preceded by the word "location":

import re
str = "Course location is London or Paris!"
mo =