Context managers are the first cousins of decorators. Like their kindred, they are tools for wrapping code around other code. However, whereas decorators wrap defined blocks of code (such as functions or classes), context managers wrap arbitrary, free-form blocks of code.
In almost every other respect, the purposes of context managers and decorators are equivalent (and, it is often the case that APIs are written to allow you to use either, as discussed later in this chapter).
This chapter introduces and explains the concept of context managers, showing how and when to use them, and enumerating the different ways of handling exceptions that may occur within context blocks.
A context manager is an object that wraps an arbitrary block of code. Context managers ensure that setup is consistently performed when the context manager is entered, and that teardown is consistently performed when the context manager is exited.
It is important to note early that the exit is guaranteed. If a context manager is entered, it will, by definition, be exited. This holds true even if the internal code raises an exception. In fact, the context manager's exit code is given an opportunity to handle such exceptions if it sees fit to do so (although it is not obligated to do so).
Therefore, context managers perform a very similar function to the try
, except
, and finally
keywords. They are often a useful mechanism to encapsulate boilerplate try
-except
-finally
constructs that you may otherwise repeat.
This is probably the most common use of context managers—they are a way to ensure cleanup.
Consider a common use case where a context manager would be useful—opening a file. You open a file in Python with the built-in open
function. When you open a file, it is your responsibility to close it again, as shown here:
try:
my_file = open('/path/to/filename', 'r')
contents = my_file.read()
finally:
my_file.close()
You use a finally
clause to ensure that, no matter what happens, my_file
will, in fact, be closed. If an error occurs when reading the file, or something else goes wrong, the finally
clause will still run, and my_file
will be closed.
So, how you do the same thing—open a file and ensure that it is properly closed—with a context manager? Context managers were introduced in Python 2.5, which adds a new keyword to the language: with
. You use the with
statement to enter a context manager.
As it happens, Python's built-in open
function can also be used as a context manager. This code is identical to what you saw previously:
with open('/path/to/filename', 'r') as my_file:
contents = my_file.read()
Essentially, what is happening here is that the with
statement evaluates the expression that comes after it (in this case, the open
call). That expression is expected to return an object with two special methods: __enter__
and __exit__
(more on those shortly). The __enter__
method returns a result that is assigned to the variable after the as
keyword.
It is important to note that the result of the expression after with
is not being assigned to said variable. In fact, it is not assigned to anything at all. It is what is returned from __enter__
that is assigned.
Simplicity is a huge reason for doing it this way. More importantly, however, remember that the exception-handling and cleanup code can sometimes be very complex, and applying it in many different places is cumbersome. As with decorators, a key reason to use context managers is to avoid repetitive code.
Remember that the with
statement's expression is responsible for returning an object that follows a particular protocol. Specifically, the object must define an __enter__
and an __exit__
method, and the latter method must take particular arguments.
The __enter__
method takes no arguments except for the traditional self
argument. It is run immediately when the object is returned, and its return value is assigned to the variable used after as
, if any (the as
clause is technically optional). Generally, the __enter__
method is responsible for performing some kind of setup.
The __exit__
method, on the other hand, takes three positional arguments (not including the traditional self
): an exception type, an exception instance, and a traceback. These three arguments are all set to None
if there is no exception, but are populated if an exception occurs within the block.
Consider the following simple class whose instances act as context managers:
class ContextManager(object):
def __init__(self):
self.entered = False
def __enter__(self):
self.entered = True
return self
def __exit__(self, exc_type, exc_instance, traceback):
self.entered = False
This context manager does very little. It simply returns itself and sets its entered
variable to True
upon entrance, and then False
upon exit.
You can observe this by looking at this context manager in the Python shell. If you create a new ContextManager
instance, you find that its entered value is False
as expected:
>>> cm = ContextManager()
>>> cm.entered
False
If you use this same ContextManager
instance as a context manager, observe that its entered
attribute becomes True
, then False
again on exit.
>>> with cm:
... cm.entered
...
True
>>> cm.entered
False
If you do not need the ContextManager
instance for anything else, you can instantiate it in the with
statement. This works because its __enter__
method just returns itself.
>>> with ContextManager() as cm:
... cm.entered
...
True
A context manager must define an __exit__
method, which may optionally handle exceptions that are raised in the wrapped code, or handle anything else needed to tear down the context manager state.
As mentioned previously, the __exit__
method must define three positional arguments: the type of the exception (called exc_type
in this chapter), the instance of the exception (called exc_instance
here), and the traceback option (called traceback
here). If no exception occurred within the context manager code, all three of these values will be None
.
If the __exit__
method receives an exception, it has the responsibility to handle that exception. Fundamentally, it has three options:
It can propagate the exception (causing it to be re-raised after __exit__
finishes).
It can suppress the exception.
It can raise a different exception.
You can propagate exceptions by having an __exit__
method that returns False
, or suppress exceptions by having an __exit__
method that returns True
. Alternatively, if __exit__
raises a different exception, it is used in place of the exceptions it was sent.
Each of these options is covered in more detail in examples throughout this chapter.
Several common reasons exist to write context managers. Generally, these involve ensuring that a certain resource is both initialized and de-initialized in an expected manner, or trying to avoid repetition.
One of the key reasons to write context managers is for situations in which you are opening and closing a resource (such as a file or a database connection). It is often important to ensure that the handle in question is closed properly, to avoid ending up with a situation where many zombie processes can build up over time.
Context managers excel here. By opening a resource in the __enter__
method and returning it, the __exit__
method is guaranteed to be run, and can close the resource before allowing the exception to bubble.
Consider the following context manager that opens a PostgreSQL database connection:
import psycopg2
class DBConnection(object):
def __init__(self, dbname=None, user=None,
password=None, host='localhost'):
self.host = host
self.dbname = dbname
self.user = user
self.password = password
def __enter__(self):
self.connection = psycopg2.connect(
dbname=self.dbname,
host=self.host,
user=self.user,
password=self.password,
)
return self.connection.cursor()
def __exit__(self, exc_type, exc_instance, traceback):
self.connection.close()
Within the context manager, it is possible to run queries against the database and retrieve results.
>>> with DBConnection(user='luke', dbname='foo') as db:
... db.execute('SELECT 1 + 1')
... db.fetchall()
...
[(2,)]
However, as soon as the context manager exists, the database cursor that you assigned to db
becomes closed, and further queries cannot be made against it.
>>> with DBConnection(user='luke', dbname='foo') as db:
... db.execute('SELECT 1 + 1')
... db.fetchall()
...
[(2,)]
>>> db.execute('SELECT 1 + 1')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
psycopg2.InterfaceError: cursor already closed
What has happened here? This context manager creates a psycopg2
connection object and returns a cursor, which the developer can use to interact with the database. What is important here, though, is that the connection is guaranteed to be closed when the context manager exits.
This is important because, as mentioned, lingering database connections not only consume memory, but they also open files or ports on both the application machine and the database machine. Additionally, some databases also have maximum connection allowances.
Note also that, unlike the previous example, this context manager does not simply return itself at the end of the __enter__
method. Instead, it returns a database cursor. This is fine, and a useful paradigm. However, it is still the context manager's __exit__
method that runs.
Most frameworks that work with databases handle opening and closing your database connections for you, but this principle remains: if you are opening a resource and must ensure that it is being properly closed, a context manager is an excellent tool.
When it comes to avoiding repetition, the most common place where this is useful is in exception handling. Context managers can both propagate and suppress exceptions, which makes them ideal for taking repetitive except
clauses and defining them in one place.
An __exit__
method that just propagates the exception up the chain can do so by returning False
. It need not interact with the exception instance at all. Consider the following context manager:
class BubbleExceptions(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_instance, traceback):
if exc_instance:
print('Bubbling up exception: %s.' % exc_instance)
return False
Running a normal block of code (that does not raise an exception) with this context manager will do nothing particularly interesting.
>>> with BubbleExceptions():
... 5 + 5
...
10
On the other hand, this block of code does actually raise an exception:
>>> with BubbleExceptions():
... 5 / 0
...
Bubbling up exception: integer division or modulo by zero.
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ZeroDivisionError: integer division or modulo by zero
A couple important things are worth noting here. The first printed line (which begins with Bubbling up exception: integer
…) was generated by the __exit__
method itself. It corresponds to the print
statement on the second line of __exit__
. This means that __exit__
did run, and complete. Because it returned False
, the exception that was sent to __exit__
in the first place is simply re-raised.
As mentioned previously, another option that the __exit__
method has is to suppress the exception that it receives. The following context manager suppresses any and every exception that might be sent to its __exit__
method (you should never actually do this, however):
class SuppressExceptions(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_instance, traceback):
if exc_instance:
print('Suppressing exception: %s.' % exc_instance)
return True
The bulk of this code is similar to the BubbleExceptions
class from earlier, with the primary difference being that now the __exit__
method returns True
instead of False
.
The example of showing normal, uninteresting code that does not raise any exception at all remains unchanged:
>>> with SuppressExceptions():
... 5 + 5
...
10
However, if you do something that raises an exception, you see a different result:
>>> with SuppressExceptions():
... 5 / 0
...
Suppressing exception: integer division or modulo by zero.
The first and most obvious thing to note is that the traceback is gone. The exception was handled (suppressed) by the __exit__
method, so program execution continues with no exception raised.
The second thing to note is that no value was ever returned. Whereas the expression 5 + 5
, when entered into the interpreter, gave a value of 10
, the exception-raising 5 / 0
simply never shows a value. The exception was raised in the process of computing a value, which triggered the running of __exit__
. A value is never actually returned. It is also worth noting that if any code was present after 5 / 0
, it would never run.
As you would expect, however, exception handlers that are defined within the context block are handled before the context block completes. Exceptions handled within a context block are considered to be dealt with and are not sent to __exit__
.
Consider the following example:
with SuppressExceptions():
try:
5 / 0
except ZeroDivisionError:
print('Exception caught within context block.')
If you run this, the “Exception caught within context block.” message will print, and no exception will be sent to __exit__
.
Although propagating exceptions is fairly straightforward, suppressing exceptions is always something that you should do carefully. Suppressing too many exceptions leads to code that is extremely difficult to debug. Simply suppressing all exceptions is fundamentally equivalent to a try
block that looks like this:
try:
[do something]
except:
pass
Suffice it to say that this is very rarely wise.
__exit__
methods can, however, conditionally suppress or handle exceptions, because they are provided the type and instance of the exception, as well as a full traceback. In fact, the exception handling is extremely customizable.
A simple exception-handling __exit__
function may simply check to see if the exception is an instance of a particular exception class, perform whatever exception handling is necessary, and return True
(or return False
) if it gets any other exception class.
class HandleValueError(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_instance, traceback):
# Return True if there is no exception.
if not exc_type:
return True
# If this is a ValueError, note that it is being handled and
# return True.
if issubclass(exc_type, ValueError):
print('Handling ValueError: %s' % exc_instance)
return True
# Propagate anything else.
return False
If you use this context manager and raise ValueError
inside the block, you see that it prints and then suppresses the exception.
>>> with HandleValueError():
... raise ValueError('Wrong value.')
...
Handling ValueError: Wrong value.
Similarly, if you use this context manager but raise a different class of exception (such as TypeError
, instead), it will bubble and you will still get your traceback.
>>> with HandleValueError():
... raise TypeError('Wrong type.')
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
TypeError: Wrong type.
By itself, this does not have a whole lot of value. After all, this is really just a substitute for a much more straightforward try
clause.
try:
[do something]
except ValueError as exc_instance:
print('Handling ValueError: %s' % exc_instance)
One way that the context manager can be valuable is when the work that must be done in the except
clause is both non-trivial and must be repeated in multiple places throughout the application. The context manager encapsulates not only the except
clause, but also its body.
There is also a little more flexibility in how the class or instance check is done. For example, suppose that you want to catch a given class of exception, but explicitly not its subclasses. You cannot do that in a traditional except
block (nor should you be able to), but a context manager is able to address such an edge case, as shown here:
class ValueErrorSubclass(ValueError):
pass
class HandleValueError(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_instance, traceback):
# Return True if there is no exception.
if not exc_type:
return True
# If this is a ValueError (but not a ValueError subclass),
# note that it is being handled and return True.
if exc_type == ValueError:
print('Handling ValueError: %s' % exc_instance)
return True
# Propagate anything else.
return False
Note that the HandleValueError
context manager has changed slightly now. It checks its type using ==
rather than the more traditional issubclass
check that the previous example used. This means that although it will handle ValueError
as before, it will not handle a ValueError
subclass such as the ValueErrorSubclass
defined previously:
>>> with HandleValueError():
... raise ValueErrorSubclass('foo bar baz')
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
__main__.ValueErrorSubclass: foo bar baz
Similarly, a context manager might decide whether to handle an exception based on not the type of the exception (which is what an except
clause must do), but rather based on an attribute of the exception.
Consider the following function designed to run shell commands conveniently, and use an exception class that is designed to be raised in response to shell errors:
import subprocess
class ShellException(Exception):
def __init__(self, code, stdout='', stderr=''):
self.code = code
self.stdout = stdout
self.stderr = stderr
def __str__(self):
return 'exit code %d - %s' % (self.code, self.stderr)
def run_command(command):
# Run the command and wait for it to complete.
proc = subprocess.Popen(command.split(' '), stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
proc.wait()
# Get the stdout and stderr from the shell.
stdout, stderr = proc.communicate()
# Sanity check: If the shell returned a non-zero exit status, raise an
# exception.
if proc.returncode > 0:
raise ShellException(proc.returncode, stdout, stderr)
# Return stdout.
return stdout
Such a function (and exception class) is very easy to use. The following is an attempt to rm
a bogus file:
run_command('rm bogusfile')
Running this will generate the ShellException
traceback as expected.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 11, in run_command
__main__.ShellException: exit code 1 - rm: bogusfile: No such file or directory
What happens when it comes time to handle these exceptions? Handling any generic ShellException
is easy, but imagine a situation where you receive a ShellException
but only want to handle a particular exit code. A context manager is one possible way to approach this.
For example, say that you want to remove a file, but you are okay with a situation where the file was already removed. (For the purpose of this example, ignore that os.remove
exists.) In this case, you would be fine with a return code of 0
, which indicates successful removal of the file, as well as a return code of 1
, which indicates that the file was already absent. On the other hand, an exit code of 64
is still problematic, because this would indicate a usage error of some kind. This should still be raised.
Here is a context manager that would allow some ShellException
instances based on their code:
class AcceptableErrorCodes(object):
def __init__(self, *error_codes):
self.error_codes = error_codes
def __enter__(self):
return self
def __exit__(self, exc_type, exc_instance, traceback):
# Sanity check: If this is not an exceptional situation, then just
# be done.
if not exc_type:
return True
# Sanity check: If this is anything other than a ShellException,
# then we do not actually know what to do with it.
if not issubclass(exc_type, ShellException):
return False
# Return True if and only if the ShellException has a code that
# matches one of the codes on our error_codes list.
return exc_instance.code in self.error_codes
This example code actually introduces a new pattern. The context manager is given the error codes that it should allow when the context manager is initiated. In this case, AcceptableErrorCodes
takes any number of integers as arguments, and those are used to determine which error codes are actually acceptable.
If you want to attempt to remove a non-existent file when using the AcceptableErrorCodes
context manager, it will work without incident.
>>> with AcceptableErrorCodes(1):
... run_command('rm bogusfile')
...
What this context manager will not do, however, is just blindly swallow up every ShellException
it gets. Consider the following case where you actually use rm
incorrectly:
>>> with AcceptableErrorCodes(1):
... # -m is not a switch available to rm (at least in Mac OS X).
... run_command('rm -m bogusfile')
...
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "<stdin>", line 11, in run_command
__main__.ShellException: exit code 64 - rm: illegal option -- m
usage: rm [-f | -i] [-dPRrvW] file ...
unlink file
So, why did this cause a traceback? Because the exit code was 64
(on Mac OS X; this may vary based on the exact operating system you are using), and you told the context manager that the only acceptable erratic exit code was 1
. Therefore, __exit__
returned False
, and the exception was bubbled as usual.
Many of the context managers explored thus far are actually very simple. Although they are fully constructed classes, their only real purpose is to provide straightforward, linear __enter__
and __exit__
functionality.
This structure is extremely powerful. It allows for the creation of very complex and context managers that can do a great deal of customizable logic. However, many context managers are very simple, and creating a class and manually defining __enter__
and __exit__
may seem like overkill.
A simpler approach is designed around handling the simple cases. The Python standard library provides a decorator that will decorate a simple function and make it into a context manager class.
This decorator is @contextlib.contextmanager
, and functions it decorates are expected to yield
a single value somewhere during the function. (The yield
statement is discussed in more detail in Chapter 3, “Generators.”)
Consider what the AcceptableErrorCodes
class might look like as a single, more straightforward function:
import contextlib
@contextlib.contextmanager
def acceptable_error_codes(*codes):
try:
yield
except ShellException as exc_instance:
# If this error code is not in the list of acceptable error
# codes, re-raise the exception.
if exc_instance.code not in codes:
raise
# This was an acceptable error; no need to do anything.
pass
This function ultimately does the exact same thing that your class did. (It is worth noting that the pass
line is for instructional purposes—it is obviously not necessary.)
>>> with acceptable_error_codes(1):
... run_command('rm bogusfile')
Similarly, error codes are still checked, and only the appropriate ones are intercepted.
>>> with acceptable_error_codes(1):
... run_command('rm -m bogusfile')
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "<stdin>", line 11, in run_command
__main__.ShellException: exit code 64 - rm: illegal option -- M
usage: rm [-f | -i] [-dPRrvW] file ...
unlink file
This simpler syntax (just declaring a function with a single yield
and using the @contextlib.contextmanager
decorator) is more than sufficient to create most simple context managers, and is easier to read later. Create a context manager class yourself when you need the power that this provides, and use the decorator with a function otherwise.
Context managers provide an excellent way to ensure that resources are handled appropriately, as well as to take exception-handling code that would be repeated in multiple different places throughout an application and giving that code a single home.
Along with decorators, context managers are tools for employing the simple principle of not repeating yourself unless you absolutely must. Where decorators encase named functions and classes, context managers are ideal for encasing arbitrary blocks of code.
Chapter 3 discusses generators, which produce values one by one when iterated, as each value is needed, rather than having to compute an entire set of values in advance.