Chapter 2
Context Managers

Context managers are the first cousins of decorators. Like their kindred, they are tools for wrapping code around other code. However, whereas decorators wrap defined blocks of code (such as functions or classes), context managers wrap arbitrary, free-form blocks of code.

In almost every other respect, the purposes of context managers and decorators are equivalent (and, it is often the case that APIs are written to allow you to use either, as discussed later in this chapter).

This chapter introduces and explains the concept of context managers, showing how and when to use them, and enumerating the different ways of handling exceptions that may occur within context blocks.

What Is a Context Manager?

A context manager is an object that wraps an arbitrary block of code. Context managers ensure that setup is consistently performed when the context manager is entered, and that teardown is consistently performed when the context manager is exited.

It is important to note early that the exit is guaranteed. If a context manager is entered, it will, by definition, be exited. This holds true even if the internal code raises an exception. In fact, the context manager's exit code is given an opportunity to handle such exceptions if it sees fit to do so (although it is not obligated to do so).

Therefore, context managers perform a very similar function to the try, except, and finally keywords. They are often a useful mechanism to encapsulate boilerplate try-except-finally constructs that you may otherwise repeat.

This is probably the most common use of context managers—they are a way to ensure cleanup.

Context Manager Syntax

Consider a common use case where a context manager would be useful—opening a file. You open a file in Python with the built-in open function. When you open a file, it is your responsibility to close it again, as shown here:

try:
    my_file = open('/path/to/filename', 'r')
    contents = my_file.read()
finally:
    my_file.close()

You use a finally clause to ensure that, no matter what happens, my_file will, in fact, be closed. If an error occurs when reading the file, or something else goes wrong, the finally clause will still run, and my_file will be closed.

The with Statement

So, how you do the same thing—open a file and ensure that it is properly closed—with a context manager? Context managers were introduced in Python 2.5, which adds a new keyword to the language: with. You use the with statement to enter a context manager.

As it happens, Python's built-in open function can also be used as a context manager. This code is identical to what you saw previously:

with open('/path/to/filename', 'r') as my_file:
    contents = my_file.read()

Essentially, what is happening here is that the with statement evaluates the expression that comes after it (in this case, the open call). That expression is expected to return an object with two special methods: __enter__ and __exit__ (more on those shortly). The __enter__ method returns a result that is assigned to the variable after the as keyword.

It is important to note that the result of the expression after with is not being assigned to said variable. In fact, it is not assigned to anything at all. It is what is returned from __enter__ that is assigned.

Simplicity is a huge reason for doing it this way. More importantly, however, remember that the exception-handling and cleanup code can sometimes be very complex, and applying it in many different places is cumbersome. As with decorators, a key reason to use context managers is to avoid repetitive code.

The enter and exit Methods

Remember that the with statement's expression is responsible for returning an object that follows a particular protocol. Specifically, the object must define an __enter__ and an __exit__ method, and the latter method must take particular arguments.

The __enter__ method takes no arguments except for the traditional self argument. It is run immediately when the object is returned, and its return value is assigned to the variable used after as, if any (the as clause is technically optional). Generally, the __enter__ method is responsible for performing some kind of setup.

The __exit__ method, on the other hand, takes three positional arguments (not including the traditional self): an exception type, an exception instance, and a traceback. These three arguments are all set to None if there is no exception, but are populated if an exception occurs within the block.

Consider the following simple class whose instances act as context managers:

class ContextManager(object):
    def __init__(self):
        self.entered = False
def __enter__(self): self.entered = True return self
def __exit__(self, exc_type, exc_instance, traceback): self.entered = False

This context manager does very little. It simply returns itself and sets its entered variable to True upon entrance, and then False upon exit.

You can observe this by looking at this context manager in the Python shell. If you create a new ContextManager instance, you find that its entered value is False as expected:

>>> cm = ContextManager()
>>> cm.entered
False

If you use this same ContextManager instance as a context manager, observe that its entered ­attribute becomes True, then False again on exit.

>>> with cm:
...   cm.entered
...
True
>>> cm.entered
False

If you do not need the ContextManager instance for anything else, you can instantiate it in the with statement. This works because its __enter__ method just returns itself.

>>> with ContextManager() as cm:
...   cm.entered
...
True

Exception Handling

A context manager must define an __exit__ method, which may optionally handle exceptions that are raised in the wrapped code, or handle anything else needed to tear down the context manager state.

As mentioned previously, the __exit__ method must define three positional arguments: the type of the exception (called exc_type in this chapter), the instance of the exception (called exc_instance here), and the traceback option (called traceback here). If no exception occurred within the context manager code, all three of these values will be None.

If the __exit__ method receives an exception, it has the responsibility to handle that exception. Fundamentally, it has three options:

  • It can propagate the exception (causing it to be re-raised after __exit__ finishes).

  • It can suppress the exception.

  • It can raise a different exception.

You can propagate exceptions by having an __exit__ method that returns False, or suppress exceptions by having an __exit__ method that returns True. Alternatively, if __exit__ raises a different exception, it is used in place of the exceptions it was sent.

Each of these options is covered in more detail in examples throughout this chapter.

When You Should Write Context Managers

Several common reasons exist to write context managers. Generally, these involve ensuring that a certain resource is both initialized and de-initialized in an expected manner, or trying to avoid repetition.

Resource Cleanliness

One of the key reasons to write context managers is for situations in which you are opening and closing a resource (such as a file or a database connection). It is often important to ensure that the handle in question is closed properly, to avoid ending up with a situation where many zombie processes can build up over time.

Context managers excel here. By opening a resource in the __enter__ method and returning it, the __exit__ method is guaranteed to be run, and can close the resource before allowing the exception to bubble.

Consider the following context manager that opens a PostgreSQL database connection:

import psycopg2

class DBConnection(object): def __init__(self, dbname=None, user=None, password=None, host='localhost'): self.host = host self.dbname = dbname self.user = user self.password = password
def __enter__(self): self.connection = psycopg2.connect( dbname=self.dbname, host=self.host, user=self.user, password=self.password, ) return self.connection.cursor()
def __exit__(self, exc_type, exc_instance, traceback): self.connection.close()

Within the context manager, it is possible to run queries against the database and retrieve results.

>>> with DBConnection(user='luke', dbname='foo') as db:
...     db.execute('SELECT 1 + 1')
...     db.fetchall()
...
[(2,)]

However, as soon as the context manager exists, the database cursor that you assigned to db becomes closed, and further queries cannot be made against it.

>>> with DBConnection(user='luke', dbname='foo') as db:
...     db.execute('SELECT 1 + 1')
...     db.fetchall()
...
[(2,)]
>>> db.execute('SELECT 1 + 1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
psycopg2.InterfaceError: cursor already closed

What has happened here? This context manager creates a psycopg2 connection object and returns a cursor, which the developer can use to interact with the database. What is important here, though, is that the connection is guaranteed to be closed when the context manager exits.

This is important because, as mentioned, lingering database connections not only consume memory, but they also open files or ports on both the application machine and the database machine. Additionally, some databases also have maximum connection allowances.

Note also that, unlike the previous example, this context manager does not simply return itself at the end of the __enter__ method. Instead, it returns a database cursor. This is fine, and a useful paradigm. However, it is still the context manager's __exit__ method that runs.

Most frameworks that work with databases handle opening and closing your database connections for you, but this principle remains: if you are opening a resource and must ensure that it is being properly closed, a context manager is an excellent tool.

Avoiding Repetition

When it comes to avoiding repetition, the most common place where this is useful is in exception handling. Context managers can both propagate and suppress exceptions, which makes them ideal for taking repetitive except clauses and defining them in one place.

Propagating Exceptions

An __exit__ method that just propagates the exception up the chain can do so by returning False. It need not interact with the exception instance at all. Consider the following context manager:

class BubbleExceptions(object):
    def __enter__(self):
        return self
def __exit__(self, exc_type, exc_instance, traceback): if exc_instance: print('Bubbling up exception: %s.' % exc_instance) return False

Running a normal block of code (that does not raise an exception) with this context manager will do nothing particularly interesting.

>>> with BubbleExceptions():
...     5 + 5
...
10

On the other hand, this block of code does actually raise an exception:

>>> with BubbleExceptions():
...     5 / 0
...
Bubbling up exception: integer division or modulo by zero.
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ZeroDivisionError: integer division or modulo by zero

A couple important things are worth noting here. The first printed line (which begins with Bubbling up exception: integer…) was generated by the __exit__ method itself. It corresponds to the print statement on the second line of __exit__. This means that __exit__ did run, and complete. Because it returned False, the exception that was sent to __exit__ in the first place is simply re-raised.

Suppressing Exceptions

As mentioned previously, another option that the __exit__ method has is to suppress the exception that it receives. The following context manager suppresses any and every exception that might be sent to its __exit__ method (you should never actually do this, however):

class SuppressExceptions(object):
    def __enter__(self):
        return self
def __exit__(self, exc_type, exc_instance, traceback): if exc_instance: print('Suppressing exception: %s.' % exc_instance) return True

The bulk of this code is similar to the BubbleExceptions class from earlier, with the primary ­difference being that now the __exit__ method returns True instead of False.

The example of showing normal, uninteresting code that does not raise any exception at all remains unchanged:

>>> with SuppressExceptions():
...     5 + 5
...
10

However, if you do something that raises an exception, you see a different result:

>>> with SuppressExceptions():
...     5 / 0
...
Suppressing exception: integer division or modulo by zero.

The first and most obvious thing to note is that the traceback is gone. The exception was handled (suppressed) by the __exit__ method, so program execution continues with no exception raised.

The second thing to note is that no value was ever returned. Whereas the expression 5 + 5, when entered into the interpreter, gave a value of 10, the exception-raising 5 / 0 simply never shows a value. The exception was raised in the process of computing a value, which triggered the running of __exit__. A value is never actually returned. It is also worth noting that if any code was present after 5 / 0, it would never run.

As you would expect, however, exception handlers that are defined within the context block are handled before the context block completes. Exceptions handled within a context block are considered to be dealt with and are not sent to __exit__.

Consider the following example:

with SuppressExceptions():
    try:
        5 / 0
    except ZeroDivisionError:
        print('Exception caught within context block.')

If you run this, the “Exception caught within context block.” message will print, and no exception will be sent to __exit__.

Although propagating exceptions is fairly straightforward, suppressing exceptions is always something that you should do carefully. Suppressing too many exceptions leads to code that is extremely difficult to debug. Simply suppressing all exceptions is fundamentally equivalent to a try block that looks like this:

try:
    [do something]
except:
    pass

Suffice it to say that this is very rarely wise.

__exit__ methods can, however, conditionally suppress or handle exceptions, because they are provided the type and instance of the exception, as well as a full traceback. In fact, the exception handling is extremely customizable.

Handling Certain Exception Classes

A simple exception-handling __exit__ function may simply check to see if the exception is an instance of a particular exception class, perform whatever exception handling is necessary, and return True (or return False) if it gets any other exception class.

class HandleValueError(object):
    def __enter__(self):
        return self
def __exit__(self, exc_type, exc_instance, traceback): # Return True if there is no exception. if not exc_type: return True
# If this is a ValueError, note that it is being handled and # return True. if issubclass(exc_type, ValueError): print('Handling ValueError: %s' % exc_instance) return True
# Propagate anything else. return False

If you use this context manager and raise ValueError inside the block, you see that it prints and then suppresses the exception.

>>> with HandleValueError():
...     raise ValueError('Wrong value.')
...
Handling ValueError: Wrong value.

Similarly, if you use this context manager but raise a different class of exception (such as TypeError, instead), it will bubble and you will still get your traceback.

>>> with HandleValueError():
...     raise TypeError('Wrong type.')
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: Wrong type.

By itself, this does not have a whole lot of value. After all, this is really just a substitute for a much more straightforward try clause.

try:
    [do something]
except ValueError as exc_instance:
    print('Handling ValueError: %s' % exc_instance)

One way that the context manager can be valuable is when the work that must be done in the except clause is both non-trivial and must be repeated in multiple places throughout the application. The context manager encapsulates not only the except clause, but also its body.

Excluding Subclasses

There is also a little more flexibility in how the class or instance check is done. For example, suppose that you want to catch a given class of exception, but explicitly not its subclasses. You cannot do that in a traditional except block (nor should you be able to), but a context manager is able to address such an edge case, as shown here:

class ValueErrorSubclass(ValueError):
    pass

class HandleValueError(object): def __enter__(self): return self
def __exit__(self, exc_type, exc_instance, traceback): # Return True if there is no exception. if not exc_type: return True
# If this is a ValueError (but not a ValueError subclass), # note that it is being handled and return True. if exc_type == ValueError: print('Handling ValueError: %s' % exc_instance) return True
# Propagate anything else. return False

Note that the HandleValueError context manager has changed slightly now. It checks its type using == rather than the more traditional issubclass check that the previous example used. This means that although it will handle ValueError as before, it will not handle a ValueError subclass such as the ValueErrorSubclass defined previously:

>>> with HandleValueError():
...     raise ValueErrorSubclass('foo bar baz')
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
__main__.ValueErrorSubclass: foo bar baz
Attribute-Based Exception Handling

Similarly, a context manager might decide whether to handle an exception based on not the type of the exception (which is what an except clause must do), but rather based on an attribute of the exception.

Consider the following function designed to run shell commands conveniently, and use an exception class that is designed to be raised in response to shell errors:

import subprocess

class ShellException(Exception): def __init__(self, code, stdout='', stderr=''): self.code = code self.stdout = stdout self.stderr = stderr
def __str__(self): return 'exit code %d - %s' % (self.code, self.stderr)

def run_command(command): # Run the command and wait for it to complete. proc = subprocess.Popen(command.split(' '), stdout=subprocess.PIPE, stderr=subprocess.PIPE) proc.wait()
# Get the stdout and stderr from the shell. stdout, stderr = proc.communicate()
# Sanity check: If the shell returned a non-zero exit status, raise an # exception. if proc.returncode > 0: raise ShellException(proc.returncode, stdout, stderr)
# Return stdout. return stdout

Such a function (and exception class) is very easy to use. The following is an attempt to rm a bogus file:

run_command('rm bogusfile')

Running this will generate the ShellException traceback as expected.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 11, in run_command
__main__.ShellException: exit code 1 - rm: bogusfile: No such file or directory

What happens when it comes time to handle these exceptions? Handling any generic ShellException is easy, but imagine a situation where you receive a ShellException but only want to handle a particular exit code. A context manager is one possible way to approach this.

For example, say that you want to remove a file, but you are okay with a situation where the file was already removed. (For the purpose of this example, ignore that os.remove exists.) In this case, you would be fine with a return code of 0, which indicates successful removal of the file, as well as a return code of 1, which indicates that the file was already absent. On the other hand, an exit code of 64 is still problematic, because this would indicate a usage error of some kind. This should still be raised.

Here is a context manager that would allow some ShellException instances based on their code:

class AcceptableErrorCodes(object):
    def __init__(self, *error_codes):
        self.error_codes = error_codes
def __enter__(self): return self
def __exit__(self, exc_type, exc_instance, traceback): # Sanity check: If this is not an exceptional situation, then just # be done. if not exc_type: return True
# Sanity check: If this is anything other than a ShellException, # then we do not actually know what to do with it. if not issubclass(exc_type, ShellException): return False
# Return True if and only if the ShellException has a code that # matches one of the codes on our error_codes list. return exc_instance.code in self.error_codes

This example code actually introduces a new pattern. The context manager is given the error codes that it should allow when the context manager is initiated. In this case, AcceptableErrorCodes takes any number of integers as arguments, and those are used to determine which error codes are actually acceptable.

If you want to attempt to remove a non-existent file when using the AcceptableErrorCodes context manager, it will work without incident.

>>> with AcceptableErrorCodes(1):
...     run_command('rm bogusfile')
...

What this context manager will not do, however, is just blindly swallow up every ShellException it gets. Consider the following case where you actually use rm incorrectly:

>>> with AcceptableErrorCodes(1):
...     # -m is not a switch available to rm (at least in Mac OS X).
...     run_command('rm -m bogusfile')
...
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "<stdin>", line 11, in run_command
__main__.ShellException: exit code 64 - rm: illegal option -- m
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file

So, why did this cause a traceback? Because the exit code was 64 (on Mac OS X; this may vary based on the exact operating system you are using), and you told the context manager that the only acceptable erratic exit code was 1. Therefore, __exit__ returned False, and the exception was bubbled as usual.

A Simpler Syntax

Many of the context managers explored thus far are actually very simple. Although they are fully constructed classes, their only real purpose is to provide straightforward, linear __enter__ and __exit__ functionality.

This structure is extremely powerful. It allows for the creation of very complex and context managers that can do a great deal of customizable logic. However, many context managers are very simple, and creating a class and manually defining __enter__ and __exit__ may seem like overkill.

A simpler approach is designed around handling the simple cases. The Python standard library provides a decorator that will decorate a simple function and make it into a context manager class.

This decorator is @contextlib.contextmanager, and functions it decorates are expected to yield a single value somewhere during the function. (The yield statement is discussed in more detail in Chapter 3, “Generators.”)

Consider what the AcceptableErrorCodes class might look like as a single, more straightforward function:

import contextlib

@contextlib.contextmanager def acceptable_error_codes(*codes): try: yield except ShellException as exc_instance: # If this error code is not in the list of acceptable error # codes, re-raise the exception. if exc_instance.code not in codes: raise
# This was an acceptable error; no need to do anything. pass

This function ultimately does the exact same thing that your class did. (It is worth noting that the pass line is for instructional purposes—it is obviously not necessary.)

>>> with acceptable_error_codes(1):
...     run_command('rm bogusfile')

Similarly, error codes are still checked, and only the appropriate ones are intercepted.

>>> with acceptable_error_codes(1):
...     run_command('rm -m bogusfile')
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 11, in run_command
__main__.ShellException: exit code 64 - rm: illegal option -- M
usage: rm [-f | -i] [-dPRrvW] file ...
       unlink file

This simpler syntax (just declaring a function with a single yield and using the @contextlib.contextmanager decorator) is more than sufficient to create most simple context managers, and is easier to read later. Create a context manager class yourself when you need the power that this ­provides, and use the decorator with a function otherwise.

Summary

Context managers provide an excellent way to ensure that resources are handled appropriately, as well as to take exception-handling code that would be repeated in multiple different places throughout an application and giving that code a single home.

Along with decorators, context managers are tools for employing the simple principle of not repeating yourself unless you absolutely must. Where decorators encase named functions and classes, ­context managers are ideal for encasing arbitrary blocks of code.

Chapter 3 discusses generators, which produce values one by one when iterated, as each value is needed, rather than having to compute an entire set of values in advance.

    Reset