The next Function

You can ask a generator for a value without using a for…in loop. Sometimes you may want to just get a single value, or a fixed number of values. Python provides the built-in next function, which can ask a generator (actually, any object with a _next_ method, called next in Python 2) for its next value.

The earlier fibonacci function yields an infinite sequence of Fibonacci numbers. Instead of iterating over the entire thing, you can ask for values one at a time.

First, you simply create your generator by calling the fibonacci function and saving its returned value. Because the function has yield statements rather than a return statement, the Python ­interpreter knows to just return the generator object.

>>> gen = fibonacci()
>>> gen
<generator object fibonacci at 0x101555dc8>

At this point, it is worth noting that none of the code within fibonacci has actually run. The only thing that the interpreter has done is recognize that a generator is present and return a generator object, which is ready to run the code once a value is requested.

You can use the built-in next function to request your first value, as shown here:

>>> next(gen)
1

Now (and only now) some of the actual code in the fibonacci function has been run. (To make the explanation as clear as possible, an explicit continue statement has been added at the end of the loop.)

def fibonacci():
    numbers = []
    while True:
        if len(numbers) < 2:  # True; numbers == []
            numbers.append(1)
        else:
            numbers.append(sum(numbers))
            numbers.pop(0)
        yield numbers[-1]
        continue

The function is entered, and it begins the first iteration of the while loop. Because the numbers list is empty at this point, the value 1 is appended to the list. Finally, you get to the yield numbers[-1] statement. At this point, the generator has been given a value to yield, so execution halts, and the value 1 is yielded. This is where the execution ends; the continue statement does not yet run.

Now, issue next(gen) again, as shown here:

>>> next(gen)
1

Execution picks up where it left off, which means the first thing to run is the continue statement.

def fibonacci():
    numbers = []
    while True:
        if len(numbers) < 2:
            numbers.append(1)
        else:
            numbers.append(sum(numbers))
            numbers.pop(0)
        yield numbers[-1]
        continue

This sends you back to the top of the while loop. Your numbers list only has one member (it is [1]), so len(numbers) is still less than 2, and that path is chosen at the if statement again. Your numbers list is now [1, 1], and the final element of the list is yielded, stopping execution.

def fibonacci():
    numbers = []
    while True:
        if len(numbers) < 2:  # True; numbers == [1]
            numbers.append(1)
        else:
            numbers.append(sum(numbers))
            numbers.pop(0)
        yield numbers[-1]
        continue

Now, issue next(gen) yet again, as shown here:

>>> next(gen)
2

Again, execution picks up where it left off, meaning the next thing to run is the continue statement.

def fibonacci():
    numbers = []
    while True:
        if len(numbers) < 2:
            numbers.append(1)
        else:
            numbers.append(sum(numbers))
            numbers.pop(0)
        yield numbers[-1]
        continue

The continue statement sends the interpreter back to the stop of the while loop. However, now it takes the else pathway when it gets to the if statement, because numbers is now a list with two elements ([1, 1]). The sum of the two elements is then appended to the end of the list, and the first element is removed. Again, you get to the yield statement, and it yields the final element of the list, which is 2.

def fibonacci():
    numbers = []
    while True:
        if len(numbers) < 2:  # False; numbers == [1, 1]
            numbers.append(1)
        else:
            numbers.append(sum(numbers))
            numbers.pop(0)
        yield numbers[-1]
        continue

If you issue next(gen) again, the interpreter will follow the same path (because the length of the numbers list is still 2). Of course, now the numbers list itself has changed from [1, 1] to [1, 2], so the result is different. The value 3 is appended to the list, the 1 is lopped off of the beginning, and 3 is yielded.

>>> next(gen)
3

If you continue to ask for more values, you see this pattern repeat. The same code runs, but against an updated numbers list, so the yielded values continue along the Fibonacci series.

>>> next(gen)
5
>>> next(gen)
8
>>> next(gen)
13
>>> next(gen)
21

Notice that a few things are not happening. You are not storing a huge list of Fibonacci numbers in memory. The only numbers that you must store are the most recent two, because they are required to find the next number in the series. The generator scraps anything that is out of date. This would matter if the generator were to continue on indefinitely, because if you needlessly held on to every previous value, eventually the list would fill up free memory.

Similarly, the generator only computes each value in the series when it is specifically requested. At this point in code execution, the generator has not bothered to determine that the next value that it will need to yield back (if asked) is 34, precisely because it may not be asked.

The StopIteration Exception

As with other functions, with generators, you may want to have more than one potential exit path. For example, the following “plain” function has multiple exit paths using multiple return statements:

def my_function(foo, add_extra_things=True):
    foo += '\nadded things'
    if not add_extra_things:
        return foo
    foo += '\n added extra things'
    return foo

This function normally returns at the end of the block. However, if the keyword argument add_extra_things is provided and set to False, the earlier return statement on the third line of the function will be hit instead, and function execution will be cut off there.

Plenty of reasons exist to do this, and generators must have a mechanism to serve a similar purpose.

>>> def my_generator():
...     yield 1
...     return
...
  File "<stdin>", line 3
SyntaxError: 'return' with argument inside generator

>>> def my_generator():
...     yield 1
...     yield 2
...     raise StopIteration
...     yield 3

>>> [i for i in my_generator()]
[1, 2]

>>> gen = my_generator()
>>> next(gen)
1
>>> next(gen)
2
>>> next(gen)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in my_generator
StopIteration

return 42

raise StopIteration(42)

yield 42
return

range

During the earlier discussion about the distinction between iterables and iterators, you learned about the range function, which returns an iterable range object.

The range object's iterator is a generator. It returns sequential values, beginning with the range object's floor, and continuing through its ceiling. By default, its sequence is simply adding one to each value to get the next value to yield. But an optional third argument to the range function, step, enables you to specify a different increment, including a negative one.

dict.items and Family

The built-in dictionary class in Python includes three methods that allow for iterating over the ­dictionary, and all three are iterables whose iterators are generators: keys, values, and items.

The purpose of these methods is to allow for iteration over the keys, values, or two-tuples of keys and values (items) of a dictionary, as shown here:

>>> dictionary = {'foo': 'bar', 'baz': 'bacon'}
>>> iterator = iter(dictionary.items())
>>> next(iterator)
('foo', 'bar')
>>> next(iterator)
('baz', 'bacon')

One value of using a generator here is that it prevents the need to make an additional copy of the dictionary (or pieces of the dictionary) in another format. dict.items does not need to reformat the entire dictionary into a list of two-tuples. It simply returns back one two-tuple at a time, when it is requested.

You can see a side effect of this if you attempt to alter the dictionary during iteration, as shown here:

>>> dictionary = {'foo': 'bar', 'baz': 'bacon'}
>>> iterator = iter(dictionary.items())
>>> next(iterator)
('foo', 'bar')
>>> dictionary['spam'] = 'eggs'
>>> next(iterator)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration

Because the items iterator is a generator that simply reads from the referenced dictionary, it does not know what it should do if the dictionary changes while it is working. In the face of ambiguity, it refuses the temptation to guess, and raises RuntimeError instead.

zip

Python includes a built-in function called zip that takes multiple iterable objects and iterates over them together, yielding the first element from each iterable (in a tuple), then the second, then the third, and so on, until the end of the shortest iterable is reached. Following is an example:

>>> z = zip(['a', 'b', 'c', 'd'], ['x', 'y', 'z'])
>>> next(z)
('a', 'x')
>>> next(z)
('b', 'y')
>>> next(z)
('c', 'z')
>>> next(z)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

The reasons to use zip are similar to the reasons to use dict.items and family. Its purpose is to yield back members of its iterables in a different structure, one set at a time. This alleviates the need to copy over the entire thing in memory if such an operation is not necessary.

map

A cousin to the zip function is the built-in map function. The map function takes a function that accepts N arguments as well as N iterables, and computes the result of the function against the sequential members of each iterable, stopping when it reaches the end of the shortest one.

Similarly to zip, a generator is used for the iterator here, precisely because it is undesirable to compute every value in advance. After all, these values may or may not be needed. Instead, each value is computed when and only when it is requested.

>>> m = map(lambda x, y: max([x, y]), [4, 1, 7], [3, 4, 5])
>>> next(m)
4
>>> next(m)
4
>>> next(m)
7
>>> next(m)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

As before, this is a trivial operation when dealing with small iterables. However, given a larger data structure, the use of a generator may entail serious savings in computation time or memory use, because the entire structure does not need to be computed and transformed at once.

File Objects

One of the most commonly used generators in Python is the open file object. Although you can interact in many ways with open files in Python, and it is common with smaller files to just call read to read the entire file into memory, the file object does support the generator pattern, which reads the file from disk one line at a time. This is very important when operating on larger files. It is not always reasonable to read the entirety of a file into memory.

For historical reasons, file objects have a special method called readline used for reading a line at a time. However, the generator protocol is also implemented, and calling next on a file does the same thing.

Consider the following simple file:

$ cat lines.txt
line 1
line 2
line 3
line 4
line 5

You read it in the Python shell by using the built-in open function. The resulting object is, among other things, a generator.

>>> f = open('lines.txt')
>>> next(f)
'line 1\n'
>>> next(f)
'line 2\n'

Note that the generator reads one line at a time and yields the entire line, including the trailing ­newline (\n) character.

If you attempt to call next after the end of the file is reached, StopIteration is raised as expected.

>>> next(f)
'line 5\n'
>>> next(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

It is worth noting that __next__ and readline are not exact aliases for one another here. Once end of file is reached, __next__ raises StopIteration as it would for any other generator, whereas readline actually catches this exception and returns an empty string:

>>> next(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>> f.readline()
''

Accessing Data in Pieces

The first (and probably most common) reason to write generators is to cover cases where you must access data in chunks, but where it is undesirable to store copies of the entire thing.

This is essentially what happens in the file object generator explored previously, as well as the dict.items (and family) methods. When dealing with small files, it is entirely reasonable to read the entire file into memory and do whatever work needs to be done against that in-memory string.

On the other hand, what if a file is large? What if you need to restructure a dictionary that is large? Sometimes, making a copy to manipulate data is not a feasible operation. This is where accessing data in pieces is a valuable capability.

When iterating over a large file with the generator method, it does not matter how large the file is. Each line will be read and yielded, one at a time. When iterating over a dictionary with dict.items, it does not matter whatsoever how large the source dictionary is. The iterator will iterate over it one piece at a time, and yield only that two-tuple.

The same principle applies to generators that you write. A generator is a useful tool in any situation where you want to iterate over a substantial amount of data, and it is unnecessary to store or copy the entirety of that data in memory at once.

Computing Data in Pieces

The second common reason to write generators is to compute data only as it is needed. Consider the range function or the fibonacci function discussed earlier in this chapter. A program that must loop over each number between zero and a googleplex need not store a list of every number between those figures. It is sufficient to simply keep adding one until the maximum is reached.

Similarly, the fibonacci function does not need to compute every Fibonacci number (an impossible task, because there exists an infinite number of them—more on this shortly). It simply must determine the single next Fibonacci number and yield it back.

This can be important because sometimes the computation of each item in a sequence can be expensive. It is not useful to compute the entire series unnecessarily.