Python Generators

What is a Python generator?

Python generators are defined as a special type of function that returns an iterator. In Python, an object that contains a countable number of items is known as the iterator. We can iterate over the iterator object, meaning we can traverse all the values using the ‘for’ loop.

In Python, creating an iterator is not an easy task. We have to implement a class with methods __iter__() and __next__() to keep track of internal states and raises StopIteration when there are no values to iterate over. In Python, Generators are a simple way to create iterators, which automatically handles the above-mentioned things.

Create a Python generator:

In Python, we define a generator just like we define a normal function. But the difference is that the generator contains a yield statement, not the return statement. So, if any function which contains at least one yield statement becomes a generator. The difference between yield and return is the return statement terminates the function. Whereas the yield statement doesn’t terminate the function. It pauses the function, saves all its states, and continues from there on for the successive function calls.

def generate_numbers():
    for i in range(5):
        yield i

output=generate_numbers()
print(output)
for i in output:
    print(i)

<generator object generate_numbers at 0x02BA0BD0>
0
1
2
3
4

Multiple yield statements:

A generator function can contain multiple yield statements.

def myGenerator():
    print("First Item")
    yield 1

    print("Second Item")
    yield 2

    print("Third Item")
    yield 3

When we execute the program in a shell, the output will be

>>> gen=myGenerator()
>>> next(gen)
First Item
1
>>> next(gen)
Second Item
2
>>> next(gen)
Third Item
3
>>> next(gen)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

The next() returns the next item in an iterator. The generator function terminates when it encounters the “StopIteration”. And one more thing to note about generators is, once we consume an item from an iterator, its gone. Hence we can iterate over the elements of the generator only once.  We have to create another generator object to reiterate over the elements.

Instead of looping over the generator object, we can directly use the generator with a ‘for’ loop. The for loop also iterates over the elements using the next() function.  The ‘for’ loop ends when it encounters StopIteration.

def myGenerator():
    print("First Item")
    yield 1

    print("Second Item")
    yield 2

    print("Third Item")
    yield 3

for i in myGenerator():
    print(i)

First Item
1
Second Item
2
Third Item
3

Functions Vs Generators:

Python Generator Expression:

Like lambda functions create anonymous functions, generator expression creates anonymous generator functions. Simple generators are easily created using generator expressions.

The syntax for generator expressions and list comprehension is similar, except that the round parentheses replace the square brackets.  And another difference between list comprehension and generator expressions is generator expression returns the generator object, unlike list comprehension which returns the entire list.

input_list=[1,2,3,4,5]
output_list=[i**2 for i in input_list]
gen=(i**2 for i in input_list)

print(output_list)
print(gen)

[1, 4, 9, 16, 25]
<generator object <genexpr> at 0x03492E70>

A generator expression is efficient than list comprehension because it produces items only when asked for. This is known as the lazy execution.

>>> next(gen)
1
>>> next(gen)
4
>>> next(gen)
9
>>> next(gen)
16
>>> next(gen)
25
>>> next(gen)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

We can also use generator expressions as arguments to a function.

>>> max(i**2 for i in input_list)
25

Advantages of Python Generator:

Easy to implement:

Generators are easier to implement compared to creating an iterator class with __iter__ and __next__ methods.

Memory efficient:

As mentioned already, generators are memory efficient. Since it produces items only when asked for it. In contrast, normal functions have to store the entire sequence in memory before returning it.

import sys

list= [i for i in range(1000)]

gen= (i for i in range(1000))

print(sys.getsizeof(list))
print(sys.getsizeof(gen))

4516
48

Generate Infinite sequence:

Since generators produce one item at a time, we can use generators to generate an infinite sequence.

Pipelining with generators:

We can also use generators to define a sequence of operations. One such example will be getting the sum of the square of  Fibonacci numbers.

def fibonocci_numbers(num):
    i,j=0,1
    for k in range(num):
        yield i
        i,j=j,i+j

def square(num):
    for i in num:
        yield i**2

print(sum(square(fibonocci_numbers(11))))

4895