Day 11. Iterators & Generators in python.

Hello Everyone..

Today, we are discussing iterators and generators in python, how is it useful and how be can perform a different task in easily manner.

Let’s Start..

1. Iterators.

What are the iterators in Python?

Iterator in Python is simply an object that can be iterated upon. An object which will return data, one element at a time.

Technically speaking, Python iterator object must implement two special methods, __iter__() and __next__(), collectively called the iterator protocol.

An object is called iterable if we can get an iterator from it. Most of the built-in containers in Python like: list, tuple, string, etc. are iterables.

The iter() function (which in turn calls the __iter__() method) returns an iterator from them.

Iterating Through an Iterator in Python

# define a list
my_list = [4, 7, 0, 3]
# get an iterator using iter()
my_iter = iter(my_list)
## iterate through it using next()#prints 4
#prints 7
## next(obj) is same as obj.__next__()#prints 0
#prints 3
## This will raise error, no items left

A more elegant way of automatically iterating is by using the for loop. Using this, we can iterate over any object that can return an iterator, for example, list, string, file, etc.

>>> for element in my_list:...     print(element)...4703

How for the loop actually works?

In fact the for loop can iterate over any iterable. Let's take a closer look at how the for loop is actually implemented in Python.

for element in iterable:# do something with element

It is actually implemented as.

# create an iterator object from that iterableiter_obj = iter(iterable)# infinite loopwhile True:try:# get the next itemelement = next(iter_obj)# do something with elementexcept StopIteration:# if StopIteration is raised, break from loopbreak

So internally, the for loop creates an iterator object, iter_obj by calling iter() on the iterable.

Ironically, this for loop is actually an infinite while loop.

Inside the loop, it calls next() to get the next element and executes the body of the for loop with this value. After all the items exhaust, StopIteration it is raised which is internally caught and the loop ends. Note that any other kind of exception will pass through.

Building Your Own Iterator in Python

The __iter__() method returns the iterator object itself. If required, some initialization can be performed.

The __next__() method must return the next item in the sequence. On reaching the end, and in subsequent calls, it must raise StopIteration.

Here, we show an example that will give us the next power of 2 in each iteration. The power exponent starts from zero up to a user set number.

class Maths:
"""Class to implement an iterator
of powers of two"""
def __init__(self, max = 0):
self.max = max
def __iter__(self):
self.n = 0
return self
def __next__(self):
if self.n <= self.max:
result = 2 ** self.n
self.n += 1
return result
raise StopIteration

Now we can create an iterator and iterate through it as follows.

>>> a = Maths(4)>>> i = iter(a)>>> next(i)1>>> next(i)2>>> next(i)4>>> next(i)8>>> next(i)16>>> next(i)Traceback (most recent call last):...StopIteration

We can also use a for loop to iterate over our iterator class.

>>> for i in Maths(5):...     print(i)...12481632

Python Infinite Iterators

Here is a simple example to demonstrate infinite iterators.

The built-in function iter() can be called with two arguments where the first argument must be a callable object (function) and second is the sentinel. The iterator calls this function until the returned value is equal to the sentinel.

>>> int()0>>> inf = iter(int,1)>>> next(inf)0>>> next(inf)0

We can see that the int() function always returns 0. So passing it as iter(int,1) will return an iterator that calls int() until the returned value equals 1. This never happens and we get an infinite iterator.

We can also built our own infinite iterators. The following iterator will, theoretically, return all the odd numbers.

class InfIter:
"""Infinite iterator to return all
odd numbers"""
def __iter__(self):
self.num = 1
return self
def __next__(self):
num = self.num
self.num += 2
return num

A sample run would be as follows.

>>> a = iter(InfIter())>>> next(a)1>>> next(a)3>>> next(a)5>>> next(a)7

The advantage of using iterators is that they save resources. Like shown above, we could get all the odd numbers without storing the entire number system in memory. We can have infinite items (theoretically) in finite memory.

Iterator also makes our code look cool.

2. Generators.

What are the generators in Python?

This is both lengthy and counter-intuitive. The generator comes into rescue in such situations.

Python generators are a simple way of creating iterators. All the overhead we mentioned above is automatically handled by generators in Python.

Simply speaking, a generator is a function that returns an object (iterator) which we can iterate over (one value at a time).

How to create a generator in Python?

If a function contains at least one yield statement (it may contain other yield or return statements), it becomes a generator function. Both yield and return will return some value from a function.

The difference is that, while a return statement terminates a function entirely, yield statement pauses the function saving all its states and later continues from there on successive calls.

Differences between Generator function and a Normal function

  • Generator function contains one or more yield statement.
  • When called, it returns an object (iterator) but does not start execution immediately.
  • Methods like __iter__() and __next__() are implemented automatically. So we can iterate through the items using next().
  • Once the function yields, the function is paused and the control is transferred to the caller.
  • Local variables and their states are remembered between successive calls.
  • Finally, when the function terminates, StopIteration is raised automatically on further calls.

Here is an example to illustrate all of the points stated above. We have a generator function named my_gen() with several yield statements.

# A simple generator function
def my_gen():
n = 1
print('This is printed first')
# Generator function contains yield statements
yield n
n += 1
print('This is printed second')
yield n
n += 1
print('This is printed at last')
yield n

An interactive run in the interpreter is given below. Run these in the Python shell to see the output.

>>> # It returns an object but does not start execution immediately.>>> a = my_gen()>>> # We can iterate through the items using next().>>> next(a)This is printed first1>>> # Once the function yields, the function is paused and the control is transferred to the caller.>>> # Local variables and theirs states are remembered between successive calls.>>> next(a)This is printed second2>>> next(a)This is printed at last3>>> # Finally, when the function terminates, StopIteration is raised automatically on further calls.>>> next(a)Traceback (most recent call last):...StopIteration>>> next(a)Traceback (most recent call last):...StopIteration

One interesting thing to note in the above example is that, the value of variable n is remembered between each call.

Unlike normal functions, the local variables are not destroyed when the function yields. Furthermore, the generator object can be iterated only once.

To restart the process we need to create another generator object using something like a = my_gen().

Note: One final thing to note is that we can use generators with for loops directly.

This is because, a for loop takes an iterator and iterates over it using next() function. It automatically ends when StopIteration is raised. Check here to know how a for loop is actually implemented in Python.

# A simple generator function
def my_gen():
n = 1
print('This is printed first')
# Generator function contains yield statements
yield n
n += 1
print('This is printed second')
yield n
n += 1
print('This is printed at last')
yield n
# Using for loop
for item in my_gen():

When you run the program, the output will be:

This is printed first
This is printed second
This is printed at last

Python Generators with a Loop

Normally, generator functions are implemented with a loop having a suitable terminating condition.

Let’s take an example of a generator that reverses a string.

def rev_str(my_str):
length = len(my_str)
for i in range(length - 1,-1,-1):
yield my_str[i]
# For loop to reverse the string
# Output:
# o
# l
# l
# e
# h
for char in rev_str("hello"):

In this example, we use range() function to get the index in reverse order using the for loop.

It turns out that this generator function not only works with string, but also with other kind of iterables like list, tuple etc.

Python Generator Expression

Same as lambda function creates an anonymous function, generator expression creates an anonymous generator function.

The syntax for generator expression is similar to that of list comprehension in Python. But the square brackets are replaced with round parentheses.

The major difference between a list comprehension and a generator expression is that while list comprehension produces the entire list, generator expression produces one item at a time.

They are kind of lazy, producing items only when asked for. For this reason, a generator expression is a much more memory efficient than an equivalent list comprehension.

# Initialize the list
my_list = [1, 3, 6, 10]
# square each term using list comprehension
# Output: [1, 9, 36, 100]
[x**2 for x in my_list]
# same thing can be done using generator expression
# Output: <generator object <genexpr> at 0x0000000002EBDAF8>
(x**2 for x in my_list)

We can see above that the generator expression did not produce the required result immediately. Instead, it returned a generator object which produces items on demand.

# Intialize the list
my_list = [1, 3, 6, 10]
a = (x**2 for x in my_list)
# Output: 1
# Output: 9
# Output: 36
# Output: 100
# Output: StopIteration

Generator expression can be used inside functions. When used in such a way, the round parentheses can be dropped.

>>> sum(x**2 for x in my_list)146>>> max(x**2 for x in my_list)100

Thank You..

Data Scientist, Blockchain enthusiast, Entrepreneur.