Premature ending of generator in list comprehension

  • A+
Category:Languages

I'm using generators in list comprehensions, and getting some unexpected behavior with one of the generators ending early. Why does creating the generator outside of the list comprehension cause the behavior to change?

The generator I created is as follows:

def inc_range(a,b):     for i in range(min(a,b), max(a,b) + 1):         yield i 

The first way of calling is as follows:

[(i,j) for i in inc_range(1,3) for j in inc_range(4,6)] 

This gives me the following result:

[(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)] 

The second way of calling it is as follows:

a = inc_range(1,3) b = inc_range(4,6)  [(i,j) for i in a for j in b] 

This gives me the following:

[(1, 4), (1, 5), (1, 6)] 

Experimenting around, the following two examples gave me the first result:

a = range(1,4) b = range(4,7)  [(i,j) for i in a for j in b] 
a = (i for i in range(1,4)) b = (i for i in range(4,7))  a = list(a) b = list(b)  [(i,j) for i in a for j in b] 

While the following gave me the second result again.

a = (i for i in range(1,4)) b = (i for i in range(4,7))  [(i,j) for i in a for j in b] 

What rule am I violating here regarding generators? Why does it make a difference when I assign the generators to variables before using them in a list comprehension, vs. using them directly?

ANSWERS

Check out the following answers which helped me understand what is occurring here:

Alex Yu mkrieger1

 


To get the desired result, the "inner" generator would have to be run as many times as the "outer" generator yields a value.

But, after the first run, the "inner" generator is exhausted and cannot be run again.

Adding a print illustrates this (simplifying the example):

>>> def inc(a, b): ...    for i in range(a, b): ...        print(i) ...        yield i ... >>> a = inc(1, 4) >>> b = inc(4, 7) >>> [(i, j) for i in a for j in b] 1  # <-- a begins to run 4  # <-- b begins to run 5 6  # <-- b exhausted here 2  # <-- a continued, but not resulting in list item, because lacking value from b 3 [(1, 4), (1, 5), (1, 6)] 

The reason why not storing the generators in variables works as expected is because a new "inner" generator is created for each iteration of the "outer" generator. Again, illustrated by some prints:

>>> def inc(a, b): ...    print('started', a, b) ...    for i in range(a, b): ...        yield i ...  >>> [(i, j) for i in inc(1, 4) for j in inc(4, 7)] started 1 4 started 4 7 started 4 7 started 4 7 [(1, 4), (1, 5), (1, 6), (2, 4), (2, 5), (2, 6), (3, 4), (3, 5), (3, 6)] 

The reason why using range objects or lists works as expected is because they can be iterated over arbitrarily many times without being exhausted.

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: