Programming Python

GeneratorExit: another reason to upgrade to Python 2.6

Ever heard of GeneratorExit? Unless you write generators, and do some unusual stuff with them, you probably haven’t encountered it.

GeneratorExit is a special exception that gets raised from within a generator when it is close()-ed.
For example, consider the following code:

def func():
	f = file('temp.txt','w')
	for i in range(10):
		f.write('%d\n' % i)
		yield i

When iterating over func(), when will f be closed? Obviously, when the generator is exhausted. What if f.write raised an exception? It’s better to make sure then (in py2.5, do from __future__ import with-statement):

def func():
    with file('temp.txt','w') as f:
        for i in range(10):
            f.write('%d\n' % i)
            yield i

(If you don’t know what with does: basically it wraps the code in a try-finally clause, and makes sure that f is closed in the finally).

This solution is clean enough, but there’s something interesting going on. What if we use func in the following manner?

for i in func():
    print i
    if i == 5:

Well, the generator func() returned is no longer used, so eventually it will be cleaned by the gc. However, the code inside func is still executing – it is “blocking” on the yield statement!

To solve this issue cleanly, the yield statement will raise an exception when the generator returned is close()-ed.

What does this have to do with upgrading to Python 2.6?
It turns out that GeneratorExit is supposed to inherit from BaseException, just like KeyboardInterrupt and StopIteration, so that except Exception won’t catch it. However, in Python 2.5 it inherits from Exception, which means that the following sample code will behave strangely:

def func():
    with file('temp.txt','w') as f:
        for i in range(10):
                f.write('%d\n' % i)
                yield i
            except Exception:
In [13]: for i in func():
   ....:     print i
   ....:     if i == 5:
   ....:         break
Exception exceptions.RuntimeError: 'generator ignored GeneratorExit' in <generator object at 0x021BEAD0> ignored

The workaround is simple: before except Exception, catch GeneratorExit and handle it correctly. Or, upgrade to Python 2.6 instead :)

One reply on “GeneratorExit: another reason to upgrade to Python 2.6”

Thanks for the post. I was having a problem because I had a generator with “except:”, and it was catching GeneratorExit and producing an error whenever I broke a loop reading from the generator. The solution was to replace that with “except Exception:”, what works because, as you mention, GeneratorExit does not inherit from Exception.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.