<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Algorithm Blogs &#187; memory leak</title>
	<atom:link href="http://www.algorithm.co.il/blogs/index.php/tag/memory-leak/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.algorithm.co.il/blogs</link>
	<description>Algorithms, for the heck of it</description>
	<lastBuildDate>Thu, 22 Apr 2010 21:04:32 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9.2</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Leaky method references</title>
		<link>http://www.algorithm.co.il/blogs/index.php/programming/python/leaky-method-references/</link>
		<comments>http://www.algorithm.co.il/blogs/index.php/programming/python/leaky-method-references/#comments</comments>
		<pubDate>Wed, 04 Nov 2009 08:39:25 +0000</pubDate>
		<dc:creator>lorg</dc:creator>
				<category><![CDATA[Python]]></category>
		<category><![CDATA[Bugs]]></category>
		<category><![CDATA[Gotcha]]></category>
		<category><![CDATA[memory leak]]></category>
		<category><![CDATA[__del__]]></category>

		<guid isPermaLink="false">http://www.algorithm.co.il/blogs/?p=472</guid>
		<description><![CDATA[After reading my last post regarding __del__, you should know that __del__ + reference cycle = leak.
Let's say that you do need to use __del__, so you decide to avoid reference cycles. You write your code in such a way as to use the minimum necessary cycles, and for the ones that remain you use [...]]]></description>
			<content:encoded><![CDATA[<p>After reading my last post <a href="http://www.algorithm.co.il/blogs/index.php/programming/python/python-gotchas-no-2-garbage-collection-oddities/">regarding __del__</a>, you should know that __del__ + reference cycle = leak.<br />
Let's say that you do need to use __del__, so you decide to avoid reference cycles. You write your code in such a way as to use the minimum necessary cycles, and for the ones that remain you use the weakref module.</p>
<p>You might still have cycles where you don't expect it - in references to methods.<br />
Consider the following piece of code. Can you spot the reference cycle?</p>
<div class="syntax_hilite">
<div id="python-3">
<div class="python"><span style="color: #00007f;font-weight:bold;">class</span> A<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> f<span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">return</span></p>
<p>a = A<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span><br />
a.<span style="color: #000000;">g</span> = a.<span style="color: #000000;">f</span></div>
</div>
</div>
<p></p>
<p>This code has the following reference cycle: a -> a.g -> a.f -> a.<br />
How come?<br />
When you call a.f like so: "a.f()" <strong>two</strong> things happen:<br />
1. A.f is bounded to a<br />
2. The bounded A.f is called with the first parameter getting the bounded value.</p>
<p>You may consider that "a.f" is syntactic sugar for the partial function application, A.f gets a as a first argument but doesn't get called yet.</p>
<p>When you use "a.g = a.f" what actually happens is a holding a reference to a bounded method, which holds a reference to a.</p>
<p>An idiom that uses these cycles is implementing state machines. Consider the following example code:</p>
<div class="syntax_hilite">
<div id="python-4">
<div class="python"><span style="color: #00007f;font-weight:bold;">class</span> MyMachine<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> <span style="color: #0000cd;">__init__</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #008000;">self</span>.<span style="color: #000000;">next_func</span> = <span style="color: #008000;">self</span>.<span style="color: #000000;">state_a</span><br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> run<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, <span style="color: #008000;">input</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">for</span> x <span style="color: #00007f;font-weight:bold;">in</span> <span style="color: #008000;">input</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #008000;">self</span>.<span style="color: #000000;">next_func</span><span style="color: black;">&#40;</span>x<span style="color: black;">&#41;</span><br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> state_a<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, value<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">print</span> <span style="color: #483d8b;">'a: '</span>, value<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #008000;">self</span>.<span style="color: #000000;">next_func</span> = <span style="color: #008000;">self</span>.<span style="color: #000000;">state_b</span><br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> state_b<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, value<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">print</span> <span style="color: #483d8b;">'b: '</span>, value<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #008000;">self</span>.<span style="color: #000000;">next_func</span> = <span style="color: #008000;">self</span>.<span style="color: #000000;">state_a</span></div>
</div>
</div>
<p></p>
<p>Of course, my code was a bit more complicated than that, but the basic idea remains. (My code usually created some kind of function table in __init__ used to lookup the next function, and lookups happened outside "state functions"). I've seen many state machine recipes include method references - and rightly so. It's a clear and easy way to code a state machine. (For example, <a href="http://code.activestate.com/recipes/574430/">this state machine recipe</a>).<br />
Be careful though - once you add __del__ to these simple recipes you might end up with a memory leak. </p>
<p>Short note: I was going to publish this post a few days ago, but <a href="http://www.kylev.com/2009/11/03/finding-my-first-python-reference-cycle/">kylev</a> beat me to it. This just goes to show that other people encountered this kind of cycle.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.algorithm.co.il/blogs/index.php/programming/python/leaky-method-references/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Python Tidbits</title>
		<link>http://www.algorithm.co.il/blogs/index.php/programming/python/python-tidbits/</link>
		<comments>http://www.algorithm.co.il/blogs/index.php/programming/python/python-tidbits/#comments</comments>
		<pubDate>Thu, 10 Apr 2008 18:07:51 +0000</pubDate>
		<dc:creator>lorg</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[Utility Functions]]></category>
		<category><![CDATA[gc]]></category>
		<category><![CDATA[memory leak]]></category>
		<category><![CDATA[reference cycle]]></category>
		<category><![CDATA[undefined]]></category>

		<guid isPermaLink="false">http://www.algorithm.co.il/blogs/index.php/programming/python/python-tidbits/</guid>
		<description><![CDATA[Memory Leaks
With the advent of the Python's garbage collector, it would seem that Python programs should not leak memory. However, this is not always the case. With just a bit of circular references, you can find yourself leaking quite a bit of memory. 
A simple example is a bidirectional data structure, like a 2-way linked [...]]]></description>
			<content:encoded><![CDATA[<p><u><strong>Memory Leaks</strong></u></p>
<p>With the advent of the Python's garbage collector, it would seem that Python programs should not leak memory. However, this is not always the case. With just a bit of circular references, you can find yourself leaking quite a bit of memory. </p>
<p>A simple example is a bidirectional data structure, like a 2-way linked list, or a tree where the nodes have a reference to their parents. To really free those, you have to break the cycle manually.  Another option is to use a weakref for some of the references. (Usually, I like to use weakref.proxy for the parent links.)</p>
<p>An easy way to test code for memory leaks is to run it in a loop (allocate, do job, deallocate, repeat), and call gc.collect() before and after. If the value that gc.collect() returns after the loop grows with the size of the loop, you might have a memory leak.</p>
<p>I found out about a special case of a potential memory leak just the other day.<br />
Let's say you want to implement some switch code. A common idiom is to write it like this:</p>
<div class="syntax_hilite">
<div id="python-9">
<div class="python"><span style="color: #00007f;font-weight:bold;">def</span> a<span style="color: black;">&#40;</span>x<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">print</span> <span style="color: #483d8b;">'a'</span>, x<br />
<span style="color: #00007f;font-weight:bold;">def</span> b<span style="color: black;">&#40;</span>x<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">print</span> <span style="color: #483d8b;">'b'</span>, x<br />
...<br />
<span style="color: #000000;">func</span> = <span style="color: black;">&#123;</span><span style="color: #ff4500;">0</span>: a, <span style="color: #ff4500;">1</span>: b<span style="color: black;">&#125;</span><span style="color: black;">&#91;</span>x<span style="color: black;">&#93;</span><br />
func<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span></div>
</div>
</div>
<p></p>
<p>This works well enough. However, if you do the same thing in a class instance, things start to get messy:</p>
<div class="syntax_hilite">
<div id="python-10">
<div class="python"><span style="color: #00007f;font-weight:bold;">class</span> A<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> <span style="color: #0000cd;">__init__</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #008000;">self</span>.<span style="color: #000000;">lookup</span> = <span style="color: black;">&#123;</span><span style="color: #ff4500;">0</span>: <span style="color: #008000;">self</span>.<span style="color: #000000;">a</span>, <span style="color: #ff4500;">1</span>: <span style="color: #008000;">self</span>.<span style="color: #000000;">b</span><span style="color: black;">&#125;</span><br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> a<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, x<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">print</span> <span style="color: #483d8b;">'a'</span>, x<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> b<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, x<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">print</span> <span style="color: #483d8b;">'b'</span>, x<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> do_something<span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; ...<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #000000;">func</span> = <span style="color: #008000;">self</span>.<span style="color: #000000;">lookup</span><span style="color: black;">&#91;</span>x<span style="color: black;">&#93;</span><br />
&nbsp; &nbsp; &nbsp; &nbsp; func<span style="color: black;">&#40;</span>x<span style="color: black;">&#41;</span></div>
</div>
</div>
<p></p>
<p>In this simple example, __init__ creates a reference cycle. This is because self.a creates a method object on the spot, with a reference to self.<br />
So, if z is an instance of A, our cycle is: z->lookup->z.a->z. </p>
<p>Take heed.<br />
(Note: Some cycles are collectible with gc.collect(). The simple cycle described here is such a case. It may get ugly though.)</p>
<p><u><strong>Undefined</strong></u><br />
While writing the <a href="http://www.algorithm.co.il/blogs/index.php/programming/python/issues-in-writing-a-vm-part-2/">VM for vial</a>, we needed some way to avoid testing assembly code for undefined values.</p>
<p>For example, after executing a div instruction, CF is undefined. We handle this in the templates by writing CF=UNKNOWN(), and we allow to VM to decide on a value. If you need the VM to just execute code, returning 0 for undefined values is fine. If, however, you are testing code, your actual CPU might return a non-zero value, and your test will fail with no real reason.</p>
<p>To solve this problem, our VM may be told to return a special singleton object for undefined values:</p>
<div class="syntax_hilite">
<div id="python-11">
<div class="python"><span style="color: #00007f;font-weight:bold;">class</span> _UndefinedNumber<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> <span style="color: #0000cd;">__eq__</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, other<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">if</span> other <span style="color: #00007f;font-weight:bold;">is</span> <span style="color: #008000;">self</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">return</span> <span style="color: #008000;">True</span><br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">return</span> <span style="color: #008000;">False</span><br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> <span style="color: #0000cd;">__ne__</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, other<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">return</span> <span style="color: #00007f;font-weight:bold;">not</span> <span style="color: #008000;">self</span> == other<br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> binary_action<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, other<span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">return</span> <span style="color: #008000;">self</span><br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> unary_action<span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">return</span> <span style="color: #008000;">self</span><br />
&nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">def</span> <span style="color: #0000cd;">__str__</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:<br />
&nbsp; &nbsp; &nbsp; &nbsp; <span style="color: #00007f;font-weight:bold;">return</span> <span style="color: #483d8b;">"Undefined"</span><br />
&nbsp; &nbsp; <span style="color: #0000cd;">__repr__</span> = <span style="color: #0000cd;">__str__</span></p>
<p>&nbsp; &nbsp; <span style="color: black;">&#40;</span><span style="color: #0000cd;">__add__</span>, <span style="color: #0000cd;">__sub__</span>, <span style="color: #0000cd;">__mul__</span>, <br />
&nbsp; &nbsp; &nbsp;<span style="color: #0000cd;">__div__</span>, __floordiv__, <span style="color: #0000cd;">__mod__</span>, <br />
&nbsp; &nbsp; &nbsp;<span style="color: #0000cd;">__pow__</span>, <span style="color: #0000cd;">__and__</span>, <span style="color: #0000cd;">__xor__</span>, <br />
&nbsp; &nbsp; &nbsp;<span style="color: #0000cd;">__or__</span>, <span style="color: #0000cd;">__lshift__</span>, <span style="color: #0000cd;">__rshift__</span>, <br />
&nbsp; &nbsp; &nbsp;<span style="color: #0000cd;">__rlshift__</span>, <span style="color: #0000cd;">__rrshift__</span>, <span style="color: #0000cd;">__radd__</span>,<br />
&nbsp; &nbsp; &nbsp;<span style="color: #0000cd;">__rsub__</span>, <span style="color: #0000cd;">__rmul__</span>, <span style="color: #0000cd;">__rdiv__</span>, <br />
&nbsp; &nbsp; &nbsp;__rfloordiv__, <span style="color: #0000cd;">__rmod__</span>, <span style="color: #0000cd;">__rpow__</span>, <br />
&nbsp; &nbsp; &nbsp;<span style="color: #0000cd;">__rand__</span>, <span style="color: #0000cd;">__rxor__</span>, <span style="color: #0000cd;">__ror__</span><span style="color: black;">&#41;</span> = <span style="color: #ff4500;">24</span>*<span style="color: black;">&#91;</span>binary_action<span style="color: black;">&#93;</span></p>
<p>&nbsp; &nbsp; <span style="color: black;">&#40;</span><span style="color: #0000cd;">__neg__</span>, <span style="color: #0000cd;">__pos__</span>, <br />
&nbsp; &nbsp; &nbsp;<span style="color: #0000cd;">__abs__</span>, <span style="color: #0000cd;">__invert__</span><span style="color: black;">&#41;</span> = <span style="color: #ff4500;">4</span>*<span style="color: black;">&#91;</span>unary_action<span style="color: black;">&#93;</span></p>
<p>Undefined = _UndefinedNumber<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span></div>
</div>
</div>
<p></p>
<p>Now, we test only the flags that are not Undefined. Note that any operation on an Undefined returns an Undefined. This is done to allow registers to be Undefined as well. This way, if AX gets Undefined, so will AH and EAX.</p>
<p>Here is an example:</p>
<div class="syntax_hilite">
<div id="python-12">
<div class="python">In <span style="color: black;">&#91;</span><span style="color: #ff4500;">2</span><span style="color: black;">&#93;</span>: Undefined + <span style="color: #ff4500;">1</span><br />
Out<span style="color: black;">&#91;</span><span style="color: #ff4500;">2</span><span style="color: black;">&#93;</span>: Undefined</p>
<p>In <span style="color: black;">&#91;</span><span style="color: #ff4500;">3</span><span style="color: black;">&#93;</span>: Undefined * <span style="color: #ff4500;">2</span><br />
Out<span style="color: black;">&#91;</span><span style="color: #ff4500;">3</span><span style="color: black;">&#93;</span>: Undefined</p>
<p>In <span style="color: black;">&#91;</span><span style="color: #ff4500;">4</span><span style="color: black;">&#93;</span>: Undefined / Undefined<br />
Out<span style="color: black;">&#91;</span><span style="color: #ff4500;">4</span><span style="color: black;">&#93;</span>: Undefined</div>
</div>
</div>
<p></p>
<p>Note that this constant might behave a bit strangely. For example, Undefined * 0 is Undefined, and Undefined - Undefined is also Undefined. Because of that, I'm not sure using this object in production circumstances is a good idea, so in the meantime, this is just a testing utility.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.algorithm.co.il/blogs/index.php/programming/python/python-tidbits/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
