0. Basics

Know the basics – especially profiling!
Just by looking at the profiling output, you can tell where does the computing time go. To get that information, I like to sort on cumulative time (i.e, time taken by a given function and all functions called from it, over all of its calls).

0.5. Knowing your goal, and your enemy

The kind of optimizations you do, and how far you’re willing to go is dependent on your code’s users. If you’re writing batch processing software, your required time for running might be a minute, an hour, or a day. So far, I had to optimize various cases of weeks to minutes for batch processing and also seconds to milliseconds for web-application UI.
Your timing should apply to typical input, and probably to your biggest probable input as well.

Create some simple benchmark you can test your code against. It’s important that your benchmark be typical ‘complexity-wise’, but smaller in size, so that running it and getting profiling results takes no more than a few seconds. You may even want multiple benchmarks, each one for a different size. That way, once you are more sure of yourself, you can run your code against the larger benchmark. If your benchmarks are real inputs – all the better.

1. Python vs. C and similar considerations

In my line of work, I usually do research oriented development. That means that it’s harder to know upfront where the bottlenecks will be. As a result, the prevailing attitude is usually “let’s write it in Python, and later, when the need arises, convert the critical code to C”.
So far, I haven’t had the chance to do that. Usually what happens is we write the code, it works well enough, and we figure that the flexibility of writing it in Python is more important than the Python to C conversion gains. Also, Python is not always the bottleneck – sometimes it’s a database, or some 3rd party API.
Usually “import pysco”, and changing the code to allow parallel processing is cheaper and simpler than the conversion to C.

2. The small time-eater

A common problem is when a relatively trivial function is taking a lot of cumulative time. That’s usually a sign you’re doing something wrong. I had this issue when I used my symbolic constants for a new project. Consider the following:

def SymbolInt(value, name):
    class _SymbolInt(int):
        def __str__(self):
            return name
        def __repr__(self):
            return 'SymbolInt(%d, "%s")' % (value, name)
        def __eq__(self, other):
            if isinstance(other, str):
                other = other.lower()
            return int(self)==other or name.lower() == other
        def __ne__(self, other):
            return not self == other
    return _SymbolInt(value)

This one is very nice for interactive interfaces. However, in the new project, I found out that __eq__ was taking *a lot* of time. Way more than it should, even when I wasn’t comparing SymbolInt-s to strings!
It turned out that ‘or name.lower() == other’ was very bad speed wise. So for that project, I removed this subcondition, and voila! My code was fast!

3. The algorithm is critical

In many cases I’ve worked on, the greatest reductions in running time were due to algorithm changes. That means that playing with issues such as variable lookups and so on should come after you’re mostly settled on your algorithm. The latest example that I can think of is the set counting problem, where using my solution got me down from two weeks to 20 something minutes on my real input.
Later I did some simpler optimizations that chopped off a few more minutes.

4. Avoiding loops

That one is easy. Everyone knows you should avoid loops, especially nested ones. Still, there are some cases where your code just has to have these loops – because that’s the essence of what your code is doing.

To make loop avoidance possible, and specifically cartesian product kind of loops, consider refactoring your code to use set intersections and unions. As simple illustration, consider:
Instead of this,

for x in a:
    for y in b:
        if x == y:
            yield (x,y)

do this:

return set(a) & set(b)

Sometimes, applying this change doesn’t quite fit your algorithm. In that cases, try to change your algorithm to accommodate. For example, it might yield less accurate results. In that case, aim for returning more than you need, and then do a second pass to filter the bad ones. The time gains from avoiding the extra loops should still be worth it.

(Note: this is similar to doing your computations in your database queries instead of in your code. Similar ideas apply.)

5. Lookups

If you spend time looking for something, use a dict. If that is not feasable, use any other data-structure that fits your problem. For example, let’s say you’re looking for given strings in a lot of files. You can build a small index beforehand, and instead of looking at the files each time, just look at this index.

(Note: this is similar to creating an index on the database column you are searching on.)

6. Memory

When dealing with large inputs, you’ll usually want to reduce your memory requirements. Consider an algorithm that requires O(n) memory, for n-sized inputs. All you need is a factor of 4, and 500 megs of input, and your code will choke on many current machines.
Also, I’ve found out that writing your code in such a way as to use drastically less memory, will sometimes force me to write it more time-efficiently as well.

There are a few techniques to dealing with the memory issue. The central idea is to have as little of your data as you need available at any time.

7. Generators

Generator expressions are usually preferable to list comprehensions. Similarly, consider replacing this kind of functions:

def myfunc(some_input):
    ...
    result = []
    for bla in foo:
        ...
        result.append(bar)
    return result

with the following idiom:

def myfunc(some_input):
    ...
    for bla in foo:
        ...
        yield bar

This has the added advantage of simplifying myfunc, as its state is kept for you. On really big inputs and outputs, this one could save you from keeping all of your output in memory.
If you are not familiar with generators, I suggest reading David Beazley’s presentation on the subject, it’s an excellent read, regardless of optimizations.

8. Outputs

If your goal is to generate output, dump it to a file as soon as possible. This is made simple by the previous idiom:

for bar in myfunc():
    #process bar
    ...
    dump(foobar)

Just make sure that dump doesn’t keep your data around for too long.
For example, I once had to insert a lot of data into a database. After I finished processing each record, I would insert it. The bottleneck was the database. I tried flushing only after several inserts (which meant inserts in chunks of N for various N), until I was introduced to the solution: bulk inserts.
The, my extraction script just dumped to a text file, which was lightning fast, and later I did the bulk insert.

9. Summing up

Do

sum(x for x in some_generator)

Instead of

for x in some_list:
    my_sum += x

Kidding!
Use your profiler, your head, psyco, and more experienced advice in the best order that suits you. As I’ve come to learn, getting advice from friends is an excellent way to avoid bashing your head against some mad bugger’s O(n²) wall.

Consider a web application, required to do lengthy operations from time to time, either IO bound, or CPU bound. To save time, and maybe also bandwidth or CPU time, we are interested in caching the results of these operations.
So, let’s say we create some database table, that has the following fields:

• Unique key: input
• output
• use_count *
• Last use date *

I’ve marked with an asterisk the fields that are optional, and are related to managing the size of the cache. These are not relevant at the moment.
Here is how code using the cache would look like (in pseudocode form):

result = cache.select(input)
if result:
    return result
result = compute(input)
cache.insert(input, result)
return result

This code will work well under normal circumstances. However, in a multithreaded environment, or any environment where access to the database is shared, there is a race-condtion: What happens if there are two requests for the same input at about the same time?
Here’s a simple scheduling that will show the bug:

Thread1: result = cache.select(input). result is None
Thread1: result = compute(input)
Thread2: result = cache.select(input) result is None
Thread2: result = compute(input)
Thread2: cache.insert(input, result)
Thread1: cache.insert(input, result) – exception – duplicate records for the unique key input!

This is a classic race condition. And here’s a small challenge: What’s the best way to solve it?

You think this might be enough, as your smart tagging algorithm can add tags and attach tags to blog posts. If you want to change it manually, then no problem, you just modify any of these tables. For example, if the algorithm makes a mistake, you just erase the mapping and/or the tag.

The problems start when you want to run the algorithm more than once.
First, the algorithm must not create duplicates on the second run. This is very easy to implement and doesn’t require any change to the DB. Now, let’s say that a taggable object (our blog post) has changed, and we want to update the tags accordingly. We might want to erase all the mappings we created for this object. No problem, also easy to do.

What about manual changes? Should these be erased as well? Probably not, at least not without alerting their creator. So we need to record the source of these mappings in an extra column of the mapping table, and use it to mark manually and algorithmically generated mappings differently.

How about deletions? What if the first time around, the algorithm made a mistake, and added a wrong tag, which was manually removed? Running the algorithm again will cause the tag to be added again. We need some way to mark “negative tags” , which are also pieces of information. The easiest way I found of doing this is adding a boolean “valid” column to the mapping table.

It’s important to note that this also applies to all mapping types and not just to many-to-many. So even when you don’t naturally need a separate table for a mapping, you should consider adding one, if the mapping is part of the actual data you keep. Also, if you need to keep extra data about the mapping itself, for example “relationship type” in a social network or “tag weight” as in our example, you would already have a separate table anyway.

I encountered this issue when I implemented my multiple source db-design. A reminder: I had data collected from various sources, and then combined together to a final merged record. The combining was done automatically.
My mistake was that I only considered the records as pieces of data, and didn’t consider that the actual grouping of raw data records is also part of the information I keep. As such, I should have represented the groupings in a separate table, with the added columns, as I outlined in this blog post.

While I was still making up my mind, I discussed the problem with a couple of friends, and to better describe the problem and the proposed solutions I wrote up a short document describing them. I decided to publish this document along with my choice and considerations. Maybe someone else will benefit from my choice, or at least from the alternatives I listed.

Problem description:
We want to to have a table with collected information from various sources.

For example, let’s say we want to collect information about paintings. We’d want to have a database holding for each painting we know about its dimensions, painter, description, link to an image file, etc. Since we collect this information from various sources (maybe harvest information from multiple websites), we would like our application to display each field either from all sources, or from the best source available.
(Note: in my original formulation, being able to display the value from the best source was enough).

Let’s say our desired, “regular” table looks like this:

{
field_a
field_b
field_c
}

Proposed Solution 1:
Add a source column, like so:

{
field_a
field_b
field_c
source
}

Keep multiple records with different source, combine on query time.

Downside:
combine on query time, really not good.

Proposed solution 1*:
final table
{
field_a
field_b
field_c
}

source table
{
field_a
field_b
field_c
source
}
Application only queries the final table, while a “source table” with a source column is added. Now combination is done during harvest time.

Downsides:
Two separate almost identical tables. Not good for DRY karma.

Proposed Solution 1**:

{
field_a
field_b
field_c
source
}

Same as solution 1, except now a null source indicates a combined (“final”) record. Record combination is done during harvest time, while the application queries this table for records with a null source.

Downsides:
Complicates queries a bit, keeps final data with raw data.

Proposed Solution 2:
{
field_a
field_a_source
field_b
field_b_source
field_c
field_c_source
}

Downsides:
Violates DRY, adds a lot of boilerplate code for handling each column. Doesn’t easily allow keeping unused data from different sources. For example, if source X was used for field_a, source Y’s field_a is discarded.
Also, doesn’t cleanly allow the final field to be a combination of sources, for example, concatenation.

Proposed solution 3:
Add another table, holding sources:

final table
{
field_a
field_b
field_c
}

source table
{
field_name
field_value
field_source
}

Queries are made against the final table, and record combination is done during harvest time.

Downsides:
Implements a database in a database.
For further reading on this solution see Entity-Attribute-Value Model, and the Inner Platform Effect.

My Choice
I decided to use solution 1**. For some time I thought I should implement solution 3, but decided against it, mostly because of the inner platform effect, but also because of the other downsides listed (see references).
I also added another column to solution 1, one that indicates for a given “source record” (a record with a non-null source) which is its combined record. Now, given a final record, I can also query for all its source records, which is desired for late data processing.
I work with this solution in the following manner:
Offline:
1. Harvest from some source.
2. Harvest from some other source.
3. Run record combination algorithm.
4. Run data analysis algorithm.
5. Harvest from yet another source.
6. Run record combination algorithm.
7. Run data analysis algorithm.
8. etc…

Online:
Query only from records with a null source.

For further reading see also Database Anti Patterns.

Consider the following situation: you are developing some big rss reader/aggregator and you want to allow users to specify handling rules. How would you keep these rules in memory, and how would you keep them on disk?
Obviously, this problem was solved before. Just consider email filters, or even packet filters in ethereal.

One way of approaching the problem is implementing simple predicate templates:
“%field contains %s” where field is subject, or body, etc.
Once that is accomplished, you can specify that a “filter” is some combination (for example logical and, or logical or) of multiple predicates. To store this, we’ll have an actual predicate table (or pickle) with their data, and a one-to-many mapping of filters to predicates.

Another option is allowing just some very simple predicates, and a filter will just “point” to (have an id/name of) the required predicate, and the required data. In this option, all data is stored with the filter.

A more complicated solution is to implement some logical serialize-able lanugage (such as the expression trees I used for diStorm or PyKoan). Using this language, the predicates can be very dynamic, and be combined and manipulated programmatically. This solution might be overkill for many projects though.

An interesting issue regarding handling of predicates, is their application to constraint solving. However, this is an issue for a future post. Suffice it to say, that when writing PyKoan I’m using a constraint solver. Since I’m representing predicates with expression trees, the ability to analyze and manipulate predicates is very handy.

Besides looking at existing solutions, I’m very curious to hear other peoples’ opinions. Feel free to write about your preferred solution in the comments.

Holding the state of registers is a bit involved by the fact that registers may overlap, as I mentioned in a previous article. To handle this, the class keeping track of the state information, upon seeing a request to change the value of a register, propagates this change to its parent and child registers.

Holding memory might be different though. At a first glance, it would seem that memory should be kept in some buffer. However, it is much easier to keep the memory in a python dict as well, and treat that dict as if it was a sparse array. This implementation allows programs to write at address 10, and at address 10000, without requiring the VM to keep track of all the addresses in between. (Of course, we are not going to implement paging just now :)

I really liked this idea of treating dicts as sparse lists. In fact, it could work even better with some tree data structure instead of a hash table. With a tree data structure your keys would be sorted, and you could do slices.
I went ahead and tried using tree data structures for this, but it seems to be more trouble than it’s worth, so I think that unless it becomes a real timing issue, I’ll let it go.

I did have some fun wrapping the memory in some class to abstract away the data structure, and to give it a few more capabilities:

In [2]: import vm
 
In [3]: m = vm.VMMemory()
 
In [4]: m.set(0, "hello world!\r\n")
 
In [5]: m.set(51, "foobar, foobar\x00")
 
In [6]: m.set(77, "a"*30)
 
In [7]: print m
0000: 68656c6c6f20776f726c64210d0a????  hello world!..??
 
0033: 666f6f6261722c20666f6f62617200??  foobar, foobar.?
 
004d: 61616161616161616161616161616161  aaaaaaaaaaaaaaaa
005d: 6161616161616161616161616161????  aaaaaaaaaaaaaa??

So how to solve the challenge? Here are the two possible solutions I thought about:

• Use a dict for lookup, and each element’s age is indicated by its position in a linked list. This is the solution Erez and I implemented.
• Keep a ‘last time of use’ indicator for each element. This could be just a regular int, incremented by 1 for each lookup. Keep the elements in a min heap, and when there are too many elements, pop them using the minimum heap.

Generally, I consider the first solution more elegant. It doesn’t rely on an integer to work, so it could work ‘indefinitely’. Of course, the second solution can be also made to work indefinitely, with some upkeep from time to time. (The added time cost of the upkeep may be amortized over other actions.)

If you can think of some other, more elegant solution, I’ll be happy to hear about it.

So, given that a linked list solution is more elegant, we come to the crux of the problem: what to do in Python? The Python standard library does not contain a linked list implementation as far as I know. As a result, Python programmers are encouraged to use the list type, which is an array. This is just as well: for most intents and purposes, the list type is good enough.

I tried to think a little about other cases where a linked list was more appropriate, and I didn’t come up with any more such cases. If you come up with any such case, I’ll be happy to hear about it.

After looking for a public implementation, and not finding one that seemed good enough, I decided to go ahead and write my own.

Out of curiousity, I also did a small comparison of runtime speeds between my implementation of a linked list, and the list data type. I tried a test where a linked list has an obvious advantage (complexity wise)- removing elements from the middle of the list. The Python list won up to somewhere in the thousands of elements. (Of course, list is implemented in C, and mine is in pure Python).

What is my conclusion from all of this? The same as the conventional wisdom: use the list data type almost always. If you find yourself in need of a linked list, think long and hard (well, not too long) about your problem and solution. There’s a good chance that either you can use the built-in list with an equivalent solution, or that a regular list will still be faster, for most cases. Of course, if you see no other way – do what you think is best.

Your thoughts?

My interest was first piqued when I encountered the following annoyance – some websites would use transparent layers to prevent you from:

1. Marking and copying text.
2. Left-clicking on anything, including:
   1. images, to save them,
   2. just the website, to view its source -
3. and so on and so forth…

Now I bet most intelligent readers would know how to pass these minor hurdles – but mostly just taking the steps is usually deterrent enough to prevent the next lazy guy from doing anything. So I was thinking, why not write a browser – or just a Firefox plugin, that will allow us to view just the top-level of any website?

This should be easy enough to do, but if it bothered enough sites (which it probably won’t), and they fought back, there would be a pretty standard escalation war. However, since the issue is not that major, I suspect it wouldn’t matter much.

Now comes the more interesting part. Unlike preventing someone from copying text, html (plus any ‘sub-languages’ it may use) may be used to display one thing, and to be read like a different thing altogether. The most common example is with spam – displaying image spam instead of text. When that was countered by spam filters, animated gif files were used. Now you have it – your escalation war, par excellence. This property of html was also used by honeypots to filter comment-spam, as described in securiteam. In this the securiteam blog post by Aviram, the beginning of another escalation war is described. There are many more examples of this property of html.

All of these examples come from html’s basic ability to specify what do display, and being able to seem to display completely different things. There are actually two parsers at work here – one is the ‘filter’ – its goal is to filter out some ‘bad’ html, and the other is a bit more complicated – it is the person reading the browser’s output (it may be considered to be the ‘browser + person’ parser) . These two parsers operate on completely different levels of html. Now, I would like to point out that having to parsers reading the same language is a common insecurity pattern. HTML has a huge space between what is expressible, and what is visible. In that space – danger lies.

As another, simpler example, consider phishing sites. These are common enough nowadays. How does your browser decide if the site you are looking at is actually a phishing site? Among other things – reading the code behind the site. However, this code can point to something completely different then what is being displayed. In this ‘invisible’ space – any misleading code can live. In that way, the spammer may pretend to be a legitimate site for the filter, but your run-of-the-mill phishing site for the human viewer. This misleading code in the ‘invisible space’ may be used to good – like a honeypot against some comment-spammer, or it may be used for different purposes – by the spammer himself.

Now comes the interesting part. The “what to do part”. For now let me just describe it theoretically, and later work on its practicality. I suggest using a ‘visibility browser’. This browser will use some popular browser (Internet Explorer, Firefox, Safari, Opera, etc.. ) as its lower level. This lower level browser will render the website to some buffer, instead of the screen. Now, our ‘visibility browser’ will OCR all of the visible rendered data, and restructure it as valid HTML. This ‘purified’ html may now be used to filter any ‘bad’ sites – whichever criterion you would like to use for ‘bad’.

I know, I know, this is not practical, it is computationally intensive etc etc… However, it does present a method to close down that nagging ‘space’, this place between readability and visibility, where bad code lies. I also know that the ‘visible browser’ itself may be targeted, and probably quite easily. Those attacks will have to rely on implementation faults of the software, or some other flaw, as yet un-thought-of. We all know there will always be bugs. But it seems to me that the ‘visibility browser’ does close, or at least cover for a time, one nagging design flaw.

After getting some experience with handling problems and writing applications that continue to work despite exceptions, I’ve come to the conclusion that the best practice approach is that each of the project’s module must raise its own type of exception. That way, in higher level modules that catch the exception you can set a concrete policy about ‘what to do’.

An example: I have some low level module named “worker”. Let’s say this module’s job is to copy files around efficiently. The higher level module, “manager” decides on ‘policy’ about what to do with files (copy them, delete them, etc…). Now, the “worker” module raised an exception. If this exception is an IndexError, the higher level module can’t really decide – was this because of a programming error, or an OS error? Even if “worker” did define an interface that allowed for throwing IndexError’s in some cases, if there is a real programming error in the module, it will be missed. The correct way to go about it is to create an exception hierarchy for each module, which will be the ‘communication channel’ for errors. This way, any unexpected programming error will percolate up as such, and not get missed – and any exception handling mechanism on the way can decide what do to about it (log, die, ignore, etc…).

Every exception handling rule has exceptions: when writing ‘low-level’ library modules yourself, such as a dict-like class, it makes sense to use IndexError. Don’t get confused with other library modules though – If you are writing some communication library, it should follow the same rules described above for a module.