Programming Python

Must Have Python Packages

Python 2.6 and 3.0 are upon us.
I helped a friend set up the new Python environment for research and development, and we thought about constructing a PythonPack: a compilation of packages every home should have lying around.

I decided I’d write down my list of generic Python packages: packages you’d want to have regardless of what you were working on. So I didn’t include any of the numerous web-development platforms, or some other very specialized packages.

What you won’t see in this list however, are IDE’s and editors. I don’t feel like joining in on the religious wars just yet :)

So without further ado, here’s my list:

  • IPython – I just can’t stress enough how useful this tool is. If you ever worked with Python’s interactive prompt, this is the same thing, just on steroids. It has completion, history, shell capabilities, you name it. If you didn’t yet work interactively, you won’t do badly to start with IPython.
  • ActiveState Python, or the win32 python extensions – if you’re developing on win32, it will allow you to get at the native win32 API, which I usually found as a blessing
  • PyGame – If you want to start game development on Python, that’s the tool for you. Even if you don’t, I found out I’ve used PyGame many times just for quick and dirty visualizations. Besides, it makes for a nice toy :)
  • numpy – Ever needed to do any computation that’s a bit more complicated than 1/(1-math.sin(x))**2? numpy is the way to go, with plenty of tools for vectors, matrices, and what not.
  • After mentioning numpy, it’s important to mention scipy and pylab (matplotlib). Just a bit of visualization strength, in the vein of matlab.
  • – Python code coverage by Ned Batchelder. An essential tool for effective testing. While you’re there I’ll also recommend checking out his blog.
  • ply – for most of your text parsing needs. Of course, this is just my preference. I heard about (but didn’t use) PyParsing and other modules
  • construct – If you intend to do some binary parsing, construct is an excellent tool for that.
  • PyOpenGL – I use it mostly for 3D visualizations, see for example my recursive spring.
  • PIL – Python Imaging Library is a very useful tool for working with images. I used it in my twindragon fractal implementation
  • nltk – Natural Language Toolkit – I realize most people won’t be using this one, but it’s generic enough, and I’ve used it enough times so far to include it in here. For example, I used it when I wanted to find rhymes.
  • WinPDB – A Python debugger especially useful for embedded Python apps (Python running from within another application).
  • nose – the framework I use for testing. Easy to start using it if you’re already using unittest.
  • psyco – if it so happens that you start to care about speed, psyco is by far the easiest way to go about doing it, short of writing your code in a different language.

Other useful but less often used packages include:

  • pefile – for parsing win32 pe files.
  • PyDBG (from within PaiMei) – an interace to the win32 debugging api (there’s an upcoming post about that one :)
  • scapy – I didn’t use it lately, but if you want to sniff packets programmatically, that’s a good way to go about doing that
  • Peach – Easy Python tool for fuzzing. Didn’t get the chance to use it myself though.
  • diStorm – the fastest disassembler for x86 and amd64 around, with a Python interface. Of course, it helps if I’m also working with gil on the next version :)
  • sqlalchemy and sqlobject – your handy ORMs. I use them mostly for web development. For other work I don’t often need a full blown db.
  • wx and PyGTK– I really don’t do that much GUI work, so I don’t have really much to say about those two. I still have them lying around though, just in case.
  • RPyC – Didn’t get a chance to use it myself, but if you want to get into distributed Python programming, I’ll look into RPyC first.
  • NetworkX – if you happen to need some graph algorithm or visualization, I’d go for networkx. Last time I used graphs I mostly wrote my own code, and discovered networkx only afterwards. Next time I’ll be sure to use it.
  • SymPy – Symbolic Python, in the vein of Maple and Mathematica. It allows you to do symbolic operations for example, integrating f(x)=sin(x) and still getting a function and not a number. I used it a little bit for my homework.

Now it’s your turn: what packages did I miss?

9 replies on “Must Have Python Packages”

I think that list is too long, there were a couple I hadn’t even heard of. My list would simply include:

* setuptools
* virtualenv
* ipython

Nose and coverage would probably be good general purpose things to add to that list.

Nice list. I thought of making one of my own, and this serves as a nice reference, because lots of the modules are common. A small difference is that I like the PyCrust shell that comes with wxPython more than IPython, and that I find wxPython essential since I develop GUIs from time to time.

It would be great if someone would package all these together into a mega-package, especially for PCs that are not connected to the net for one reason or another.

I actually didn’t yet use virtualenv, and I didn’t use setuptools for my own modules. When I did need to distribute something “properly”, I just used distutils. I still have easy_install around though, but I consider it an installation tool rather than a programming tool.

Originally before I wrote this post, my intention was to build such a meta-package. However, except for an organization to do it internally, it probably won’t be worth the effort. First, because you’ll have to construct different meta-packages for each platform (Linux, Windows 32, Windows 64, etc, also for different Python versions). Then, you’ll have to keep updating it, probably at least once a month, whenever one module changes.
However, the list serves almost the same purpose, works just fine, and will probably continue working for months.


I read you blog and always like what you write

I found the other day ProtoBuf Google’s data serializer which has a python interface. It might belong on the list.


Thanks! Regarding ProtoBuf, I never used it, so I can’t really say if it belongs on this list.

Actually, I know about cheetah (but didn’t use it myself). I didn’t put it in because:
1. It’s mostly specific to web development.
2. Even in web development, I’m not really sure it would be my template engine of choice. Right now there are plenty of template engines, (kid, genshi, Django’s to name a few), so I didn’t think it right to put it in.
3. For other common uses I usually just hack something simple using Python’s built-in capabilities. For example, I once wrote a preprocessor for some language.
4. For code generation I had a bit of a good experience with cog.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.