Writing a VM capable of executing expression trees is different from writing a VM for executing assembly instructions. Here I’ll cover several issues stemming from this difference.
The first group of issues involve generality. Supporting a specific instruction set is a straightforward job, even if a hard one. Vial has to support multiple platforms and that’s a little tricky. These are just a few of the problems:
- Different registers for each instruction set. This one is easy enough to solve. Just have a node in the expression tree with some value to indicate the register.
- Register overlaps. A change to one register implies a change to its parents and its children. Consider RAX->EAX->AX->AH,AL. Changing EAX will affect all of the registers in the hierarchy.
To handle this, we wrote a CPU class to keep all the info about the platform, including register overlaps.
- Delayed branches. Some platforms have branch delay slots. This means that after any branch instruction, right before the branch is taken, instructions in the delayed slots are executed anyway. For instance, SPARC has three delay slots, while MIPS has just one. This isn’t an easy issue to solve, and for now we didn’t tackle it. We’ve got a few ideas though.
To make sure that our implementation is generic enough, we decided to write a skeleton disassembler implementation for MIPS as well.
The second group of issues involve the nature of expression trees versus instructions:
- Stepping over statements or instructions? Each expression tree for an instruction usually holds more than one statement. For example, dec eax changes eax as well as the zero flag. Since some instructions like rep stosd may contain a long loop, being able to step over statements instead of expressions is preferable.
The problem is, executing expression trees is done with a DFS-like walk. If implemented with recursion it makes pausing the walk for each step a bit complicated. However, recursion is the clearest way to write the execution code, and I’d rather not give it up.
My solution was to use Python generators. Each time a statement was executed, the function would yield, thus returning control to the calling function, while keeping its state intact.
- Instruction Pointer changes. The lower-level disassembler returns expression trees for each instruction. However, it does not return the jump to the next instruction as well. While this is the required behavior, it means that the VM should change the instruction pointer after executing each instruction.
This is easier said than done: What should we do after jumps? Several possible solutions emerged.
The first was to append an IP change to each instruction’s expression. Those changes will have to be appended only where needed.
A second solution was to check if the IP was changed, and if it was not, change it. This solution however will not support instructions that jump to themselves.
The last and our preferred solution was to check if the IP was touched. This solution is simple and very straightforward.
There are many more issues I didn’t write about, for example, self modifying code. I’ll leave those issues for future articles.