In this short article I'd like to focus on a weird way of how IDA decompiles floating point comparisons between two variables. On normal x86 architecture, the normal way of comparing two integer values involves executing the cmp instruction, and after that the following jump, depending on the circumstances and set bits inside EFLAGS register (more on that in this article, x86 Registers). However when it comes to floating-point values, the FPU (Floating Point Unit) comes in place with it's own x87 instruction set. More on the x87 register set in this article, x87 FPU.

The instruction that takes place when comparing two floating-point values is called, surprisingly, fcomp. When executed, the result is stored in a similar way, like when calling x86's cmp. In this case, the result of the operation is stored in three bits inside the status word, that is a part of the x87 FPU instruction set. These are called condition flags, and depending on the result of the operation, certain bit is set in one of these. The following figure shows the table with associated conditions and results.

The ST(0) register is also a part of the x87 FPU instruction set, and the value in this register is set through the fld instruction. After the value to ST(0) is loaded, the comparison can begin. The fcomp instruction is executed with an argument described above as ARG. The following code snipped shows side-by side comparison of assembly code to C equivalent.

The value of val1 is loaded into ST(0), then the comparison is performed, and particular bits in the status word are set. This is where it becomes interesting. The next instruction called fnstsw loads up the status word into register ax (Half of EAX). This is just the contents of the status word that also contains C0 through out C3.

And then probably the most confusing stuff happens. An AND operation on the lower part of the EAX register. When we look at the following figure:

We can see that the bits 8, 9, 10 and 14 are the Condition bits described above (C0 through C3). These are the bits that indicate the state of the comparison, as described in the table below. For those that are familiar with the AND operation probably knows where this is going.

So why does and operation with such weird hexadecimal value even exists there, and why is it needed? Lets see this value in binary.

Let's compare these two bits that are set to the status word and see what happens. If you see the status word bits, the C0 and C3 are stored at these positions in our number 0x4100.

Remember what the assembly showed?

The status word is loaded into ax and the ax is then masked with value 0x4100 which we've just discussed. If we see which bits are masked, we can see that bits C3 and C0 have the value 1. If we look at our table, we can see that this exact combination is used when using the <= comparison.

And this the entire magic behind all of this. The magic value actually represents the bits to be set inside the status word.