Thomas:
One interesting simple fact of reversing I learned the hard way, about five years before you were born
, is that packed and/or encrypted code will (fortunately) not run on a computer. At the time I was working on PACE protection on music software written for the MAC. The code on the disk, was checked to "validate" the software each time it was started. That code was encrypted and, just starting out in reversing, I had no clue what assembly language was or did.
Fortunately, I had a debugger which would at least allow me to "watch" the code executing as I spent hour after hour stepping through the code, watching and trying to understand what was happening. Eventually, it would reach a point where the debugger would just quit and I didn't know why or how this was made to happen. So off to the book store I went and bought a book on programing MACs in assembly language. After study this for a long time and watching more code go by, I eventually figured out
how it was breaking the debugger. I had already discovered "where" this was happening through stepping, learning if I stepped over a particular jump or subroutine, the debugger was screwed. Eventually, I figured "what" exactly it was doing to break the debugger and how to "fix" it so I could get past that point. Then I "owned" the software, even if I had not yet figured out how it actually worked.
At that time MACs worked quite differently from the way PCs do. Their code was kept in "code sections" and called only when needed, to reduce memory requirements. All the code sections of my program were encrypted, so all of them had to be "decrypted" before they could be executed by the machine and hence, the program. Eventually I discovered that the encryption was made by using a starting phrase (in this case BEEFABAD) in hex and adding in the hex of each byte in the code section and using that checksum as the starting phrase for the next section, etc, etc, until they reached the end. At the end they had a master checksum hex phrase which the program had to store somewhere to use to "decrypt" the code sections, which had been encrypted using this master checksum hex phrase.
Eventually, I figured out how all this worked and how the decryption process worked. When needed, the encrypted code section was called by the program and the program had hijacked the MAC code section loader to make the encrypted section run through the "decoder" each time it was called to be placed into memory. When it was no longer needed, it was simply "released" and, when needed again, it was loaded through the 'decrypter." Now I had the decryption code and eventually I figured how to run it myself from within the program and forced it to write the decrypted code back to my HD so I then had the decrypted code.
The point of this history lesson, is that the processor can not "execute" encrypted/packed code itself. It needs to decrypt/unpack the code, either entirely or partially or part by part to allow the processor to understand what the "real" opcode is and execute it. This means a "packer" has to unpack the "real" code at some point, if the program is going to actually run. Your job is to find the point where this is/has occurred and then "capture" the "real" code to play with. This also means that "at some point" all the individual tricks the protectors have attempted to make your journey as difficult and tedious as possible "have to fix themselves" or the damn thing isn't going to work as it was intended before the protection was applied.
The protectors job is to make that journey take so long and be so tedious you will give up before you reach your goal. Their best defense is to try to blow up or screw with your debugger or disassembler in some way that you can't see what they are doing to muck with the code. This being true, one of your first lines of "offense" is to try to understand as much as you can about methods of screwing with your "tools of the trade." If you can learn how to protect your tools, you can eventually figure out what they have done.
Even if, at the beginning, you do not completely understant what it was they did to wreck your tools, if you know "where" they did it, you can try to figure some way to get past it. For example, maybe you can bypass a branch or a jump which bombs the debugger. You quickly learn that if you execute "that" particular step you are toast. Try skipping it and see what happens. Screwed again? OK. Go back and single step through the routine and watch what happens and see where things crash your world. When you figure out "what", now you you can work on "why." The more time you spend learning how your tools work, the better prepared you will be to prevent the protection from making you blind. If you can see what they are doing, eventually you can figure out how to "fix' it.
Regards,