Exetools - Highly self modifying code

Greetings,

Is anyone aware of any resources regarding highly self-modifying code? I mean code that implements something like a cryptography algorithm but in an innovative and clever way where the self-modifications are vital to the algorithm and non-trivial. As such, flattening it would require unwrapping it using directed acyclic graph and not the more simpler temporal methods, as it would not depend on time but also how the program had modified itself previously based on input.

It would be interesting to see a practical example of this, if its ever been successfully done. It could be processor e.g. x86 centric. It is interesting to me that we have not seen models of how to look at the processor opcodes in terms of generating self-modifying algorithms. Compilers have not really focused much on it beyond overlays in the old days which is generally trivial to unwrap. I suppose the data-flow and code-flow analysis required would be very sophisticated. But as protection and anti-protection mechanisms advance, I predict this field of study will inevitably gain a lot of information at some point.