Any pointers on this troublesome algorithm?

[Cryo]

Originally Posted by Syoma

From the description you provide I can guess it is something asymmetric.
16 steps may be x^2 16 times, then step 17 +x. Totally 65537 prime.
Then full algo may be powmod(data, 0x10001, N), where N == magic?

void decode_1(unsigned int *src, unsigned int iter, unsigned int *dest) = bigint.mul
void decode_2(unsigned int *src, unsigned int *key) = bigint.sub

...Woah. I think that just kind of blew my mind. It makes a lot more sense, now that I look at it and step through it as a whole instead of as individual functions.

I manually calculated the return values in each step of the process using your advice, with the first 16 iterations being (x^2)%N where x is the data, and the last being (x*y)%N where x is the data and y is the original data, and sure enough, the result was exactly as expected!

Checking your proposed algorithm of (data ^ 0x10001) % N, it also yields the exact same results, which is kind of unreal to me at this point that it can be simplified so easily. Thank you so much!

You described it at such a base level and with such ease that I can't help but be impressed; for the sake of additional learning opportunities, how would you suggest that I build upon these kinds of skills? Would reversing crackme challenges be helpful, or would you recommend something else?