Any pointers on this troublesome algorithm?

[Syoma]

From the description you provide I can guess it is something asymmetric.
16 steps may be x^2 16 times, then step 17 +x. Totally 65537 prime.
Then full algo may be powmod(data, 0x10001, N), where N == magic?

void decode_1(unsigned int *src, unsigned int iter, unsigned int *dest) = bigint.mul
void decode_2(unsigned int *src, unsigned int *key) = bigint.sub