Find the secret key for this function

[UniSoft]

Originally Posted by Syoma

RSA powmod primitive is
c = m^e mod N, where c is ciphertext, m - message, e - public exponent, N - modulus
m = c^d mod N, where d is private exponent

our case is
0x90de757572b51cd3 = key^7 mod (max_uint64+1)

This is not the RSA !!!
In RSA N = p * q, where p and q are prime numbers!!!
max_uint64+1 = 0x10000000000000000 is not the product of two prime numbers, therefore the equation m = (m^e mod N)^d mod N will not work (mainly because of the large number of collisions).