Git, you said
"a data file protected by RSA, RC6 and MD5 checks."
so do you mean that these numbers could be as a result of one of strong crypto algo?
you need to figure out what algo used exactly (if you lucky - you cuold monitor the Win API, is it windows app?
you could for example sniff the crypto api from ms...... etc
without this knowledge your teaser = find black cat in black room