Trying to identify crypto algorithm

[SiNTAX]

I'm working on a target with crypto and am trying to identify the algorithm.. the target uses RC4 to encrypt communication. At the start it takes the random RC4 encryption key does some XORs with SHA256 hashes on it and then encrypts the entire shebang.

Most logical choice would be for this key exchange packet to be encrypted with public key encryption, although tracing through the code I have my doubts it's RSA.

For one.. it only takes a single 512 bit input key... pubkey encryption would need a private and public key.. so unless I missed some code somewhere, it seems doubtful it actually is public key encryption.

One of the first steps it does is take the unencrypted data block and shifts it into another memory block byte by byte, but the destination memory block only uses 28 bits per 32 bits (0x0FFFFFFF), so for example with input bytes:

01 02 03 04 05

the resulting destination block is (word display):

00030405 00000102

Lateron there's a step where 2 of these block get matrix multiplied.


I've looked at a couple of encryption algorithms but haven't really found one that seems to operate like this, then again.. I'm no crypto jock

Any clues?