Yeah, the first thing you need to do is gather as much information about the target as you can. Check the version tab of the file's properties first. Usually that will tell you what installer is being used. If that doesn't work, try looking at it with a hex editor or loading it into Olly or IDA. Check for string references from there that might tell you what installer they used.

It's always possible that they used some homemade installer, but the much greater possibility is that they used a common installer like Installshield, Vbox, WISE, etc. Once you know what installer is used, I'm sure you can find tutorials and/or decompilers and/or password crackers for that installer. I mean, you could just start trying to reverse it like any other program. And I'm not saying that isn't a worthy pursuit if you just want to figure out how to reverse it without using any tuts or tools. But if your goal is just to get the installer open so you can work on the payload, there may not be a lot of point in reinventing the wheel.

At any rate, it's really hard for anyone else to give you meaningful advice if they don't know what installer you're working on.