Difficult debugging situation
 

A GUI exe calls several console apps for transitory functions. Some of those exe's call A.dll and A.dll calls exports from B.dll. The exports from B.dll are wrappers around flexlm 9.2 library functions.

Any suggestions on how this can be debugged please? (with the aim of finding all the flexlm keys).


Git

(Admin _ i've put this here because I thought it could be edducational for other too. Please move to Requests if you think it more appropriate. )

In B.dll set "int 03h" in interesting for you place, and use JIT Debugger or SoftIce with "I3HERE ON". Don't foget to correct code (int 3 -> right code).
And of course h++p://www.woodmann.com/crackz/Flexlm.htm

Wouldnt
1) breakpointing CreateProcess in GUI exe
2) on every break
2.1) changing the flags to suspended and tracing over CreateProcess
2.2) attaching a separate debugger to the new process
2.3) breakpoint every export in A.dll
work?
I'm not sure if 2.2 will work, dont know if you can attach a debugger to a process that's created suspended.

i agree with the last post's either set the int3 in the places you need them, you should have a rough idea where the flexlm functions are located, then just let it crash and debug from there, or like the last person said you can trace the createprocess call's, either emulate them if they are doing (OldProc->CreateNewProc->KillOldProc) (NewProc->FlexLM), or just attach as after they are supended.

Its possible for you to msg me the target name and url ?
id be intrested to look :)

I'm persuing the INT3 approach which seem to be a good idea. Not quite ther yet though. Thanks for your help guys.

Git