Well, a few weeks ago a decrypted version of whmcs is exposed, but as you can guess it is illegal to use it. I suggest you to stay with the proper licensed version.
Assuming you will continue with the latest whmcs version, I can say that what you can do is limited. You cannot change the core functionalities but you can easily expand it with hooks and custom modules.
You can change or create you own template.
You can create your own pages(http://docs.whmcs.com/Creating_Pages)
You can use hooks to trigger events within whmcs(http://docs.whmcs.com/Hooks).
You can also create you own modules(http://docs.whmcs.com/Addon_Module_Developer_Docs)
So, if you really need to be independent from the core of WHMCS, I suggest you use your custom software which can communicate WHMCS API only. Use the things you need WHMCS with API only, and if you need something which you cannot do in WHMCS, do that in you custom software. But in this case, you need more time and effort to code your software(probably with your own client interface)
top layer : Your Software(you can code whatever you need)
middle layer: WHMCS API
engine layer: WHMCS(you can use the functions of it)
And please be careful, WHMCS is exposed many security problems(sql injection and xss) . So I strongly suggest to use your own software(which you can set your own security), and allow to communicate it with WHMCS API. Don't allow any client's to reach whmcs client area.
