Cannot comment much on this one. Though a few obvious things that are not clear...
project.UI and I am not sure why your .wxs contains any dialogs at all. Are you building your own dialog sequence instead of relying on WixUI_InstallDir?
To be honest I found native MSI dialogs extremely limiting and abandoned any effort to improve developer experience for them. You just cannot make a racing horse out of a donkey. Thus it is only available as a "lab" effort.
It was the reason why I implemented ManagedUI where you have both full control and full power. And also you don't need to abandon your programming experience and learn something rather obsolete (MSI/WiX UI markup).
Have a look at ManagedUI may be it is what you need?