Can i install the unifi controller on my zyxel nas542?..

Theoretically yes. I downloaded the Debian install package, and looked at the content. As far as I can see it doesn't contain any binaries. The software runs in Java. So you'll need to install a Java Virtual Machine, change some configuration files and scripts because you can't install it at it's default place, due to firmware collision.

In practice it might be simpler to install it in a chroot-ed Debian installation on your NAS. In that case the default paths can be used, and the package manager will automatically pull the dependencies. The only problem I see is the startscript. The controller is supposed to be started by systemd, which is problematic in chroot. So just write a shell script to start it.

Of course there might be problems I don't see at the moment.