Connecting USB-to-RJ45 2.5/5Gbps dongle to NAS542

Is this possible to get worked USB-to-RJ45 5Gbps dongle, connected to NAS542 through rear USB slot?

Probably not. In Linux all drivers are distributed in the kernel source tree. When I'm not mistaken that box runs a 3.2 kernel, which means it won't support any hardware which needs newer drivers than available in 2012. I don't think that dongle already existed, back then.

Further not all available drivers were compiled, just them needed to let the box do it's designed job. I have compiled some extra modules using the firmware GPL sources, when I wanted to connect an USB sound card, which you can find here. I didn't test most of the modules, of course.

If not, is there a possibility to overcome 1Gbps LAN speed at NAS542?

Yes, there is. You can trunk the 2 ports, and connect them to a managed switch which also supports trunking. This will give you a 2Gbps connection. Per stream you have only 1Gbps available, but you could have 2 1Gbps streams simultaneously. That's how trunking works.