This article covers method to resolve Unable to connect to VNC Server using your chosen security setting.
Basically, the unable to connect error could trigger due to incompatible encryption settings or due to the version of the VNC running in the remote server.
To fix this VNC error, simply apply the following:
1. On the remote computer, change the VNC Server Encryption parameter to something other than AlwaysOff.
2. Change the VNC Viewer Encryption parameter to Server, PreferOn or PreferOff.