Authorization to allow a Horizon Client user to access the virtual desktop and remote application directly is controlled within a local group called View Agent Direct-Connection Users.
This local group is created and contains the Authenticated Users group when the plug-in is first installed. This means that anyone who is successfully authenticated by the plug-in is authorized to access the desktop.
If you want to restrict access, then modify the membership of this group to specify a list of users and user groups. These can be local or domain users and user groups. If a Horizon Client user is not specified in this group, the user will get a message after authentication saying that they are not entitled to access this desktop.