Edit the .htaccess file located at the root of your site and copy this code at the end:

# BEGIN Protect
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_REFERER} !^https://www.mysite.com/.*$ [NC]
RewriteRule . - [F] [L]
# END Protect

If you don't have an .htaccess file, create it.

Replace "www.mysite.com" by your domain.

Your users and you will always have access to your files by your site but it will not be possible to download them by their URL.

If you do tests do not forget to clear the cache of your browser.

My knowledge is limited in this domain, suggestions from pros are welcome.

There are only 2 plugins for managing user access: "Advanced Access Manager" and "User Access Manager" (more oriented Group)