Proper Permissions are Required to View Artifacts
The "raw" URL, which is available if you right-click and "copy link" on the artifacts list, does not allow unauthenticated access. So even if someone has this URL, they will not be able to access it without being logged in with permissions for your repo.
To access an artifact, the URL must contain a valid token. The only way to generate one of these authenticated URLs is to be logged in with the correct permissions, and click on the artifact required and copy the URL from the address bar of the browser.
This URL will allow anyone, authenticated or not, to access the artifact. However, these tokens are only valid for a short period of time, so after a while a new link would need to be generated.
CircleCI artifacts can also be accessed via the API, which requires an access token, and is the recommended way of accessing artifacts outside of the CircleCI UI