You might experience builds that are pull requests not running with the error message "This project is configured to only run builds that have open pull requests associated with them. Update the 'Only build pull requests' setting to run this build."
This happens due to a race condition when a branch is created and a pull request is opened simultaneously. CircleCI will receive notification the branch was created, and will not run the build because it is not a pull request. We will then recieve a second notification of a pull request being opened, with the same commit hash. Since we will never run the same commit hash twice, the build is updated in the UI, but never ran.
To ensure this doesn't happen builds with "Only build pull requests" turned on, can disable the "Branch or tag creation" setting in Github. This will ensure the only webhooks we receive are when the pull request is open.
How to find your webhook settings: https://support.circleci.com/hc/en-us/articles/360021511153-How-to-view-your-GitHub-WebHook-deliveries
Find this option, and uncheck the box.