When do you see this? For me, I just go to System Settings → Privacy & Security.
Scroll down to Security and look for the message about the blocked app, click Allow Anyway, and then reopen the app.
If I install Xournall++[1], just opening the application will fail and MacOS will ask me to move it into trash. I am required to execute `xattr -c /Applications/Xournal++.app` to "remove quarantine".
I just reinstalled and can confirm that I don't see anything in the System Settings as you say.
