An atheist, call him Smith, will sometimes argue, in at attempt to evade any burden of proof, that he simply lacks the belief that God exists.
In reply to that, Smith can be asked whether he also lacks the belief that God does not exist. If the answer is yes, then Smith is not an atheist at all but an agnostic: insofar as knowledge is justified true belief or something close to that, lack of a determinate belief as to whether God does or does not exist causes him to fail to know whether or not God exists which is precisely the dictionary definition of an agnostic.
If, however, he says that he does have the belief that God does not exist, then he needs reasons for this belief, just as a theist needs reasons for his beliefs.
I agree that it’s pretty unlikely that unicorns exist. But I wouldn’t be so bold as to assert with 100% certainty that unicorns definitely do not exist. New species are discovered all the time, and perhaps unicorns do thrive in some remote and still unexplored part of the world.
Thus, even if an atheist thinks that none of the theistic arguments for the existence of God are convincing, he would be foolish to deny that God may still exist despite that.
In other words, this is a situation in which absence of evidence is not evidence for absence. If Smith wants to make a positive case for God’s non-existence, then he needs to present precisely the evidence for absence of God.
And many atheists do just that, when they bring up the only real argument for God’s non-existence which is the problem of evil. The second argument, namely that God is nowhere to be found in the causal structure of the world, that the mode of First Cause cannot be viewed either as physical or teleological causation, also tends toward agnosticism, not atheism.
We can see that the starting positions of the atheist and theist are identical, and it is only true agnostics who can remain staunchly neutral.