1. If we do not pick an idealized perfect circle for our geometry, then there are an infinitude of imperfect circles to choose from. What justifies preferring one imperfect circle over another? Any choice is equally arbitrary.
It is not even clear that the choice is possible at all: as I write in my book, "It is impossible to consider for selection every member of an infinite set."
2. If we manage to choose at random, then the geometry based on whatever imperfect circle we have picked is going to be more complex that one based on a perfect circle.
Idealized physics in which, say, the motion of a pendulum is circular, is also much simpler and neater and more tractable than a more "realistic" physics.
Finally, if I am manufacturing bolts (to be used in the construction of a bridge, say), then a bolt with a perfectly circular head will be both more in demand (that is, more useful to my customers) and cheaper to make. I have an incentive then to make the head as circular as I possibly can. Perfect circle then represents a more sensible ideal to aspire to.
Consider an ideal right angle. “The sum of the angles of a triangle is equal to two right angles.” The difference between a 90° angle and 89° angle is not “1 degree” but qualitative, for example, because a perpendicular line can be constructed with only a compass and straightedge, while a line at an 89° angle cannot be (I think). Perhaps, an ideal circle is similarly priviledged.