In 1960, the physicist Eugene Wigner wrote an essay called "The Unreasonable Effectiveness of Mathematics in the Natural Sciences" that physicists and philosophers still find deeply unsettling.
His observation: mathematicians develop mathematical structures purely for reasons of internal logical interest, following where the mathematics leads, with no particular physical application in mind. Then, decades or centuries later, physicists discover that these purely abstract mathematical structures describe physical reality with extraordinary precision.
The clearest example is non-Euclidean geometry. In the 19th century, mathematicians like Riemann developed alternative geometries, geometries in which space is curved rather than flat, purely as mathematical exercises. What would geometry look like if the parallel postulate were denied? Interesting mathematical structures emerged. No one thought these structures described real space. Then Einstein developed general relativity, and it turned out that physical space actually is curved in the way Riemann's mathematics described. The abstract mathematical structure, developed with no physical motivation, turned out to be the correct description of the universe.
Other examples abound. Complex numbers (involving the square root of -1) were invented as a mathematical trick for solving certain equations. Physicists found they were essential to quantum mechanics, you cannot even write down the Schrödinger equation without them. Group theory, developed as abstract algebra, turned out to be exactly the right framework for describing symmetries in particle physics. The list goes on, and it is genuinely remarkable.
Wigner's puzzle: why should this be? If mathematics is a purely human construction, a game we play with symbols, why does it describe physical reality so well? A purely invented game has no reason to match a mind-independent physical world. This seems to support some form of mathematical realism: mathematics keeps describing physical reality because mathematical structures are the deep structure of reality. The physical universe is, at some level, mathematical.
Max Tegmark has taken this to its logical extreme with the Mathematical Universe Hypothesis: the physical universe is not merely described by mathematics, it literally is a mathematical structure. There is no "physical substance" underlying the mathematics, the mathematics is all there is. This is a minority position, but it shows where the unreasonable effectiveness argument leads if you follow it consistently.
The anti-realist response: the fit between mathematics and physics is not as mysterious as it looks. We select the mathematical tools that fit, there is survivor bias in the examples Wigner cites. Enormous quantities of mathematics have no physical application at all. And the fit is never perfect, physical theories always require approximation, and the mathematics must be adjusted to match observations. The unreasonable effectiveness is partly a retrospective construction.
But the unease persists. When Paul Dirac derived the Dirac equation for the electron from purely mathematical symmetry requirements and found it predicted the existence of antimatter, before antimatter had ever been observed, it is hard not to feel that something profound is being touched.