The Materia Caves are four caves scattered throughout Gaea that each contain a different piece of Materia. Each of the four caves are identified in the map below, but you should probably refer to the Materia Caves side quest section using the link below for more information on how to reach each of them:
The Materia Caves contain the following items:
Quadra Magic Materia
Mime Materia
HP <-> MP Materia
Knights of the Round Materia
You can only reach the Materia Caves by completing part or all of the Chocobo Racing and Breeding side quest. Click on the link below for more information:
The map below shows the location for Materia Caves on the World Map: