There are limestone caves at Yarrangobilly, just off the Snowy Mountains Highway a bit past the Selwyn/Cabramurra turnoff (travelling from Cooma) and at Buchan Caves in Gippsland.
They are both lovely sets of caves, with the usual collection of stalactites, stalagmites, shawls, flowstone and of course the slightly twisted helictites (my favorite). Both sets of caves have guided tours and Yarrangobilly has an unguided cave. Both probably have wild caves that are accessible to speleologists, but check this before you mount an expotition.
Yarrangobilly has wild caves, as with all wild caves in National Parks in NSW, except Bungonia, a permit is required from NSW National Parks and Wildlife Service before entering. These usually need to be obtained beforehand and can involve a notice period and a fee. Some caves can be closed during bat breeding season or for the protection of speleotherms. Entry is on condition that the Australian Speleological Federation code of ethics is observed. It is recomended that caving is done as a member of an AFS affiliated club.
Yarrangobilly has a thermal pool that stays in the low 20s all the time. There are rumours of skinny dipping by X-C skiers in the depths of winter and of nudie skis into the pool, but lentil munchers would never do anything that interesting. The bad news is that the pool is at the bottom of a hill that induces sufficient sweat in summer that the only rational response at the top is to turn around and have another swim. And so it goes.
Buchan has an idyllic camping area (at least it seems idyllic at a distance of 4 decades - I suppose the fact that I still remember it says something).