Two lower levels of ancient caves date back to the the Middle Ages and extend under the church.  The stairs were the only route to the Baptistery for this part of town in the 4th century.
 
Ancient caves
Cave 1
Cave 2
Cave 1