Kataragame is along with Adam's Peak the most important religious pilgrimage site in Sri Lanka and is a holy place for Buddhists, Muslims and Hindus alike. It's about 1,5 h away by car and worth the trip since it's much less touristic than the temples within the cultural triangle and the influence of the different religions makes it very special. You should plan on one day to go there and return. You can also go there if you are planing to visit the Yala national park (but you can hadly do this on the same day).