If you are a male headed to Sri Lanka, not to worry. You don't really have to adjust your wardrobe unless you only wear tank tops and shorts. That being said, men mostly wear collared shirts with pants/trousers or sarongs (!?) so t-shirts are a bit casual for day-to-day wear but no one is going to say anything because you're a guy. Also, if you are Western, you are likely going to be taller than the general population of Sri Lankan men. James seems to tower over everyone and he is about 6 feet tall. Therefore, people are unlikely to step to you for any reason unless you are being disrespectful to certain religious beliefs. Who would do that anyway!?
LOTS of ladies (I mean, everyone) has a much-needed accessory. What is it? An UMBRELLA. I was pretty taken aback by this because we're here in the dry season. It is literally 94F+ everyday with 89% humidity. [Diatribe: It seems even hotter because there is no consistent air conditioning in Sri Lanka and the common mode of transportation is the tuk tuk that is open-air. When you sit for 10 minutes at a red light or traffic jam you think you are going to die.] The reasons the ladies use the umbrellas are two fold: 1. It is very hot and it helps to keep the sun off of them while walking from one shop to the next. 2. They don't want their skin to darken. I had a hard time understanding that reason because in our culture tan skin is embraced. I prefer to keep the sun off of my skin because I'm old and I don't want skin problems so I love the idea of the umbrella! Here an umbrella is essential - especially when it's not raining.
Women wear saris (a common fashion here) and I also regularly see ladies wearing hijab and in other parts of Sri Lanka, somewhat rarely, burqas.