Depends a lot on how professional you want it to look and how old the car is.
When you sand them down you're making it easier for the paint to stick and look natural, but you're also removing any caked on road debris.
Personally I would lightly sand it regardless, as I've seen these tiny jobs gone terribly wrong because someone didn't take 10 minutes to go the extra mile.
Sent from my Autoguide iPhone app