It is the synchro!
I have the same problem with mine but on 4th. When I cruise around town and put the 4 one in, or accelerating for the highway from 3 to 4, no problem. But riding on the Highway wanting to pass someone I shift from 5 to 4 and it grinds a bit...
But there is a solution
You just have to do what people did when they didn't even have synchros!!!!!
... just Double Clutch.
If you do it right, it'll be smooth as new! And you won't be needing a new tranny in a year!
Damn these old Cam's are plenty of fun

William