Whats the time between difficult starts? Within the same day? If thats true I'd say you have a pretty big drain problem. You commented on the battery testing OK, but knowing the age may be helpful. I've had batteries for only a year that would go bad.
And by go bad I mean a cracked cell or similar. It'll throw you for a loop, because it will test fine for a short time, but under a few seconds of load the batteries output capacity drops dramatically.
However, if your battery stays at 10 volts while cranking over the engine, I would say your problem is elsewhere. Possibly correded terminals on the battery or starter.
I had one battery, cheap one, charging voltage was 14V, engine off was 12.2 volts, but had a problem starting. Load test the battery, one time, two time, all fine. Go to test it a third time, voltage is appropiate at 10-11 volts, then all of a sudden drops to 7 volts. And it was 2 months out of warrenty, curses!
If you do replace your battery, do yourself a favor and opt for an AGM type. It'll handle drains better atleast.