If ear pain most likely it is the window being down. This is caused by variations of air pressure (same as sound waves), but these waves are more extreme.

With the window down, a low pressure area is being created inside the car. The issue is similar to being in an airplane and having ear pain as the plane climbs and descends. It is a combination of extreme air pressure changes and noise. But, you can have the air pain without the noise as on the airplane.

Something like a wheel bearing would vibrate through the frame. Might also be some exhaust system and road noise.

Save your ears and leave the window up.

Also on some cars if the window is down exhaust fumes can be sucked into the car.

 

Toyomoho is correct, I have experienced this problem on different vehicles.... However, if you 'crack' at least one of the front windows while the rear are down, it should eliminate the pressure and your ride will be more comfortable.

 

Loud Noise

Yes, I have the same problem. With front windows up and both center windows down it gets very bad. I notice that below 40 mph it becomes bearable and approaching 70 mph it seems to increase very fast in intensity. Someone else who has a RAV4 wonders if it has something to do with the cargo space becoming a resonant chamber since the rear window cannot be opened. Other than opening the front windows or driving slow there appears to be no solution.

 

I read in another forum that window visors/deflectors can help. I have had this same problem in other vehicles but to a lesser degree. Not using windows seems like a bad solution and if this is the only answer it seems like a major flaw.