Here is an excellent thread on how to replace the spark plug tube seals. I used this technique on our '98 Sienna which I believe has the same 1MZ-FE engine as your Avalon. If the seals have never been replaced, or if they were replaced and installed incorrectly, they will leak, allowing oil to seep around the seal and
down into the spark plug tube. In my case, I also had oil seeping down the top of the valve cover. I had replaced the spark plug tube seals previously and had done a lousy job of putting the new ones in. It's been 15,000 miles now since I used the technique in the link below, and no leak whatsoever.
DIY: 1MZ V6 spark plug tube seals replacement 1MZFE
Regarding the smoke at start-up, how many miles are on your engine? When was the last time the PCV valve was replaced?