Given the rather poor reliability of the manual in these particular models, you're probably better off keeping the auto and taking care of it with a pan drop/ATF change and spending the time/money on other things.
It'd be a different story if you had two vehicles, one with a good motor and manual trans but bad body, and the other a blown/worn out motor with auto trans but good body, and were just swapping between the two. Also would be different if the manual were stronger/more reliable than the auto, as is the case with some other vehicles. In this case, actually taking the time to seek out a manual trans and doing the work to put it in would likely not be worth it.
You could always keep an eye out though, maybe one day a rear-ended 2005-2008 will show up when you have more time, money, and ability and can swap it out into what you have now.
I'd agree with the above as well too, you'd be better off just finding and buying something in good condition that already has a manual if you want a manual. Of course, it doesn't have to be a 9th gen or even a Corolla. Instead of doing a manual swap, you can spend the time and money getting and installing a new clutch.