I got a factory service manual and tried to properly remove it and trust me -- it's a pain in the ass. I tried it for about 20 minutes with absolutely no result.
At the end - I followed POLO708's instruction -- just cut out the factory speaker, and then cut out the 5 plastic spokes that were holding the speaker magnet. Then I put my speaker and attached the screws - voila! 5 minutes work with 100% success
Thanks to POLO708 once again
Next project will be to replace the rear speakers... By the way, are the rear speakers similarly crappy like the front speakers also?