I've done a lot of work on diesels, but I can't say I have ever heard of using engine oil to flush the sludge....
Do you have a bad problem with build up? If so I think a rebuild would be more the answer than attempting to flush it.
The only way I would imagine you could flush it would be to presure spray the oil through there. And that seems kind of expensive.
Either way, I say it's worth the work. If you have a diesel Camry, you own a pretty rare car. I would only dream of one