Just as an FYI, there have been tons of people who have posted in various threads that they wanted to swap with someone to get rid of (or to get) one or the other. I don't think that I've seen anyone post that the swap was successful or if they had trouble with the electronics. Keep in mind that Toyota has sensors built into the seatbelts/seats to help ensure that the driver is informed about the seatbelt status of each passenger. There's no guarantee that you wouldn't get a bunch of error messages if you tried to do the swap... which is entirely possible because you'd either be removing a seatbelt from the system or adding one. Given how annoying that seatbelt alarm can be if it thinks that someone doesn't have a seatbelt on... I'd be hesitant to swap captains chairs for a bench (or vice versa) until I was 100% sure that it wouldn't cause electrical headaches.
Also, the XLE doesn't come with leather seating. The XLE has Softex seating, which is different than leather. I'd hate for you to arrange to meet with someone and potentially drive a great distance thinking that the material would match... only to find out that it doesn't.
Good luck!