Caveats in advance on my own limited experience, but for my two cents I would offer the 2001 Z3 M coupe. I test drove one a few years ago, and the handling was phenomenal. The car stuck to turns like monkey crap flung against a wall and the structure felt rock solid.
Having never driven the E46 M3 (although I am working on a possible change to that this weekend), I cannot comment on whether its the king or not, but OT I think based on all reviews I have read the hands down champ for under 50K appears to be the Lotus Elise.
Sure would love to drive all three cars back to back to judge for myself