I think you guys are giving the M3 too much credit. We all here have the "trained eye" to easily spot an M3 over a regular 3-Series, but the average fool would probably barely know the difference. The little differences of the M3 are HUGE to us because we understand how the small differences give the M3 such an aggressive appearance. But in the grand scheme of things, the M3's profile and looks are almost nothing like a sports car. It is squared off a lot and has a very boxy profile...it is not very low and long and I don't think it screams "sports car" at all, unless you know BMW's well.
If you put a Corvette and an M3 side-by-side, the Corvette is EASILY the definition of a sports car, whereas the M3 looks more like a luxury sports coupe.
Case in point: The E46 M3 is in fact a sleeper in the auto industry as a whole and I expect to the next-generation M3 to be a sleeper as well.