Not just that, but if you get a flat on one of the rear tires, and you put your smaller spare on and drive for a long time on it, it'll destroy your differential. Having two different tire diameters on the rear makes the two different wheels turn at different RPMs, and so the spider gears in the differential will constantly be turning when going straight (normally, when driving straight, the spider gears are stationary). This can overheat the differential (particularly the spider gears) due to the added constant friction and lead to excessive spider gear wear or worse. I imagine that in a car with a limited slip differential, this would cause tremendous problems - the differential "thinks" the smaller diameter wheel is slipping since it is rotating faster. As such, the clutches will try to equalize the speeds between the two wheels. However, they won't be able to equalize the speeds since the wheel is not actually slipping, and it'll burn out your limited slip clutches.
Shouldn't be a problem in a car with an open differential if it's just a short term thing. Don't drive across the country with mismatched tire diameters or continue driving on them for weeks after getting a flat. On the other hand, if you have a limited slip differential, I'd be very careful of it.