I think the bit that we're in danger of forgetting is that BinaryFormatter is designed to be safe and to be version independent.

Being fast or space-efficient are really quite far down the list of features - and changing the format now runs the risk of not being able to deserialize objects that were serialized using an older version of BinaryFormatter.