Binary​Formatter Slow and Bloaty

Just an interesting observation I made while having to serialize/deserialize a large List<> of structs: Using the default serializing functionality, the serialized data is 2.5 times larger than marshalling the data to a byte array, and also more than 30 times slower.

Let's say I have a struct like this:

public struct Sample
{
    public ushort Operation;
    public TimeSpan StartTime;
    public TimeSpan EndTime;
    public int Data1;
    public int Data2;
}

Then if I have a List<Sample> of 500,000 items and use the default BinaryFormatter functionality, the resulting file is 27MB and it takes 30.5 seconds to serialize, 20.8 seconds to deserialize.

Now when I have a custom serializer like this (in the container class, which I now mark with ISerializable)...

public unsafe void GetObjectData(SerializationInfo info, StreamingContext context)
{
    var sampleSize = Marshal.SizeOf(typeof(Sample));
    var size = sampleSize * m_samples.Count;
    var bytes = new byte[size];

    fixed (byte* ptr = bytes)
    {
        var intPtr = (IntPtr)ptr;
                    
        for (var idx = 0; idx < m_samples.Count; idx++)
        {
            Marshal.StructureToPtr(m_samples[idx], intPtr, false);
            intPtr += sampleSize;
        }
    }

    info.AddValue("Samples", bytes);
}



public unsafe ContainerClass(SerializationInfo info, StreamingContext context)
{
    var sampleSize = Marshal.SizeOf(typeof(Sample));
    var bytes = info.GetValue("Samples", typeof(byte[])) as byte[];
    var sampleCount = bytes.Length / sampleSize;

    m_samples = new List<Sample>(sampleCount);

    fixed (byte* ptr = bytes)
    {
        var intPtr = (IntPtr)ptr;
        var sampleType = typeof(Sample);

        for (var idx = 0; idx < sampleCount; idx++)
        {
            m_samples.Add((Sample)(Marshal.PtrToStructure(intPtr, sampleType)));
            intPtr += sampleSize;
        }
    }
}

... The resulting data is 2.5 times smaller and it takes less than 1 second to serialize or deserialze. Why this huge time difference?

Now I know that with the default serialization we get the ability to add/remove/re-arrange members in the structures without affecting the deserialization process (makes versioning control easier), but you'd think that if the serializer detects all value types it would at least create some sort of "template" in memory to map all entries (which all have to have the same layout).

Something else I noticed: If you do:

var samples = new List<Sample>(1000);

...it will serialize/deserialize samples.Capacity items, instead of samples.Count items. So even if you add just one item to that list, it will still serialize/deserialize 1000 items.

Just seems a bit slow and bloaty to me.

Specialized serialization is always going to outperform, by several factors, any generalized serialization. The built-in serializer has to take into considertion many factors that you're ignoring.

1. Handling polymorphism, which means every member has to include type information.
2. Circular graphs of objects.
3. Addition/removal/rearrangement of members.

I doubt it would be possible to optimize this to be any better than it is today, but even if there were ways to optimize further it could never approach what you've hand coded. This is precisely why the framework allows you to optimize the serialization. Most of the time you're going to want/need those features and won't be able to do much better, and in any case the built-in serialization will be good enough, but when you need to, you can roll specialized serialization logic to get that last ounce of performance/space.

The reason for the slowdown is that the binary formatter uses reflection to perform all of its internal serialization.

The reason for the bloat is that the binary formatter includes meta data to cope with the possibility that you're deserializing into a different type than you began with.

If you want to serialize stuff in a cross-platform way, quickly or with as little overhead as possible, don't use the .NET binary serializer.

Be careful about using Marshal.StructureToPtr - by using unsafe code you are putting your application at heightened risk of attack, for example if (sampleSize * m_samples.Count) > int.MaxValue, you'll end up with an overflow that might lead to bytes being unexpectedly small leading to a heap read-overflow in the loop.

It's called unsafe for a reason.

Well, I can think of at least one way it can be improved. Since this is a value type list, and you can't derive from value types, they will all be exactly the same. So what the formatter can do is store a layout description of the original value type, and then use the marshaller to basically do what I did. Then when deserializing, compare the new structure layout with the old layout and create a mapping function that will map old->new. It's not rocket science.

What possible reason is there to store metadata for each and every instance of the value type?

Also, the .Net framework does this type of stuff internally all the time so if this was done internally there would be no need to use "unsafe".

EDIT: Well that, and it should also only serialize List<>.Count items as opposed to List<>.Capacity items.

1 hour ago, BitFlipper wrote

What possible reason is there to store metadata for each and every instance of the value type?

In case you serialize a MyObject and try and deserialize it as a MyObject2. Or you save it as a MyObject in product version1, and during your upgrade to product version2 you swap, say, the order of two fields. By storing meta information about the class (obtained via reflection) the binary formatter can be reasonably sure that the object you're deserializing came from an object that looks the same as the object you're expecting.

Also, the .Net framework does this type of stuff internally all the time so if this was done internally there would be no need to use "unsafe".

I don't follow. How is it possible to do Marshal.PtrToStructure() without using unsafe* code?

* unsafe in the context of "uses pointers" rather than "uses the unsafe keyword in C#"

31 minutes ago, evildictait​or wrote

*snip*

In case you serialize a MyObject and try and deserialize it as a MyObject2. Or you save it as a MyObject in product version1, and during your upgrade to product version2 you swap, say, the order of two fields. By storing meta information about the class (obtained via reflection) the binary formatter can be reasonably sure that the object you're deserializing came from an object that looks the same as the object you're expecting.

You missed my explanation of the deserializer mapping the values to an updated type (it starts with "So what the formatter can do is store a layout description..."). I specifically address your comment about deserializing to a changed type.

It still doesn't explain why the data is so bloated, since for a value type array by definition should  all contain exactly the same types, so there is zero need to store per-instance metadata.

I don't follow. How is it possible to do Marshal.PtrToStructure() without using unsafe* code?

* unsafe in the context of "uses pointers" rather than "uses the unsafe keyword in C#"

What I am saying is that classes like BinaryFormatter etc already work with pointers internally, and many probably use the Marshal class, yet I don't need to specify "unsafe" when I use any of those classes. So if BinaryFormatter internally did what I did in my 1st post, I won't need to specify "unsafe" at all.

3 hours ago, wkempf wrote

Specialized serialization is always going to outperform, by several factors, any generalized serialization. The built-in serializer has to take into considertion many factors that you're ignoring.

1. Handling polymorphism, which means every member has to include type information.

Irrelevant for value types.

       2.  Circular graphs of objects.

Irrelevant for types that only contain value types.

      3.  Addition/removal/rearrangement of members.

So if BinaryFormatter did #1 and #2 for my value type array, it missed two obvious optimizations.

For #3, as I suggested it can track the layout of the original type and compare it to the type's new layout, and map the values accordingly. Why store per-instance metadata for value types that are all forced to be the exact same type?

My guess is that it treats ref types and value types the same way, hence the bloat and slow performance. The problem is you can't optimize it since it will break backwards compatibility.

You missed my explanation of the deserializer mapping the values to an updated type (it starts with "So what the formatter can do is store a layout description..."). I specifically address your comment about deserializing to a changed type.

Either that would require you to provide a Func<TNew, TOld> to the function for conversion (which doesn't work if the new version of the DLL actually doesn't have a TOld), or you run the risk of performing a partial conversion. In either case, you probably solve this by demarshalling the old structure and then performing your own conversion to the new structure, so you don't really gain anything with this implementation beyond an inline function that does something like

NewType t = ConvertOldToNew((OldType)BinaryFormatter.Deserialize(memStream));

It still doesn't explain why the data is so bloated, since for a value type array by definition should  all contain exactly the same types, so there is zero need to store per-instance metadata.

It's easier (and faster) to implement a serializer recursively. The serializer is designed to make a flat representation of your data. It makes no claims that it will store it as the most efficient representation. If you want guarrantees like that, either do the serialization yourself or compress the stream.

What I am saying is that classes like BinaryFormatter etc already work with pointers internally, and many probably use the Marshal class, yet I don't need to specify "unsafe" when I use any of those classes. So if BinaryFormatter internally did what I did in my 1st post, I won't need to specify "unsafe" at all.

They don't use pointers internally. They use reflection. Using pointers would run the risk of getting screwed over by field alignment differences between 32-bit and 64-bit .NET apps, and little-endian versus big-endian problems (since .NET is technically endian-agnostic). It also runs the risk of leaking information that is hidden in the padding between fields which might be usable in applications like Silverlight to attack the process.

If you want to perform a C-like serialization, use Marshal.StructureToPtr. BinaryFormatter is not a C-like serialization, and doesn't claim to be.

Have you tried Protobuf-net? It's a project Mark Gravell from StackExchange created.

http://code.google.com/p/protobuf-net/

@evildictaitor:

I'm not sure why such an elaborate system is required to map old->new types. At the beginning of the binary data, simply store the struct's field names, types and offsets. Then when deserializing, examine the new structure (it has to be able to instantiate it so it obviously has full access to it - yes, using reflection), and map the previous names, types and offsets to the new structure. Ignore fields that are missing in the new structure, set new fields in the new structure to its default values, and store the new values at the new byte offsets. If a field with the same name now has a different type (say float to double), try to see if there is a conversion available, and if not, use the default value.

I fail to see what information is missing from the above mechanism that would require me to provide something like Func<TNew, TOld>? Can you elaborate a bit on why it would be required?

They don't use pointers internally. They use reflection. Using pointers would run the risk of getting screwed over by field alignment differences between 32-bit and 64-bit .NET apps, and little-endian versus big-endian problems (since .NET is technically endian-agnostic). It also runs the risk of leaking information that is hidden in the padding between fields which might be usable in applications like Silverlight to attack the process.

Most of the data manipulation inside the .Net classes happens via native calls (for instance use ILSpy to look at Marshal.Copy, Array.Copy or even string.Compare), so why would this be any different? And why would I be forced to use "unsafe" in this particular case?

@ZippyV: Yes that looks interesting. However with the custom serializer/deserializer shown in the 1st post, I find it is fast enough and compact enough for my purposes. Also since this is for an internal debugging tool only, using unsafe code isn't really an issue.

@BitFlipper: Off a tangent.

Binary serialization is a double edged sword, regardless of how you do it. I have lost countless hours fixing problems with structs dumped via a pointer (you know, the poisonus _write/sizeof combo) that later fail for problems when the struct layout changes, or endianness, bitness or even just alignment. In general, I wouldn't recommend binary serialization to persist anything you can either not afford to lose or that needs to last longer than a very short time.

With that in mind, I have a hunch that 500K records don't qualify: a database - even something like SQL Server Compact - sounds a lot better IMHO.

What you are doing here will break if the field order changes, if memory alignment changes (e.g. on 64 bit; even more troublesome if the struct contains IntPtr values), if you try to load the data with different endian-ness. And it can't work at all if the struct contains even a single reference type member.

So the answer to your question of why the BinaryFormatter doesn't do that: it would be a specialization to optimize only one of the myriad of cases that the BinaryFormatter must be able to handle. Having an entirely separate code path from the other, more general, serialization code just to handle one edge case is a terrible idea, and should only be considered if you have plenty of evidence that a very large percentage of applications using the BinaryFormatter would benefit greatly from such an optimization, which I doubt is the case.

BinaryFormatter is more about convenience than performance anyway. If you want performance, you can write your own code, doing for instance what you're doing (although I would only do that for very short term storage because of the issues mentioned) or alternatively something like custom serialization using BinaryReader/Writer (where you control the formatting so you can ensure binary compatability across systems and application versions).

@Sven Groot: I'm not sure serializing structs that contain only value types is such an edge case as you make it out to be. You can find these structs all over the place.

Yes I know my simple implementation will break when the structure changes, but as I've said numerous times now, fixing that is trivial if you store the original structure's layout in the binary stream and then adjust for the new structure layout at deserialization time. It'll probably take me less than an afternoon to come up with something that can handle that properly.

This is such an easy optimization with such a big impact (and certainly not an edge case) that I'm surprised no-one implemented it in BinaryFormatter. The .Net framework needs all the optimizations it can get, so why skip on such low-hanging fruit for something (BinaryFormatter) that must be used quite often?

6 minutes ago, BitFlipper wrote

This is such an easy optimization with such a big impact (and certainly not an edge case) that I'm surprised no-one implemented it in BinaryFormatter. The .Net framework needs all the optimizations it can get, so why skip on such low-hanging fruit for something (BinaryFormatter) that must be used quite often?

Do you have numbers to prove that introducing a separate code path and more complicated parsing rules is worth it? What percentage of .Net applications are performing this operation in a performance-critical part of their application? Don't forget that you're potentially making the deserializer much more complicated, because not only do you probably want it to be able to read the data back fast on the original system, but if you read it on another system the deserializer must now be able to deal with adjusting for different memory layout, endian-ness, etc. And since this is dealing with external input and is used for various aspects of e.g. .Net Remoting and (I think) WCF, it needs to be very resilient and secure against malformed data too.

And then there's breaking the principle of least-surprise, because you're creating a situation reading data on some systems could be potentially much slower if it happens to use a different memory layout than the system on which the data was written.

If you can prove it's worth all those risks in general, post a bug report to the .Net team and they might listen. But I suspect they'll simply tell you that if performance is that important to you, you probably shouldn't be using the BinaryFormatter in the first place.

6 minutes ago, Sven Groot wrote

*snip*

And then there's breaking the principle of least-surprise, because you're creating a situation reading data on some systems could be potentially much slower if it happens to use a different memory layout than the system on which the data was written.

I'm not sure this can be any worse that what we have today. The current implementation also needs to deal with the issues you mention (hidden via reflection), so how will a specialized implementation be any worse? I think the "surprise" is that the current implementation is so slow.

If you can prove it's worth all those risks in general, post a bug report to the .Net team and they might listen. But I suspect they'll simply tell you that if performance is that important to you, you probably shouldn't be using the BinaryFormatter in the first place.

As I said previously, this can't be changed now simply because it will break backwards compatibility for BinaryFormatter. At this point it is simply an academic discussion.

But at least, can we fix it so that it serializes List<>.Count items instead of List<>.Capacity items? That won't break backwards compatibility, and someone please explain why that was ever a good idea at any point in time.

For the record, BinaryFormatter doesn't use Marshal.StructureToPtr to marshal non-linear structures because doing so introduces information leaks in structures that use padding between fields.

This is because structures and new'ed objects are only guaranteed to be cleared by the CLR at every readable location - i.e. in every variable slot. The CLR doesn't make any guarantee that memory between fields (i.e. in the padding of the structure) that the stack garbage there will be zeroed.

If you serialize this structure using Marshal.StructureToPtr, you're actually just doing a memcpy of the structure directly to the pointer - this is fine if you're sending that structure to a Pinvoke that knows to ignore those padding bytes, but is unacceptable if you're transmitting those bytes over the Internet or to an untrusted application, because those padding bytes might be providing an information leak to an attacker that would allow them to, say, remotely circumvent the ASLR in the process.

3 hours ago, BitFlipper wrote

It'll probably take me less than an afternoon to come up with something that can handle that properly.

This is such an easy optimization with such a big impact that I'm surprised no-one implemented it.

If it's so easy for you to do, and has such a big impact on your application, then implement it.

Often it requires you to actually implement something to realize it's not quite as trivial as you'd previously thought.

3 hours ago, BitFlipper wrote

*snip*

At this point it is simply an academic discussion.

But at least, can we fix it so that it serializes List<>.Count items instead of List<>.Capacity items? That won't break backwards compatibility, and someone please explain why that was ever a good idea at any point in time.

I would imagine due to a distaste for null list items... maybe for easier paging... who knows.