There's no difference in the rendering engine of IE5.5 and IE6, as far as I know there were only security improvements (SSL and a lot of warning mesage boxes) added to the browser part.
Actually IE6 changed things massively from IE5, including massive changes in support for CSS2 features.
I understand "IE6" isn't an option because IE7 Compatibility Mode is essentially the same as IE6 sans serious layout bugs. These aren't mere differences in rendering that poorly made applications might rely upon, but actually broken rendering that was fixed in IE7 (they spent their time fixing this in IE7 rather than completing CSS2.1 support, that's what they did in IE8). If an application was written against IE6 it's going to work in IE7 compatibility mode.
But the inclusion of IE5 mode is interesting, I wonder if it's just a rebranding for IE6's own Quirks Mode, I don't believe Microsoft would actively put effort into reproducing IE5's rendering behaviour: IE5 was only out for just over a year and a bit before IE6 came out (not forgetting IE5.5 in the middle) and IE6 was widely accepted by enterprises.