Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

C9 Lectures: Stephan T Lavavej - Advanced STL, 3 of n

Download

Right click “Save as…”

Many thousands of you have watched Stephan T. Lavavej's great introductory series on the STL on Channel 9. If you haven't, then you should.

There are two STLs: the Standard Template Library and Stephan T. Lavavej Smiley You will get to know a lot about each STL over the course of these lectures.

Advanced STL covers the gory details of the STL's implementation -> you will therefore need to be versed in the basics of STL, competent in C++ (of course), and be able to pay attention! Stephan is a great teacher and we are so happy to have him on Channel 9—the only place you'll find this level of technical detail regarding the internals of the STL. There are no books. There are no websites. This is Stephan taking us into what is uncharted territory for most, even those with a more advanced STL skill set.

In this third part of the n-part series, STL digs into the _ITERATOR_DEBUG_LEVEL "New World Order," as he says, which powers the STL's comprehensive correctness checks. You will also learn about the history of _SECURE_SCL and _HAS_ITERATOR_DEBUGGING.

STL explains #pragma detect_mismatch as a defense against ODR violations (One Definition Rule).

As an extra treat, you will learn about the undocumented compiler option  /d1reportSingleClassLayout for looking at data structure representations. Well, now it's "documented" Smiley Thank you, STL!

[Advanced STL]

Part 1 (shared_ptr - type erasure)

Part 2 (equal()/copy() - algorithm optimizations)

Part 3 (_ITERATOR_DEBUG_LEVEL, #pragma detect_mismatch, and /d1reportSingleClassLayout)

Part 4 (rvalue references v2.1 and associative container mischief)

Part 5 (deduplicator, using Boost.Bimap/Filesystem/ScopeExit) - see Stephan's deduplicate.cpp

Part 6 (container pretty printer) - see Stephan's pretty_printer.cpp

Tags:

Follow the Discussion

  • new2STLnew2STL xkcd.com

    Nice coincidence, I was reading a short text from Giovanni Dicanio about SCL to gain performance but lost protection.

    Thanks again Stephan come with a razor sharp explanation, and plus insight of new IDL. The 'undocumented' feature that text out structure are very nice, my students (and I included) will like have fun visual info when I ask then to deal with command prompt only Big Smile

    Pardon my ignorance, but the source code still compare pointers to 0 (zero), its not the case to use nullptr?

  • felix9felix9 the cat that walked by itself

    /d1reportSingleClassLayout is super

  • STLSTL

    new2STL> Pardon my ignorance, but the source code still compare pointers to 0 (zero), its not the case to use nullptr?

    Because our C++ Standard Library implementation has to compile under native, /clr, and /clr:pure, and the "native nullptr" and "managed nullptr" haven't been unified yet, we'd have to use __nullptr.  (We define std::nullptr_t as decltype(__nullptr).)  We could also use std::nullptr_t(), which is a sneaky way to be portable.

    Aside from that, and having to coordinate our sources with Dinkumware's master sources, we could begin using nullptr, but we're incredibly busy and have much higher priority things to do!

  • Sunil JoshiSunil Joshi

    Can the linker detect ODR violations when you use whole program optimisation?

  • Eric AguiarHeavens​Revenge Know Thyself

    STL: To zoom into those dialogs you can press the Windows Key + "+" to zoom in on the screen using software zooming, it's a Win7 feature not many people use Tongue Out but could be useful if u remember it and even a few of the Windows key + # shortcuts in the future like opening that console after breakpoints as long as its <10 icons in the taskbar.

    OMG... /d1reportSingleClassLayout* is just awesome Big Smile I feel like a newb again looking at some of my whacky class datastructure diagrams from older code Big Smile That is officially awesome++.
    I also think that undname is pretty usefull too, sometimes mangled names arent so easy to decipher on 1st glance to me, especially looking at video convertters and filesystem structs where their... enormous. You should probly update it though for not needing the "> >" template space like at the end like what it used to require.

    Liked this video more than the others because "undocumented feature examples" are valuable when their this usefull, since I tend to always optimize beyond Debug level help and have always needed to just picture it in my brain which isnt so hard but using this just makes it easy enough a 4th grader could grok Tongue Out

  • Nice lecture [ though perhaps I've enjoyed the prior (more Standard C++) ones a bit more Wink ].

    BTW, /d1reportAllClassLayout is also pretty cool Smiley

  • petkepetke

    That linker pragma missmatch feature is very important. 
    Frankly I have find it a bit embarrasing that the C++ language standard says nothing about the linker. Version managing binary compatibility between libraries is unforgivably hard to do in C++. All this standarisation effort and type safety improvements that is done on the compiler, an none on the linker.
    I wish this linker missmatch feature was somehow automated even for user defined classes. Maybe if the linker could somehow automatically calculate a short checksum for every type (maybe based on  structures, size, or compiler switches) and then include that in its mangled name. That way missmatches of binary compability could be found at link time. Anyone see a reason why this is not possible or desired? 
     
     

  • STLSTL

    Sunil Joshi> Can the linker detect ODR violations when you use whole program optimisation?

    VC10's linker doesn't have that ability for native code.

    HeavensRevenge> To zoom into those dialogs you can press the Windows Key + "+" to zoom in on the screen using software zooming

    Thanks! I didn't know about that.

    petke> Version managing binary compatibility between libraries is unforgivably hard to do in C++.

    I've heard that C++0x's "inline namespaces" can help with library versioning, but I haven't looked at them closely.

  • XeoXeo

    Great video again STL! Finally I know why the names are Containerbase12 and Containerbase0! I like to lurk through the STL headers and was always confused with this proxy stuff, thanks for going into all those details. But is it just me or do you get less time every video? :P

  • kachchhukachchhu kachchhu

    this video size is too much large so listen to it thats very critical.

  • petke> STL> In theory, in the next platform / major binary breakage that Microsoft gets, they could change the name mangling scheme so that non-built in types include a CRC.  This could detect a lot of binary breakers pretty easily.  This would cause some amount of binary bloat (import and export tables would be larger), it wouldn't help for extern "C" interfaces, and it wouldn't help with COM like situations where clients just call non-exported virtual functions, but it would be something. 

    It would have to wait until a platform break, or be strictly opt-in though, because it does break binary.  Maybe it could be tackled with some __declspec tags?

     

    STL> I would have thought that _ITERATOR_DEBUG_LEVEL would have given itself a bit more room to grow.  For example, SCL probably should be at 5, and HID at 10.  This way values could have been set in between.  One thing that I could see going in at less than 5 would be some /GS like modifications.  Basically, vector and string could put some security cookies before and after their "real" allocation, and check those values during each deallocation of their memory range.  That particular mitigation may not actually solve anything though, as an exploit may have already been run before a check ever happens.

  • Thanks!

    Good Video and Lecture.

     

  • Thanks, Strephan!

    Great, as always!

  • PhilhippusPhilhippus

    That looks like heavy use of the pre-processor, which gives ammo to the C stalwarts who berate the perf of C++. Isn't there a way of getting the effect and value checking with some clever use of templates at compile time to shut them up? :P

  • petkepetke

    > Ben. Thanks for the reply. Makes sense.

    STL> I did some reading on "inline namespaces". Very interesting. Thanks for the tip.

    For instance I found this description of the feature: "At any time, in a given translation unit, one wants to pretend that a set of names that are defined in a namespace N, are also members of a namespace M, for all purpose but linkage (i.e. name mangling)."

    (I'm mostly worried about my libraries being binary compatible even though they are of different (product wide) versions. Say if the implementation of a function has changed but the interface has not. I would like object files, or static and dynamic libraries from different versions to be impossible to link together without being told there is a version conflict.)

    With I could imagine the name of namespace N being set to a version string by the preprocessor. Callers of the code would not have to know about the name of the N namespace, they could just use the fixed name of the enclosing namespace M. The implementors of the functions would only have to know about the not yet preprocessed name of the N namespace. That seems very clean. Does VC10 support inline namespaces?

    Cheers, and good job by the way. I always look forward with anticipations to your videos.

  • STLSTL

    Xeo> I like to lurk through the STL headers and was always confused with this proxy stuff, thanks for going into all those details.

    Cool, that's exactly what I wanted to hear.

    Xeo> But is it just me or do you get less time every video? Tongue Out

    This one was short for some reason (38:50). I don't have an internal chronometer, so I rely on the video guys to start waving at me from off-camera when I'm getting close to the time limit.

    Ben_Craig> This way values could have been set in between.

    Heh! Modes are incredibly expensive for us to support; for example, we have to build 5 static libraries. More would give me a heart attack.

    Ben_Craig> One thing that I could see going in at less than 5 would be some /GS like modifications.

    First, we just rely on HeapAlloc() to do whatever it does (and I have been informed that Windows has security-hardened it). Second, such checking would either be cheap enough to do unconditionally, or expensive enough to belong with the other IDL=1 checks.

    Philhippus> That looks like heavy use of the pre-processor, which gives ammo to the C stalwarts who berate the perf of C++.

    Non sequitur. We mainly use the preprocessor to simulate variadic templates (such simulation demands either macros or external scripts, and currently we rely on the former) and to conditionally compile code. Simultaneously, we are extremely heavy users of C++, relying on templates/overloading/etc. to do compile-time work that requires more precision than blunt text manipulation. We rely on compiler optimizations (especially inlining) to make calling tiny helper functions, passing empty arguments, etc. free. There is no contradiction here. As modern C++ programmers, we aren't really fans of the preprocessor, and we generally don't use it when better C++ mechanisms are available. In the case of _ITERATOR_DEBUG_LEVEL, we actually do want blunt text manipulation, if for no other reason than it should be possible to compile in debug mode with no optimizations and IDL=0, and not see any checking machinery whatsoever.

    petke> Does VC10 support inline namespaces?

    It does not. You can consult the magic decoder ring at http://blogs.msdn.com/b/vcblog/archive/2010/04/06/c-0x-core-language-features-in-vc10-the-table.aspx . Anything that doesn't have a row in that table, isn't supported by VC10.

  • GoogilyGoogily

    @petke:
    the open source compiler GCC have this support if i remember correctly
    (yet another reason open source rules, always up to date and no hidden agendas that halt development and releases, take that microsoft!)

  • ok i will repeat since charles obviously didn't get it the last time: Make make these videos longer, please ! come on don't be so cheap ! For instance, craplight (silverlight) and windows phone 7 have hours upon hours of video content *cough* marketing, breaking c9 rules! *cough* but c++ have like one video barely 40 minutes long every 3 weeks ? Only expressing that comes to mind is: What the hell ! Where are all the c++ videos we where promised, charles ?
  • CharlesCharles Welcome Change

    , Mr Crash wrote

    ok i will repeat since charles obviously didn't get it the last time: Make make these videos longer, please ! come on don't be so cheap ! For instance, craplight (silverlight) and windows phone 7 have hours upon hours of video content *cough* marketing, breaking c9 rules! *cough* but c++ have like one video barely 40 minutes long every 3 weeks ? Only expressing that comes to mind is: What the hell ! Where are all the c++ videos we where promised, charles ?

    Actually, you're wrong. I have released a C++ interview almost every week (sometimes, 2 weeks goes by, but come on...). For one thing, the C++ team is very busy and they have a schedule for releasing information to you which I have to follow. I have a number of interviews scheduled to dig into what the C++ team will share - according to their schedule. Perhaps you could share what type of C++ content you want to see? Not everything will be in lecture format.

    In terms of these lectures lasting longer, well, STL only needed ~39 minutes to get across what he intended to get across to you in this episode.

    If I could do a C++ interview every day, then I would. The problem is you need C++ people to talk to and good topics, too. You need to get on other people's calenders, etc.  As I said, the VC++ team is cranking hard right now and it's not right for me to randomize cranking developers to produce videos for Channel 9 when they should be writing code, fixing bugs, writing code, fixing bugs, writing code, ++

    I don't think I've made any empty promises. From Jan '11 - March '11, 17 C++ videos have been produced for C9. When the time is right, you will see an increase in the amount of C++ content streaming through 9.

    C

  • C64C64

    , Mr Crash wrote

    ok i will repeat since charles obviously didn't get it the last time: Make make these videos longer, please ! come on don't be so cheap ! For instance, craplight (silverlight) and windows phone 7 have hours upon hours of video content *cough* marketing, breaking c9 rules! *cough* but c++ have like one video barely 40 minutes long every 3 weeks ? Only expressing that comes to mind is: What the hell ! Where are all the c++ videos we where promised, charles ?

     

    Mr Crash: I think you are too harsh in judgment.

    I think both Charles and Stephan are doing a great job. I prefer few quality videos (that just take the time they need, no more, no less) instead of a lot of marketing/useless videos. I prefer quality instead of quantity. And STL's videos do have quality.

    Charles: in addition, some videos I'd be interested in watching would be related to the use that several teams in Microsoft (Windows, Office, XBox...) do of C++. I hope that you (and the other team members) can find a proper schedule for these.

    Thanks, and keep up your good job on C9.

     

  • StingzStingz

    I also prefer quality over quantity. One thing I would like to ask though is to please post these videos to the VC++ Team blog. I check that much more often than I check C9 so I was two videos behind even though I read the VC blog weekly.

  • CharlesCharles Welcome Change

    @C64: Great suggestions. Coming up is a video with one of the developers of WP7 (80% of his time is spent writing C++ and we talk about that). I like the idea of seeing how devs across MS use C++ in its various "forms" (meaning C with classes on the one hand and, say, template metaprogramming on the other, etc...).

    @Stingz: Stephan is writing up a post for this as I type... The VC blog posting schedule and the C9 release schedule aren't synched (and they needn't be). Perhaps we should align closer, however. I'll bring this up to Diego and team.

    C

  • StingzStingz

    Actually I totally missed the 2nd part of the advanced series. I thought it had been ignored by the VC blog... my fault!

  • CharlesCharles Welcome Change

    @Stingz: Here's STL's latest VC blog post on the matter: http://blogs.msdn.com/b/vcblog/archive/2011/04/05/10150198.aspx

    BTW, Stingz, you might as well bookmark this URL for your C++ pleasure 9: http://channel9.msdn.com/Tags/c++

    C

  • , Charles wrote

    *snip*

    Actually, you're wrong. I have released a C++ interview almost every week (sometimes, 2 weeks goes by, but come on...).

     

    Those interviews doesn't count becasue it's just chitchat.
    Except this one "Parallel Programming for C++ Developers: Tasks and Continuations"
    its was well acceptable yet watching it i felt they brushed over a lot that could have been interesting to talk about.

     

    For one thing, the C++ team is very busy and they have a schedule for releasing information to you which I have to follow.

    Well busy is a strong word relative to what they actually produce / release.
    But yes they are there to work not make videos i understand that.



    I have a number of interviews scheduled to dig into what the C++ team will share - according to their schedule. Perhaps you could share what type of C++ content you want to see? Not everything will be in lecture format.

     

    Many have given suggestions of topics you could talk about already, i think i may have given some suggestions too.



    In terms of these lectures lasting longer, well, STL only needed ~39 minutes to get across what he intended to get across to you in this episode.

    Actually STL have mentioned in the videos many times that the limited time have forced him to not talk about related things, in this video too.

    Secondly, i like STL and how he describes, etc, so given a few more minutes with him i think is a good thing.


    If I could do a C++ interview every day, then I would. The problem is you need C++ people to talk to and good topics, too. You need to get on other people's calenders, etc.  As I said, the VC++ team is cranking hard right now and it's not right for me to randomize cranking developers to produce videos for Channel 9 when they should be writing code, fixing bugs, writing code, fixing bugs, writing code, ++



    I understand but there is no reason for you to not plan ahead, ex. book time with devs.
    So the random part can be removed completely if needed with planing ahead.


    I don't think I've made any empty promises. From Jan '11 - March '11, 17 C++ videos have been produced for C9. When the time is right, you will see an increase in the amount of C++ content streaming through 9.



    It's been an empty promis so far.
    So i say "put the content where your mouth is"



  • 13 hours ago, C64 wrote



    Mr Crash: I think you are too harsh in judgment.



    As i have said before, i don't beat around the bush. It's a waste of time and may dilute / corrupt the info i want you to know, etc..

    Just think how much time you could save if politicians would skip the *-kissing, etc and get to the point.




    I think both Charles and Stephan are doing a great job.


    Hey !

    i said nothing about Stephan, he's doing a fine job,  he's actually very good at this.
    Don't assume stuff.
    Read the text i have typed and take it for what it is, no need to read between the lines, it will only confuse you and you will get it wrong

      I prefer few quality videos (that just take the time they need, no more, no less) instead of a lot of marketing/useless videos. I prefer quality instead of quantity. And STL's videos do have quality.

    I like quality too.
    Yes there have been too much of marketing / PR / useless videos.
    That is why i'm pushing for some good proper stuff.

    You could say they somewhat owe us for force feeding marketing / pr videos too use..

  • Stephan, thanks a lot for this info.  You mentioned that you could manually inject the detect_mismatch pragma into your import library.  I haven't been able to find any information on this.  Can you give a pointer?

  • STLSTL

    Filmed Part 4 today!

    GregM: You need to use the "LIB" tool to add an object file (containing the pragma) to your import library. See http://msdn.microsoft.com/en-us/library/7ykb2k5f.aspx . It's a very simple process on the command line.

  • Thanks Stephan!

     

  • jalfjalf

    How come you have the allocator as a class member, instead of deriving from it? The empty base class optimization would allow you to get the vector size down to the "expected" 12 bytes then (for stateless allocators), and I'm not aware of any downside to it.

  • STLSTL

    Our Empty Base Class Optimization isn't always activated in the presence of multiple inheritance, so deriving from allocators/comparators while preserving our current inheritance hierarchy could be very tricky.

    In VC11, we've avoided storing empty allocators/comparators through different means.  unique_ptr derives from empty deleters, but we've been working through various gotchas there (e.g. the deleter's function call operator was appearing on the unique_ptr, which is something we don't want).

  • SergeyKSergeyK

    First of all thanks Stephan - very well presented lectures.

    As for the optimizing for the empty allocator case (or empty deleter for unique_ptr) you can make one of the fields inherit from allocator/deleter instead of the main class thus solving unintentionally exposed interface problem. To make it generic you can use something like boost's compressed_pair.

    Another question: your approach on keeping a list of iterators to invalidate - I was thinking of an alternative way to accomplish the same end result. What about keeping revision ID in proxy object that gets incremented on each change to the container invalidating existing iterators? Each iterator keeps revision ID value the container had when the iterator was created. So you can get rid of “next” pointer you had for the list and if you make revision ID 32-bit you don’t even increase iterator size.

  • STLSTL

    > First of all thanks Stephan - very well presented lectures.

    Thanks for watching!

    > To make it generic you can use something like boost's compressed_pair.

    That's a good idea, but we don't have a compressed_pair implementation (yet). As I mentioned earlier, we've already solved this with a different approach for the containers. For unique_ptr, we've used private inheritance to conceal the function call operator (although it took me a couple of tries to get it right).

    > What about keeping revision ID in proxy object that gets incremented on each change to the container invalidating existing iterators?

    Ah, but iterator invalidation is more fine-grained than all-or-nothing. For example, erasing a list element invalidates iterators to that element, but preserves all other iterators.

  • XeoXeo

    Any updates on where in your encoding / uploading chain the 4th video currently is? I'm craving for it. ;)

  • STLSTL

    I just heard from Charles that Part 4 should be uploaded on Wed 4/27, assuming that I parsed "next Wednesday" correctly.

  • Michael KilburnMichael Kilburn

    Stephan, I have a question about previous lecture -- you've mentioned getting rid of pointer arithmetic -- why compiler did not do it for you? Usually MSVC is pretty good with micro-optimizations -- what about asking someone from compiler development team?

    (Or maybe compiler decided that that extra multiplication overlaps with surrounding operations and causes no performance loss?)

  • STLSTL

    The compiler's pretty smart, but it's not infinitely smart. In this case, we're performing a pointer subtraction to get a count of elements, which involves a division (really a shift, since the size is going to be a power of 2 - although if we generalized our use of memmove() to include trivially copyable structs, then we could end up with real divisions, which are even worse).  Then we multiply the element count by sizeof(T) (again, just a shift) to get a byte count.  Instead, we could have subtracted byte addresses to get a byte count, eliminating the division and multiplication.  The compiler may or may not be smart enough to figure that out in isolation (I haven't checked), but what's stopping it is probably the third bit - we perform pointer addition by adding the element count to the destination pointer (which involves a multiplication, i.e. a shift).  Again, with a byte count, we could do straight up addition to get the same value.  My vague guess, without actually asking anyone from the compiler team, is that the compiler sees that it needs the element count for two separate computations, and doesn't notice that all three locations could be changed simultaneously.

    I could talk to the compiler back-end team about this, and maybe I will, but this situation seems very specific to our memmove() wrapper. In any event, it's much easier for me to change the libraries. The back-end team is very busy, and I want other things from them. :->

  • Michael KilburnMichael Kilburn

    I see your point, but still -- these microscopic things should be in compiler's domain.
    It might be a good idea to check that compiler-generated code actually performs worse than hand-written case that treats all pointers as 'unsigned char const*'
    ;-)

  • IvanIvan

    Grat video as always. One question. I read your old article about vector swaptimization. What is the current state of that? BTW if you run out of ideas for AdvSTL you can always do a video on swaptimization. :D

  • STLSTL

    Ivan> I read your old article about vector swaptimization. What is the current state of that?

    Removed and superseded by rvalue references in VC10.

  • STLSTL

    Part 4: http://channel9.msdn.com/Shows/Going+Deep/C9-Lectures-Stephan-T-Lavavej-Advanced-STL-4-of-n

  • The documentation for MSVC10's "Checked Iterators" and "Debug Iterator Support" make no mention of _ITERATOR_DEBUG_LEVEL.

  • - wrong page -

  • Ryan KawickiRyan Kawicki

    Just a quick question.

    I am curious as to how the pragma detect_mismatch actually works with the IDL. I guess what I am really after is the rules by which IDL is placed into an object by the pragma detect_mismatch. Let me explain with some code.

    I have two projects. One builds an executable and the other builds a static library. In the static library, I deliberately set IDL to be 1 from the project settings. The executable is left to the default value of 2 under debug builds.

    Static Lib:
    Header: ///////////////////////
    #ifndef _IDL_LIB_H_
    #define _IDL_LIB_H_

    #define USE_STL 0

    #if USE_STL == 1
    #include <string>
    std::string IDLDblToStr( double val );
    #elif USE_STL == 2
    #include <string>
    const char * IDLDblToStr( double val, char * const buf );
    #else
    const char * IDLDblToStr( double val, char * const buf );
    #endif

    #endif // _IDL_LIB_H_

    Source: ///////////////////////
    #include "stdafx.h"
    #include "IDLLib.h"

    #include <map>
    #include <vector>
    #include <sstream>
    #include <iostream>

    #if USE_STL == 1
    std::string IDLDblToStr( double val )
    {
    std::stringstream ss;
    ss << val;
    return ss.str();
    }
    #elif USE_STL == 2
    const char * IDLDblToStr( double val, char * const buf )
    {
    std::stringstream ss;
    ss << val;
    strcpy(buf, ss.str().c_str());
    return buf;
    }
    #else
    const char * IDLDblToStr( double val, char * const buf )
    {
    std::vector< int > v;
    v.push_back(1);
    v.push_back(2);
    v.push_back(3);

    for (int i = 0; i < 3; ++i)
    {
    std::cout << v[i];
    }

    std::stringstream ss;
    ss << val;
    strcpy(buf, ss.str().c_str());
    return buf;
    }
    #endif

    Executable: ///////////////////////
    #include "stdafx.h"
    #include "IDLLib.h"

    int _tmain(int argc, _TCHAR* argv[])
    {
    #if USE_STL == 1
    std::string s = IDLDblToStr(123.456);
    #elif USE_STL == 2
    char buf[128];
    std::string s = IDLDblToStr(123.456, buf);
    #else
    char buf[128];
    IDLDblToStr(123.456, buf);
    #endif

    return 0;
    }

    If I define USE_STL as 1 or 2, then I get the expected linker error message of IDL 1 does not equal IDL 2, but if I define USE_STL as 0, I do not get any linker errors at all. I would have expected, since the static library source file still references STL, that the pragma would have inserted itself in the resulting object file.

    Thanks for any clarification you can provide.

    - Ryan

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.