Comments

@C64: The big thing I personally got out of last week's event and chats was that Standard C++ now does have a safe subset, but we need a concrete description -- ideally a mode. It has made me think again about doing a scrub of all standard C++ language and library features and marking some as "unsafe" in some way (possibly that allows overloading, such as for vector::op[] overloads for safe and unsafe instead of the current hack of providing op[] and at() which almost nobody uses), then supporting a switch that enables only safe mode. I believe I know how to do that very efficiently now that we have C++11.

I'll have to look into this more over the summer. I think saying "use this (possibly standardized) switch/mode and your modern C++ code is type- and memory-safe" would be a big deal and an important missing piece to completely answer and dispel this question. The question "what subset of C++ is that exactly and is it a usable subset" is a legitimate question now, and IMO wants a more concrete answer than we have today. I did look at this about four years ago, but we know more now, and (b) C++11 has already added most of the then-missing pieces now.

@Mort: "Interesting, as Herb Sutter explains, the direction received by his team from management was: R1. Enable full power and control, including both consumption and authoring. R2. Be as simple and elegant as possible for the developer. Right from the outset, we see that there was no requirement to conform to the language standard."

Because that's so obvious it goes without saying. As I continued (and you then quoted) a key goal of our management and my design team was, and the boldface is in the original: "Minimize the actual amount of non-portable non-ISO C++ code/tools developers have to write/use to use WinRT." You can call it failed, that's your prerogative, but it's disingenuous above to declare that I said we didn't have it as a goal -- it was one of the three I boldfaced.

Here's a clear counterexample: Look at C++ AMP. The same management and same design team in the same product cycle did a complete extension of C++ for GPGPU programming as almost a pure C++ library that follows C++ STL conventions faithfully and relies on C++11 features like lambdas -- with only a single language extension because one was necessary. We'd have been happy to do it without any language extensions, but that's not possible because current GPUs cannot handle the entire C++ language -- nevertheless, even with some extensions necessary, we're very proud of having been able to find a single general extension that enabled the rest of it to be written as a library.

There is not another GPGPU product for C++ out there in the marketplace today that does it with as few language extensions to ISO C++ as we did in C++ AMP in this same product cycle -- just one extension, the rest as a pure library. If you want to draw any conclusions from that, it would be fair to say that in the GPGPU space Microsoft values C++ more than any other company (all others primarily target C99 dialects, with at best C++ wrapper libraries) and placed higher priority on ISO C++ compatibility and minimizing language extensions than any other major GPGU vendor (they all have way more language extensions).

I asked:

"consider:

ref class Screen {
public:
    int DrawLine( Point x, Point y );
};

...

What is the minimal code syntax and build steps it would it take to express this in a MIDL-style approach, and across how many source/intermediate files? What's the best we could do in that kind of alternative approach?"

@PFYB answered: "For the sake of completeness:

// ISO C++
class Screen {
public:
int DrawLine( Point x, Point y );
};"

If that's your answer, you didn't understand the question.

Let me try to explain, at least with a few basic examples, why I don't think the above begins to answer the question because there's just not enough information:

• Is Screen is supposed to be a WinRT class, not a C++ class? How does the compiler know? How do I say so?
• Should it publicly inherit IInspectable, or not? (Your answers are inconsistent -- in your first example you show inheritance from IInspectable, whereas here you don't.)
• What is the Screen type's accessibility -- public or private? How do I say so?
• Does Screen aggregate the FTM? How do I say so?
• ... lots more but that's a start ...

You also didn't answer "what... build steps it would take" -- what are the build steps and tools involved, in what order are they invoked, and what does each one do? Also, "and across how many source/intermediate files" -- I think your answer to the source is no more source files, but what about intermediate files?

I'm just trying to explain why the above doesn't begin to answer my question. To me this isn't an alternative design, it's writing plain C++ code and waving "do the right thing to make it work" hands.

I asked: "Before I go further, let me sanity-check something: Do we agree that the route of "extending TLB #import" leads to a full ATL + MIDL like approach?"

@PFYB answered: "No."

Then you're clearly mistaken. WinRT is COM (and then some), and in general you can't author COM types without all the additional information about the types you can specify in an IDL file -- whether those information-carrying extensions go into a separate IDL file (the MIDL approach) or into the main .cpp file (the C++/CX approach, though it could be done with different syntax, but it must necessarily be just as rich in order to to the job).

@PFYB: "ATL + MIDL is a reasonable way to write code, but we are talking about something else. We are talking about not only "extending TLB #import" to cover WinRT modules, but also about doing the reverse of #import, that is, parsing regular C++ code ..."

Again, "regular C++ code" doesn't carry sufficient information.

@PFYB: "...and generating wrapper C++ code in order to expose regular code to WinRT.

Here is an example:

// ISO C++
struct I1 : public IInspectable {
virtual void F() = 0;
virtual std::wstring G(int) = 0;
};
struct I2 : public I1 {
virtual winrt_ptr<I1> H() = 0;
};
class C : public I2 {
public:
C();
...
};"

Short answer: There's not enough information in that code. Also, this doesn't avoid the need to generate wrappers, you will be generating wrappers anyway such as to change the std::wstring type to HSTRING and change all the function signatures to HRESULT returns and generate code to catch exceptions and translate them to HRESULTs and a dozen more things (I think you know this, but others on this thread think wrappers are unnecessary so I'm noting this for completeness; wrappers are necessary, and the goal above would be to generate the wrappers automatically from the above code).

To take just the first line, a few problems I can see are:

• ": public IInspectable" is not really C++ inheritance if you want it to have WinRT semantics.
• There's not enough information to decide whether the intent is to define I1 as an interface or a class, since a class could be named I1 and could have all pure virtual methods. (I hope you won't suggest a heuristic like 'if it has all pure virtual methods, implicitly assume it's an interface.' I tried that in early C++/CLI, in desperation to stick with ISO C++ even though I knew the heuristic approach was brittle, and it fails spectacularly anyway in several ways that are all ultimately rooted in that the programmer didn't get to express his intent explicitly.) Note that this lack of clarity about whether it's an interface or a class is not a problem in ISO C++ because ISO C++ has no separate concept of an interface -- everything is a class so it's all the same -- but it is a problem for any object model that does distinguish the two and has to generate significantly different code and semantics depending on which it is.
• As I've said before, the code doesn't carry enough information to generate metadata, do code generation, etc. The user still needs to provide all that information somewhere.
• As for the rest of I1, note that I1::G that returns a std::wstring will be slow if you provide only that much information. You can't actually use a native wstring on the boundary (it's not ABI-safe) so you will at minimum have to automatically convert it to be a WinRT HSTRING instead, and then how will you generate the code so that you don't do a needless conversion and temporary value on every call (if you say that last one is easy, I think you don't understand the problem -- performance will generally be unacceptable if you create a new HSTRING on every call to I1::G)?

Similarly, in the other two cases with the pattern "class X : public Y":

• Again, there's not enough information to do WinRT code generation. For example:
  • We don't know whether each of X and Y is intended to be a class or an interface.
  • WinRT requires X to have a default interface, but there's nothing in the code that says whether Y is to be the default interface for X. We could assume it must be, but then when you have "class X : public Y, public Z" how do you specify which interface should be the default interface? The user has to be able to state that information somewhere, and lots of other information like it.
• The meaning and code generation is completely different depending on whether each of X and Y is intended to be an interface or a class. There are three possible legal combinations, and all of them have very different code generation.
  • If X and Y both are interfaces, we have to emit a requires relationship and every cast must QI.
  • If X is a class and Y is an interface, we have to implement the interface, which is not as easy as it sounds because it then further depends whether I1 is to be the default interface or not:
    • If Y is the default interface, we can inherit directly, and every cast can just do a normal derived-to-base conversion. (We could just QI all the time, but that would be slow and a good example of the kind of pessimization you get when you don't have enough information.)
    • If it isn't, we can't and have to use a different representation.
  • If X and Y and both classes, we have to implement a complex COM aggregation-like pattern with a bunch of helper interfaces.

So where exactly is the syntax that lets the user state whether X is intended to be a class or an interface? what its default interface is? whether to actually inherit and do pointer conversions under the covers or emit a requires relationship into metadata and QI all the time? whether to aggregate the FTM (free threaded marshaler)? and a dozen more things like that.

Disclaimer: This is just scratching the surface. It is not by any means a complete list.

By the way, as I mentioned last night, I tried exactly this approach early in the C++/CLI effort -- there, ": public IInspectable" was spelled ": System::Object". Quoting myself: "For example, I asked "why do we need a new class category, why doesn't it just work to have 'class X : System::Object' be an inheritance tag from a magic type that the compiler can treat specially?" and spent several weeks trying to make it work, and several dozen similar questions -- many similar to questions in this thread."

@PFYB: "I hope the above is sufficiently clear."

I do understand what you want. I was in the same place in 2004 and tried many variations on this theme, and then we tried it again in 2010-11 in case WinRT might have removed enough of the constraints we had with .NET so as to make it possible (it didn't).

I think it's clear that we aren't going to be successful designing this together in a comment thread, and if you refuse to accept that it isn't that simple, from the people who tried and understand the requirements and constraints deeply, then the only alternative is for you to invest the time to make the technical effort yourself -- not just sketch things that look plausible, but build the prototype (including correct and complete binding to WinRT features and high-quality VS product tooling integration) and demonstrate a proof of concept.

This isn't science, it's engineering. There are many plausible ideas, but the only proof that an idea works is to show working code. As you do the exercise, you will discover a number of constraints and problems.

But sometimes the only way to explore is to go into the wilderness in person to see for oneself, if one won't accept maps from people who've been before. And that person on his own trek may indeed find something new that the earlier explorers overlooked -- that's how we make progress -- but one won't know until he actually goes, one can't decide that by the Victorian method of making a pencil-and-paper sketch sitting in the living room armchair by the fireplace.

@Glen: "Hurrahh that some smart people went to lengths to avoid using simple keywords; but it "befounds" the rest of us average joes that anyone would have ever have wasted time proposing or considering attributes for those roles in the first place, even given the sheer fugly factor alone!!"

Forgive me, but this is a perfect example: "I can't imagine why it was so hard to get ISO C++ to not use simple keywords virtual override/final" is a perfect parallel to the questions about "I can't imagine why it should be so hard to surface WinRT in ISO C++" -- you have to know the many social and technical constraints on the problem to see why things that everyone initially believes ought to be easy (or easier) really aren't.

The short answer in this case is that:

• WG21 places a very high priority on minimizing potential breakage of existing code, which really is important because a lot of people depend on third-party headers they can't edit and so forth. So WG21 wouldn't take "nice" common words as new reserved words, because that would break any code that used those words as identifiers.
• WG21 had never done contextual keywords before and is allergic to novelties like that. I'm still happy-but-amazed that they accepted contextual keywords into the standard.
• The only remaining major alternatives were to use uglified_keywords that would be very unlikely to break existing code, or to use the new [[attribute]] syntax (but it is entirely nonobvious to many people at first why disguising language extensions using attributes is a horrible idea that leads to an awful place).

I personally led the charge on this one, at multiple meetings. If I hadn't done that, today we would have a C++11 with [[override]] and [[virtual]] -- not only in that hideous style, but in weird grammar positions. That's what was the default solution already voted into the draft standard, and what we would ship with unless there was consensus to switch to some other specific alternative (which is a very high bar).

@Glen: "Not to sound ungrateful though, the reality of the man on the street is this: as long as the WG21 team can agree, that the concepts keywords are serving are useful - which should have been relatively easy since Java et al had already proved the ones we were talking about (i.e. nullptr, override, etc., which was my point) - the rest of us mere mortals would have accepted even full keywords in this context and dealt with the consequences! I don't mean to downplay your invention of a super keyword but to say anything other elides these facts."

That's simply not true or realistic. There are billions and billions of lines of working C++ code in the world. Breaking even 0.1% of the code in the world is a major breaking change that would directly hurt adoption of the new standard. Compatibility is king for C++ -- it always has been, because if C++ had not been essentially 100% C-compatible, essentially a strict superset of C, it would never have been widely used outside of AT&T. Even now C++ faces great pressure to remain a nearly perfectly compatible superset of C, to the point where WG21 felt obliged to formally add C99's preprocessor extensions and the entire C99 standard library to C++11.

@Glen: "That microsoft firmed up an implementation of some of those ideas, I am willing to accept, but your comments tend to suggest that the majority of invention came from C++/CLI and now that C++/CLI was required to proove them for the cases we were talking about. I think you will find the mainstream view is that the core ideas were not new (as you agree) but also that Java piloted those features sufficiently to see the value of them and that it elides a lot of reality to credit C++/CLI with the origin of those features, so you shouldn't do it. Still, like I said it's a minor point, we can agree to disagree. I still appreciate the work done on it."

The point wasn't the feature, it was the difficult legwork of how to expose the feature specifically in C++, and specifically with perfect backward compatibility with existing C++ code. The answers were not obvious.

@Glen: "I take your point about fusing C++/CX and WinRT. But never the less, their existance is fused. I don't see that one exists without the other? ref class has no future without WinRT?"

Not only does it have a future, but "ref class" already had a past even before C++/CX and WinRT in C++/CLI and .NET -- it has now successfully bound to two very different (if related) object models, namely .NET and WinRT. And we have enough experience now to be pretty sure if we need to ever bind to a new static OO object model, we can reuse the same language extensions with a new switch to signify the new target (just as today with /clr they target .NET, and with /ZW they target WinRT), not invent something new. (However, I don't know yet if they would be sufficient to bind to a dynamic type system's object model, which is both similar and different.)

@Glen wrote: "If the problem can't be solved in a transparent way, either the platform model is too complicated or it will harm C++."

I get it -- you want it to be true that ISO C++'s great flexibility is so powerful that it's sufficient to directly express all object models. It's frustrating to be told by people who it seems ought to know better that it isn't that way. I understand; I wish it were true, too.

You may not realize how much I understand your frustration.

Let me tell you story.

In the Way-Back Machine

In December 2003, an effort to revise the Managed Extensions to C++ was already underway (an effort that would later be called C++/CLI), but I was not participating in it at the time and in fact had heard little about it. I had joined Microsoft just the year before and was still figuring out the huge company and its platforms. I already had a lot of C++ experience and love, as did the other people on the team, but was still drinking from the fire hose of information and trying to absorb as fast as I could all the deep details of the platform and tools that people around me already knew.

That December, the C++/CLI effort had hit a technical and organizational roadblock, and people started coming to me to look for help. In the end I asked for, and was given, the job of leading the design team. As you can imagine, this always presents tensions and challenges for a project team already in midstream, and especially when the newcomer has some pretty hard-nosed design sensibilities about doing things the ISO C++ way.

For the first several months I made a real pain of myself, insisting on understanding the deep rationale for every single feature because I just couldn't believe such a set of language extensions was needed. For example, I asked "why do we need a new class category, why doesn't it just work to have 'class X : System::Object' be an inheritance tag from a magic type that the compiler can treat specially?" and spent several weeks trying to make it work, and several dozen similar questions -- many similar to questions in this thread. And my probing was most of the time not specifically whether a language extension was needed; the questions were deeper, to find the goals and constraints -- 'what semantics need to be achieved?', 'what does the code generation have to be?', 'what is the IL we have to map to and round-trip with?', 'how does the compacting garbage collector work?', 'why can't regular pointers point into the GC heap?', 'what are the exact semantics of pinning an object's memory?', and so forth.

In the end, once I grokked the main design constraints, I like to think that I did make a number of contributions that improved and simplified the design quite a lot. But at first I was really wishing it could be 'just ISO C++,' and it was painful when I really began to understand the deep reasons why wishing wouldn't make it so.

Over the course of those first several months, I gradually realized that a few patterns were emerging in my thinking, notably (but not only): 1. I kept trying to make a language feature look like a library, but it still required compiler knowledge and so it was still a language feature in library-syntax clothing and I was only fooling myself that that somehow made it more like a library. 2. I kept discovering differences where this really was a foreign object model that couldn't be expressed directly in ISO C++ any more than full ISO C++ classes with virtual functions could be expressed directly in ISO C.

I see the same patterns here in this thread, and I understand completely because I said the same kinds of things for months in early 2004 during the C++/CLI design effort when I didn't understand the full problem. I appreciate the patience the team had with me as I absorbed it, but even then it took months in daily meetings in rooms with whiteboards and high-bandwidth interaction.

In this comment format, I can't possibly convey all that learning derived from months of face-to-face interaction with the best experts on the technologies, but maybe I can distill some key learnings and why I felt they were so important in changing my understanding of the nature of the problem.

Pursuing an Analogy: A General Answer to "Why Language Extensions"

Let me pursue the analogy with expressing virtual functions in ISO C, because it's a very good one that directly applies to expressing foreign object models in ISO C++.

In the past, people could have (and sometimes did) argue at great length that C is enough (heck, it's Turing-complete and you can write OSes in it) so "you don't need C++ classes and 'virtual' etc. added to the C language, you can write your own vtables in C" and look here are three projects I can cite that did that...

... Yes, clearly, you can. But that "you can" completely glosses over the costs, including the usability and other drawbacks of doing it by hand, including the quantity of code you write, the quality of the code you write, the quality of error messages, the run-time performance, and the tooling difficulty and quality. If that list sounds familiar because I've mentioned those same things already earlier in the thread, it's because it's what always happens when you fail to expose high-level abstractions and instead use low-level abstractions plus coding conventions.

So how would you answer someone who presses you with the question, "well, can you tell us why it isn't possible to just write our own vtables in C"? Of course it's possible -- and at this point, before you can say more, they may tend to interrupt with 'ha! it's possible, you admit it! so you can have no valid reason not to have done it in just ISO C' -- ; but that doesn't change the fact that it's not a good solution because you're missing abstractions that belong in the language because they require compiler knowledge.

Key point: As soon as something requires compiler knowledge, it's a language feature. Pushing the language extensions somewhere else, like to a MIDL language+tool, is just squeezing the toothpaste tube and moving the extension to somewhere else in the design space; it's not removing the need for language extensions.

Corollary: Magic libraries and compiler code generation tools are language extensions. As soon as something requires compiler knowledge, it's not a normal library type, and wishing won't make it so. Neither will giving it a library-like syntax.

Note that the answer to nearly all questions of the form "why did you need a language extension for X" is "because X required compiler knowledge." The reason I say "nearly all" is:

• In a few cases the answer is "for usability." For example, usability is the reason C++11 added "auto" and "range-for" and C++/CLI and C++/CX added ^. They're only conveniences, but deemed important enough to bake into the language because their convenience is so useful and frequent that the sugar they provide yields a high benefit even though they're just sugar over other existing language features.
• But nearly always it's "because X required compiler knowledge." For example, that's the reason C++11 added decltype (the twin of auto), override, final, lambdas, nullptr, enum class, move semantics and rvalue references, uniform initialization, initializer lists, constexpr, variadic templates, and nearly everything else. Libraries simply would not do. (Before you say "but initializer_list is a library feature," I slyly and deliberately included that in the list to underscore the point -- initializer_list is not fully a library feature, because the language is aware of the type and gives it special semantics and code generation; you cannot write initializer_list as a library in a C++98 compiler and get its full semantics).

Limitations of the World's Most Flexible Object Model

Finally, the thing that was perhaps hardest to accept was that this beautifully flexible ISO C++ object model that seemed like it could express anything, this so wonderfully controllable and flexible and efficient object model, could not after all directly express all interesting object models. Some object models would stay foreign (at least until added to some future ISO C++) -- not just dynamically typed object models, but even statically typed object models that have different rules and constraints. So the realization was the following:

Key point: There are interesting object models that are not directly expressible in C++. "Directly" means in the language without relying deeply on convention and discipline; you can express anything in assembler via convention and discipline, but that observation is neither helpful nor interesting. "Other object models" mean ones that reflect different object layout assumptions (e.g., hiding data rather than exposing it the way the C++ object model assumes and requires), different static and dynamic restrictions, and even different language rules (e.g., deep virtual dispatch in constructors). When you need to bind to them, and you want to do it well with best quality and performance and toolability, you need language extensions.

Corollary: Using those object models well from C++ requires a language-level binding. It can be done without a language binding to the same degree as virtual calls can be done using hand-coded vtables in C.

Let's return one more time to virtual functions expressed in ISO C.

Consider what would happen if you really tried hard to do virtual functions in C without language extensions. You could imagine creating a wonderful set of C macros (just like #import-enhancing macros) that automates a lot of the repetitive coding for you, and the resulting code may even look reasonably nice. (After all, when we're in this "imagine if" mode before doing real detailed systems prototyping things always look rosier and feasible because you haven't found the problems and limits yet; but let's assume what I just described was actually possible in this case for C macros simulating virtual functions.) (And I said "you could imagine creating" on purpose -- that's what this thread is often doing.)

Here's what we know you will never get without a language feature:

• You will never get the code elegance and readability. People using those macros will always write more verbose code than just adding the one word "virtual" to a function signature. (The same is true in C++/CX of, say, "ref".)
• You will never get as robust code. The code that people write will definitely be more brittle -- if nothing else, there will be many more places to spell it wrong because it relies on more-complex conventions and so there are more points of typing/thinking failure. Further, when you say "virtual" you declare your intent and language rules kick in to do two things on your behalf: 1. The language rules do a lot of work and code generation automatically, and they'll get those detailed vtable mechanics right every time (modulo compiler bugs) because it's the compiler that's writing the code, not you. 2. The language rules can often make it impossible to write certain classes of errors even by mistake, because the language can often be designed to make it impossible to say some nonsensical things. When you roll your own vtables, you're not declaring any intent, all the code generation is by hand or at best semiautomated, and there are no language guard rails to help you do it right.
• You will never get the same high quality of error messages. When "virtual" is a keyword you are clearly declaring your intent, the language has rules that enforce you did it right, and the compiler can tell you far more exactly when you got it wrong -- at compile time. When you roll your own vtables, most errors will be at link time or run time rather than compile time, and will be far more cryptic and less actionable.
• You will never get the same run-time performance optimizability. It's important to think about why this is fundamentally true: It too derives from expressing intent. When you don't declare your intent in a language-understood way, you cripple your compiler's and optimizer's ability to help you. For example, in VC++11 we are doing aggressive devirtualization which can result in some major performance wins. Good luck getting that optimization without the "virtual" keyword and instead using hand-coded vtables in C, because the compiler and optimizer will have no idea what you're trying to do. (Note that this is the same reason why C++ templated algorithms taking iterators and lambda functions are routinely faster than hand-coded C loops with direct pointer increment and dereference instructions! That higher-level abstractions just naturally enable performance optimization is just a fact of life, an inherent benefit that results from the fact that letting your tools know more about what you're doing directly increases their ability to help you.)
• You will never get the ease of producing high-quality tooling. Yes, you can code your vtables in C by convention -- but don't expect to get as nice a debugger or object browser, and possibly no class wizard or class inspector at all.

That's what (well-designed) language abstractions always give you over coding convention + discipline: More concise, more elegant code. More robust code. Higher quality error messages. More performance optimization opportunities. Better toolability. This is not wishful idealism; it's inherently true, every time.

All of these things -- said above about language extensions to C for dealing with virtual functions -- apply to language extensions for binding C++ to foreign object models, for they are simply always true for all language extensions that eliminate conventions + discipline.

ISO C just isn't suited to natively doing OO programming with virtual functions, any more than ISO C++ is suited to doing a component object model with true encapsulation and ABI safety. ISO C and C++ are both incredibly useful, powerful, and flexible languages; but an important part of using a power tool is to understand its limits. Sometimes that takes time and deep experience at the boundaries.

@Glen wrote: "How many other languages are you going to extend or recommend get two notions of class?"

Let me ask you a counter-question, please.

What would you say if you were proposing virtual functions for C, and someone asked, "Why do you recommend getting two notions of functions? Functions are functions, darn it, everyone knows that, virtual functions are just functions and I'm sure you can express them in ISO C without extensions, we have function pointers and everything you need."

Two questions:

1. How would you answer that?
2. When your answer begins with, "Well yes, you can, but..." -- now how do you keep the person listening after that first four-word sound bite that may tell him what he wants to hear, but isn't anywhere close to being the complete truth?

It took me a while to get all this through my head, so I know it takes time.

I don't know if this will help, but maybe it will.

@Glen:

Re: "I believe the extensions you attribute to being "thanks to C++/CLI" are really more "thanks to Java" or someone prior to C++/CLI. I am surprised you keep stating this claim, I know you helped push them through to C++ but that's different from originating the basic concepts."

Of course many languages, including before Java, had some of these concepts (spelled in C++/CLI and C++11 as nullptr, enum class, override and sealed/final). But ISO C++11 had many alternatives for each one (choice of keyword, grammar position, attribute or not, etc.), and standardized exactly C++/CLI's syntax and semantics with few or no tweaks. The C++11 keyword "nullptr" and the "enum class" syntax and semantics are C++/CLI. For the other two it's even clearer: "override" et al. are ISO C++'s very first ever contextual keywords; there were many other options, from attributes to fully reserved words, and C++11 followed directly the trail blazed by C++/CLI to make them contextual even though that was a major novelty for C++11, but because it had been shown to be a great option that works seamlessly in practice based on the field experience with C++/CLI it was considered safe for standardization. I consider that a big win for the whole C++ community, because it would have been beyond awful to have override and final be [[override]] and [[final]] attribute, or based_check_override and base_check_final deliberately-uglified-to-not-break-existing-code keywords, which were the other major options n the table.

You really don't get field experience with future-standard language features unless someone implements them as an extension, and so C++/CLI / C++/CX has turned out to be a sort of "Boost for language features" and I expect more features to be proposed for ISO C++.

Re: "Regarding properties, events and delegates, if they were of value to ISO, it's stunning that they aren't already in the language or weren't more widely debated this time around and more stunning why MS didn't at least propose them so either could happen or advance their definitions further - you already had all the experience you needed with them in C++/CLI and they feature again in C++/CX unchanged."

Good question, two reasons (caveat: I'm going from memory here, which can be dangerous):

1. They were cut from C++0x for scope/interest: As I recall, deferring properties/delegates/events was more of a C++0x scope issue and the availability of experts willing to contribute their time to develop the proposal. Only so much was going to fit into C++0x, and a lot of good ideas had to be cut early on because we couldn't afford the development time and/or didn't have the right people available. Even at that, we didn't cut enough -- x was hex.
2. The timing was wrong: If I recall correctly, properties/delegates/events were cut/deferred from C++0x before C++/CLI existed. Remember, C++0x work began in 2002, Bjarne's wishlist that I linked to was as of 2004, and the C++/CLI standard was not finished until December 2005.

Re: "Regarding goals B1, and B2."

I'm not clear from your answer which you prefer, B1 or B2?

Re: "C++/CX as it stands requires me to wrap all of my libraries - my classes, i.e. the ones that *I* produce; at the class level"

C++/CX doesn't do that, WinRT does. How is this different from COM or CORBA?

ISO C++11 doesn't have ABI-safe components or modules; it can't express those. That's not surprising, because the design requirements for C++ classes and ABI-safe classes are inherently different and incompatible:

• C++ is all about efficiency and direct access, and so C++ classes and the C++ object model guarantee that the compiler has direct access to the bits of an object to enable full optimization and "you don't pay for what you don't use" etc.
• A component object model is all about decoupling and truly encapsulating, and so ABI-safe classes and the ABI-safe object model are all about hiding the bits of an object to decouple caller and callee.

So to have ABI-safe component types inherently requires non-ISO C++ code somewhere.[*] The only question I can see on this thread is where to put that non-ISO C++ -- and putting it in a separate MIDL file with its own language extensions and compiler is no less of an extension and no more portable than putting the extensions in the same source file.

Re: "Unlike, say, a Win32 GUI API, that you wrap, C++/CX wraps you, at the class level"

(Again, this should say WinRT.)

Right, like COM and CORBA and the .NET CLR, WinRT is a runtime environment with its own object model, not just a library. The task my team was given was how to enable C++ programmers to consume and author types in that universe with as minimal impact to C++ code as possible by minimizing the amount of code/tools that programmers would have to write/use to work with the extensions.

Re: "If Bjarne really thinks this is a good idea"

Just to be clear, I was not claiming any of those participants endorse C++/CLI, I was saying that they actively invested effort to contribute to the design of C++/CLI to make it the best quality it could be and as compatible and conflict-free with ISO C++ as possible. Bjarne prefers no extensions at all whenever possible, and wishes we needed to do fewer, while recognizing that extensions are necessary to deal with different operating environments (such as in this case).

Re: "Whatever the means, wrapping is what makes C++/CX not viable in of itself IMO. Anything that doesn't virtually completely remove the burden of wrapping your own classes (even if you could justify the need to do it to start with, which I am not yet sold on either), is wrong; because it harms C++ productivity no matter how you do it - unless you don't do it. That's my stance, don't do it. If the problem can't be solved in a transparent way, either the platform model is too complicated or it will harm C++."

See above, the fundamental difference, starting from basic requirements, between C++ types and ABI-safe component types.

How can you expose an API surface to any of WinRT, COM, CORBA, or .NET CLR without using their types? The question isn't whether wrapping is necessary; it is. (If it weren't, then generations of COM and CORBA vendors have been somehow naive.) The question is only how to spell the extensions and where to put them.

Herb

Interesting; I'm glad to see how well that interview has held up over the course of a decade where my own understanding of Microsoft products/platform/ecosystem, as well as the course of the industry through through a major transformation or two, has changed a lot. I still agree with pretty much everything I said, including the need for further conformance to the new C++ standard now that it has stabilized and was published last month.

@PFYB asked:

1. Isn't a third extension (on top of countless things marked as "Microsoft-specific" which we take as a given) in 10 years "a lot"?
2. Was there really "a clear need" for C++/CX given that a strict C++ alternative as suggested in this thread was possible as well (if you disagree it was possible, I'd appreciate knowing why, but I have already asked about that a number of times to no avail, so if you disagree but won't say why, just skip the question)?
3. Was there "some community agreement" around C++/CX?
4. Is C++/CX one of the things the community "has been clamoring for"?
5. Is C++/CX something that is "not proprietary but rather the opposite: thing we already know is certain or likely to come" in the next version of the C++ standard and which you "hope all compiler vendors will provide too"?
6. Are you encouraging other vendors to implement C++/CX "to the benefit of everybody no matter whose tools they're using"?

Taking them one at a time:

1. It's not a brand-new set of extensions, rather it's essentially the same as C++/CLI. This was deliberate; once we realized were were on a path of doing language extensions again for the reasons I tried again to clarify in my note a minute ago, I did not want to create yet another different set of extensions. (There were several attempts from other people and teams within Microsoft to propose other extensions including many variants of the @ character, and despite considerable pressure to do that we steered away from the shoals of inventing yet another thing where you'd see code like @IFoo everywhere. There were so many that by the end of last year "hey, why don't we just use @ here" had become a running joke within our design team, because it had been proposed by so many people using so many different potential syntaxes for such a long time.)

2. We think so, given the constraints I tried again to describe in my last comment. I haven't given up trying to find and answer the question here. Referring to that comment, it comes down to whether design requirement R2 is important to you, and whether you prefer choice B1 vs. B2 which are different ways to advocate for ISO C++ purity and minimize impurity.

3. Yes, because C++/CX is C++/CLI (modulo spelling gcnew as ref new, and very minor differences such as when WinRT doesn't support one or two things that .NET does, like generic classes). C++/CLI is the only set of non-ISO C++ language extensions in the world that was developed openly in an international standardization process with the participation of a who's-who of the participants and officers of ISO C++ (WG21), including Bjarne, IBM, Intel, EDG, Dinkumware, Plum Hall, and others including notably experts from UK and France, whose contributions were very helpful and changed C++/CLI to make it better than it would otherwise have been. This involved holding design and standardization meetings around the world, co-located with WG21, over the course of two years, plus telecons, sharing all documents openly (a first for Ecma), and other open engagement. C++/CLI is now an international standard (Ecma-372) that anyone can freely implement; that's a good thing now, because I've already heard about other compiler vendors investigating implementing C++/CX, which to my knowledge nobody ever did for C++/CLI.

4. Partly.

• From a feature point of view, it does include things like properties, delegates, events, and other things on Bjarne's 2004 wishlist, including extensions that originated in C++/CLI and are now in ISO C++11, such as nullptr, enum class, override, and sealed/final all with exactly or very nearly C++/CLI syntax and semantics.
• From a platform point of view, some developers were clamoring for doing better COM integration (e.g., to be able to do everything in one source file without extra files/tools), and despite that there will always be some who dislike any given design choice, we've had a lot of positive feedback from people who are glad that we finally let them do COM without MIDL.

5. Yes, see above; parts of it have already been adopted in ISO C++11, and if the door opens for another round of language extensions then among the first to be submitted are likely to be things like properties/delegates/events which didn't make C++0x's priority list last time partly for lack of time and desire for more field experience with library vs. language approaches (which C++/CLI / C++/CX is helping to generate), but have always been of interest (see again Bjarne's 2004 wishlist).

6. Yes. Just like with C++ AMP, which we also announced earlier this year as a VC11 feature; I know of multiple investigations/implementations in progress for non-Windows platforms, and we are actively helping those implementers with their implementations. This is a definite change from the past; our viewpoint is that anything we ship, including library interfaces (not necessarily the Windows-specific implementations), should be something that we'd be willing to encourage other implementations and to submit to standardization if there's interest. That's one reason many of our new libraries in VC++2010 and VC++11, like PPL and AMP, are using c_style_naming (the naming convention preferred by ISO C++ and Boost) except where we were specifically binding to something that already had a different strong naming convention (e.g., the Platform:: namespace VC++-specific WinRT wrappers we provide we chose to PascalCase for consistency since they were specific to WinRT).

I've been thinking about this over the weekend. (Yes, I did read parts the thread last week and since then, though it really is too long to read in detail.) When two people express themselves and try to listen, and both still feel unheard, it's usually because of a fundamental disconnect. In this case, I suspect it's a difference about design goals.

Before I go further, let me sanity-check something: Do we agree that the route of "extending TLB #import" leads to a full ATL + MIDL like approach?

This seems to be a key point since early in the thread:

• @Glen actually complained of us doing IDL in the past. Amid a list of our technologies: "... Then we had COM. Object based, but hardly useable and hardly C++, just IDL and a vtable. In reality, it was even less friendly than C. ... Where in that great mass of development is a simple ISO C++ API? I don't see one! Now I realise I never had one!!"
• @C64 asked: "What would be an ISO C++ alternative to C++/CX? Would it require using preprocessor macros? Would it require a separate compiler (kind of like MIDL) for metadata or other platform specific stuff? I'm not sure this would be better than a platform-specific language extension like C++/CX."
• @PFYB replied: "Yes, macros, base classes, as well as tools (or new logic in the existing tools) to generate and use metadata."

Later, when I replied saying in part that this path leads to growing a full MIDL language/compiler and toolchain model" (which we completely agree is feasible; we've done it in the past), two of the responses were:

• @Dennis wrote: "You are willing to do a full-blown compiler and toolchain for C++/CX, but not for C++? How is parsing C++ code and generating other C++ code significantly more difficult than parsing C++ code and generating metadata?"
• @PFYB wrote: "We suggest you generate wrapper code for exposing C++ classes to WinRT in the compiler, using a logic similar to the one you employ for generating metadata out of ref classes. Alternatively, move that logic into a separate tool. Before you say so, yes, we realize this logic is going to be different than the one you are using for ref classes right now."

So I'll assume (please correct me if I'm wrong) that we agree that extending the #import model for the consumption side means that the authoring side would employ something like a MIDL language and compiler/toolchain -- so that programmers could specify the additional information that is needed for metadata -- plus a supporting structure of macros, base classes, and/or ATL-like smart pointers. Correct?

If that's the question, let me do my best to shed light on it...

At the outset of the project, my team was asked by our management to take as primary design requirements that we find a programming model that should:

R1. Enable full power and control, including both consumption and authoring. It's C++'s turf to leave no room for a language lower than C++ to get at the underlying platform, and we wanted that to be true also for WinRT. There should never be a reason to use another language to be able to control or express WinRT-like things. I think(?) this is noncontroversial on this thread (correct me if I'm wrong).

R2. Be as simple and elegant as possible for the developer. In particular, the model should minimize both the quantity of the code the user has to write and the complexity of the build system he has to deal with, comparable to the other WinRT languages. The main reasons for this goal were:

• To minimize the amount of non-ISO C++ code the user has to write. Whether via language extensions or MIDL extensions, the user has to express information to author extra information about a type, this is inherently nonportable WinRT-specific code, and we wanted the user to have to write as little of it as possible, and as easily and quickly as possible, to spend as little time as possible on that necessarily evil on the wrapper around his portable ISO C++ code.
• To not create a disincentive to use (V)C++ on WinRT. When writing most of the program code in C++ is appropriate, we don't want "but I want to also use WinRT" to create an incentive to use a different Microsoft language/compiler instead of VC++. It would be a disaster if we now had all this wonderful native (not-managed) WinRT environment but created an incentive to use it from a non-native language (i.e., not C++) for usability reasons alone. As one litmus test, if in side-by-side WinRT examples on MSDN the VC++ example is routinely twice the code of the C# example, we have a serious problem of people using the C# compiler instead, and often not just for the boundary code -- it would actively hurt ISO C++ use on the platform by creating an incentive to use another language compiler instead. We felt it's essential not to dilute the message that "modern [ISO] C++ code is as clean and safe as code written in other modern languages" by adding things like "except when you use WinRT or want to write a modern Metro app or do other modern things with it."

I suspect that the tension in this thread is around R2, because that goal/constraint is in conflict with an ATL + MIDL like approach. As a simple example that doesn't even use properties/delegate/events, consider:

ref class Screen {
public:
    int DrawLine( Point x, Point y );
};

Screen s;
s.DrawLine( Point(10,40), Point(20,50) );

In this admittedly simple authoring example, the only syntax delta from ISO C++ is "ref" and the code is all in one place; no extra tool is required, and the user just wrote a few more letters and moved on.

Two questions:

A. What is the minimal code syntax and build steps it would it take to express this in a MIDL-style approach, and across how many source/intermediate files? What's the best we could do in that kind of alternative approach?

B. How do you feel about these two goals?

1. Minimize the actual amount of non-portable non-ISO C++ code/tools developers have to write/use to use WinRT.
2. Make the code in the .cpp file itself appear to be using a C++ library, even though it actually is not (it is generated from Windows-specific language/extensions that are just located in a different non-.cpp file).

Both goals are pro-ISO C++ statements, but they are different and actually in conflict -- at least, I don't know of a way to achieve both simultaneously, and we tried.

So it sounds like perhaps the fundamental tension is that this time we aimed for answer B1 but some other people would prefer answer B2 (which we aimed for in the past with ATL + MIDL). I think either choice will unfortunately disappoint some subset of developers. We have lots of experience shipping B2 and know its weaknesses and the reasons why some customers have kept telling us they don't like it; for example, people have complained equally that that's not ISO C++ either (I agree; I don't think either C++/CX style extensions or MIDL extensions/tools ends up being any more "ISO C++," and I think there's feedback about that even on this thread such as @Glen's and @C64's comments I quoted above).

Either way, it's still all nonportable proprietary extensions that are not ISO C++, just in different places and styles, and so I have sympathy for prioritizing the B1 goal of at least minimizing the quantity of it (as a subgoal of requirement R2 above). The primary reason we couldn't convince ourselves to do down the B2 path again was because every time we generated side-by-side WinRT-using examples we found it made the user's code more verbose even in the .cpp file, and added more complexity and extensions in the .idl file and compiler that would be needed, and therefore (we felt) failed to achieve R2 because it created a disincentive against using VC++ for WinRT projects which includes a disincentive against using C++ for even the non-boundary code.

There are other design goals and constraints we already mentioned, such as supporting properties/delegates/events which begs for language support and supporting tools like debuggers and design tools well, but even those are secondary to these two fundamental requirements R1 and R2 above.

@PFYB wrote: "I suspect Herb didn't read the thread. I suspect he won't. Such is the "discussion" we are having here."

Note that this is now a 32,000-word thread. That's half the word count of Exceptional C++.

So I appreciated that @PFYB agreed to identify a key question, which can reasonably be answered:

@PFYB asked: "The main question raised in the thread is: *** Why, instead of doing C++/CX, did you not use a design that requires no language extensions, eg, by generating C++ wrapper code for consuming WinRT components (similarly to #import) and generating C++ wrapper code for exposing WinRT components (the reverse of #import)? ***"

Yesterday I put aside other work to spend a couple of hours writing a thoughtful answer, without evasion and in reasonable detail, including: a pointer to detailed discussion of many of these points in my 54-page A Design Rationale for C++/CLI; that we had tried and prototyped this and other options; and summarizing some highlights of what we learned over the course of about 18 months (end to end, the intensive part was about a year), during which time WinRT itself was still in flux and we had to both influence its design and accommodate the directions it might go.

@PFYB responded in part: "I am deeply dissatisfied with the post from Herb [that answered exactly your question]. I am tired of answering the same old points ad infinitum [so evidently others have already given you the same answers to the same question]. I suspect Herb didn't read the thread. I suspect he won't. Such is the "discussion" we are having here."

Wow.

You're welcome. Sorry that I can't be helpful.

Herb