Jim Hogg: Phoenix Framework
- Posted: May 02, 2006 at 4:03 PM
- 55,855 Views
- 24 Comments
Download
How do I download the videos?
- To download, right click the file type you would like and pick “Save target as…” or “Save link as…”
Why should I download videos from Channel9?
- It's an easy way to save the videos you like locally.
- You can save the videos in order to watch them offline.
- If all you want is to hear the audio, you can download the MP3!
Which version should I choose?
- If you want to view the video on your PC, Xbox or Media Center, download the High Quality WMV file (this is the highest quality version we have available).
- If you'd like a lower bitrate version, to reduce the download time or cost, then choose the Medium Quality WMV file.
- If you have a Zune, WP7, iPhone, iPad, or iPod device, choose the low or medium MP4 file.
- If you just want to hear the audio of the video, choose the MP3 file.
Right click “Save as…”
- Mid Quality WMV (Lo-band, Mobile)
- WMV (WMV Video)
We recently caught up with Jim Hogg, program manager for the Phoenix Framework, a very robust backend compiler platform co-created by the VC++ team and MSR. Ever wanted to extend the functionality of a backend compiler? Well, now you can, and it's much easier than you'd think given the great work that's gone into "Phoenix" (it's a code name at this point...).
Comments Closed
Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation,
please create a new thread in our Forums,
or
Contact Us and let us know.
Follow the Discussion
EDIT: sorry i tought something else from the title. Is it posibble to extend a existing language like C++ and add to it new features? using Phoenix.
C
Good work!
Thank very much to MS for allowing this to happen and I hope many great plugins for it come out.
Before I download it, as it was mentioned in the video this is a framework for the backend, does that mean support for stuff like parsers, lexers etc, that could help develop a whole new language are out of this Framework?
The subject is slightly esoteric and might have benefited from having a possible int/external customer for the Phoenix to throw in few leading questions. When the interviewer comes off as trying to think up a question on the spot it just doesn't come out very well in terms of audiences confidence on the information gained and time spent.
Though since I don't write compilers I feel bad complaining here about the free peak into latest research..
Typically, the interviewer (me) knows about as much of the details of a given subject as the audience, which is by design. We feel it makes for a more interesting video than having domain experts asking all the right questions...
That said, I do like the idea of having internal/external consumers of a given platform technology involved in an interview focusing on the platform.
Good feedback. Thanks.
C
Thanks for this video, nice to see a fellow Scot flying the flag! I have to say this was one of the most intriguing interviews yet...and to be honest the real implications of this are eluding me somewhat.
I guess it means that...
a) you get fast response to new chip architectures - and so new platforms, probably most important for you guys with all the convergence thats going on.
b) you get more flexibility in analysing and tweaking code and introducing funky bits and bytes not handled by the source language.
c) The ability to optimise code in a bespoke manner if you are into highly specialised applications....
Are you going to be doing another interview focusing on managed code? I'd be interested to see what kind of improvements you can expect for managed code compared to C / C++.
Question time!*
1) Wouldn't it be possible to check for buffer overflows on the front end of the compiler? Maybe somewhere between the lexer/parser stages and the backend? Hopefully that is done before the optimization phase.
2) Any thoughts about running Phoenix itself through the Phoenix compiler?
3) How does it handle hand-optimized assembly code in the C++? I know that isn't done all that often anymore, but it does happen.
4) In theory, you could pretty much target any processor you want (not just x86 related ones). All you'd have to do is make the compiler emit its machine code into a text file and then take that file to whatever system you want. Er, right? While you're at, do it for the 6502.
5) His diagram threw me off a bit. If the .NET code is run by the JIT part of Phoenix, it would not produce a machine excecutable, correct? I hope I phrased that right.
6) I sensed there was some sort of "reverse engineering" ability with it? Is that a correct assessment? I thought at one point in the video he talked about taking a binary executable as input. If that is the case, does it backtrack to the point where it will crank out C++ code given a particular binary executable for input? Isn't that opening up a whole Pandora's box if people start reverse engineering everything in sight?
7) I have not done assembly language level optimization in a long, long, long time (like the 6502 days). I have a mediocre handle on x86 assembly (and can figure it out if I'm asked to) but the way the Pentium is put together internally is sort of goofy...at least the way the registers were sort of "added on to" over the years in terms of bits. Is that the case with the multi-core processors, too? Sort of like multiplying that several times over? I know the how/why of the register design additions over the years, but I can't imagine having to write assembly for a multi-core system.
8) I think the way parsers work is rather archaic, but that's just me. It seems incredibly inefficient to process code one character at a time (and then string them together into tokens, and then compare those tokens to predefined grammar, and then...). Any thoughts about changing that in the future? I have some ideas on how to do it, and if I find the time I might start messing around with that.
* yes, I watched the video
Here's my writeup of this interview.
Phoenix IR is a "linear", assembly-like language. Internally, it's binary, of course. A doubly-linked list of instructions, where each instruction has two single-linked lists hanging off - one for source operands, one for destination operands.
Take a look at "Advanced Compiler Design" by Muchnick to get a flavour of what it is, and the sorts of analysis/transformations you can apply.
Jim
http://research.microsoft.com/phoenix
You need a version of Visual Studio 2005 to use it - again, it works just fine with the Express editions - free downloads at:
(http://msdn.microsoft.com/vstudio/express
We update the RDK every 6 months or so. Next one is about ready, and should 'hit the streets' in a few weeks' time.
[needless to say, the RDK is for experimenting/researching - you can't ship a commercial product based on it - it's not yet ready for prime time]
Lexers and parsers - no, Phoenix doesn't help there. That whole area is already well covered with lex/yacc, flex/bison - versions by the dozen for whatever language you want to build your front-end compiler with.
Jim
Can Phoenix compile the il down to native code? - yes.
Can that native code run without a .NET Framework? - no.
Why? Even "pure" native programs require runtime support. For example, if your program writes a file, the OS does all the hard work, on your behalf, of finding free blocks on the disk to store that output, and remembering how to find them when you reopen the file tomorrow.
And it's the same with .NET programs. They can use features, provided (at runtime), by the OS. But they also use a bunch of extra features provided by the .NET Framework - eg:
a) memory management (allocation, and automatic garbage collection);
b) code access security
c) threadpool
d) appdomains
e) reflection
and a bunch more. Arguably, the most fundamental is the first - GC - but that's a different discussion.
Oh - I've made it sound like Phoenix does all this clever stuff, of converting a .NET assembly into native code and saving it. It's mostly done by the Ngen tool (also known, confusingly, as "prejit"). Phoenix can provide the IL-to-Native compile engine, but laying out classes, and persisting runtime data structures (thing of vtables, and such-like), etc is done by Ngen.
Out of interest - why would you like to run programs without the .NET Framework installed? - I'm not suggesting it's a crazy idea, just wondering which of several possible reasons you're interested in.
Jim
Target new chips (or extensions to an existing chip's instruction set, like when SSE2 came along)? Yes.
Think of the job: get existing user programs, written in C++ or Fortran or whatever, to run on a new chip - the results must be correct (no compiler codegen bugs!), and the code must run fast. Well, with a framework like Phoenix, you start that job 90% already done! Keep all of the analysis and optimizations and codegen it already provides (ie, all those parts that are independent of target chip). Just define the new chip's instruction set (opcodes, addressing modes, etc) and plug it in. (how easy this is depends upon the chip - eg Itanium is much harder than the x64 chips from Intel AMD).
Can you add 'features' using Phoenix, that the front-end language does not provide? Yes. Think of things like adding code to gather runtime code-coverage by-function/by-block/by-edge/etc. Or all of the features attacked by "Aspect Oriented Programming".
Custom/Bespoke optimization? Yes - write a "plugin" that does what you need. Replace an existing phase, or provide new ones. Again, no need to build an entire optimizing compiler yourself from scratch before you get the job done - use 98% of what Phoenix already does - just add your bespoke 2%.
This is starting to sound like "Yes, Phoenix can do that - now what's the question". But it's not - it's just that so far, most of the questions have been directed-enough, that I can say "yes".
Jim
On the other hand, if you use "safe" languages like C#, C++/CLI, etc then you're good. They all demand you specify the size of buffers (in their verifiable subset).
2) Yes. We call it "self-build" - done every full build (nightly).
3) We keep the assembly code as-is. We don't mess with it.
4) In theory, for unmanaged code you want compiled to native, yes. For example, with a C source program, run it thru our C front-end, and into Phoenix backend. Then write yourself a "dump" phase that transforms Phoenix IR into assembly-source for your target chip. You could insert that phase wherever you want - before we optimize, or after. The task for a managed language, like C#, is "more challenging" - you'd also need to write the equivalent of a .NET Framework for the 6502
5) No, all JITs for .NET are handed an MSIL method and told to compile it into native code. There's an API that the JIT uses to ask the 'surrounding' .NET engine (the "CLR") for info it needs to do its job - things like "tell me at what byte-offset you have decided to locate the field called "bal" in the class called "AccountDetails". The CLR reuses the native code for that method, each time that method is called subsequently in this session. But it does not save the result to-disk. If you run the App over again, all those methods get JIT'd afresh.
6) Kind of. Phoenix can read a .NET assembly, or a native binary, and convert into Phoenix IR (plus symbol table / type table). You can then analyze it (eg build the flowgraphs) and view the info.
For multi-core - currently, it's a hardware feature. How to take a program, written assuming a single cpu, either in assembler, or some high-level language, and change it to make use of as many cores as available - is a big challenge for the industry. Initial steps will likely introduce language extensions so the programmer can direct/suggest what needs to be done. Auto parallelization would magically figure this out without hints.
8) I can't think of ways to avoid the classic approach you describe: scan characters one-at-a-time and assemble into tokens, pass them into the parser, etc. Improvements for the future, I'd suggest, lie in devising better languages. Programming in today's OO languages is just too much typing - the compiler should figure out 50% of the nonsense we have to supply as programmers. For example, why do I need to tell the compiler the type of every variable? - the compiler can infer the right answer most of the time. Take a look at ML or Haskell - clear, powerful, no needless typing (in the sense of hitting the keyboard).
Jim
Thanks for your response Jim,
I have to admit that compilers are somewhat of a blackbox to me..or rather I treat them as such, it's going to take a change in perspective and a bit of thinking on my (and others i suspect) part to get the most out of this.
Very inspiring stuff though, I heard *rumours* that AMD are going to be making changes to their architecture for the "k9" that might take some of the sting out of coding parallel processing friendly apps. Is this going to be a future focus of effort between cpu designers and compiler spcecialists?
Guess you cant talk about that though
. Keep up the good work!
Charles, thanks again for this video. I think you need to interview compiler people more often.
:O
I love studying compilers (although I still don't have a total handle on yac, bison, etc.). I'll experiment when I get some free time with other types of parsers. It's hard to explain what I am envisioning in a parser without some pictoral explanations or a working demo.
I'm sure I'll come up with more questions, and maybe eventually I'll get around to downloading Phoenix.
Thanks again!
Actually, I interviewed one of the developers of VC++'s backend compiler (he's been working on the VC++ compiler team for about 15 years!) a few weeks ago. You'll meet him soon right here on C9.
Thanks Jim. The framework does not bother me - I like it. However, as an MVP and NG fan, the primary "shots" I see are:
1) Easy to decompile and see code (i.e. obfuscation problem).
2) The tax of installing the framework (and over again for each new version)
So the primary question I see (as y'all do I am sure) is why not just compile everything needed to native code and ship a native exe to kill both issues above? I have to admit at times that would be nice to have. Not sure if just running ngen would solve #1 and/or be practical? I am not complaining, I enjoy the .Net framework.
Mmm ... obfuscation ... the system needs to keep a copy of the original metadata and msil around, even after Ngen is done. Because if any of its Ngen-time dependencies change (eg, new version of a dependent assembly), the CLR will silently fall back to JITting the methods you need (the assembly is also enqueued to be 'fixed' by the Ngen service, I believe - tho' I'm not up-to-date with the details).
No easy solution on that one (altho' there's research afoot to improve obfuscators)
Jim
We're not aiming to do this. Binary translation (yes, it's source text, but machine-specific, so effectively 'binary') presents a whole raft of different problems. We already have enough challenges
Jim
Remove this comment
Remove this thread
close