Entries:
Comments:
Posts:

Loading User Information from Channel 9

Something went wrong getting user information from Channel 9

Latest Achievement:

Loading User Information from MSDN

Something went wrong getting user information from MSDN

Visual Studio Achievements

Latest Achievement:

Loading Visual Studio Achievements

Something went wrong getting the Visual Studio Achievements

Pex - Automated Exploratory Testing for .NET

Download

Right click “Save as…”

  • High Quality WMV (PC)
  • MP3 (Audio only)
  • MP4 (iPhone, Android)
  • Mid Quality WMV (Lo-band, Mobile)
  • WMV (WMV Video)
Pex is a tool being developed by Microsoft Research which has the potential to dramatically improve the quality of software testing while requiring minimal, if any, effort on the part of the developer. Pex can automatically generate a set of inputs for a paramaterized unit test which can effectively excercise most, if not all, possible code paths.

I visited Nikolai Tillmann and Peli de Halleux on the Pex team for a closer look at this cool technology.

Follow the Discussion

  • Allan LindqvistaL_ Kinect ftw

    thats so awsome Smiley

    i like the floating "we catch your bugs" note in the background too Wink

  • Stefán Jökull SigurðarsonStebet Internet spaceships is serious ​business​3;
    This is the coolest thing I've seen in a long time.

    For those interested here are the blogs from Nikolai and Peli:
    http://blogs.msdn.com/nikolait
    http://blog.dotnetwiki.org/default.aspx

    I can't wait for this to hit 1.0!
  • Nikolai and Peli,

    Very interesting technology - looking forward to v1.0 and a 64-bit version. Speaking of 64-bit, when will you release a beta that supports that version of the framework?

  • As far as I am aware, 64bit version isn't planned any time soon. My suggestion - use a 32bit virtual machine.
  • stevo_stevo_ Human after all
    Been using pex a little (not had time to really get into it yet), but its really impressive.. great work on this guys!
  • Too bad about the 64-bit version. We're more and more developers running in 64-bit environments (even in development), so it would really really nice to have access to the entire suite of tools. Running in a virtual environment when doing unit testing is probably one possible workaround, but I would definitely welcome 64-bit support.

    I've subscribed to the PEX list but haven't seen any trafic yet.

    Automated unittesting is worth dying for if done right. Can't really think of any code that wouldn't benefit (I suspect they even used Pex to test Pex Smiley).
  • Peli de Halleuxpeli Peli

    Good news: we are working on a WOW64 version of Pex  (support for exploring 32-bit .net processes on 64-bit).

  • Allan LindqvistaL_ Kinect ftw
    ive been playing around a lille with pex now and i just have to say.. WOW :O
    its GREAT! it sounded cool in the interview and all but in reality its even more awsome!
    if you havent already, you Have to check this out! the vs integration is sweeet Smiley
    the last time i had this kind of experience was when i tried out wpf! Smiley unit testing is no longer a horrible bore but actually something interesting Smiley

    again, great work pex team! cant wait for the next release Smiley

    i do have a qustion though Smiley is there a way to get pex to try a generic method with a bunch of diffrent types? i guess i could use like a list of instances of the various types and get pex to use diffrent lists or diffrent indexes in the list (like with a static list and a int argument to the parameterized method)

    but im sure there is a smarter way Smiley
  • Peli de Halleuxpeli Peli

    Pex actually supports generic unit tests: add generic arguments to the unit test and specify the instanciation types using 'PexUseGenericArgumentsAttribute':

        [PexMethod]
        [PexUseGenericArguments(typeof(int))]
        public void AddItem(List<T> target, T item) {
           ...
        }

  • Allan LindqvistaL_ Kinect ftw

    aaah, awsome Smiley and i see it also takes multiple parameters too, sweet Smiley

    is there also a way to have a staging method be called before each test or at the start of each test run?
    by stageing method i mean a static method that sets up the environment in diffrent ways. i guess i could call it from within the parameterized test but maybe the pex team thought of something more clever here as well Smiley
     
    im sorry if thats in the docs somewhere or in the ms test api thats built into vs, im sort of a noob with this kind of testing... Smiley im used to writing eeeverything myself.. Perplexed that is also why pex makes me ever so happy Big Smile

    maybe that isnt even a smaart thing to do.. Smiley i guess im sort of trying to do some mini system tests in my unit tests..Smiley but i figure, whats a unit anyway.. a part of a system could be a unit even if its actually a system in it self right? Smiley

    oh i got another question too
    is there a way to have the parameterized test pick some of the parameters from somewhere (like a collection) and try and figure the other ones out? like if i have a method that takes some arguments and a file path to like a config file and i want it to try out a few diffrent config files with various permutations of the other arguments to the method
    so basically what i waant to do is databind to some of the arguments but have pex figure out the others Smiley

  • Peli de Halleuxpeli Peli

    staging method: if the code that is being run in the method is relevant, you really want  to call it from the parameterized test. Otherwise, you can use [TestInitialize], [TestCleanup] methods, they are supported by Pex.

    Since you are mentioning the environment, be aware that Pex does not understand the environment (i.e. Pex cannot build constraint over the state of the file system, network, current time, etc...).

    Partial databinding: interresting idea but it's not supported as is right now. Currently, create multiple copies of the test or use the 'Abstract Test Pattern'.

     

  • Allan LindqvistaL_ Kinect ftw

    --
    staging method: if the code that is being run in the method is relevant, you really want  to call it from the parameterized test. Otherwise, you can use [TestInitialize], [TestCleanup] methods, they are supported by Pex.
    --

    alright i'll try that Smiley 

    --
    Since you are mentioning the environment, be aware that Pex does not understand the environment (i.e. Pex cannot build constraint over the state of the file system, network, current time, etc...).
    --
    no i understand that Smiley that would be kind of spooky.. Perplexed and awsome Tongue Out that is why it would be cool if one could supply a list of alternatives somehow Smiley

    --
    Partial databinding: interresting idea but it's not supported as is right now. Currently, create multiple copies of the test or use the 'Abstract Test Pattern'.
    --
    alright Smiley one could work around it by adding an int to te parameterized test, assume its within the range of a collection defined somewhere and use the int to look up the alternatives from that collection, but i guess pex wouldnt know that its supposed to use all the ints in the range..  you could also use if statements though, then pex would try to satisfy them Smiley but if there are alot of alternatives that might be cumbersome..
    perhaps a dictionary with an enumeration as a key? im just letting my mind wander.. Big Smile


    again, great work pex team Smiley

  • Peli de Halleuxpeli Peli
    If you're willing to dig deeper, Pex is extensible. You could implement your own [PexMethod] variation that performs the custom databinding. Unfortunately, there's no sample on this for now Smiley
  • Allan LindqvistaL_ Kinect ftw
    had some more fun with pex Smiley

    is there a way to get a certin pexmethod to use one factory and another (or all others) to use some otherway of exploring classes?

    or perhaps that factory souhld be a pex factory but just a regular static method that is called from the parameterized test?

    also, if you want your custom classes to appear in the table of results and arguemnts in visual studio, should you use PexValue.AddForValidation? (for arguments that are custom classes as well?) or is there some other way to do that?
  • Peli de Halleuxpeli Peli
    > is there a way to get a certin pexmethod to use one factory and another (or all others) to use some otherway of exploring classes?

    There is no automated way and only 1 factory per type is supported.

    > or perhaps that factory souhld be a pex factory but just a regular static method that is called from the parameterized test?
    Exactly.

    > also, if you want your custom classes to appear in the table of results and arguemnts in visual studio, should you use PexValue.AddForValidation?

    You can simply use PexValue.Add:

             PexValue.Add("name", myobject.ToString());

    By default, Pex does not call the ToString() method of user defined types, because it might stack overflow... and kill the process. Moreover, it might have sideeffect that would change the test behavior.

    PexValue.AddForValidation works similarly (it adds the value in the parameter table) but it also tells Pex to encode the observed value in the generated test for regression testing.
  • Allan LindqvistaL_ Kinect ftw

    great stuff!
    thanks Smiley

  • I've been playing with it for a while. Even though I know I shouldn't use it with production code (this is a research project), but I had to run it against real, production code to see some real results. Seems like me and my fellow team members forgot to write some very important tests Smiley

    I hope to find this in a future release of Visual Studio.
  • Anand ManikiamAnand Manikiam Newer, faster, Stronger ...
    I really like the concept. However I have some doubts ...

    1. How would Pex fare if the system is a state machine ? How would it limit the number of constraints it would need to test out. ?
    2. What about unsafe blocks in code ?
    3. Also when you say bugs these are purely functional right ..

    Also is there any place I can have a look at a sample test report .
    I am really looking forward to a fully functional release of Pex.

    Regards,
    Anand
  • Peli de Halleuxpeli Peli

    1. Not sure what you mean there. Pex does not generate API sequence calls currently if that's what you mean.
    2. Pex can reason about unverifiable IL, i.e. unsafe C# for example. However, Pex cannot monitor native code (i.e. C++ code). In that case, one needs to provide a 'managed' implementation.
    3. Not sure what you mean there.
    4. You can download the academic release of Pex. Nikolai has published a blog post where Pex is applied to the ResourceReader (http://blogs.msdn.com/nikolait/archive/2008/06/04/fun-with-the-resourcereader.aspx). The resource reacher touches a lot of classes in the BCL, including Encoding classes which are mostly unsafe.

  • Why can't constraints be constructs that can be applied on the class and methods and automatically enforced on the caller by the framework? That is, one should be able to decorate a method with:

    [Pre-Constraint: i>=1, i<=int.MaxValue, c!=null]
    [Post-Constraint: i==i, c==c]

    //Pre-constraint indicates value of i accepted from caller must be greater or equal to 1; and that c cannot be null.
    //Post-Constraint imply that on method exit the constraints should also be respected otherwise the method cannot successfully complete - constraint exception condition

    The above constraint would be enforced on calling code at design or compile time (if a known value or a variable whose value is known is being passed). The constraint on class instances being used in a method should then be passed up the chain to all callers. This would at least reduce a lot of run-time time issues since the constraints are enforced at design or compile time.

    In fact the i<=int.MaxValue constraint would be superfluous if int already defines constraint on acceptable values of between MinValue and MaxValue.

    ---
    Taiwo
  • This looks really interesting, but obviously can only work to a point due to undecidability constraints.  So how sophisticated does it get?  If I have in my code:

    if (SeventhOrderPolynomial(x) < 0)...

    will Pex actually try to solve the polynomial and put in values of x where it is larger than zero and others where it is smaller?  Does it give up at a certain order?  Linear?  Diophontine equations?  Does it use some sort of Monte Carlo algorithm to guess at values which may or may produce different branches in the code?

    I'm impressed regardless.  Very nice work.
  • Peli de Halleuxpeli Peli
    Hey you're asking too much! Pex relies on Z3 (http://research.microsoft.com/projects/z3/) an SMT solver developed in our group as well to 'solve' those problems. Z3 has limited support for non-linear arithmetic and generally does not implement optimization techniques (no simplex, no montecarlo, no simulated annealing or other). Z3 is really geared towards problems that are encountered in most problem... which are not usually 7th order equations Smiley

    Note that you can download Z3 yourself, and ask it questions.

Remove this comment

Remove this thread

close

Comments Closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums,
or Contact Us and let us know.