Comment Feed for Pex - Automated Exploratory Testing for .NET (briankel on Channel 9)

Re: Pex - Automated Exploratory Testing for .NET

peli — Tue, 04 Nov 2008 15:23:14 GMT

Hey you're asking too much! Pex relies on Z3 (http://research.microsoft.com/projects/z3/) an SMT solver developed in our group as well to 'solve' those problems. Z3 has limited support for non-linear arithmetic and generally does not implement optimization techniques (no simplex, no montecarlo, no simulated annealing or other). Z3 is really geared towards problems that are encountered in most problem... which are not usually 7th order equations :)

Note that you can download Z3 yourself, and ask it questions.

Re: Pex - Automated Exploratory Testing for .NET

darrellp — Tue, 04 Nov 2008 06:18:56 GMT

This looks really interesting, but obviously can only work to a point due to undecidability constraints. So how sophisticated does it get? If I have in my code:

if (SeventhOrderPolynomial(x) < 0)...

will Pex actually try to solve the polynomial and put in values of x where it is larger than zero and others where it is smaller? Does it give up at a certain order? Linear? Diophontine equations? Does it use some sort of Monte Carlo algorithm to guess at values which may or may produce different branches in the code?

I'm impressed regardless. Very nice work.

Re: Pex - Automated Exploratory Testing for .NET

taiwo — Sat, 25 Oct 2008 19:16:47 GMT

Why can't constraints be constructs that can be applied on the class and methods and automatically enforced on the caller by the framework? That is, one should be able to decorate a method with:

[Pre-Constraint: i>=1, i<=int.MaxValue, c!=null]
[Post-Constraint: i==i, c==c]

//Pre-constraint indicates value of i accepted from caller must be greater or equal to 1; and that c cannot be null.
//Post-Constraint imply that on method exit the constraints should also be respected otherwise the method cannot successfully complete - constraint exception condition

The above constraint would be enforced on calling code at design or compile time (if a known value or a variable whose value is known is being passed). The constraint on class instances being used in a method should then be passed up the chain to all callers. This would at least reduce a lot of run-time time issues since the constraints are enforced at design or compile time.

In fact the i<=int.MaxValue constraint would be superfluous if int already defines constraint on acceptable values of between MinValue and MaxValue.

---
Taiwo

Re: Pex - Automated Exploratory Testing for .NET

peli — Sat, 19 Jul 2008 14:59:02 GMT

1. Not sure what you mean there. Pex does not generate API sequence calls currently if that's what you mean.
2. Pex can reason about unverifiable IL, i.e. unsafe C# for example. However, Pex cannot monitor native code (i.e. C++ code). In that case, one needs to provide a 'managed' implementation.
3. Not sure what you mean there.
4. You can download the academic release of Pex. Nikolai has published a blog post where Pex is applied to the ResourceReader (http://blogs.msdn.com/nikolait/archive/2008/06/04/fun-with-the-resourcereader.aspx). The resource reacher touches a lot of classes in the BCL, including Encoding classes which are mostly unsafe.

Re: Pex - Automated Exploratory Testing for .NET

Anand Manikiam — Sat, 19 Jul 2008 12:27:02 GMT

I really like the concept. However I have some doubts ...

1. How would Pex fare if the system is a state machine ? How would it limit the number of constraints it would need to test out. ?
2. What about unsafe blocks in code ?
3. Also when you say bugs these are purely functional right ..

Also is there any place I can have a look at a sample test report .
I am really looking forward to a fully functional release of Pex.

Regards,
Anand

Re: Pex - Automated Exploratory Testing for .NET

JorisW — Tue, 15 Jul 2008 22:53:40 GMT

I've been playing with it for a while. Even though I know I shouldn't use it with production code (this is a research project), but I had to run it against real, production code to see some real results. Seems like me and my fellow team members forgot to write some very important tests :)

I hope to find this in a future release of Visual Studio.

Re: Re: Pex - Automated Exploratory Testing for .NET

aL_ — Sat, 12 Jul 2008 15:47:59 GMT

great stuff!
thanks :)

Re: Pex - Automated Exploratory Testing for .NET

peli — Fri, 11 Jul 2008 13:55:18 GMT

> is there a way to get a certin pexmethod to use one factory and another (or all others) to use some otherway of exploring classes?

There is no automated way and only 1 factory per type is supported.

> or perhaps that factory souhld be a pex factory but just a regular static method that is called from the parameterized test?
Exactly.

> also, if you want your custom classes to appear in the table of results and arguemnts in visual studio, should you use PexValue.AddForValidation?

You can simply use PexValue.Add:

PexValue.Add("name", myobject.ToString());

By default, Pex does not call the ToString() method of user defined types, because it might stack overflow... and kill the process. Moreover, it might have sideeffect that would change the test behavior.

PexValue.AddForValidation works similarly (it adds the value in the parameter table) but it also tells Pex to encode the observed value in the generated test for regression testing.

Re: Pex - Automated Exploratory Testing for .NET

aL_ — Fri, 11 Jul 2008 10:42:56 GMT

had some more fun with pex :)

is there a way to get a certin pexmethod to use one factory and another (or all others) to use some otherway of exploring classes?

or perhaps that factory souhld be a pex factory but just a regular static method that is called from the parameterized test?

also, if you want your custom classes to appear in the table of results and arguemnts in visual studio, should you use PexValue.AddForValidation? (for arguments that are custom classes as well?) or is there some other way to do that?

Re: Pex - Automated Exploratory Testing for .NET

peli — Mon, 07 Jul 2008 14:41:06 GMT

If you're willing to dig deeper, Pex is extensible. You could implement your own [PexMethod] variation that performs the custom databinding. Unfortunately, there's no sample on this for now :)

Re: Re: Pex - Automated Exploratory Testing for .NET

aL_ — Mon, 07 Jul 2008 07:38:47 GMT

--
staging method: if the code that is being run in the method is relevant, you really want to call it from the parameterized test. Otherwise, you can use [TestInitialize], [TestCleanup] methods, they are supported by Pex.
--

alright i'll try that :)

--
Since you are mentioning the environment, be aware that Pex does not understand the environment (i.e. Pex cannot build constraint over the state of the file system, network, current time, etc...).
--
no i understand that :) that would be kind of spooky.. :S and awsome :P that is why it would be cool if one could supply a list of alternatives somehow :)

--
Partial databinding: interresting idea but it's not supported as is right now. Currently, create multiple copies of the test or use the 'Abstract Test Pattern'.
--
alright :) one could work around it by adding an int to te parameterized test, assume its within the range of a collection defined somewhere and use the int to look up the alternatives from that collection, but i guess pex wouldnt know that its supposed to use all the ints in the range.. you could also use if statements though, then pex would try to satisfy them :) but if there are alot of alternatives that might be cumbersome..
perhaps a dictionary with an enumeration as a key? im just letting my mind wander.. :D

again, great work pex team :)

Re: Pex - Automated Exploratory Testing for .NET

peli — Mon, 07 Jul 2008 05:29:49 GMT

staging method: if the code that is being run in the method is relevant, you really want to call it from the parameterized test. Otherwise, you can use [TestInitialize], [TestCleanup] methods, they are supported by Pex.

Since you are mentioning the environment, be aware that Pex does not understand the environment (i.e. Pex cannot build constraint over the state of the file system, network, current time, etc...).

Partial databinding: interresting idea but it's not supported as is right now. Currently, create multiple copies of the test or use the 'Abstract Test Pattern'.

Re: Re: Pex - Automated Exploratory Testing for .NET

aL_ — Sun, 06 Jul 2008 23:19:35 GMT

aaah, awsome :) and i see it also takes multiple parameters too, sweet :)

is there also a way to have a staging method be called before each test or at the start of each test run?
by stageing method i mean a static method that sets up the environment in diffrent ways. i guess i could call it from within the parameterized test but maybe the pex team thought of something more clever here as well :)

im sorry if thats in the docs somewhere or in the ms test api thats built into vs, im sort of a noob with this kind of testing... :) im used to writing eeeverything myself.. :S that is also why pex makes me ever so happy :D

maybe that isnt even a smaart thing to do.. :) i guess im sort of trying to do some mini system tests in my unit tests..:) but i figure, whats a unit anyway.. a part of a system could be a unit even if its actually a system in it self right? :)

oh i got another question too
is there a way to have the parameterized test pick some of the parameters from somewhere (like a collection) and try and figure the other ones out? like if i have a method that takes some arguments and a file path to like a config file and i want it to try out a few diffrent config files with various permutations of the other arguments to the method
so basically what i waant to do is databind to some of the arguments but have pex figure out the others :)

Re: Pex - Automated Exploratory Testing for .NET

peli — Sun, 06 Jul 2008 20:58:02 GMT

Pex actually supports generic unit tests: add generic arguments to the unit test and specify the instanciation types using 'PexUseGenericArgumentsAttribute':

    [PexMethod]
    [PexUseGenericArguments(typeof(int))]
    public void AddItem(List<T> target, T item) {
       ...
    }

Re: Pex - Automated Exploratory Testing for .NET

aL_ — Sun, 06 Jul 2008 17:53:11 GMT

ive been playing around a lille with pex now and i just have to say.. WOW :O
its GREAT! it sounded cool in the interview and all but in reality its even more awsome!
if you havent already, you Have to check this out! the vs integration is sweeet :)
the last time i had this kind of experience was when i tried out wpf! :) unit testing is no longer a horrible bore but actually something interesting :)

again, great work pex team! cant wait for the next release :)

i do have a qustion though :) is there a way to get pex to try a generic method with a bunch of diffrent types? i guess i could use like a list of instances of the various types and get pex to use diffrent lists or diffrent indexes in the list (like with a static list and a int argument to the parameterized method)

but im sure there is a smarter way :)

Re: Pex - Automated Exploratory Testing for .NET

peli — Sun, 06 Jul 2008 06:21:36 GMT

Good news: we are working on a WOW64 version of Pex (support for exploring 32-bit .net processes on 64-bit).

Re: Pex - Automated Exploratory Testing for .NET

Borum.NET — Sat, 05 Jul 2008 20:48:16 GMT

Too bad about the 64-bit version. We're more and more developers running in 64-bit environments (even in development), so it would really really nice to have access to the entire suite of tools. Running in a virtual environment when doing unit testing is probably one possible workaround, but I would definitely welcome 64-bit support.

I've subscribed to the PEX list but haven't seen any trafic yet.

Automated unittesting is worth dying for if done right. Can't really think of any code that wouldn't benefit (I suspect they even used Pex to test Pex :-)).

Re: Pex - Automated Exploratory Testing for .NET

stevo_ — Sat, 05 Jul 2008 18:07:20 GMT

Been using pex a little (not had time to really get into it yet), but its really impressive.. great work on this guys!

Re: Re: Pex - Automated Exploratory Testing for .NET

ben2004uk — Fri, 04 Jul 2008 23:03:42 GMT

As far as I am aware, 64bit version isn't planned any time soon. My suggestion - use a 32bit virtual machine.

Re: Pex - Automated Exploratory Testing for .NET

Borum.NET — Fri, 04 Jul 2008 20:32:21 GMT

Nikolai and Peli,

Very interesting technology - looking forward to v1.0 and a 64-bit version. Speaking of 64-bit, when will you release a beta that supports that version of the framework?

Re: Pex - Automated Exploratory Testing for .NET

Stebet — Fri, 04 Jul 2008 11:43:02 GMT

This is the coolest thing I've seen in a long time.

For those interested here are the blogs from Nikolai and Peli:
http://blogs.msdn.com/nikolait
http://blog.dotnetwiki.org/default.aspx

I can't wait for this to hit 1.0!

Re: Pex - Automated Exploratory Testing for .NET

aL_ — Thu, 03 Jul 2008 22:38:40 GMT

thats so awsome :)

i like the floating "we catch your bugs" note in the background too ;)