Using Smart Unit Tests To Find Bugs Earlier In The Development Cycle

Play Using Smart Unit Tests To Find Bugs Earlier In The Development Cycle
Sign in to queue

Description

Smart Unit Tests help you find bugs earlier in the development cycle. See how this Microsoft Research project walks through code paths of existing applications to create these new unit tests.

Embed

Download

The Discussion

  • User profile image
    satish860

    Please don't release this . It will create unnecessary confusion between writing a unit test and generating. I dont think this will help the application in any way and as well it has potential to create a very adverse effect in the code 

  • User profile image
    BrianLowe

    What am I missing? It looks like you've built a robot that looks at my current code (including logic errors) and writes tests that assert what it finds by traversing all possible paths, then runs them to show how clever it is.

    If it finds a path that multiplies two given arguments won't your robot write a test to verify that? even though the intent was to add the arguments? If you don't know the requirement, how can you verify the implementation?

    A test that passes when it should fail is worse than no test.

     

  • User profile image
    Algorithms​AreCool

    I'll contend Satish and Brian's assertions about this being harmful. I think it can be a big timesaving tool for getting off the ground with unit testing. I agree that this obviously can't generate tests to validate business rules, but this does generate thorough tests that can find bugs for malformed input. 

    More importantly, if you have inherited code that has very few or no unit tests, this can assist in creating those tests before you modify unfamiliar areas of the codebase. It can provide a safety net for validating that odd code flows are still correct. If your new code doesn't pass the auto generated tests you can validate that your changes are in fact correct before altering the tests to reflect the new implementation.

  • User profile image
    dgschrei

    Can we please call this feature "Stupid Unit Tests"?

    There is nothing smart about this. All it does is generate unit tests that assert that whatever is written in the method is properly executed by the CLR. 

    It has no idea whether the code inside the method is faulty or not. It might as well just generate tests that have an Assert.True() at the end. It would not make a difference.

    If you have the following:

    public int Multipy(int a, int b)
    {  return Math.Pow(a,b); }

    your tool will not find the error. All it will do, is write tests that prove that the framework's Pow function works as expected. It will stay completely oblivious to the fact that the method was originally supposed to multiply the two parameters. 

    And that's just a very very obvious example. Let's assume you have amethod that is supposed to filter an input string for sql injection attacks. This method calls several subroutines for the different attack possibilities. And somehow you forgot to add the call to the "CheckForDropDatabase" subroutine. Again your tool will fail to detect this. But it will happily report 100% code coverage for the method and suggest that everything is ok and working. 

    That is until your customer's database was wiped in an attack and whatever service he was running was disrupted until the hopefully very recent backup of the database has been restored to the system. 

    Please like satish860 said: Don't release this. You derive tests from acceptance criteria. You do NOT derive them from a potentially faulty state of code by running some algorithm. 

     

  • User profile image
    pvlakshm

    @Satish360, @BrianLowe, @dgschrei,
    Let me try to clarify:
    "Smart Unit Tests" discovers all the code paths in your code, synthesizes precise test input values that exercise those code paths, and then records what was the output from your code for the said inputs. It then persists these as a compact test suite with high coverage.

    "Smart Unit Tests" can be used at two levels:
    (1) By default, if you do nothing more than just run it on a piece of code, the test suite captures the observed behaviour of your code for each of the synthesized inputs. At this stage, except for tests causing runtime errors, the remaining are deemed to be passing tests - after all that is the observed behaviour. This is especially useful when dealing with a code base without tests or a test oracle. You can baseline the current/observed behaviour of your code as a suite of tests for use as a regression suite.
    (2) Additionally, if you write assertions expressing the correctness properties of your code, then it will come up with test inputs that can cause the assertions to fail as well (each such input representing a bug in your code, and thereby a failing test). Smart Unit Tests cannot come up with such correctness proprieties by itself. You as a developer would write them based on your domain knowledge.

    As @AlgorithmsAreCool mentions, Smart Unit Tests are meant to augment any existing testing practices you follow, not replace them.

    Here are some more links for context:
    The recent Connect session on Evolving software engineering practices: https://channel9.msdn.com/Events/Visual-Studio/Connect-event-2014/015 (especially from 17:20s to 19:45s)
    MSDN Documentation: https://msdn.microsoft.com/library/dn823749%28v=vs.140%29.aspx

    Stay tuned for more videos, documentation, and blog posts.

  • User profile image
    Mohamed​Abdeen

    it is really amazing feature i like in visual studio 2015 also it will encourage more developers to go and user unit test.

  • User profile image
    ttt

    Regarding "stupid tests": This is simply Microsoft Pex renamed. Search for that name to find what it is and any it is useful.

    This tool allows you to assert properties over *all* possible inputs, not just over a few hard-coded ones.

    The demo was really bad. It focuses on totally unimportant parts of the product such as factories. Totally not the point of what Pex does.

  • User profile image
    pvlakshm

    Thank you for the feedback. We will work on more content that showcases capabilities of the feature beyond that shown in the video.

    Regarding factories you might have already observed this, but I thought I would mention in nonetheless: Creating an object from a class can get quite involved. For e.g. there might be private fields that can only be set through the constructor and/or through some invocation sequence of other state-modifying methods (such was the case with the Employee class used in the video). "Smart Unit Tests" helps you out by trying to figure that out on your behalf, and reports that as the factory it will use. You can then augment this as you deem appropriate.

  • User profile image
    savagemonit​or

    This will be a very good tool for exercising code paths to ensure that scenarios are covered.  Good programmers will likely leverage this as a tool to close unit test gaps for scenarios they did not consider.  Unfortunately, I feel that this will also produce lazy programmers who simply write a unit of code, auto-gen the tests, and then move on having "completed" the coding task (presuming you require unit tests as part of check-ins).  After all, developers get acclaims for delivering features and not for having thorough unit tests.

  • User profile image
    KubuS

    IMHO it is a very useful feature. It can be abused, true, but it also helps to automate some of the more mundane tests.

    It might be useful to rename the tool from 'Smart Unit Tests' to 'Infer Unit Tests':

    1. it changes adjective into a verb i.e. meaningless and somewhat misleading description into a very descriptive action
    2. it shows better the true nature of the feature, after all PEX does exactly that - it infers inputs

    It would be great if this feature could use information contained in Code Contracts as they provide an additional information about the requirements. I use Code Contracts whenever I can in all my new code and it would be fantastic if at some point Visual Studio would just pick this information up and helped me with unit testing.

  • User profile image
    Patrick

    I really like that feature. Of course, this should not be used without having some human to create additional unit tests. People that are complaining seems to think that tool replace all unit tests. In fact, it is a tool. A tool is there to help you be more productive. These simple tests generated help you test basic scenario. For more complex one, you have to create them. Nice job Visual Studio. Keep innovating.

  • User profile image
    deef

    @satish860: It's a VS Ultimate only feature from what I can see on the MSDN page. So they are releasing it to that many people. Which I think is very unfortunate, but gotta sell the higher SKUs also right....

  • User profile image
    devlife3

    @Patrick:Agreed!

  • User profile image
    Monomachus

    How is this different from what we had years ago https://research.microsoft.com/en-us/projects/pex/? 

    Thank you,

    Timur

  • User profile image
    pvlakshm

    @Timur, please see reply to a similar question from @NickWilton_TullettPrebon on the post here: https://blogs.msdn.com/b/visualstudioalm/archive/2014/11/19/introducing-smart-unit-tests.aspx#10575495.

    Also we talk about this on the .NET Rocks! show as well. You can listen in here: http://www.dotnetrocks.com/default.aspx?showNum=1065.

Add Your 2 Cents