Continuous Testing for Visual Studio

Posted on by

The other night is was playing around with a side project. I try to use a rather strict TDD approach for these projects, and so I run my tests a lot of times as I move forward, and spend quite some time waiting for the result before I move forward. This is a tedious and frankly unnecessary manual step; what I needed was continuous testing – unit tests that test themselves continuously, making sure I don’t break anything.

I remembered reading about JUnit Max by Kent Beck, a continuous testing plugin for Eclipse, that runs your unit tests in the background and unobtrusively tells you when a test fails, allowing you to do what you do best: write code. JUnit Max seems like a great thing, and now I needed the same thing for Visual Studio. A quick Google didn’t yield any add-ins, extensions or packages, so I decided to create one.

The result is Continuous Testing for Visual Studio, a small extension which runs your unit tests each time you build your solution, and reports failing tests to the error list so you can navigate to the line that failed and make the test pass. The extension significantly improves my workflow by removing a tedious manual step of running unit tests, so I encourage you to take it for a spin. Continuous Testing can be downloaded for Visual Studio 2008 and Visual Studio 2010. Future updates and versions will be announced on the Continuous Testing home page.

UPDATE Jun 17th, 2010: I’ve receive a lot of feedback through various solutions online. To be able to help you and/or improve Continuous Testing for Visual Studio, I need samples from you that reproduce the problems you are experiencing. Do not hesitate to leave a comment here, and provide your e-mail address when commenting, and you will receive a reply.

  • http://hex.lostechies.com Eric Hexter

    I installed the vs2010 version and I cannot figure out why it is not running my mstests in my project. I cannot figure out what needs to be done. It is enabled through the extension manager but guess I could use a little help getting going.

    Also, would you consider open sourcing this?

  • http://ox.no Håvard

    @Eric Hexter It might be a bug. If you could provide me a sample solution reproducing your problem I would be happy to look into it. :)

  • http://makeitfaster.wordpress.com Sebastian Vilstrup

    I tried to install both the VS2008 and VS2010 versions. Both seem to install fine, but when I start op VS there’s no trace of the program, and nothing tin the “addins” section either.

    Not sure how to provide a sample for that… since the plugin isn’t even loading.

    I’m using default installs of both VS on a 64 bit windows 7

  • http://ox.no Håvard

    @Sebastian Vilstrup: They don’t show up as add-ins. Under VS10, see the Extension Manager. Under VS08, see the Package Load Analyzer. You should see Continuous Testing appearing in both.

    To test your installation, simply build a solution containing a failing test and watch it appear in the error list. :)

  • Valamas

    Thank you for a wonderful product. It was only weeks ago I though it would be nice to have something like this, then presto, i find it on the VS2010 extensions list!

    I am emailing you to ask about the error list.

    If the error list is closed or auto-hidden when the continuous test runs automatically; I do not see the failed test. Is there a way to fix this?

    Steps to reproduce. 1. Create a simple test “Assert.IsTrue(false);” 2. From the menu, view / errorlist to ensure the error list is showing. Ensure it is not auto-hiding. 3. Build and then view the error of the test failing. 4. Now close or auto-hide the error list. 5. Build and note that the error list does not show.

    thanks

  • http://makeitfaster.wordpress.com Sebastian Vilstrup

    After a few mails back and forth with Håvard, we got it working for me in both VS2008 and VS2010. Good job, Håvard, it works fine now :)

  • yoosiba

    Hey. I have installed your plugin into my MSVS2008. I also added to VS templates for NUniTestProject and NUnitTestFile. Now when I’m creating simple NUnit test, just from template, with all methods failing. When I’m building project/solution I get message ‘Test run failed. Enable tracing, rebuild, and check trace file for more information.’ but results are not displayed in VS. If I enable trace file in options it is creates file with results. If I run NUnit (with GUI) manually I see results. What I would like is co know how to configure your plugin, so I would get results in VS (no results are displayed in ErrorList or TestResults windows).

  • http://ox.no Håvard

    @yoosiba Could you send me the trace file and a sample solution that reproduces your problem?

  • rmiesen

    I attempted to use the continuous integration plugin for VS 2010 on a project that has NUnit test defined in a separate assembly (named EnumUtilTests.dll) and whenever the continuous integration plugin attempts to run, nothing happens. I checked the trace logs and this is the output that gets produced:

    – Trace started at 08/28/2010 17:37:18 – Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1 Copyright (c) Microsoft Corporation. All rights reserved.

    Loading C:\Users\rmiesen\src-personal\PersonalProjects\Active\enumutils\trunk\src\EnumUtils\bin\Debug\EnumUtils.dll… Starting execution… No tests to execute. Process exited with code 0. – Trace closed at 08/28/2010 17:37:21 –

    I did some digging on my own and found that my core project, EnumUtils (generates an assembly called “EnumUtils.dll”) was referencing the Microsoft.VisualStudio.QualityTools assembly. When I deleted that and recompiled, the continuous testing plugin still didn’t run.

    Additional information: — I installed the Continuous Testing plugin 08/19/2010 — I am running Visual Studio 2010. — The project in question is called the Enum Utilities Library and runs on the .NET framework, v4.0.

  • http://ox.no Håvard

    @rmiesen Thanks for reporting your problem. Sadly, I am not even able to run your test projects successfully in NUnits own test runner. I’ve tried running with both version 2.5.5 and 2.5.7 of NUnit, and both fail (miserably) with a runtime exception (2.5.5 fails when loading your test project, 2.5.7 fails miserably with an engine execution exception when running tests). I don’t have time to research your test project issues further, but please make contact again when you have your EnumUtils test project running properly in NUnit. Again, thanks for reporting!

  • rmiesen

    That’s very strange. On my machine, the Enum Utilities Library passes all regression tests. Since my tests pass on my machine but seem to be failing on your machine, I’d definitely appreciate hearing more from you about what specific problems your having. If you have time, could you review the system requirements and make sure you have all required software installed on your system? If you don’t, could you try it again? I’d also like to know what OS are you running.

    I would appreciate any help you can offer.

  • http://ox.no Håvard

    @rmiesen Installed code contracts, currently works fine for me on the latest development snapshot. I suspect your problem with v1.0.3 is an issue with solution folders that has been reported. Basically, if your test project has folders, Continuous Testing will fail during test detection. We will release a fix for this soon.

  • rmiesen

    @Håvard: thank you for your follow up. I am glad to hear that the regression tests all pass now. I’ll be sure to keep following your Continuous Testing add-in.

  • http://blog.soerenlarsen.dk Søren Larsen

    Hi Håvard,

    I have noticed that the extension only runs test in the first test project in a solution. In larger projects there often are 2 or more test projects.

    Do you think you can fix this issue?

    Here is a link to a sample solution including a test log: http://dl.dropbox.com/u/137976/TestProject2.zip

    Regards Søren

  • http://ox.no Håvard

    Hi Søren,

    Thanks for reporting! Problem has been reproduced, and a fix has been made and will be included in the next release.

  • Adis

    Hi Håvard

    according to my understanding an extension expects tests have to live in same project where the code they test lives in. Is that by design or is there any setting in order to share code and test on two different projects?

    thanks Adis

  • http://ox.no Håvard

    @Adis No, Continuous Testing can run tests from any test project organization. It does not require test code and code under test to be in the same assembly.

  • Robert Hanson

    No unit tests run under the following scenario 1) Create a ASP.NET website. 2) Follow instructions here (http://msdn.microsoft.com/en-us/library/ms182526(v=vs.80).aspx) to add unit tests 3) Compile and run unit tests using visual studio 4) Build website – no unit tests apparently run; I tried creating a failing test too just to demonstrate that it is not working.

    Does the CT runner work with a website build? or do I need to build the test project separately?

  • http://ox.no Håvard

    @Robert Hanson:

    Thank you for reporting this issue.

    It is a known bug, which will be fixed in the next version.

    The bug causes Continuous Testing to fail for solutions containing web sites to fail. Until the next version is released, you need to avoid working with solutions containing web sites to use Continuous Testing.

    Obviously, this is not a great solution for you. The next version, which fixes the bug, is just around the corner, so we hope you will bear with us until then, and give us a new chance! Until then, you can disable Continuous Testing simply by unchecking “Run tests on successful build” under Tools > Options > Test Tools > Continuous Testing.

  • http://richardminerich.com Rick Minerich

    Tests don’t seem to run in my F# projects using xUnit 1.6.1. There’s no error, I don’t think it’s even getting triggered. Here’s an example of a simple F# test assembly:

    module XunitTest

    open Xunit

    [] let newtest () = Assert.Equal (null, null)

  • http://richardminerich.com Rick Minerich

    It appears that the comment system mangled my submission. Let’s try again with html formatting:

    module XunitTest

    open Xunit

    [<Fact>] let newtest () =     Assert.Equal (null, null)

  • http://ox.no Håvard

    @Rick Minerich For reference, I reply here as well. As we discussed on Twitter, the reason for your problem is that your project contains folders. The current version has issues with projects with folders in them. We have a fix upstream which will be in the next version. Thanks for reporting! :)

  • James Bradt

    Fatal unit testing error

    Good day, I’m attempting to try out this add-in (vs2010).

    1) create a new vb command line project 2) add a class 3) create a new vb test project 4) add a new test 5) build the solution

    At the bottom of the IDE, the following message flashes quickly

    Fatal unit testing error: Unable to cast COM object of type ‘System._ComObject’ to interface type ‘VSLangProj.VSProject’. This operation failed because the QueryInterface call on the COM component for the interface with IID ‘{2CFBB26F-F6BF-480D-A546-95A0381CC411}’ failed due to the following error: No such interface supported (Excepti…

    That is all I can capture.

    Manually clicking the ‘Run All Tests in the Solution’ button on the toolbar does work (and the simple default mstest passes).

    Any suggestions on how to correct this?

    Thanks, JB

  • Necip Karakas

    Hi, As I wrote in my twit, could not install this tool for VS2008. I have XP on my computer an get the following error. “Could not access network location: \\mydomainname\users\myusername\\Visual Studio 2008\Addins.” I have my Addin folder on a network drive. I think the problem happens because of the double backslashes after username . Thanks necip

  • http://ox.no Håvard

    @necip Thanks for reporting. This is a general issue relating to Visual Studio add-in installations (prior to VS 2010). You’ll need to work around by temporarily pointing your My Documents folder to a local drive, and then copying the add-in folder contents back to your network location after installing. See here for a tutorial.

  • Tony Wang

    I think I may have come across a bug with reporting the correct file in the Error List window. We have written extension methods around the NUnit Assert class to make our tests more “fluent.” When an assert fails, the file reference in the Error List is not the file containing the test, but rather the file of the extension method containing the Assert.

    I have prepared a sample project that illustrates this issue. Thanks!

  • http://ox.no Håvard

    @Tony Thanks for reporting. Technically this is not a bug, since the error list reference points to where the exception originated. We do however understand the expectation that you should end up at the line in the test from which the exception originated, and we will consider changing this behavior.