The other night is was playing around with a side project. I try to use a rather strict TDD approach for these projects, and so I run my tests a lot of times as I move forward, and spend quite some time waiting for the result before I move forward. This is a tedious and frankly unnecessary manual step; what I needed was continuous testing – unit tests that test themselves continuously, making sure I don’t break anything.
I remembered reading about JUnit Max by Kent Beck, a continuous testing plugin for Eclipse, that runs your unit tests in the background and unobtrusively tells you when a test fails, allowing you to do what you do best: write code. JUnit Max seems like a great thing, and now I needed the same thing for Visual Studio. A quick Google didn’t yield any add-ins, extensions or packages, so I decided to create one.
The result is Continuous Testing for Visual Studio, a small extension which runs your unit tests each time you build your solution, and reports failing tests to the error list so you can navigate to the line that failed and make the test pass. The extension significantly improves my workflow by removing a tedious manual step of running unit tests, so I encourage you to take it for a spin. Continuous Testing can be downloaded for Visual Studio 2008 and Visual Studio 2010. Future updates and versions will be announced on the Continuous Testing home page.
UPDATE Jun 17th, 2010: I’ve receive a lot of feedback through various solutions online. To be able to help you and/or improve Continuous Testing for Visual Studio, I need samples from you that reproduce the problems you are experiencing. Do not hesitate to leave a comment here, and provide your e-mail address when commenting, and you will receive a reply.
I installed the vs2010 version and I cannot figure out why it is not running my mstests in my project. I cannot figure out what needs to be done. It is enabled through the extension manager but guess I could use a little help getting going.
Also, would you consider open sourcing this?
@Eric Hexter It might be a bug. If you could provide me a sample solution reproducing your problem I would be happy to look into it.
I tried to install both the VS2008 and VS2010 versions. Both seem to install fine, but when I start op VS there’s no trace of the program, and nothing tin the “addins” section either.
Not sure how to provide a sample for that… since the plugin isn’t even loading.
I’m using default installs of both VS on a 64 bit windows 7
@Sebastian Vilstrup: They don’t show up as add-ins. Under VS10, see the Extension Manager. Under VS08, see the Package Load Analyzer. You should see Continuous Testing appearing in both.
To test your installation, simply build a solution containing a failing test and watch it appear in the error list.
Thank you for a wonderful product. It was only weeks ago I though it would be nice to have something like this, then presto, i find it on the VS2010 extensions list!
I am emailing you to ask about the error list.
If the error list is closed or auto-hidden when the continuous test runs automatically; I do not see the failed test. Is there a way to fix this?
Steps to reproduce. 1. Create a simple test “Assert.IsTrue(false);” 2. From the menu, view / errorlist to ensure the error list is showing. Ensure it is not auto-hiding. 3. Build and then view the error of the test failing. 4. Now close or auto-hide the error list. 5. Build and note that the error list does not show.
thanks
After a few mails back and forth with Håvard, we got it working for me in both VS2008 and VS2010. Good job, Håvard, it works fine now
Hey. I have installed your plugin into my MSVS2008. I also added to VS templates for NUniTestProject and NUnitTestFile. Now when I’m creating simple NUnit test, just from template, with all methods failing. When I’m building project/solution I get message ‘Test run failed. Enable tracing, rebuild, and check trace file for more information.’ but results are not displayed in VS. If I enable trace file in options it is creates file with results. If I run NUnit (with GUI) manually I see results. What I would like is co know how to configure your plugin, so I would get results in VS (no results are displayed in ErrorList or TestResults windows).
@yoosiba Could you send me the trace file and a sample solution that reproduces your problem?
I attempted to use the continuous integration plugin for VS 2010 on a project that has NUnit test defined in a separate assembly (named EnumUtilTests.dll) and whenever the continuous integration plugin attempts to run, nothing happens. I checked the trace logs and this is the output that gets produced:
– Trace started at 08/28/2010 17:37:18 – Microsoft (R) Test Execution Command Line Tool Version 10.0.30319.1 Copyright (c) Microsoft Corporation. All rights reserved.
Loading C:\Users\rmiesen\src-personal\PersonalProjects\Active\enumutils\trunk\src\EnumUtils\bin\Debug\EnumUtils.dll… Starting execution… No tests to execute. Process exited with code 0. – Trace closed at 08/28/2010 17:37:21 –
I did some digging on my own and found that my core project, EnumUtils (generates an assembly called “EnumUtils.dll”) was referencing the Microsoft.VisualStudio.QualityTools assembly. When I deleted that and recompiled, the continuous testing plugin still didn’t run.
Additional information: — I installed the Continuous Testing plugin 08/19/2010 — I am running Visual Studio 2010. — The project in question is called the Enum Utilities Library and runs on the .NET framework, v4.0.
@rmiesen Thanks for reporting your problem. Sadly, I am not even able to run your test projects successfully in NUnits own test runner. I’ve tried running with both version 2.5.5 and 2.5.7 of NUnit, and both fail (miserably) with a runtime exception (2.5.5 fails when loading your test project, 2.5.7 fails miserably with an engine execution exception when running tests). I don’t have time to research your test project issues further, but please make contact again when you have your EnumUtils test project running properly in NUnit. Again, thanks for reporting!
That’s very strange. On my machine, the Enum Utilities Library passes all regression tests. Since my tests pass on my machine but seem to be failing on your machine, I’d definitely appreciate hearing more from you about what specific problems your having. If you have time, could you review the system requirements and make sure you have all required software installed on your system? If you don’t, could you try it again? I’d also like to know what OS are you running.
I would appreciate any help you can offer.
@rmiesen Installed code contracts, currently works fine for me on the latest development snapshot. I suspect your problem with v1.0.3 is an issue with solution folders that has been reported. Basically, if your test project has folders, Continuous Testing will fail during test detection. We will release a fix for this soon.
@Håvard: thank you for your follow up. I am glad to hear that the regression tests all pass now. I’ll be sure to keep following your Continuous Testing add-in.