On Continuous Testing, bling, and the quest for immediate feedback

My usual MO is not to engage in debates such as the recent heated HerdingCode/NCrunch vs Mighty Moose debate, but I can’t resist sharing my view regarding the issue at hand, as it concerns a tool I wrote called Continuous Testing for Visual Studio.

First of all, I want to emphasize that I think both of the aforementioned tools are great efforts, and most certainly provide value to those using them.

Second, the reason for being a passive listener is that I believe debates of this kind are utterly meaningless and provide no value whatsoever; making and improving the tools at hand is what provides value. I will make an exception this time as this debate (among other things) discusses a (if not the) fundamental motivation for writing Continuous Testing for Visual Studio, and highlights one of the significantly differentiating aspects of the approach it takes.

Third, and most importantly, the statement from the Herding Code podcast is objectively true. Put in even clearer words: It is impossible for a Continuous Testing tool to do static analysis and achieve perfect and correct test minimization. Even near perfect test minimization is impossible. The only generally possible thing is simple dependency resolution. Don’t let anyone tell you otherwise.

That is why Continuous Testing for Visual Studio does not do static analysis1, but relies purely on statistical heuristics to prioritize and give as immediate feedback as it possibly can, while still guaranteeing you the correct feedback.

Note the highlit words in the previous sentence. Immediate while correct. That is the core philosophy of Continuous Testing for Visual Studio. In the end, correct feedback matters infinitely much more than fast feedback when it comes to testing. Continuous Testing for Visual Studio tries to give you as immediate feedback as it can, while still giving you correct feedback.

It is up to you, dear reader, to try out the tools and find out which works best for you in your real world project. Try Continuous Testing for Visual Studio. While you do that, I’ll continue improving it.

1 We obviously do static analysis to resolve dependencies etc., but not for optimizing away tests.

Continuous Testing 2.0 is here

I am happy to announce the immediate availability of Continuous Testing 2.0 for Visual Studio 2010 and 2008.

From now on, Continuous Testing for Visual Studio will be available in two flavors: A full-featured Professional version and a free version staying more or less as it is today.

Continuous Testing 2.0 Buy Professional Download for VS2010 Download for VS 2008

The new Professional version features a number of improvements over the free version:

  • Smart Test Run Prioritization records the outcome of each of your tests, and orders the next test runs to minimize your wait for feedback even further. Smart Test Run Prioritization will run your tests in order of most recently failing and most often failing. This provides a significant speed-up for larger projects.
  • Abort on first failing test: You can choose to have Continuous Testing abort as soon as a test fails. This is useful for freeing up resources acquired by the test run as soon as feedback is available. This is another must-have feature in real world software development on large projects.
  • Priority support: Send an e-mail and get a response within 24 hours.

The combined features of Continous Testing Professional make for a very powerful and efficient TDD workflow.

Ensuring the last failing test runs first and aborting as soon as a test fails enables you to do as much as possible of what you are supposed to do: Write code.

The free version will continue to stay free with at least the current features. Here are the most notable new features of the free version:

  • A new integrated version of the test runners for MSTest and NUnit speeds up test runs. The executable test runners are still available. If you experience problems with running your tests using the integrated runners, try running your tests using the executable test runners instead.
  • Continuous Testing can be told to ignore specific projects when scanning for tests. This is useful if you are working on a solution with many test projects, and only want to run some of those projects.
  • Continuous Testing now has its own Output Window pane. Watch this pane to see what Continuous Testing is doing.

In addition, many improvements and bug fixes have made it into the release, ensuring a more stable and productive experience than before.

Happy testing and TDD-ing!

Introducing JSUT – Cross-platform JavaScript Unit Testing

I write quite a bit of JavaScript these days, and one observation I have made with regards to Line-of-Business apps is that platforms like Node.js makes me lean more towards sharing business logic between the server and client side (meaning Node.js and the browsers, in practice). This creates an immediate need to unit test “universally” (across all platforms and environments), and I couldn’t find a single tool that quite did it for me.

JSUT (JavaScript Unit Testing) is an attempt at making such a tool for fun and profit (non-monetary).

JSUT stems from me growing tired of the hurdles of unit testing my JavaScript code. There are tons of alternative unit testing tools and libraries readily available, but my experience was that they all exhibited one or more of the following shortcomings:

  • Imposing a “framework” or “philosophy” or “style” on my tests
  • Bringing me additional dependencies to worry about
  • Working only in the browser or inside Node.js

In addition, I feel that many of the available tools and libraries are either undermaintained or overly featureful, and buggy and/or painful to use as a result of one or both of those. I can’t and won’t claim that this will be different in the case of JSUT, so it’s not really an argument, but more a motivation for me.

Goals

I sat down and wrote a few goals for JSUT:

  • Universal, cross-platform
  • Minimalistic, simple, non-constraining
  • Accessible

I then took to writing code and brushing up my rather rusty shell scripting skills, and a few hours later I wound up with what I present to you now.

JSUT – JavaScript Unit Testing – An introduction

Writing a test in JSUT is about as simple as it can get. You write a function:

function myTest(test) {
  // JSUTs only requirement:
  // Call test.done() when the test is done
  test.done(); 
}

Now, let’s say you save this function to a file called test.js. You can then move to your preferred shell and write the following:

jsut -b chrome test.js

This will run your test in Chrome. To run your test in Node.js, do:

jsut -n test.js

You can combine the -b and -n flags any way you like, of course.

Getting JSUT

Getting JSUT is very simple. Assuming you have npm, just do:

npm install jsut

Which will install jsut locally. If you want a global installation, issue:

npm install jsut -g

Of course, with the goals for JSUT, you can also just download and copy deploy JSUT. Visit http://github.com/havard/jsut for source code and further instructions.

Requirements

As mentioned in the above sample, the only requirement JSUT imposes on you is to call test.done() when your test is done. Obviously, this seems redundant for the simple case of a synchronously executing function, but since most JavaScript functions these days are asynchronous at some level, there has to be a way to tell JSUT that a test is done. I am actively considering easing this requirement for simple cases, but this has not been a priority so far.

For Node.js, you are required to add the functions you want to test to the exports of each test file. Building on the example above, you can maintain cross-platform compatibility by doing the following:

if (typeof(exports) !== 'undefined') {
  exports.myTest = myTest;
}

This will add exports only if the exports variable is defined. Now, if you wanted to run your test in Node.js, you simply do:

jsut -n test.js

This will run your test in Node.js. You are free to combine the -n and -b flags any way you like, so that you can target all required platforms with a single command.

Assertions

JSUT automatically injects a CommonJS compatible assert module in your browser tests, so you can write stuff like assert.ok(true); anywhere in your test.

Planned features

JSUT currently only runs browser tests on OS X. I plan to support Linux and Windows as soon as possible. Follow me and JSUT on GitHub!

Problems running .NET 4.0 tests using NUnit 2.5 with (or without) Continuous Testing?

NUnit 2.5 up to and including 2.5.4 fails to properly detect certain revisions of .NET 4.0, causing it to crash. The issue is documented in this bug report on LaunchPad. The bug has been fixed in 2.5.5 and later, so go grab the latest version of NUnit 2.5 and run your tests smoothly again, preferably on each build using Continuous Testing, of course. :)

Continuous Testing for Visual Studio

The other night is was playing around with a side project. I try to use a rather strict TDD approach for these projects, and so I run my tests a lot of times as I move forward, and spend quite some time waiting for the result before I move forward. This is a tedious and frankly unnecessary manual step; what I needed was continuous testing – unit tests that test themselves continuously, making sure I don’t break anything.

I remembered reading about JUnit Max by Kent Beck, a continuous testing plugin for Eclipse, that runs your unit tests in the background and unobtrusively tells you when a test fails, allowing you to do what you do best: write code. JUnit Max seems like a great thing, and now I needed the same thing for Visual Studio. A quick Google didn’t yield any add-ins, extensions or packages, so I decided to create one.

The result is Continuous Testing for Visual Studio, a small extension which runs your unit tests each time you build your solution, and reports failing tests to the error list so you can navigate to the line that failed and make the test pass. The extension significantly improves my workflow by removing a tedious manual step of running unit tests, so I encourage you to take it for a spin. Continuous Testing can be downloaded for Visual Studio 2008 and Visual Studio 2010. Future updates and versions will be announced on the Continuous Testing home page.

UPDATE Jun 17th, 2010: I’ve receive a lot of feedback through various solutions online. To be able to help you and/or improve Continuous Testing for Visual Studio, I need samples from you that reproduce the problems you are experiencing. Do not hesitate to leave a comment here, and provide your e-mail address when commenting, and you will receive a reply.