On Continuous Testing, bling, and the quest for immediate feedback

My usual MO is not to engage in debates such as the recent heated HerdingCode/NCrunch vs Mighty Moose debate, but I can’t resist sharing my view regarding the issue at hand, as it concerns a tool I wrote called Continuous Testing for Visual Studio.

First of all, I want to emphasize that I think both of the aforementioned tools are great efforts, and most certainly provide value to those using them.

Second, the reason for being a passive listener is that I believe debates of this kind are utterly meaningless and provide no value whatsoever; making and improving the tools at hand is what provides value. I will make an exception this time as this debate (among other things) discusses a (if not the) fundamental motivation for writing Continuous Testing for Visual Studio, and highlights one of the significantly differentiating aspects of the approach it takes.

Third, and most importantly, the statement from the Herding Code podcast is objectively true. Put in even clearer words: It is impossible for a Continuous Testing tool to do static analysis and achieve perfect and correct test minimization. Even near perfect test minimization is impossible. The only generally possible thing is simple dependency resolution. Don’t let anyone tell you otherwise.

That is why Continuous Testing for Visual Studio does not do static analysis1, but relies purely on statistical heuristics to prioritize and give as immediate feedback as it possibly can, while still guaranteeing you the correct feedback.

Note the highlit words in the previous sentence. Immediate while correct. That is the core philosophy of Continuous Testing for Visual Studio. In the end, correct feedback matters infinitely much more than fast feedback when it comes to testing. Continuous Testing for Visual Studio tries to give you as immediate feedback as it can, while still giving you correct feedback.

It is up to you, dear reader, to try out the tools and find out which works best for you in your real world project. Try Continuous Testing for Visual Studio. While you do that, I’ll continue improving it.

1 We obviously do static analysis to resolve dependencies etc., but not for optimizing away tests.