API changes

The API of the entire thing has changed. All callbacks exposed and used by the library are now on the form callback(error, args). This convention is familiar to Node.js developers, and gives us the advantage of simpler integration with other libraries, as well as a slight increase in the fidelity of error messages.

What has changed?

• authenticate now expects a callback which will be called with (error, authUrl). Existing code which expects (authUrl) will not get an authUrl (but possibly an error instead), and so authentication won’t work after upgrade until you change your code.
• verifyAssertion also expects a callback of the same form, which will be called with (error, result). Again, existing code depends on receiving a (result) call, and will break.
• loadAssociation and saveAssociation must now accept a callback on the form (error, result) as an additional parameter. Previously, these functions were synchronous, so existing implementations will have to adapt so they call the callback rather than return.

What is new?

As usual, we have fixed several bugs. The library is gaining a community, and there have been several contributions along the 0.1 release path leading up to 0.2.0. Thank you all for your participation!

Apart from the changes above, 0.2.0 also introduces a discovery cache for caching discovered information. This cache is used to avoid additional HTTP requests when verifying assertions from the OpenID providers. The default cache is an in-memory cache, but you can implement your own by overloading two functions:

• loadDiscoveredInformation(claimedIdentifier, callback) is expected to look for and load a cached provider for the given claimedIdentifier. It is expected to call callback(error, provider) when finished, with either an error or a provider (duh).
• saveDiscoveredInformation(provider, callback) is expected to cache the given provider. The key for this object is provider.claimedIdentifier, which loadDiscoveredInformation uses for lookup. After saving, the function is expected to call callback(error) with an error if something failed, or nothing if all went well.

I have been very reluctant to make these significant API changes, but the recent development, reported issues, and feature requests have convinced me that a unified API, and a more familiar API for Node.js developers, is the right way. Also, adapting to the new API is a pretty easy process. I hope you agree with me that the introduction of these changes is best in the long run.

A brief API overview:

• crypto.createDiffieHellman(prime_length)
  • Creates a Diffie-Hellman key exchange object and generates a prime of the given bit length. The generator used is 2.
• crypto.createDiffieHellman(prime, encoding='binary')
  • Creates a Diffie-Hellman key exchange object using the supplied prime. The generator used is 2. Encoding can be 'binary', 'hex', or 'base64'.
• diffieHellman.generateKeys(encoding='binary')
  • Generates private and public Diffie-Hellman key values, and returns the public key in the specified encoding. This key should be transferred to the other party. Encoding can be 'binary', 'hex', or 'base64'.
• diffieHellman.computeSecret(other_public_key, input_encoding='binary', output_encoding=input_encoding)
  • Computes the shared secret using other_public_key as the other party’s public key and returns the computed shared secret. Supplied key is interpreted using specified input_encoding, and secret is encoded using specified output_encoding. Encodings can be 'binary', 'hex', or 'base64'. If no output encoding is given, the input encoding is used as output encoding.
• diffieHellman.getPrime(encoding='binary')
  • Returns the Diffie-Hellman prime in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.getGenerator(encoding='binary')
  • Returns the Diffie-Hellman prime in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.getPublicKey(encoding='binary')
  • Returns the Diffie-Hellman public key in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.getPrivateKey(encoding='binary')
  • Returns the Diffie-Hellman private key in the specified encoding, which can be 'binary', 'hex', or 'base64'.
• diffieHellman.setPublicKey(public_key, encoding='binary')
  • Sets the Diffie-Hellman public key. Key encoding can be 'binary', 'hex', or 'base64'.
• diffieHellman.setPrivateKey(public_key, encoding='binary')
  • Sets the Diffie-Hellman private key. Key encoding can be 'binary', 'hex', or 'base64'.

NOTE: The API is still subject to change.

I would appreciate getting a note if you actually do something useful with it. Play around with it and let me know what you think!

Installation

Installing is very simple using npm:

npm install openid 

If you don’t use npm, you can of course just download and require. Remember dependencies from the lib folder, and remember adding paths to require.paths if necessary.

Usage

One of the main design goals for the library has been simplicity. Using OpenID for node.js is pretty simple, here’s a minimal sample server:

var openid = require('openid');
var url = require('url');
var server = require('http').createServer(
    function(req, res)
    {
        var parsedUrl = url.parse(req.url, true);
        if(parsedUrl.pathname == '/verify')
        {
            // Verify identity assertion
            var result = openid.verifyAssertion(req); // or req.url
            res.writeHead(200);
            res.end(result.authenticated ? 'Success  ' : 'Failure  ');
        }
        else if(parsedUrl.pathname == '/authenticate')
        {
            // Resolve identifier, associate, build authentication URL
            openid.authenticate(
                parsedUrl.query.openid_identifier, // user supplied identifier
                'http://example.com/verify', // our callback URL
                null, // realm (optional)
                false, // attempt immediate authentication first?
                function(authUrl)
                {
                    res.writeHead(302, { Location: authUrl });
                    res.end();
                });
        }
        else
        {
            // Deliver an OpenID form on all other URLs
            res.writeHead(200);
            res.end('<!DOCTYPE html><html><body>'
                + '<form method="get" action="/authenticate">'
                + '<p>Login using OpenID</p>'
                + '<input name="openid_identifier" />'
                + '<input type="submit" value="Login" />'
                + '</form></body></html>');
        }
    });
server.listen(80);

Source

I use GitHub for source control, so you can follow me and the node-openid repository there.

Licensing

OpenID for node.js is licensed under the MIT license. Details can be found in the LICENSE file on GitHub. The library includes third-party code released under the MIT and BSD licenses, see README for details.

Wishes for node.js

I’ve come across a couple of missing features which I’d love to see in node.js in the near future:

• The crypto module should support Diffie-Hellman key exchange (OpenSSL supports it, so please provide bindings)
• A big integer library should be available alongside the crypto module, and preferably seamlessly integrated with the crypto module – perhaps bindings to the GMP library could suffice?
• More seamless unit testing libraries would be good

In addition, I would love to see a tailored IDE for node.js applications.

public static HtmlHelper CreateHtmlHelper(ViewDataDictionary vd)
{
    Mock<ViewContext> mockViewContext = new Mock<ViewContext>(
        new ControllerContext(
            new Mock<HttpContextBase>().Object,
            new RouteData(),
            new Mock<ControllerBase>().Object),
        new Mock<IView>().Object,
        vd,
        new TempDataDictionary(),
        new Mock<TextWriter>().Object);
    var mockViewDataContainer = new Mock<IViewDataContainer>();
    mockViewDataContainer.Setup(v => v.ViewData)
        .Returns(vd);
    return new HtmlHelper(mockViewContext.Object,
                            mockViewDataContainer.Object);
}

I remembered reading about JUnit Max by Kent Beck, a continuous testing plugin for Eclipse, that runs your unit tests in the background and unobtrusively tells you when a test fails, allowing you to do what you do best: write code. JUnit Max seems like a great thing, and now I needed the same thing for Visual Studio. A quick Google didn’t yield any add-ins, extensions or packages, so I decided to create one.

The result is Continuous Testing for Visual Studio, a small extension which runs your unit tests each time you build your solution, and reports failing tests to the error list so you can navigate to the line that failed and make the test pass. The extension significantly improves my workflow by removing a tedious manual step of running unit tests, so I encourage you to take it for a spin. Continuous Testing can be downloaded for Visual Studio 2008 and Visual Studio 2010. Future updates and versions will be announced on the Continuous Testing home page.

UPDATE Jun 17th, 2010: I’ve receive a lot of feedback through various solutions online. To be able to help you and/or improve Continuous Testing for Visual Studio, I need samples from you that reproduce the problems you are experiencing. Do not hesitate to leave a comment here, and provide your e-mail address when commenting, and you will receive a reply.

The version I put up includes several enhancements from the latest release:

• It uses FormatterServices.GetUninitializedObject and hence does not depend on a parameterless constructor or custom instance provider (but you can of course still create an instance provider if you want to control object initialization)
• The bug with copy semantics for already visited objects submitted by Walter Oesch has been fixed
• The bug with inherited fields found by Alex, and the patch submitted for it, has been incorporated

Bleeding edge Copyable can be found at http://github.com/havard/copyable. The clone URL is git://github.com/havard/copyable.git. Now go fix your own bugs! Or even better, enhance the framework.

What is MapReduce?

For those of you who don’t know what MapReduce is: MapReduce is a simplified interface for parallel data processing. MapReduce was initially described by the Google engineers Jeffrey Dean and Sanjay Ghemawat in the 2004 paper titled MapReduce: Simplified data processing on large clusters.

MapReduce processes data by splitting the processing in to a set of transformations (in functional programming, this is called the “map” function (it maps or transforms an input to an output)). The results of the transformations are then combined into a single result (in functional programming, this is called the “reduce” function (it reduces a set of values to a single value)). On a sidenote, Linq has equivalent functions, but the names are different, presumably to make them more familiar to people with SQL knowledge. In Linq, map is called Select, and reduce is called Aggregate.

Shortly put, to process a huge set of data, you split the data into chunks and process each chunk in parallel. This eventually creates a new set of intermediary results, which is reduced to a single result.

Implementing a minimalistic MapReduce in .NET 4.0

The signature of my MapReduce function is

static Task<TResult> Start<TInput, TPartial, TResult>(
  Func<TInput, TPartial> map, 
  Func<TPartial[], TResult> reduce, 
  params TInput[] inputs);</pre>

In other words, to start a MapReduce run, you supply a map function, a reduce function, and a set of inputs. Each input will be turned into an intermediate result (of type TPartial). Inputs are transformed concurrently. When all inputs are transformed, the reduce function is called to transform the partial results into a final result (of type TResult). Cool!

The map part is implemented by starting a task for each supplied input using Task.Factory.StartNew().

Task.Factory.StartNew(() => map(input));

The reduce part is implemented as a continuation of all the map tasks, meaning that the reduce task waits for all the map tasks to complete, and then executes. This is achieved using Task.Factory.ContinueWhenAll.

Task.Factory.ContinueWhenAll(
  mapTasks, 
  tasks => PerformReduce(reduce, tasks));

As you can see, the implementation is minimalistic and simple, and usage is likewise.

Here’s a simple example using MapReduce to calculate the root mean square (MSE) of a set of values:

var task = MapReduce.Start<int, int, double>(
  i => i * i,
  s => Math.Sqrt(s.Aggregate((a, b) => a + b) / 5),
  1, 2, 3, 4, 5);
// Wait for result
task.Wait();
// Prints 3.3166...
Console.WriteLine(task.Result);

Actual applications of MapReduce are of course far more interesting than this simple example.

Applications of MapReduce

MapReduce can essentially be applied to any problem where you need a number of things to be done in parallel. It can even be applied in cases where you don’t need a final result. Just return an arbitrary value as the result (or even better, implement a variant of my MapReduce which uses Action<T>).

A few obvious use cases:

• Distributed search
• Distributed sort
• Tokenization
• Indexing
• Log processing
• Machine learning
• General artificial intelligence
• General data mining
• Large scale image processing
• …

The list goes on and on, these are just a few things off the top of my head.

You can grab the source code for MapReduce here. Since this is done in .NET 4.0, it requires Visual Studio 2010 Beta 2 or later.

As usual, play around with it, have fun, and let me know if you find it useful!

NOTE: The general cryptographical security of the presented method is beyond the scope of the article. The code presented is not cryptographically secure for large data sets. If you’re here looking for a way to do cryptographically secure RSA in the general case, you should look into more complicated approaches including padding, blinding, and more sophisticated block cipher modes. Cryptography is a topic undergoing constant research, so stay up to date and be sure to evaluate the strength of your solution for the scenarios in which you apply it.

BouncyCastle provides flexibility and control over your encryption approach, which comes at a cost. The BouncyCastle API might be a bit hard to cope with at first, but if you know encryption in general you should be able to find your way around the API without too much effort. This post will be focusing on RSA, since that was my original need, but it should be mentioned that BouncyCastle provides many other asymmetric (and symmetric) algorithms for which the usage is similar to what you find below.

Creating RSA keys

Creating RSA keys is a simple task. The method below lets you specify the key size in bits, and creates a key pair for you.

public AsymmetricCipherKeyPair GenerateKeys(int keySizeInBits)
{
  RsaKeyPairGenerator r = new RsaKeyPairGenerator();
  r.Init(new KeyGenerationParameters(new SecureRandom(),
    keySizeInBits));
  AsymmetricCipherKeyPair keys = r.GenerateKeyPair();
  return keys;
}

That’s all there is to it.

Encryption

Now that we have a key pair, we are ready to encrypt and decrypt using RSA. In the example below, we use a key (public or private) to encrypt a byte sequence. To encrypt a string, simply convert the string to a byte array using Encoding.GetBytes.

public byte[] Encrypt(byte[] data, AsymmetricKeyParameter key)
{
  RsaEngine e = new RsaEngine();
  e.Init(true, key);</p>

<p>int blockSize = e.GetInputBlockSize();</p>

<p>List<byte> output = new List<byte>();</p>

<p>for (int chunkPosition = 0; chunkPosition &lt; data.Length; 
    chunkPosition += blockSize)
  {
    int chunkSize = Math.Min(blockSize, data.Length - 
      (chunkPosition * blockSize));
    output.AddRange(e.ProcessBlock(data, chunkPosition,
      chunkSize));
  }
  return output.ToArray();
}

The approach above uses a list to gather output for the sake of simplicity. Note that the RSA engine can only process a limited block size at a time (block size depends on the key size). The approach above processes a data set of an arbitrary size.

The above method does not impose constraints on which key you use for encryption. Use the public key or the private key as you see fit for your solution.

Decryption

The Decrypt method is very similar to the Encrypt method:

public byte[] Decrypt(byte[] data, AsymmetricKeyParameter key)
{
  RsaEngine e = new RsaEngine();
  e.Init(false, key);</p>

<p>int blockSize = e.GetInputBlockSize();</p>

<p>List<byte> output = new List<byte>();</p>

<p>for (int chunkPosition = 0; chunkPosition &lt; data.Length;
    chunkPosition += blockSize)
  {
    int chunkSize = Math.Min(blockSize, data.Length - 
      (chunkPosition * blockSize));
    output.AddRange(e.ProcessBlock(data, chunkPosition,
      chunkSize));
  }
  return output.ToArray();
}

Again, it’s up to you which key you choose to use. If you want to use the common approach, encrypt using a symmetric cipher, hash the data, and sign the hash with your private key using the above Encrypt method. If you want to use another approach like encrypting the actual data using your private key, you are free to do so.

I hope this post helps those of you who want to apply RSA (or any other asymmetric cipher) to more subtle cases than those supported by the .NET framework.

The ViewDataDictionary that is passed to the HtmlHelper can be empty, or made to contain the data you want for your test.

public static HtmlHelper CreateHtmlHelper(ViewDataDictionary vd)
{
  var mockViewContext = new Mock<ViewContext>(
    new ControllerContext(
      new Mock<HttpContextBase>().Object,
      new RouteData(),
      new Mock<ControllerBase>().Object),
    new Mock<IView>().Object,
    vd,
    new TempDataDictionary());</p>

<p>var mockViewDataContainer = new Mock<IViewDataContainer>();
  mockViewDataContainer.Setup(v => v.ViewData)
    .Returns(vd);</p>

<p>return new HtmlHelper(mockViewContext.Object, 
    mockViewDataContainer.Object);
}