Cook Computing

Doug Stewart has posted that version 3.0 of Microsoft's Application Verifier has shipped. In case you've not heard of this useful debugging tool:

Application Verifier is a runtime verification tool for unmanaged code that assists in quickly finding subtle programming errors that can be extremely difficult to identify with normal application testing.

Application Verifier is designed specifically to detect and help debug memory corruptions and critical security vulnerabilities. It makes it easier to create reliable applications by monitoring an application's interaction with the Windows operating system, profiling its use of objects, the registry, the file system, and Win32 APIs (including heaps, handles, locks, and more). It also includes checks to predict how well the application will perform under Least-privileged User Account operation.

Running Application Verifier is easy; simply turn on the tool then run your project and go through your normal testing scenarios. When your tests are completed, view the Application Verifier logs for any errors that may have been detected.

I've been using it over the last year or so and have found several obscure bugs.

I use Lotus Notes daily to access a development group bug database and Lotus Domino server is one of the mail server backends supported by the product I work on, so the first thing that caught my attention in Ray Ozzie's announcement of Simple Sharing Extensions (SSE) was the point that the SSE's replication mechanism is based on that used by Notes:

This got me to thinking about simplicity. Notes had just about the simplest possible replication mechanism imaginable. After all, we built it at Iris in 1985 for use on a 6Mhz 286-based IBM PC/AT with incredibly slow-seeking 20MB drives. We were struggling with LIM EMS trying to make effective use of more than 1MB of memory. Everything about the design was about implementation simplicity and efficiency. So if simple is the goal, why not just adapt the Notes replication algorithm to this need? Notes "notefiles" could be analogous to RSS "feeds"; and Notes "notes" could be analogous to RSS "items"; and Notes "items" could be analogous to XML "elements".

Notefiles replicate by using a very simple mechanism based on GUID assignment, with clocks and tie-breakers to detect and deterministically propagate modifications. Something like this could easily be represented in XML. Notefiles replicate with one another in a decentralized, masterless manner; feeds could be "cross-subscribed" in a similar manner. There's no magic to it once you know specifically what you're trying to accomplish, but it certainly helped to have an existence proof.

So I studied the spec last night to find out how this works, in particular conflict detection.

Each version of an item in a feed is identified by a version number, and either or both of the time when the version was created (the when attribute) and the identifier for the endpoint that created the version (the by attribute).

Using the example from the draft spec, the first version of an item will look like this:

<item>
  <description>This is a test item</description>
  <sx:sync id="0a7903db47fb0ae8" version="1">
    <sx:history when="Sat, 21 May 2005 09:43:33 GMT" by="REO1750"/>
  </sx:sync>
</item>

When subsequent versions are created locally the information about previous versions is stored in <update> elements:

<item>
  <description>This is a test item</description>
  <sx:sync id="0a7903db47fb0ae8" version="4">
    <sx:history when="Tue, 24 May 2005 09:43:33 GMT" by="REO1750">
      <sx:update when="Mon, 23 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sun, 22 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sat, 21 May 2005 09:43:33 GMT"
          by="REO1750" />
    </sx:history>
  </sx:sync>
</item>

Note that the <update> elements are ordered most recent first and although they do not have a version attribute this can be deduced from the version attribute of the <sync> element. If this is n then the implicit version numbers of the <update> elements are n-1, n-2, n-3, and so on. So the versioning information in the previous example above can be represented as:

(4, Tue, 24 May 2005 09:43:33 GMT, REO1750)
(3, Mon, 23 May 2005 09:43:33 GMT, REO1750)
(2, Sun, 22 May 2005 09:43:33 GMT, REO1750)
(1, Sat, 21 May 2005 09:43:33 GMT, REO1750)

When a subscriber reads a feed and the feed contains items with the same id as items held locally, the subscriber must use a globally consistent algorithm to decide which instance of the item is the "winner" which will be included in the local copy of the feed. The spec define this as follows:

a. When comparing the incoming item to its corresponding local item, the "winner" is determined by a collation using comparison of these four keys, in order:

1. the version attribute of the sx:sync element (larger version number wins)

2. the when attribute of the sx:history element (later date/time wins; if it is present on one and not present on the other, present wins)

3. the by attribute of the sx:history element ("greater" normalized text wins; if it is present on one and not present on the other, present wins)

b. If the collation results in equality on all tests, the two items are deemed to be identical and no action is required on this item.

Continuing with the same example, say the item downloaded from the remote server looks like this:

<item>
  <description>This is a test item</description>
  <sx:sync id="0a7903db47fb0ae8" version="5">
    <sx:history when="Wed, 25 May 2005 09:43:33 GMT" by="JIM234">
      <sx:update when="Tues, 24 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Mon, 23 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sun, 22 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sat, 21 May 2005 09:43:33 GMT"
          by="REO1750" />
    </sx:history>
  </sx:sync>
</item>

In this case a new version of the item has been created by JIM234 on the remote feed. The test on version number indicates the remote item is the winner and the local item should be replaced. If it turns out that there is no winner we assume that the items are equal and no action is required. Otherwise:

c. Now that a "winner" and a "loser" have been declared, one must detect to see if this was a normal sequence of edits, of if the endpoints separately edited the same version of the same item and "stepped on" each others' changes. This is called conflict detection.

4.4.1 Conflict Detection

In order to detect conflicts, endpoints MUST follow this algorithm on the modification history of two updated items:

1. If there is an sx:history element:

a. Let n equal the version attribute of the winner's sx:sync element.

b. Decrement n by the current version attribute of the loser's sx:sync item.

c. Compare the when and by attributes of the loser's sx:history element with the corresponding attributes of the nth (1-based) sx:update sub-element of the winner. If n is 0, compare against the sx:history element of the winner.

d. If the nth sx:update sub-element does not exist, assume NO CONFLICT; we are done processing conflict detection.

e. If attribute values are equal, there is NO CONFLICT; we are done processing conflict detection.

f. If they are not equal it is a CONFLICT; continue with step 2.

2. If a conflict is detected, the conflict attribute on the sx:sync element should be set to "true"; else if a conflict is not detected, the value of the conflict attribute MUST be preserved.

All this is saying is that when the most recent version of the item for the loser does not have the same when and by attributes as the similarly numbered version of the winner's item, the remote and local versions are out of sync. For example in the example above the versioning information is:

Local - Loser
(4, Tue, 24 May 2005 09:43:33 GMT, REO1750)
(3, Mon, 23 May 2005 09:43:33 GMT, REO1750)
(2, Sun, 22 May 2005 09:43:33 GMT, REO1750)
(1, Sat, 21 May 2005 09:43:33 GMT, REO1750)

Remote - Winner
(5, Wed, 25 May 2005 09:43:33 GMT, JIM234)
(4, Tue, 24 May 2005 09:43:33 GMT, REO1750)
(3, Mon, 23 May 2005 09:43:33 GMT, REO1750)
(2, Sun, 22 May 2005 09:43:33 GMT, REO1750)
(1, Sat, 21 May 2005 09:43:33 GMT, REO1750)

The most recent version for the loser is version 4, and if we compare this with version 4 of the winner we see that they are the same and so there is no conflict. However say the versioning information looked like this:

Local - Loser
(4, Tue, 24 May 2005 09:43:33 GMT, REO1750)
(3, Mon, 23 May 2005 09:43:33 GMT, REO1750)
(2, Sun, 22 May 2005 09:43:33 GMT, REO1750)
(1, Sat, 21 May 2005 09:43:33 GMT, REO1750)

Remote - Winner
(5, Wed, 25 May 2005 09:43:33 GMT, JIM234)
(4, Mon, 23 May 2005 11:01:12 GMT, JIM234)
(3, Mon, 23 May 2005 09:43:33 GMT, REO1750)
(2, Sun, 22 May 2005 09:43:33 GMT, REO1750)
(1, Sat, 21 May 2005 09:43:33 GMT, REO1750)

Now the most recent version for the loser - version 4 - has different attribute values to the winner's version 4 and so there is a conflict. Up to version 3 the two feeds were in sync but the endpoints made their own version 4 changes and conflict was introduced.

The spec specifies how to handle cases where full versioning information is not available. One point to note is that the version number takes precedence over when the version was created which might seem a bit counter-intuitive. In the following example the remote item is the winner even though the local item was updated over two days later:

LOCAL

<item>
  <description>This is a test item</description>
  <sx:sync id="0a7903db47fb0ae8" version="5">
    <sx:history when="Thu, 26 May 2005 20:00:00 GMT" by="REO1750">
      <sx:update when="Tues, 24 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Mon, 23 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sun, 22 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sat, 21 May 2005 09:43:33 GMT"
          by="REO1750" />
    </sx:history>
  </sx:sync>
</item>

REMOTE

<item>
  <description>This is a test item</description>
  <sx:sync id="0a7903db47fb0ae8" version="6">
    <sx:history when="Tues, 24 May 2005 13:00:00 GMT" by="JIM234">
      <sx:update when="Tues, 24 May 2005 11:00:00 GMT"
          by="JIM234" />
      <sx:update when="Tues, 24 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Mon, 23 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sun, 22 May 2005 09:43:33 GMT"
          by="REO1750" />
      <sx:update when="Sat, 21 May 2005 09:43:33 GMT"
          by="REO1750" />
    </sx:history>
  </sx:sync>
</item> 

The consistently interesting Stumbling and Mumbling reports on a paper which puts a price on job satisfaction, coming up with a surprisingly large value. S&M quotes the paper:

To reduce job satisfaction from 9 to 8 on the 10-point scale...would, for a family with $65,000 income, have to be matched by an income increase of more than $30,000 a year [to leave life satisfaction unchanged]...Moving from the middle to the 75% percentile in job satisfaction would have a personal income equivalence, for someone of median income, of $17,000 per annum. These dollar amounts would be correspondingly lower for families with lower incomes.

But then maybe it will not be so surprising to readers of this blog who are likely to be software developers and so typically endowed with high IQ brains which have a voracious appetite for intellectual stimulation. Being stuck in a dull job is going to be very tedious for that sort of person and would have to be financially very rewarding to make life bearable.

This does suggest the corollary that moving someone into a role they don't want to do, for example moving a developer into a support role working on legacy code, is going to increase the chance that they will look for another job, because they will be prepared to take lower income for increased job satisfaction, so widening the range of job opportunities they will consider. Or maybe that never happens because other instincts or fears come into play making it very hard to give up income for increased happiness.

This post by James Clarke reminded me that I should mention the excellent Fiddler HTTP debugging proxy, not least because it is a very useful tool for viewing XML-RPC requests and responses.

Fiddler is a HTTP Debugging Proxy which logs all HTTP traffic between your computer and the Internet. Fiddler allows you to inspect all HTTP Traffic, set breakpoints, and "fiddle" with incoming or outgoing data. Fiddler is designed to be much simpler than using NetMon or Achilles, and includes a simple but powerful JScript.NET event-based scripting subsystem.

Fiddler ships with several request and response inspectors installed by default. The following snapshots show the output for an XML-RPC request.

Request Headers Inspector:

Request TextView Inspector:

Request HexView Inspector:

Request Raw Inspector:

Request XML Inspector:

There are a couple of articles on MSDN which provide a comprehensive introduction to what you can do with Fiddler: HTTP Debugging and HTTP Performance.

I've just returned from a week on the Isle of Tiree for my annual walking holiday. The photo above shows the view across Balephuil Bay to the township of Balephuil where I stayed. The weather was very windy when I arrived and then settled down to several days of sunshine. I did some very enjoyable walks along the coastline, watching seals and many species of birds. The beaches are particularly beautiful. I will definitely return.

I mentioned a while ago that I'm adding support for XML-RPC methods with a variable number of parameters in XML-RPC.NET. I recently enhanced the XmlRpcProxyGen class to add the ParamArrayAttribute where required. I wrote the following sample program to explore how to do this (just a few lines of new code) and to get some practice using Reflection.Emit (the implementation of the Foo method).

using System;
using System.Diagnostics;
using System.Reflection;
using System.Reflection.Emit;

public interface IFoo
{
  int Foo(params int[] args);
}

class Program
{
  static void Main(string[] args)
  {
    AppDomain ad = AppDomain.CurrentDomain;
    Type itf = typeof(IFoo);
    AssemblyName asmName = new AssemblyName();
    asmName.Name = "FooAssembly";
    AssemblyBuilder asmBldr = ad.DefineDynamicAssembly(
      asmName, AssemblyBuilderAccess.Run);
    ModuleBuilder modBldr = asmBldr.DefineDynamicModule(
      asmName.Name);
    TypeBuilder typeBldr = modBldr.DefineType(
      "FooClass", TypeAttributes.Class | TypeAttributes.Public,
      null, new Type[] { itf });
    MethodBuilder mthdBldr = typeBldr.DefineMethod("Foo",
      MethodAttributes.Public | MethodAttributes.Virtual,
      typeof(int), new Type[] { typeof(int[]) } );

    // add ParamArrayAttribute to Foo parameter
    ParameterBuilder paramBldr = mthdBldr.DefineParameter(1,
      ParameterAttributes.In, "args");
    ConstructorInfo ctorInfo 
      = typeof(ParamArrayAttribute).GetConstructor(new Type[0]);
    CustomAttributeBuilder attrBldr =
        new CustomAttributeBuilder(ctorInfo, new object[0]);
    paramBldr.SetCustomAttribute(attrBldr);

    // generate the IL
    ILGenerator ilgen = mthdBldr.GetILGenerator();
    Label Next = ilgen.DefineLabel();
    Label Done = ilgen.DefineLabel();
    LocalBuilder total = ilgen.DeclareLocal (typeof (int));
    LocalBuilder idx = ilgen.DeclareLocal (typeof (int));
    
    // get count of arguments in array
    ilgen.Emit(OpCodes.Ldarg, 1);
    ilgen.Emit(OpCodes.Ldlen);
    ilgen.Emit(OpCodes.Stloc, idx);
    
    // return here for subsequent arguments
    ilgen.MarkLabel (Next);
    
    // decrement index - if result less than zero we're done
    ilgen.Emit(OpCodes.Ldloc, idx);
    ilgen.Emit(OpCodes.Ldc_I4_1);
    ilgen.Emit(OpCodes.Sub);
    ilgen.Emit(OpCodes.Dup);
    ilgen.Emit(OpCodes.Stloc, idx);
    ilgen.Emit(OpCodes.Ldc_I4_0);    
    ilgen.Emit(OpCodes.Blt, Done);
    
    // use index to get argument and add to total
    ilgen.Emit(OpCodes.Ldarg, 1);
    ilgen.Emit(OpCodes.Ldloc, idx);
    ilgen.Emit(OpCodes.Ldelem_I4);
    ilgen.Emit(OpCodes.Ldloc, total);
    ilgen.Emit(OpCodes.Add);
    ilgen.Emit(OpCodes.Stloc, total);
    
    // go back for next argument
    ilgen.Emit(OpCodes.Br_S, Next);

    // all arguments added - put total on stack and return
    ilgen.MarkLabel (Done);
    ilgen.Emit(OpCodes.Ldloc, total);
    ilgen.Emit(OpCodes.Ret);
    
    // build the type
    Type fooType = typeBldr.CreateType();

    // check IFoo.Foo has ParamArrayAttribute
    MethodInfo mi = fooType.GetMethod("Foo");
    ParameterInfo pi = mi.GetParameters()[0];
    Debug.Assert(Attribute.IsDefined(pi, 
      typeof(ParamArrayAttribute)));
    
    // create instance of type and test it
    IFoo ifoo = (IFoo)Activator.CreateInstance(fooType);
    Debug.Assert(ifoo.Foo() == 0);
    Debug.Assert(ifoo.Foo(1) == 1);
    Debug.Assert(ifoo.Foo(1,2) == 3);
    Debug.Assert(ifoo.Foo(1,2,3) == 6);
  }
}

Good news from the MSBuild team. Work on support for targetting .NET 1.1 has started under the project codename MSBee - MSBuild Everett Environment (Everett is the codename for Visual Studio 2003). I'd like to move to MSBuild for building XML-RPC.NET and this could be very useful. Craig Lichtenstein has more details. I actually have been building for .NET 1.0 but that might be asking too much.

I just noticed that friend and ex-colleague Andy McMullan has at last started blogging. Andy has been maintaining a .NET FAQ and a C# for C++ Programmers FAQ for over five years, both of which have been translated into French and Chinese.

In May I reported to the MSN Product Feedback Center a problem with .NET 2.0 beta 2 which broke existing XML-RPC.NET applications. The bug report was later marked as fixed, in particular "This bug will be fixed in RTM milestone.". Unfortunately the problem still exists in the release version of .NET 2.0.

Therefore any applications which create a XML-RPC.NET proxy based on an interface which is not declared as public will fail on the call to XmlRpcProxyGen.Create. For example, this example will fail:

[XmlRpcUrl("http://www.cookcomputing.com/sumAndDiff.rem")] 
interface ISumAndDiff
{ 
  [XmlRpcMethod] 
  SumAndDiffValue SumAndDifference(int x, int y);
}

static void Main(string[] args)
{
  ISumAndDiff proxy = (ISumAndDiff)XmlRpcProxyGen.Create(typeof(ISumAndDiff));
  SumAndDiffValue ret = proxy.SumAndDifference(2, 3);
}

The workaround is to specify that an earlier version of the .NET runtime should be used with the application, by creating or modifying the application config file. For example, if your application's executable is myapp.exe, the application config file is called myapp.exe.config and is placed in the same directory as the executable. It needs to specify the supported runtime as follows:

<?xml version ="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v1.1.4322" /> 
  </startup>
</configuration> 

This example uses version 1.1 of the runtime. To check which pre-2.0 versions of the runtime are installed on your machine, run the following commands in a command prompt window:

cd %windir%\microsoft.net\framework
dir v*

If you are building an application, the solution is to ensure that any interfaces used in XmlRpcProxyGen.Create are public.

One phrase that has stuck in my mind over the years, in relation to software development team management, has been interchangeable morons school of resource management. I finally tracked it down last weekend when I was archiving all my C++ books to a bookcase in the garage. It was used by Bjarne Stroustrup in the 2nd edition of "The C++ Programming Language", published 1991. In chapter 11 on Design and Development he wrote:

As in all other areas of software development it is necessary to consider the longer term. It is essentially impossible to judge the performance of an individual on the basis of a single year's work. Most individuals do, however, have consistent long term track records that can be reliable predictors of technical judgement and a useful help in evaluating immediate past performance. Disregard of such records - as is done where individuals are considered merely as interchangeable cogs in the wheels of an organization - leaves managers at the mercy of misleading quantity measurements.

One consequence of taking a long term view and avoiding the "interchangeable morons school of resource management" is that individuals (both developers and managers) need longer to grow into the more demanding and interesting jobs. This discourages job hopping as well for job rotation for "career development". A low turnover of both key technical people and key managers must be a goal. No manager can succeed without a rapport with key designers and programmers and some recent and relevant technical knowledge. Conversely, no group of designers and developers can succeed in the long run without support from competent managers and a minimum of understanding of the larger non-technical context in which they work.

As another example of aggregator serendipity, the day after I posted a code snippet relating to generics, I came across a Serverside post pointing to several MSDN FAQs on .NET Generics:

• Fundamentals
• .NET Framework
• Tool Support
• Best Practices

For future reference, sample code on how to get the generic argument of a nullable type in C#:

Type type = typeof(int?);

bool isNullable = type.IsGenericType
  && (type.GetGenericTypeDefinition() == typeof(Nullable<>));

Type genArg = type.GetGenericArguments()[0];