out >> m_Conscientia; » Software Engineering

fatal error C1859

Alex Peck — Mon, 22 Aug 2011 22:14:24 +0000

Once, a friend of mine set the welcome message on his Nokia 5110 to ‘fatal error’. This totally baffled the salesmen in Carphone Warehouse.

I was equally baffled when I returned to an old C++ project and tried to compile it in a freshly installed Visual Studio 2008 SP1. I was greeted with a series of these errors:

fatal error C1859: ‘Debug\blah.pch’ unexpected precompiled header error, simply rerunning the compiler might fix this problem

Not surprisingly, simply rerunning the compiler did not fix the problem. In fact, this is caused by address space layout randomization on Windows7/Server2008, as noted here.

The hotfix is available here: Fix for Visual C++ 2008 SP1 compiler error C1859

How to do unit test reviews

Alex Peck — Fri, 08 Jul 2011 23:39:11 +0000

At work we have begun to push unit testing and even TDD. Consequently code reviews now include more unit tests. Often, reviewers are very focused on product code, and neglect to review tests in detail.

Letting poor quality test code slip in is a big problem. Fragile tests erode confidence in unit testing as a valuable exercise, and tests which are hard to maintain just slow you down when the inevitable changes are required.

When I review unit test code, I consider the following:

Unit tests must favour readability. They must be compact, have a clear relationship between cause and effect, and use descriptive and meaningful phrases (DAMP).
Unit tests should assert once. There must be a clear relationship between test case method name and the assertion(s). This way, it is easy to determine the purpose of the test at a glance, and it is obvious what it means if the test fails.
Prefer state verification to behaviour verification. If you must verify behaviour (because there is no state, or it is unreasonably difficult to test state), be very careful not to over specify expectations.
Unit tests must be based on the public interface of a class. Testing internals makes your tests fragile. This is related to behaviour verification, sometimes you need to verify internal interactions, but this is not the same as invoking private methods in order to test them.
Unit tests must not contain (or contain very little) conditional logic, such as branches or loops. Conditional logic makes code harder to read.
Unit tests must be easy to maintain. As far as possible, DRY by applying the following techniques: use test setup/tear down methods to execute code common to all tests; implement custom verification functions or classes; use framework features like CollectionAssert, IEqualityComparer or IComparer.

Extension method to compute standard deviation of IEnumerable

Alex Peck — Wed, 22 Jun 2011 18:24:27 +0000

Recently, I have been writing some quick and dirty performance tests. I repeat each test scenario a few times and take an average of the execution time. In order to verify that I have a reasonably stable measurement, I wanted to also display the coefficient of variation, which is defined as the ratio of the standard deviation to the mean.

Given a set of TimeSpan values, I want to get the standard deviation as a TimeSpan. I implemented this based on the extension method I found here on StackOverflow. I implemented both sample (with Bessel’s correction) and population standard deviation.

public static class StandardDeviationExtensions
{
    public static double SampleStandardDeviation(this IEnumerable<double> values)
    {
        int count = values.Count();
 
        if (count > 1)
        {
            double average = values.Average();
            double sumOfSquaredDifferences = values.Sum(v => Math.Pow(v - average, 2));
            return Math.Sqrt(sumOfSquaredDifferences / (count - 1));
        }
 
        return 0.0;
    }
 
    public static double PopulationStandardDeviation(this IEnumerable<double> values)
    {
        if (values.Count() > 1)
        {
            double average = values.Average();
            return Math.Sqrt(values.Average(v => Math.Pow(v - average, 2)));
        }
 
        return 0.0;
    }
 
    public static TimeSpan SampleStandardDeviation(this IEnumerable<TimeSpan> values)
    {
        return new TimeSpan(
            (long)values
                .Select(v => (double)v.Ticks)
                .SampleStandardDeviation());
    }
 
    public static TimeSpan PopulationStandardDeviation(this IEnumerable<TimeSpan> values)
    {
        return new TimeSpan(
            (long)values
                .Select(v => (double)v.Ticks)
                .PopulationStandardDeviation());
    }
}

And here are the tests:

[TestClass]
public class StandardDeviationTest
{
    private readonly double[] EmptySet = new double[] {};
    private readonly double[] SingleValue = new double[] { 1.0 };
    private readonly double[] SameValues = new double[] { 5.0, 5.0, 5.0 };
    private readonly double[] ScenarioValues = new double[] { 2.0, 4.0, 4.0, 4.0, 5.0, 5.0, 7.0, 9.0 };
 
    private const double ScenarioValuesSampleDeviation = 2.13809;
    private const double ScenarioValuesPopulationDeviation = 2.0;
 
    [TestMethod]
    public void SampleStandardDeviation_EmptySet_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, EmptySet.SampleStandardDeviation());
    }
 
    [TestMethod]
    public void SampleStandardDeviation_SingleValue_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SingleValue.SampleStandardDeviation());
    }
 
    [TestMethod]
    public void SampleStandardDeviation_SetOfSameValues_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SameValues.SampleStandardDeviation());
    }
 
    [TestMethod]
    public void SampleStandardDeviation_ScenarioValues_HaveExpectedDeviation()
    {
        Assert.AreEqual(ScenarioValuesSampleDeviation, Math.Round(ScenarioValues.SampleStandardDeviation(), 5));
    }
 
    [TestMethod]
    public void SampleStandardDeviation_TimeSpanScenarioValues_HaveExpectedDeviation()
    {
        var timeSpanValues = ScenarioValues.Select(seconds => new TimeSpan(0, 0, (int)seconds));
        var result = timeSpanValues.SampleStandardDeviation().TotalSeconds;
        Assert.AreEqual(ScenarioValuesSampleDeviation, Math.Round(result, 5));
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_EmptySet_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, EmptySet.PopulationStandardDeviation());
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_SingleValue_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SingleValue.PopulationStandardDeviation());
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_SetOfSameValues_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SameValues.PopulationStandardDeviation());
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_ScenarioValues_HaveExpectedDeviation()
    {
        Assert.AreEqual(ScenarioValuesPopulationDeviation, Math.Round(ScenarioValues.PopulationStandardDeviation(), 5));
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_TimeSpanScenarioValues_HaveExpectedDeviation()
    {
        var timeSpanValues = ScenarioValues.Select(seconds => new TimeSpan(0, 0, (int)seconds));
        var result = timeSpanValues.PopulationStandardDeviation().TotalSeconds;
        Assert.AreEqual(ScenarioValuesPopulationDeviation, Math.Round(result, 5));
    }
}

Programming Windows Phone 7

Alex Peck — Fri, 05 Nov 2010 11:45:23 +0000

Programming Windows Phone 7 by Charles Petzold is available for free here.

Charles Petzold: Branded

Natural User Interface TechTalk

Alex Peck — Tue, 19 Oct 2010 16:43:01 +0000

Bill Buxton gives a great historical perspective on user interfaces and the pace of innovation.

Ballmer and Ozzie at D8

Alex Peck — Mon, 19 Jul 2010 17:55:10 +0000

SSD Compiler Benchmark

Alex Peck — Sat, 05 Jun 2010 16:49:55 +0000

Anandtech has some good articles illustrating the type of performance gains you might expect using an SSD disk for everyday tasks. Using Visual Studio, and in particular, compilation, are not considered. I did a cursory inspection of the rest of the Internet, and found no decent benchmarks investigating the impact of SSDs on compile time. So, I decided to perform my own. It’s well known that a lot of the operations performed by Visual Studio are disk bound, so it seems like there is some potential.

I discussed this with colleagues at work, and someone suggested that compiling managed code is more CPU intensive than native. This would suggest that it might be CPU bound, rather than disk bound. I therefore decided to test both native and managed code compilation, using both a conventional hard disk drive and a solid state drive.

Test Setup

I used my ageing PC running Windows 7 x64 Ultimate and Visual Studio 2010 Ultimate. For each disk I did a clean install and didn’t apply any updates.

I ran my tests on the following:

A Western Digital Raptor 150gb (WD1500ADFD), a 10k RPM conventional hard disk
An OCZ Vertex 2 100gb SSD

I selected the following test code to compile, based on it being relatively fast and self contained (I didn’t want to spend hours doing this!), and be fairly representative of a small system (compiling multiple dependent binaries):

Microsoft Enterprise Library 5.0 – a small set of managed components written by the Microsoft Patterns & Practices team.
LAME, the open source MP3 encoding library. I used version 3.98.4, and did a little hacking to get it to build against a recent GTK+.

Results

I compiled each test program three times and took the average result. I turned on build timing in Visual Studio by going to Tools->Options->Projects and Solutions->Build and Run->MSBuild project build output verbosity=Detailed. VS2010 now uses MSBuild for native code.

Compile time in seconds. Lower is better.

So, using an SSD we get around a 21% gain in speed on native code, and a 16% gain compiling managed. Not bad. Although my Raptor isn’t the most recent model, it is representative of the fastest conventional SATA hard disk available. On this basis, I would expect a RAID setup based on 10k RPM disks to outperform the SSD.

P/Invoke Interop Assistant

Alex Peck — Sat, 08 May 2010 23:45:28 +0000

After messing about for hours trying to p/invoke GetTokenInformation only for it to return what appeared to be sinographs, I came accross the P/Invoke interop assistant on CodePlex.

The P/Invoke Interop Assistant: it generates some useful stuff

Although this didn’t solve the my problem (which was correctly marshalling a fixed size char array), it would doubtlessly have saved me time writing the boiler plate structs, enums and method signatures. Highly recommended.

Parsing HTML tables into System.Data.DataTable

Alex Peck — Thu, 06 May 2010 20:50:08 +0000

What follows is a quick and dirty class I made to parse HTML tables into DataTables. As usual, it is the result of internet search/run/bug fix/refactor.

In use, it looks little like this:

WebClient client = new WebClient();
string html = client.DownloadString(@"http://www.table.co.uk");
DataSet dataSet = HtmlTableParser.Parse(html);

Here is the implementation. It’s not optimised for runtime performance, but it works.

/// 
/// HtmlTableParser parses the contents of an html string into a System.Data DataSet or DataTable.
/// 
public class HtmlTableParser
{
    private const RegexOptions ExpressionOptions = RegexOptions.Singleline | RegexOptions.Multiline | RegexOptions.IgnoreCase;
 
    private const string CommentPattern = "";
    private const string TablePattern = "]*>(.*?)

"; private const string HeaderPattern = "]*>(.*?)"; private const string RowPattern = "]*>(.*?)"; private const string CellPattern = "]*>(.*?)"; ///

/// Given an HTML string containing n table tables, parse them into a DataSet containing n DataTables. ///

/// An HTML string containing n HTML tables /// A DataSet containing a DataTable for each HTML table in the input HTML public static DataSet ParseDataSet(string html) { DataSet dataSet = new DataSet(); MatchCollection tableMatches = Regex.Matches( WithoutComments(html), TablePattern, ExpressionOptions); foreach (Match tableMatch in tableMatches) { dataSet.Tables.Add(ParseTable(tableMatch.Value)); } return dataSet; } ///

/// Given an HTML string containing a single table, parse that table to form a DataTable. ///

/// An HTML string containing a single HTML table /// A DataTable which matches the input HTML table public static DataTable ParseTable(string tableHtml) { string tableHtmlWithoutComments = WithoutComments(tableHtml); DataTable dataTable = new DataTable(); MatchCollection rowMatches = Regex.Matches( tableHtmlWithoutComments, RowPattern, ExpressionOptions); dataTable.Columns.AddRange(tableHtmlWithoutComments.Contains(") ? ParseColumns(tableHtml) : GenerateColumns(rowMatches)); ParseRows(rowMatches, dataTable); return dataTable; } ///

/// Strip comments from an HTML stirng ///

/// An HTML string potentially containing comments /// The input HTML string with comments removed private static string WithoutComments(string html) { return Regex.Replace(html, CommentPattern, string.Empty, ExpressionOptions); } ///

/// Add a row to the input DataTable for each row match in the input MatchCollection ///

/// A collection of all the rows to add to the DataTable /// The DataTable to which we add rows private static void ParseRows(MatchCollection rowMatches, DataTable dataTable) { foreach (Match rowMatch in rowMatches) { // if the row contains header tags don't use it - it is a header not a row if (!rowMatch.Value.Contains(")) { DataRow dataRow = dataTable.NewRow(); MatchCollection cellMatches = Regex.Matches( rowMatch.Value, CellPattern, ExpressionOptions); for (int columnIndex = 0; columnIndex < cellMatches.Count; columnIndex++) { dataRow[columnIndex] = cellMatches[columnIndex].Groups[1].ToString(); } dataTable.Rows.Add(dataRow); } } } ///

/// Given a string containing an HTML table, parse the header cells to create a set of DataColumns /// which define the columns in a DataTable. ///

/// An HTML string containing a single HTML table /// A set of DataColumns based on the HTML table header cells private static DataColumn[] ParseColumns(string tableHtml) { MatchCollection headerMatches = Regex.Matches( tableHtml, HeaderPattern, ExpressionOptions); return (from Match headerMatch in headerMatches select new DataColumn(headerMatch.Groups[1].ToString())).ToArray(); } ///

/// For tables which do not specify header cells we must generate DataColumns based on the number /// of cells in a row (we assume all rows have the same number of cells). ///

/// A collection of all the rows in the HTML table we wish to generate columns for /// A set of DataColumns based on the number of celss in the first row of the input HTML table private static DataColumn[] GenerateColumns(MatchCollection rowMatches) { int columnCount = Regex.Matches( rowMatches[0].ToString(), CellPattern, ExpressionOptions).Count; return (from index in Enumerable.Range(0, columnCount) select new DataColumn("Column " + Convert.ToString(index))).ToArray(); } }

As always, here are the tests. They yield 100% coverage but I still need to add some asserts on the column names.

/// 
/// Tests for the HtmlTableParser class
/// 
[TestClass]
public class ParserTest
{
    private TestContext testContextInstance;
 
    /// 
    /// Verify that HtmlTableParser can parse an HTML file containing a single table. The
    /// test file includes a commented out table which should be ignored. Note some tags use
    /// attributes (we test we can parse tags with and without attributes).
    /// 
    [TestMethod]
    [DeploymentItem(@"data\singleTable.txt")]
    public void TestParseSingleTable()
    {
        string html = File.ReadAllText("singleTable.txt");
        DataTable table = HtmlTableParser.ParseTable(html);
 
        AssertTable(GetExpectedData(), table);
    }
 
    /// 
    /// Verify that HtmlTableParser can parse an HTML file containing multiple tables. The
    /// test file includes a commented out table which should be ignored. The test file
    /// contains tables both with and without headers.
    /// 
    [TestMethod]
    [DeploymentItem(@"data\multipleTables.txt")]
    public void TestParseMultipleTables()
    {
        string html = File.ReadAllText("multipleTables.txt");
        DataSet dataSet = HtmlTableParser.ParseDataSet(html);
        Assert.AreEqual(3, dataSet.Tables.Count);
 
        var expected = GetExpectedData();
 
        foreach (DataTable table in dataSet.Tables)
        {
            AssertTable(expected, table);
        }
    }
 
    private static string[][] GetExpectedData()
    {
        return new[]
        {
            new[] { "row 1, cell 1", "row 1, cell 2" },
            new[] { "row 2, cell 1", "row 2, cell 2" }
        };
    }
 
    private static void AssertTable(string[][] expected, DataTable table)
    {
        Assert.AreEqual(expected.Count(), table.Rows.Count, "Table did not contain the expected number of rows");
 
        for (int i = 0; i < expected.Count(); i++)
        {
            for (int j = 0; j < expected[i].Count(); j++)
            {
                string actualElement = (table.Rows[i][j] as string).Trim();
                string expectedElement = expected[i][j];
 
                Assert.AreEqual<string>(expectedElement, actualElement, "Table did not contain the expected element");
            }
        }
    }
}

These are the test files, which are just some basic HMTL.




      
    
      Title

 

      
 
      
      
            
                  Heading 1
                  Heading 2
            
            
                  row 1, cell 1
                  row 1, cell 2
            
            
                  row 2, cell 1
                  row 2, cell 2




      
    
      Title

 

      
      
            
                  row 1, cell 1
                  row 1, cell 2
            
            
                  row 2, cell 1
                  row 2, cell 2
            
       
      
      
            
                  Heading 1
                  Heading 2
            
            
                  row 1, cell 1
                  row 1, cell 2
            
            
                  row 2, cell 1
                  row 2, cell 2
            
       
      
            
                  row 1, cell 1
                  row 1, cell 2
            
            
                  row 2, cell 1
                  row 2, cell 2

Filtering sqlmetal output using XSLT

Alex Peck — Thu, 04 Mar 2010 21:50:41 +0000

I needed to generate LINQ to SQL O/RM classes for a subset of the objects in a large database. Unfortunately, sqlmetal doesn’t provide a mechanism to filter it’s output, so I made my own using XSLT, a batch file and a custom build target. This post explains how.

Overview

When you invoke a build, a custom “BeforeBuild” build target runs to do the code generation. This code generation is driven by a batch file, this is what it does:

Run sqlmetal to output a dbml file for the entire database schema
Using an XML configuration file and an XSLT transform, generate a test .cs file containing a reference to each expected table. When included in our project, this gives us a compile time test that we generated the expected classes (we make this internal).
Run a second XSLT transform based on the configuration file and the output a dbml file from step 1. In this step we prune the xml according to the config file. The result is a dbml file containing only the tables we specify in our configuration.
Run sqlmetal again using the result of step 3 as input. This time sqlmetal outputs our CSharp code for our Linq to Sql classes.

The XSLT transforms are run from the command line using MSXSL.exe, which is available here.

The generated CSharp files are part of my Visual Studio project. After the BeforeBuild target is run (which does the code generation), the generated code is compiled into an Assembly.

Custom build target

These are the pertinent parts of my project file. You can see that inside my project directory, I created a directory called Prebuild where the work happens. This is where the Configuration.xml, DbmlPruner.xslt, GenerateTestTypes.xslt and generate.bat files live.

 version="1.0" encoding="utf-8"?>
 ToolsVersion="3.5" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  
   Name="BeforeBuild" DependsOnTargets="GenerateDbClasses;"> >
   Name="GenerateDbClasses" Inputs="Prebuild\Configuration.xml;Prebuild\DbmlPruner.xslt;Prebuild\GenerateTestTypes.xslt" Outputs="MasterDb.Generated.cs;TestMasterDbTypes.cs">
     Command="$(ProjectDir)Prebuild\generate.bat $(ProjectDir) $(TargetName) master.dbml prunedmaster.dbml TestMasterDbTypes.cs MasterDb Configuration.xml master" />
  >
>

Configuration.xml

 version="1.0" encoding="utf-8"?>

 Name="Master Tables" SourceXml="master.dbml" Namespace="Master.Data.Linq" TestClassName="TestMasterTypes">
   SqlName="dbo.TestTable" DataContextPropertyName="TestTable" ClassName="TestTableRow"/>
>

GenerateTestTypes.xslt

 version="1.0" encoding="utf-8"?>

 xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
   method="text" />
   match="/">
//-----------------------------------------------------------------------
// ]]>
//     This code was generated by a tool.
//
//     Changes to this file may cause incorrect behavior and will be lost 
//     if the code is regenerated.
// ]]>
//-----------------------------------------------------------------------
 
// Disable warning CS0169: The private field 'foo' is never used. This is 
// by design.
#pragma warning disable 0169
 
namespace  select="/Configuration/@Namespace" />
{
    /// ]]>
    /// This class is provided as a compile time test for the linq to SQL
    /// classes specified in Prebuild/configuration.xml. It will fail to
    /// compile if one of the dependent classes is not generated (or not
    /// generated with the expected name). This is by design.
    /// ]]>
    internal class  select="/Configuration/@TestClassName" />
    {      
 select="/Configuration/Table">>	>>	>private  select="@ClassName" />  select="' '" />  select="concat(@ClassName, 'Member')" />;>
>>>	>}
}
 
#pragma warning restore 0169
  >
>

DbmlPruner.xslt

 version="1.0" encoding="utf-8"?>

 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
	 xmlns:sql="http://schemas.microsoft.com/linqtosql/dbml/2007"
     version="1.0" >
  	 method="xml" version="1.0" encoding="utf-8" indent="yes"
  	  omit-xml-declaration = "no"/>
 
	 match="Configuration">
 
		
		 name="database" select="document(@SourceXml)"/>
     name="namespace" select="namespace-uri($database/sql:Database)"/>
 
		> =====================================================================================
   select="@Name" /> generated from database  select="$database/sql:Database/@Name" /> ( select="@SourceXml" />)
========================================================================================== >
 
		
		 name="Database" namespace="{$namespace}">
			 name="Name">
        		 select="$database/sql:Database/@Name" />
      		>
 
			 select="/Configuration/Table">
 
				 name="sqlName" select="@SqlName"/>
 
				
				 test="$database/sql:Database/sql:Table[@Name=$sqlName]">
 
					
					 name="Table" namespace="{$namespace}">
						 name="Name">
							 select="@SqlName"/>
						>
						 name="Member">
							 select="@DataContextPropertyName"/>
						>
 
						 name="Type" namespace="{$namespace}">
							 name="Name">
								 select="@ClassName"/>
							>
 
							
							 select="$database/sql:Database/sql:Table[@Name=$sqlName]/sql:Type/*"/>
						>
 
					>
 
				>
			>
		>
	>
 
>

generate.bat

This is my entire generate.bat file, you can glean the input arguments from the custom build target above.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:: Generate LINQ to SQL classes based on tables defined in an xml config
::
 
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
 
setlocal 
 
set SQLMETAL=\sdk\Win2008\Bin\sqlmetal.exe
set MSXSL=\msxsl.exe
 
set PRJDIR=%1%
set PREDIR=%PRJDIR%Prebuild\
 
set NAMESPACE=%2
 
set METALOUT=%PREDIR%%3
set METALIN=%PREDIR%%4
 
set COMPILETESTCLASS=%PRJDIR%%5
set CONTEXTCLASS=%6
set CONTEXTCLASSFILE=%PRJDIR%%CONTEXTCLASS%.Generated.cs
 
set CONFIGPATH=%PREDIR%%7
set DATABASE=%8
 
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
 
echo ================ Generation of LINQ to SQL classes started ================
echo Using %CONFIGPATH%
 
echo 1. Building %METALOUT% for entire %DATABASE% database 
%SQLMETAL% /conn:"server=localhost; database=%DATABASE%;Integrated Security=SSPI" /dbml:%METALOUT%
if errorlevel 1 goto :Failed
 
echo 2. Generating %COMPILETESTCLASS% to test linq to sql types were generated correctly at compile time
%MSXSL% %CONFIGPATH% %PREDIR%GenerateTestTypes.xslt -o %COMPILETESTCLASS%
if errorlevel 1 goto :Failed
 
echo 3. Building %METALIN% from %CONFIGPATH%
%MSXSL% %CONFIGPATH% %PREDIR%DbmlPruner.xslt -o %METALIN%
if errorlevel 1 goto :Failed
 
echo 4. Generating %CONTEXTCLASSFILE% using %METALIN%
%SQLMETAL% %METALIN% /code:%CONTEXTCLASSFILE% /language:csharp /context:%CONTEXTCLASS% /namespace:%NAMESPACE% /serialization:Unidirectional
if errorlevel 1 goto :Failed
 
echo ======== Generation of LINQ to SQL classes completed successfully =========
endlocal
goto :EOF
 
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
:Failed
 
echo Configuration %CONFIG% FAILED
endlocal
exit /B 1