out >> m_Conscientia; » ienumerable

Extension method to compute standard deviation of IEnumerable

Alex Peck — Wed, 22 Jun 2011 18:24:27 +0000

Recently, I have been writing some quick and dirty performance tests. I repeat each test scenario a few times and take an average of the execution time. In order to verify that I have a reasonably stable measurement, I wanted to also display the coefficient of variation, which is defined as the ratio of the standard deviation to the mean.

Given a set of TimeSpan values, I want to get the standard deviation as a TimeSpan. I implemented this based on the extension method I found here on StackOverflow. I implemented both sample (with Bessel’s correction) and population standard deviation.

public static class StandardDeviationExtensions
{
    public static double SampleStandardDeviation(this IEnumerable<double> values)
    {
        int count = values.Count();
 
        if (count > 1)
        {
            double average = values.Average();
            double sumOfSquaredDifferences = values.Sum(v => Math.Pow(v - average, 2));
            return Math.Sqrt(sumOfSquaredDifferences / (count - 1));
        }
 
        return 0.0;
    }
 
    public static double PopulationStandardDeviation(this IEnumerable<double> values)
    {
        if (values.Count() > 1)
        {
            double average = values.Average();
            return Math.Sqrt(values.Average(v => Math.Pow(v - average, 2)));
        }
 
        return 0.0;
    }
 
    public static TimeSpan SampleStandardDeviation(this IEnumerable<TimeSpan> values)
    {
        return new TimeSpan(
            (long)values
                .Select(v => (double)v.Ticks)
                .SampleStandardDeviation());
    }
 
    public static TimeSpan PopulationStandardDeviation(this IEnumerable<TimeSpan> values)
    {
        return new TimeSpan(
            (long)values
                .Select(v => (double)v.Ticks)
                .PopulationStandardDeviation());
    }
}

And here are the tests:

[TestClass]
public class StandardDeviationTest
{
    private readonly double[] EmptySet = new double[] {};
    private readonly double[] SingleValue = new double[] { 1.0 };
    private readonly double[] SameValues = new double[] { 5.0, 5.0, 5.0 };
    private readonly double[] ScenarioValues = new double[] { 2.0, 4.0, 4.0, 4.0, 5.0, 5.0, 7.0, 9.0 };
 
    private const double ScenarioValuesSampleDeviation = 2.13809;
    private const double ScenarioValuesPopulationDeviation = 2.0;
 
    [TestMethod]
    public void SampleStandardDeviation_EmptySet_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, EmptySet.SampleStandardDeviation());
    }
 
    [TestMethod]
    public void SampleStandardDeviation_SingleValue_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SingleValue.SampleStandardDeviation());
    }
 
    [TestMethod]
    public void SampleStandardDeviation_SetOfSameValues_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SameValues.SampleStandardDeviation());
    }
 
    [TestMethod]
    public void SampleStandardDeviation_ScenarioValues_HaveExpectedDeviation()
    {
        Assert.AreEqual(ScenarioValuesSampleDeviation, Math.Round(ScenarioValues.SampleStandardDeviation(), 5));
    }
 
    [TestMethod]
    public void SampleStandardDeviation_TimeSpanScenarioValues_HaveExpectedDeviation()
    {
        var timeSpanValues = ScenarioValues.Select(seconds => new TimeSpan(0, 0, (int)seconds));
        var result = timeSpanValues.SampleStandardDeviation().TotalSeconds;
        Assert.AreEqual(ScenarioValuesSampleDeviation, Math.Round(result, 5));
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_EmptySet_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, EmptySet.PopulationStandardDeviation());
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_SingleValue_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SingleValue.PopulationStandardDeviation());
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_SetOfSameValues_HasZeroDeviation()
    {
        Assert.AreEqual(0.0, SameValues.PopulationStandardDeviation());
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_ScenarioValues_HaveExpectedDeviation()
    {
        Assert.AreEqual(ScenarioValuesPopulationDeviation, Math.Round(ScenarioValues.PopulationStandardDeviation(), 5));
    }
 
    [TestMethod]
    public void PopulationStandardDeviation_TimeSpanScenarioValues_HaveExpectedDeviation()
    {
        var timeSpanValues = ScenarioValues.Select(seconds => new TimeSpan(0, 0, (int)seconds));
        var result = timeSpanValues.PopulationStandardDeviation().TotalSeconds;
        Assert.AreEqual(ScenarioValuesPopulationDeviation, Math.Round(result, 5));
    }
}

Combinations of IEnumerable

Alex Peck — Wed, 25 Nov 2009 21:16:46 +0000

Today I needed to generate all the combinations of elements in a collection. After some fruitless searching I was forced to engage my brain, and came up with the following iterator block. I’m making no extravagant claims about performance (skip, take and count could all be faster).

/// 
/// Compute the set of combinations, where a combination is an un-ordered collection 
/// of elements taken from the candidate set. Each element must appear zero or once. 
/// For set [1,2,3], the combinations are [1], [1,2], [1,2,3], [2], [2,3] and [3].
/// 
/// The type of objects whose combinations we evaluate
/// The set from which to combine elements
/// The set of combinations
public static IEnumerable<IEnumerable<T>> Combinations<T>(this IEnumerable<T> set)
{
    int n = set.Count();
 
    for (int skip = 0; skip < n; skip++)
    {
        for (int take = 1; take <= n - skip; take++)
        {
            yield return set.Skip(skip).Take(take);
        }
    }
}

Just to prove it works, here are my tests.

[TestMethod]
public void CombinationsOfNone()
{
    var candidates = new List<int>();
    var combinations = new List<List<int>>();
 
    AssertEqualSetsOfSets(combinations, candidates.Combinations());
}
 
[TestMethod]
public void CombinationsOfOne()
{
    var candidates = new List<int>() { 1 };
 
    var combinations = new List<List<int>>() 
    { 
        new List<int>() {1}
    };
 
    AssertEqualSetsOfSets(combinations, candidates.Combinations());
}
 
[TestMethod]
public void CombinationsOfThree()
{
    var candidates = new List<int>() { 1, 2, 3 };
 
    var combinations = new List<List<int>>() 
    { 
        new List<int>() {1}, 
        new List<int>() {1, 2},
        new List<int>() {1, 2, 3},
        new List<int>() {2},
        new List<int>() {2, 3},
        new List<int>() {3}
    };
 
    AssertEqualSetsOfSets(combinations, candidates.Combinations());
}
 
private void AssertEqualSetsOfSets(List<List<int>> expected, IEnumerable<IEnumerable<int>> actual)
{
    Assert.AreEqual<int>(expected.Count, actual.Count());
 
    var expectedIter = expected.GetEnumerator();
    var actualIter = actual.GetEnumerator();
 
    while (expectedIter.MoveNext() && actualIter.MoveNext())
    {
        Assert.IsTrue(OrderIndependentEqual(expectedIter.Current, actualIter.Current));
    }
}
 
public bool OrderIndependentEqual<T>(IEnumerable<T> first, IEnumerable<T> second)
{
    return !first.Except(second).Concat(second.Except(first)).Any();
}

Iteration

Alex Peck — Wed, 23 Sep 2009 21:17:14 +0000

Iteration in .NET can be acomplished in two ways: indexing and using the IEnumerable interface. Indexing is an elementary programming concept covered elsewhere.

IEnumerable & IEnumerator

The IEnumerable interface would perhaps have been better named IForwardIteratable, since it defines a scheme for forward only iteration. IEnumerable specifies a single method, GetEnumerator, which returns an IEnumerator, the actual forward iterator. IEnumerator specifies three members:

Current property: the collection element to which the iterator is pointing.
MoveNext method: moves the iterator to the next element in the collection. Returns false when the iterator has passed the final element.
Reset method: set the iterator to the initial state, where calling MoveNext results in the iterator pointing at the first element.

You may then traverse a collection as follows:

IEnumerator enumerator = collection.GetEnumerator();
while (enumerator.MoveNext())
{
   Console.WriteLine(enumerator.Current.ToString());
}

foreach

The code snippet above might seem a little cumbersome. To improve matters both VB and C# provide the foreach language level construct. Using foreach the example above becomes simply:

foreach (object element in collection)
{
   Console.WriteLine(element.ToString());
}

Notice that this scheme is based on object rather than the derived type in the collection. So, if you don’t use generic collections, there might be an implicit cast from IEnumerator.Current to the target type.