C# Benchmark Programs

You are wondering if the C# programming language is fast. You question whether it is worth it to benchmark programming constructs. What else can you learn from careful benchmarks? Here's a benchmarking overview using the C# language that touches on some important things about benchmarking.

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Donald Knuth

Examples

Here we look at the kinds of loops that Dot Net Perls benchmarks are usually done with. This simple framework is what I do my experiments with. You will want to change m depending on the code of each iteration; it is best to start smaller and push the limit up.

Tips: Use the two loops here to time code. The two stopwatches are set up to print out the time. Run in a new C# console application. Adjust _max higher or lower based on how slow your iterations are. Always run in Release mode, never in VS; click the .exe yourself. Change the order of the tests.

Benchmarking Console application, 2 loops [C#]

using System;
using System.Diagnostics;
using System.Runtime.CompilerServices;

class Program
{
    const int _max = 1000000;
    static void Main()
    {
	var s1 = Stopwatch.StartNew();
	for (int i = 0; i < _max; i++)
	{
	}
	s1.Stop();
	var s2 = Stopwatch.StartNew();
	for (int i = 0; i < _max; i++)
	{
	}
	s2.Stop();
	Console.WriteLine(((double)(s1.Elapsed.TotalMilliseconds * 1000000) /
	    _max).ToString("0.00 ns"));
	Console.WriteLine(((double)(s2.Elapsed.TotalMilliseconds * 1000000) /
	    _max).ToString("0.00 ns"));
	Console.Read();
    }

    static int _temp1;
    static int _temp2;

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void Method1()
    {
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void Method2()
    {
    }
}

Benchmarking Console application, 3 loops [C#]
    This prints out milliseconds.

using System;
using System.Diagnostics;

class Program
{
    static void Main()
    {
	const int m = 1000000;
	Stopwatch s1 = Stopwatch.StartNew();
	for (int i = 0; i < m; i++)
	{
	}
	s1.Stop();
	Stopwatch s2 = Stopwatch.StartNew();
	for (int i = 0; i < m; i++)
	{
	}
	s2.Stop();
	Stopwatch s3 = Stopwatch.StartNew();
	for (int i = 0; i < m; i++)
	{
	}
	s3.Stop();
	Console.WriteLine("{0},{1},{2}",
	    s1.ElapsedMilliseconds,
	    s2.ElapsedMilliseconds,
	    s3.ElapsedMilliseconds);
	Console.Read();
    }
}

Benchmarking aspx file [ASP.NET and C#]

using System;
using System.Diagnostics;
using System.Web.UI;

public partial class _Default : Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
	const int m = 1000000;
	Stopwatch s1 = Stopwatch.StartNew();
	for (int i = 0; i < m; i++)
	{
	}
	s1.Stop();
	Stopwatch s2 = Stopwatch.StartNew();
	for (int i = 0; i < m; i++)
	{
	}
	s2.Stop();
	Response.Write("Loop 1: ");
	Response.Write(s1.ElapsedMilliseconds);
	Response.Write("<br/>" +
	    "Loop 2: ");
	Response.Write(s2.ElapsedMilliseconds);
    }
}

Notes. The benchmarking code has some problems, such as the second block rarely taking less time to execute due to unknown causes. Repeat and swap two loops if you are doubtful. The first program shown converts the results to nanoseconds, which are easier to understand and provide a way to compare different operations together easier. It is usually best to report results in nanoseconds or microseconds with the number of iterations used to divide the result.

Importance

Is benchmarking important? Yes—however, it is often not important in a practical sense for your current project. It encourages you to examine your code and find out what it is really doing.

Example optimization. The Dictionary collection in the base class library is a huge optimization and reduces lookup time to a constant O(1). However, developers sometimes write code that results in twice as many lookups. What did I learn from this?

How to dig deeper. The JIT optimizer in .NET makes code really fast, but it can't fix everything. By benchmarking, I am able to tell what code is optimized. I gain insight into JITting.

Knowledge

With the IL Disassembler tool, we can see the internals of our C# code. However, the best way to tell that code is doing less on your Pentium is to time it. As computer scientists, we need to understand every part of how computer languages work. With benchmarking, I improved my knowledge of pipelining and locality of reference.

Locality of reference

Locality of reference is important to benchmark. This gives us insight into how CPUs work, and what is essentially the core of computer science. Things that are near are faster to access. Astronomers will tell you the speed of light. Fundamental to science is locality of reference, and this carries over into the C# language.

Natural languages

Here we look at optimization from another point of view. As I have progressed as a writer, I have worked on making my English clearer and shorter to read. I think of this as optimization in the same way as with the C# language.

Slow English What I want to show to you next is this code, which adds one to a variable in each iteration of the loop.

Fast English The loop increments the variable.

Summary

We looked at example code for benchmarking the C# programming language. Benchmarking encourages careful thinking about your code. It saves nanoseconds from your software but greatly improves the depth of your understanding. With benchmarking, I am able to grasp pipelining, file system caches, and locality of reference.