BenchmarkDotNet - Baseline Styling

Introduction

One of my absolute favorite libraries for .Net Core is BenchmarkDotNet. BenchmarkDotNet is a .Net Core nuget package that allows engineers to perform statistical benchmarks on the CPU execution time and RAM performance of C# functions through the simple use of [Benchmark] function attribute. In this blog post, we are going to review BenchmarkDotNet Baseline Styling. For reference, if you are interested in a more detailed introduction to BenchmarkDotNet, see my introductory blog post here .

If this isn’t your first rodeo with the wonderful library, check out how to run benchmark comparisons of common EF6 and EF Core 6 database query situations.

Benchmark Baselines and Summary Styling

When building a benchmark suite to test performance improvements to a code base, the first-place engineers should start as they examine their code is by establishing the baseline execution metrics. BenchmarkDotNet allows this to be easily done by adding [Baseline(true)] to any method in the benchmark suite. Once the baseline has been set, the BenchmarkDotNet summary graph will show the ratio of all benchmarked functions to the baseline. See the following example -

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<BearBenchmark>();

/// <summary>
/// Benchmarking files
/// </summary>
public class BearBenchmark
{
    [Benchmark(Baseline = true)]
    public void Hibernate() => Thread.Sleep(10);

    [Benchmark]
    public void HibernateImproved() => Thread.Sleep(5);

    [Benchmark]
    public void HibernateSuperImproved() => Thread.Sleep(2);
}

1
2
3
4
5
6
BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19043.2130/21H1/May2021Update)
Intel Core i5-6200U CPU 2.30GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
.NET SDK=6.0.402
  [Host]     : .NET 6.0.10 (6.0.1022.47605), X64 RyuJIT AVX2
  DefaultJob : .NET 6.0.10 (6.0.1022.47605), X64 RyuJIT AVX2

While the default summary table is very nice and BenchmarkDotNet will gladly create markdown files and fancy R graphs for your benchmarks, one great improvement to summary table is styling the ratio column to output as trends rather than basic ratios. This can easily be accomplished by adding a BenchmarkDotNet styling [ManualConfig] attribute class and defining the our own manual and custom styling for the baseline ratio. Check out the following improvement -

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<BearBenchmark>();

/// <summary>
/// Benchmarking files
/// </summary>
[Config(typeof(StyleConfig))]
public class BearBenchmark
{
    [Benchmark(Baseline = true)]
    public void Hibernate() => Thread.Sleep(10);

    [Benchmark]
    public void HibernateImproved() => Thread.Sleep(5);

    [Benchmark]
    public void HibernateSuperImproved() => Thread.Sleep(2);
}


/// <summary>
/// Style config for baseline summary style
/// </summary>
public class StyleConfig : ManualConfig
{
    public StyleConfig()
    {
        SummaryStyle = SummaryStyle.Default.WithRatioStyle(RatioStyle.Trend);
    }
}

1
2
3
4
5
6
BenchmarkDotNet=v0.13.2, OS=Windows 10 (10.0.19043.2130/21H1/May2021Update)
Intel Core i5-6200U CPU 2.30GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
.NET SDK=6.0.402
  [Host]     : .NET 6.0.10 (6.0.1022.47605), X64 RyuJIT AVX2
  DefaultJob : .NET 6.0.10 (6.0.1022.47605), X64 RyuJIT AVX2

Conclusion

While the styling is simple, it adds greatly to the readability of the benchmark and rather than have to do the ratio math, you can simply read from the graph to determine if a benchmark is actually faster or slower than before! Thanks for checking in and feel free to leave some comments below! Until then, happy coding!

Resources

Benchmark Style Config Source Code

BenchmarkDotNet Baseline Styling Docs