BenchmarkDotNet Introduction

This post is an introduction to a series of posts on benchmarking C# .Net code using BenchmarkDotNet. Be on the lookout for future posts on BenchmarkDotNet where I will explore using BenchmarkDotNet to benchmark and compare EF Core vs EF6

 

1. See BenchmarkDotNet and EF Core vs EF6 - Part 1 if you are familiar with BenchmarkDotNet and want to learn to configure BenchmarkDotNet for use with EF Core and EF6 See BenchmarkDotNet and EF Core vs EF6 - Part 1 for the configuration of benchmarks and database setup.

2. See BenchmarkDotNet and EF Core vs EF6 - Part 2 if you are looking for a comparison between real world EF Core 6 vs EF Core application scenarios.

3. For all code in this post and all subsequent posts on EF Core 6 vs EF6 Benchmarking, see the following project on my GitHub at https://github.com/BrianMikinski/EfPerformance

Suppose that you are a software engineer and you’ve been tasked with improving the performance of a specific set of functions within your application. As a diligent engineer, you quickly set out rewriting the ill performant code, save off a couple of commits, submit a pull request for review and claim victory. The following day, you login and see that a colleague of yours has blocked your PR and added comments claming that your code fix does not improve performance but in fact makes it worse! How dare he or she make such a claim. Luckily for you, you are reading this blog and know exactly what to do next and that is benchmark your code using BenchmarkDotNet to prove once and for all that you’re new functions are indeed faster that the code that lay there before.

In this post I’ll be exploring the fantastic BenchmarkDotNet Library library and showing you how to get started benchmarking your code in a simple demo example.This post will act as a introduction to subsequent posts on EF Core 6 vs EF6 Benchmarking.

To jump straight in to the code, see the following project on my github at https://github.com/BrianMikinski/EfPerformance and hop right to it.

An Old Friend… .Net Stopwatch Class!

If you’re doing some casual benchmarking and need a rough idea of how much faster one method implementation is vs the other, I would suggest using the tried and true .Net Stopwatch.

Here’s an example from a simple ProfilingService.cs that I wrote to double check the exciting new benchmarking tool we’ll be discussing next.

1
2
3
4
5
6
Stopwatch stopWatch = new();
stopWatch.Start();

_blogContext.Posts.FirstOrDefault();

stopWatch.Stop();

 

In this example, we simply do the following

  1. Create a new .Net Stopwatch object
  2. Start() the Stopwatch timer
  3. Run our code and then Stop() the Stopwatch timer

 

That’s it, that’s all there is to it. It’s simple, easy and works great…but there is another .Net profiling tool that we should try out.

 

Introducing…BenchmarkDotNet!

This is my first time exploring BenchmarkDotNet but my initial impressions are that it is a fantastic library for digging deep into C# code to fully understand performance from both a CPU and Memory perspective. BenchmarkDotNet allows for easy, simple (and advanced) statistical analysis through the use of method attributes on your benchmark functions. There are of course many advanced features of BenchmarkDotNet but in the most barebones use cases, BenchmarkDotNet is nothing more than adding a couple of method attributes.

benchmark_dot_net_logo.png

   

BenchmarkDotNet Driver Console App

In my limited experience with BenchmarkDotNet, it seems like the correct way to structure your benchmarking is to use a dotnet console app as your benchmark driver and then build your benchmarks in separate classes or separate projects all together. Going forward, for my EF Core vs EF6 examples, this is the structure I will be using.

 

efperformanceSolutionExplorer.png

 

Here is the Program.cs file that can be found in EFPerformance project. The BenchmarkDotNet driver console app is pretty straight forward.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using Blog.Benchmarks;

BenchmarkSwitcher.FromAssembly(typeof(BenchmarkBase).Assembly).Run(args, GetGlobalConfig());

DebugBenchmarks();

DbDiagnostics();

/// <summary>
/// Programatic configuration of the jobs.
/// This can be overwridden by passing in arguments
/// </summary>
static IConfig GetGlobalConfig()
    => DefaultConfig.Instance.AddJob(Job.Default
            .WithWarmupCount(1)
            .AsDefault());

/// <summary>
/// Method for debugging individual benchmark functionality
/// </summary>
static void DebugBenchmarks(bool isEnabled = false)
{
    if (isEnabled)
    {
        var allPostsBenchmark = new RetrieveAllPostsBenchmark();

        allPostsBenchmark.GlobalSetup();

        allPostsBenchmark.EfCorePooled();

        allPostsBenchmark.Ef6();
    }
}

/// <summary>
/// Print database diagnostics to the console
/// </summary>
static void DbDiagnostics(bool isEnabled = false)
{
    if (isEnabled)
    {
        Console.WriteLine("Post profiling statistics: ");
        var ef6ProfilingService = new ProfilingService();

        var (coreBlogDiagnostics, blogDiagnostics) = ef6ProfilingService.TableDiagnostics();

        coreBlogDiagnostics.PrintTableDiagnostics();
        blogDiagnostics.PrintTableDiagnostics(false);
    }
}

 

Most of the file can be ignored but the main line to focus in on is line 7 or the one that begins with BenchmarkSwitcher

BenchmarkSwitcher.FromAssembly(typeof(BenchmarkBase).Assembly).Run(args, GetGlobalConfig());

The BenchmarkSwitcher is what BenchmarkDotNet utilizes to know it is configuring this console app to run benchmarks. Another part of the BenchmarkSwitcher is to keep track of is the FromAssembly(typeof(BenchmarkBase).Assembly).

FromAssembly allows us to tell BenchmarkDotNet where to look for our benchmark methods. In this case, BenchmarkBase is referencing BenchmarkBase.cs that I have created in a separate project called BlogBenchmarks where all of my benchmarks are grouped into classes around functionality.

Now let’s get onto the good stuff. Some actual code using BenchmarkDotNet!

Benchmark Demo Class

Here is an DemoBenchmark.cs class to demonstrate some baseline functionality.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
using BenchmarkDotNet.Attributes;

namespace Blog.Benchmarks;

public class DemoBenchmark : BenchmarkBase
{
    [GlobalSetup]
    public void GlobalSetup()
    {
       // code that is run once prior to all BenchmarkDotNet iterations
    }

    [IterationSetup]
    public void IterationSetup()
    {
        // code that is run once prior to each iteration
    }

    [Benchmark(Baseline = true)]
    public void OldFunction()
    {
        Thread.Sleep(50);
    }

    [Benchmark]
    public void NewFunction()
    {
        Thread.Sleep(100);
    }

    [IterationCleanup]
    public void IterationCleanup()
    {
        // code run once after each iteration
    }

    [GlobalCleanup]
    public void GlobalCleanup()
    {
       // code run once after all iterations have run
    }
}

 

In the DemoBenchmark.cs file we can see the [Benchmark] attribute for identifying methods that we want to be benchmarked.

BenchmarkDotNet Automation with PowerShell

Because this project holds many Benchmarks, and running all of the benchmarks at once would take quite a long time (due to the fact that we are going to be continually hitting a database for our tests) I have my benchmarks automated to run specific categories using PowerShell scripts.

 

Here is an example of the demo.Benchmark.ps1 file

1
2
# run benchmarking code
dotnet run -c Release --filter *DemoBenchmark*

 

In the code above, the –filter argument get’s passed to the .Run(args, GetGlobalConfig()). line of the Program.cs console app. This can be traced through the following BenchmarkSwitcher.FromAssembly code

1
BenchmarkSwitcher.FromAssembly(typeof(BenchmarkBase).Assembly).Run(args, GetGlobalConfig());

 

And, from the previous section above showing the full DemoBenchmark class, we have our legacy benchmark functions marked with attributes the [Benchmark(Baseline = true)] baseline attribute telling BenchmarkDotNet that we want the OldFunction to be the baseline for displaying ratios against the new Benchmark.

1
2
3
4
5
6
7
8
9
10
11
public class DemoBenchmark : BenchmarkBase
{
    // ...
    // setup code
    // ...
    [Benchmark(Baseline = true)]
    public void OldFunction()
    {
        Thread.Sleep(50);   
    }
}

   

So far, this setup for running benchmarks on methods has worked very well. I am able to quickly and repeatedly test specific benchmark methods yet still maintain a clean level of organization across the benchmark’s containing classes.  

Results of the DemoBenchmark Class

Another great thing about BenchmarkDotNet is the ease at which one can share the results of their benchmarks. My blog is written in Markdown and lucky for me, BenchmarkDotNet automatically generates results in multiple formats - one of which is an .md Markdown file that I have included below.

1
2
3
4
5
6
7
8
BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1586 (21H1/May2021Update)
Intel Core i5-6200U CPU 2.30GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
.NET SDK=6.0.201
  [Host]     : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT
  Job-HWYLMR : .NET 6.0.3 (6.0.322.12309), X64 RyuJIT

InvocationCount=1  UnrollFactor=1  WarmupCount=1  
Categories=DemoBenchmark  

 

Method Mean Error StdDev Min Max Ratio RatioSD
OldFunction 58.80 ms 1.124 ms 1.500 ms 56.09 ms 61.17 ms 1.00 0.00
NewFunction 104.05 ms 1.375 ms 1.148 ms 102.20 ms 106.09 ms 1.75 0.03

As you can see from the results in this contrived example, the NewFunction (Thread.Sleep(100);) code is about ~1.75 times slower than the OldFunction code (Thread.Sleep(50);). This is exactly what we would expect from our the benchmark code we are analyzing. If we were rewriting this method in a real world scenario, we would not want to use the new implementation over the older one!

BenchmarkDotNet Conclusion

In conclusion, BenchmarkDotNet is an absolutely incredible tool and one that I will no doubt continue to explore. In the follow up series of posts, I intend to put EF Core 6 and EF6 to the test using BenchmarkDotNet to dig deep and understand under what scenarios which EF version is faster than the other. Until next time, have a Wonderful Day and Happy Coding!  

If you want to view all of the code show in this post, you can view the source at the following github repo https://github.com/BrianMikinski/EfPerformance

 


Gravatar Image

Brian Mikinski

Software Architect - Houston, Tx