BenchmarkDotNet and EF Core vs EF6 - Part 1

I am an unabashed fanboy of Entity Framework. EF Core 6 is my current tool of choice but I fondly remember stumbling upon EF4 as a fresh, starry eyed engineer many years ago and being amazed that something as easy as LINQ could serve data from a database. Sure, sometimes it’s necessary to drop into SQL Land and reconnect with your past but like milk and cookies, peanut butter and jelly or even Siegfried and Roy – Entity Framework and databases just go good together.

These days, with EF Core 6 being almost at full parity with EF6, I exclusively use EF Core for database access but I’ve always been interested in how much more performant EF Core is than legacy EF6.

Putting my curiosity into action and utilizing the fantastic BenchmarkDotNet Library, I am going to be pitting EF Core against EF6 in this and the next series of blog posts.

For all code in this post and all subsequent posts on EF Core 6 vs EF6 Benchmarking, see the following project on my github at https://github.com/BrianMikinski/EfPerformance

What we’re testing - FirstOrDefault()

One LINQ method that engineers can use in both EF Core and EF 6 is FirstOrDefault(). FirstOrDefault will return the following -

1. A single entity from a table in the database or
2. A null value if the table contais no records

While FirstOrDefault() isn’t necessarily the most useful method for writing real world EF queries. It does offer a great starting place for us to learn how to configure EF Core tests.

Solution Layout

Before diving into how we can automate the creation and teardown of our benchmarks, let’s first get the lay of the land for the solution where we will be running our benchmarks. There are four C# projects in our solution -

• Blog
• Core.Blog Project
• Blog.Benchmarks
• EFPerformance

Blog

This project contains the C# entities and a BlogDbContext for building out a simple blog database using legacy EF6. Our database contains the following entities -

• Post
• Category
• Tag
• PostTag

For the DbContext, we’re creating a database model that contains a 1 to 1, 1 to many and many to many relationship. This will give us a good base to work from and cover many of the types of database operations and queries that we will run with Entity Framework.

Core.Blog

The Core.Blog project contains an identical db table structure and entity relationships as the EF6 Blog project. The db designs are kept identical between both EF6 and EF Core projects to ensure that we are comparing apples to apples when running our benchmarks. The EF Core DbContext is named CoreBlogContext to help keep the DbContext’s easily identifiable and typable when writing code.

Blog.Benchmarks

Blog.Benchmarks contains all of the actual EF Core and EF6 benchmark classes, database configuration and setup code, as well as database seeding methods.

• Ef Core and EF6 benchmark classes
  • All of the C# classes and files ending in *Benchmark.cs
• BlogContext and CoreBlogContext db creation/deletion
  • See BenchmarkBase.cs
• Seed data methods
  • See BenchmarkBase.cs

EFPerformance

The final project required to run benchmarks is the EFPerformance project. This project is a simple console application that acts as a driver for running our benchmarks. For a more indepth explanation of working with the BenchmarkSwitcher, see the following post Introduction to BenchmarkDotNet

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using Blog.Benchmarks;

BenchmarkSwitcher.FromAssembly(typeof(BenchmarkBase).Assembly).Run(args, GetGlobalConfig());

DebugBenchmarks();

DbDiagnostics();

/// <summary>
/// Programatic configuration of the jobs.
/// This can be overwridden by passing in arguments
/// </summary>
static IConfig GetGlobalConfig()
    => DefaultConfig.Instance.AddJob(Job.Default
            .WithWarmupCount(1)
            .AsDefault());

...

FirstOrDefault() Benchmark

The actual code for running the EF6 vs EF Core Benchmark is very simple. Much of this simplicity can be attributed to the use of the abstract base class BenchmarkBase. The abstract base class BenchmarkBase.cs allows us to split out our benchmark tests across multiple C# classes and yet still share all required setup and configuration code for both the EF 6 and EF Core databases.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
using BenchmarkDotNet.Attributes;
using Blog.Models;
using CoreBlog.Models;

namespace Blog.Benchmarks;

public class FirstOrDefaultBenchmark : BenchmarkBase
{
    [GlobalSetup]
    public void GlobalSetup()
    {
        ConfigDatabases();
        AddPostsToSeedLimit();
    }

    [Benchmark(Baseline = true)]
    public void Ef6()
    {
        using var context = new BlogContext();
        _ = context.Posts.FirstOrDefault();
    }

    [Benchmark]
    public void EfCore()
    {
        using var context = new CoreBlogContext(CoreBlogContext.NewDbContextOptions());
        _ = context.Posts.FirstOrDefault();
    }

    [Benchmark]
    public void EfCorePooled()
    {
        using var context = _corePooledDbContextFactory.CreateDbContext();
        _ = context.Posts.FirstOrDefault();
    }
}

Configuring Entity Framework for BenchmarkDotNet

In order to run our to EF6 vs EF Core benchmarks for the FirstOrDefault() command, we need to perform a couple of setup tasks prior to each benchmark iteration set. The first thing we need to do is build out new databases on our local machine. This is accomplished by the following code snippets -

FirstOrDefaultBenchmark.cs - GlobalSetup()

1
2
3
4
5
6
7
8
9
10
11
public class FirstOrDefaultBenchmark : BenchmarkBase
{
    [GlobalSetup]
    public void GlobalSetup()
    {
        ConfigDatabases();

        // igore this line for now
        //AddPostsToSeedLimit();
    }
  ...

BenchmarkBase.cs - ConfigureDatases()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
// BenchmarkBase.cs
using BenchmarkDotNet.Attributes;
using Blog.Models;
using CoreBlog.Models;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;

namespace Blog.Benchmarks;

[MemoryDiagnoser]
[RPlotExporter]
[MinColumn]
[MaxColumn]
public abstract class BenchmarkBase
{
    protected const string EF_CORE_CATEGORY = "EF Core";
    protected const string EF_6_CATEGORY = "EF 6";

    protected int SeedLimit = 1000;

    protected PooledDbContextFactory<CoreBlogContext> _corePooledDbContextFactory;

    ...

    /// <summary>
    /// Configure database setup
    /// </summary>
    public void ConfigDatabases()
    {
        // ef 6
        BlogContext _blogContext = new();

        _blogContext.Database.Delete();
        _blogContext.Database.Create();

        // ef core
        _corePooledDbContextFactory = new PooledDbContextFactory<CoreBlogContext>(CoreBlogContext.NewDbContextOptions());

        CoreBlogContext _coreBlogContext = new CoreBlogContext(CoreBlogContext.NewDbContextOptions());

        _coreBlogContext.Database.EnsureDeleted();
        _coreBlogContext.Database.EnsureCreated();
    }

    ...

BenchmarkBase.ConfigDatabases()

BenchmarkBase.ConfigDatases() is a particularly interesting method. This call does a number of cool things that allow us to stay within the realm of Entity Framework yet still perform all of the necessary database work to configure clean databases for each BenchmarkDotNet run. It is very important to keep our tests clean and repeatable across the same size dataset on every benchmark and it’s subsequent iterations.

Another interesting aspect of this config method is that it introduces the EF Core concept of PooledDbContexts. While I won’t go into the details of a pooled DbContext in this post, as you will see, pooled DbContexts are a fantastic feature added to EF Core 2+ and I highly suggest you review the Advanced Performance section of the Microsoft docs to learn more about EF Core’s DbContext pooling.

BenchmarkBase.SeedData()

The second required configuration for our application is to seed data into our EF6 and EF Core databases. Here’s the code that does just that. Be sure to focus in on the BenchmarkBase.AddPostsToSeedLimit() method.

FirstOrDefaultBenchmark.cs - GlobalSetup()

1
2
3
4
5
6
7
8
9
public class FirstOrDefaultBenchmark : BenchmarkBase
{
    [GlobalSetup]
    public void GlobalSetup()
    {
        ConfigDatabases();
        AddPostsToSeedLimit();
    }
  ...

BenchmarkBase.cs - AddPostsToSeedLimit()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
using BenchmarkDotNet.Attributes;
using Blog.Models;
using CoreBlog.Models;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Infrastructure;

namespace Blog.Benchmarks;

[MemoryDiagnoser]
[RPlotExporter]
[MinColumn]
[MaxColumn]
public abstract class BenchmarkBase
{
    protected const string EF_CORE_CATEGORY = "EF Core";
    protected const string EF_6_CATEGORY = "EF 6";

    protected int SeedLimit = 1000;

    protected PooledDbContextFactory<CoreBlogContext> _corePooledDbContextFactory;

    // constructors + other code unrelated to data seeding
    ... 

    /// <summary>
    /// Seed ef 6 and ef core databases
    /// </summary>
    public void AddPostsToSeedLimit(int? overridenLimit = null, bool isBulkInsert = false)
    {
        if (overridenLimit.HasValue)
        {
            SeedLimit = overridenLimit.Value;
        }

        if (!isBulkInsert)
        {
            PostsAddEfCore();
            PostsAddEf6();
        }
        else
        {
            PostsAddBulkInsertEfCore(); ;
            PostsAddBulkInsertEf6();
        }
    }

    /// <summary>
    /// Seed ef 6 and ef core databases
    /// </summary>
    public void AddPostsToSeedLimit(int? overridenLimit = null, bool isBulkInsert = false)
    {
        if (overridenLimit.HasValue)
        {
            SeedLimit = overridenLimit.Value;
        }

        if (!isBulkInsert)
        {
            PostsAddEfCore();
            PostsAddEf6();
        }
        else
        {
            PostsAddBulkInsertEfCore(); ;
            PostsAddBulkInsertEf6();
        }

    }

    /// <summary>
    /// EF Core multiple posts using native Add
    /// </summary>
    protected void PostsAddEfCore()
    {
        using var context = _corePooledDbContextFactory.CreateDbContext();

        for (int i = 0; i < SeedLimit; i++)
        {
            var post = PostCore.NewPost();
            context.Posts.Add(post);
            context.SaveChanges();
        }
    }

    /// <summary>
    /// EF Core multiple posts with bulk insert
    /// </summary>
    protected void PostsAddBulkInsertEfCore(int? overrideLimit = null)
    {
        using var context = _corePooledDbContextFactory.CreateDbContext();

        var limit = overrideLimit ?? SeedLimit;

        List<PostCore> posts = new();

        for (int i = 0; i < limit; i++)
        {
            posts.Add(PostCore.NewPost());
        }

        context.BulkInsert(posts);
    }

    /// <summary>
    /// EF6 multiple post with native add
    /// </summary>
    protected void PostsAddEf6()
    {
        using var context = new BlogContext();

        for (int i = 0; i < SeedLimit; i++)
        {
            var post = Post.NewPost();

            context.Posts.Add(post);
            context.SaveChanges();
        }
    }

    /// <summary>
    /// EF6 multiple psots add
    /// </summary>
    protected void PostsAddBulkInsertEf6(int? overrideLimit = null)
    {
        using var context = new BlogContext();

        var limit = overrideLimit ?? SeedLimit;

        List<Post> posts = new();

        for (int i = 0; i < limit; i++)
        {
            posts.Add(Post.NewPost());
        }

        context.BulkInsert(posts);
    }

    ...

BenchmarkBase.AddPostsToSeedLimit()

On line 28, the public void AddPostsToSeedLimit(int? overridenLimit = null, bool isBulkInsert = false) method is where we are inserting records for our benchmarks. As you can see, this method contains default parameters that can be used to override the default 1000 records being inserted into the database. Additionally, it allows you to change the method of insertion into the table. For the most part, you can ignore the isBulkInsert = false method until we review more complicated EF benchmark scenarios and actually need to use that parameter.

Benchmark Results and Analysis

At this point, we’ve built out the EF Benchmark functions, configured the database creation/deletion and enabled database seeding prior to each function iteration so it’s time to run the benchmarks. As previously mentioned, I like to run my benchmarks from the command line using PowerShell scripts to hone in on the individual class I am attempting to benchmark. In my previous post Introduction to BenchmarkDotNet, I go a little more indepth in running BenchmarkDotNet benchmarks from the command line but here is a copy of the FirstOrDefault.benchmark.ps1 script that I am using for this run.

1
2
# run benchmarking code
dotnet run -c Release --filter *FirstOrDefaultBenchmark*

And with that simple command, all of our benchmarking code will be run for testing the performance of FirstOrDefault() in both EF6 and EF Core. As you can see below, the results are very interesting…

1
2
3
4
5
6
7
8
BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1706 (21H1/May2021Update)
Intel Core i5-6200U CPU 2.30GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
.NET SDK=6.0.203
  [Host]     : .NET 6.0.5 (6.0.522.21309), X64 RyuJIT
  Job-EQYWMU : .NET 6.0.5 (6.0.522.21309), X64 RyuJIT

WarmupCount=1  

One of the most surprising aspects of our benchmark is the fact that EF Core performs WORSE (677.1 μs vs 547.0 μs) than EF6 for a simple FirstOrDefault() query?

Why is this the case? I was very interested in this myself and I didn’t initially have a great explanation so I filed a Github issue for this and the response I received was very informative. It also and led me down the path of adding the 3rd EfCorePooled benchmark that you see above here. Effectively, EF Core performs worse than EF6 for a single entity because it must configure it’s own internal services everytime a new context is created. That performance hit isn’t insignificant and adds roughly 100 ms to most calls.

In our 3rd benchmark with Pooled DbContext’s, we can see that the Mean time for a Pooled EF Core context is over 2x faster than the EF6! What is happening? Well, in the case of Pooled EF Core contexts, we are effectively utlizing a singleton db context that is created from a factory. Everytime the factory creates a new db context for our benchmark, the same singleton context is reused but it’s state is resut to the initial state with no entities attached nor any other data loaded into the context. In this manner we are able to save substantially on the EF Core internal service setup.

But that’s not the only benefit that EF Core pooled contexts bring us. Additionally, the pooled contexts allocate far less memory! Overally, EF Core pooled DbContexts crush the performance of EF6 from both a time and space complexity perspective.

EF Benchmarking Best Practices and Conclusion

Overall, the more I work with BenchmarkDotNet the more useful I continue to find the library. By configuring BenchmarkDotNet for any application, we can produce hard evidence in an easily digestable medium to prove out the performance improvements or regressions in our code. In building out this first EF6 vs EF Core benchmarking example, here are some of the best practices that I’ve come across -

1) Do NOT use [IterationSetup] for DbContext instantiation! Why? As we have seen from the results, DbContext instantiation is probably going to be a CPU expensive operation. Because it is an expensive operation, if we run it during every iteration setup we can skew our results for EVERY benchmark run. This will make our results uninterpretable because the contexts consume a relatively large amount of CPU time throughout the benchmark compared to the query execution.

1
2
3
4
5
6
7
8
9
// DO NOT INSTANTIATE DBCONTEXTS IN ITERATION SETUP CODE
[IterationSetup]
public void IterationSetup()
{
    // code that is run once prior to each iteration
    
    // DO NOT DO THIS!
    using var context = new BlogContext();
}

2) Do NOT reuse DbContexts outside of their benchmarking function! Why? First off, DbContexts are generally suggested to have a short and sweet lifetime. Sharing the DbContext across benchmark iterations is an anti-pattern. Create them, destroy them, and if you need another one just new() it up. Resuing your DbContexts can also complicate your benchmark setup because it means you have to ensure you are resetting the context, detaching/removing entities, or using AsNoTracking() on your sql queries.

1
2
3
4
5
6
// DO NOT USE SINGLETON CONTEXTS FOR YOUR BENCHMARK ANALYSIS
[Benchmark(Baseline = true)]
public void Ef6()
{
    _ = _blogContextSingleton.Posts.FirstOrDefault();
}

3) DO instantiate DbContexts in individual benchmarks! Why? DbContext instantiation has both a CPU and Memory cost and that will affect the performnce of your code. Instantiating a DbContext in your benchmarks insures that this cost is included in your results. Additionally, managing the state of contexts’ is much easier.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// DO INSTANTIATE YOUR CONTEXTS IN EACH BENCHMARK
[Benchmark(Baseline = true)]
public void Ef6()
{
    using var context = new BlogContext();
    _ = context.Posts.FirstOrDefault();
}

[Benchmark]
public void EfCore()
{
    using var context = new CoreBlogContext(CoreBlogContext.NewDbContextOptions());
    _ = context.Posts.FirstOrDefault();
}

4) Pooled DbContexts give the best CPU and memory performance! Why? When using pooled DbContexts, EF effectively creates a singleton DbContext that is properly managed by Entity Framework. When the PooledDbContextFactory.CreateDbContext() is called, the factory creates a reference from the singleton DbContext for little CPU and Memory overhead. This reference is guaranteed by EF to be correctly reset to it’s default state, with no entities attached to the context insuring a clean execution of your EF queries.

1
2
3
4
5
6
7
// DO USE DBCONTEXT POOLING TO CREATE YOUR CONTEXTS
[Benchmark]
public void EfCorePooled()
{
    using var context = _corePooledDbContextFactory.CreateDbContext();
    _ = context.Posts.FirstOrDefault();
}

In conclusion, even when testing intricate libraries such as Entity Framework 6 and Entity Framework Core, BenchmarkDotNet continues to shine. It is an incredible library that allows you to truly dig deep into the CPU and Memory peformance of your code as well as scientifically prove enhancements or diagnose performance bottlenecks that you wouldn’t have otherwise had the tools or data to do so. I hope you enjoyed this post on BenchmarkDotNet, and Entity Framework. In the meanwhile, stay tuned for more posts on EF6 vs EF core performance and Happy Coding!

If you want to view all of the code show in this post, you can view the source at my github repo https://github.com/BrianMikinski/EfPerformance