AppInsights and logging with Serilog

Overview

For our web applications, we use Application Insights for log messages and metrics. We also want to use Azure Log Stream and Live Metrics. How to set this up usually changes between major version of ASP.NET Core, so finding relevant and up to date documentation is hard.

Since our team is not the only one who has struggled with this, I decided to document how I got it to work for ASP.NET Core 3.1.

If you want to know how to set it up using the build-in Microsoft logger. You can find it here: https://arvehansen.net/codecave/2020/02/28/appinsights-and-logging

What we are going to do

I assume that you already have setup an Application Insights resource

1. Create a new ASP.NET Core Web Application
2. Replace built-in logger with Serilog
3. Hook up Application Insights using the instrumentation key
4. Publish it to Azure
5. Verify that log message gets to Application Insights
6. Add Request logging and metrics
7. Setup Azure Log Stream

Create new ASP.NET Core Web application

Open Visual Studio 2019 (I’m using version 16.4.5)

Create a new ASP.NET Core Web Application. Choose ASP.NET Core 3.1 and Web Application (MVC).

Remove the appSettings.Development.json file (not necessary, but if you want to do some local debugging you don’t have to change settings in two files)

Replace built-in logger with Serilog

We’re more or less following these instructions: https://github.com/serilog/serilog-aspnetcore, but we’ll use the configuration to configure the logger, and we will only use the Application Insights sink.

Install the following NuGet packages

• Serilog.AspNetCore
• Serilog.Sinks.ApplicationInsights
• Serilog.Enrichers.Environment (nice to have)
• Serilog.Enrichers.Process (nice to have)

Create a helper class for reading the configuration:

public static class ConfigurationHelper
{
    public static IConfigurationRoot GetConfiguration(string userSecretsKey = null)
    {
        var envName = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
        var builder = new ConfigurationBuilder()
            .SetBasePath(Directory.GetCurrentDirectory());

        builder.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);

        if (!string.IsNullOrWhiteSpace(envName))
            builder.AddJsonFile($"appsettings.{envName}.json", optional: true);

        if (!string.IsNullOrWhiteSpace(userSecretsKey))
            builder.AddUserSecrets(userSecretsKey);

        builder.AddEnvironmentVariables();

        return builder.Build();
    }
}

Configure and setup the logger in Program.cs:

public class Program
{
    public static void Main(string[] args)
    {
        var config = ConfigurationHelper.GetConfiguration();
        var loggerConfiguration = new LoggerConfiguration()
            .ReadFrom.Configuration(config);
        var telemetryConfiguration = TelemetryConfiguration
            .CreateDefault();
        telemetryConfiguration.InstrumentationKey = 
            "9****015-****-****-a9c1-c669****fca3";

        loggerConfiguration.WriteTo
            .ApplicationInsights(telemetryConfiguration, 
                TelemetryConverter.Traces);
        Log.Logger = loggerConfiguration
            .Enrich.WithMachineName()
            .Enrich.WithProcessId()
            .Enrich.FromLogContext()
            .CreateLogger();

        try
        {
            Log.Information("Starting web host");
            CreateHostBuilder(args).Build().Run();
        }
        catch (Exception ex)
        {
            Log.Fatal(ex, "Host terminated unexpectedly");
        }
        finally
        {
            Log.CloseAndFlush();
        }
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.UseStartup<Startup>();
            })
            .UseSerilog();
}

In ConfigureServices, add the logger we just created to the dependency injection container:

public void ConfigureServices(IServiceCollection services)
{
    services.AddSingleton(Log.Logger);
    services.AddControllersWithViews();
}

Replace the Logging section in appSettings.json:

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "System": "Warning",
        "Microsoft": "Warning",
        "Microsoft.EntityFrameworkCore": "Warning"
      }
    }
  },
  "AllowedHosts": "*"
}

In the home controller, replace the Microsoft logger with the Serilog logger, and create a log message using the logger:

private readonly ILogger _logger;

public HomeController(ILogger logger)
{
    _logger = logger.ForContext<HomeController>();
}

public IActionResult Index()
{
    _logger.Information("*** SERILOG *** We just called the Index action");
    return View();
}

Publish

Let’s publish it now, just to check that everything is good so far:

Check that Web App works:

Check that you got the log message in Application Insights. Go to your Application Insights and press Search:

Add Request logging and metrics

To get request logging, add the following middleware:

app.UseSerilogRequestLogging();
app.UseHttpsRedirection();
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.UseEndpoints(endpoints =>
{
    endpoints.MapControllerRoute(
        name: "default",
        pattern: "{controller=Home}/{action=Index}/{id?}");
});

To get telemetry data, install the following NuGet:

• Microsoft.ApplicationInsights.AspNetCore

and add AddApplicationInsightsTelemetry in Startup.cs. Use the instrumentation key from Application Insights:

public void ConfigureServices(IServiceCollection
{
    services.AddSingleton(Log.Logger);
    services.AddApplicationInsightsTelemetry(
        "9****015-****-****-a9c1-c669****fca3");
    services.AddControllersWithViews();
}

Click Home a few times to see that you get Live Metrics:

And that your requests are also logged:

Setup Azure Log Stream

Currently Azure Log Stream will not work out-of-the-box by just adding the Console sink or AzureApp sink. You have to follow the Serilog instructions found at the bottom of this page: https://github.com/serilog/serilog-aspnetcore.

Add the NuGet package Serilog.sinks.file to your project. Then add the File sink in Program.cs:

loggerConfiguration.WriteTo
    .ApplicationInsights(telemetryConfiguration, 
        TelemetryConverter.Events);
loggerConfiguration.WriteTo.File(
    @"D:\home\LogFiles\Application\trace.log",
    fileSizeLimitBytes: 5_000_000,
    rollOnFileSizeLimit: true,
    shared: true,
    flushToDiskInterval: TimeSpan.FromSeconds(1));

Log.Logger = loggerConfiguration
    .Enrich.WithMachineName()
    .Enrich.WithProcessId()
    .Enrich.FromLogContext()
    .CreateLogger();

Publish your changes and turn on Application Service Logs:

The go to the Log Stream blade, click Home a few times and see your log messages:

To get better performance, you should also use the Async Sink together with your file sink, but I will not show that. You can find information here: https://github.com/serilog/serilog-sinks-async