Use service-to-service correlation in Azure Service Bus solutions

The concept of service-to-service correlation is big and complex and spans multiple Arcus libraries. This user-guide will walk through all the available Arcus features that work together to facilitate service-to-service correlation in your Azure Service Bus solutions. For a general introduction, see the introduction page on service-to-service correlation.

To use service-to-service correlation in Azure Service Bus solutions, both the sending and receiving side of the separate components have to be adapted. The correlation will be passed through from one service to another via the Azure Service Bus message application properties which Arcus will use to link your services together.

The following diagram shows how a Web API calls a Service Bus with correlation tracking:

How does this work?

Three kind of correlation ID's are used to create the relationship:

• Transaction ID: this ID is the one constant in the diagram. This ID is used to describe the entire transaction, from beginning to end. All telemetry will be tracked under this ID.
• Operation ID: this ID describes a single operation within the transaction. This ID is used within a service to link all telemetry of that service together.
• Operation Parent ID: this ID is create the parent/child link across services. When service A calls service B, then service A is the so called 'parent' of service B.

The following list shows each step in the diagram:

1. The initial call in this example doesn't contain any correlation headers. This can be seen as a first interaction call to a service.
2. Upon receiving at service A, the application will generate new correlation information. This correlation info will be used when telemetry is tracked on the service.
3. When a call is made to service B, the transaction ID is sent but also the operation parent ID in the form of a hierarchical structure.
4. The jkl part of this ID, describes the new parent ID for service B (when service B calls service C, then it will use jkl as parent ID)
5. The user receives both the transaction ID and operation parent ID in their final response.

💡 Additional configuration is available to tweak this functionality, see the dedicated Arcus Messaging feature documentation for more in-depth information on Azure Service Bus correlation.

Service Bus demonstration

In this user-guide, a fictive API and Service Bus application will be used to represent Service A and Service B of the diagram. Both services will be adapted to use service-to-service correlation.

• Order API (Service A): receives an request to order a product, sends the order to process.
• Order Worker (Service B): receives the order request and processes the order.

The user interacts with the Product API to order their product. Internally, the Order API will contact the Order Worker to further process the order request.

⚠ Take into account that this sample should only be used for demonstration purposes and does not reflect a fully production-ready implementation. We have chosen to only provide the bare bones of the application to be able to focus on the changes required for service-to-service correlation.

Order API: startup code

First, lets look at the initial code for the Order API. The startup code has the hosting and routing functionality to use API controllers. The Azure.Messaging.ServiceBus package is added so that the AddServiceBusClientWithNamespace becomes available. This will inject a ServiceBusClient in the application which will contact the Order Worker.

using Microsoft.Extensions.Azure;

public class Program
{
    public static void Main(string[] args)
    {
        WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
        builder.Services.AddRouting();
        builder.Services.AddControllers();

        builder.Services.AddAzureClients(clients =>
        {
          clients.AddServiceBusClientWithNamespace("<fully-qualified-servicebus-namespace>")
                 .WithName("Order Worker")
                 .WithCredential(new ManagedIdentityCredential());
        });

        WebApplication app = builder.Build();
        app.UseRouting();
        app.UseEndpoints(endpoints => endpoints.MapControllers());
        app.Run("http://localhost:787");
    }
}

The sole API controller in the Order API makes sure that we receive the product order request and contact the Order Worker to further process the order request.

using Azure.Messaging.ServiceBus;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Azure;

[ApiController]
[Route("api/v1/order")]
public class OrderController : ControllerBase
{
    private readonly ServiceBusSender _serviceBusSender;
    private readonly ILogger<OrderController> _logger;

    public OrderController(
        IAzureClientFactory<ServiceBusClient> clientFactory,
        ILogger<OrderController> logger)
    {
         ServiceBusClient client = clientFactory.CreateClient("Order Worker");
        _serviceBusSender = client.CreateSender("orders");
        _logger = logger;
    }

    [HttpPost]
    public async Task<IActionResult> Post([FromBody] ProductOrderRequest productRequest)
    {
        var order = new Order(productRequest);
        var data = BinaryData.FromObjectAsJson(order);
        var message = new ServiceBusMessage(data);

        await _serviceBusSender.SendMessageAsync(message);
    }
}

Order Worker: startup code

The Order Worker application will use mostly Arcus functionality, so for now, the startup code is only creating the hosting functionality:

using Microsoft.Extensions.Azure;

public class Program
{
    public static void Main(string[] args)
    {
       Host.CreateDefaultBuilder(args)
           .Build()
           .Run();
    }
}

Right now, we haven't used any Arcus functionality, only common Microsoft-available features.

Order API: add Arcus correlation

Now that we have explained the application code, we can add the Arcus functionality that will provide clear telemetry for the API interaction.

First, these packages need to be installed:

PM > Install-Package Arcus.WebApi.Logging -MinimumVersion 1.6.1
PM > Install-Package Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights -MinimumVersion 2.6.0

For more information on Arcus.WebApi.Logging, see these dedicated feature docs, for more information on Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights, see these dedicated feature docs.

The following additions have to be made to the startup code for the Product API to be able to track the incoming/outgoing HTTP requests, and to log to Application Insights.

using Microsoft.Extensions.Azure;
using Serilog;
using Serilog.Configuration;
using Serilog.Events;

public class Program
{
    public static void Main(string[] args)
    {
        WebApplicationBuilder builder = WebApplication.CreateBuilder(args);
        builder.Services.AddRouting();
        builder.Services.AddControllers();

        // [Add] Makes HTTP correlation available throughout application
        builder.Services.AddHttpCorrelation();

        builder.Services.AddAzureClients(clients =>
        {
          clients.AddServiceBusClientWithNamespace("<fully-qualified-servicebus-namespace>")
                 .WithName("Order Worker")
                 .WithCredential(new ManagedIdentityCredential());
        });

        // [Add] Serilog configuration that writes to Application Insights
        builder.Host.UseSerilog((context, provider, config) =>
        {
            config.MinimumLevel.Verbose()
                  .MinimumLevel.Override("Microsoft", LogEventLevel.Information)
                  .Enrich.WithComponentName("Order API")
                  .Enrich.WithHttpCorrelationInfo(provider)
                  .WriteTo.AzureApplicationInsightsWithConnectionString("<connection-string>");
        });

        WebApplication app = builder.Build();
        // [Add] Gets HTTP correlation from incoming request
        app.UseHttpCorrelation();
        app.UseRouting();
        // [Add] Tracks incoming request
        app.UseRequestTracking();
        app.UseEndpoints(endpoints => endpoints.MapControllers());
        app.Run("http://localhost:787");
    }
}

Following additions are made:

• builder.Services.AddHttpCorrelation(): adds the HTTP correlation services to the application services. This registers the IHttpCorrelationInfoAccessor that is used to get/set the HTTP correlation throughout the application. (more info)
• WriteTo.AzureApplicationInsightsWithConnectionString: Serilog's sink that writes all the logged telemetry to Application Insights (more info)
• app.UseHttpCorrelation(): retrieves the HTTP correlation from the incoming request or generates a new set (first request). This correlation information is set into the IHttpCorrelationInfoAccessor. (more info).
• app.UseRequestTracking(): tracks the incoming HTTP request as a telemetry request. (more info).

⚠ Note that when initializing the Application Insights Serilog sink, you should use the Arcus secret store to retrieve this connection string. Setting this up requires you to reload the logger after the application is build. For more information, see this dedicated section that describes how to do this.

⚠ Note that the order of the middleware component registrations is important. The HTTP request tracking needs the endpoint routing to figure out if the request should be tracked, for example. For more information on our different middleware components, see our Web API feature documentation.

With these HTTP correlation additions, we can alter our API controller to include the correlation tracking when a new product order request is placed on the Service Bus queue:

using Azure.Messaging.ServiceBus;
using Arcus.Observability.Correlation;
using Arcus.WebApi.Logging.Core.Correlation;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Azure;

[ApiController]
[Route("api/v1/order")]
public class OrderController : ControllerBase
{
    private readonly ServiceBusSender _serviceBusSender;
    private readonly ILogger<OrderController> _logger;
    private readonly IHttpCorrelationInfoAccessor _correlationAccessor;

    public OrderController(
        // [Add] Inject the application's available correlation accessor
        IHttpCorrelationInfoAccessor correlationAccessor,
        IAzureClientFactory<ServiceBusClient> clientFactory,
        ILogger<OrderController> logger)
    {
         ServiceBusClient client = clientFactory.CreateClient("Order Worker");
        _serviceBusSender = client.CreateSender("orders");
        _correlationAccessor = correlationAccessor;
        _logger = logger;
    }

    [HttpPost]
    public async Task<IActionResult> Post([FromBody] ProductOrderRequest productRequest)
    {
        var order = new Order(productRequest);

        // [Add] Retrieve the current application correlation
        CorrelationInfo correlationInfo = _correlationAccessor.GetCorrelationInfo();

        // [Add] Send the application correlation and logger when the message is send, using one of Arcus' extensions
        await _serviceBusSender.SendMessageAsync(order, correlationInfo, _logger);
    }
}

Following additions are made:

• IHttpCorrelationInfoAccessor: is the correlation accessor implementation for Web API applications. This instance is used to get and set the correlation for a scoped HTTP request. For more information on HTTP correlation, see our Web API feature documentation.
• SendMessageAsync(message, correlationInfo, _logger): is an extensions on the ServiceBusSender that allows consumers to track the Service Bus message. This will make sure that the send operation is tracked as an Azure Service Bus dependency and that the Order Worker will be able to link the correlation. For more information on Azure Service Bus tracking, see our Service Bus feature documentation.

Order Worker: add Arcus functionality

The Order Worker didn't had any implementation, so let's add this now. We want to receive a message on an Azure Service Bus queue, and track that as a linked request in Application Insights. We will be using the Arcus message pump for this, as it allows us to focus solely on the message handling and does request tracking built-in.

First, let's install these packages:

PM > Install-Package Arcus.Messaging.Pumps.ServiceBus -MinimumVersion 1.3.0
PM > Install-Package Arcus.Observability.Telemetry.Serilog.Sinks.ApplicationInsights -MinimumVersion 2.6.0
PM > Install-Package Serilog.Extensions.Hosting

These packages allows us to register the message pump and the Serilog logging:

using Microsoft.Extensions.Azure;
using Microsoft.Extensions.DependencyInjection;
using Serilog;
using Serilog.Configuration;
using Serilog.Events;

public class Program
{
    public static void Main(string[] args)
    {
        // [Add] Serilog configuration that writes to Application Insights
        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Verbose()
            .MinimumLevel.Override("Microsoft", LogEventLevel.Information)
            .Enrich.FromLogContext()
            .Enrich.WithComponentName("Order Worker")
            .WriteTo.AzureApplicationInsightsWithConnectionString("<connection-string>")
            .CreateLogger();

       Host.CreateDefaultBuilder(args)
           .ConfigureServices(services =>
           {
                // [Add] Arcus message pump on Service Bus 'orders' queue
                services.AddServiceBusQueueMessagePumpUsingManagedIdentity("orders", "<fully-qualified-servicebus-namespace>")
                        // [Add] Arcus message handler that processes the received 'order' on the Service Bus queue
                        .WithServiceBusMessageHandler<OrderMessageHandler, Order>();
           })
           .Build()
           // [Add] Register the Serilog logger as the application's logger
           .UseSerilog(Log.Logger)
           .Run();
    }
}

Following additions are made:

• WriteTo.AzureApplicationInsightsWithConnectionString: Serilog's sink that writes all the logged telemetry to Application Insights (more info)
• AddServiceBusQueueMessagePumpUsingManagedIdentity: registers an Arcus message pump listening on an Azure Service Bus 'orders' queue. Received messages will automatically be tracked as Service Bus requests in Application Insights.
• WithServiceBusMessageHandler: registers a custom OrderMessageHandler to process the deserialized Order message.

⚠ Note that when initializing the Application Insights Serilog sink, you should use the Arcus secret store to retrieve this connection string. Setting this up requires you to reload the logger after the application is build. For more information, see this dedicated section that describes how to do this.

The OrderMessageHandler is currently doing nothing:

using Arcus.Messaging.Abstractions;
using Arcus.Messaging.Abstractions.ServiceBus;
using Arcus.Messaging.Abstractions.ServiceBus.MessageHandling;
using Microsoft.Extensions.Logging;

public class OrderMessageHandler : IAzureServiceBusMessageHandler<Order>
{
    private readonly ILogger<OrderMessageHandler> _logger;

    public OrderMessageHandler(ILogger<OrderMessageHandler> logger)
    {
        _logger = logger;
    }

    public async Task ProcessMessageAsync(
        Order message,
        AzureServiceBusMessageContext messageContext,
        MessageCorrelationInfo correlationInfo,
        CancellationToken cancellationToken)
    {
        _logger.LogInformation("Order processed!");
    }
}

💡 For more information on how Arcus handles message handling in message pumps, see our dedicated messaging feature documentation.

Run the solution

When both services are adapted, we can run the solution. Arcus makes sure that the link between the API and Service Bus is visible in Application Insights, without much effort from the consumer. Note that the OrderMessageHandler also receives a correlation model in its ProcessMessageAsync method which allows users to interact further with the passed-along correlation.

Once the applications run, we send a request to the Order API, like:

curl -Method POST `
   -Headers @{'Content-Type'='application/json'}  `
   'http://localhost:5000/api/v1/order' `
   -Body '{ "ProductName": "Fancy desk", "Amount": 3 }'

// StatusCode : 202
// Headers:   : X-Transaction-ID=fb088835-a8f7-4cfe-9434-5ea4892e63b9
//              X-Operation-ID=585ea273-eaa9-44bb-905a-6805bd418566

The real result, though, happens in Application Insights.

The application map (Application Insights > Investigate > Application Map) shows a clear relationship between the two services:

If you copy the X-Transaction-ID from the response (585ea273-eaa9-44bb-905a-6805bd418566) and past it in the transaction search (Application Insights > Investigate > Transaction search), you'll see this relationship in more detail when you select the initial HTTP request to the Order API. You clearly see how the initial request to the Order API is the caller of the dependency towards the Order Worker:

Further reading

• Arcus Service Bus messaging documentation
• Arcus Application Insights Serilog sink documentation
• Messaging blogs
  • Taking messaging to the next level with Arcus Messaging v1.0
• Web API blogs
  • Out-of-the-box Request Tracking, Simplified HTTP Correlation & JWT Authorization in Arcus Web API v1.0
  • Enhanced request tracking in Arcus Web API v1.3
• Observability blogs
  • Announcing Arcus Observability
  • Measure a Variety of Azure Dependencies with Observability v0.2
  • Service Correlation Preparation & .NET 6 Support in Arcus Observability v2.4