.Net Core console applications are by far the most popular applications most developers will routinely develop in order to implement some kind of automation-based task.
I have previously discussed how by using IHost in .net core console applications developers can create asynchronous Linux Daemons and how to use Configuration API in .net core console application .
In this post, I will expand on both of these examples and provide an example of how to use .net core console applications and the HttpClient to develop a Linux daemon to interact with a Web-Based API to extract and process data. a
What is the HttpClient
In .net core the HttpClient Class provides a base class for sending HTTP requests and receiving HTTP responses from a resource identified by a URI.
In previous versions on.net framework developers misused the HttpClient by typically wrapping it in a using block, primarily because the HttpClient implements a Dispose method.
The using statement is a C# language feature for dealing with disposable objects. Once the using block is complete then the disposable object. In fact, many refactoring tools like Jetbrains Resharper including my personal favourite cross-platform .net IDE Jetbrains Rider will explicitly warn if you don't. However, In this case disposing of HttpClient was the wrong thing to do.
It is unfortunate that HttpClientimplements IDisposable which encourages the wrong behaviour.
The dispose method is called and whatever resources are in use are cleaned up. This is a typical pattern in .NET and is used for everything from database connections to stream writers. Any object which has external resources that must be cleaned up using the IDisposable interface.
You will undoubtedly find code similar to this in almost all code bases, where the HttpClient is used to download resources or connect to API's.
using (var client = new HttpClient())
{
var content = await client.GetStringAsync(Url);
return JsonConvert.DeserializeObject<List<Model>>(content);
}
The underlying issue with this code is that while this HttpClient class is disposable, using it with the using statement is not the best choice because even when you dispose HttpClient object, the underlying socket is not immediately released and can cause a serious issue named sockets exhaustion, which I do also discuss a little further in What is Socket Exhaustion
HttpClient is intended to be instantiated once and reused throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads.
Possible approaches to solve the problem are based on the creation of the HttpClient object as singleton or static. However, the HttpClient was not designed to be used as a SIngleton or static object as it doesn't respect DNS changes.
In .net core 2.1 , a new HttpClientFactory was introduced to be used to implement resilient HTTP calls by integrating Polly, which is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner.
What is the HttpClientFactory
The HttpClientFactory is a new mechanism to create HttpClient instances that are properly managed behind the scenes. The HttpClientFactory manages the lifetime of the handlers so that we have a pool of them which can be reused, while also rotating them so that DNS changes are respected.
The expensive part of using HttpClient is actually creating the HttpClientHandler and the connection. By Pooling these within the HttpClientFactory enables more efficient use of the connections. Using the HttpClientFactory to request a HttpClient, creates a new instance each time, meaning no need to worry about mutating its state. The HttpClient may or may not use an existing HttpClientHandler from the pool and therefore use an existing open connection.
How to use the HttpClientFactory
Like most objects in the .net core framework, the HttpClientFactory has been optimised so it can be made use of by Dependency Injection. I will illustrate a typical approach take when making use of the HttpClientFactory, in this approach I'll be developing a Linux Daemon which will be used to contact a Rest API end-point too extract and process data.
Add the following reference to your project
dotnet add package Microsoft.Extensions.Http --version 2.2.0
It is worth checking out using IHost in .net core console applications and expand further on typical usage scenarios. We will also see an example how to use Configuration API in .net core console application .
In this scenario, we generate a simple Console Application then modify the code to make use of the IHost to enable dependency injection, logging and configuration. You will need to add a number of packages, details of which can be found in the articles referenced.
I have also added the requisite json files:
- appsettings.json
- appsettings.Development.json
- appsettings.Production.json
- hostsettings.json
class Program
{
private const string EnvironmentVariablePrefix = "ADZUNA_FEED_";
public static async Task Main(string[] args)
{
var host = new HostBuilder()
.ConfigureHostConfiguration(hostConfig =>
{
hostConfig.SetBasePath(Directory.GetCurrentDirectory());
hostConfig.AddJsonFile("hostsettings.json", true);
hostConfig.AddEnvironmentVariables(EnvironmentVariablePrefix);
hostConfig.AddCommandLine(args);
})
.ConfigureAppConfiguration((hostingContext, config) =>
{
config.SetBasePath(hostingContext.HostingEnvironment.ContentRootPath);
config.AddJsonFile("appsettings.json", true, true);
config.AddJsonFile($"appsettings.{hostingContext.HostingEnvironment.EnvironmentName}.json");
})
.ConfigureServices((hostContext, services) =>
{
services.AddLogging();
})
.Build();
await host.RunAsync();
}
Information
In .net core 3.0 you will be able to generate projects which will have all this boiler plated code available as the Worker Service Template
Now that the basic project template has been configured we're all good to go. As you might of deduced from the code, the service we're going to develop is going to connect to the Adzuna API service to extract data. However, I will attempt to make the concepts as generic as possible so you can follow along and connect to any Web API service of choice.
The first thing we need to do before we wire up our HttpClientFactory is to edit our appsettings.[Environment].json files so the contain the configuration data we'll need
{
"adzuna": {
"key" : "",
"secret": "",
}
}
You will notice we are going store our configuration details that we will need these are credentials you receive when you register to use the Adzuna API.
The Adzuna API enables developers to get the very latest ads, job, property and car ads, and data with Adzuna's API. Adzuna's up-to-the-minute employment data to power your own website, reporting and data visualisations.
In particular scenario I also wanted to create a Configuration class to contain the Base Urls that I will be using in my application. So I created another POCO class as follows.
[JsonObject("baseUrls")]
public class BaseUrls
{
[JsonProperty("adzuna")]
public string Adzuna { get; set; }
}
We add the details to our environment specific App Settings file as follows:
{
"adzuna": {
"key" : "",
"secret": ""
},
"baseUrls": {
"adzuna" : "http://api.adzuna.com/v1/api/"
}
}
Information
Always ensure you include the trailing / when including a base address because this is the only permutation that seems to work.
With the details we can now edit our Program.cs to enable the dependency injection and to retrieve the configuration details from our App Settings file
using System;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using AdzunaFeed.Config;
using AdzunaFeed.Interfaces;
using AdzunaFeed.Services;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
namespace AdzunaFeed
{
class Program
{
private const string EnvironmentVariablePrefix = "ADZUNA_FEED_";
public static async Task Main(string[] args)
{
var host = new HostBuilder()
.ConfigureHostConfiguration(hostConfig =>
{
hostConfig.SetBasePath(Directory.GetCurrentDirectory());
hostConfig.AddJsonFile("hostsettings.json", true);
hostConfig.AddEnvironmentVariables(EnvironmentVariablePrefix);
hostConfig.AddCommandLine(args);
})
.ConfigureAppConfiguration((hostingContext, config) =>
{
config.SetBasePath(hostingContext.HostingEnvironment.ContentRootPath);
config.AddJsonFile("appsettings.json", true, true);
config.AddJsonFile($"appsettings.{hostingContext.HostingEnvironment.EnvironmentName}.json");
})
.ConfigureServices((hostingContext, services) =>
{
var baseUrls = new BaseUrls();
hostingContext.Configuration.GetSection("baseUrls").Bind(baseUrls);
services.Configure<Adzuna>(options =>
hostingContext.Configuration.GetSection("adzuna").Bind(options));
services.AddHttpClient<IDownloadService, DownloadService>(client =>
{
client.BaseAddress = new Uri(baseUrls.Adzuna);
client.DefaultRequestHeaders
.Accept
.Add(new MediaTypeWithQualityHeaderValue("application/json"));
});
services.AddLogging();
}
)
.Build();
await host.RunAsync();
}
}
You'll notice on line 42 we configure the HttpClient making use of the AddHttpClient method. If you are unable to access this method you may need to add the dependency Microsoft.Extensions.Http to your project.
dotnet add package Microsoft.Extensions.Http --version 2.2.0
We're now ready to start writing code for our service,add a new class we'll call JobProcessingService which will implement the IHostedService interface which defines two methods for objects that are managed by the host.
The daemon we're going to be be developing here, will potentially be running several times an hour and will be running several jobs at least, so it is important for us to build in some kind of cancellation process using tokens. For this purpose of this post, I will implement a "Good Enough"approach but this is by no means a complete approach and it simply has the basics, but provides at least some understanding of what required.
using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;
using Threenine.Job.Feed.Entity;
using Threenine.Job.Feed.Interfaces;
namespace Threenine.Job.Feed.Services
{
public class JobProcessingService : IHostedService, IDisposable
{
private CancellationTokenSource _cts;
private Task _currentTask;
private readonly IDownloadService<AdzunaFeed> _service;
public JobProcessingService(IDownloadService<AdzunaFeed> service)
{
_service = service;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
// In later version of this task we will get a list of
//skills which we will be make subsequent calls
//however for now I just provided an example
_currentTask = _service.GetFeed("C#",_cts.Token);
return _currentTask.IsCompleted ? _currentTask : Task.CompletedTask;
}
public async Task StopAsync(CancellationToken cancellationToken)
{
if (_currentTask == null)
{
return;
}
try
{
_cts.Cancel();
}
finally
{
await Task.WhenAny(_currentTask, Task.Delay(Timeout.Infinite, cancellationToken));
}
}
public void Dispose()
{
_cts.Cancel();
}
}
}
Now I background task is complete we'll move onto the actual implementation of our Service which implements the HttpClient, again the implementation here is just enough to provide a flavour of the implementation for demo purpose, but you will see enough of how to implement the HttpClient and make use of it, in a real-world scenario.
The major drawback with the code in this implementation is that we make a call retrieve data and deserialize it but we don't actually do anything with it! But it this just provides and example of how to use the HttpClient and the HttpClientFactory
using System;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Options;
using Threenine.Job.Feed.Config;
using Threenine.Job.Feed.Entity;
using Threenine.Job.Feed.Interfaces;
namespace Threenine.Job.Feed.Services
{
public class AdzunaDownloadService : IDownloadService<AdzunaFeed>
{
private readonly HttpClient _client;
private readonly AdzunaCredentials _creds;
public AdzunaDownloadService(HttpClient client, IOptions<AdzunaCredentials> creds)
{
_client = client;
_creds = creds.Value;
}
public async Task<AdzunaFeed> GetFeed(string criteria, CancellationToken token)
{
var requestUrl = $"jobs/gb/search?app_id={_creds.Key}&app_key={_creds.Secret}&what={criteria}";
var response = await _client.GetAsync(requestUrl, token);
if (!response.IsSuccessStatusCode) throw new ApplicationException();
var stream = await response.Content.ReadAsStreamAsync();
return stream.DeserializeFromJson<AdzunaFeed>();
}
}
}
You may have noticed I am using an extension method on the Stream class to DeserializeFromJsonI thought I would include the extension method for clarity. The code for this method is in a new class I called StreamExtensions.cs
using System;
using System.IO;
using Newtonsoft.Json;
namespace Threenine.Job.Feed
{
public static class StreamExtensions
{
public static T DeserializeFromJson<T>(this Stream stream)
{
if (stream == null) throw new ArgumentNullException(nameof(stream));
if (!stream.CanRead) throw new NotSupportedException("can't read from stream");
using (var sread = new StreamReader(stream))
using (var jsonText = new JsonTextReader(sread))
{
var serialiser = new JsonSerializer();
return serialiser.Deserialize<T>(jsonText);
}
}
}
}
The code for these sections is almost complete, we now just need to wire up our application startup.
We'll now edit the Program.cs to look like the following
using System;
using System.IO;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Threenine.Job.Feed.Config;
using Threenine.Job.Feed.Entity;
using Threenine.Job.Feed.Interfaces;
using Threenine.Job.Feed.Services;
namespace Threenine.Job.Feed
{
class Program
{
private const string EnvironmentVariablePrefix = "ADZUNA_FEED_";
public static async Task Main(string[] args)
{
var host = new HostBuilder()
.ConfigureHostConfiguration(hostConfig =>
{
hostConfig.SetBasePath(Directory.GetCurrentDirectory());
hostConfig.AddJsonFile("hostsettings.json", true);
hostConfig.AddEnvironmentVariables(EnvironmentVariablePrefix);
hostConfig.AddCommandLine(args);
})
.ConfigureAppConfiguration((hostingContext, config) =>
{
config.SetBasePath(hostingContext.HostingEnvironment.ContentRootPath);
config.AddJsonFile("appsettings.json", true, true);
config.AddJsonFile($"appsettings.{hostingContext.HostingEnvironment.EnvironmentName}.json");
})
.ConfigureServices((hostingContext, services) =>
{
var baseUrls = new BaseUrls();
hostingContext.Configuration.GetSection("baseUrls").Bind(baseUrls);
services.Configure<AdzunaCredentials>(options =>
hostingContext.Configuration.GetSection("adzuna").Bind(options));
services.AddHttpClient<IDownloadService<AdzunaFeed>, AdzunaDownloadService>(client =>
{
client.BaseAddress = new Uri(baseUrls.Adzuna);
client.DefaultRequestHeaders
.Accept
.Add(new MediaTypeWithQualityHeaderValue("application/json"));
});
services.AddHostedService<JobProcessingService>();
services.AddLogging();
}
)
.Build();
await host.RunAsync();
}
}
}
Summary
We have seen an example of how to make use HttpClientFactory within a .net console application to create a HttpClient instance to be used within a HostedService application.
- What is this Directory.Packages.props file all about? - January 25, 2024
- How to add Tailwind CSS to Blazor website - November 20, 2023
- How to deploy a Blazor site to Netlify - November 17, 2023