.Net Core console applications are by far the most popular applications most developers will routinely develop in order to implement some kind of automation-based task.
I have previously discussed how by using IHost in .net core console applications developers can create asynchronous Linux Daemons and how to use Configuration API in .net core console application .
In this post, I will expand on both of these examples and provide an example of how to use .net core console applications and the HttpClient to develop a Linux daemon to interact with a Web-Based API to extract and process data.
What is the HttpClient
In .net core the HttpClient Class provides a base class for sending HTTP requests and receiving HTTP responses from a resource identified by a URI.
In previous versions on.net framework developers misused the HttpClient by typically wrapping it in a using block, primarily because the HttpClient implements a Dispose method.
using statement is a C# language feature for dealing with disposable objects. Once the
using block is complete then the disposable object. In fact, many refactoring tools like Jetbrains Resharper including my personal favourite cross-platform .net IDE Jetbrains Rider will explicitly warn if you don’t. However, In this case disposing of
HttpClient was the wrong thing to do.
It is unfortunate that
IDisposable which encourages the wrong behaviour.
dispose method is called and whatever resources are in use are cleaned up. This is a typical pattern in .NET and is used for everything from database connections to stream writers. Any object which has external resources that must be cleaned up using the
You will undoubtedly find code similar to this in almost all code bases, where the HttpClient is used to download resources or connect to API’s.
The underlying issue with this code is that while this HttpClient class is disposable, using it with the
using statement is not the best choice because even when you dispose
HttpClient object, the underlying socket is not immediately released and can cause a serious issue named ‘sockets exhaustion’.
HttpClient is intended to be instantiated once and reused throughout the life of an application. Instantiating an
HttpClient class for every request will exhaust the number of sockets available under heavy loads.
Possible approaches to solve the problem are based on the creation of the
HttpClient object as singleton or static. However, the HttpClient was not designed to be used as a SIngleton or static object as it doesn’t respect DNS changes.
In .net core 2.1 , a new
HttpClientFactory was introduced to be used to implement resilient HTTP calls by integrating Polly, which is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner.
What is the HttpClientFactory
HttpClientFactory is a new mechanism to create HttpClient instances that are properly managed behind the scenes. The HttpClientFactory manages the lifetime of the handlers so that we have a pool of them which can be reused, while also rotating them so that DNS changes are respected.
The expensive part of using HttpClient is actually creating the HttpClientHandler and the connection. By Pooling these within the HttpClientFactory enables more efficient use of the connections. Using the HttpClientFactory to request a HttpClient, creates a new instance each time, meaning no need to worry about mutating its state. The HttpClient may or may not use an existing HttpClientHandler from the pool and therefore use an existing open connection.
How to use the HttpClientFactory
Like most objects in the .net core framework, the HttpClientFactory has been optimised so it can be made use of by Dependency Injection. I will illustrate a typical approach take when making use of the HttpClientFactory, in this approach I’ll be developing a Linux Daemon which will be used to contact a Rest API end-point too extract and process data.
Add the following reference to your project
It is worth checking out using IHost in .net core console applications and expand further on typical usage scenarios. We will also see an example how to use Configuration API in .net core console application .
In this scenario, we generate a simple Console Application then modify the code to make use of the IHost to enable dependency injection, logging and configuration. You will need to add a number of packages, details of which can be found in the articles referenced.
I have also added the requisite json files:
In .net core 3.0 you will be able to generate projects which will have all this boiler plated code available as the Worker Service Template
Now that the basic project template has been configured we’re all good to go. As you might of deduced from the code, the service we’re going to develop is going to connect to the Adzuna API service to extract data. However, I will attempt to make the concepts as generic as possible so you can follow along and connect to any Web API service of choice.
The first thing we need to do before we wire up our HttpClientFactory is to edit our
appsettings.[Environment].json files so the contain the configuration data we’ll need
You will notice we are going store our configuration details that we will need these are credentials you receive when you register to use the Adzuna API.
The Adzuna API enables developers to get the very latest ads, job, property and car ads, and data with Adzuna’s API. Adzuna’s up-to-the-minute employment data to power your own website, reporting and data visualisations.
In particular scenario I also wanted to create a Configuration class to contain the Base Urls that I will be using in my application. So I created another POCO class as follows.
We add the details to our environment specific App Settings file as follows:
Always ensure you include the trailing / when including a base address because this is the only permutation that seems to work.
With the details we can now edit our
Program.cs to enable the dependency injection and to retrieve the configuration details from our App Settings file
You’ll notice on line 42 we configure the
HttpClient making use of the
AddHttpClient method. If you are unable to access this method you may need to add the dependency
Microsoft.Extensions.Http to your project.
We’re now ready to start writing code for our service,add a new class we’ll call
JobProcessingService which will implement the
IHostedService interface which defines two methods for objects that are managed by the host.
The daemon we’re going to be be developing here, will potentially be running several times an hour and will be running several jobs at least, so it is important for us to build in some kind of cancellation process using tokens. For this purpose of this post, I will implement a “Good Enough”approach but this is by no means a complete approach and it simply has the basics, but provides at least some understanding of what required.
Now I background task is complete we’ll move onto the actual implementation of our Service which implements the
HttpClient, again the implementation here is just enough to provide a flavour of the implementation for demo purpose, but you will see enough of how to implement the HttpClient and make use of it, in a real-world scenario.
The major drawback with the code in this implementation is that we make a call retrieve data and deserialize it but we don’t actually do anything with it! But it this just provides and example of how to use the
HttpClient and the
You may have noticed I am using an extension method on the Stream class to
DeserializeFromJsonI thought I would include the extension method for clarity. The code for this method is in a new class I called
The code for these sections is almost complete, we now just need to wire up our application startup.
We’ll now edit the Program.cs to look like the following
We have seen an example of how to make use HttpClientFactory within a .net console application to create a HttpClient instance to be used within a HostedService application.