Getting started with .NET Core and the Serverless Framework

The serverless framework is a cloud provider agnostic toolkit designed to aid operations around building, managing and deploying serverless components for either full-blown serverless architectures or disparate functions as a service (FaaS).

I have previously posted about using Build a Twitter bot with AWS lambda serverless framework and node.js

In this post, we'll explore how to adopt a Serverless Computing approach using the Serverless Framework with .net core and provide an example of how to develop AWS Lambda functions using the Serverless framework and .net core.

The key objective of the Serverless Framework is to provide developers with an interface that abstracts away Cloud Vendor specific APIs and configuration whilst simultaneously providing additional tooling to enable the testing and deployment functions with ease, perfect for rapid feedback or being able to integrate into your CI/CD pipelines.

This article is based on using the Serverless Framework Open Source and .net core on Ubuntu, but the process should be similar using whichever operating system you use.

Install the Serverless Framework

The Serverless Framework is a node based package, therefore required node to be installed on your system.

I typically use and have Node Version Manager installed on my Ubuntu development desktops which enables me to manage install multiple different versions of the node framework on my development machine.

Installing Serverless Framework is really easy and is well documented on the but I will reproduce it here for reference.

npm install -g serverless

Once installed you are good to go. I do recommend you open an account on the Serverless Framework website, but it is essential to get started to play with the framework. To create an account you can use the terminal shortcut, which will open a browser and ask to create an account or login

Serverless Enterprise

Although it is not absolutely necessary to have a Serverless Enterprise account, you can still develop and deploy Serverless Applications using the Serverless Framework without one, you may find the Serverless Enterprise to be quite a handy platform to help you on your company on the path to adopting Serverless Architecture.

Gain and share visibility across all serverless applications, the cloud resources they contain, and their deployments, in a single dashboard.
Serverless Framework

Information

You will need the AWS CLI installed.
Instructions to install AWS CLI on Ubuntu Desktop

Serverless Framework enables you to seemlessly deploy applications to AWS. However, this does require AWS Credentials to do so. There are a few ways to supply your credentials, my preferred approach it to use the AWS CLI to configure my details.

In order to use Serverless on AWS you will need an AWS Account. If you don't already have one, you can sign up for a free trial that includes 1 million free Lambda requests per month

Advice

For detailed information of the aws configure check out the Configuring the AWS CLI

Create Lambda Project

The serverless framework has a number of templates available to help create lambda projects quickly and easily.

Information

You can use either, serverless or the shorthand, sls to execute Serverless Framework statements. I prefer to use sls and will continue to do so throughout this article

To see the complete list yo can use:

sls create --help

Which will return a list similar too:

"aws-nodejs",
"aws-python",
"aws-python3",
"aws-groovy-gradle",
"aws-java-maven",
"aws-java-gradle",
"aws-scala-sbt",
"aws-csharp",
"aws-fsharp",
"azure-nodejs",
"openwhisk-nodejs",
"openwhisk-python",
"openwhisk-swift",
"google-nodejs"

To create a lambda project and instruct Serverless to use the C# template we provide a --template switch

sls create -t aws-csharp -n hello-world -p hello-world

This will generate a project for us comprising a number of files to a directory created hello-world.

.gitignore 
Handler.cs 
aws-csharp.csproj 
build.cmd build.sh 
serverless.yml

We can open the project in whichever IDE or Text Editor of your choice to view the files , in my case I use Jetbrains Rider - A fast & powerful, cross-platform .NET IDE

Lets go through some of the important files here, in order to get a better understanding of the project before we start coding.

serverless.yml

The serverless.yml is basically the schema which defines the configuration of your serverless application and how it will interact with components included in your wider serverless architecture.

When you deploy your serverless application the framework generates a Cloud Formation template with AWS - in this case - will use to provision the appropriate infrastructure.

Serverless provides a comprehensive Reference of all the available properties to use when configuring your application with AWS.

It's well worth taking the time to read through this, in order to gain a broader understanding of what is available.

build.sh

The build.sh is important for when you want to build your serverless application in preparation for deployment.

The build.cmd is the build file for windows based developers.

Deploy and Test

Your lambda can be deployed and executed at this stage. If you would like to deploy and test your lambda at this stage Serverless Framework have a great quick start guide to take you through this process.

Serverless Lambda in Action

I wanted to create a bit more of meaningful example to illustrate why developers would use Lambdas with a real world example and just how the Serverless Framework simplifies this process.

In the example I am going to build a Lambda which is notified by S3 storage when a new file is created. The Lambda then parses the file to a JSON format then dumps the JSON data to a DynamoDB database.

I will focus on the Lambda specifics of this process and won't be discussing the other code involved in this process in depth.

The problem the Lambda I will be developing will solve, primarily focuses around the processing of Word Documents, primarily Docx.

service: rea-docx-parser
app: #use your application bucket
tenant: #use your tenant name
custom:
  bucket: rea-cv-upload
provider:
  name: aws
  runtime: dotnetcore2.1
  stage: dev
  region: eu-west-2

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - s3:*
      Resource: "*"
package:
  artifact: bin/release/netcoreapp2.1/deploy-package.zip

functions:
  process:
    handler: CsharpHandlers::AwsDotnetCsharp.Handler::Process
    events:
      - s3:
          bucket: ${self:custom.bucket}
          event: s3:ObjectCreated:*
          rules:
            - suffix: .docx

The lambda will be configured to monitor a S3 bucket and if a new file is created in the bucket it will process the file.

Processing the file includes parsing the file extracting keywords, phrases and other bits of information contained in the file. Creating a JSON representation of the data contained and saving it to the DynamoDB table for further processing.

We have included libraries in our lambda which will assist in the actual parsing of documents and creating the relevant JSON objects to be stored in dynamoDb.

using Amazon.Lambda.Core;
using System;

[assembly:LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace AwsDotnetCsharp
{
    public class Handler
    {
       public Response Hello(Request request)
       {
           return new Response("Go Serverless v1.0! Your function executed successfully!", request);
       }
    }

    public class Response
    {
      public string Message {get; set;}
      public Request Request {get; set;}

      public Response(string message, Request request){
        Message = message;
        Request = request;
      }
    }

    public class Request
    {
      public string Key1 {get; set;}
      public string Key2 {get; set;}
      public string Key3 {get; set;}

      public Request(string key1, string key2, string key3){
        Key1 = key1;
        Key2 = key2;
        Key3 = key3;
      }
    }
}

All the code for a first test is now complete. We can now build our application and deploy it.

In a terminal window we need to ensure we are in the directory where our build.sh then we can simply execute our build file

sudo ./build.sh

Once the application has completed building we can then deploy it. Using the Serverless Framework command

sls deploy -v

We added the -v to display Verbose information regarding the deployment operation. Sometimes this may be necessary to display more information to help troubleshoot deployment issues.

We're not expecting any errors but its a good opportunity to get acquainted with the output and see what information is available.

After our deployment is complete lets first check our service logs, to ensure ourLambada has not executed. Using a terminal window in the directory of our lambda code.

sls logs -f process

We have used the logging functionality of the Serverless Framework and provided the -f which is the shorthand for --function then provided the function name we wanted to query.

This is also the function name we defined in our serverless.yml. At this stage, you should see a blank return.

In order to test, at this stage lets manually load a document to our S3 Storage. Any .docx will suffice at this stage.

Once the file is uploaded we can now re-check our logs to see if our service executed

sls logs -f process

This time we should see something returned similar too

START RequestId: 987dacc8-f3c5-11e8-87ac-a78036c929c9 Version: $LATEST END RequestId: 987dacc8-f3c5-11e8-87ac-a78036c929c9 REPORT RequestId: 987dacc8-f3c5-11e8-87ac-a78036c929c9  Duration: 419.31 ms     Billed Duration: 500 ms         Memory Size: 1024 MB    Max Memory Used: 54 MB

This essentially tells us our function has executed. Its a great short cut to have when you're testing your function. However, if you would like to see more detail logs or logs for previous versions of your lambda, you can view the logs generated in your AWS CloudWatch, the Serverless Framework has created this all for you!

Navigating to your CloudWatch in your AWS Services console and Selecting Logs for for Lambda service you will be able to see the detailed view.

Add ReadMe

Although not essential lets add a README.md to our code. This will help us to document our code and leave important information for other developers.

At this stage we'll simply add some basic information, but we will keep this file up to date during our development process.

### REA CV Parser Docx

Lambda to monitor S3 Bucket and detect when new .docx files are added and process them.

We can now rebuild and deploy our Lambda. Once our deployment is finished lets go to the Serverless Platform and go an inspect the information that is now Available.

If you have a Serverless Framework Enterprise Account your ReadMe will be displayed as follows along with a break down of all the services and features and usage statistics of your Lambda.

Configure Lambda Service

We will start by introducing some of the basic features of the Serverless Framework that as a developer you will more than likely use during your development process.

I have to admit, that although I do like the Serverless Framework, I do find the documention to quite light in general and even more so when it comes to C#, there are very few if any samples available.

First, we will configure our Lambda using the serverless.yml.

service: rea-docx-parser # NOTE: update this with your service name
app: docxparser
tenant: garywoodfine

custom:
  bucket: rea-cv-upload-dev
  candidatesTableName: Candidates
  documentsTableName: Documents

provider:
  name: aws
  runtime: dotnetcore2.1
  environment:
    CANDIDATE_DYNAMODB_TABLE: ${self:custom.candidatesTableName}
    DOCUMENT_DYNAMODB_TABLE: ${self:custom.documentsTableName}
  stage: dev
  region: eu-west-2
  iamRoleStatements:
  - Effect: "Allow"
    Action:
    - s3:*
    - dynamodb:*
    Resource: "*"
package:
  artifact: bin/release/netcoreapp2.1/deploy-package.zip

functions:
  process:
    handler: CsharpHandlers::AwsDotnetCsharp.Handler::Process
    events:
    - s3:
        bucket: ${self:custom.bucket}
        event: s3:ObjectCreated:*
        rules:
        - suffix: .docx
          
resources:
   Resources:
     CandidateDynamoDbTable:
       Type: 'AWS::DynamoDB::Table'
       DeletionPolicy: Retain
       Properties:
         AttributeDefinitions:
           - AttributeName: id
             AttributeType: S
         KeySchema:
           - AttributeName: id
             KeyType: HASH
         ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
         TableName: ${self:provider.environment.CANDIDATE_DYNAMODB_TABLE}
     DocumentDynamoDbTable:
         Type: 'AWS::DynamoDB::Table'
         DeletionPolicy: Retain
         Properties:
           AttributeDefinitions:
             - AttributeName: id
               AttributeType: S
           KeySchema:
             - AttributeName: id
               KeyType: HASH
           ProvisionedThroughput:
             ReadCapacityUnits: 1
             WriteCapacityUnits: 1
           TableName: ${self:provider.environment.DOCUMENT_DYNAMODB_TABLE}

There is a lot going on in this code snippet, so it is worth explaining. The service tag is used to define a name for the service.

The custom is just a handy area to declare any variables that you would like to use later.

In my case I have used it to define a name for my S3 Bucket, which I want to name rea-cv-upload.

The provider is the area which you use deine attributes of your application, with things like the runtime platfrom, region and whether the application is dev or production.

iamRoleStatements is where you define the permissions for your lambda, check out IAM in serverless documentation and the What is IAM? for more details.

The functions is where the important bits for our application in its present form. I have defined a process function and have delegated it's handler to a method in my Handler.cs which is also name Process

The events is where I define that my Lambda will watch my S3 bucket , the name of which I get from my Custom Variable defined earlier, if any files are created with a .docx extension then my lambda will leap into action.

We can now go edit the Handler.cs slightly in order to conduct a test and highlight another feature of the Serverless Framework. Our updated Handler now looks like this:

using System;
using System.Threading.Tasks;
using Amazon;
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon.Lambda.Core;
using Amazon.Lambda.S3Events;
using Amazon.Lambda.Serialization.Json;
using Amazon.S3;
using Threenine.DocumentParser.Factory;
using Threenine.DocumentParser.Input.Docx;

[assembly: LambdaSerializer(typeof(JsonSerializer))]

namespace AwsDotnetCsharp
{
    public class Handler
    {
        private readonly AmazonDynamoDBClient _amazonDynamoDbClient;
        private readonly AmazonS3Client _amazonS3Client;

        private readonly string CandidateTableName = "Candidates";
        private readonly string FullDocumentTableName = "Documents";

        public Handler()
        {
            _amazonS3Client = new AmazonS3Client(RegionEndpoint.EUWest2);
            _amazonDynamoDbClient = new AmazonDynamoDBClient();
        }

        public async Task Process(S3Event s3event, ILambdaContext _context)
        {
            foreach (var record in s3event.Records)
                using (var objectResponse =
                    await _amazonS3Client.GetObjectAsync(record.S3.Bucket.Name, record.S3.Object.Key))
                using (var responseStream = objectResponse.ResponseStream)
                {
                    var reader = new DocxStreamReader();
                    var content = reader.Read(responseStream);
                 
                    var fullDocumentParserFactory = new FullDocumentParserFactory(record.S3.Object.Key);

                    var fullDocument = fullDocumentParserFactory.Parse(content);

                    var candidateParserFactory = new CandidateParserFactory(record.S3.Object.Key);
                    var candidate = candidateParserFactory.Parse(content);

                    var candidateTable = Table.LoadTable(_amazonDynamoDbClient, CandidateTableName);
                    var documentsTable = Table.LoadTable(_amazonDynamoDbClient, FullDocumentTableName);

                    var documents = Document.FromJson(fullDocument);
                    var candidates = Document.FromJson(candidate);

                    await documentsTable.PutItemAsync(documents);
                    await candidateTable.PutItemAsync(candidates);
                }
        }
    }
}

Advice

If you don't like the idea of instantiating objects within your Lambda Functions check out how to implement dependency injection in AWS Lambdas

You can now see the added value that the Serverless platform provides in how it also helps us to document our service. At a glance on the left hand side, we're able to see what resources our lambda uses, how many functions are defined and the subscriptions to other services it has.

The README file we created will help to relay any other important information we need to share with other developers on our team.

CloudFormation

You have probably noticed that Serverless and specifically the serverless.yml takes care of all your AWS configuration details. So when creating Serverless applications with the Framework there is no need to go an create and manage all your dependent resources individually. This can all be done by editing the serverless.yml

For instance, if I wanted to use DynamoDB functionality within my serverless application. I could quite simply edit my serverless.yml and add the additional resources as follows.

service: rea-docx-parser # NOTE: update this with your service name
app: docxparser
tenant: garywoodfine

custom:
  bucket: rea-cv-upload-dev

provider:
  name: aws
  runtime: dotnetcore2.1
  environment:
    DYNAMODB_TABLE: "Candidates"
  stage: dev
  region: eu-west-2

  iamRoleStatements:
  - Effect: "Allow"
    Action:
    - s3:*
    - dynamodb:*
    Resource: "*"
package:
  artifact: bin/release/netcoreapp2.1/deploy-package.zip

functions:
  process:
    handler: CsharpHandlers::AwsDotnetCsharp.Handler::Process
    events:
    - s3:
        bucket: ${self:custom.bucket}
        event: s3:ObjectCreated:*
        rules:
        - suffix: .docx
          
resources:
   Resources:
     CandidateDynamoDbTable:
       Type: 'AWS::DynamoDB::Table'
       DeletionPolicy: Retain
       Properties:
         AttributeDefinitions:
           - AttributeName: id
             AttributeType: S
         KeySchema:
           - AttributeName: id
             KeyType: HASH
         ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
         TableName: ${self:provider.environment.DYNAMODB_TABLE}

I could then simply deploy my update and DynamoDB table will be available. You can also inspect your CloudFormation details in your AWS Console

The core key benefit of the the Serverless Framework is that it helps you to configure and manage your serverless stack. Enabling you to focus on developing the functionality you need without having to worry about infrastucture details.

When you deploy your Serverless framework application, it handles everything for you! That is true serverless mindset at work!

Warning

Don't try to manage your Resources created using your Serverless Stack in your AWS console as it can get out of sync with your CloudFormation stack and you will experience deployment errors.
If you need to remove resources or change some configuration details, using Serverless.yml.
If you want to completely remove your serverless application simply use the sls remove

Using AWS SDK

The thing that stumped me most when I started using the Serverless Framework, is that out the box, it didn't come with support AWS Events etc. Although I could easily manage and configure my AWS serverless stack, the library didn't seem to come shipped with the AWS SDK built in. After thinking about this for a while I concluded that this actually makes sense. In all probability most of the functionality in your Lambda won't be focused on interacting with AWS components, and it would probably be in-efficient to ship the library with every single Lambda. So for good reason, it is up to the developer to include this should they need it.

For developers including libraries, it is just a case in importing a package using nuget package manager. In my particular case, I wanted to listen to events when documents are added to S3, my lambda is already configured for that, and whenever a Word Document is added to S3 my lambda is triggered. However, currently my lambda is not able to get any details about the document etc.

So I need to add a reference to the Amazon.Lambda.S3Events and while I'm at it, I need to add a reference to the AWSSDK.DynamoDBv2 library so I can save whatever details I need to my DynamoDB Bucket.

It is basically a simple case of using nuget to add these packages. I'll do so by either using the command line or in my particular case I using Jetbrains Rider - A fast & powerful, cross-platform .NET IDE great built in Nuget capability.

 dotnet add package AWSSDK.DynamoDBv2 
 dotnet add package Amazon.Lambda.S3Events

I can implement an initial version of my Lambda code, using some additional libraries I have developed to handle the parsing of documents.

using Amazon.Lambda.Core;
using System.IO;
using System.Threading.Tasks;
using Amazon;
using Amazon.DynamoDBv2;
using Amazon.Lambda.S3Events;
using Amazon.S3;
using Threenine.DocumentParser.Input.Docx;
using Amazon.DynamoDBv2.DocumentModel;
using Threenine.DocumentParser.Factory;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace AwsDotnetCsharp
{
    public class Handlerurg
    {
        private readonly AmazonS3Client _amazonS3Client;
        private readonly AmazonDynamoDBClient _amazonDynamoDbClient;
        private readonly string TableName = "Candidates";

        public Handler()
        {
            _amazonS3Client = new AmazonS3Client(RegionEndpoint.EUWest2);
            _amazonDynamoDbClient = new AmazonDynamoDBClient();
        }

        public async Task Process(S3Event s3event, ILambdaContext _context)
        {
            foreach (var record in s3event.Records)
            {
                using (var objectResponse =
                    await _amazonS3Client.GetObjectAsync(record.S3.Bucket.Name, record.S3.Object.Key))
                using (Stream responseStream = objectResponse.ResponseStream)

                {
                    var reader = new DocxStreamReader();
                    var parser = new ProfileParserFactory(record.S3.Object.Key);
                    var parsedDocument = parser.Parse(reader.Read(responseStream));

                    var table = Table.LoadTable(_amazonDynamoDbClient, TableName);
                    var item = Document.FromJson(parsedDocument);
                    await table.PutItemAsync(item);
                }
            }
        }
    }
}

Information

We get the details of the S3 Event and process it, then save the data to the DynamoDB instance we created. We have kept the implementation detail of our service as simple as possible.

Conclusion

Hopefully this post has covered some important aspects of creating Serverless Application using the Serverless Framework and .net core. The Serverless Framework is a great framework to increase developer productivity.

About
Latest Posts

Gary Woodfine

Technical Director at threenine.co.uk

Gary is Technical Director at threenine.co.uk, an independent software vendor specialising in IoT, Field Service and associated managed services,enabling customers to be efficient, productive, secure and scale-able.