Software Development is a complex process and one where aspects of the solutions that have been implemented continually evolve and the only constant in software development is change.
In many cases, code may continually be tweaked, modified, refactored or even new requirements may need implementation. Developers will integrate new changes into existing applications and get them deployed to production regularly.
In most cases they will want to avoid as much as possible from doing what is known as a Big Bang integration. A Big Bang integration is when all components or modules are integrated simultaneously, after which everything is deployed as a whole.
one of main motivations is to avoid big bang integrations, where multiple people and/or multiple teams build their part of the project and it is all fused together before release. The benefits of having an automated continuous build are huge, since problems become visible early. The build server polls the source control server and produces a build and runs the appropriate tests when new code is committed to central source code repository.
What is Continuous Integration
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
The CI process is generally comprised of automatic tools that assert the new codes correctness before integration. A source code version control system is the crux of the CI process. The version control system is also supplemented with other checks like automated code quality tests, syntax style review tools, and more.
CI is a valuable and well-established practice in modern, high performance software engineering organizations helping to scale up headcount and delivery output of engineering teams .
Introducing CI enables software developers to work independently on features in parallel. Upon completion they are ready to merge these features into the end product.
CI is generally used alongside an agile software development workflow. An organisation will compile list of tasks that comprise a product road-map. These tasks are then distributed amongst software engineering team members for delivery. Using CI enables these software development tasks to be developed independently and in parallel amongst the assigned developers.
Once one of theses tasks is complete, a developer will introduce that new work to the CI system to be integrated with the rest of the project. The CI process is a mechanism to find and eliminate problems from code changes as early as possible.
In The Pragmatic Programmer: your journey to mastery, 20th Anniversary Edition (2nd Edition) there is a discussion of what leads to what is known as Software Entropy or as us developers tend to call Software rot, and how developers and teams should ensure that every developer should be responsible and take proactive steps to ensure that software rot does not start to sneak into their projects.
Quality is a team issue. The most diligent developer placed on a team that just doesn’t care will find it difficult to maintain the enthusiasm needed to fix niggling problems. The problem is further exacerbated if the team actively discourages the developer from spending time on these fixes.
The Pragmatic Programmer: your journey to mastery
The Pragmatic Programmer
your journey to mastery
20th Anniversary Edition (2nd Edition)
Team members should integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build including tests to detect integration errors as quickly as possible. I personally prefer to make use of TeamCity by Jetbrains a free and powerful enterprise lever Continuous Integration and Continuous Deployment server.
Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.
The key for developers to be able to get the most out of CI without it causing to much friction is to be disciplined about their Check In Dance.
Source code Repository design
In Git Source Code Repository Design Considerations I discussed some of the common Source Code repository practices and what organisations typically consider when structuring their source code repositories.
Git Workflow Considerations
Gitflow
Gitflow is a legacy Git workflow that was originally a disruptive and novel strategy for managing Git branches. Gitflow has fallen in popularity in favor of trunk-based workflows, which are now considered best practices for modern continuous software development and DevOps practices. Gitflow also can be challenging to use with CI/CD.
If you are using a typical Git Flow - a popular git branching and merging strategy, then ideally any commit regardless of whether it occurred on feature/* , hotfix/* , bug/* or release/* should trigger a CI build.
Github flow
GitHub Flow is another popular lightweight, branch-based workflow and I have come across several other different implementations in the Enterprise software development arena. Where each consultancy or Company has pioneered their own approach that works for their particular environment or ways of working.
Release flow
The "Release Branching Strategy" is an industry-standard approach which advocates that the only type of long-lived branches - apart from the mainline - are release branches. This strategy is documented and recommended in the Continuous Integration book.
The mainline branch (e.g. master or main) is used for cutting edge development which is also more commonly known as "Develop on mainline".
Trunk-based workflows
A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’ *, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after.
Branch and Pull Request triggers
In Continuous Integration processes, builds are the first formal check during the software development feedback cycle. They should be fast, reproducible and a reliable form of feedback. CI build processes should be run as frequently as possible to ensure that any new errors or issues can be flagged up as soon as possible.
Developers should also be able to run as much as possible of the CI build processes on their local machines in order to preform sanity checks before committing code to central repositories.
Irrespective of the approach you take it is important that all Pull Requests (PR) should trigger a CI build too. This ensures validation of your changes before merging.
What is a Check-in Dance ?
The basic set best practice steps developers should undertake for effective CI is to perform the integration on a developer workstation before committing code to Version Control and ultimately to the build and deployment pipeline.
- Inform the team a change is coming, typically done by raising a Pull Request (PR).
- Get the latest code from source control.
- Merge on any conflicts.
- Run the build and Unit Tests locally and fix any problems found.
- Commit the changes to source control.
- Stop coding until the build passes.
- If the build breaks, drop everything else and fix the build.
A software development team will typically have some kind of process they will require member to follow when it comes to Raising PR's etc. and this will almost always be primarily dictated by their Ticketing, Project Management and Version Control Systems of choice, therefore how steps 1 to 3 are conducted may always be up to the team.
Steps 5 to 7 are also typically managed by a mix of the the chosen CI/CD and the version control system. Which typically leaves step 4 to be responsibility of the Developer. This also the step that more than likely adds the most friction for developers.
Automate Build and Test
Done correctly, build automation reduces manual labour by developers, ensures builds are consistent and complete, and improves product quality. The problem is that builds can take time, the aim of the game is to try ensure your builds are efficient and take minimum amount of time possible.
The problem is the more lines of code you maintain, and the more tests and routines you run as part of your build process- the longer the build will take to run. So by ensuring you have developers, constantly running the same Build and Test script that your CI/CD process will run, helps to ensure it is efficient.
Why Automate your Build and Test matters?
In CI/CD environments, builds are expected to be run frequently and the organisation depends on build output to guide development work. Builds that take a long time can be a big problem, and one that can steadily “creep up†on a development team.
If your build just takes 10-15 minutes longer than it should, it will still lead to significant overhead expenses and could seriously affect your organisations financial performance.
There are a few reasons why developers should be mindful and constantly evaluate the build and test performance :
- Developer wait time - Many surveys indicate that developers spend between 2 - 10 hours a week waiting for builds to complete, usually doing nothing more than twiddling thumbs!
- Context shift impact - If a build takes a long time to complete developers may lose context and lose concentration on related problem they were originally trying to fix, which reduces their productivity.
- Product Quality - The quicker a build is the quicker it is to verify that a bug has been fixed.
Over the years, I have developed many Bash and other types of scripts to automate it. However, I have for the past few years specifically for C# .net core projects taking to use Cake, to automate this task for me.
One of the primary reasons I like using Cake, is that I can also use exactly the same script as my Build Script on my CI/CD server. Therefore ensuring that my Development Process and Build Process use the same script.
What is Cake
Cake (C# Make) is a cross-platform build automation system with a C# DSL for tasks such as compiling code, copying files and folders, running unit tests, compressing files and building NuGet packages.
How to use Cake to automate your Check-in Dance
I make use of Cake and TeamCity in my Threenine.Data - Generic Repository for EF Core and in the source code you will find a typical example of a build.cake I write to build my projects and run my Unit Tests.
I have previously posted How to write a Cake Build script for ASP.net core project and despite the post being a few years old many of the principles still apply. In developing Threenine.Data is just merely expanded on the basic build script.
These days I mostly use Jetbrains Rider as my Integrated Development Environment (IDE) of choice for all my .net core projects, and you'll notice that my build.cake and build.sh files are in the root of my project or solution folder. This is also the default location the bootstrapper will place the files when setting up a New Project when using Cake.
This enables me to make use of the terminal window and run the ./build.sh to execute my build script and run all my unit tests in my project to ensure my code, compiles, builds and tests are run to verify my changes and the project is ready to commit.
A handy feature of Jetbrains Rider is that it enables other developers to develop plugins for the IDE to extend functionality, and Anna Dolbina has developed the really handy CakeIntegration plugin, that lists all the Cake tasks in a tool window. These tasks can be launched by double-click, and Cake output will appear in the console.
Unit Test Coverage
A great future of Jetbrains Rider that I really like is the built in Unit Test Coverage which provides a really great visual aid and illustration of where I need to focus my attention to implement Unit Tests.
As discussed in Pragmatic Programmer, I usually don't exactly follow the mantra of Test Driven Development, and always write my tests first then code to implement. On occasions, I don't always write tests, however I do always try write my code so that it is easily testable by attempting as much as possible to follow Agile coding with design patterns and SOLID principles, defined in Adaptive Code
Adaptive Code
Agile coding with design patterns and SOLID principles
Applying principles from this book, will help you create code that accommodates new requirements and unforeseen scenarios without significant rewrites.
Build Server Integration
As I mentioned previously we prefer to use TeamCity Ci/CD for our build server, we have used it for years and it works really well. We also predominantly use JetBrains based IDE's for all our development, We use Rider, PHPStorm, WebStorm, PyCharm, Goland, RubyMine and IntelliJ. As result, TeamCity is able to easily integrate and Jetbrains provide a great plugin, which helps a lot of time finding failing builds on different feature branches on TeamCity. Also triggering release-builds from IDE.
using the plugin, I am able to easily monitor the status of my build, and take any actions required should the build fail, without ever leaving my IDE.
Project Management
We also make heavy use of the Jetbrains YouTrack and Space for all our project management and as you may suspect there are great plugins available to remove the friction of using these tools from within our IDE's. So Developers never have to leave their IDE's to update tickets and status of the feature they are working on.
Conclusion
Using this approach I am able to quickly and easily run the Build & Test elements of of my project prior to check in dance, which significantly reduces the risk of breaking the build, as I am using exactly the same build script and test script on my local machine that is being used on the build server, so I at least have some consistency.
Building an application that can be deployed at any time, you will find that it’s no longer a big deal to try and get ready for a deployment. Having the ability to run a single script or click a single button to deploy the code to a staging and/or production environment.
Increased confidence in deployments, you should be able to deploy more frequently, resulting in a tighter feedback loop and more value being delivered more rapidly, as well as issues from the field coming in much closer to when the code that caused them was written (making bugs easier to reproduce and correct).
- What is this Directory.Packages.props file all about? - January 25, 2024
- How to add Tailwind CSS to Blazor website - November 20, 2023
- How to deploy a Blazor site to Netlify - November 17, 2023