Creating A Build Process

Jeff Atwood had a great post today about automating your build process (The F5 Key Is Not a Build Process) and it inspired me to finally write this post (I promised it a while ago - Making the Build).

As mentioned in my post Making the Build, an automated build process is very important. However, most software teams have a tight schedule and budget and find it difficult to justify spending time creating a proper build process, especially when nobody on the team has previous experience creating an automated build process.

Before I get to the benefits of an automated build, let me first describe the build for our product.

Our build process is fully automated. We have it scheduled to run on a dedicated machine every night. The build includes many different things, including compiling the source code, building the database from the previous version of our product using scripts, generating some source code, updating assembly attributes (such as version, company, etc), running unit tests, and more (too project specific to bother mentioning).

The benefits that we have found for our team are as follows:

1. Frequent builds -  Before we automated our build process, it was run manually (though we've always had scripts and utilities to help). Unfortunately it was sometimes weeks between builds. If there was a problem with the build it was very difficult at times to track it down.
2. Standard builds - Back when we ran our build process manually, it was not always run the same way. Often times steps were skipped because they took too long and didn't seem important (sometimes it wasn't, but when it was, it could take a lot of time to figure out what the problem was).
3. Build status visibility - Every developer gets an email every day about the status of the build. If the build fails due to something they checked in, it is usually easy to find the problem because it is only a single days worth of code and was checked in only yesterday - hopefully they still remember what they worked on yesterday :).
4. Easily get the latest build - The build process places the completed build (at least the parts of it the developer cares about) into a shared directory that all developers have access to. To run the build, they can simply copy the build onto their machine and run a simple utility to attach the database. This makes it very easy to debug problems with the build.

If you decide to create an automated build process, here are a few suggestions (some of the advice is specific to building .Net projects, but I think much of it should hold true for other technologies as well):

1. Fully automated - I can't seem to mention this enough :). The process should be able to run in the middle of the night on an unattended machine without any manual setup or teardown process.
2. Run regularly - The sooner a problem is found, the easier it is to fix. We build nightly, but many people build even more often than that.
3. Run in a clean environment - In order to prevent false positives with a build (the build succeeds but shouldn't have), the first task in your build should be to create a clean environment (create a new directory or clear out the directory that you are using). This is especially important in finding circular references.
4. Use proper tools - You don't want  We use FinalBuilder. The tool provides a visual interface for creating your build. I would not recommend using batch files, MSBuild (as your primary build tool anyway), or NAnt. These can be very difficult to maintain. You don't want to get into a situation where every time you want to make a tweak you have to relearn the whole build process. If your team has a dedicated build engineer, then MSBuild or NAnt are great tools.
5. K.I.S.S. - Keep It Simple, Stupid. The simpler you make the build process, the more likely it will be for people to use it correctly. For example, in our build process, the only thing developers need to do to integrate their project with the build is to check it into the correct location in VSS. We have a utility that will make sure all the projects get compiled in the correct order based on the references in the project file (Compiling Multiple Projects Without a Solution).
6. Don't underestimate the effort - It took me a couple of weeks full time to create our current build process. Of course once it's setup, and assuming you used the proper tools, it should only take a few minutes to make adjustments when necessary.

Many people also advocate that build should be able to be run by any developer on their machine. This hasn't been very realistic for our project due to the size of the project and the cost of the software we chose to use (FinalBuilder isn't exactly free). However, it is a goal that I approve of.

If your project doesn't currently have an automated build process, I hope this article has encouraged you to create one. Good luck.

Compiling Multiple Projects Without a Solution

I've posted a new article on Code Project. It is titled Compiling Multiple Projects Without a Solution. The article shows how to compile a multi-project application without using solution files (yes, it includes source code).

The code creates a MSBuild project file that will allow MSBuild to compile a list of VB.Net or C# projects in the correct order. The list can be loaded based on a root directory that contains .vbproj and/or .csproj files or a text file that can contain a combination of directories, project files, and/or other text files.

Making the Build

I am of the belief that an automated build process is perhaps the most essential element in producing a quality software application. The build process can be used to enforce good practices within the development team and also detect any issues within the application in a timely manner (the sooner you know about a problem, the less code you have to look at to determine what caused it). Even a small team with a simple application can benefit from an automated build process.

Recently I've gotten the opportunity to recreate the build process for the company I work for. The product is over 5 years old, is fairly large and complex (well over 200 separate .Net projects as well as database and legacy code), and has grown a fairly complicated build process using a combination of batch files and custom executables.

The main reason for recreating the build process is due to the difficulty of maintaining it. There were quite a few batch files and it wasn't easy to run. We averaged a build about once a week (builds should be run several times a day, nightly at most) and the unit tests weren't run very often and if they were, the results often times weren't published (it could be months before you find out a unit test no longer works and then who knows what changed that broke it).

I started building my own build software using Windows Workflow (see my April 28th post, Windows Workflow and Your Next Build System). Although it probably could work and is a great way to learn Windows Workflow, I soon remembered one of my life rules, always use the right tool for the job.

I decided to try FinalBuilder instead and boy am I glad I did. FinalBuilder (FB from here on) is a great build tool. They have dozens (maybe hundreds) of different actions (that's what they call activities or tasks) available to use, such as MSBuild, get from VSS, run SQL, update AssemblyInfo, NUnit, send email, etc. They also allow you to write script (either VBScript or JavaScript) that is run during the build. If that isn't enough for you, you can also create your own custom actions using .Net (or other supported languages).

I've looked at several of the build tools available out there today (especially the free ones :) such as MSBuild and NAnt and the thing that strikes me about these tools is that they seem to be designed for full time build engineers. I don't have time to figure out how to setup these tools let alone write a complete build process using them. Perhaps even more frustrating is that I actually did take the time to learn MSBuild at one point and had done some interesting things with it, however, I couldn't tell you the first thing about MSBuild anymore. I wouldn't use MSBuild to create an entire build process for the same reason I wouldn't use Perl to write an entire ERP system (write only code doesn't work well in a complex, constantly evolving product).

I am not sure if it would have taken longer to create the build process using another technology (such as MSBuild or batch files) or not, but the big benefit in using FB is in the ability to maintain it months later and, if I'm lucky, by somebody other than myself (I definitely don't want to be the "build" guy).

I'm sure there are other great build tools out there. If you know of one, feel free to leave a comment (I always love to find out about new tools :). I plan on writing another article within the next week or two that has more concrete tips for creating a build process (perhaps even some source code), so stay tuned.