Operating Dev

Blurring the line between software development and operations

Operating Dev - Blurring the line between software development and operations

Evolving codelines – mastering the runtime

Image from http://www.techgatherer.com/newvem-unveils-first-ever-iphone-app-for-managing-amazon-cloud-services/In last week’s article, I have argued that most software code is at what I called maturity level 1-3 and only those teams that practice coding as a deliberate discipline achieve level 4. At this level they are controlling every aspect of their deployment, like configuring the application, managing shared libraries and third-party dependencies, deploying or migrating their database, etc. They are also taking advantage of scripted builds, continuous integration and automated testing, which are pre-requisites for successful adoption of agile and lean methodologies.

But there are two more levels (and maybe more can be achieved through innovation) that are pushing the limits of what is possible with code. Arguably, these levels are only possible thanks to advances in virtualization and cloud technologies and may be beyond reach for certain type of software products that have to deal with traditional infrastructure or where the maturity of the tools available is making it harder to go beyond level 4. (Those projects are possibly limited more by tradition and mindset than tools, though.)

Let’s look at each new level in more detail this time as I think it is important to understand the value of moving forward with your code and the associated changes required to do so.

Level 5 – mastering multiple skills

If you’re happy with how your code behaves and you’ve had good successes in deploying a number of software releases that brought valuable new features to your users, why would you make any changes? I have few things in mind but one of them is competitive advantage. Getting features to market faster is as important as ever and if you can reduce the cost of adding new features, while maintaining or improving your quality, you stand to gain a lot. Scaling your teams so you can get more features done is hard, but there is a way to do it without necessarily having to double your dev team. Enter the lean code!

I am not happy with the term lean as it is used in software development – I much prefer to talk about continuous improvement and learning, but it emphasises an important aspect, which is the concept of waste reduction by optimizing the whole. If you think about the end-to-end process for introducing new features for your customers, you’ll quickly realize that you’re doing a lot of analysis, design, coding, building, deploying, testing, redesign, more coding, rebuilding, redeploying, retesting, etc.

Some of these steps are quite costly so you’re likely doing them once or twice per release to keep your sprints within the two week target you set for yourself. If you could only do some of these steps in an automated fashion the cost of doing them repeatedly would be close to zero and thus you can afford to do them often and reduce the need for big efforts at the end of each release to prepare you application before it goes to the customers.

To automate, you typically need to write more code and use suitable tools to run your automation. You’re already using some at level 4 as you’re doing some automated testing. One ingredient that is missing is full control over the environment you’re using for your builds and tests.

Thanks to virtualization and cloud technologies, now your can turn your build and test infrastructure into code too. At this level, your code now includes much more than just features. Besides the build scripts, unit tests and other automated tests, install and database scripts added in previous evolutions, it now includes templates, snapshots, scripts and launch configurations for creating your infrastructure from scratch.

You’re likely leveraging the APIs available from your VM or cloud provider in a combination of Chef cookbooks or Puppet templates and can build almost anything from scratch. Best of all, you are able to use your continuous integration framework to prepare fresh new environment so you can deploy and test every build – you can even deploy the build in multiple environments and run different test suites in parallel.

The time from introducing a bug in your code to catching that bug has been reduced dramatically and you can effectively deploy any sprint to production as the time to validate the product before release is virtually zero – the difference between releases and sprints starts to fade over time.

Your code is finally at the peek of adulthood and it feels like it can do almost anything that you desire it to do. Heck, you even managed to script the creation of the build environment and you can move your builds anywhere you want by a simple git clone git@github.com:acmecorp/BuildSystem call and then running a bootstrap script from within the BuildSystem folder to do the rest of the job. There is really nothing else you can think of that can be improved – or is there?

Level 6 - transcendence and freedom

Your product owners are encouraged by all of the great work you have put in implementing continuous integration, deployment and testing and now they want to release individual features to customers so they can stay ahead of the competition. Batching features into releases every few months is no longer good enough. You devise a strategy by generously using feature branches to support the development of many new features in parallel and train people how to properly merge features that are ready for release into the master branch for final validation and push into production.

At the start it seems to go well and you’ve had few successful incremental updates. But managing many branches, dealing with merge conflicts, re-educating developers on the difference between rebase and merge, etc. feels like a waste. Spinning up builds for each feature branch and environments to deploy the different software versions to run regression testing also feels inefficient and increases your costs as you need more and more VMs and cloud instances to sustain it all. Furthermore, each of those builds goes through the same steps – stage, compile, package, deploy, test – even though only a tiny portion of the code has changed in each. There must be a better way.

Luckily there is one, but it requires you to throw away the rules and structure you so painstakingly have put in place as you were maturing your code through the past levels. These are some of the things that can help you regain freedom and transcend the processes that impede innovation and are blocking you from doing rapid releases:

  • Rethink your architecture. Are you correctly applying patterns that support decoupling and service-oriented systems? Can you draw the boundaries between the different components so that feature development doesn’t require changes in many levels of hierarchies? Can you abstract away the business logic and encapsulate different computations into separate services? What prevents you to improve the reuse across different components?
  • Rethink your builds. Instead of recompiling every line of code, detect the changes and rebuild the affected components only. Better yet, take advantage of your decoupled architecture and service model and focus on managing dependencies and versions better so you can rebuild only the services with changed code. Make the build process part of your deployment by blurring the lines between building and installing your services – let the build process prepare and update the environment and your components based on what is already running and what had to be rebuilt.
  • Build the deployment into the fabric of your application. You’re already controlling the environment – throw away your install packages that can handle zillion types of deployment scenarios as the world in which your app is now deployed is under your control and doesn’t vary. Bootstrap your services by letting them deploy themselves using dependency management tools. Build service discovery mechanisms in your code and decouple the location of various services and the infrastructure they’re running on independent of the rest of the system. Don’t upgrade if your product is SaaS – install on a freshly built environment and throw away the old one once you switch your users over.
  • Version your data and objects. Minimize the changes to your database schema by abstracting away the objects stored there (e.g. keep only metadata and keys in a SQL DB and store your objects either in a NoSQL database or in files on a cloud storage like Amazon S3 using standard format like JSON). Version your objects and write your code to properly upgrade or downgrade the object when incompatible version is found. Data migration is a waste – let your code handle the data gently. Storage is cheap – keep all versions of your data and implement mechanism to propagate content changes to one object version to others, so upgrades or downgrades don’t cause data loss.

These are only few of the things that will help you crank up the maturity of your code. If you’re working on SaaS products and are deploying your system in a cloud environment like AWS, think of the steps you need to take to build a truly modern architecture and deployment that is self-monitoring, self-healing, and scales up and down based on demand. Think of the services built into your application and how many of them add true business value, while others can be replaced by similar services offered by your cloud provider.

Moving up through the levels

Software development, with exception of a small number of trivial applications,  is tough. Most software applications are complex and as features get added to them become virtually impossible to fully understand and control. I hope this and the previous post help you realize some of the things that are important to improve in the code in order to deal with complexity.

It is exciting to think about the opportunities when every aspect of your application and the environment in which it works can be controlled through code. Long time ago, testing was manual and labor intensive – today companies hire developers to work on automating tests. Not so long ago, IT was in charge of the environment in which software got to be deployed – nowadays DevOps is changing the way Dev & IT work together, while cloud models like IaaS and PaaS change the skillset required in IT.

In my experience, it is almost impossible to achieve levels 4, 5 or 6 without going through the initial levels – the best way to learn is to fail or at least live through some pain. I have been able to bring one of my past projects to level 5 and have been able to get an inkling of what is possible if you start moving towards level 6. Where would you rate your codeline to be in this model? Do you think you can push it to level 4 at minimum? What do you anticipate would be a roadblock for achieving level 5? How about 6?

Your email address will not be published. Required fields are marked *