Operating Dev

Blurring the line between software development and operations

Operating Dev - Blurring the line between software development and operations

Evolving codelines – maturity levels for software products

Scene from the movie The MatrixI am often asked to give advice on the most important processes a software team needs to implement for managing their code. The discussion usually moves from version control, to branching and merging strategies, unit testing, code reviews, static code analysis, build automation, continuous integration, test automation, code coverage, stress testing, etc. But one aspect I find as important as all of the items above is the scope your code controls. That is, how many things sit outside your codeline and how can you ensure your code is complete.

When you think of code, what comes to mind first? The lines that define your application features I presume. But those features don’t come into life in a vacuum, solely from your code — unless you’re coding The Matrix that is ;-)

In order to run your application, you need machines or VMs in your network or hosted in the cloud; an operating system like Linux or Windows deployed on them and providing the basic environment to run applications; a variety of useful libraries that provide lower level functionality like network access or sending email; maybe a framework providing baseline structure and patterns for your application to implement the business logic on top; maybe an application server that handles your user sessions; some kind of persistent storage for configuration or user data (e.g. XML files, SQL or NoSQL databases, etc.); external web services or other third-party APIs to provide integrations or bring complex functionality into your app; …

While you clearly don’t need to worry about the codes bringing these components and pieces that support or allow your application to run, you do need to worry about the steps you need to take in order to make your application work well with them or use them. In working with various software development teams and products for the past 15+ years, I have come to recognize the following maturity levels a code typically goes through in order to achieve mastery and take full control over all of the moving pieces mentioned above:

Level 0 – the unconscious

At this level, you only write code directly related to the features you’re trying to build. You do write code to load useful libraries or call third-party APIs and web services, but those lines are essential for the behaviour of your application. No consideration is given to the environment or the data (aside from making sure you can read or write that data correctly).

Your code is like a newborn baby – completely at the mercy of the environment and the caregivers. It often fails to recognize messages it receives from them (time for bed, not for playing) or returns back responses that frustrate them (wailing and crying). Just like with babies, a lot of care goes into maintaining your application and making sure it runs properly.

Your codeline might look like this:

<project>
   + src
      + feature 1
      + feature 2

Level 1 – awareness emerges

You realize that too many things outside your control can impact your application. When inexpertly deployed and configured, your QA or product people are screaming or worse, customers threaten to leave for the competition. You start adding lines of code to validate assumptions about the environment — e.g. your app won’t start if a required library is not present or the wrong version is found; you add extensive exception handling to the methods that deal with data loaded from files or the database to handle cases when the data is incorrect, etc.

At this level, your app is like a six months old baby that stuffs everything in her mouth to check if she can eat it and learn about its basic properties like texture, smell, taste, hardness or softness, etc. Your code acquires capability to recognize danger and avoid it. It might even be capable of working around simple problems (e.g. retry on connection timeout, convert wrong data read from a file to an expected value) and still run properly and provide regular or at least limited functionality to its users, but this is still haphazard and mistakes are common, often bringing the application on its head when such an attempt fails.

Your codeline might look like this:

<project>
   + src
      + feature 1
      + feature 2
      + feature 3
      + healthcheck

Level 2 – early attempts at control

After going through several long nights dealing with deployment issues or discovering corrupted data after few bad migration experiences, you decide enough is enough and you write code to deploy configuration and other data or to configure certain aspects of the environment — e.g. scripts to create the database or deploy certificates. You also write a number of tools to help you with data migration and similar and you even hire your first QA person to help you test the deployment on the environments you support.

Congratulations, your code is now a toddler! When left without the caregivers, she can navigate the environment on their own and even make some changes to satisfy her needs or make it easier to achieve what her mind set to do.  While there may still be some bruises and scratches from time to time when load is applied or asked to deal with larger data or similar, the application can mostly deploy nicely and even work flawlessly with old data after the migration tools are run – though migration is not always easy.

Your codeline is expanding and might look like this:

<project>
   + src
   + conf
   + scripts
   + db
   + migration

Level 3 – working with structure and rules

You’ve learned a lot since you started. Your application is mostly stable and you’ve been able to more or less consistently add new features and expand the customers and environments your application supports. But you found that adding new features is getting harder and harder. Your team struggles with fixing bugs, spends too much time constantly refactoring the code and the codeline is generally harder to manage. It is time to organize the code better!

Your code is now a child rapidly moving through preschool, grade K-12 and University to emerge as a young adult with good knowledge in one or few areas and a fairly strong confidence in managing their future prospects. You’ve started keeping track of all your external dependencies and third-party libraries; you automated your build and started running a continuous integration server; you even made all developers write unit tests for any new code and slowly started adding tests for legacy code whenever someone had to modify it when fixing bugs or enhancing features.

Things have evolved and now your codeline might look like this:

<project>
   + <component 1>
      + conf
      + src
      + test
   + <component 2>
      + conf
      + src
      + test
   + third_parties
      + <lib 1>
         + v1
         + v2
      + <lib 2>
   + build
   + db
   + migration
   + scripts

Level 4 – becoming an expert

Progress has been encouraging and you really started feeling like you’ve been able to gain efficiency again in building out new features, but something is still missing. While unit tests are great for catching errors breaking other components when one makes a change, there’s been other more subtler problems that still make it hard to put the application together when trying to get ready for a release. Deploying into production is also a many day effort plagued with problems due to migration errors, missed steps, etc. It is time to automate the database updates fully, polish your packaging and install scripts and introduce more rigorous testing by building out automation for integration tests, UI tests, load tests, etc.

The going gets harder for your twenty-something in this period. There are many failures and implementing the new releases feels like an uphill battle – even if your team has been churning out new features while consistently meeting the sprint deadlines prior to the release. But the reward at the end makes it all worth it. Your code have emerged as an expert in managing its environment and dealing with almost any issue that it meets as people are trying to do more and more with it and they’re pushing it beyond the design specs and documented limitations.

Your codeline might look like this:

<project>
   + <component 1>
   + <component 2>
   + <component 3>
   + <sql_db>
      + schema
      + config
      + upgrade
      + rollback
   + <nosql_db>
      + sets
      + config
      + upgrade
      + rollback
   + third_parties
   + build
      + framework
      + scripts
   + deploy
      + package
      + install
      + uninstall
   + test
      + integration
      + ui
      + load

If your code is at this level, congratulations! Very many organizations, particularly startups or software development or IT departments within organizations whose core business is not software product development are at level 1-3. Getting to level 4 requires your team to adopt programming as a deliberate practice – it is a level at which hackers and software engineers differentiate. It also requires architecture skills that not many developers have.

Unfortunately, achieving level 4 is not enough. There are still many aspects that your code is not controlling. If you’re trying to improve your efficiency by implementing agile or lean processes you will find that you need to further push your skills and take full control of all aspects of your application runtime. But that is a topic for the next article in which I’d like to propose two additional maturity levels and their benefits. Stay tuned till next week.

Your email address will not be published. Required fields are marked *

*