While advising a client how to properly implement agile methodologies like scrum recently, I was talking about the importance of automation to their team and mentioned that automation starts with standardization. What I meant was that to reduce the cost of automation they need to standardize the platforms and technologies they use for development and production. The reaction from their people got me thinking. Their response was ‘Are you saying we should ask everyone to use only one programming language? We are a small company and we have been trying to attract talent by letting them choose the best technology they think will help them do their work.’
I am used to looking at the cost of automation through the time and effort required for an automation team to build and implement frameworks for automated builds, continuous integration and deployment, automated testing, etc. When one framework is expected to deal with a multitude of technologies and platforms the cost can exponentially grow and often put a stop to the effort to automate altogether. This is what I thought as I was finalizing my presentation and trying to plant a seed to get the people in the room to think what it means for them to standardize.
Before I started working with SaaS and cloud infrastructure I used to think that investment in proper installation technology is a must for anyone who wants to develop serious software. If you got your builds going but you manually install your software by lugging zip files to servers and downloading configuration files from the team wikis, followed by manually adjusting few settings to match your environment, you’re opening yourself to human errors. In a production environment this could stop you from releasing and all of your investment in continuous integration and automated testing will be wasted.
Packaging your software in an installer and using the same installer to deploy to your test and production environments is surely a great strategy to avoid this problem. You can then make your continuous server deploy the software for you with every build so you get yourself an automated and continuous deployment. Even better make your developers use the same method to deploy to their dev environments and voila, all of a sudden no one can say ‘it was working on my machine’ when things go wrong in test or production.
But this leads to a problem. If your installer doesn’t work on, say, Linux, you will either alienate your Linux fans on the team by forcing them to move to Mac or run Windows VMs or they will work around your process and keep on manually deploying to their development environments. What would you do? Will you make your packaging tools produce .rpm or .deb files for their flavour of Linux? It depends is a common answer and most people start talking about cost at this point.
You see, during the last year I came to recognize that I have been thinking of the installer problem from the wrong perspective. What matters is your ability to automate your deployment, not your packaging! Actually, if you can get rid of the packaging altogether and simply deploy your builds to any environment you wish even better.
(Of course, producing an installer could be a business requirement if you are producing on premise software or need to comply to standards to deploy mobile apps to “app stores”. I realize that my example here won’t work in such a case, but I think the discussion around standardization still applies — when your business requirements drive the technology choices then you may not be able to choose to reduce the options without sacrificing the business but you can surely still standardize in other areas.)
It may sound like a blasphemy to some, but if you can “build” your software on your target environment directly from source you free yourself from many platform choices you had to make in the past by building the binaries in one place, packaging potentially in another and then installing on a third.
I am sure those with experience working on build automation will rage now and mention things like polluted environments – previous builds might leave files lying around that can cause issues; security holes – the more unnecessary compilers and other tools you leave in production the more targets for exploits are available to hacker; escrow for compliance certification or audits – you could fail your certification if you can’t prove that the code you deliver to the auditor is used to produce what is in production; etc.
Addressing those is probably a topic for another post and I won’t worry too much about them for now. Just think that maybe those questions come from the wrong perspective. Just like I was thinking about deployment in terms of packaging and installers, you may be thinking about building software in the wrong way. No one said you can’t build your environment too as part of your build. Heck, you can even ‘build’ your build framework as part of your build too if you can bootstrap it with some simple scripts.
Keeping the discussion around builds and deployment at the back of your mind, let’s go back to the topic of automation in software development in general and the need to standardize before you automate (or as part of your automation effort). I won’t deny that an organization that chooses to deploy only on one OS and build software with only one technology for each application layer will have a drastically reduced cost of automation.
But as the picture above depicts, the technology choices available are immense and it is hard to pick the ones that will ultimately allow you to build successful products and grow your business. Questions like ‘Do you hire Java or PHP developers?’, ‘Do you need SQL or Hadoop experts?’, ‘Do you invest in moving your existing Delphi developers to C# or do you let them go?’ take much of your time and influence the culture you build at your company. You may even feel like you’re not only standardizing your technology but your people too — you know you’ve done that when it is easy to write job descriptions.
The truth is, you’re probably right. I’ve experienced companies trying to switch from an old legacy technology to a more modern choice and in all cases it was pretty hard to do it. When you standardize the people that work for you no one knows how to handle the newcomers that don’t fit the old standard when you try to make the switch and the newcomers get frustrated that no one is willing to change. You could try to avoid this problem by forming a separate team to focus on (or outsource) the rebuilding of your product but that will only make things worse as you will now have two cultures and because each follows a different “standard” they can’t find any compatible grounds to cooperate.
Does this mean it is better not to standardize? No, I am not arguing against standardization. I am just having second thoughts about some typical standardization choices I am used to seeing in software companies. I think it is important to think about the impact they have on the people in your organization. If you can standardize the technology without standardizing your people, great, go ahead and pick your technology. However, if you can hire outstanding developers who can pick up any technology as needed, thus making the need to standardize obsolete, don’t think twice, hire them. You benefit from bringing in people eager to learn and grow, the two characteristics you need to build into your company culture if you want to build a successful business.
But we conveniently put automation under the carpet by discussing the pros and cons of standardization alone. If standardization is not possible or desirable for you organization, does that mean you will have to deal with costly automation or maybe give up on it altogether? I think not. I think you just need to start asking different questions than ‘how much will it cost’. Questions like ‘Can I automate certain processes so I gain freedom from technology choices?’, ‘Can I build automation that adds value to my team and/or products?’, ‘Which platforms should I use that have been designed with automation in mind and allow me to extend their capacity by integrating all my tools?’, ‘What can I do to make automation part of the team/company culture?’.
No one said it will be easy, though. But is you wanted easy you would not have started a company or worked at a startup, would you?