Personal Opinion
All I really want is something that actually helps me do my job better, faster and with a smaller mental load. Being able to complete features faster with higher quality and more consistency, reducing my stress and allow me to have more balance. In the end that’s what I want most. To do my job well and also have time for everything else that is important. I think this should be the end goal of developer tooling. Creating better products for users while at the same time providing developers a better quality of life (Yes, I know it’s already a very good life but why not make it better when possible).
Ideally what would you want development to look like?
We want to have something that allows you to be productive and efficient while also supporting you and helping you catch and correct mistakes that will inevitably happen. Allow people to focus on things where a person adds more value and allow computers to focus on things where they add more value. Combining these two things means people get to do more of the things they find engaging and worry less about the things that are error prone but expected to be done perfectly and repeatably. Combining the two doesn’t mean job loss it should mean higher quality of work for a longer more sustainable period of time.
What do we want from development?
The ability to complete your tasks without having to have background or be an expert in multiple tools, operating systems, ci systems, deployment systems as well as how to undo all of that in the event that something goes wrong on one of those steps. What if experts in those areas could share their knowledge with you all the way to updating and improving your workstation and defaults so that you would not have to change but would still get the benefits. That is a better way to grow.
What if you could log on from any computer. No constraints or requirements on operating system (outside of teams creating things for specialized hardware, ios app). Simply login to your work environment from anywhere through the web, ios, android, mac, windows, whatever you would like to use as your daily driver. Then once you are logged in your environment is the same as everyone else. Same tools, versions and everything else for you to be productive, all without having to install, update, and learn all of these tools for yourself to begin with. Greatly reducing the barrier to entry and making it easier to actually accomplish something.
This means it would be far easier to help others fix and resolve problems. This means that knowledge and experience can be easily and quickly shared with everyone so that the effort required to complete tasks can continue to fall for everyone.
This is a great positive for everyone in an organization that embraces this. The more people you have sharing their insights and improving everyone’s developer experience the better it will become.
To do things faster, more reliability and if possible with more ease. Because the mindless toil that stands between the feature you just created based on a user request and getting that feature out isn’t productive for anyone. Everyone would be happier with that time removed and given back to other tasks that are meaningful for your users or product.
What do you actually want?
- Easy and consistent setup that is mostly done for you
- Pair programing partner (get ideas, fill in boiler plate, remember syntax, help with multiple languages)
- Fast and easy way to search your code base ( find code you can reuse or get ideas from)
- Intuitive source control for a main branch that has lots of changes that are small and concise
- Fast feedback, for yourself and with others in the loop as well
- Better code review and a system that rewards small understandable pieces of work that are built on top of each other.
- Easy and safe deployments (automatic rollback and with roll forward so the next working deployment takes over, that can also build in complexity)
- Faster easier ways to test and get feedback on new ideas
In the end I want to be able to do my job well and a developer experience should make that as easy as possible. You shouldn’t have to constantly fight your tools to stay consistent with best practices. Your tools should reward you for following best practices.
History and current situation
In the past most companies in software have used the technique of just throwing more people at the problem.
We aren’t going to assume there is one single magical change that will make your team 10x more efficient because that’s not the case. But what if you could keep drilling down on each individual part of the process reclaiming 10 minutes here and 5 minutes there. And potentially removing tasks from people and passing them off to other systems. While individually these things may not seem like much as you add them together the end result can be surprising.
So lets go through what software development consists of.
1. Coming up with an idea of what to do
This means coming up with all the tasks and changes that you or your customers want. There isn’t a huge amount to be done in this area. People and teams have their preference.
My personal preference is to explore different areas with inexpensive experiments that can be used to gain feedback and insights about next steps. No one has a crystal ball of what will work so make it easy to try many things.
2. Selecting tasks to do.
This one also comes down to people and teams. Flexibility for teams seems to work. Your best guess will most likely be wrong. Maybe just provide an open environment where people can be honest about why things are taking longer and so they can ask for help from relevant co workers or others. Don’t shame people because the estimate that is always wrong was wrong.
Try to break tasks down into small incremental pieces in a way that you can get feedback on them along the way. This can help with course correction from other members of the team or from users.
3. Implementing ideas.
Quickly plan something out so that you have an idea of what your end goal is and how to do that. Then create the simplest version possible to meet those requirements and get something working. Then it can be improved upon if needed. You can also do with by setting up tests first so that when you pass the tests you know you have the thing you wanted.
How do you make this better? Lets try to see what the best way to do this would be. For this you would be working with someone who has more experience than you and would help guide you as you created an implementation. You would also have tools that would help catch any small errors that could cause problems. Another thing to consider is a search tool to find if someone has already implemented something similar and the choices they made and what they learned. This can be very useful if you already have a large code base and people have already solved similar problems.
So what does this translate into. Well what if you could have that pair programming partner but they were simply a code completion or injection tools in your ide so that you would be prompted with potential implementations. Along with a code search tool for your code base so that you could find similar or relevant code you could reuse to help you implement your idea faster.
You would also want to make sure that your build and test times were as fast as possible as well. This means making sure that everything is consistent and easy to build quickly for everyone making changes to the project or that could need to make changes to the project. New people to the project, others working on the project who just want to make small changes. These types of changes can add value but the friction for changes like this needs to be greatly reduced.
Extending from a consistent build and test environment there should also be a consistent work environment so that everyone can build, use common tools and practices that can improve the best contributors while also lifting up less experienced people so that they can see and learn from the practices of others.
There area also some special cases here where you project can become too large or take too long to build on a single machine, for ambitious teams this could simply mean reducing the build time even further. In this case build actions can be pushed off to a remote execution service for those actions. Meaning the build can take place across as many machines as required to bring your build times down. This type of system can also use a cache so things that have already been built and have not changed do not need to be built again.
System like this also make it easier to share code across teams as well so code can easily and consistently be reused.
Another thing to consider when looking at your development time is interruptions or other requirements on your time that might break up your productive time. With this in mind it could be useful to provide developers with assistance that can take care of simple tasks for them and make things easier.
Another useful tool in this area is spelling and grammar assistance when creating documentation. This would make it much more approachable for everyone on your team to help keep your documentation up to date.
The idea here it to reward common tasks that you would like to happen often by ensuring there is as little friction as possible reducing any barriers. Common tools can make it much easier, give people more confidence and most importantly people should not be punished for trying to improve things. If they find it difficult to improve things there is probably a larger opportunity for improvement there.
As much assistance and tooling should be provided here as possible to make things easier, faster, more consistent, and reduce the load on the person responsible for the task.
4. Code Review.
Code review is a very important piece of software development and can sometimes make or break your teams ability to do quality work quickly and efficiently. There are some best practices for code review that should be layout out first. Pieces of work up for review should be presented in concise and pointed pieces. Each piece being easy to understand and review on their own. To create larger changes small concise diffs should be stacked on top of each other so that each diff retains a clear goal and can be review independently while at the same time allowing a larger piece of work to be completed quickly by a contributor. Each diff in a stack should be independently reviewable and the changes should only be dependant on previous diffs, they should not depend on future diffs as well. The key here is that the diffs should be independently deployable in one direction. Base commit and then moving up the stack through changes that rely on changes made previously. With work like this it is much easier to get many small changes landed and it is also easy to revert a change in a stack that may cause an error while still keeping previous work that did not cause any problems. It is also a much more productive workflow for developers as they can continue to work on changes and continue to stack diffs while others are being review without invalidating the reviews of previous work. This is great for everyone involved in reviews as it means far less rework and wasted time.The code review UI needs to reward this type of work though to make is efficient and easy for everyone involved. This can be difficult but the improvements can be amazing.
Also this step should be an opportunity not just for human review to provide feedback but for automated systems and other programs or checks you may want to run. These should provide easy and actionable feedback that could be included in changes.
5. Continuous Integration.
This should be used to check that tests and builds pass. Consistency and reproducibility is key here. You should not fall into the tap of just running things again because they might pass next time. This will waste money and time for everyone involved. The end result of your CI system should be a reproducible artifact that was created from a certain commit and it can be traced back to that commit. After the artifact is created and then most likely sent somewhere where it can be easily and scalably retrieved when needed the CI system is done. The most important thing from the CI system is speed and visibility for both the user and the system as a whole. This means that it should be easy for different types of users to get the relevant information out of the CI system. This can be different information for a developer, build infrastructure team, team manager and higher level managers as well. It should be simple and consistent to show relevant information for different parties.
For example a developer would like to be able to quickly see if their commit passed CI or if it failed and why. This can mean showing errors from this run along with the history for the failure to see if this change was the culprit or had it already been broken. For someone on the build infrastructure team there could be a completely different set of information and trends that would be relevant for them about build time, cache hits, amount of data transferred and other breakdowns of builds. Again a different view for a manager my include what commit are associated with different features and what stage of completion they are at along with how a team is doing overall or how certain members on a team are doing. This can be valuable for many reasons. Again I would not recommend using this as something to change behavior off of as most of these metrics can be manipulated, which pushed people will start doing. Use these things as a tool to find where help should be provided or things could be learned and improvements could be taken and rolled out to everyone. This information can also be sent up to an overall view that includes local development, code review, ci, cd, experimentation and then completion of the task.
Also your ci should work on every platform that you need to test against from a single and coherent place.
6. Continuous Delivery
Continuous delivery should take the artifacts created by continuous integration and then consistently deploy. This can be very difficult to do and so far most implementations just cover a very basic set of features for this.
The first place to start is tracking changes as they are deployed. This should be easy for a user to find what it’s current state of deployment is. With this it should be easy to visually track your work all the way through the different stages of deployment.
It should also be comfortable for your developers to rely on your cd system to catch and mitigate mistakes that make it through. Mistakes will happen to everyone the best thing to do is have systems that will help mitigate those mistakes and allow you to find and fix them simply and easily. With cd that means stopping and rolling back a deployment if/when there is a problem, while also retaining the logs and making them easy to find later. The safety that comes from you system automatically rolling back and potentially mitigating the problem without any input from an engineer is great.
Your cd system should also have the ability to mark artifacts as bad. That way you will not accidentally automatically rollback to a broken artifact if there is ever a problem.
Your cd system should also allow for roll forward in the event of an error and your team already has new changes that could fix the problem. This is very useful when you have many changes and deployments happening. So instead of stopping all deployments for that service. You can continue to work if a fix is already being deployed. In this scenario instead of rolling a change all the way back it is just rolled back as far as the next deployment has reached. This means you have less changes in your system and you could have potentially closed the time it takes to resolve your problem almost without any human intervention. Making everyone more productive and less stressed along the way.
Your cd system should also be able to deploy to multiple systems and show you the state of everything from one central location. This is key so that someone might have a chance at understanding the state of your system at any time. This can be very useful if you have systems that include vms, containers and serverless functions or any number of ever expanding ways and locations to deploy software.
A big piece is that you should be able to trust your cd system and that the default actions it takes should make your job and life easier.
Your cd should also be able to grow with you from deploying to a single vm, serverless function, or container service to deploying gradually across multiple data centres across the world with multiple regions in each one. The setup, defaults and abstractions should be allow you to grow as your system does without having a massively complex onboarding or crazy customization to make things work at scale.
CD should just be responsible for deploying new software to the correct environment. Actually enabling that software should be the responsibility of a different system.
7. Experimentation and feature flags
Enabling or disabling features along with creating experiments should be the last piece of your software system that actually controls what people see or how they interact with your system. This can take many forms but lets break it down into two pieces that have some commonality but also some distinct differences.
First lets go over feature flags as more people seem to be using them. These should be used as a way to turn a feature on or off and can be long running. This means that you might want to be able to turn this feature on or off for a long period of time. Lets say you have essential functionality in your services and then pieces that are nice to have. A good example of a long lasting feature flag is one that turns the non essential pieces off under heavy load so that your essential services continue to work as expected delivering value just without extras that don’t stop that main value from being delivered. This can be a very useful tool to ensure reliability in difficult times. It also allows changes to happen immediately without having to deploy anything. Feature flags can be applied in many areas but the cost associated with maintenance should not be ignored with long standing feature flags.
Experimentation should and could be used for most other case. This is when you are not sure what the end user actually wants (you should assume you do not know). This takes the form of potentially trying many variations of changes. Simple examples are different colors, fonts, copywriting but this can also go far deeper. What if you want to generate layouts and find the one that people use more. How about picking the best implementation for a microservice that might have a few ways to implement. How about all of the different combinations for flags or settings you can set in your builds or compilers? All of these things can be tested if you are willing try many combinations or variations so you can end up with the best result. There is nothing wrong with that and in my eyes the alternative of believing you are correct without any proof or results can cause poor results. This also allows anyone to contribute ideas and test them if they have time and can then show the result is positive (hurdle rates for accepting a change can vary). Importantly if something consistently shows to not have a positive result or neutral result then that line of changes should be reevaluated and possibly abandoned so that time and work can be put toward something that can create more value.
Doing this consistently at scale is also a key. If you find this adds value then it’s only a matter of time until you are running multiple experiments at the same time on possibly interacting pieces. With this it becomes key that you can get results from all of this that isolate a single change as much as possible so that you know if it really had a positive impact or not. Without this you may throw away a valuable change because it also happened along side a different very bad change. The ability to separate these results while still allowing many experiments to happen at the same time is key.
Also experimentation should be common practice throughout your entire stack not just for frontend. It is nearly impossible to correctly reason about all of the interactions in a very complex system like software. So why don’t we stop pretending we know the answers ahead of time. How about we take all the time and energy that is usually used up debating a solution and put a good chunk of that towards creating different solutions and then experimenting with them to get their actual positives and negatives from actual deployment into the system.
What is a better and more reproducible way to reach the best result. To debate the “best” implementation usually ending with what the highest ranking person in the room wants or to implement multiple solutions quickly get feedback on them and then learn and implement a better version that includes the best tradeoffs as measured for your goal. The second option should get you to a better result far more consistently and also allow you to take full advantage of the collective knowledge of the team. The first option may feel better when you get to have the final say but at the risk of not knowing if this actually is the best solution which is negative for the whole team. The second option also has the added benefit of pushing a team to default to action and implementation vs creating the “correct” solution. I have nothing against trying for the best solution possible, I think that should always be the goal but rarely is the best solution the one that is implemented the first time, especially if you have to learned from others mistakes or attempts at doing the same thing.
I think the key to understanding if experimentation is some that you should implement is: Are you always going to be right about everything you ever try to do? Is that really the bet that you want to make? Or would you rely on a team to find the best result efficiently and consistently after trying a few different options and learning from them?
Summary
Then when you put it all together in a way that makes sense and can be navigated easily you have a productive way to do development. That is what we should be trying to make. Something that makes development more focused, consistent, and provides consistent and productive feedback along the whole development process.
If you can break things down and continue to make them easier and faster, rewarding best practices then you have a far better chance of actually reaching the goal of creating an amazing developer experience. Just waiting for it to magically happen or for a product to solve all of your problems is not likely.