In November of 2025 we were working on goals for the next year and one of them was to move to Kubernetes (k8s). I had not had the opportunity to learn k8s yet but understand container orchestration and management from working with nomad. So I was intrigued at working on that task but was worried because the last two people that did were let go. I started learning about k8s and service meshes and had a light bulb moment when I realized we could route traffic based on client ID and that would allow us to route our internal clients to new code and test it in production and then the deploy is just changing routing to let all clients use the exact same set of containers the internal clients were using.
I started researching service meshes and quickly decided istio would be the best choice because it was extermely powerful and allowed chaos testing and had a huge community. Once I had the service mesh picked out I started researching how to install it. We had already decided as a team to use eks and later we switched to eks auto. This allowed me to focus on the service mesh/service side of things exclusively and figure out how to implement my design.
I started off using istioctl to install istio and as I started working on running our services was using kubectl to install resources. I had heard of helm charts and that we should have one to install our services. So I started looking at that and created our own helm chart to install services that was pretty opionated. Then a teammate told me about helmfile and everything clicked for me. I can use helmfile to represent what an environment looks like, the helm chart to represent a single piece, and kubectl to view it all.
Our existing deploy process is to have a json file named after the release and everything that gets deployed has the tag to deploy in the json. I wanted to maintain this same workflow where the delpoy pipeline looks at that one file and then determines everything to deploy. This is why we did not go the argocd and git-ops route. I also wanted the values files for helmfile to be in the repo with the code. I have always like the ideo of everything being in one place. I’ve recently had the idea of expanding this even further with crossplane to include the aws resources you need.
When helmfile is run it is passed values for every service that is deployed in k8s and then a script determines what has changed and makes sure them come up healthy.