Introduction rant
Throughout the various places I've worked, how to setup a development environment has always been a subject of contention and argument. Each developer always has their preference as to what to use to develop which is fine in most cases; however when it comes to how the application you are all working on runs and how consistently it runs that application is of great importance in my mind.
This comes down to a few simple things that need to be met for development when I am building out a project.
- consistency
- test-ability
- flexibility
- developer/ops friendliness.
It is hard to get all of these, and truth be told you won't get all 4 perfect in any project. The important thing is that you get at least good with all of them. This article came from my setup of a micro service infrastructure that I had developed for a company to support high volume traffic without a service discovery solution. This made both developer and ops management of a distributed computation application with micro services a serious challenge.
The problem
The issue here is that without a solid service discovery solution in use such as Consul ( An article will come on that later ) but still using micro services there was no consistent way for the developers setup all the services and manage them in a reasonable way. Before I came on board to where this project happened devs had to manually keep track of everything running on their box and they didn't have a choice but to start the entire set of services. It was a situation that wasted a lot of time and effort if anything went wrong in a single service.
The solution
The solution to this issue is long winded, but included converting Java services into Node.js ones, and introducing PM2 as a process manager/runner. But then the issue of how to create consistency and a service management cropped up for developers.
To this end I embarked to find a reasonable and easy way for developers to manage these services on their boxes. I did a fair amount of tools soul searching for a while before dedicating myself to a type of solution but finally just settled on setting up a command line tool that developers could install via NPM from our internal Nexus.
The solution ideals
The idea behind the solution is that each service we build has a PM2 config that contains all the details for running that service. On top of that the PM2 Json configuration format allows me to define an instance to be run by name. This is a great setup for developers to be able to run things such as pm2 restart exampleService to manage their environment easily.
The ideal flow is that a developer can checkout any service from our Git servers put them in a single directory and use a tool to generate a PM2 configuration that handles the running of all the services in that directory for them.
The solution code bits
The tool assumes you have a single directory on your developer box with all the services under it. For example :
- AllMyServices
- service1
- service2
This is so our tool can walk through each service project and extract the configuration for PM2 and use the application definition to build a service cluster configuration. To build the tool I used the following packages and technologies.
- Vorpal.js - A Node library for creating command line tools.
- Babel JS - For modern JS hotness.
- Node
- Npm
lets take a look at the actual code behind this tool, be gentle I wrote in about an hour :
Alright, so this isn't perfect, and honestly this is the unpolished but tried and trusted code that is currently being used. But that being said lets take a look at what is happening here. It's a little long winded but pretty straight forward. In essence all this does is :
- Walks through the directory that it is given look for files named pm2.config.json
- Loads that configuration file into a JSON object
- Modifies the pathing so that you can run the configuration in any directory
- Dumps that config out in the out directory given.
This will provide you with a PM2 config that can startup and manage each node process that had a pm2.config.json file in the search directory which is pretty sweet. Granted this could use a fair amount of improvement, I won't say otherwise, but I think some of the most useful code is the raw concepts that get a developer moving in the right direction. Lets take a look at a configuration that was generated by this tool :
If you are familiar with the PM2 configuration structure this will look pretty familiar if not a little boring, but ultimately boring is kind of our goal here; a simple way to manage your node instances. With this configuration you are able to issue commands to specific instance being run, or the entire stack of instances/services.
For example, using this tool to build a config and run things looks like this for one of my own projects that is smaller and doesn't have the support of large infrastructure, it's just a 2 instance application that lives on my own box currently.
As shown above though, it provides a huge amount of usability for a developer as your distributed application grows and becomes more and more separate pieces. With this kind of setup the developer can load the config once and then flip instances off and on as they are needed for development.
Conclusion
Again PM2 allows us to manage modern Node.js projects with ease and provides us quick ways to build out tools. Though this is not a perfect solution, it has worked great for my team thus far in our state of development. This can also be a great alternative to happening to set up an orchestration system on every developer machine you have which is an absolute nightmare from my experience. There are alternatives to this as always, Docker images being the most common to come up.
While I love Docker and am currently actually building these services into Docker images for an orchestration system in my current project; I feel that after you hit a threshold of instances that need to exist on the developers box it becomes unmanageable resource requirements wise.
Being able to create this type of managable ecosystem on your developers machines ultimately leads to more felxability everywhere that your application goes. There are a lot applications here for QA and testing as well, being able to single out specific instances for debugging within the stack or even multiple versions of the same service for debugging becomes an easy and relatively painless task.