Introduction rant

Throughout the various places I've worked, how to setup a development environment has always been a subject of contention and argument. Each developer always has their preference as to what to use to develop which is fine in most cases; however when it comes to how the application you are all working on runs and how consistently it runs that application is of great importance in my mind. 

This comes down to a few simple things that need to be met for development when I am building out a project.

• consistency 
• test-ability
• flexibility
• developer/ops friendliness. 

It is hard to get all of these, and truth be told you won't get all 4 perfect in any project. The important thing is that you get at least good with all of them. This article came from my setup of a micro service infrastructure that I had developed for a company to support high volume traffic without a service discovery solution. This made both developer and ops management of a distributed computation application with micro services a serious challenge. 

The problem

The issue here is that without a solid service discovery solution in use such as Consul ( An article will come on that later ) but still using micro services there was no consistent way for the developers setup all the services and manage them in a reasonable way. Before I came on board to where this project happened devs had to manually keep track of everything running on their box and they didn't have a choice but to start the entire set of services. It was a situation that wasted a lot of time and effort if anything went wrong in a single service. 

The solution

The solution to this issue is long winded, but included converting Java services into Node.js ones, and introducing PM2 as a process manager/runner. But then the issue of how to create consistency and a service management cropped up for developers.

To this end I embarked to find a reasonable and easy way for developers to manage these services on their boxes. I did a fair amount of tools soul searching for a while before dedicating myself to a type of solution but finally just settled on setting up a command line tool that developers could install via NPM from our internal Nexus. 

The solution ideals

The idea behind the solution is that each service we build has a PM2 config that contains all the details for running that service. On top of that the PM2 Json configuration format allows me to define an instance to be run by name. This is a great setup for developers to be able to run things such as pm2 restart exampleService to manage their environment easily.

The ideal flow is that a developer can checkout any service from our Git servers put them in a single directory and use a tool to generate a PM2 configuration that handles the running of all the services in that directory for them.

The solution code bits

The tool assumes you have a single directory on your developer box with all the services under it. For example : 

• AllMyServices
• service1
• service2

This is so our tool can walk through each service project and extract the configuration for PM2 and use the application definition to build a service cluster configuration. To build the tool I used the following packages and technologies. 

• Vorpal.js - A Node library for creating command line tools.
• Babel JS - For modern JS hotness.
• Node
• Npm

lets take a look at the actual code behind this tool, be gentle I wrote in about an hour :

Alright, so this isn't perfect, and honestly this is the unpolished but tried and trusted code that is currently being used. But that being said lets take a look at what is happening here. It's a little long winded but pretty straight forward. In essence all this does is :

• Walks through the directory that it is given look for files named pm2.config.json
• Loads that configuration file into a JSON object
• Modifies the pathing so that you can run the configuration in any directory
• Dumps that config out in the out directory given. 

This will provide you with a PM2 config that can startup and manage each node process that had a pm2.config.json file in the search directory which is pretty sweet. Granted this could use a fair amount of improvement, I won't say otherwise, but I think some of the most useful code is the raw concepts that get a developer moving in the right direction. Lets take a look at a configuration that was generated by this tool : 

If you are familiar with the PM2 configuration structure this will look pretty familiar if not a little boring, but ultimately boring is kind of our goal here; a simple way to manage your node instances. With this configuration you are able to issue commands to specific instance being run, or the entire stack of instances/services.

For example, using this tool to build a config and run things looks like this for one of my own projects that is smaller and doesn't have the support of large infrastructure, it's just a 2 instance application that lives on my own box currently.

As shown above though, it provides a huge amount of usability for a developer as your distributed application grows and becomes more and more separate pieces. With this kind of setup the developer can load the config once and then flip instances off and on as they are needed for development.

Conclusion 

Again PM2 allows us to manage modern Node.js projects with ease and provides us quick ways to build out tools. Though this is not a perfect solution, it has worked great for my team thus far in our state of development. This can also be a great alternative to happening to set up an orchestration system on every developer machine you have which is an absolute nightmare from my experience. There are alternatives to this as always, Docker images being the most common to come up. 

While I love Docker and am currently actually building these services into Docker images for an orchestration system in my current project; I feel that after you hit a threshold of instances that need to exist on the developers box it becomes unmanageable resource requirements wise. 

Being able to create this type of managable ecosystem on your developers machines ultimately leads to more felxability everywhere that your application goes. There are a lot applications here for QA and testing as well, being able to single out specific instances for debugging within the stack or even multiple versions of the same service for debugging becomes an easy and relatively painless task.

Links

• Vorpal.js - A Node library for creating command line tools.
• Babel.js - A lovely JS compiler that gives us access to next gen JS
• PM2 - The lovely Node.js process management tool

Introduction rant

Over the course of the last few months I have been in charge of developing a sustainable and observable project structure for Node micro services. While I have done many things like this before, each environment and ecosystem that you develop in provides different challenges. Using PM2 for your node projects is not anything revolutionary if you live in the world of Node development; however using PM2 to its fullest can be a bit mystifying depending on your development requirements. 

One requirement I came across with my latest endeavors is when your employer doesn't want to invest into Kyemetrics as a service, there is a limiting technical factor that doesn't allow direct reporting to the Keymetrics platform, or you simply get dismissed by the higher ups but there is still the requirement for the existing technology ( such as an orchestration system ) to be able to pull metrics from your service/app.

It is the last situation above, that lead me into the deeper parts of the PM2 ecosystem tools and how to expose things not only that came stock with the metric reporter, but very specific things that I wanted to expose for one reason or another.

At this point if you don't have a understanding of what PM2 is or what it does for you, I suggest you read this before moving forward.

What is PMX?

PMX is a module for the PM2 runner that implements an API that allows exposure of metrics tied to the process sandboxes and layer 4 of the OSI model. This allows for the monitoring of various metrics associated with each application process being run.

That is really just a lot of talk to say that it can let you know when your Node instances are on fire. Also it can reveal useful patterns that can show you data that can lead to change on how you scale your instance or utilize certain resources.

Starting a small example

Download Sample Code

Let us get to the good stuff and actually show how it is used and the code implemented. It is actually a pretty painless process assuming you are already using PM2 to manage your project. If you don't want to bother going through the steps of creating your own project you can download the sample code above.

Installation and setup

For this particular article we will assume our project structure looks like this, you can also just download the code example for this article.

• myProject
• metricsDefinitions
• count.js
• middleware
• countMetric.js
• node_modules
• package.json
• bootstrap.js

and it will be using the following libraries for our example build out : pm2, pmx, and express. You can install these by running

 npm install pm2 pmx express --save 

After your dependencies have been installed modify your package.json scripts section to mimic the following

This adds the ability for your npm start command to also start the metrics API if your app start up was successful. Really we are just wrapping the PM2 startup commands here but leveraging the npm scripts section to ensure that PM2 exits successfully for starting your app is just good practice in my book. You can just do the whole thing in the start script command as well with pm2 start bootstrap.js; pm2 web which will always bring up the metrics API regardless of your app state which can be useful if you are going to use it to check the status of you Node instance(s).

Finally writing some code

Now we can get to actually adding code to files. We will be creating a small server that keeps track of how many requests have come Into it over the course of its life span. We will then use a custom metric probe, and the PMX API to expose that information.

Create, if you haven't already, a bootstrap.js file and add the following to it :

Bootstrap.js walk through

In our bootstrap.js we setup a few things for our app. First we do the normal requires just to get the express app up and running. On line 3 however we import the pmx library and configure it. There are a few options that are boolean flags

• http - HTTP routes logging (default: false)Enable pm2 to perform http watching for http related metrics
  
• errors - Exceptions logging
• custom_probes - Auto expose JS Loop Latency and HTTP req/s as custom metrics
• network - Network monitoring at the application levelEnable calculation of lower level network statistics
  
• ports - Shows which ports your app is listening on
  

There are a lot more options that the pmx interface can be configured with which can be found here , for now however these are the ones we need to ensure are setup for this example. The following is the rest of the steps taken in our bootstrap.js file

• Then we pull in our metric middleware which we will go over in just a bit. 
• Then we insert that middleware into the Express app request life cycle so it can intercept all requests that come into our app. 
• Then we add a health check route so we can check to see if our app is running, but also it's a simple route that we can hit to trigger our metric. 
• Then we start our server and output a success message for us to see in the console.

countMetric.js

In our bootstrap.js we pulled in a countMetric.js , if you haven't created that file, do so now. Then add the following to it :

While this file is long or complex it does act as a glue layer between your probe definitions from PMX to the your actual applications use of the metrics. This will allow you to change out metric calculation, implementation, or even library in the future without breaking your entire Express request life cycle. In this file we do the following :

• Pull in a countProbe - Will be covered next.
• Create a module export function that is pretty standard
• We tell our metric that we want to increment it's count by 1.
• Then we tell the life cycle it can continue with next()

count.js

count.js is a wrapper for the actual PMX probe creation that then gets exported. Again it's not complex or long, but breaking your modules out like this will prevent a lot of pain down the line when these metrics inevitably change. If you haven't already create this file, do so and add the following to it :

This will do the following for us :

• Pull in the PMX module
• Instantiate a new probe definition within PMX for us
• Export a defined probe type

PMX provides various metric types, not just counts. In this example we are only using count because it's simple and doesn't require high volumes of requests to show it working. PMX supports the following metric types all of which expose different methods once their type is defined by calling that metric type method on a new probe instance.

• Simple metrics: Values that can be read instantly
  • eg. Monitor variable value
• Counter: Things that increment or decrement
  • eg. Downloads being processed, user connected
• Meter: Things that are measured as events / interval
  • eg. Request per minute for a http server
• Histogram: Keeps a resevoir of statistically relevant values biased towards the last 5 minutes to explore their distribution
• eg. Monitor the mean of execution of a query into database

In our instance we are creating a counter type metric probe that we are simple adding to which expose the inc() and dec() methods that increments and decrements the count of the metric probe respectively. The agg_type param is optionnal, it can be sum, max, min, avg (default) or none. It will impact the way the probe data is aggregated within the Keymetrics/PMX report backend. Use none if this is irrelevant (eg: constant or string value).

Starting Our Project

Right, it's been a bit of a long winded explanation but we are finally there. Lets run our project, issue a npm start in your project directory. You should see something similar to this in your terminal :

Confirming it all works

If nothing errored out, then your little application is now running! You can now check your custom metric status by hitting the PMX metric report API by going to : localhost:9615 

I highly suggest you have a JSON parsing extension setup for your browser because it will give you way more information then you thought you ever needed for you instances. The pieces of this massive metrics dump that we are interested however are under the following JSON path : processes[].pm2_env.axm_monitor

Under that path you will find in the axm_monitor object there is a field name "Request Count" which is the name that was gave our counter probe in count.js , that is our custom metric value. Now to test it out you can go to localhost:3000/health , let that load. Now reload your metrics url, go back the same path and take note that the "value" property has been increased by 1. 

Conclusion

There is a lot of things you can get PM2 and PMX to do for you in the world of reporting, metrics and monitoring. This article hasn't even scratched the surface of what all can be done but hopefully it has peaked your interest enough to have you learn more about what you can do with PM2 to make managing your applications and services easier in everyday dev life.

If you want to poke around more feel free to download the sample code and mess around with all that PM2 and PMX has to offer. In addition I will link to all the resources that I used to create my first metrics and everything that was pulled into this article.

Links

https://expressjs.com/en/starter/hello-world.html

https://github.com/keymetrics/pmx

https://github.com/Unitech/pm2

Welcome to the chronicles of yet another developer blog.

Lets get one thing straight from the get go, I do not know all things, let alone most things. My goal with this blog is to simply document my findings while working in the development world so that some soul may find it useful.  With that in mind I will attempt to be as clear and concise as I can be with examples and reasoning while going over whatever it might that I am going over.

This blog will be mostly about the world of JavaScript but will also contain things that I feel will be relevant to all developers these days. I hope someone will find this project and it will save them time and effort.

Thanks for reading the first entry in what will hopefully be a long journey with mistakes, learning, and building software worth a damn.