Troubleshooting Latency Issues on App Platform

Introduction

Latency in web applications refers to the time a server takes to process a request and send a response.

In this tutorial, you will use PM2, a powerful process manager for Node.js, and its command line tools to troubleshoot latency issues in applications deployed on the DigitalOcean App Platform. Latency can significantly impact user satisfaction and overall application performance, making its management crucial.

This detailed tutorial will guide you through deploying a sample Express.js application on DigitalOcean’s App Platform and leveraging PM2 to monitor and troubleshoot latency issues.

Prerequisites

You will need the following:

• A DigitalOcean account.
• A sample Express.js application deployed on App Platform.
• Artillery installed on your App Platform instance for simulating load testing.
• Pm2 installed on your App Platform Instance for troubleshooting latency issues.

Deploy Your Express.js Application on the App Platform

You can either follow the steps to deploy a Sample Express.js application or use this github repo to create an Express.js sample app on App Platform.

If you use the latter, you will need to fork this GitHub repository to your Github account so that you have a copy of it stored to the cloud. Click the Fork button in the GitHub repository and follow the on-screen instructions.

Now login to DigitalOcean cloud portal and navigate to Create App -> App Platform.

Next, choose your forked Express.js GitHub repository and click on Next.

Follow the on-screen instructions to set your application’s resources, general configuration, and Environment variables, and click Next. Once done, review the overall configuration of the app. Click on Create Resources when done. Once the application is created, it will take some time to deploy on the App Platform.

Once the application is deployed and the build is done. You can check the app’s overview and visit the deployed app in your browser.

Latency Load Testing Using Artillery

Load testing is crucial to understand how your application behaves under significant stress. Tools like Apache JMeter, Artillery, or Loader.io can be used for this purpose.

In this case, you will integrate a simple load test using Artillery.

You will need to navigate to the Console tab to install Artillery and other Node.js packages inside the App platform instance.

To install Artillery use the below command:

npm install -g artillery

Once, Artillery is installed let’s go ahead and install PM2.

Using PM2 for Monitoring and Troubleshooting

PM2 is a powerful tool that not only helps in managing your Node.js applications but also provides detailed insights into their performance.

You can install PM2 on the App Platform instance using the below command:

npm install pm2 -g

First, let’s list the application files in the App Platform instance’s console:

ls

Output
README.md  app.js  node_modules  package-lock.json  package.json

You can start your Express.js application under PM2’s watch:

pm2 start app.js

The output will be as follows:

Output
[PM2] Starting /workspace/app.js in fork_mode (1 instance)
[PM2] Done.
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 0  │ app                │ fork     │ 0    │ online    │ 0%       │ 102.3mb  │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘

Now, let’s use Artillery to load test this application. Let’s create a simple load test script for the App Platform instance.

vi load_test.yaml

And copy and paste the below in the load_test.yaml file:

config:
  target: 'http://your-app-url.com/'
  phases:
    - duration: 300
      arrivalRate: 20
      rampTo: 100

scenarios:
  - flow:
      - think: 1
      - get:
          url: "/"

In the above file replace the http://your-app-url.com/ with your app’s URL.

The example configuration file simulates a load test with 100 virtual users and 20 requests per second for 5 minutes(300s). You have also added a think phase with a duration of 1 second to simulate users thinking before making a request.

To run this load test you will use the below command:

artillery run load_test.yaml

Now, let’s use the pm2 commands below to get a comprehensive health report of the Express.js application environment under load.

Each of the below pm2 commands provides vital information:

• pm2 list - shows all currently running processes.
• pm2 show - displays detailed information about a specific process managed by PM2.
• pm2 dashboard - displays a real-time dashboard.
• pm2 logs - consolidates logs for all processes.
• pm2 report - generates a diagnostic report.

Let’s deep dive into each of these commands:

1. pm2 list - Shows all currently running processes managed by PM2.

pm2 list

output
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 0  │ app                │ fork     │ 10   │ online    │ 6%       │ 120.8mb  │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘

The ↺ in the 4th column indicates the number of restarts.

2. pm2 show command displays useful information about a specific process managed by PM2.

pm2 show app

Output
Describing process with id 0 - name app 
┌───────────────────┬────────────────────────────────────┐
│ status            │ online                             │
│ name              │ app                                │
│ namespace         │ default                            │
│ version           │ 1.0.0                              │
│ restarts          │ 10                                │
│ uptime            │ 62m                                 │
│ script path       │ /workspace/app.js                  │
│ script args       │ N/A                                │
│ error log path    │ /workspace/.pm2/logs/app-error.log │
│ out log path      │ /workspace/.pm2/logs/app-out.log   │
│ pid path          │ /workspace/.pm2/pids/app-0.pid     │
│ interpreter       │ node                               │
│ interpreter args  │ N/A                                │
│ script id         │ 0                                  │
│ exec cwd          │ /workspace                         │
│ exec mode         │ fork_mode                          │
│ node.js version   │ 20.13.0                            │
│ node env          │ production                         │
│ watch & reload    │ ✘                                  │
│ unstable restarts │ 0                                  │
│ created at        │ 2024-05-10T09:28:07.075Z           │
└───────────────────┴────────────────────────────────────┘
 Revision control metadata 
┌──────────────────┬─────────────────────────────────────────────────────────────┐
│ revision control │ git                                                         │
│ remote url       │ https://github.com/digitalocean/sample-nodejs.git           │
│ repository root  │ /workspace                                                  │
│ last update      │ 2024-05-10T09:40:18.972Z                                    │
│ revision         │ 11145279d540077bbfd2f71bb21adf9b549705c8                    │
│ comment          │ Merge pull request #35 from digitalocean/ElanHasson-patch-1 │
│                  │                                                             │
│ branch           │ HEAD                                                        │
└──────────────────┴─────────────────────────────────────────────────────────────┘

 Actions available 
┌────────────────────────┐
│ km:heapdump            │
│ km:cpu:profiling:start │
│ km:cpu:profiling:stop  │
│ km:heap:sampling:start │
│ km:heap:sampling:stop  │
└────────────────────────┘
 Trigger via: pm2 trigger app <action_name>

 Code metrics value 
┌────────────────────────┬───────────────┐
│ Used Heap Size         │ 22.70 MiB     │
│ Heap Usage             │ 46.06 %       │
│ Heap Size              │ 49.28 MiB     │
│ Event Loop Latency p95 │ 2.58 ms       │
│ Event Loop Latency     │ 0.95 ms       │
│ Active handles         │ 252           │
│ Active requests        │ 0             │
│ HTTP                   │ 38.78 req/min │
│ HTTP P95 Latency       │ 946.55 ms     │
│ HTTP Mean Latency      │ 553 ms        │
└────────────────────────┴───────────────┘
 
 Divergent env variables from local env 


 Add your own code metrics: http://bit.ly/code-metrics
 Use `pm2 logs app [--lines 1000]` to display logs
 Use `pm2 env 0` to display environment variables
 Use `pm2 monit` to monitor CPU and Memory usage app

In the above output, the section Code metrics value is what we should be most concerned about while troubleshooting any latency issues and is the key performance metric for the application’s performance.

Let’s understand each metric in more detail:

• Used Heap Size: The Node.js heap is an area of memory used to store objects and data. The heap size is the amount of memory currently used by the Node.js process to store objects and data in the heap. In the above output, the used heap size is 22.70 MiB, which means that the Node.js process uses 22.70 megabytes of memory to store objects and data.

• Heap Usage: The heap usage is the percentage of the total heap size currently used. In the above output, the heap usage is 46.06%, which means that the Node.js process uses 46.06% of the total heap size. This metric can help you understand how much of the available heap memory is being used and can help you identify potential memory issues.

• Heap Size: The heap size is the total size of the heap memory that has been allocated to the Node.js process. In the above output, the heap size is 49.28 MiB, which means that the Node.js process has been allocated 49.28 megabytes of heap memory. This metric can help you understand the amount of memory available to the Node.js process for storing objects and data.

• Event Loop Latency p95: The Node.js event loop is responsible for handling I/O operations, timers, and other asynchronous events. Event loop latency is the time it takes for the event loop to process a single event. The p95 value is the 95th percentile of event loop latency, meaning 95% of events are processed within this time. In the above output, the event loop latency p95 is 2.58 ms, meaning 95% of events are processed within 2.58 milliseconds.

• Event Loop Latency: The event loop latency is the average time it takes for the event loop to process a single event. In the above output, the event loop latency is 0.95 ms, meaning it takes an average of 0.95 milliseconds to process each event. This metric can help you understand the performance of the Node.js event loop and can help you identify potential bottlenecks or issues.

• Active handles: Handles are objects that are used to manage I/O operations, timers, and other asynchronous events. Active handles are handles that are currently being used by the Node.js process. In the above output, there are 252 active handles. This metric can help you understand the number of I/O operations and other asynchronous events currently managed by the Node.js process.

• Active requests: Active requests are HTTP requests currently handled by the Node.js process. In the above output, there are no active requests. This metric can help you understand the number of HTTP requests currently handled by the Node.js process.

• HTTP: The HTTP value is the number of HTTP requests the Node.js process can handle per minute. The Node.js process can handle 38.78 requests per minute in the above output. This metric can help you understand the throughput of the application and can help you identify potential bottlenecks or issues.

• HTTP P95 Latency: The HTTP P95 latency is the 95th percentile of the time for the Node.js process to respond to an HTTP request. In the above output, the HTTP P95 latency is 946.55 ms, which means that 95% of requests are responded to within 946.55 milliseconds.

• HTTP Mean Latency: The HTTP mean latency is the average time it takes for the Node.js process to respond to an HTTP request. In the example output, the HTTP mean latency is 553 ms, meaning it takes an average of 553 milliseconds to respond to each request. These metrics can help you understand the response time of the application and can help you identify potential bottlenecks or issues.

You can better understand the application’s performance and identify potential latency issues by monitoring these metrics.

3. pm2 dashboard - Provides a real-time dashboard for monitoring CPU and memory usage for the application.

output
┌─ Process List ─────────────────────┐┌──  app Logs  ─────────────────────────────────────────────────────────────────────────┐
│[ 0] app      Mem: 122 MB    CPU:6% ││                                                                                       │
│                                    ││                                                                                       │
│                                    ││                                                                                       │
│                                    ││                                                                                       │                                                               │
└────────────────────────────────────┘└───────────────────────────────────────────────────────────────────────────────────────┘
┌─ Custom Metrics ───────────────────┐┌─ Metadata ────────────────────────────────────────────────────────────────────────────┐
│ Used Heap Size          42.18 MiB  ││ App Name              app                                                             │
│ Heap Usage                85.06 %  ││ Namespace             default                                                         │
│ Heap Size               49.28 MiB  ││ Version               1.0.0                                                           │
│ Event Loop Latency p95      1.60   ││ Restarts              10                                                              │
└────────────────────────────────────┘└───────────────────────────────────────────────────────────────────────────────────────┘

left/right: switch boards | up/down/mouse: scroll | Ctrl-C: exit                      To go further check out https://pm2.io/  

This interface helps identify spikes in resource usage that may correlate with latency issues.

4. pm2 logs - Displays logs for all managed applications, which is essential for identifying error patterns and issues in real-time.

pm2 logs app

output
[TAILING] Tailing last 15 lines for [all] processes (change the value with --lines option)
/workspace/.pm2/pm2.log last 15 lines:
PM2        | 2024-05-10T09:33:55: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:33:59: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:33:59: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:33:59: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:34:02: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:34:02: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:34:02: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:34:06: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:34:06: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:34:06: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:34:09: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:34:09: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:34:09: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:34:13: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:34:13: PM2 log: App [app:0] starting in -fork mode-

/workspace/.pm2/logs/app-out.log last 15 lines:
/workspace/.pm2/logs/app-error.log last 15 lines:
PM2      | App [app:0] exited with code [0] via signal [SIGINT]
PM2      | App [app:0] starting in -fork mode-
PM2      | App [app:0] online
PM2      | App [app:0] exited with code [0] via signal [SIGINT]
PM2      | App [app:0] starting in -fork mode-
PM2      | App [app:0] online

These logs provide insights into application behavior, such as startup messages and error reports.

5. pm2 report - Generates a comprehensive status report of the PM2-managed application.

pm2 report app

output
--- PM2 report ----------------------------------------------------------------
Date                 : Fri May 10 2024 09:31:24 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.3.1
node version         : 20.12.2
node path            : /layers/heroku_nodejs-engine/nodejs/bin/pm2
argv                 : /layers/heroku_nodejs-engine/nodejs/bin/node,/layers/heroku_nodejs-engine/nodejs/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : undefined
uid                  : 1000
gid                  : 1000
uptime               : 3min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.3.1
node version         : 20.12.2
node path            : /layers/heroku_nodejs-engine/nodejs/bin/pm2
argv                 : /layers/heroku_nodejs-engine/nodejs/bin/node,/layers/heroku_nodejs-engine/nodejs/bin/pm2,report,app
argv0                : node
user                 : undefined
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : unknown
cpus nb              : 16
freemem              : 296103936
totalmem             : 536870912
home                 : /workspace
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 0  │ app                │ fork     │ 54   │ online    │ 0%       │ 120.1mb  │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/workspace/.pm2/pm2.log last 20 lines:
PM2        | 2024-05-10T09:31:00: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:31:00: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:31:03: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:31:03: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:31:03: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:31:07: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:31:07: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:31:07: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:31:11: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:31:11: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:31:11: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:31:14: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:31:14: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:31:14: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:31:18: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:31:18: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:31:18: PM2 log: App [app:0] online
PM2        | 2024-05-10T09:31:21: PM2 log: App [app:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-05-10T09:31:21: PM2 log: App [app:0] starting in -fork mode-
PM2        | 2024-05-10T09:31:21: PM2 log: App [app:0] online

This report helps in understanding the health and behavior of the application over time.

Accessing the PM2 Dashboard

The PM2 dashboard provides a web-based interface to monitor your applications. Access it by running:

pm2 plus

Output
[PM2 I/O] Using non-browser authentication.
[PM2 I/O] Do you have a pm2.io account? (y/n) n
[PM2 I/O] No problem ! We just need few informations to create your account
[PM2 I/O] Please choose an email : xxxxxx@gmail.com
[PM2 I/O] Please choose a password : **********

[PM2 I/O] Do you accept the terms and privacy policy (https://pm2.io/legals/terms_conditions.pdf) ?  (y/n) y


[PM2 I/O] Successfully authenticated
[PM2 I/O] Successfully validated
[PM2 I/O] Successfully created the bucket
[PM2 I/O] Using: Public key: yiixesp0wfnzi03 | Private key: o0pdb8rtk52l6ou | Machine name: sample-expressjs-7ffbb5fcd9-wsr9g-8d7e
[+] PM2+ activated!
[PM2 I/O] Successfully connected to bucket PM2 Plus Monitoring
[PM2 I/O] You can use the web interface over there: https://app.pm2.io/#/bucket/6642fb21221ecc22db0045e2

You will need to sign in, and it’s free to use. You can copy the link to the web interface to open the monitoring dashboard in your browser.

Conclusion

In this tutorial, you learned to use PM2 to effectively monitor and troubleshoot latency issues in an Express.js application on the DigitalOcean App Platform. By regularly analyzing the detailed metrics and logs these tools provide, you can maintain optimal application performance and proactively address any issues.