Starting Express.js applications with PM2

I recently ran into an issue with PM2. I have created a “helloworld” applications using Express.js.

[root@nodejs projects]# express helloworld
warning: the default view engine will not be jade in future releases
warning: use --view=jade' or--help' for additional options
create : helloworld/
create : helloworld/public/
create : helloworld/public/javascripts/
create : helloworld/public/images/
create : helloworld/public/stylesheets/
create : helloworld/public/stylesheets/style.css
create : helloworld/routes/
create : helloworld/routes/index.js
create : helloworld/routes/users.js
create : helloworld/views/
create : helloworld/views/error.jade
create : helloworld/views/index.jade
create : helloworld/views/layout.jade
create : helloworld/app.js
create : helloworld/package.json
create : helloworld/bin/
create : helloworld/bin/www
change directory:
$ cd helloworld
install dependencies:
$ npm install
run the app:
$ DEBUG=helloworld:* npm start

The application started without issues with npm start.

Next I started the application with PM2

[root@nodejs helloworld]# pm2 start app.js

PM2 displayed the application as “online”, but it constantly restarted and was not accessible.

I found the solution here

[root@nodejs helloworld]# pm2 start bin/www

did the trick for me.


nginx + node.js + CentOS 7 = 502 Bad Gateway

I have setup a new Node.js / Express development environment on a CentOS 7 VM. I ‘ll describe the details in another post later.

To test my setp, I created a new Express application “helloworld”. The application listens on port 3000 and I was able to connect to the application using a browser.

Next, I configured NGINX as reverse proxy to use port 80 to access the helloworld application.

But I got an error

I checked the logs

[root@nodejs ~]# cat /var/log/audit/audit.log | grep nginx | grep denied

and got

type=AVC msg=audit(1546783734.750:239): avc:  denied  { name_connect } for  pid=11084 comm="nginx" dest=3000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket permissive=0

My best guess was SELinux.I checked, if SELinux was enabled.

[root@nodejs ~]# sestatus

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31

Next I checked the settings for httpd.

[root@nodejs ~]# getsebool -a | grep httpd
httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off

So, httpd_can_network_connect was set to “Off”. This blocks the connection from the reverse proxy to the node.js application. As a result, you get the 502 Bad gateway error.

To enable the setting, execute the following command from the shell.

[root@nodejs ~]# setsebool -P httpd_can_network_connect on

You do not need to reboot the machine or SELinux.


Finding And Fixing Node.js Memory Leaks: A Practical Guide


Fixing memory leaks may not be not the shiniest skill on a CV, but when things go wrong on production, it’s better to be prepared!
After reading this article, you’ll be able to monitor, understand, and debug the memory consumption of a Node.js application.


Kévin Maschtaler

https://marmelab.com/blog/2018/04/03/how-to-track-and-fix-memory-leak-with-nodejs.html

In addition to this very good article , here are a couple of tipps how to enable remote debugging.

node –inspect=192.168.178.133:9229 yourapp.js

This will bind the debugger to a different IP address:port. Otherwise only LOCALHOST will be available

If you are using PM2

pm2 start –node-args=”–inspect=192.168.178.133:9229″ yourapp.js

Then open Chrome browser and navigate to chrome://inspect

Check “Discover network targets” and click “Configure”

Your application is ready for inspect


Start node.js application using systemd on RHEL 7

In a test or development environment we can start a node.js application with npm start . But this is not, what we want to do in production.

We need to start the application as soon as the server is up and running.

Here is a quick and easy way to achieve this goal.

Create a new file in /etc/systemd/system ( domino-db.service for example )

[Unit]
Description=domino-node-list sample

[Service]
ExecStart=/git/domino-node-list/app.js
Restart=always
User=root
Group=root
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/git/domino-node-list/

[Install]
WantedBy=multi-user.target

Modify ExecStart and WorkingDirectory to point to the correct path in your installation.

Save domino-db.service and set execution rights to 744.

Check app.js for proper execution right (744).

Edit app.js and add

#!/usr/bin/env node

as the first line in the code.

Enable the service

systemctl enable domino-db

Created symlink from /etc/systemd/system/multi-user.target.wants/domino-db.service to /etc/systemd/system/domino-db.service.

Check with

systemctl status domino-db

● domino-db.service - domino-node-list sample
   Loaded: loaded (/etc/systemd/system/domino-db.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2018-12-30 10:24:04 CET; 4min 15s ago
 Main PID: 7163 (node)
    Tasks: 7
   CGroup: /system.slice/domino-db.service
           └─7163 node /git/domino-node-list/app.js

Dec 30 10:24:04 serv02.fritz.box systemd[1]: Started domino-node-list sample.
Dec 30 10:24:04 serv02.fritz.box systemd[1]: Starting domino-node-list sample...
Dec 30 10:24:05 serv02.fritz.box app.js[7163]: Example app listening at http://:::3000

You can now start and stop the service at any time using

systemctl start domino-db
systemctl stop domino-db

[Solved] – PROTON_SSL=1

This is a follow up to my recent post https://www.eknori.de/2018-12-28/problem-using-proton-with-proton_ssl1-enabled/

Due to my misunderstanding and also a lack in documentation, I was not able to get PROTON with PROTON_SSL=1 and PROTON_AUTHENTICATION=client_certs running.

But thanks to our great community and a little help from Jan Krejcárek (SUTOL), I was finally able to solve the puzzle.

I must admit that I am not an expert in this field. So please bare with me if things are obvious for you. They are (not yet) for me.

My understanding of using TLS/SSL with PROTON was that I could use my existing Let’s Encrypt certificate to communicate via HTTPS instead of HTTP with my server URL.

But this is not the case. TLS/SSL encryption only enables a secured communication between the domino.db module and the PROTON addin sitting on the Domino server.

Here is an image.

The second thing that I got wrong was the fact that the existing certificate and it’s CA would be enough to enable the secured communication. In my first test, I had set PROTON_AUTHENTICATION=anonymous.

So why bother with client certificates if they are not needed at this point?

I followed Jan’s advice to try with a new generated self-signed certificate. The AppDevPack contains shell scripts to create some sample certs and also lets you create a keyring file that can be used with PROTON.

I have modified the make_keyring.sh file a bit because it complained about missing environment variables.

# added UKR, 12/2018
export NOTESDATA=/local/notesdata
export NOTESBIN=/opt/ibm/domino/notes/10000000/linux

I then created the certs and keyring and copied them to the proper location.

The original source code has used

// proton config
const serverConfig = {
    "hostName": "serv02.fritz.box",
    "connection": {
        "port": 3002
    }
}

to define the serverConfig. This must be enhanced, if you are going the secure way.

I created a new module using Jan’s code

const fs = require('fs');
const path = require('path');

/**
 * Internal functions that reads a content of a file to a buffer.
 * @param {string} fileName 
 */
const readFile = fileName => {
        return fs.readFileSync(path.resolve(fileName));
}

/**
 * (c) Jan Krejcárek
 * Creates an object in a structure required for the DominoDB module
 * to initialize o connection to the Domino server.
 * @param {string} serverHostName Host name of the Domino server
 * @param {string} port TCP port number where a Proton task listens for connection requests
 * @param {string} rootCertificatePath Path to the certificate used to establish a TLS connection
 * @param {string} clientCertificatePath Path to the application certificate used to authenticate the application
 * @param {string} clienKeyPath Path to the private key of the application
 */
const config = (serverHostName, port, rootCertificatePath, clientCertificatePath, clienKeyPath) => {
    const rootCertificate = readFile(rootCertificatePath);
    const clientCertificate = readFile(clientCertificatePath);
    const clientKey = readFile(clienKeyPath);
    
    const serverConfig = {
            hostName: serverHostName,
            connection: {
                    port: port,
                    secure: true
            },
            credentials: {
                    rootCertificate,
                    clientCertificate,
                    clientKey
            }    
    };

    return serverConfig;
};

module.exports = config;

All parts in the credentials section are mandatory. Even if you use PROTON_AUTHENTICATION=anonymous with PROTON_SSL=1, you must have certificates and keys for the client as well.

And we can now use this module in app.js

const protonConfig = require('./protonConfigSSL.js');
const serverConfig = protonConfig("serv02.fritz.box", "3002", "./certs/proton-self/ca.crt", "./certs/proton-self/app1.crt", "./certs/proton-self/app1.key");

My PROTON configuration is

[029971:000009-00007F78D8293700] PROTON_AUTHENTICATION=client_cert
[029971:000009-00007F78D8293700] PROTON_KEYFILE=proton-self.kyr
[029971:000009-00007F78D8293700] PROTON_LISTEN_ADDRESS=serv02.fritz.box
[029971:000009-00007F78D8293700] PROTON_LISTEN_PORT=3002
[029971:000009-00007F78D8293700] PROTON_SSL=1
[029971:000009-00007F78D8293700] PROTON_TRACE_REQUEST=0
[029971:000009-00007F78D8293700] PROTON_TRACE_SESSION=0

And after restarting PROTON ( restart task proton ) and starting my application ( npm start ), I was able to open hp.nsf in the browser.

I also found an interesting article by Sven Hasselbach about how to protect PROTON keys.

I hope that this will help others starting with this stuff to save some time.

Recommended reading:

Heiko Voigt: DominoDB and a big NO-NO !

Sven Hasselbach ( response to Heiko’s post ): DominoDB and a big NO-NO?