NSD -monitor on Linux

NSD is a great tool to find the cause of a crash or hang ( if you are familiar with how to do the analysis ).
But sometimes the log file created by NSD does not include enough information to find the root cause of a problem.

In this case, you can start NSD in monitor mode.

If you are on Windows you would

  • stop the Domino server
  • from inside the Domino data directory invoke the command nsd -monitor
  • start the Domino server ( as normal application )

After you have reproduced the problem

  • type detach to detach NSD from running processes
  • quit NSD
  • restart your Domino server ( as a service )

I tried to do this on LINUX ( and inside a Docker container ) but the NSD -monitor command always returned an error

[notes@serv07 notesdata]$ /opt/hcl/domino/bin/nsd -monitor

INFO: NSD Monitor : Started
ERROR: NSD Monitor : No Processes Found or Specified
/opt/hcl/domino/notes/latest/linux/nsd.sh: line 8020: 11040 User defined signal 1 "$Nsd" "$@" -wrapper

I opened a case with HCL support and got a reply.

Here is what you have to do.

If you are running Domino 12 as participant of the HCL Domino Early Access program, you have to get into your Domino V12 container first

[root@docker ~]# docker exec -it container-id bash

The next steps are the same for Docker and Non-Docker environments

Identify the running Domino processes. Switch to /local/notesdata and issue the command

ps -fu $USER

You will get something like that

[notes@serv07 notesdata]$ ps -fu $USER
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
notes 2636 0.0 0.0 19200 2332 pts/1 Ss 11:58 0:00 bash
notes 2651 0.0 0.0 51808 1928 pts/1 R+ 11:59 0:00 _ ps -fu
notes 1 0.0 0.0 11988 1720 pts/0 Ss+ 11:52 0:00 /bin/bash /local/start.sh
notes 843 0.0 0.0 11988 1032 pts/0 S+ 11:52 0:00 /bin/bash /local/start.sh
notes 895 0.4 1.1 1466332 92908 pts/0 Sl+ 11:52 0:01 _ /opt/hcl/domino/notes/latest/linux/server
notes 903 0.0 0.3 339272 29448 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/logasio NOTESLOGGER reserved
notes 913 0.2 0.6 1374008 48860 pts/0 Sl+ 11:52 0:01 _ /opt/hcl/domino/notes/latest/linux/event
notes 1732 0.0 0.4 417372 38232 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/replica
notes 1733 0.1 0.5 624500 42024 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/router
notes 1734 0.0 0.4 486132 39296 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/update
notes 1735 0.0 0.5 422108 41700 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/amgr -s
notes 1975 0.0 0.4 421952 36988 pts/0 Sl+ 11:52 0:00 | _ /opt/hcl/domino/notes/latest/linux/amgr -e 1
notes 1737 0.1 0.5 619644 43332 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/adminp
notes 1738 0.0 0.4 487588 35024 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/daosmgr
notes 2424 0.0 0.5 417392 42164 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/cldbdir
notes 2595 0.2 0.5 890580 47804 pts/0 Sl+ 11:52 0:01 _ /opt/hcl/domino/notes/latest/linux/clrepl

Write down the PIDs of interest. In my case it was 1735 and 1975 for the Agent Manager (amgr)

Now you can start NSD in monitor mode with

[notes@serv07 notesdata]$ /opt/hcl/domino/bin/nsd -monitor -pidlist 1735,1975

You can now start to reproduce the problem. In addition to the well known nsd … .log files in IBM_TECHNICAL_SUPPORT you will also find additional information in the nsd.notes folder


Get Linux Version Information

To get some basic information about the installed Linux version you can use uname -a.

This will give you something like

Linux serv03.fritz.box 3.10.0-957.1.3.el7.x86_64 #1 SMP 
Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Well, we get the kernel version and the architecture.

rpm -qia ‘release gives much more information

Name        : nodesource-release
Version     : el7
Release     : 1
Architecture: noarch
Install Date: Thu 29 Nov 2018 09:01:53 AM CET
Group       : System Environment/Base
Size        : 3191
License     : MIT
Signature   : RSA/SHA256, Thu 04 Jan 2018 09:42:19 PM CET, Key ID 5ddbe8d434fa74dd
Source RPM  : nodesource-release-el7-1.src.rpm
Build Date  : Thu 04 Jan 2018 09:40:35 PM CET
Build Host  : luthien
Relocations : (not relocatable)
URL         : http://rpm.nodesource.com/pub_8.x/
Summary     : N|Solid repository configuration
Description :
This package contains the NodeSource NodeJS repository
GPG key as well as configuration for yum.
Name        : redhat-release-server
Version     : 7.4
Release     : 18.el7
Architecture: x86_64
Install Date: Tue 27 Nov 2018 01:50:07 PM CET
Group       : System Environment/Base
Size        : 43146
License     : GPLv2
Signature   : RSA/SHA256, Wed 28 Jun 2017 08:59:19 PM CEST, Key ID 199e2f91fd431d51
Source RPM  : redhat-release-server-7.4-18.el7.src.rpm
Build Date  : Wed 28 Jun 2017 08:36:32 PM CEST
Build Host  : x86-037.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
Summary     : Red Hat Enterprise Linux Server release file
Description :
Red Hat Enterprise Linux Server release files
Name        : epel-release
Version     : 7
Release     : 11
Architecture: noarch
Install Date: Wed 28 Nov 2018 07:54:48 AM CET
Group       : System Environment/Base
Size        : 24834
License     : GPLv2
Signature   : RSA/SHA256, Mon 02 Oct 2017 07:52:02 PM CEST, Key ID 6a2faea2352c64e5
Source RPM  : epel-release-7-11.src.rpm
Build Date  : Mon 02 Oct 2017 07:45:58 PM CEST
Build Host  : buildvm-ppc64le-05.ppc.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : http://download.fedoraproject.org/pub/epel
Summary     : Extra Packages for Enterprise Linux repository configuration
Description :
This package contains the Extra Packages for Enterprise Linux (EPEL) repository
GPG key as well as configuration for yum.


java.lang.UnsatisfiedLinkError: lotus/domino/local/Database.NcreateDQuery()J

UPDATE 05-FEB-2019: Issue is being tracked under SPR # VRARB94KAQ

When executing db.createDominoQuery(); in a Java agent on the server, I see the following error message on the Domino console:

te amgr run "ec11.nsf" 'dql.java'
[021963:000035-00007F2BE8DFD700] 02/03/2019 05:41:29 AM AMgr: Start executing agent 'dql.java' in 'ec11.nsf'
[021963:000037-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent printing: Version: Release 10.0.1 November 29, 2018
[021963:000037-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent printing: Db Title: singultus's Directory
[021963:000038-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent error: Exception in thread "AgentThread: JavaAgent"
[021963:000039-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent error: java.lang.UnsatisfiedLinkError: lotus/domino/local/Database.NcreateDQuery()J
[021963:000041-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent error: at lotus.domino.local.Database.createDominoQuery(Unknown Source)
[021963:000043-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent error: at JavaAgent.NotesMain(JavaAgent.java:19)
[021963:000045-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent error: at lotus.domino.AgentBase.runNotes(Unknown Source)
[021963:000047-00007F2BE816F700] 02/03/2019 05:41:29 AM Agent Manager: Agent error: at lotus.domino.NotesThread.run(Unknown Source)
[021963:000035-00007F2BE8DFD700] 02/03/2019 05:41:29 AM AMgr: Agent 'dql.java' in 'ec11.nsf' completed execution

According to John Curtis (HCL) “… We haven’t seen this to date.”.

I have tested on different OS.

serv02: 
Red Hat Enterprise Linux Server release 7.4 (Maipo)
Linux serv02.fritz.box 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
IBM Domino (r) Server (64 Bit) (Release 10.0.1 for Linux/64) 02/03/2019 05:48:40 AM
serv02 has been upgraded from V10.0
serv03:
CentOS Linux release 7.6.1810 (Core)
Linux serv03.fritz.box 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
IBM Domino (r) Server (64 Bit) (Release 10.0.1 for Linux/64) 02/03/2019 05:49:15 AM (Community server)
serv03 is "fresh" install and not upgraded from earlier Domino versions.

Here is the code I used

import lotus.domino.AgentBase;
import lotus.domino.Database;
import lotus.domino.DominoQuery;
import lotus.domino.Session;

public class JavaAgent extends AgentBase {

	public void NotesMain() {

		try {
			Session session = getSession();
			System.out.println("Version: " + session.getNotesVersion());
			
			Database db = null;
			db = session.getDatabase(null, "names.nsf");
			System.out.println("Db Title: " + db.getTitle());
			
			DominoQuery dql = null;
			dql = db.createDominoQuery();

			dql.recycle();
			db.recycle();
			
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

As it works on Domino 10.0.1 on WINDOWS, I investigated nlsxbe.dll and liblsxbe.so

nlsxbe.dll contains the function in question

Java_lotus_domino_local_Database_NcreateDQuery 0x0000000180017cd0 0x00017cd0 732 (0x2dc) nlsxbe.dll W:\Domino\nlsxbe.dll Exported Function

liblsxbe.so is missing the function. At least, it is not exported.

nm -D liblsxbe.so | grep createD

0000000000096a03 T Java_lotus_domino_local_Database_NcreateDocColl
0000000000095587 T Java_lotus_domino_local_Database_NcreateDocument
000000000012bf7d T Java_lotus_domino_local_DateRange_NrecreateDateRange
00000000000e33e1 T Java_lotus_domino_local_DateTime_NrecreateDateTime
0000000000110ad2 T Java_lotus_domino_local_DbDirectory_NcreateDatabase
00000000000d5d6d T Java_lotus_domino_local_Session_NcreateDateRange
00000000000da091 T Java_lotus_domino_local_Session_NcreateDateTime
00000000000d6bd1 T Java_lotus_domino_local_Session_NcreateDxlExporter
00000000000d6d71 T Java_lotus_domino_local_Session_NcreateDxlImporter
00000000000957a0 T Java_lotus_notes_Database_NcreateDocument
0000000000110d7e T Java_lotus_notes_DbDirectory_NcreateDatabase
00000000000d5f40 T Java_lotus_notes_Session_NcreateDateRange
00000000000da360 T Java_lotus_notes_Session_NcreateDateTime
U _ZN11XmlDocument14createDocumentEv
U _ZN11XmlDocument22cr
eateDocumentFragmentEv

The version and date seem to be OK.

ls -al liblsxbe.so
-rwxr-xr-x. 1 root root 14510400 Nov 29 07:48 liblsxbe.so

Perhaps someone else can confirm this behaviour. I have already created case #TS001863705 with HCL.


Start node.js application using systemd on RHEL 7

In a test or development environment we can start a node.js application with npm start . But this is not, what we want to do in production.

We need to start the application as soon as the server is up and running.

Here is a quick and easy way to achieve this goal.

Create a new file in /etc/systemd/system ( domino-db.service for example )

[Unit]
Description=domino-node-list sample

[Service]
ExecStart=/git/domino-node-list/app.js
Restart=always
User=root
Group=root
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/git/domino-node-list/

[Install]
WantedBy=multi-user.target

Modify ExecStart and WorkingDirectory to point to the correct path in your installation.

Save domino-db.service and set execution rights to 744.

Check app.js for proper execution right (744).

Edit app.js and add

#!/usr/bin/env node

as the first line in the code.

Enable the service

systemctl enable domino-db

Created symlink from /etc/systemd/system/multi-user.target.wants/domino-db.service to /etc/systemd/system/domino-db.service.

Check with

systemctl status domino-db

● domino-db.service - domino-node-list sample
   Loaded: loaded (/etc/systemd/system/domino-db.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2018-12-30 10:24:04 CET; 4min 15s ago
 Main PID: 7163 (node)
    Tasks: 7
   CGroup: /system.slice/domino-db.service
           └─7163 node /git/domino-node-list/app.js

Dec 30 10:24:04 serv02.fritz.box systemd[1]: Started domino-node-list sample.
Dec 30 10:24:04 serv02.fritz.box systemd[1]: Starting domino-node-list sample...
Dec 30 10:24:05 serv02.fritz.box app.js[7163]: Example app listening at http://:::3000

You can now start and stop the service at any time using

systemctl start domino-db
systemctl stop domino-db

Ubuntu – reset root password

I wanted to update a virtual machine from Ubunto 9.10 to 10.04 but I could not recall the root password. None of my standard passwords worked.
Google to the rescue, I found a way to reset the root password.

1) Login as non-root user
2) sudo su –
3) type in the password for the non-root user
4) passwd
5) set new root password

Now updating to Ubuntu 10.04


ESXi 3.5, SSH And FTP

VMware ESXi ships with the ability to run SSH, but the feature is disabled by default and not supported. Even FTP is disabled so you cannot upload images to the server for installation.

To enable SSH and FTP, you have to access the console. Obviously it is not intended by VMware to access the console as you would do in a regular Linux system. But, as always, there is a little magic to accomplish the aim.

At the console of the VMware ESXi host, press ALT+F1 to access the console window. Enter the word unsupported in the console and then press Enter. You will not see the text you type in.

If you typed in unsupported correctly, you can enter the password for the root login. You should then see the prompt ~ #.

To enable SSH and/or FTP, edit /etc/inetd.conf and remove the comments on #SSH and #FTP. Save the file and restart the server or inetd process.

Now you can access the host via SSH and/or FTP.