NotesException: Object has been removed or recycled

A few days ago, a customer reported that one of our applications was not working properly and sent the associated logs.

It was very quickly clear under what conditions the error occurred. So it was reproducible, which is the first important step to troubleshooting.

Nevertheless, it took some time to locate the cause. Finally, I was able to find the “error”. To send it ahead. The “error” is not limited to a specific Domino version. The problem also occurs under Domino 9.0.1 FP10.

To illustrate the problem I wrote a small Java agent.
When we work with Java in the Domino environment, one of the best practices is to recycle Domino objects. We do this in countless places in our applications, and it hasn’t really caused a problem yet.

Not so in this particular case. Let’s take a look at the code of the agent.

import lotus.domino.*;

public class JavaAgent extends AgentBase {

	Database db = null;

	public void NotesMain() {

		DocumentCollection col = null;
		Document doc = null;

		try {
			Session session = getSession();
			db = session.getCurrentDatabase();
			col = db.getAllDocuments();
			
			doc = db.getDocumentByUNID(col.getFirstDocument().getUniversalID());
			
			if (null != doc) {
				System.out.println("MAIN: " + doc.getItemValueString("Subject"));
				foo(doc);
				System.out.println("MAIN_2: " + doc.getItemValueString("Subject"));
			}

		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	public void foo(Document doc) {
		try {

			System.out.println("FOO: " + doc.getItemValueString("Subject"));
			bar(doc.getUniversalID());

		} catch (NotesException e) {
			e.printStackTrace();
		}
	}

	public void bar(String unid) {
		Document doc2 = null;

		try {
			doc2 = db.getDocumentByUNID(unid);
			System.out.println("BAR: " + doc2.getItemValueString("Subject"));

		} catch (Exception e) {
			e.printStackTrace();
		} finally {

			try {
				doc2.recycle();
			} catch (NotesException e) {
				// fail silent
			}
		}
	}
}

My test database contains a single document. The document has an item “Subject”.
In the MAIN method of the agent we fetch the document and get the UNID.
Now we get the document via this UNID and assign it to the LOCAL variable doc.

On the Domino console we display the content of the item Subject.
Then we branch to the method FOO, and pass the object doc as parameter.
Here, too, we display the contents of Subject.

Now we branch to another method BAR. Since the method is called from many places in the real application, and the document is not available everywhere, we pass the UNID and assign the document found via db.getDocumentByUNID to the LOCAL variable doc2.

Again, we display the contents of Subject. So far everything is OK. When leaving BAR, the LOCAL variable doc2 is recycled and we return to MAIN, where we again display the contents of the item Subject.

And exactly at this point we get an error message. (line 22)

AMgr: Start executing agent 'test' in 'java.nsf' 
Agent Manager: Agent printing: MAIN: I am a sampe document 
Agent Manager: Agent printing: FOO: I am a sampe document 
Agent Manager: Agent printing: BAR: I am a sampe document 
Agent Manager: Agent  error: NotesException: Object has been removed or recycled 
Agent Manager: Agent  error:   at lotus.domino.local.NotesBase.CheckObject(Unknown Source) 
Agent Manager: Agent  error:   at lotus.domino.local.Document.getItemValueString(Unknown Source) 
Agent Manager: Agent  error:   at JavaAgent.NotesMain(JavaAgent.java:22) 
Agent Manager: Agent  error:   at lotus.domino.AgentBase.runNotes(Unknown Source) 
Agent Manager: Agent  error:   at lotus.domino.NotesThread.run(Unknown Source) 
AMgr: Agent 'test' in 'java.nsf' completed execution

If we remove the doc2.recycle() in the finally block of the BAR method, then the agent works as expected.

Looks like a db.getDocumentByUNID() references to the same cAPI backend Object and the object is discarded when it is recycled anywhere in the code flow, regardless if it is assigned to different LOCAL variables.

Not sure, if this is a BUG in Domino, or if it is considered as work as designed. Will open a case with HCL support.

I am totally baffled that no one has encountered this problem yet.
I have also always been under the (obviously) mistaken assumption that LOCAL variables are treated as such.


Java agents with imported .jar files

Many of us Notes developers know the problem. You have developed a Java agent that contains methods from imported .jar files in addition to the actual code.
When the agent is executed on a HCL Domino server, it works for a while ( sometimes longer, sometimes shorter ), but then causes problems due to memory leaks.
The problem lies in the architecture of Domino. The imported .jar files are reloaded every time the agent is started and over time they use more and more RAM.

The workaround was to dump the .jar files to the server’s file system. This works, but always causes problems where you as a developer do not always have access to the server.
Or there is already a .jar in the file directory; but not in the required version. Copying your own .jar into the file system can then lead to undesirable side effects.

I had therefore created a case at HCL to get a fix for the problem. The problem was tracked under SPR # BHUY8PRMKK. Yesterday I received an update from support:

The Product Development Team had developed a possible fix that mitigates the issue but the fix was of medium risk and it was decided not to submit the fix. As an alternative they have come up with a workaround. Set the following notes.ini, this will cache the agentloader objects and reuse them.

EnableJavaAgentCache=2

I have set the parameter on my test system. At the moment it looks like it solves the problem. I will continue testing.

Many thanks to HCL Development & HCL Support ( especially to Abhaysingh Shirke ).


NSD -monitor on Linux

NSD is a great tool to find the cause of a crash or hang ( if you are familiar with how to do the analysis ).
But sometimes the log file created by NSD does not include enough information to find the root cause of a problem.

In this case, you can start NSD in monitor mode.

If you are on Windows you would

  • stop the Domino server
  • from inside the Domino data directory invoke the command nsd -monitor
  • start the Domino server ( as normal application )

After you have reproduced the problem

  • type detach to detach NSD from running processes
  • quit NSD
  • restart your Domino server ( as a service )

I tried to do this on LINUX ( and inside a Docker container ) but the NSD -monitor command always returned an error

[notes@serv07 notesdata]$ /opt/hcl/domino/bin/nsd -monitor

INFO: NSD Monitor : Started
ERROR: NSD Monitor : No Processes Found or Specified
/opt/hcl/domino/notes/latest/linux/nsd.sh: line 8020: 11040 User defined signal 1 "$Nsd" "$@" -wrapper

I opened a case with HCL support and got a reply.

Here is what you have to do.

If you are running Domino 12 as participant of the HCL Domino Early Access program, you have to get into your Domino V12 container first

[root@docker ~]# docker exec -it container-id bash

The next steps are the same for Docker and Non-Docker environments

Identify the running Domino processes. Switch to /local/notesdata and issue the command

ps -fu $USER

You will get something like that

[notes@serv07 notesdata]$ ps -fu $USER
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
notes 2636 0.0 0.0 19200 2332 pts/1 Ss 11:58 0:00 bash
notes 2651 0.0 0.0 51808 1928 pts/1 R+ 11:59 0:00 _ ps -fu
notes 1 0.0 0.0 11988 1720 pts/0 Ss+ 11:52 0:00 /bin/bash /local/start.sh
notes 843 0.0 0.0 11988 1032 pts/0 S+ 11:52 0:00 /bin/bash /local/start.sh
notes 895 0.4 1.1 1466332 92908 pts/0 Sl+ 11:52 0:01 _ /opt/hcl/domino/notes/latest/linux/server
notes 903 0.0 0.3 339272 29448 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/logasio NOTESLOGGER reserved
notes 913 0.2 0.6 1374008 48860 pts/0 Sl+ 11:52 0:01 _ /opt/hcl/domino/notes/latest/linux/event
notes 1732 0.0 0.4 417372 38232 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/replica
notes 1733 0.1 0.5 624500 42024 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/router
notes 1734 0.0 0.4 486132 39296 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/update
notes 1735 0.0 0.5 422108 41700 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/amgr -s
notes 1975 0.0 0.4 421952 36988 pts/0 Sl+ 11:52 0:00 | _ /opt/hcl/domino/notes/latest/linux/amgr -e 1
notes 1737 0.1 0.5 619644 43332 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/adminp
notes 1738 0.0 0.4 487588 35024 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/daosmgr
notes 2424 0.0 0.5 417392 42164 pts/0 Sl+ 11:52 0:00 _ /opt/hcl/domino/notes/latest/linux/cldbdir
notes 2595 0.2 0.5 890580 47804 pts/0 Sl+ 11:52 0:01 _ /opt/hcl/domino/notes/latest/linux/clrepl

Write down the PIDs of interest. In my case it was 1735 and 1975 for the Agent Manager (amgr)

Now you can start NSD in monitor mode with

[notes@serv07 notesdata]$ /opt/hcl/domino/bin/nsd -monitor -pidlist 1735,1975

You can now start to reproduce the problem. In addition to the well known nsd … .log files in IBM_TECHNICAL_SUPPORT you will also find additional information in the nsd.notes folder


Update HTTPPassword item in names.nsf (backend)

Yesterday, we ran into an issue with the HTTP Password in the person record in names.nsf.

The problem occured after we upgraded the customers Domino server from V 9.0.1 to V11.0.1FP1.

The customer has some backend processes installed that let them delegate the process of register, update and delete users and groups to different departments. One part of the process is a piece of code that sets the HTTP password in the person record.

The issue was that the password was stored in clear text after upgrading the server. I looked into the design and could spot the root of the issue.

In pubnames_9.ntf, the HTTPPassword item has an input translation formula that encodes the password.

In V11 of the pubnames.ntf, the HTTPPassword item is missing and so is the input translation formula. The password encoding has been moved to the “Enter Password” button.

As a condequence, if you set the password in a backend agent, the String is not encoded and visible in clear text to others.

The fix is simple. We changed our agent code from

...
doc.HTTPPassword = pwdDoc.getItemValue("pwd").text
...

to

...
Dim result As Variant
result = Evaluate(|@Password("|+ pwddoc.getItemValue("pwd").text + |")|)
doc.HTTPPassword.result(0)
...
call doc.save(true, false)

to encode the password. This tipp might be useful, if you have similar processes implemented.

I have not looked into the design of Domino 10. But chances are that HCL has changed the design also in this release.


[Solved] – PROTON_SSL=1

This is a follow up to my recent post https://www.eknori.de/2018-12-28/problem-using-proton-with-proton_ssl1-enabled/

Due to my misunderstanding and also a lack in documentation, I was not able to get PROTON with PROTON_SSL=1 and PROTON_AUTHENTICATION=client_certs running.

But thanks to our great community and a little help from Jan Krejcárek (SUTOL), I was finally able to solve the puzzle.

I must admit that I am not an expert in this field. So please bare with me if things are obvious for you. They are (not yet) for me.

My understanding of using TLS/SSL with PROTON was that I could use my existing Let’s Encrypt certificate to communicate via HTTPS instead of HTTP with my server URL.

But this is not the case. TLS/SSL encryption only enables a secured communication between the domino.db module and the PROTON addin sitting on the Domino server.

Here is an image.

The second thing that I got wrong was the fact that the existing certificate and it’s CA would be enough to enable the secured communication. In my first test, I had set PROTON_AUTHENTICATION=anonymous.

So why bother with client certificates if they are not needed at this point?

I followed Jan’s advice to try with a new generated self-signed certificate. The AppDevPack contains shell scripts to create some sample certs and also lets you create a keyring file that can be used with PROTON.

I have modified the make_keyring.sh file a bit because it complained about missing environment variables.

# added UKR, 12/2018
export NOTESDATA=/local/notesdata
export NOTESBIN=/opt/ibm/domino/notes/10000000/linux

I then created the certs and keyring and copied them to the proper location.

The original source code has used

// proton config
const serverConfig = {
    "hostName": "serv02.fritz.box",
    "connection": {
        "port": 3002
    }
}

to define the serverConfig. This must be enhanced, if you are going the secure way.

I created a new module using Jan’s code

const fs = require('fs');
const path = require('path');

/**
 * Internal functions that reads a content of a file to a buffer.
 * @param {string} fileName 
 */
const readFile = fileName => {
        return fs.readFileSync(path.resolve(fileName));
}

/**
 * (c) Jan Krejcárek
 * Creates an object in a structure required for the DominoDB module
 * to initialize o connection to the Domino server.
 * @param {string} serverHostName Host name of the Domino server
 * @param {string} port TCP port number where a Proton task listens for connection requests
 * @param {string} rootCertificatePath Path to the certificate used to establish a TLS connection
 * @param {string} clientCertificatePath Path to the application certificate used to authenticate the application
 * @param {string} clienKeyPath Path to the private key of the application
 */
const config = (serverHostName, port, rootCertificatePath, clientCertificatePath, clienKeyPath) => {
    const rootCertificate = readFile(rootCertificatePath);
    const clientCertificate = readFile(clientCertificatePath);
    const clientKey = readFile(clienKeyPath);
    
    const serverConfig = {
            hostName: serverHostName,
            connection: {
                    port: port,
                    secure: true
            },
            credentials: {
                    rootCertificate,
                    clientCertificate,
                    clientKey
            }    
    };

    return serverConfig;
};

module.exports = config;

All parts in the credentials section are mandatory. Even if you use PROTON_AUTHENTICATION=anonymous with PROTON_SSL=1, you must have certificates and keys for the client as well.

And we can now use this module in app.js

const protonConfig = require('./protonConfigSSL.js');
const serverConfig = protonConfig("serv02.fritz.box", "3002", "./certs/proton-self/ca.crt", "./certs/proton-self/app1.crt", "./certs/proton-self/app1.key");

My PROTON configuration is

[029971:000009-00007F78D8293700] PROTON_AUTHENTICATION=client_cert
[029971:000009-00007F78D8293700] PROTON_KEYFILE=proton-self.kyr
[029971:000009-00007F78D8293700] PROTON_LISTEN_ADDRESS=serv02.fritz.box
[029971:000009-00007F78D8293700] PROTON_LISTEN_PORT=3002
[029971:000009-00007F78D8293700] PROTON_SSL=1
[029971:000009-00007F78D8293700] PROTON_TRACE_REQUEST=0
[029971:000009-00007F78D8293700] PROTON_TRACE_SESSION=0

And after restarting PROTON ( restart task proton ) and starting my application ( npm start ), I was able to open hp.nsf in the browser.

I also found an interesting article by Sven Hasselbach about how to protect PROTON keys.

I hope that this will help others starting with this stuff to save some time.

Recommended reading:

Heiko Voigt: DominoDB and a big NO-NO !

Sven Hasselbach ( response to Heiko’s post ): DominoDB and a big NO-NO?


Restart Task HTTP – Unable To Open Notes Message Queue”

Just a quick hint for those running Lotus Notes Travel(l)er. When you try to restart the HTTP task on the Domino, you have to stop Travel(l)er prior to restarting the HTTP task ( restart task HTTP )

Otherwise you will see the following message on the server console

Restart Task HTTP
and the HTTP task will not restart.

UPDATE: The problem does not occur when you type “tell http quit” followed by “load http” at the server console.