Tag: DOTS

DOTS – “An error occurred while processing the command”

If you are using DOTS in Domino 12.x, you might have seen some NullPointer Exceptions after you have updated a DOTS plugin. Recommendation was to delete the complete workspace-dots folder prior to starting DOTS after the upgrade.

This issue has been addressed in SPR# SJAAC3BNWV: DOTS – Workspace need to be deleted on every change and add of new plugin.

With this fix it is no longer necessary to delete the workspace-dots folder. Not sure, when the fix was provided. There is no information about it in the fix list database.

But this is only true for Domino on WINDOWS. If you are using DOTS with Domino on LINUX, you will see an error on the Domino server console ( “An error occurred while processing the command”) or even worse, the NullPointer Exceptions.

The problem is that the underlying API in Domino does not delete files starting with a dot. (hidden files on UNIX systems).

There is a HF67 available for Domino 12.0.1 FP1 that fixes this issue. SPR# DNADCHM8VJ: Deleting directory fails on Linux/UNIX when directory contains files starting with dot (UNIX hidden file).

The fix has already found it’s way into Domino 12.0.2 (Danube).

If you need the fix, open a case with HCL support.


Load JDBC SQL driver at runtime in DOTS and amgr

Recently there was a discussion in the German Notes forum about an error when loading a JDBC driver. https://atnotes.de/index.php/topic,63166.0.html To be clear, the problem has so far only occurred with the driver for MySQL.
The driver used is the same as the one used for HCL Traveler when Traveler uses a MySQL database as backend datastore in HA mode.
I could not reproduce the described error in Traveler, but in a DOTS plugin and in a Java agent.
My tests were performed with Domino 12. But I assume that the behavior is also reproducible in V11.0.1.x.
Here is a description of my tests and instructions on how to solve the problem.
Once again. The problem is only with the JDBC MySQL driver and is not with Domino. Rather, it is a problem in the driver’s code, but it is affecting Domino.

Here is the code I used in a Java agent. I also use the same code in my DOTS plugin.

import java.io.File;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
import java.sql.Driver;
import java.sql.DriverManager;

import lotus.domino.AgentBase;

public class JavaAgent extends AgentBase {
	static final String MYSQL_DRIVER = "com.mysql.cj.jdbc.Driver";
	static final String MYSQL_DRIVER_FILEPATH = "Traveler\\lib\\mysql-connector-java-8.0.22.jar";

	public void NotesMain() {

		try {
			File dbFile = new File(MYSQL_DRIVER_FILEPATH);
			updateClasspath(dbFile.toURI().toURL());
			Class.forName(MYSQL_DRIVER);

			Driver driver = DriverManager.getDriver("jdbc:mysql:");

			System.out.println(String.format("%s loaded and registered. Version: %d.%d", dbFile.getName(),
					driver.getMajorVersion(), driver.getMinorVersion()));
		} catch (Exception e) {
			e.printStackTrace();
		}
	}

	private void updateClasspath(URL path) {
		try {
			URLClassLoader cl = (URLClassLoader) ClassLoader.getSystemClassLoader();
			Method m = URLClassLoader.class.getDeclaredMethod("addURL", new Class[] { URL.class });
			m.setAccessible(true);
			m.invoke(cl, new Object[] { path });
		} catch (Exception e) {
			e.printStackTrace();
		}
	}
}

In the simplest variant, the drivers to be used are simply copied to DominoPrgmDir/ndext and can then be accessed via

Class.forName(MYSQL_DRIVER);

The directory ndext is already in the classpath. But I use the drivers from the HCL Traveler installation. The directory is not in the classpath because Traveler brings its own JVM. So we need to extend the classpath before using the driver. This is done by the updateClasspath() method.

When you run the agent, you will get the following error.

AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent  error: Exception in thread "AgentThread: JavaAgent" 
Agent Manager: Agent  error: java.lang.ExceptionInInitializerError
Agent Manager: Agent  error:     at java.lang.J9VMInternals.ensureError(J9VMInternals.java:146)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:135)
Agent Manager: Agent  error:     at sun.misc.Unsafe.ensureClassInitialized(Native Method)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.initialize(J9VMInternals.java:87)
Agent Manager: Agent  error:     at java.lang.Class.forName(Class.java:347)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.NonRegisteringDriver.(NonRegisteringDriver.java:98)
Agent Manager: Agent  error:     at sun.misc.Unsafe.ensureClassInitialized(Native Method)
Agent Manager: Agent  error:     at java.lang.J9VMInternals.initialize(J9VMInternals.java:87)
Agent Manager: Agent  error:     at java.lang.Class.forName(Class.java:347)
Agent Manager: Agent  error:     at JavaAgent.NotesMain(Unknown Source)
Agent Manager: Agent  error:     at lotus.domino.AgentBase.runNotes(Unknown Source)
Agent Manager: Agent  error:     at lotus.domino.NotesThread.run(Unknown Source)
Agent Manager: Agent  error: Caused by: 
Agent Manager: Agent  error: java.security.AccessControlException: Access denied ("java.lang.RuntimePermission" "setContextClassLoader")
Agent Manager: Agent  error:     at java.security.AccessController.throwACE(AccessController.java:176)
Agent Manager: Agent  error:     at java.security.AccessController.checkPermissionHelper(AccessController.java:238)
Agent Manager: Agent  error:     at java.security.AccessController.checkPermission(AccessController.java:385)
Agent Manager: Agent  error:     at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
Agent Manager: Agent  error:     at lotus.notes.AgentSecurityManager.checkPermission(Unknown Source)
 Agent Manager: Agent  error:     at java.lang.Thread.setContextClassLoader(Thread.java:840)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.lambda$static$0(AbandonedConnectionCleanupThread.java:77)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread$$Lambda$8/0x0000000000000000.newThread(Unknown Source)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932)
Agent Manager: Agent  error:     at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367)
Agent Manager: Agent  error:     at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
Agent Manager: Agent  error:     at com.mysql.cj.jdbc.AbandonedConnectionCleanupThread.(AbandonedConnectionCleanupThread.java:80)
Agent Manager: Agent  error:     … 10 more
AMgr: Agent 'driver' in 'driver.nsf' completed execution

The relevant line in the stack trace is 

Agent Manager: Agent  error: java.security.AccessControlException: Access denied ("java.lang.RuntimePermission" "setContextClassLoader")

A little causal research, and the reason was found. It is an error in the code of the driver.Here is the source https://bugs.mysql.com/bug.php?id=88172.
To solve the problem I have added the following lines to the java.policy in DominoPrgmDir/jvm/lib/security

grant codeBase "file:${notes.binary}/Traveler/lib/-" {
	permission java.lang.RuntimePermission "setContextClassLoader";
};

This fixed the problem in the DOTS plugin, but the error in the Java agent persisted. So i moved the “permission” to the 

// default permissions granted to all domains

grant {

section at the top of the file.

This now solved the issue for DOTS and the Java agent. To be more precise. This solved it for the Java agent only, if you have the driver copied to the ndext directory. If you use the updateClasspath() method, you must add additional permissions to make your code work.

grant {
	permission java.lang.RuntimePermission "getClassLoader";
	permission java.lang.RuntimePermission "setContextClassLoader";
	permission java.lang.RuntimePermission "accessDeclaredMembers";
	permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

Modifying the java.policy file directly is generally not a good idea. The customizations may be lost during an upgrade.
HCL Domino uses the parameter javaOptionsFile= to make adjustments to the JVM.
Using this file we can now include our own custom.policy file. The content of the file is added to the DEFAULT java.policy at runtime. My custom.policy file has the following content

// =============================================================
//  custom.policy file added at runtime
//  -Djava.security.manager -Djava.security.policy=custom.policy
// =============================================================

grant {
	permission java.lang.RuntimePermission "getClassLoader";
	permission java.lang.RuntimePermission "setContextClassLoader";
	permission java.lang.RuntimePermission "accessDeclaredMembers";
	permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};

Save the file to the DominoDataDir for example.

Create a new file in the DominoDataDir javaOptionsFile.opt and add the following line to the file.

-Djava.security.manager -Djava.security.policy=./data/custom.policy

Save the file and add the following line to the server notes.ini

JavaOptionsFile=<DominoDataDir>/javaoptions.opt

Replace <DominoDataDir> according to your environment. After restarting Domino, you should now be able to load the driver without issues.

But wait. Here is one more thing. When you run the agent to load the driver, it will be loaded but you will see the following on the console.

te amgr run "driver.nsf" 'driver'
JVM: Java Virtual Machine initialized.
AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent printing: mysql-connector-java-8.0.22.jar
Agent Manager: Agent printing: Driver loaded.
Agent Manager: Agent  error: Error cleaning up agent threads
Agent Manager: Agent  error: Exception in thread "mysql-cj-abandoned-connection-cleanup" 
Agent Manager: Agent  error: java.lang.IllegalMonitorStateException
Agent Manager: Agent  error: 	at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
Agent Manager: Agent  error: 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
Agent Manager: Agent  error: 	at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
Agent Manager: Agent  error: 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:449)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
Agent Manager: Agent  error: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Agent Manager: Agent  error: 	at java.lang.Thread.run(Thread.java:823)
AMgr: Agent 'driver' in 'driver.nsf' completed execution

The errors only occur when the JVM is not yet initialized. Any subsequent agent run does not show the output. Also, running the code in a DOTS plugin does not show any errors.

lo dots
JVM: Java Virtual Machine initialized.
Domino OSGi Tasklet Container started ( profile DOTS )
te dots run de.eknori.driver.main com.mysql.cj.jdbc.Driver C:\\Domino\\Traveler\\lib\\mysql-connector-java-8.0.22.jar
[DOTS] (de.eknori.driver.main) [driver]: Main started.
[DOTS] (de.eknori.driver.main) mysql-connector-java-8.0.22.jar
[DOTS] (de.eknori.driver.main) [driver]: Main disposed.

te amgr run "driver.nsf" 'driver'
AMgr: Start executing agent 'driver' in 'driver.nsf'
Agent Manager: Agent printing: mysql-connector-java-8.0.22.jar
Agent Manager: Agent printing: Driver loaded.
AMgr: Agent 'driver' in 'driver.nsf' completed execution

So far I can’t see why the error doesn’t occur when loading the JDBC MySQL driver in HCL Traveler. HCL uses the exact same code in Notes.jar.

I can only assume that the required adjustments to the java.policy are already in place when initializing the HCL Traveler JVM. I have not yet run my tests in a V11 environment. That is still pending.

Maybe someone is fancy to do this in a V11 environment and give feedback.


Domino v 12 brings back Domino OSGi Tasklet Service (DOTS)

Domino 12 brings back DOTS. The feature was removed for an unclear reason in v 10.

I had created a new idea in the HCL #dominoforever Product Ideas Portal on 26-MAR-2019.

Barely 2 years later, DOTS is back. The first version was available in Domino v 12 Beta3 codedrop. It was not exactly, what I expected, and I had some discussion with Thomas Hampel and HCL Development.

HCL listened and the gave me a preview what they have changed based on the input provided during Beta testing.

  • In Beta3, DOTS needs an additional notes.ini parameter to locate the launcher.jar and to start the JVM. This requirement has been removed in the final version.
  • The ndots.exe delivered with Domino 9.x had a fixed value for the max. memory allocation pool size of only 64M. If you build complex DOTS applications that is way to small. DOTS v 12 now launches a JVM with Xmx = 1024M and Xms = 64M. There are 2 new notes.ini parameters (DOTSJavaMaxHeapSize & DOTSJavaMinHeapSize) to give you full control over this settings
  • DOTS v 12 supports JavaUserOptionsFile for any non DEFAULT JVM parameters.
  • A very unpleasant behavior of the installer was that it completely deleted the existing DOTS installation under <DominoPrgmDir>/osgi-dots when upgrading Domino to v 12. Thus all custom plugins were gone. This is handled now and the upgrade will not remove existing plugins in <DominoPrgmDir>/osgi-dots/shared/eclipe/plugins.

For developers it is important to know that plugins developed for the existing DOTS version can no longer be executed under Domino v 12. For all tasklets created using previous DOTS versions, you will need to update the package names and recompile all plugins created with a version of DOTS binaries integrated with Domino 12. Preferably using Eclipse SDK 4.6.2.