Tag: HCL

Error: Index does not exist. (No such file or directory) – UPDATE

This is a follow up article to Error: Index does not exist. (No such file or directory)

Over the past days, I worked with HCL support to find out what causes the issue. With help from the extended support team we finally found the answer.

In my test environment I have an application that stores order data in 250k documents. If the application has no FT index, I can run an agent to find all orders that have a quantity below 30.

	public void sample1(){
		Instant start = null;
		Instant finish = null;
		
		DocumentCollection coll = null;
		DominoQuery dql1 = null;

		try {
			dql1 = order2018.createDominoQuery();
			dql1.setMaxScanDocs(Integer.MAX_VALUE);
			dql1.setMaxScanEntries(Integer.MAX_VALUE);

			String query = "quantity < 30";
			
			start = Instant.now();
			coll = dql1.execute(query, "named_java_rslt_1",true,24);
			finish = Instant.now();
			System.out.println(String.format("%s found %d results in %d msec.", query, coll.getCount(),
					Duration.between(start, finish).toMillis()));
			
			
		} catch (NotesException e) {
			e.printStackTrace();
		} 
	}

When I run the code, I get 

[212C:051A-32D0] 06.04.2022 07:33:01   AMgr: Start executing agent 'java.test' in 'orders\dql.nsf'
[212C:051C-30E8] 06.04.2022 07:33:15   Success: overwriting named foundset "named_java_rslt_1"
[212C:051C-30E8] 06.04.2022 07:33:15   Agent Manager: Agent printing: quantity < 30 found 724 results in 14000 msec.
[212C:051A-32D0] 06.04.2022 07:33:15   AMgr: Agent 'java.test' in 'orders\dql.nsf' completed execution

To speed up things a bit, I then created a FT index. I used the new feature in Domino V12 to exclude specific items to keep the index as small as possible. I was only interested in filtering by quantity. So I disabled index creation for all $ items as well as all items except quantity.

The index was created without issues.

[2AAC:0005-2474] FTGIndex: Finished: 0 ms. for [h:\orders\2018.ft] 
[2AAC:0005-2474] 250000 documents added, 0 updated, 0 deleted: 0 text bytes; 8000000 numeric bytes for [h:\orders\2018.ft]
[2AAC:0005-2474] FTGIndex: All Done: 1108 ms for [h:\orders\2018.ft]
[2AAC:0005-2474]  OUT FTGIndex rc = 0 (No error) - for [h:\orders\2018.ft]

When I now run the agent I get

[212C:0520-1FD0] 06.04.2022 07:46:47   AMgr: Start executing agent 'java.test' in 'orders\dql.nsf'
[212C:0522-2328] 06.04.2022 07:46:47   Full Text message: Index does not exist. (No such file or directory)
[212C:0522-2328] 06.04.2022 07:46:47   GTR search error for "h:\orders\2018.ft\ftgi": Index does not exist.: Full text index not found for this database
[212C:0522-2328] 06.04.2022 07:46:47   Agent Manager: Agent  error: NotesException: Domino Query execution error:   Full text index not found for this database -  error during planning and tree generation     quantity < 30   (Call hint: FTCalls::FTSearchExt, Core call #0
)   ****************** 
[212C:0522-2328] 06.04.2022 07:46:47   Agent Manager: Agent  error:  at lotus.domino.local.DominoQuery.execute(Unknown Source)
[212C:0522-2328] 06.04.2022 07:46:47   Agent Manager: Agent  error:  at de.eknori.DQLTest.sample1(DQLTest.java:50)
[212C:0522-2328] 06.04.2022 07:46:47   Agent Manager: Agent  error:  at JavaAgent.NotesMain(Unknown Source)
[212C:0522-2328] 06.04.2022 07:46:47   Agent Manager: Agent  error:  at lotus.domino.AgentBase.runNotes(Unknown Source)
[212C:0522-2328] 06.04.2022 07:46:47   Agent Manager: Agent  error:  at lotus.domino.NotesThread.run(Unknown Source)
[212C:0520-1FD0] 06.04.2022 07:46:47   AMgr: Agent 'java.test' in 'orders\dql.nsf' completed execution

And here is WHY. There wasn’t a single piece of text indexed. That is because the field quantity is not a “text” field. It is a NUMBER field an that goes into a different index and would not be in FullMain.

I changed the Full Text Index Rules to include at least on TEXT item.

and re-created the index

[2AAC:0005-2474] FTGIndex: Finished: 0 ms. for [h:\orders\2018.ft] 
[2AAC:0005-2474] 250000 documents added, 0 updated, 0 deleted: 1500000 text bytes; 8000000 numeric bytes for [h:\orders\2018.ft]
[2AAC:0005-2474] FTGIndex: All Done: 1087 ms for [h:\orders\2018.ft]
[2AAC:0005-2474]  OUT FTGIndex rc = 0 (No error) - for [h:\orders\2018.ft]

As you can see, we now have a FULLMAIN folder under ftgi.

When I now run my agent, everything works fine

[212C:0526-3348] 06.04.2022 07:52:48   AMgr: Start executing agent 'java.test' in 'orders\dql.nsf'
[212C:0528-0914] 06.04.2022 07:52:48   Success: overwriting named foundset "named_java_rslt_1"
[212C:0528-0914] 06.04.2022 07:52:48   Agent Manager: Agent printing: quantity < 30 found 724 results in 234 msec.
[212C:0526-3348] 06.04.2022 07:52:48   AMgr: Agent 'java.test' in 'orders\dql.nsf' completed execution

Documentation for (Notes)DominoQuery.execute (…) method is wrong

The documentation under https://help.hcltechsw.com/dom_designer/12.0.0/basic/H_EXECUTE_METHOD_NDQ.html#reference_ivd_gz3_cgb and https://help.hcltechsw.com/dom_designer/12.0.0/basic/H_EXECUTE_METHOD_JAVA.html is incomplete, because it does not list all variants for the execute method.

The Lotusscript documentation only shows this syntax

Set doccol = NotesDominoQuery.Execute(Query$, Optional String ResultName, Optional Long ExpireHours)

while the JAVA documentation lists almost all possible syntax variants

DocumentCollection execute(String query) throws NotesException
DocumentCollection execute(String query, String resultname, boolean replace) throws NotesException
DocumentCollection execute(String query, String resultname, boolean replace, int expirehours) throws NotesExceptionn

The correct documentation would be

Set doccol = NotesDominoQuery.Execute(Query$)
Set doccol = NotesDominoQuery.Execute(Query$, Optional String ResultName)
Set doccol = NotesDominoQuery.Execute(Query$, Optional String ResultName, Optional boolean Replace)
Set doccol = NotesDominoQuery.Execute(Query$, Optional String ResultName, Optional boolean Replace, Optional Long ExpireHours)

for Lotusscript, and for Java

DocumentCollection execute(String query) throws NotesException
DocumentCollection execute(String query, String resultname) throws NotesException
DocumentCollection execute(String query, String resultname, boolean replace) throws NotesException
DocumentCollection execute(String query, String resultname, boolean replace, int expirehours) throws NotesException

Be aware that there is NO type check in the Lotusscript for the REPLACE parameter. It is possible to pass i.e. 42 to the method, which would work without any hint that 42 is not a valid value for true or false.


Error: Index does not exist. (No such file or directory)

I recently ran into an issue with the new feature in Domino 12 to exclude fields from full-text search indexes.

You must enable this feature by adding the notes.ini setting FT_DEBUG_DISABLE_DB_RULES=0 to the Domino server AND restart the Domino server.

Navigate to Application Properties -> Full Text Index and select the items to exclude fron the FT index.

You can then delete an existing FT Index and create a new one. The size should be smaller now. So far so good.

If you now do a search either from the Notes Client or progammatically ( LS or Java ) you will get an error.

Y03/26/2022 05:43:54 AM  AMgr: Start executing agent 'qry' in 'ftdemo.nsf'
03/26/2022 05:43:54 AM  Agent Manager: Agent printing: initial query.
03/26/2022 05:43:54 AM  Full Text message: Index does not exist. (No such file or directory)
03/26/2022 05:43:54 AM  GTR search error for "/local/notesdata/ftdemo.ft/ftgi": Index does not exist.: Full text index not found for this database
03/26/2022 05:43:54 AM  Agent Manager: Agent  error: NotesException: Domino Query execution error: 
Full text index not found for this database -  error during planning and tree generation  


idNumber > 20 AND idNumber < 30

	(Call hint: FTSearchExt, Core call #0)


******************

03/26/2022 05:43:54 AM  Agent Manager: Agent  error: 	at lotus.domino.local.DominoQuery.execute(Unknown Source)
03/26/2022 05:43:54 AM  Agent Manager: Agent  error: 	at JavaAgent.NotesMain(JavaAgent.java:35)
03/26/2022 05:43:54 AM  Agent Manager: Agent  error: 	at lotus.domino.AgentBase.runNotes(Unknown Source)
03/26/2022 05:43:54 AM  Agent Manager: Agent  error: 	at lotus.domino.NotesThread.run(Unknown Source)
03/26/2022 05:43:54 AM  AMgr: Agent 'qry' in 'ftdemo.nsf' completed execution

The issue is reproducible on Windows and Linux. I have opened a support case with HCL.

If you disable the feature, restart the server and recreate the index, everthing works good.


NotesDominoQuery sample

I have put together a small sample to demonstrate how to use NotesDominoQuery from LotusScript.

I created a new Class DQLWrapper. A little bit over the top, I know.

%REM
	Library 10010.dql
	Created Dec 30, 2018 by Ulrich Krause/singultus
%END REM
Option Declare

%REM
	Class DqlWrapper
%END REM
Public Class DqlWrapper
	
	m_query As String
	m_session As NotesSession
	m_db As NotesDatabase
	m_ndq As NotesDominoQuery
	
	%REM
		Sub New
	%END REM
	Public Sub New(strDbFilePath As String)
		Set me.m_session = New NotesSession
		Set me.m_db = me.m_session.Getdatabase("",strDbFilePath, False)
		If ( me.m_db.Isopen ) then
			Set me.m_ndq = me.m_db.Createdominoquery()
		Else
			' // do some error handling
		End if
	End Sub

	%REM
		Public function executeQuery()
	%END REM	
	Public function executeQuery() As NotesDocumentCollection
		If ( me.m_query <> "" ) then
			Set executeQuery = me.m_ndq.Execute(me.m_query)
		Else
			Set executeQuery = nothing
		End If
	End Function

	%REM
		Public Function explainQuery()
	%END REM	
	Public Function explainQuery() As String
		If ( me.m_query <> "" ) Then
			explainQuery = me.m_ndq.Explain(me.m_query)
		Else
			explainQuery = ""
		End If
		
	End Function
	
	%REM
		Public Function explainQuery()
	%END REM	
	Public Function parseQuery() As String
		If ( me.m_query <> "" ) Then
			parseQuery = me.m_ndq.parse(me.m_query)
		Else
			parseQuery = ""
		End If
		
	End Function	

	%REM
		Property query
	%END REM	
	Public Property Set query As String
		me.m_query = query
	End property
	
End Class

The query itself is executed from an agent that runs on the server. At the moment it is not possible to run a query client/ server.

Here is the code for the agent

%REM
	Agent dql.execute
	Created Dec 30, 2018 by Ulrich Krause/singultus
%END REM
Option Public
Option Declare
Use "10010.dql"

Sub Initialize
	Dim query As String
	Dim col As  NotesDocumentCollection
	query = "firstname = 'Ulrich' And lastname = 'Krause'"
	
	Dim dql As New DQlWrapper("names.nsf")
	dql.query = query
	
	If (  LCase(dql.parseQuery()) ="success" ) Then
		
		Set col = dql.executeQuery()
		MsgBox "QRY returns # docs: " + CStr(col.count)
		
		If ( col.count > 0 ) then
			Dim doc As NotesDocument
			Set doc = col.Getfirstdocument()
			MsgBox "UNID of first doc: " + doc.Universalid
		End if
	Else 
		
		MsgBox dql.explainQuery()
	End If
	
End Sub

You can now start the agent from the server console. You will get the number of documents for this query and the UNID of the first document found.

te amgr run "ec11.nsf" 'dql.execute'
 [0DFC:001F-0FFC] 30.12.2018 13:49:10   AMgr: Start executing agent 'dql.execute' in 'ec11.nsf'
 [0DFC:001F-0FFC] 30.12.2018 13:49:10   Agent Manager: Agent message: QRY returns # docs: 1
 [0DFC:001F-0FFC] 30.12.2018 13:49:10   Agent Manager: Agent message: UNID of first doc: D8436D0F4E546BA3C12573FE0070AE88
 [0DFC:001F-0FFC] 30.12.2018 13:49:10   AMgr: Agent 'dql.execute' in 'ec11.nsf' completed execution

If your query contains errors / is not understandable, you will see an output similar like this on your console

[0DFC:0020-11D0] 30.12.2018 13:59:45   Agent Manager: Agent 'dql.execute' error: Domino Query execution error:   Query is not understandable -  syntax error     - processing or expecting operator (=, <, <= …) token syntax    (Call hint: OSCalls::OSLocalAllc, Core call 
 0)  firstname = 'Ulrich' And lastname IS 'Krause' …………………………….^………..  ****
 [0DFC:0020-11D0] 30.12.2018 13:59:45   AMgr: Agent 'dql.execute' in 'ec11.nsf' completed execution


[Solved] – PROTON_SSL=1

This is a follow up to my recent post https://www.eknori.de/2018-12-28/problem-using-proton-with-proton_ssl1-enabled/

Due to my misunderstanding and also a lack in documentation, I was not able to get PROTON with PROTON_SSL=1 and PROTON_AUTHENTICATION=client_certs running.

But thanks to our great community and a little help from Jan Krejcárek (SUTOL), I was finally able to solve the puzzle.

I must admit that I am not an expert in this field. So please bare with me if things are obvious for you. They are (not yet) for me.

My understanding of using TLS/SSL with PROTON was that I could use my existing Let’s Encrypt certificate to communicate via HTTPS instead of HTTP with my server URL.

But this is not the case. TLS/SSL encryption only enables a secured communication between the domino.db module and the PROTON addin sitting on the Domino server.

Here is an image.

The second thing that I got wrong was the fact that the existing certificate and it’s CA would be enough to enable the secured communication. In my first test, I had set PROTON_AUTHENTICATION=anonymous.

So why bother with client certificates if they are not needed at this point?

I followed Jan’s advice to try with a new generated self-signed certificate. The AppDevPack contains shell scripts to create some sample certs and also lets you create a keyring file that can be used with PROTON.

I have modified the make_keyring.sh file a bit because it complained about missing environment variables.

# added UKR, 12/2018
export NOTESDATA=/local/notesdata
export NOTESBIN=/opt/ibm/domino/notes/10000000/linux

I then created the certs and keyring and copied them to the proper location.

The original source code has used 

// proton config
const serverConfig = {
    "hostName": "serv02.fritz.box",
    "connection": {
        "port": 3002
    }
}

to define the serverConfig. This must be enhanced, if you are going the secure way.

I created a new module using Jan’s code

const fs = require('fs');
const path = require('path');

/**
 * Internal functions that reads a content of a file to a buffer.
 * @param {string} fileName 
 */
const readFile = fileName => {
        return fs.readFileSync(path.resolve(fileName));
}

/**
 * (c) Jan Krejcárek
 * Creates an object in a structure required for the DominoDB module
 * to initialize o connection to the Domino server.
 * @param {string} serverHostName Host name of the Domino server
 * @param {string} port TCP port number where a Proton task listens for connection requests
 * @param {string} rootCertificatePath Path to the certificate used to establish a TLS connection
 * @param {string} clientCertificatePath Path to the application certificate used to authenticate the application
 * @param {string} clienKeyPath Path to the private key of the application
 */
const config = (serverHostName, port, rootCertificatePath, clientCertificatePath, clienKeyPath) => {
    const rootCertificate = readFile(rootCertificatePath);
    const clientCertificate = readFile(clientCertificatePath);
    const clientKey = readFile(clienKeyPath);
    
    const serverConfig = {
            hostName: serverHostName,
            connection: {
                    port: port,
                    secure: true
            },
            credentials: {
                    rootCertificate,
                    clientCertificate,
                    clientKey
            }    
    };

    return serverConfig;
};

module.exports = config;

All parts in the credentials section are mandatory. Even if you use PROTON_AUTHENTICATION=anonymous with PROTON_SSL=1, you must have certificates and keys for the client as well.

And we can now use this module in app.js

const protonConfig = require('./protonConfigSSL.js');
const serverConfig = protonConfig("serv02.fritz.box", "3002", "./certs/proton-self/ca.crt", "./certs/proton-self/app1.crt", "./certs/proton-self/app1.key");

My PROTON configuration is

[029971:000009-00007F78D8293700] PROTON_AUTHENTICATION=client_cert
[029971:000009-00007F78D8293700] PROTON_KEYFILE=proton-self.kyr
[029971:000009-00007F78D8293700] PROTON_LISTEN_ADDRESS=serv02.fritz.box
[029971:000009-00007F78D8293700] PROTON_LISTEN_PORT=3002
[029971:000009-00007F78D8293700] PROTON_SSL=1
[029971:000009-00007F78D8293700] PROTON_TRACE_REQUEST=0
[029971:000009-00007F78D8293700] PROTON_TRACE_SESSION=0

And after restarting PROTON ( restart task proton ) and starting my application ( npm start ), I was able to open hp.nsf in the browser.

I also found an interesting article by Sven Hasselbach about how to protect PROTON keys.

I hope that this will help others starting with this stuff to save some time.

Recommended reading:

Heiko Voigt: DominoDB and a big NO-NO !

Sven Hasselbach ( response to Heiko’s post ): DominoDB and a big NO-NO?