xGrid And Performance Optimization

Yesterday, I took a closer look at the xGrid custom control, that has recently been posted on OpenNTF by Pablo Solano.
The control uses jqGrid, a jQuery plugin. It lets you display data from a view in a grid. jqGrid has some very nice features. For more details take a look at the demo page.
The download from OpenNTF also contains a set of demo data ( 40.000 person documents ).

I copied the sample application to my server ( Domino 8.5.3 64bit, 16GB RAM, 4 Core AMD ), opened every view in client to create the indexes, signed application and finally opened xContactsSSJS.xsp in Firefox 12.

Next, I took a closer look at the design.

xContactsSSJS.xsp contains a duplicate line of code, which can be deleted
var docs:NotesDocumentCollection = database.search(query);

 

Although the grid was rendered almost immediately in the browser, I wanted to do some time measurement to see how long the retrieval of 40.000 documents would last and if there is some room for optimization.
So I added two lines of code into xJsonContacts.xsp at the beginning and the end of the code that simply calculates the time difference in milliseconds from start to end.

try{
	var start = new Date().getTime();
	var externalContext = facesContext.getExternalContext();
	var writer = facesContext.getResponseWriter();
	var response = externalContext.getResponse();

	// Set content type
	response.setContentType("application/json");
	response.setHeader("Cache-Control", "no-cache");

	// Get all Contacts
	var query = 'Form = "Contact"';

	var docs:NotesDocumentCollection = database.search(query);

	json = "";
	var doc = docs.getFirstDocument()

	while (doc != null) {
		json = json + '{"@unid":"'+ doc.getUniversalID() + '","FirstName":"' + doc.getItemValueString("FirstName") +
		 '","LastName":"' + doc.getItemValueString("FirstName") +  '","State":"' + doc.getItemValueString("State") +
		 '","City":"' + doc.getItemValueString("City") + '"},'		

		// Get next doc and recycle
		tempdoc = docs.getNextDocument();
		doc.recycle();
		doc = tempdoc;
	}

	json  = "[" + @Left(json, @Length(json) - 1) + "]";	

	writer.write(json);
	writer.endDocument();
	var elapsed = new Date().getTime() - start;
	print("xContactsSSJS.xsp ->" + elapsed +" ms");

} catch(e){
	_dump(e);
}

I then opened the page 10 times in the browser. here are the results

The code builds a JSON string at runtime. So, for every element, it has to access a document from the document collection and get the item values one after the other from the document.

I changed the code; first of all, I copied the Contacts view an deleted all columns except one. I put the following code into the column formula.

_fld:="FirstName":"LastName":"State":"City";

"{\"@unid\":\""
+@Text(@DocumentUniqueID)+"\","
+ @Implode (
@Transform (
_fld; "_fn" ; "\"" + _fn + "\":\"" + @Text ( @GetField ( _fn) ) + "\"" ) ; "," ) + "},"

The formula computes a JSON string for every document in the view. This avoids the need to build the string at runtime in javascript.

The code for the XAgent looks like this:

try{
	var start = new Date().getTime();
	var externalContext = facesContext.getExternalContext();
	var writer = facesContext.getResponseWriter();
	var response = externalContext.getResponse();

	// Set content type
	response.setContentType("application/json");
	response.setHeader("Cache-Control", "no-cache");
	json = ""
	  var v:NotesView = database.getView("ContactsSingleCol");
	  //do not do AutoUpdates
	  v.AutoUpdate = false;
	  var nav:NotesViewNavigator = v.createViewNav();
	  nav.setEntryOptions(
	  NotesViewNavigator.VN_ENTRYOPT_NOCOUNTDATA);
	  //enable cache for max buffering
	  nav.BufferMaxEntries = 400
	  var entry:NotesViewEntry = nav.getFirst();

	  while (entry != null) {
	    json=json + entry.getColumnValues().elementAt(0).toString();
	    var tmpentry:NotesViewEntry = nav.getNext(entry);
	    entry.recycle();
	    entry = tmpentry;
	  }

  	writer.write('[' + @Left(json, @Length(json) - 1) + ']');
	writer.endDocument();
	var elapsed = new Date().getTime() - start;
print("xContactsSSJSViewNav.xsp ->" + elapsed +" ms");

} catch(e){
	_dump(e);
}

As you can see, the code uses a NotesViewNavigator to iterate thru the view entries and concat the values from the first column of the view containing the pre-build JSON string.

Once again, I opened the page in the browser; here are the results from my simple time measurement.

Conclusion: Building the JSON in advance and using a NotesViewNavigator speeds up the loading of the data.

In the above code, the new value is added to the existing JSON String using the “+” sign. Using the ‘+’ operator for concatenation isn’t bad per se though.

It’s very readable and it doesn’t necessarily affect performance. Each time you append something via ‘+’ a new String is created, the old stuff is copied, the new stuff is appended, and the old String is thrown away. The bigger the String gets the longer it takes – there is more to copy and more garbage is produced.

An alternative way to concat strings is using a java.lang.StringBuilder. As you might know, you can use Java in your server-side javascript.

xJsonContactsViewNavSb.xsp shows, how to use a stringbuffer

try{
	var start = new Date().getTime();
	var externalContext = facesContext.getExternalContext();
	var writer = facesContext.getResponseWriter();
	var response = externalContext.getResponse();

	// Set content type
	response.setContentType("application/json");
	response.setHeader("Cache-Control", "no-cache");

	  var json:java.lang.StringBuilder = new java.lang.StringBuilder();
	  var v:NotesView = database.getView("ContactsSingleCol");
	  //do not do AutoUpdates
	  v.AutoUpdate = false;
	  var nav:NotesViewNavigator = v.createViewNav();
	  nav.setEntryOptions(
	  NotesViewNavigator.VN_ENTRYOPT_NOCOUNTDATA);
	  //enable cache for max buffering
	  nav.BufferMaxEntries = 400
	  var entry:NotesViewEntry = nav.getFirst();

	  while (entry != null) {
	    json.append( entry.getColumnValues().elementAt(0).toString());
	    var tmpentry:NotesViewEntry = nav.getNext(entry);
	    entry.recycle();
	    entry = tmpentry;
	  }

  writer.write('[' + @Left(json.toString(), @Length(json.toString()) - 1) + ']');
	writer.endDocument();
	var elapsed = new Date().getTime() - start;
print("xContactsSSJSViewNavSb.xsp ->" + elapsed +" ms");

} catch(e){
	_dump(e);
}

Opening the page in the browser shows the following results

Impressive, isn’t it?

As a fazit, even in XPages programming there is room for views and @formulas.

Uses these elements to pre-calculate values and move away this work from the runtime. Use a NotesViewNavigator to access data from the view. Read this article to find out more about NotesViewNavigator.
Conclusion: If you have to concat a large number of strings, use a java.lang.StringBuilder instead of the “+” operator.

Having done all these steps to optimize performance, it is only a small step to use Java instead of javascript to do the heavy lifting.

Here is the code from xJsonContactsJava.xsp

try{
	var start = new Date().getTime();
	var externalContext = facesContext.getExternalContext();
	var writer = facesContext.getResponseWriter();
	var response = externalContext.getResponse();

	// Set content type
	response.setContentType("application/json");
	response.setHeader("Cache-Control", "no-cache");
	var out:de.eknori.ViewColumn = new de.eknori.ViewColumn();
	writer.write(out.getViewColumnValueJSON("ContactsSingleCol",0));
	writer.endDocument();
	var elapsed = new Date().getTime() - start;
	print("xContactsSSJSJava.xsp ->" + elapsed +" ms");

} catch(e){
	_dump(e);
}

The getViewColumnValueJSON() method is located in the de.eknori.ViewColumn class.

package de.eknori;

import static com.ibm.xsp.extlib.util.ExtLibUtil.getCurrentDatabase;
import lotus.domino.NotesException;
import lotus.domino.View;
import lotus.domino.ViewEntry;
import lotus.domino.ViewNavigator;

public class ViewColumn {
	private static final String MSG_STRING_ERROR = "ERROR: ";
	private static final String MSG_STRING_NOT_FOUND = " not found";

	public ViewColumn() {
	}

	public String getViewColumnValueJSON(String viewname, int pos) {
		ViewNavigator nav = null;
		StringBuilder json = new StringBuilder();
		json.append('[');
		String strValue = "";
		try {
			View view = getCurrentDatabase().getView(viewname);
			if (null != view) {
				view.setAutoUpdate(false);
				nav = view.createViewNav();
				nav.setEntryOptions(ViewNavigator.VN_ENTRYOPT_NOCOUNTDATA);
				nav.setBufferMaxEntries(400);
				ViewEntry entry = nav.getFirst();

				while (entry != null) {
					json.append(entry.getColumnValues().elementAt(pos)
							.toString());
					ViewEntry tmpentry = nav.getNext(entry);
					entry.recycle();
					entry = tmpentry;
				}
				strValue = json.toString();
				strValue = strValue.substring(0, strValue.lastIndexOf(","))
						+ "]";
				view.setAutoUpdate(true);
			} else {
				System.out.println(MSG_STRING_ERROR + viewname
						+ MSG_STRING_NOT_FOUND);
			}
		} catch (NotesException e) {
			System.out.println(MSG_STRING_ERROR);
			strValue = "[{}]";
		}
		return strValue;
	}
}

As you can see from the entries in the screenshot, there is another gain in performance when using Java.

If your application has a large amount of data, I strongly advice to use Java instead of JavaScript.

I’m doing Java for no longer than 6 month by now. If you are familiar with LotusScript, you will be able to learn the basic java stuff real quick. There is no excuse to not start learning Java right now.

You can download the application including my modifications here.  (design only!)