DCT – more confusion

Don’t get me wrong; I really like DCT. Over the past 2 days I digged deeper into the configuration and the coding. I can imagine, how much time was invested in building this tool. Kudos to the team.

Going thru the config.xml I found a lot of wrong settings. Some of them I have already sent to Scott O’Keefe of IBM.

I’m asking myself what would be the best way to report bugs found? Opening a PMR, sending mails to Scott? Any suggestions?


Folder has been damaged. Please close and reopen database to have it repaired

Earlier this morning, I saw the following message on the console of one of our Domino servers.

Error reading OperationTime for folder operations in database d:\Programme\IBM\Lotus\Domino\data\mail\xxxx.nsf. Run Fixup on the database.: Invalid Time or Date Encountered

The database belongs to a user that have quit the company a few days ago and I was not very concerned about this message. As usual I would make a new copy of the database and store it on a different server and let adminp do the job of deleting the user and the according database and replicas.

I created a new copy and opened the database.  A messagebox appeared:

Folder has been damaged.  Please close and reopen database to have it repaired.

Needless to say that a fixup did not solve the problem. Nice to know that “a folder” is damaged, but why does the message does not tell me which folder is damaged?

My only hope was, that the server log would be more verbose. So I opened the log. And there it was:

Database marked corrupt: Folder ($Inbox) corrupt
06.08.2009 11:07:33   Database collection error: d:\Programme\IBM\Lotus\Domino\data\xout\xxxxx.nsf: Folder has been damaged.  Please close and reopen database to have it repaired.

The strange thing is that the database on the source server opens without any error message…

I deleted the copy from the server and ran a fixup against the source database. Then I replaced the design and deleted all view indexes.

Once again I created a new copy … same error occured.

I opened the source database and moved all documents from the inbox to a temporary folder. I deleted the inbox and replaced the design to recreate the inbox folder

Guess what happened after the new copy had been created .

I created a new database on the source server just to be sure that it is not the template that is damaged. The new copy on the target server opens without any complaint.

Next thing I tried is to create a new copy on the target server after deleting the inbox and before replacing the design. I replaced the design after the database was copied to the target.

And … drumroll … No error message in the client or in the server’s log.

Can anyone explain this behaviour, please??


Confusing DCT results

The Domino Configuration Tuner (DCT) evaluates server settings according to a growing catalog of  best practices. All servers in a single domain can be evaluated together. DCT generates reports that explain the issues DCT uncovers, suggest mitigations, and provide references to supporting publications.

DCT is intended to provide customers with easy-to-use self-service configuration analysis so that their installations are more robust and experience better performance.

Today I wanted to show this tool to a new administrator and ran a scan. After the scan completed, I reviewed the result and saw this entry:

NagleOFF1

Well, ok, so let us do what is recommended …

After a new scan I found this entry in the scan results:

NagleOFF2

Huh!! How confusing is this?

I have seen this in an earlier version of DCT, but before I ran the first scan, I updated both, the rules and the design of the DCT database.


More Nagios, More DAOS

I have configured 2 more services on my Nagios server to check the size and filecount of the DAOS repository.

The filecount is determined by the check_file plugin by Tevfik Karagulle and I use a simple script on my Windows 2003 host to get the folder size in KB.

Const intOK = 0
Const intWarning = 1
Const intCritical = 2
Const intUnknown = 3

strDrivePathFolder = Wscript.Arguments.Unnamed.Item(0)
strWarningValue = Wscript.Arguments.Unnamed.Item(1)
strCriticalValue = Wscript.Arguments.Unnamed.Item(2)

Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFolder = objFSO.GetFolder(strDrivePathFolder)

FolderSize = CDbl(objFolder.Size)
strWarningValue = CDbl(strWarningValue)
strCriticalValue = CDbl(strCriticalValue)

If ( FolderSize > strWarningValue ) or ( FolderSize > strCriticalValue ) Then 
  'Wscript.Echo "Inside outer If statement."

  If (FolderSize > strCriticalValue) Then
    'Wscript.Echo "Inside CriticalValue If statement."
    Wscript.Echo "CRITICAL: Folder Size is " & FolderSize &"|Folder.Size=" & FolderSize & ";" &  strWarningValue & ";" & strCriticalValue
    Wscript.Quit(intCritical) 
  End If

  If (FolderSize > strWarningValue) Then
    'Wscript.Echo "Inside WarningValue If statement."
    Wscript.Echo "WARNING: Folder Size is " & FolderSize &"|Folder.Size=" & FolderSize & ";" &  strWarningValue & ";" & strCriticalValue
    Wscript.Quit(intWarning)
  End If

End If

Wscript.Echo "OK: Folder Size is " & FolderSize &"|Folder.Size=" & FolderSize & ";" &  strWarningValue & ";" & strCriticalValue
Wscript.Quit(intOK)

Both programs are invoked from the nagios server by starting the check_nrpe plugin .

The result looks like this:

nagiosDaos

Once there is enough data collected, you can create nice looking graphs.


Nagios, Centreon and DAOS

heA few days ago, I started to play with Nagios. Nagios is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes.

I’ve setup a Linux VM using CentOS 5. As always when you install a package on Linux, there are several dependencies and sometimes it is hard to get it all done. I found a small script in the web that has been created to do the setup of Nagios and Nagios Plugins as well as ndoutils and mysql and stuff.

It also installs Centreon. Centreon (a.k.a Oreon) is a GUI for Nagios.  As you can see from the screenshot, I have successfully installed and configured all components.

nagiosplugin

Nagios and Centreon are Open Source products and can be used at no charge. There are several plugins for all kinds of checks available.

The most flexible wy to check a process state or get statistic values is using SNMP. With SNMP you do not need to install any client on the server. Read the Administration help on how to enable SNMP for Domino.

All of the plugins that came with Nagios uses a threshold to determine the state of a value ( OK, CRITICAL, WARNING … ).

I wanted to check the DAOS Catalog state on my Domino but was not able to do so, because the SNMP request return a non-numeric value. Non-numeric values cannot be passed as a parameter to the plugin.

I decided to write my own plugin from scratch. You can write plugins for Nagios in Perl. The Nagios::Plugin module from CPAN makes it easy to write a Nagios plugin even when you are an unexpierienced Perl developer like me.I learned the basics from the 15 minutes screencast by Ton Voon.

You can download the source code for check_domino_eknori.pl here.

To perform a normal check, start the script with the following syntax

./check_domino_eknori.pl -H 192.168.178.20 -C SINGULTUS -o Mail.Dead -c 9 -w 6

If you want to check for non-numeric return values, you have to tweak the -o parameter

./check_domino_eknori.pl -H 192.168.178.20 -C SINGULTUS -o Synchronized~DAOS.Engine.Catalog -c “Needs Resync” -w Resyncing

Simply put the value that will reflect the OK state before the object and seperate both values with a ~  (Tile).

The output looks like this

DOMINO OK – DAOS.Engine.Catalog – Synchronized

If you ere attending AdminCamp 2009 and you want to get more information about how to monitor you Domino server environment with Nagios, visit Christoph Stoettner’s session.

And if you want to watch a developer talking admin stuff, please vistit my session on DAOS.  ( Read agenda ) Not yet registered? Read here how to save money and register today for AdminCamp 2009 in Gelsenkirchen, Germany!


Back again

I have been asked a few times within the last two days if my blog is dead or so.

I was on vaccation and there were some events I cannot and/or do not want to talk about in public.

This blog is not dead at all and will be updated soon.


Draw structured diagrams for free

Dia is a GTK+ based diagram creation program for Linux, Unix and Windows released under the GPL licens

dia

It can load and save diagrams to a custom XML format (gzipped by default, to save space), can export diagrams to a number of formats, including EPS, SVG, XFIG, WMF and PNG, and can print diagrams (including ones that span multiple pages).


Klingon Anti-Virus

Use Sophos’s Klingon Anti-Virus to quickly perform an on-demand scan and find viruses, spyware, adware, zero-day threats, Betazoid sub-ether porn diallers and Tribbles that your existing protection might have missed. The software can be run without deactivating your current anti-virus software. Phasers can be left set to stun.


AdminCamp 2009 – Jetzt anmelden und € 150,- sparen

Das 4. AdminCamp findet vom 28. bis 30. September 2009 im Maritim Hotel in Gelsenkirchen statt.

Als neuer Teilnehmer erhalten Sie einen zusätzlichen Bonus in Höhe von € 50,- auf den Konferenzpreis, wenn Sie bei ihrer Anmeldung im Feld “Besonderheiten” meinen Namen, also “Ulrich Krause” eingeben.

“Neu”  sind solche Personen, die bisher noch nicht auf einem AdminCamp als Teilnehmer oder Referent vertreten waren.

Melden Sie sich bis zum 06.07.2009 an und sparen Sie insgesamt € 150,- !!


Server Console LOCKED!!

I had created a PMR regarding a little problem with Domino 8.5 and Designer. After a while I got an email from support, kindly asking for more information. I should create a console log and therefore enable some settings on the server.

  • set config console_log_enabled=1
  • set config debug_threadid=1
  • start consolelog

This is some kind of standard procedure for IBM support … I did as I was told and typed the first command on the console and confirmed by hitting the return key.

WHOOPS !!! The console is totally locked.

noaccess

After the first shock was over, I mailed IBM support that I cannot send a console log file because blah blah …

IBM immediately called me on the phone and explained that this can happen and I now have to restart the server.

There is an internal technote for this issue. I’m not sure if I can publish it here, but it says that the error occurs, because debug_outfile= is still in the notes.ini.

After investigating the notes.ini on the server it was found that the parameter in question is not in the ini file. …

Your turn now, IBM.


Attempt to perform folder operation on non-folder note

Every time I edit a view in Domino Designer 8.5, I see the following message on the server console.

05.05.2009 07:17:37   Database note open error: NT0000011A document in database d:\Programme\IBM\Lotus\Domino\data\test.nsf opened by CN=Heinz Ulrich Krause/O=Witte/C=de: Attempt to perform folder operation on non-folder note.

This happens on new created databases as well as old ones.

Steps to reproduce

  • Create a new database on the server ( 8.5 ). No template assigned
  • Open the database in Domino Designer and navigate to the view category
  • Double click the “unknown” view to open.

You can also create a new view and save it. Each time you edit a view the error occurs.

Anyone else seen this, too ? I have already created a PMR ( # 61149 668 668 ) with IBM

UPDATE 07-MAY-2009:

Dear Mr Krause,

I just wanted to send you a quick mail to let you know that your PMR has reached L2 support.
This will be assigned to one of our engineers tomorrow and you will be contacted by them then.

UPDATE 21-MAY-2009:
Dear Mr Krause,

I just wanted to send you a quick mail to keep you up to date regarding your issue. I have since been researching this error message and I have
come across an issue documented in Software Problem Report that seems to match.

This is SPR # MNIR7RSTYU: Error message does appear when copying views: “Attempt to perform folder operation on non-folder note” It documents the exact same error message as you for 85 when changing
properties of a view.

Error message:

Database note open error: NT000363B2 document in database E:\Lotus\Domino\data\……._Template.nsf opened by CN=…..: Attempt to perform folder operation on non-folder note.

This is currently under investigation by development.

I have gone ahead and created a customer report for you to add weight to this SPR. This has APAR # LO40706. The more customer reports are added to this the higher the priority this SPR will become.


SNTT – Determine DAOS FolderSize and NLO Count

After we successfully implemented DAOS on our productive servers, I wanted to create some statistics on how the DAOS repository changes over the time.

I could not find any build in statistics, so I had to find a way to get the data I wanted to collect and build some nice looking charts and graphs from this data.

Lets say, we want to have a historical overview about how the DAOS repository ( i.e. the DAOS Base Directory ) changes in size due to prunes and adding new attachments and we want to know, how many files are stored in the base folder and all subfolders.

If you follow IBM’s recommondation, you have a seperate drive for your DAOS repository. Then you can create a statistic for this drive. But does this stat will show you, how many NLO files are stored on this drive? No!

If you have a DAOS repository on a  drive aside of other data, the drive statistic will not work well for you, because it does not show the data only for the DAOS repository.

Conclusion; we have too determine the DAOS folder size and the file count for the base folder and all subfolders.  I searched Google and found many, many solutions for this.

1. Lotusscript and WMI

Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFolder = objFSO.GetFolder("d:\DAOS")
strFolderSize = Cstr(objFolder.Size)

2. Java

import lotus.domino.*;
import java.io.File;
import org.apache.commons.io.FileUtils;

public class DAOSRepSize extends AgentBase {

	public void NotesMain() {

		try {
			Session session = getSession();
			AgentContext agentContext = session.getAgentContext();
        		String DAOSBaseFolder = "d:/DAOS";

			File f = new File(DAOSBaseFolder);
        		long size = FileUtils.sizeOfDirectory ( f );
			System.out.println("Size: " + size + " bytes");

		} catch(Exception e) {
			e.printStackTrace();
		}
	}
}

Both, Lotusscript + WMI and Java have the disadvantage, that you have to scan all subdirectories seperately. This is very time consuming and took about 1 hour on the live system.
So i tried another approach and finally found the/my solution:

3. Good old DIR and FIND commands

The DIR command combined with the /S parameter displays the content of a folder and all its subdirectories.

The DIR command creates somthing like this on the display; each subdirectory reports FileCount and Size and at the bottom you find a summary of all Files and its size.

10.03.2009 09:25 12.226 FE882EC6BCA2169B4D48B584
2E99.nlo
470 Datei(en) 352.749.584 Bytes

Anzahl der angezeigten Dateien:
470 Datei(en) 352.749.584 Bytes
5 Verzeichnis(se), 54.691.553.280 Bytes frei

You can use the output from the Dir /S as input for the FIND command.

C:\WINDOWS\system32\cmd.exe /C “dir D:\Programme\IBM\Lotus\Domino\data\DAOS /s | find /I “File(s)” > c:\temp\DAOS_SERV01_%Date%.txt”

This commandline creates a textfile with the following content:

0 File(s) 0 bytes
40000 File(s) 14.765.039.757 bytes
40000 File(s) 16.137.319.939 bytes
39982 File(s) 15.409.206.655 bytes
39977 File(s) 12.258.017.367 bytes
39981 File(s) 12.953.697.877 bytes
39911 File(s) 13.429.407.386 bytes
39998 File(s) 12.668.978.548 bytes
39918 File(s) 12.735.446.658 bytes
39939 File(s) 13.405.251.936 bytes
39999 File(s) 14.543.864.272 bytes
40000 File(s) 18.486.530.559 bytes
39997 File(s) 15.527.680.149 bytes
39990 File(s) 11.587.766.203 bytes
39975 File(s) 13.679.295.122 bytes
39984 File(s) 15.774.269.341 bytes
39990 File(s) 14.484.397.076 bytes
39937 File(s) 14.256.113.580 bytes
39999 File(s) 13.542.828.429 bytes
39998 File(s) 15.007.972.785 bytes
39885 File(s) 12.793.036.296 bytes
28249 File(s) 7.986.253.672 bytes
827709 File(s) 291.432.373.607 bytes

Where method 1 and 2 needed almost 1 hour to complete, the DOS commands only run 20 minutes. I run this command as a scheduled task at 1:00 am.

A small agent now reads the files and imports the data into a Notes database.

Sub Initialize
	Dim s As New NotesSession
	Dim db As NotesDatabase
	Dim doc As NotesDocument
	Dim item As NotesItem
	Dim fileNum As Integer
	Dim fileName As String
	Dim pathName As String
	Dim FilePattern As String
	Dim ServerName As String
	Dim FileDate As String
	Dim FileCount As String
	Dim DirSize As String
	Dim text As String
	Dim dummy As Variant

	Set db = s.CurrentDatabase

	pathName = "c:\temp\"
	FilePattern = "DAOS_*.txt"
	fileName = Dir$(pathName & FilePattern, 0)

	Do While fileName <> ""
		Msgbox filename
		dummy = Split(FileName,"_")
		ServerName = dummy(1)
		FileDate = Replace(dummy(2),".txt","")

		fileNum% = Freefile()
		Open PathName & fileName For Input As fileNum%

		Do Until Eof(1)
			Line Input #fileNum%, text
		Loop

		Set doc = db.CreateDocument
		dummy = Split ( Trim(text)," " )
		With doc
			.Form = "DAOS.Folder.Size"
			.FolderSize = Replace(dummy(2),".","")
			.CreatedBy = s.UserName
			.ServerName = ServerName
			.FileCount = dummy(0)
			.FileDate = FileDate
		End With
		Call doc.Save (False, False)		

		Close fileNum%
		Kill PathName & fileName
		fileName = Dir$()
	Loop

End Sub

Another agent which is invoked via a web browser will create a Line Chart. The agent uses Macromedia Fusion Charts.

Sub Initialize
	Dim s As New NotesSession
	Dim db As NotesDatabase
	Dim doc As NotesDocument
	Dim head As Variant
	Dim graph As Variant
	Dim i As Integer
	Dim v As NotesView
	Dim hSize As String
	Dim vSize As String
	Dim yAxisMinValue As String
	Dim yAxisMaxValue As String
	Dim xAxisName As String
	Dim caption As String

	hSize = "800"
	vSize = "600"

	caption = "DAOS"
	xAxisName ="Date"
	yAxisMinValue = "280000000000"
	yAxisMaxValue = "350000000000"

	Set db= s.CurrentDatabase
	Set v =db .GetView("DAOS.Folder.Size")

	head= _
	{}
	Print head
End Sub

Here is what you’ll see in the browser
daosrepository

The chart shows, what the log file on the domino server does not shows.


Lotus Notes Traveler 8.5 performance

Found this article about IBM Lotus Notes Traveler 8.5 performance on developerworks. The article reports on IBM® Lotus Notes® Traveler 8.5 performance for both 32-bit and 64-bit environments. It also compares the performance of the Lotus Notes Traveler 8.5 (32-bit) server and the Lotus Notes Traveler 8.0.1 (32-bit) server.