Monitoring Performance

Below are recommended monitoring procedures for system administrators to follow to ensure proper monitoring of FairCom DB:

• Use system tools and c-tree Server utilities to monitor the state of the FairCom DB process. The ctadmn and ctstat utilities can be used for this purpose.
• Use system tools and c-tree Server utilities to monitor the state of c-tree clients and client activity. The ctadmn utility and server configuration keywords can be used for this purpose.
• Use system tools and FairCom DB utilities to monitor system and FairCom DB resource usage. The ctstat utility can be used to collect historical system resource usage and performance statistics in order to detect unexpected changes in resource usage or performance.
• Monitor the c-tree Server status log for unexpected warning and error messages. The ctsysm utility can be used for this purpose.
• Monitor c-tree API function return codes. The application should log occurrences of unexpected errors.

Consider connecting a ctadmn client to the server and leaving it connected so that client activity can be monitored even if the server enters a state in which it refuses client connections.

Monitoring System Resource Usage

FairCom DB relies upon system resources such as the CPU, disk, memory and the network. Typically each system includes its own tools for monitoring system resources. The following sections provide an overview of system-specific monitoring tools for system resources.

Monitoring CPU Usage

The system administrator should know the expected pattern of CPU use by FairCom DB during normal operation of the system. CPU metrics such as user time, system time, I/O wait time, idle time, and context switches for the server process should be tracked so that unexpected changes in CPU usage can be detected, analyzed, and corrected.

Solaris supports the following utilities for monitoring CPU usage. Other Unix systems support these and similar utilities:

• mpstat reports per-processor statistics including counts for interrupts, context switches, spins on mutexes and reader/writer locks, system calls, and user/system/wait/idle times.
• prstat reports active process statistics including process state, priority, number of lightweight processes, and CPU usage.
• top displays the top processes on the system and periodically updates this information. Raw CPU percentage is used to rank the processes.
• ps prints information about active processes.

The Windows Performance Monitor utility (perfmon) can be used to monitor CPU usage. The Processor performance object maintains counters for each CPU. The available counters include the following (descriptions taken from the Performance Monitor’s explanatory text):

• % Processor Time is the percentage of elapsed time that the processor spends to execute a non-Idle thread. It is calculated by measuring the duration of the idle thread is active in the sample interval, and subtracting that time from interval duration. (Each processor has an idle thread that consumes cycles when no other threads are ready to run). This counter is the primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval. It is calculated by monitoring the time that the service is inactive, and subtracting that value from 100%.
• % Privileged Time is the percentage of elapsed time that the process threads spent executing code in privileged mode. When a Windows system service in called, the service will often run in privileged mode to gain access to system-private data. Such data is protected from access by threads executing in user mode. Calls to the system can be explicit or implicit, such as page faults or interrupts. Unlike some early operating systems, Windows uses process boundaries for subsystem protection in addition to the traditional protection of user and privileged modes. Some work done by Windows on behalf of the application might appear in other subsystem processes in addition to the privileged time in the process.
• % User Time is the percentage of elapsed time the processor spends in the user mode. User mode is a restricted processing mode designed for applications, environment subsystems, and integral subsystems. The alternative, privileged mode, is designed for operating system components and allows direct access to hardware and all memory. The operating system switches application threads to privileged mode to access operating system services. This counter displays the average busy time as a percentage of the sample time.

Monitoring Disk Usage

The system administrator should know the expected pattern of disk use by FairCom DB during normal operation of the system. Expected data and index file sizes and disk I/O should be tracked so that unexpected changes in disk usage can be detected, analyzed, and corrected.

Solaris supports the following utilities for monitoring disk usage. Other Unix systems support these and similar utilities:

• vmstat reports virtual memory statistics regarding process, virtual memory, disk, trap, and CPU activity.
• iostat iteratively reports terminal, disk, and tape I/O activity, as well as CPU utilization.

The Windows Performance Monitor utility (perfmon) can be used to monitor system disk usage. The PhysicalDisk performance object maintains counters for each physical disk. The available counters include the following (descriptions taken from the Performance Monitor’s explanatory text):

• % Disk Read Time is the percentage of elapsed time that the selected disk drive was busy servicing read requests.
• % Disk Write Time is the percentage of elapsed time that the selected disk drive was busy servicing write requests.
• % Idle Time reports the percentage of time during the sample interval that the disk was idle.
• Avg. Disk Queue Length is the average number of both read and write requests that were queued for the selected disk during the sample interval.

The LogicalDisk performance object maintains counters for each logical disk. The available counters include the following:

• Free Megabytes displays the unallocated space, in megabytes, on the disk drive in megabytes. One megabyte is equal to 1,048,576 bytes.
• Current Disk Queue Length is the number of requests outstanding on the disk at the time the performance data is collected. It also includes requests in service at the time of the collection. This is a instantaneous snapshot, not an average over the time interval. Multi-spindle disk devices can have multiple requests that are active at one time, but other concurrent requests are awaiting service. This counter might reflect a transitory high or low queue length, but if there is a sustained load on the disk drive, it is likely that this will be consistently high. Requests experience delays proportional to the length of this queue minus the number of spindles on the disks. For good performance, this difference should average less than two.

The Cache performance object maintains counters for the file system cache. The available counters include the following:

• Data Flushes/sec is the rate at which the file system cache has flushed its contents to disk as the result of a request to flush or to satisfy a write-through file write request. More than one page can be transferred on each flush operation.
• Lazy Write Flushes/sec is the rate at which the Lazy Writer thread has written to disk. Lazy Writing is the process of updating the disk after the page has been changed in memory, so that the application that changed the file does not have to wait for the disk write to be complete before proceeding. More than one page can be transferred by each write operation.

In addition to system tools available for monitoring disk usage, FairCom DB supports options to limit disk usage. The server’s Disk Full feature offers three levels of control over disk full checks:

• The DISK_FULL_LIMIT keyword provides checking on all files.
• The DISK_FULL_VOLUME keyword sets a volume-specific limit that overrides the system-wide limit.
• The file-specific check (set by creating a file using an Xtd8 create function with the dskful member of the XCREblk structure set to the desired file size limit in bytes) overrides the system-wide and volume-specific checks.

If extending the size of the file would leave less than the specified threshold, then the write operation causing the file extension fails, returning SAVL_ERR (583).

Monitoring Memory Usage

The system administrator should know the expected memory use by FairCom DB during normal operation of the system. Memory use should be tracked so that unexpected changes in memory usage can be detected, analyzed, and corrected.

Solaris supports the following utility for monitoring memory usage. Other Unix systems support similar utilities:

• vmstat reports virtual memory statistics regarding process, virtual memory, disk, trap, and CPU activity.

The Windows Performance Monitor utility (perfmon) can be used to monitor system memory usage. The Memory performance object maintains counters for memory usage. The available counters include the following (descriptions taken from the Performance Monitor’s explanatory text):

• AvailableMBytes is the amount of physical memory available to processes running on the computer, in Megabytes, rather than bytes as reported in Memory\\Available Bytes. It is calculated by adding the amount of space on the Zeroed, Free, and Stand by memory lists. Free memory is ready for use; Zeroed memory are pages of memory filled with zeros to prevent later processes from seeing data used by a previous process; Standby memory is memory removed from a process’ working set (its physical memory) on route to disk, but is still available to be recalled. This counter displays the last observed value only; it is not an average.
• Pages/sec is the rate at which pages are read from or written to disk to resolve hard page faults. This counter is a primary indicator of the kinds of faults that cause system-wide delays. It is the sum of Memory\\Pages Input/sec and Memory\\Pages Output/sec. It is counted in numbers of pages, so it can be compared to other counts of pages, such as Memory\\Page Faults/sec, without conversion. It includes pages retrieved to satisfy faults in the file system cache (usually requested by applications) non-cached mapped memory files.

In addition to the available system monitoring tools, FairCom DB supports monitoring server memory usage using the FairCom DB SystemConfiguration() API function. See the section "Monitoring FairCom DB Using SystemConfiguration API" for more details.

See Also

• Memory Use of Linux Processes
   

Monitoring Network Usage

The system administrator should know the expected network use by FairCom DB during normal operation of the system. Network use should be tracked so that unexpected changes in network usage can be detected, analyzed, and corrected.

Solaris supports the following utility for monitoring network usage. Other Unix systems support similar utilities:

• netstat displays the contents of network-related data structures in various formats, depending on the specified options.

The Windows Performance Monitor utility (perfmon) can be used to network usage. The Network Interface performance object maintains counters for network usage. The available counters include the following (descriptions taken from the Performance Monitor’s explanatory text):

• Bytes Received/sec is the rate at which bytes are received over each network adapter, including framing characters. Network Interface\\Bytes Received/sec is a subset of Network Interface\\Bytes Total/sec.
• Bytes Sent/sec is the rate at which bytes are sent over each each network adapter, including framing characters. Network Interface\\Bytes Sent/sec is a subset of Network Interface\\Bytes Total/sec.
• Output Queue Length is the length of the output packet queue (in packets). If this is longer than two, there are delays and the bottleneck should be found and eliminated, if possible. Since the requests are queued by the Network Driver Interface Specification (NDIS) in this implementation, this will always be 0.
• Packets Outbound Errors is the number of outbound packets that could not be transmitted because of errors.
• Packets Received Errors is the number of inbound packets that contained errors preventing them from being deliverable to a higher-layer protocol.
• Packets Received/sec is the rate at which packets are received on the network interface.
• Packets Sent/sec is the rate at which packets are sent on the network interface.

Other System Monitoring Options

In addition to the system monitoring utilities described in the above sections that are target specific resource monitoring, there are also other system utilities that can be used to monitor a variety of system resources.

On Solaris and Unix systems, the sar utility is useful for monitoring system activity, including system buffer transfer activity, cache hit ratios, system calls, and device activity.

The Windows Performance Monitor includes performance objects such as the Object, Process, Server, System, and Thread objects that can be used to monitor system resource usage in specific ways. Also, the Windows Task Manager can be used to monitor system resource usage.

Monitoring FairCom DB Internal Resource Usage

FairCom DB supports utilities and APIs that can be used to monitor its internal resources (such as data and index caches, lock table, transaction and file activity). The following sections explain how to use these utilities and APIs to monitor the server’s internal resources.

Monitoring FairCom DB Using Snapshot Support

The FairCom DB Performance Snapshot capability enables the capture of performance monitoring data using a combination of configuration file options and calls to the SnapShot() function. The performance data can be captured automatically at specified intervals or on demand.

Performance data maintained and made available by the server’s snapshot capability includes the following: File lock statistics, transaction time histogram, transaction commit delay statistics, transaction log I/O, function timings, data and index cache details, disk read and write statistics, and communication read and write statistics.

FairCom DB supports the snapshot capability on systems that support 8-byte integers. On systems that do not support a high-resolution timer, some aspects of the snapshot are not available.

Snapshot Configuration File Options

Server configuration file options can be used to capture automatic system snapshot data.

When SNAPSHOT_INTERVAL <interval in minutes> is specified in the server configuration file, an automatic snapshot thread performs a system snapshot after each interval. The automatic system snapshot is written to FairCom DB’s system event log (SYSLOG) files.

The following configuration entries can be used multiple times to include snapshot details to for users and files in the snapshot output. Use a pattern of ‘*’ to activate monitoring for all user ID’s or files.

SNAPSHOT_USERID <user ID>
SNAPSHOT_FILENAME <file name>

Specifying DIAGNOSTICS SNAPSHOT_SHUTDOWN in the server configuration file causes the server to log system snapshot details to the file SNAPSHOT.FCS when the server shuts down.

Snapshot API Function Options

The SnapShot() API function can be used to control automatic snapshots, overriding configuration options (if any), and can capture on-demand server performance data to the SYSLOG files, to the human-readable SNAPSHOT.FCS file, or as a return to the calling program. See the FairCom DB Function Reference Guide for full details on using the SnapShot() API function.

Monitoring FairCom DB Using ctstat Utility

The ctstat utility is a client utility used to display statistics collected by FairCom DB. Depending on the specified command-line options, ctstat displays file, system, user, or function timing statistics. ctstat uses the SnapShot API function to log server statistics to the server’s system event log, then reads the statistics from the logs and displays them in human-readable format.

See Performance Monitoring Using the ctstat Utility in the FairCom DB server Administrators Guide for full details on using the ctstat utility.

Monitoring FairCom DB Using SystemConfiguration API

The FairCom DB API function SystemConfiguration() returns an array of LONGs describing the current configuration, as well as some of the important dynamic aspects of the system such as the memory usage an the number of files in use. The following sections list SystemConfiguration() return values relevant to specific types of monitoring such as server lock table, file usage, and cache usage. For a detailed list of the values returned by SystemConfiguration(), see the FairCom DB Function Reference Guide.

Monitoring FairCom DB Memory Use

The SystemConfiguration() API function can be used to display current FairCom DB memory usage figures as shown in the following table:

Monitoring FairCom DB Lock Table

FairCom DB uses a memory-resident lock table for managing locks on FairCom DB data records. The following sections discuss options for monitoring the state of the server’s lock table.

LockDump API Options

The FairCom DB API function LockDump() creates a diagnostic dump of the FairCom DB internal lock table. Locks can be dumped for a particular file or user or for all files in the system.

For a FairCom DB with snapshot support enabled, the LockDump() function also logs disk I/O statistics for each file included in the lock dump. These disk I/O statistics consist of the number of read and write operations and bytes read and written for each file.

Below is sample lock dump output for the c-tree data file mark.dat:

----------------
----------------
mark.dat>>
        0000-00094480x T013 write/1: W18 W19
        0000-00094580x T014 write/1
        0000-00094200x T010 write/1
        0000-00094600x T012 write/1
cumulative lock attempts: 9462(4731)  blocked: 0(0)  dead-lock: 0  denied: 0
Current file lock count: 4
----------------
cumulative I/O: read ops: 76 bytes: 614528    write ops: 38 bytes: 614400

The output shows four write locks held on mark.dat. The log lists the following details for each lock currently held on a file:

• The record offset of the locked record (0000-00094480x)
• The ID of the thread that is holding the lock (T013)
• The type of lock (write/1)
• The IDs of any threads that are waiting to acquire the lock (W018 W019)

The possible lock types are shown in the following table. Note that the first 5 lock types in the table are supported only for a FairCom DB built with strict serialization support:

See also:

• LockDump Output

SnapShot API Options

The SnapShot() function can be used to retrieve lock details. See the discussion of the SNAPSHOT.FCS file format for details.

SystemConfiguration API Options

The SystemConfiguration() function can be used to display the current number of pending locks:

Monitoring FairCom DB Client Activity

FairCom DB provides a variety of ways to monitor FairCom DB client activity. The following sections discuss the available options.

ctadmn Options for Monitoring Client Activity

The FairCom DB Administration Utility, ctadmn, can be used to monitor FairCom DB client activity by following these steps:

1. Start ctadmn. When prompted, enter the user ID of a member of the ADMIN group, the user’s password, the optional FAIRCOM.FCS file password, and name of FairCom DB.
2. Select option 4, “Monitor Clients”, from the ctadmn main menu.
3. Select option 1, “List Attached Clients”, from the Monitor Clients menu. ctadmn displays an entry such as the following for each connected client:

UserID: GUEST                             NodeName:
Task 11                                   Communications: F_TCPIP
         Memory:  25K       Open Files: 2      Logon Time:   0:02
         Tran Time:   0:01  Rqst Time:   0:00  InProcess  Rqst# 13  TRANEND

The meaning of each field is shown in the following table:

Server Configuration Options

The following server configuration options can be used to monitor client activity:

• Specifying DIAGNOSTICS LOGON_COMM in the server configuration file causes the server to write detailed logon status messages to its console. This option is useful for tracking down the cause of failed client connection attempts.
• Specifying DIAGNOSTICS TRAP_COMM in the server configuration file causes the server to log all communication messages received from FairCom DB clients to the file TRAPCOMM.FCS. This option provides a way to capture the exact sequence of client requests, and if a copy of the initial data and index files are preserved, the cttrap utility can be used to replay this TRAPCOMM.FCS log in order to reproduce the original client activity. Note that using this option may impact server performance.
• Specifying FUNCTION_MONITOR YES in the server configuration file causes FairCom DB to display details about each client request to the function monitor window or standard output. Another variation is to specify FUNCTION_MONITOR <logfile>, which causes the server to log function details to the file named <logfile>. When function monitor output is directed to a log file, the function return codes are included, which makes this option useful for tracking down the occurrence of unexpected FairCom DB errors. Note that using this option may impact server performance.

SystemConfiguration Options

The FairCom DB SystemConfiguration() API function can be used to monitor FairCom DB client activity. SystemConfiguration() returns three values referenced with the following constants used as subscripts in the output array of LONG values as shown in the following table:

Monitoring FairCom DB Transaction Activity

This section discusses the available options for monitoring FairCom DB’s transaction activity.

Server Configuration Options

The CHECKPOINT_MONITOR server configuration keyword can be used to cause the server to log the starting and completion times of checkpoint operations to the server status log and the server console. This option can be used to determine how often checkpoints occur. If checkpoints occur very frequently, the server’s performance can be affected. In such a situation, the CHECKPOINT_INTERVAL and LOG_SPACE keywords can be used to reduce the frequency of checkpoint operations.

SnapShot API Options

The SnapShot() function can be used to retrieve transaction activity details. See the discussion of the SNAPSHOT.FCS file format for details.

Monitoring FairCom DB Transaction Numbers and Transaction File Numbers

The server assigns a transaction number to each transaction and it assigns a transaction file number each time a transaction file is modified. The limits for these numbers are very high, so it should take years to reach them even on a system with heavy traffic. Nonetheless, these numbers should be monitored to ensure that they do not reach their limits:

1. Transaction numbers can go up to: 1,125,899,906,842,623
2. Transaction file numbers can go up to: 4,294,963,200

Both numbers can be easily monitored by looking at CTSTATUS.FCS. The server prints error messages to CTSTATUS.FCS well before it reaches limits for these numbers.

The limitation to this method is that it does not give any percentage value or any estimation on how far away the limit is. The current values of these numbers can be monitored using the ctstat -vat option. This option is documented FairCom Server Administration Guide in the ctstatTransaction Statistics Example.

The last two values returned by the ctstat -vat option, tranno (transaction number) and tfil (transaction file number), indicate the next available numbers. Knowing the max value for these numbers, it should be quick to calculate the status.

See also:

• 6-Byte Transaction Numbers
• Transaction Number Limitations
• Configurable Transaction Number Overflow Warning Limit
• ctstat Transaction Statistics Example

Monitoring FairCom DB File Usage

This section discusses options for monitoring FairCom DB data and index file usage by FairCom DB and its clients.

Server Configuration Options

The following server configuration options support monitoring FairCom DB’s usage of data and index files:

• DIAGNOSTICS LOWL_FILE_IO is useful in troubleshooting file open, create, close, delete, and rename errors. This keyword causes the server to log to the server status log, CTSTATUS.FCS, the filename and system error code for failed file open, create, close, delete, and rename operations.
• DIAGNOSTICS WRITETHRU causes FairCom DB to log to the server status log the names of all PREIMG or non-transaction FairCom DB data and index files that are opened or created during server operation without the WRITETHRU filemode in effect.
• DIAGNOSTICS EXTENDED_TRAN_NO causes each physical open of a non-extended-transaction-number file to be listed in CTSTATUS.FCS. The reason to check for a file that does not support extended transaction numbers is that if all files do not support extended transaction numbers, then the exceptions could cause the server to terminate if the transaction numbers go out of the original 4-byte range and one of these files are updated. By “all files” we mean superfile hosts and indexes; data files are not affected by the extended-transaction-number attribute.

SystemConfiguration API Options

The FairCom DB SystemConfiguration() API function can be used to monitor FairCom DB data and index file usage by FairCom DB. SystemConfiguration() returns three values referenced with the following constants used as subscripts in the output array of LONG values as shown in the following table:

Monitoring FairCom DB Dynamic Dumps

FairCom DB logs dynamic dump error messages and error codes to the server status log, CTSTATUS.FCS. In the event of a failed dynamic dump, examine the server status log.

FairCom DB can be configured to log more detailed dynamic dump progress entries to the server status log, including an entry for each file included in the dump, by adding DIAGNOSTICS DYNDUMP_LOG to the server configuration file before starting the server.

Monitoring FairCom DB Automatic Recovery

If RECOVER_DETAILS YES is present in the server configuration file when the server is started, the server logs the time spent for each phase of recovery to the server status log. This option provides a way to more specifically monitor the progress of automatic recovery. The server writes the entry for each recovery phase upon completion of that recovery phase, so the current recovery phase can be determined by examining the contents of the server status log.

See the section "Server Startup Hangs or Takes Excessive Time" of this document for more details on monitoring automatic recovery.

Monitoring FairCom DB Cache Usage

The FairCom DB SystemConfiguration() API function can be used to monitor the usage of FairCom DB data and index caches. SystemConfiguration() returns nine values referenced with the following constants used as subscripts in the output array of LONG values an application can use to capture system-wide cache and buffer statistics, (cache pages hold data record images and buffers hold index nodes), allowing an application to track the use of these resources.

Monitoring FairCom DB Status Log Messages

During server operation, FairCom DB writes messages to the server status log, CTSTATUS.FCS. The messages the server logs may be informational, warning, or error messages. The system administrator should monitor the server status log to detect situations in which the server writes unexpected warning or error messages to the status log.

The ctsysm utility can be used to monitor status log messages. This utility reads a configuration file containing the possible status log messages and associated actions depending on the context of the message. As the server logs messages to the status log, the utility examines the messages and outputs the corresponding message code, which can be matched to the appropriate action, if any, for the message. For details on using the ctsysm utility to monitor FairCom DB status log, see FairCom Server Status Monitoring Utility, ctsysm in the FairCom DB System Administrators Guide.

Monitoring FairCom DB Process State

The state of the FairCom DB process can be monitored using system utilities and FairCom DB utilities. System utilities commonly available on Unix systems include the following:

• ps lists the current processes on the system. This utility can be used to confirm that the server process exists and is in a running state.
• truss traces system calls and signals for a command or process. This utility can be used to determine what operations the server is currently performing.
• pstack prints a hex and symbolic stack trace for each lwp in a process. This utility can be used to determine what each server thread is currently doing. Also see the proc man page for details on other process monitoring tools.
• gcore creates a core image of the specified process. This utility can be used to view a detailed snapshot of the server’s operation at a point in time.

The Windows Performance Monitor and Task Manager can be used to observe the state of the FairCom DB process and its current activity.

The ctstat utility can be used to monitor server activity. By observing FairCom DB statistics over time, the server’s activity can be measured. For example, transaction commits, communication read/write operations, and disk read and write operations can be examined to determine what the server is currently doing.

The ctadmn utility can be used to monitor client activity. Details can be listed for all client connections, including current request, elapsed request time, and elapsed transaction time.

Monitoring FairCom Server with strace

A trace of the server process can yield valuable information about performance bottlenecks. A good strategy is to use a utility to monitor the system call times while running a test utility, such as the cttctx test.

On Linux and some Unix platforms, the strace utility can be used to detail system calls. The -p flag will attach strace to a running process, which will be FairCom Server in this case. This allows you to monitor a specific instance of FairCom Server to analyze performance. Similar tools are available on other platforms such as Windows and Unix. Some Unix systems provide a similar utility named truss, although arguments may differ.

You will need to know the process ID (PID) of the FairCom Server thread that will be monitored, as explained in the sections below. To find the PID, you may need to know the binary name of the FairCom Server:

• The binary name for the ISAM server is ctsrvr for Unix/Linux and ctsrvr.exe for Windows.
• The binary name for the SQL server is ctreesql for Unix/Linux and ctreesql.exe for Windows.

To run strace on FairCom Server, use the following command:

strace -f -p PID -c

• where PID is the FairCom Server process ID (the sections below explain how to obtain the process ID).
• -f - traces child processes.
• -c - adds call timing information.

This command will run in the background until you press <CTRL>+C, and then it will output the system call counts and times.

Finding the FairCom Server Process ID

To use strace you will need to know the process ID (PID) of the FairCom Server thread you need to monitor. The sections below show several methods of determining the FairCom Server process ID.

Searching CTSTATUS.FCS

You can obtain the FairCom Server process ID by looking at the most recent start of FairCom Server within CTSTATUS.FCS. If you have more than one FairCom Server running on the machine, the best way to find the process ID is to look in CTSTATUS.FCS. You can search for "Process ID:" or simply "ID:" as shown below:

Fri May  8 08:08:00 2015
 - User# 00001  Process ID: 18305

Viewing the Process Status (Unix/Linux)

The process status command, ps, shows information about the currently-running processes. That output can be searched, by piping it to grep, to find the FairCom Server process.

For the FairCom DB SQL Server use:

ps -ef | grep ctreesql | grep –v grep

For the FairCom DB ISAM Server use:

ps -ef | grep ctsrvr | grep –v grep

The following snippet shows an example of executing the ps command on a Unix/Linux system and searching the results for "ctree":

ps -ef | grep ctree
fctech 18305 13447 6 08:08 pts/4 00:00:00 ./ctreesql
fctech 18324 13447 0 08:08 pts/4 00:00:00 grep ctree

In this example, the second row containing "grep" is the grep command, which can be ignored. The first row is the ctreesql process.

Using Task Manager (Windows)

On Windows, you can load Task Manager to find the FairCom DB process.

Monitoring System Call Times

The cttctx test can be used to monitor call times. To see the system call times for the thread that is running the test, follow these steps:

1. Start FairCom Server and note its process ID (see the earlier sections).
2. Run top with the following command-line to see the FairCom Server threads:

top -H -p CTREE_SERVER_PROCESS_ID

3. While top is running, start the cttctx test. Note the process ID of the thread that is at the top of the top list. For example:

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
7591 fctech    20   0 1095m  33m 3800 D  1.3  0.9   0:00.11 ctsrvr
7567 fctech    20   0 1095m  33m 3800 S  0.7  0.9   0:00.09 ctsrvr
7568 fctech    20   0 1095m  33m 3800 S  0.0  0.9   0:00.00 ctsrvr
7569 fctech    20   0 1095m  33m 3800 S  0.0  0.9   0:00.00 ctsrvr

4. Run strace for process ID of the thread identified in the previous step:

strace -p 7591 -c

When the test finishes, stop strace and save the output. Collect the strace output for all cases of interest.

Using ctstat to Capture Statistics

The procedures below can be conducted to analyze performance when the system is running slow:

1. Use the following command to enable the internal FairCom DB function call to capture metrics.

ctstat -wrktime on -u ADMIN -p ADMIN -s FAIRCOMS

where "FAIRCOMS" should be replaced with the name of your FairCom Server.

2. Use this command to log all FairCom DB Server statistics (including function call timings) to the text file SNAPSHOT.FCS every minute:

ctstat -text -u ADMIN -p ADMIN -s FAIRCOMS -i 60

This command will not return to the command line. Allow it run for at least 10 minutes so you have several traces to analyze.

To stop execution, press CTRL+C to interrupt the program.

3. After the test in step 2 is terminated, disable the "wrktime" server metrics using this command:

ctstat -wrktime off -u ADMIN -p ADMIN -s FAIRCOMS

The SNAPSHOT.FCS file can be analyzed to locate performance problems.

The Windows procdump Utility

The procdump utility on Windows can be set to trigger on a performance counter which might be handy if the response times are available in that form. The default procdump dump (without process memory) should be sufficient to see what is going on.

The procdump utility generates stack traces. For generating system call traces, use the Process Monitor utility. You can find more information by searching Microsoft Technet.