FairCom — Diagnostics

Diagnostic keywords are used to provide additional detailed FairCom DB monitoring of specific internal parameters and metrics. Typically, these keywords should only be used under the specific advice of a FairCom engineer as they may negatively impact the performance of your FairCom DB Server. Please do not hesitate to contact your nearest FairCom office should you have any questions regarding use of any of these keywords.

WARNING: The keywords listed below should be used ONLY on the advice of your application developer. They can seriously alter the operation of the FairCom DB Server.

CHECK_TRNLOG_BUFFER_WRITES

CHECK_TRNLOG_BUFFER_WRITES YES|NO

This option by default is NO.

If set to YES in the ctsrvr.cfg file, the contents of transaction log writes are checked to see if a log write is skipping bytes; that is, writing at an offset that skips bytes from the last written offset. And the server checks if the last log entry written has an invalid transaction type. If either check fails, the server creates a stack dump and logs a message to CTSTATUS.FCS.

DIAGNOSTIC_INT

The default internal buffer for trapping the communications is 128K bytes. An entry of the following form will override the default buffer size.

DIAGNOSTIC_INT <buffer size in K bytes>

Each time the buffer fills, it is dumped to the trap file. For example,

DIAGNOSTIC_INT 48

would create a 48K byte buffer. The name of the trap file is derived from the communication protocol name by adding .FCS.

Note: Any number of DIAGNOSTIC_INT entries can be included in the configuration file. These create an internal dynamically allocated array used for providing input to various debugging routines. These options should only be used under advice from a FairCom engineer.

DIAGNOSTIC_STR

By default, the name of the communications trap file is TRAPCOMM.FCS and will be located in the server directory. To prepend a path onto the trap file name (say to route it to a separate disk), add an entry of the form

DIAGNOSTIC_STR <trap file path>

For example, to append the file system name /bigdisk/ to the filename, include the following entry in the configuration file:

DIAGNOSTIC_STR /bigdisk/

The trap file would be then be located in /bigdisk/TRAPCOMM.FCS

Note: Any number of DIAGNOSTIC_STR entries can be included in the configuration file. These create an internal dynamically allocated array used for providing input to various debugging routines. Additional entries should only be used under advice from a FairCom engineer.

DIAGNOSTICS LDAP

LDAP diagnostic logging is now available. LDAP diagnostic log messages are written to CTSTATUS.FCS and start with "LDAP_DIAG:"

Recommendation: It is best to use this feature only when resolving connection issues and then turn it off for production use. This practice minimizes the increase in information this feature writes to the log.

Specify DIAGNOSTICS LDAP in ctsrvr.cfg to enable the diagnostic logging at server startup. This logging can be enabled at runtime using ctadmn:

10. Change Server Settings

9. Change a DIAGNOSTICS option

Enter the DIAGNOSTICS option to enable or disable >> LDAP

(or ~LDAP to turn it off)

The annotated example below shows LDAP diagnostic logging messages that are written when a user connects and its LDAP groups are checked and updated in FAIRCOM.FCS (when the LDAP_GROUP_CHECK configuration option is used):

The LDAP_GROUP_CHECK setting:

- User# 00020 LDAP_DIAG: chkldapusr: LDAP_GROUP_CHECK=[(objectclass=groupOfNames)]

- User# 00020 LDAP_DIAG: chkldapusr: pgroupflt=[(&(objectclass=groupOfNames)(member=cn=jeff,ou=people,dc=faircom,dc=com))]

The group base and filter that are sent to the LDAP server:

- User# 00020 LDAP_DIAG: getldapusergroups: grp=[ou=groups,dc=faircom,dc=com], flt=[(&(objectclass=groupOfNames)(member=cn=jeff,ou=people,dc=faircom,dc=com))]

- User# 00020 LDAP_DIAG: getldapusergroups: ngroups=[5]

The groups read from the LDAP server:

- User# 00020 LDAP_DIAG: getldapusergroups: group[0]: dn=[cn=dev,ou=groups,dc=faircom,dc=com]

- User# 00020 LDAP_DIAG: getldapusergroups: dp=[dev]

- User# 00020 LDAP_DIAG: getldapusergroups: group[1]: dn=[cn=support,ou=groups,dc=faircom,dc=com]

- User# 00020 LDAP_DIAG: getldapusergroups: dp=[support]

- User# 00020 LDAP_DIAG: getldapusergroups: group[2]: dn=[cn=qalongerthanoursupportedmaximumgroupname,ou=groups,dc=faircom,dc=com]

- User# 00020 LDAP_DIAG: getldapusergroups: dp=[qalongerthanoursupportedmaximumg]

- User# 00020 LDAP_DIAG: getldapusergroups: group[3]: dn=[cn=ctreeisamusers,ou=groups,dc=faircom,dc=com]

- User# 00020 LDAP_DIAG: getldapusergroups: dp=[ctreeisamusers]

- User# 00020 LDAP_DIAG: getldapusergroups: group[4]: dn=[cn=ctreesqlusers,ou=groups,dc=faircom,dc=com]

- User# 00020 LDAP_DIAG: getldapusergroups: dp=[ctreesqlusers]

The groups returned to the calling function:

- User# 00020 LDAP_DIAG: chkldapusr: numldapgroups=5

- User# 00020 LDAP_DIAG: chkldapusr: ldapgroup[0]=[dev]

- User# 00020 LDAP_DIAG: chkldapusr: ldapgroup[1]=[support]

- User# 00020 LDAP_DIAG: chkldapusr: ldapgroup[2]=[qalongerthanoursupportedmaximumgctreeisamusers]

- User# 00020 LDAP_DIAG: chkldapusr: ldapgroup[3]=[ctreeisamusers]

- User# 00020 LDAP_DIAG: chkldapusr: ldapgroup[4]=[ctreesqlusers]

The updating of groups in FAIRCOM.FCS:

- User# 00020 LDAP_DIAG: updatectreeusergroups: ctreegroups=0, ldapgroups=5

- User# 00020 LDAP_DIAG: updatectreeusergroups: deleted all c-tree groups for user [JEFF]

- User# 00020 LDAP_DIAG: updatectreeusergroups: added c-tree group [CTREEISAMUSERS]

- User# 00020 LDAP_DIAG: updatectreeusergroups: added c-tree group [CTREESQLUSERS]

- User# 00020 LDAP_DIAG: updatectreeusergroups: added c-tree group [DEV]

- User# 00020 LDAP_DIAG: updatectreeusergroups: added c-tree group [QALONGERTHANOURSUPPORTEDMAXIMUM]

- User# 00020 LDAP_DIAG: updatectreeusergroups: added c-tree group [SUPPORT]

DIAGNOSTICS ABEND_ABORT (Deprecated)

DEPRECATED beginning with V13.0

DIAGNOSTICS ABEND_ABORT

Enables a method for generating a process core on an abnormal shutdown by having FairCom DB call abort(). This does cause the attempt at closing files and cleaning up to be skipped, so recovery may take a longer time to run when this keyword is specified.

DIAGNOSTICS ABORT_NODE_LIST

Enables a circular memory buffer to hold information relevant to debugging a L59 error. In the event of a L59 failure, the memory log will be dumped to the text file ABNODLST.FCS. In particular, placing in the server configuration file turns on the memory log. Since it is a circular buffer, only the most recent entries are maintained. Each entry requires 36 bytes.

The default number entries can be overridden through the configuration entry

DIAGNOSTIC_INT <override value>

If the L59 occurs only in a well-defined number of indexes, then the DIAGNOSTIC_STR key word can also be used to list the relevant index files. For example, the following server configuration entries would cause the memory log to hold 20,000 entries for the listed indexes:

DIAGNOSTICS ABORT_NODE_LIST
DIAGNOSTIC_INT 20000
DIAGNOSTIC_STR customer.idx
DIAGNOSTIC_STR invoice.idx
DIAGNOSTIC_STR product.idx

See Also

DIAGNOSTICS AUTO_PREIMG_CHECKLOCK / AUTO_PREIMG_CHECKREAD

DIAGNOSTICS AUTO_PREIMG_CHECKLOCK

DIAGNOSTICS AUTO_PREIMG_CHECKREAD

When either of these diagnostic options are used, the file mode for files affected by AUTO_PREIMG configuration entries will be augmented by one or both of ctCHECKLOCK and ctCHECKREAD. Missing lock calls will then generate DADV_ERR (57) errors.

Note: It is not the intention of these diagnostic options to change the file mode stored on disk for the files in question. The ctCHECKLOCK and/or ctCHECKREAD modes should not be added to the files’ header images.

See Also

AUTO_PREIMG

AUTO_TRNLOG

AUTO_TRNLOG_LIGHT

DIAGNOSTICS AUTO_TRNLOG_CHECKLOCK

DIAGNOSTICS AUTO_TRNLOG_CHECKREAD

PREIMAGE_DUMP

DIAGNOSTICS AUTO_TRNLOG_CHECKLOCK / AUTO_TRNLOG_CHECKREAD

DIAGNOSTICS AUTO_TRNLOG_CHECKLOCK

DIAGNOSTICS AUTO_TRNLOG_CHECKREAD

When either of these diagnostic options are used, the file mode for files affected by the AUTO_TRNLOG configuration entry will be augmented by one or both of ctCHECKLOCK and ctCHECKREAD. Missing lock calls will then generate DADV_ERR (57) errors.

See Also

AUTO_PREIMG

AUTO_TRNLOG

AUTO_TRNLOG_LIGHT

DIAGNOSTICS AUTO_PREIMG_CHECKLOCK

DIAGNOSTICS AUTO_PREIMG_CHECKREAD

PREIMAGE_DUMP

DIAGNOSTICS CHECK_KEY_MARKS

In V11.8 and later, a new diagnostic option, DIAGNOSTICS CHECK_KEY_MARKS, causes c-tree Server to check for unexpected transaction marks in a leaf node when it is about to be written to disk. If any unexpected marks are found, a message is written to CTSTATUS.FCS and a process stack trace is created. Sample message:

- User# 00032 CHECK_KEY_MARKS: index file lstrip.idx member 18 node 0x160000 mark 01 tranno 1 lowexc 0 key 1

The message shows the index file name, member number, node offset, transaction mark, transaction number, and the element number within the node of the key that has the unexpected mark.

The server now logs a process stack trace if it detects a call to set a transaction mark for a pending add with a transaction number of 1.

The diagnostic option can be changed at runtime.

DIAGNOSTICS CHECK_UDEFER

Used to tune UDEFER options.

Note: Internal FairCom engineering use only. Not intended for production use.

DIAGNOSTICS COMM_LEVEL_X

DIAGNOSTCIS COMM_LEVEL_X

Enables communication diagnostics. Valid level:

COMM_LEVEL_1

DIAGNOSTICS CTQUIET

When a ctQuiet call is aborted with error 843, this keyword causes a stack trace to be generated that can be sent to FairCom.

DIAGNOSTICS DADV_ERR

The configuration option DIAGNOSTICS DADV_ERR can be used to enable logging of a diagnostic message to CTSTATUS.FCS when a DADV_ERR error occurs. The message includes the name of the data file, the record offset at which the update was attempted without holding a lock, the c-tree function number and subfunction number, the caller's user ID, and the caller's node ID. Sample message:

Fri Mar 31 10:17:28 2017

- User# 00022 DADV_ERR: file=qaopnfil.dat offset=18863 func=121(0) userid=ADMIN nodeid=qaopnfil043: 57

This diagnostic logging can be enabled and disabled at runtime.

To enable diagnostic logging:

ctSETCFG(setcfgDIAGNOSTICS, "CHECKLOCK");

To disable diagnostic logging:

ctSETCFG(setcfgDIAGNOSTICS, "~CHECKLOCK");

Using the ctadmn utility to enable or disable diagnostic logging:

10. Change Server Settings

9. Change a DIAGNOSTICS option

Enter the DIAGNOSTICS option to enable or disable >> CHECKLOCK

Successfully changed the DIAGNOSTICS option.

DIAGNOSTICS DBGSEMTIM

Enables high-resolution measurement of the time threads wait at each mutex and semaphore call.

DIAGNOSTICS DEBUG

Enables a general method to signal debugging intent and to input data for diagnostic or debugging use.

DIAGNOSTICS EXTENDED_TRAN_NO

This keyword forces the server to log each physical open of a non-extended transaction number file to the CTSTATUS.FCS file. The reason to check for a file that does not support extended transaction numbers is that if all files do not support extended transaction numbers, then the exceptions could cause the server to terminate if the transaction numbers exceed the original 4-byte range and one of these files is updated. By “all files” we mean superfile hosts and indexes; data files are not affected by the extended transaction number attribute.

See Also

COMPATIBILITY 6BTRAN_NOT_DEFAULT

COMPATIBILITY EXTENDED_TRAN_ONLY

DIAGNOSTICS FALG_ERR

An issue was reported receiving error #767 (FALG_ERR) which indicates “fixed length record offset not aligned”. The error indicates a read request at an unexpected data file offset. The error was returned from GTEREC(). After reindexing the problem cleared.

To diagnose an underlying cause, this diagnostic option was added. When present in ctsrvr.cfg, a stack trace will be generated when a FALG_ERR is encountered. There is no anticipated overhead with this tracking enabled, until an issue occurs. And then, the overhead is the time to generate the stack trace file.

DIAGNOSTICS FILE_CLOSE_FLUSH

Calls by a client to open or close files or connect to c-tree Server sometimes took an unexpectedly long time. This could happen when a file that had many updated data or index cache pages was being physically closed. It was caused by flushing updated data/index cache pages to disk when closing a file. The flushing occurred while the thread was holding a global mutex that was acquired during the file open and close operations. The logic has been modified to prevent this delay.

Note that even though this change avoids a system-wide delay, the physical close of a file still requires its updated cache pages to be flushed. A call to close a file that ends up physically closing the file will not return until the writes to the file have completed. If the file has a lot of updated data or index cache pages, this could potentially take a noticeable amount of time. The use of the background flush options (NONTRAN_DATA_FLUSH_SEC, NONTRAN_INDEX_FLUSH_SEC, TRAN_DATA_FLUSH_SEC, TRAN_INDEX_FLUSH_SEC) can help reduce this delay by flushing updates in the background before the last connection closes a file.

Configuration Options:

The configuration option FILE_CLOSE_FLUSH can be specified in ctsrvr.cfg or changed at runtime in order to enable or disable this feature. FILE_CLOSE_FLUSH YES enables the feature and FILE_CLOSE_FLUSH NO disables the feature.

The configuration option DIAGNOSTICS FILE_CLOSE_FLUSH can be specified in ctsrvr.cfg or changed at runtime to enable diagnostic logging to CTSTATUS.FCS messages indicating the start and end of the file close flush. Sample messages:

- User# 00033 Begin pre file close flush for file mark.dat

- User# 00033 End pre file close flush for file mark.dat; flushes= 3199

- User# 00033 Begin pre file close flush for file mark.idx

- User# 00033 End pre file close flush for file mark.idx; flushes= 2973

Affected Components: c-tree Server

DIAGNOSTICS FILE_LOGON

When the DIAGNOSTICS FILE_LOGON keyword is added to the FairCom DB Server configuration file, upon each logon and logoff, four file counters are output: physical files open, logical files open, File Control Blocks (FCBs) in use, and FCBs available. The logical files count will be greater than physical files if c-tree Superfiles are in use. FCBs in use count will be greater than logical files if index files contain additional index members. These values reflect counts generated by all applications using the FairCom DB Server.

Default: OFF

DIAGNOSTICS FLUSH_BLM

Enables at file close, a check to see if any buffer or cache pages have been missed when flushing and clearing the buffer/cache space. If a page is missed then CTSTATUS.FCS will contain one or messages beginning with the line:

DIAGNOSTIC: orphaned BLM ...

DIAGNOSTICS FORCEI_SHADOWUPD

Enables a check for unexpected instances of shadow updates during shadow entry clean-up and will list such updates in a server's CTSTATUS.FCS.

DIAGNOSTICS FULL_DUMP

Enables the generation of a mini-dump file with a full memory dump, which are significantly larger in size.

Please note, if you have the DIAGNOSTICS FULL_DUMP keyword active, then any Windows Mini Dump Files (*.mdmp) created will include a full memory dump of the c-tree Server process, which may include end-user data. If this keyword is not active, the dump contains only the c-tree Server stack trace, which does not include end-user data.

Note: This is only available on Windows.

DIAGNOSTICS KEY_COMPARE

Enables an index tree-walk history which contains the nodes visited as well as the key comparisons made. Each time a new tree-walk is begun, the history is refreshed. If a terr() occurs, the history is dumped into the file LOGTREE.FCS. This information is not maintained during high speed index loads, or during cleaning transaction information from an index.

DIAGNOSTICS KLLX

Enables, at the end of automatic recovery, all leaf nodes to be checked to see if any uncommitted key values were inadvertently treated as committed. If so, the server console displays the index name and the suspect transaction number.

DIAGNOSTICS L59

Prior to V8, logic to detect flaws in abort node list processing generated an L59 failure with server termination. New logic post V8 will mark the index file bad and close the low level physical fail failing with error BIDX_ERR (527) and output the following message to CTSTATUS.FCS:

"Trouble processing new index node"

This keyword reverts server to the prior behavior of a server termination with an L59 failure code.

DIAGNOSTICS LOCK_LOGON

The DIAGNOSTICS LOCK_LOGON keyword displays the net lock count upon each FairCom DB Server logon and logoff. This count is system wide, not just for the process logging off or on and incurs very low overhead.

Default: OFF

DIAGNOSTICS LOGON_COMM

(Legacy) Enables the logon communications sequence to be output by the server.

DIAGNOSTICS LOWL_CRC_ON

Enables low-level CRC communication checks.

DIAGNOSTICS MEMORY_LEAK

Enables debug information upon each memory alloc() (get) and free() (put) output to MEMLEAK.FCS. A typical line of output looks like:

003b860cx 0000070 getmem O01 B0026 72

The line is interpreted as follows:

entry interpretation
----- --------------
1st memory address
2nd sequence number to aid debugging
3rd action
4th thread ID
5th excess of allocations over frees.

Separate "balance" counts maintained for mballc's and ctgetmem's
6th memory size (not given for mbfree's)

Note: Use of this option will result in extreme performance degradation.

DIAGNOSTICS MEMTRACK

Deprecated: This configuration option enabled the FairCom Server memory tracking feature.

Originally, enabling c-tree Server's memory allocation call stack collection required specifying DIAGNOSTICS MEMTRACK in ctsrvr.cfg because memory allocation call stack collection could increase memory use and slow performance.

An earlier modification made it possible for an administrator to enable and disable memory allocation call stack collection on a per-suballocator list basis at runtime, which meant that the collection was not necessarily a big impact on performance.

The DIAGNOSTICS MEMTRACK configuration option is now deprecated. This is now tracked internally without impact on performance. See the ctstat utility and SnapShot() function memory tracking, in addition to the new HEAP_DEBUG_LEVEL memory tracking.

Behavior Prior to V12

DIAGNOSTICS MEMTRACK increases memory use for each allocation by the size of the MEMALC structure (80 bytes for 32-bit compiles and 152 bytes for 64-bit compiles). The maximum number of stack frames to collect per call stack is set at compile time to MEMSTKLMT, which is defined to 15 in memtrk.c.

If a client attempts to read or log memory statistics using the ctMEMSTAT() API function, however, FairCom DB is running without DIAGNOSTICS MEMTRACK in ctsrvr.cfg, ctMEMSTAT() returns error code FTYP_ERR (53, file mode inconsistent with c-tree config).

Note: This option should be used only when recommended by a FairCom engineering team member as it will impact performance, and the collected data is only meaningful for internal diagnostics. It is included here for completeness.

In V11 and later, the memory allocation tracking feature of FairCom Server is supported on Linux systems. To use this feature, add DIAGNOSTICS MEMTRACK to ctsrvr.cfg.

The MEMTRACK keyword allows the ctstat utility to be used to log current memory allocations to a file. The following parameters can be used with ctstat:

-mf logfile - Log all memory allocations to the specified file
-ma logfile - Log aggregate memory allocations to the specified file
-mr min,max - Log only memory allocations in the range min,max
-ms - Output memory allocation statistics

Examples

C:\>ctstat -ms -h 10 -s FAIRCOMS

memseq memalc

1267 992

1289 997

To log all memory allocations (with each allocation listed separately) to the file memfull.log in the FairCom Server's working directory:

C:\>ctstat -mf memfull.log -i 1 1 -s FAIRCOMS

To log all memory allocations (with allocations having the same call stack listed only once each) to the file memaggr.log in the FairCom Server's working directory:

C:\>ctstat -ma memaggr.log -i 1 1 -s FAIRCOMS

To log all memory allocations that have sequence numbers between 1900 and 2000 to the file memaggr.log in the FairCom Server's working directory:

C:\>ctstat -ma memaggr.log -mr 1900,2000 -i 1 1 -s FAIRCOMS

See Also

DIAGNOSTICS SUBALLOCATOR_OFF

DIAGNOSTICS NO_EXCEPTION_HANDLER

The FairCom DB Server for Windows includes a Win32 Exception Handler to take care of any error situations. This keyword disables the Exception Handler in case the addition of this error handler created any unexpected behavior. FairCom cannot foresee any situation where the keyword will be needed, but in the interest of safety, added a method for disabling this Exception check.

Default: Handler Enabled

DIAGNOSTICS NO_LOG_EXTENSION

Forces the transaction log extension logic to skip the writing of 0xff fill. The logs are then extended only as actual log writes take place.

DIAGNOSTICS NODE_REQUEST_TIME

Enables tracking the cumulative time spent finding each index node in the index cache. The LockDump() function includes the node request times in its output. The average and total node request times are reported in milliseconds.

Example

Buffer request count profile:

Avg req time Tot req time Request count Block count Node offset Filno Type Filename

------------ ------------ ------------- ------------ ------------------ ----- ---- --------------------

NRQ: 3 605087 200000 67078 0x000000000033e000 26 R admin_sys_001_000001613.idx

NRQ: 1 298947 200000 33912 0x0000000000008000 21 R admin_sys_001_000001693.idx

NRQ: 0 13977 94150 1544 0x000000000033c000 26 I admin_sys_001_000001613.idx

NRQ: 0 7762 94149 740 0x000000000000c000 26 I admin_sys_001_000001613.idx

NRQ: 0 177 11701 11 0x00000000005fc000 26 I admin_sys_001_000001613.idx

NRQ: 0 64 269 0 0x00000000004f6000 26 L admin_sys_001_000001613.idx

NRQ: 0 60 269 0 0x00000000003c8000 26 L admin_sys_001_000001613.idx

NRQ: 0 60 269 0 0x00000000001f6000 26 L admin_sys_001_000001613.idx

NRQ: 0 57 269 0 0x00000000002b2000 26 L admin_sys_001_000001613.idx

NRQ: 0 56 269 0 0x00000000004ae000 26 L admin_sys_001_000001613.idx

DIAGNOSTICS NODEQ_MESSAGE

Enables the delete node thread to log the following message to CTSTATUS.FCS when it finds a discrepancy between a delete node queue entry and the current index file on disk having the same filename:

ctdnode: could not process empty node

This message is suppressed by default.

The delete node thread logs the following message to CTSTATUS.FCS when it finds a discrepancy between a delete node queue entry and the current index file on disk having the same filename:

"ctdnode: could not process empty node"

For example, if an index file is deleted and recreated, a delete node queue entry for the original index file will not match the file ID values in the new index file.

As another example, if an index file is blocked using the file block feature, the delete node thread will not be able to access the file. Because this is a normal message, we now suppress it by default.

If desired, the message can be re-enabled by adding the option DIAGNOSTICS NODEQ_MESSAGE to ctsrvr.cfg.

In V11.8 and later this option has been enhanced to provide additional detail when queue entries are not processed. The System Snapshot counter sctdnd_non ("delete node thread queue no action") now accounts for all cases when a queue entry is not processed. We now expect the following relationship to be true:

"Leaf nodes pruned" = sctdnd_red - (sctdnd_abn +sctdnd_non + sctdnd_rwt).

DIAGNOSTICS NONE

This option is used in conjunction with the tamper-proof settings file under the server. Configuration options that are in the encrypted settings file, ctsrvr.set, cannot be overridden in the ctsrvr.cfg file.

The DIAGNOSTICS, COMPATIBILITY, and CONSOLE keywords do not automatically block use in a subsequent stage of configuration loading. To explicitly block any of these keywords present in a later stage, add entries in the form: <keyword> NONE where <keyword> is DIAGNOSTICS, COMPATIBILITY, or CONSOLE.

Default: Not present

DIAGNOSTICS PCRP_ERR

Enables at any time a PCRP_ERR occurs, a message is logged to CTSTATUS.FCS and a process stack trace or minidump of the process is created.

Example messages:

PCRP_ERR: ctpartno(), file <filename>

PCRP_ERR: ctunfoldpartno(), file <filename> loc <location>

PCRP_ERR: subno=<subno> dnum->ptcur=<ptcur> dnum->ptmbr=<ptmbr>

DIAGNOSTICS POPN_ERR

DIAGNOSTICS POPN_ERR.

When enabled and the PENDING_FILE_OPEN_RETRY_LIMIT configuration limit is exceeded during file open, a stack dump is generated.

DIAGNOSTICS PROCESS_EXIT

Enables FairCom DB to take the following actions before exiting (even on a normal shutdown):

Dumps a process stack trace, logging the name of the stack dump file to CTSTATUS.FCS.
Displays one of several dialog boxes on Windows.

Note: This feature requires a custom build to enable.

DIAGNOSTICS QUEUE_LOGON

The DIAGNOSTICS QUEUE_LOGON provides the current number of items in the FairCom DB Server queues. Three system wide counts are given for QUEUE_LOGON. These three counts, listed below, are preceded by the letters M, C and D, respectively. This option requires very low overhead.

The number of pending messages in FairCom DB Server monitors.
The number of pending messages in checkpoint queue.
The number of pending messages in the delete node.

Default: OFF

DIAGNOSTICS READ_ERR

Rare, but unexpected READ_ERR (36) messages have been reported. When the operating system denies a read request from the file system, normally, a system error code is reported such as "permissions denied" or otherwise. However, rare and sporadic cases have been reported where this additional error return was zero (0). Good practice dictates we should determine why the OS file system is denying a read operation for any reason. A diagnostic option is now available to generate additional context information, along with a server stack trace when this condition occurs.

DIAGNOSTICS READ_ERR

This option enables additional diagnostic logging of read errors. When this option is in effect and an unexpected read error occurs (for example, a READ_ERR with a 0 sysiocod value), FairCom Server logs the following message to CTSTATUS.FCS and creates a process stack dump of the FairCom Server process (on systems that support that ability):

Mon Sep 14 12:23:19 2015

- User# 00020 READ_ERR: loc 5 file <FILENAME> offset <OFFSET> iosize <READ_SIZE_IN_BYTES> syserr <SYSTEM_ERROR_CODE> [physical file size <FILE_SIZE_ON_DISK>]

Mon Sep 14 12:23:21 2015

- User# 00020 Dumped stack for server process 20788, log=1, loc=0, rc=0

DIAGNOSTICS READ_ERR can be enabled and disabled at runtime by calling ctSETCFG() or using ctadmn (menu option 10, Change Server Settings, then option 9, Change a DIAGNOSTICS option).

To enable this additional logging in standalone, the application must set the following after initializing c-tree (INITISAM, etc):

ct_diagflg3 |= ctDiagnose3READ_ERR;

Note: The presence of the READ_ERR log messages does not necessarily indicate an error that an application would see. In some cases, c-tree expects and handles the READ_ERR internally.

DIAGNOSTICS REMAINING_THREADS

This keyword, when added to the server configuration file, will result in listing threads still active at server shutdown, and causing the final system checkpoint to be skipped, to the CTSTATUS.FCS log file. Any other internal threads will also be listed. If the final checkpoint is not skipped because of persistent client threads, then no such information is placed in CTSTATUS.FCS.

Example Output

Thu Dec 18 14:34:29 2003

- User# 11 Remaining Thread: ThrdID 01 ADMIN| Atributes:00000028x

Thu Dec 18 14:34:33 2003

- User# 11 Remaining Thread: ThrdID 09 ADMIN| Atributes:0000000ax

Thu Dec 18 14:34:46 2003

- User# 11 Remaining Thread: ThrdID 11 | Atributes:00000000x Shutdown Thread

Note: Administrative threads may be expected to exist at this point in server processing. Their userid will be ADMIN. In the above example, all of these ADMIN threads were routine, and were not causing any shutdown problems. Also, the thread processing the shutdown is listed, and will be annotated as the Shutdown Thread. The attributes are for determining the identity of the administrative threads within the server.

DIAGNOSTICS REPL_READ_BUFFER

Enables a check that the log data is being correctly read into the replication buffer.

DIAGNOSTICS SUBALLOCATOR_OFF

Forces FairCom DB to allocate memory directly from the heap instead of using its internal memory suballocator.

See Also

DIAGNOSTICS MEMTRACK

DIAGNOSTICS THREAD_DUMP

(Legacy) Enables low-level thread diagnostics.

DIAGNOSTICS TR_TRAN_ERR

When the local/master replication model experiences out-of-sync data between the master and local servers, this server configuration option enables logging of errors during the two-phase commit sequence. The log entries are written to the file TR_TRAN.FCS in the server's LOCAL_DIRECTORY directory.

Sample log messages:

Wed Mar 1 09:35:46 2017 Transaction operation TRANABT failed: loc=1 master_err=127 local_err=0

Wed Mar 1 09:37:44 2017 Transaction operation TRANEND failed: loc=5 master_err=128 local_err=71

Wed Mar 1 09:43:25 2017 Transaction operation TRANEND failed: loc=5 master_err=128 local_err=0

Wed Mar 1 10:30:45 2017 Transaction operation TRANEND failed: loc=6 master_err=0 local_err=768

The log message includes the function that failed, location code where the error occurred, and master and local server error codes.

This feature can be enabled/disabled at runtime using ctSETCFG() or the ctadmn option to change a DIAGNOSTICS option.

DIAGNOSTICS TRACK_LOGON

The DIAGNOSTICS TRACK_LOGON option provides a net count of memory allocation requests. This count is system wide, not just the particular process logging off or logging on and requires very little overhead. See also MEMORY_TRACK.

Default: OFF

DIAGNOSTICS TRAN_RECOVERY

Causes transaction recovery log messages to be written to the file RECOVERY.FCS (off by default).

In the standalone model, this feature is enabled by activating #define ctDBGRCVR before compiling the c-tree library. In such cases, log messages are written to the file RECOVERY.FCS.

DIAGNOSTICS TREE_WALK

DIAGNOSTICS TRNLOG_WRITE

Writes a diagnostic message to CTSTATUS.FCS for each write that the server makes to the transaction logs.

Enabling this option might impact the performance of ctTRNLOG transactions, because it writes to CTSTATUS.FCS each time it writes the transaction log buffer to disk,

DIAGNOSTICS UNOD_ERR

Enable UNOD_ERR (51) diagnostic stack dump.

In V11.5 and later, a server keyword can be used to enable UNOD_ERR (51) diagnostic stack dump. The keyword is DIAGNOSTICS UNOD_ERR. It can be turned on/off at runtime using the ctSETCFG() function or option 10 of the ctadmn utility.

DIAGNOSTICS UPDFLG

File corruption with FairCom DB should be a very rare event. Transaction control over files prevents most file corruption. However, non-transaction controlled files always have some risk. Detecting a change in integrity state is important for identifying root causes. A message is now logged to CTSTATUS.FCS at the time a file is marked corrupt and process stack trace is created. Sample message:

- User# 00027 UPDFLG: mark.dat updLoc=1016 updflg=52x oldflg=0x

This message identifies the file name, the code location where the update flag was set, and the new and old values of the update flag. If the file is an index member, the index member number follows the filename. For example "mark.idx[1]" indicates the first index member.

For detailed analysis, DIAGNOSTICS UPDFLG enables logging of every change to a file's update flag.

The flag values are single hex bytes. updLoc is a four-digit location code to later determine which c-tree function was actually responsible for the update.

DIAGNOSTICS WRITE_ERR_DUMP

The potential possibility of a disk write failure may go unreported with the FairCom DB Server. With a data or index file write failure, the FairCom DB Server could possibly continue operations, and assume the cache has been properly flushed to disk. While the transaction logs maintain data integrity, particular transactions may be marked as not flushed to disk, holding previous transaction logs from being discarded.

This server configuration keyword will cause the contents of the write request to be appended to the WRITE_ERR.FCS file. If the write is for more than 32K bytes, then only the first 32K bytes are output to the file.

To detect this situation may occur, a notification is now logged to CTSTATUS.FCS when and if a WRITE_ERR occurs. The log entry will include the file name, the offset of the write attempt, the system error code, the size of the write request and the number of bytes written. Optionally, the contents of the write request can be output to a file as well.

Note: These entries should be extremely rare!

This behavior is on by default. If this is undesirable, a new server configuration keyword will inhibit these messages:

CTSTATUS MASK_WRITE_ERR

DIAGNOSTICS WRITETHRU

The FairCom DB Server writes a warning message to the CTSTATUS.FCS file if it detects a file that is not under transaction control and does not have WRITETHRU defined. This warning is to notify the developer that the file is being maintained in a vulnerable mode. Because of the overhead of writing this message to the log, and because FairCom does allow this “dangerous-cached-buffered-type” mode, the warning message is only issued once, for the first file detected. In other words, it simply tells the developer that there was “at-least-one” vulnerable file.

To help developers detect the file names for all vulnerable files, this keyword has been added. By placing this keyword in your ctsrvr.cfg, all file names will be listed in the CTSTATUS.FCS file.

Default: Disabled

Diagnostics

CHECK_TRNLOG_BUFFER_WRITES

DIAGNOSTIC_INT

DIAGNOSTIC_STR

DIAGNOSTICS LDAP

DIAGNOSTICS ABEND_ABORT (Deprecated)

DIAGNOSTICS ABORT_NODE_LIST

DIAGNOSTICS AUTO_PREIMG_CHECKLOCK / AUTO_PREIMG_CHECKREAD

DIAGNOSTICS AUTO_TRNLOG_CHECKLOCK / AUTO_TRNLOG_CHECKREAD

DIAGNOSTICS CHECK_KEY_MARKS

DIAGNOSTICS CHECK_UDEFER

DIAGNOSTICS COMM_LEVEL_X

DIAGNOSTICS CTQUIET

DIAGNOSTICS DADV_ERR

DIAGNOSTICS DBGSEMTIM

DIAGNOSTICS DEBUG

DIAGNOSTICS EXTENDED_TRAN_NO

DIAGNOSTICS FALG_ERR

DIAGNOSTICS FILE_CLOSE_FLUSH

DIAGNOSTICS FILE_LOGON

DIAGNOSTICS FLUSH_BLM

DIAGNOSTICS FORCEI_SHADOWUPD

DIAGNOSTICS FULL_DUMP

DIAGNOSTICS KEY_COMPARE

DIAGNOSTICS KLLX

DIAGNOSTICS L59

DIAGNOSTICS LOCK_LOGON

DIAGNOSTICS LOGON_COMM

DIAGNOSTICS LOWL_CRC_ON

DIAGNOSTICS MEMORY_LEAK

DIAGNOSTICS MEMTRACK

DIAGNOSTICS NO_EXCEPTION_HANDLER

DIAGNOSTICS NO_LOG_EXTENSION

DIAGNOSTICS NODE_REQUEST_TIME

DIAGNOSTICS NODEQ_MESSAGE

DIAGNOSTICS NONE

DIAGNOSTICS PCRP_ERR

DIAGNOSTICS POPN_ERR

DIAGNOSTICS PROCESS_EXIT

DIAGNOSTICS QUEUE_LOGON

DIAGNOSTICS READ_ERR

DIAGNOSTICS REMAINING_THREADS

DIAGNOSTICS REPL_READ_BUFFER

DIAGNOSTICS SUBALLOCATOR_OFF

DIAGNOSTICS THREAD_DUMP

DIAGNOSTICS TR_TRAN_ERR

DIAGNOSTICS TRACK_LOGON

DIAGNOSTICS TRAN_RECOVERY

DIAGNOSTICS TREE_WALK

DIAGNOSTICS TRNLOG_WRITE

DIAGNOSTICS UNOD_ERR

DIAGNOSTICS UPDFLG

DIAGNOSTICS WRITE_ERR_DUMP

DIAGNOSTICS WRITETHRU

Was this article helpful?

Related Articles

FairCom DB Configuration Options

Subsystem

Basic Configuration

Client Communication