FairCom — FairCom DB Database Engine

The FairCom Server provides a flexible, high performance implementation of the FairCom DB index and data file functions for multi-tasking and network systems. The clean separation between the Server and the implementation-dependent message handling allows implementation of the FairCom Server in different environments by changing only the message routines in the client-side library.

FairCom Servers are licensed separately for development and production. The developer license activates the FairCom Server with eight connections for development purposes only. When the application is fully developed, tested, and ready for distribution into commercial service, a production FairCom Server license is required for each Server host machine, typically one per network.

The FairCom Server is licensed on a concurrent connection basis. Each connection to the FairCom Server counts against the total until the connection is closed. A client establishes a connection with every successful FairCom DB initialization call, with multi-threaded application potentially establishing many connections.

Additional information is located in the FairCom Server Administrator’s Guide, including installation information.

In This Chapter

Client/Server Advantages

FairCom Client/Server Communication

Calculating Memory Usage

Controlling Server Memory

Calculating File Storage Space

Copying Server-Controlled Files

Copy Files Between FairCom DB Servers

Multiple Server Support per Client

Local Library Support

Server Utilities

Performance

FairCom Server Cache Subsystem

File Modes

Deployment Options

Server System Event Log

Create a List of Arbitrary Names in Server Configuration File

Server Internals

Diagnostic Tools

Client/Server Advantages

The client/server model offers the following features:

Reduced disk I/O: In the non-server mode, applications cannot perform local file buffering, or applications would have inconsistent views of the files. Therefore, all disk I/O is forced to and from the disk. In the Server mode, the FairCom Server performs all disk I/O allowing user controllable caching.
Reduce traffic on Networks: In network systems the FairCom Server reduces the amount of network message traffic, provided the node running the FairCom Server is also the node where the FairCom DB files are stored. In the non-server mode, all disk I/O must be sent over the network. In Server mode, only the application request and the Server response go over the network.
Transaction Processing: A sophisticated set of functions provides industrial quality on-line transaction processing, including full commit and rollback, savepoints, and complete logging. Transaction processing is covered in more detail in “Data Integrity”.
File Security: File encryption and passwords, user passwords and ID’s, groups, and other sophisticated features provide flexible and powerful file security. See “Security and Encryption (File Security and Encryption, /doc/ctreeplus/FileSecurityandEncryption.htm)”.
File mirroring: Selected files are mirrored to separate physical devices or partitions, offering extreme data protection. See “Data Integrity”.
On-line backups: On line backups make it easier for administrators to perform regular file backups. See the FairCom Server Administrator’s Guide.
Multi-threading: All FairCom Servers are multi-threaded, regardless of whether the operating system offers multi-threaded functionality. On platforms where efficient multi-threading is available FairCom Server uses O/S provided threading calls. On other platforms that don’t offer threading the FairCom Server uses a low-level, rapid context switching threading kernel written by FairCom.

The multi-threaded design of the client/server model, the data/index caching and the reduced network traffic all assist in minimizing the load on the network as more users are added. The degradation in user response times as more users are added to the FairCom Server is much less than with other multi-user models.
Relaxed File Numbering: On systems that do not support record write locks, and for which a FairCom Server is NOT used, FairCom DB can use a dummy file to permit exclusive locks to operate as write locks. However, this requires the same file to be assigned the same file number by different applications, and different files to be assigned different file numbers. The FairCom Server permits different applications to assign the same file different file numbers, or to assign different files the same file number without causing problems.
No Operating System Lock Limits: Many operating systems limit the number of total locks or locks per application. The FairCom Server avoids this restriction by maintaining internal lock tables and minimizing system-level locks.

In This Section

Server-Application Relationship

The FairCom Server executes all FairCom DB functions. When you link your application to a FairCom DB application library set up for the FairCom Server, your application becomes bound with the required system dependent message handling functions. The resulting executable module can only run on a system designed to support the message handling routines included in the link step. This executable module cannot run as a stand-alone, single-user application, unless it is specifically built with local library support as discussed later in this chapter.

FairCom Client/Server Communication

The client/server communication is client driven. For every attached user to the FairCom Server there is a corresponding thread created inside the server to service the client requests. The thread created inside the server waits on a blocking socket for communication from the client. The thread will effectively wait forever unless the communications channel is terminated. If the OS determines that the communications channel is no longer valid, the BLOCK on the read is freed and a read is attempted by the server thread. However, if there is nothing in the buffer, the server will log an error 128 ‘CNTIO READ ERROR’ in the CTSTATUS.FCS file followed by a 161 error indicating that the server is assuming that the client has gone away and will clean up the connection.

The client thread proceeds to shut down the socket performing the necessary clean up and closing of the socket connection. In this situation a socket can take up to 3 or 4 minutes depending on the OS to completely shut down.

A client should shut down by calling CLISAM() or STPUSR() to allow the server to process the client connection termination in an orderly and efficient manner.

Logging On and Off the Server

When using the FairCom Server, applications must establish communications with the FairCom Server. This is called logging on. The following ISAM routines automatically log on: InitISAM(), OpenISAM(), or CreateISAM(). When using low-level functions, InitCTree() logs on.

When applications finish, they need to disconnect from the FairCom Server. This allows the FairCom Server to reallocate the memory used for this application to other uses. The ISAM routine CloseISAM() logs off. When using low-level or ISAM functions, use StopUser() to log off.

Calculating Memory Usage

The FairCom Server startup memory requirements can be reasonably approximated with the following equation:

Base line Memory =

Server EXE size + 1 MB +

Communications Library size (if applicable) +

Data cache (DAT_MEMORY) +

Index buffer (IDX_MEMORY) +

(900 bytes * Number of files (FILES)) +

(325 bytes * Maximum number of connections (CONNECTIONS)) +

(16000 bytes * Number of actual connections) +

(256 bytes per connection per file actually opened))

Note the following points:

DAT_MEMORY and IDX_MEMORY default to 225,000 bytes each.
FILES defaults to 100.
CONNECTIONS default to 128.
IDX_MEMORY is the MAX of:
- IDX_MEMORY or
- 3 * CONNECTIONS * (PAGE_SIZE + 400), where PAGE_SIZE defaults to 8192.

The following locking/transaction processing related items should be considered when approximating the FairCom Server dynamic memory requirements:

Each record locked consumes 24 bytes.

For transaction processing files only:

Each data record written consumes (record length + 42) bytes.
Each index operation consumes (key length + 42) bytes.

Controlling Server Memory

This section is a summary of the FairCom Server memory limit behavior and associated transaction controls. The Server has two means of controlling memory use:

User memory limits
Total memory limit

The user memory limit is specified through system-wide defaults using the USR_MEMORY and GUEST_MEMORY configuration keywords, which take as their arguments the memory threshold defaults at the user level. The GUEST_MEMORY limit applies to applications that connect to the FairCom Server without a user ID. The USR_MEMORY limit applies to applications that do provide a user ID as part of their Server logon. These thresholds default to zero, which implies no limit on user memory. The system-wide user memory defaults can be overridden by specifying a user memory limit when adding a user ID to the Server administrative database with the ctadmn program.

The user memory limit can be either a guideline or an absolute limit. The memory limit defaults to a guideline. In the guideline mode, when a user exceeds the user memory limit (if set), an attempt is made to flush preimage memory to disk. However, the preimage control block still resides in memory. In any event, the user is able to continue getting more memory even if the user limit is exceeded. The effect of exceeding the guideline limit is two-fold:

Preimage memory is swapped to disk, and
The OPS_MEMORY_SWP bit in the user’s FairCom DB status word is turned on, (see SetOperationState (SetOperationState, SetOperationState) in the FairCom DB Programmers Reference Guide). This bit stays on until a new transaction is started. In absolute mode, memory is denied the user if the limit is exceeded, which will lead to aborted transactions and/or errors during update operations.

TOT_MEMORY is a deprecated keyword. FairCom does not recommend setting a total memory limit.

To avoid large amounts of memory use by an application performing large transactions, a special automatic commit mode can be enabled through Begin().

Calculating File Storage Space

Approximate the file storage space used by the FairCom Server with the following equation:

Storage requirement in bytes =

server size +

communication libraries (if applicable) +

utility program size(s) +

transaction log files +

data files +

index files

The transaction log files initially increase in size as adds, deletes and updates are performed on the FairCom Server controlled files. The FairCom Server configuration keyword LOG_SPACE, which defaults to 10MB, indicates the amount of disk space allocated to store active transaction logs. Generally, the active transaction log files will not exceed the LOG_SPACE value. If transaction processing is used, it is not recommended to decrease the LOG_SPACE keyword.

Copying Server-Controlled Files

When a file open is attempted, FairCom DB checks to see if either a file with the same name has already been opened, or if a file with the same unique ID has already been opened. In either case, the match means that a physical file open is not required. Instead the open count for the file is incremented. Checking the unique file ID permits different applications and/or client nodes to refer to the same file with different names. For example, consider the situation where independent clients operate from different drive or path mappings.

Fri Aug 20 07:56:17 2014

- User# 00021 Matching file IDs: ./data/\ctreeSQL.dbs\custmast.dat <> /data/\ctreeSQL.dbs\custmast.dat

Fri Aug 20 07:56:17 2014

- User# 00021 Reassigning file ID ...

Fri Aug 20 07:56:17 2014

- User# 00021 /data/\ctreeSQL.dbs\custmast.dat

If two different files have the same file ID, (a 12-byte value consisting of a Server ID, a time stamp, and a transaction log sequence number), problems will arise as the second file would not actually be opened. The ID is constructed such that no two files can have the same ID unless someone copies one file on top of another. See the warning below.

When a file without a matching name does match the unique file ID, a message is sent to the system console indicating the names of the two files involved. This message can be suppressed by adding the following entry to the FairCom Server configuration file:

MONITOR_MASK MATCH_FILE_ID

Warning: Do NOT copy, move, or delete files controlled by the server while the server is operational!

As discussed above, copying a file to a new name is typically the only way the file IDs can match. The Update File ID utility, ctfileid, is available to update the file ID number should this become absolutely necessary. This utility should only to be run on a file when the server is stopped.

Copy Files Between FairCom DB Servers

FairCom DB users have long enjoyed the ease of managing their physical data and index files alongside their applications. Easily moving and copying files between servers is a consistent need required by our customers, especially as they transition applications to centrally processed FairCom DB server technology. A file copy function makes this a seamless effort directly from the application layer. File transfer is now done server-to-server in addition to a client/server transfer feature introduced in V10.

The function ctCopyFile() copies a c-tree data or index file, or a c-tree data file and its associated index files. The caller specifies the source and destination file names and options that affect the behavior of the file copy. c-tree copies the files synchronously. That is, the copy call does not return until the file copy is completed.

To copy a file, a caller must be authorized by c-tree (including the file system) to open a file for read access. ADMIN user is always granted read permission for all files. (c-tree file permissions are only supported in the client/server model and not with standalone usage.)

c-tree attempts to open the file in exclusive mode, while still allowing other connections to open the file for read access. If the file is already open, c-tree flushes updates to disk and sets the update flag to zero to indicate that the file is in a clean, consistent state. If c-tree cannot open the file in exclusive mode, the copy operation fails.

Note: Files with an .FCS extension (such as FAIRCOM.FCS, SYSLOGDT.FCS, L0000001.FCS, SNAPSHOT.FCS, etc.) are considered to be a c-tree internal system housekeeping files. The file copy function does not allow copying from or to c-tree internal system files. An attempt to copy such a file fails with error code 1013 (SCPY_ERR).

Copying c-tree files

When copying a c-tree file, c-tree assigns a new file ID to the copy of the file so that the file ID does not match the original file’s ID, and c-tree sets the data and index file and directory names in the data file’s IFIL resource to the names of the new copies of the files. Other than those differences, the copied file is identical to the original file: it contains the same active and deleted data records or index nodes and key values, resources, and has the same properties as the original file, including:

security (file owner, file group, file permission mask, file password)
data encryption
data compression
transaction control

The caller indicates what action to take when a file already exists with the same name as the specified destination file. By default, the file copy fails in this situation, but the caller can specify that the existing file should be deleted instead.

When copying multiple files (for example when copying a data file and its associated index files), if an error occurs c-tree stops the file copy operation and deletes all files that were successfully copied. If desired, the caller can specify an option to keep all the successfully copied files in case of an error.

An optional callback mechanism is available for supporting user feedback progress bars based on percentage of file copy complete.

See also

Multiple Server Support per Client

ctNOGLOBALS support (see “Multiple Instance Support”) allows client applications to connect to multiple FairCom Servers simultaneously. Using ctNOGLOBALS and the five instance functions, NextCtree(), RegisterCtree(), SwitchCtree(), UnRegisterCtree(), and WhichCtree(), a single client process can connect to multiple FairCom Servers. For example, connect a Windows client to an Mac OSX Server, an AIX Server, and a Sun SPARC Server simultaneously.

This support requires each connection to a FairCom Server to have its own unique FairCom DB instance. The instance is an allocated structure in memory that contains the FairCom DB global variables for the specific instance. Multiple connections can be made to the same FairCom Server or any combination of FairCom Servers on the network.

With multiple FairCom Server connections, it is possible to have multiple active transactions (one per instance), as long as the transactions do not depend on each other. In other words, transaction control is supported on a “one-to-one” basis with the client. Transactions cannot be spread across multiple FairCom Servers. Under no circumstances should related transactions be started across multiple instances. This feature, which involves multi-phase commits in a distributed environment, is being considered for a future FairCom Server release.

With FairCom’s sophisticated Netformat, (automated byte flipping between dissimilar platforms), communication logic the multiple FairCom Server support allows data to be seamlessly transferred from machine to machine. This requires a DODA resource in the data file that details the record structure for that file. See “Record Schemas” for information on DODA resources.

For example, to move records from an SCO Unix Server to a Mac OSX Server, a simple FairCom DB read/write loop could switch FairCom DB instances between the read and write. The Netformat logic automatically transposes the numeric data “on the fly”. Of course, if you wish to flip the data yourself, FairCom allows the Netformat feature to be disabled. See the ctlxmg sample program discussion in “ctlxmg - LOCLIB Sample Program”.

Local Library Support

In addition to connecting to multiple FairCom Servers, it is possible to connect a client side application to a local FairCom DB single-user, (NOTFORCE), or Standalone Multi-user, (FPUTFGET), library. This feature is made possible by the ctLOCLIB define discussed in ctoptn.doc, in your root FairCom DB directory and will open the door to many sophisticated application uses, for example, moving data from a local machine to a FairCom Server machine.

During program execution, your logic must establish a separate FairCom DB instance for each local and/or FairCom Server connection with the RegisterCtree() instance function. The five FairCom DB instance functions documented in the function reference section are:

RegisterCtree()	Register an instance to FairCom DB.
UnRegisterCtree()	Unregister an instance from FairCom DB.
SwitchCtree()	Make the supplied registration ID active.
NextCtree()	Change to the next registered instance.
WhichCtree()	What is the current FairCom DB instanceID.

The enhanced ctNOGLOBALS support allows an application to establish multiple “instances” of FairCom DB. A FairCom DB instance can direct file I/O to either local c‑tree files or Server based files. It is possible to have multiple instances of both local and/or Server based FairCom DB files in the same application. FairCom DB distinguishes between local and Server calls based on the “user profile” used by the extended FairCom DB initialization calls, (InitCTreeXtd(), InitISAMXtd(), and OpenISAMXtd()). When the profile mode is OR-ed with USERPRF_LOCLIB, the initialization call establishes a local connection. For implementation instructions and a good working example of this logic, consult the sample program ctlxmg discussed in “ctlxmg - LOCLIB Sample Program”.

Server Utilities

These utilities are detailed in the FairCom Server Administrator’s Guide.

ctadmn - Server Administrator Utility

The FairCom Server Administrator Utility, ctadmn, is used by an Administrator to manage general server operations including configuring users, groups and files. It can also be used to monitor the users logged on to the FairCom Server and to disconnect any user from the c-FairCom Server.

To use this utility, do the following (which may vary between environments, depending on user-interface specifics):

Start running the program, ctadmn, as any other program in the environment.
Enter an ADMIN group User ID. Initially, only ADMIN will exist until you create others.
Supply the current password for the User ID given.

Note: If this is the first time ctadmn has been run and the password for the User ID ADMIN is still ADMIN, change it before leaving ctadmn to ensure secure access to this program in the future. You can also change your password using ctpass (see Controlling FairCom Server Access).

You will be prompted for the (optional) file password for the file FAIRCOM.FCS. We recommend you do NOT give this file a password since you should already be the only one who can run this utility. To confirm the absence of a password, press the return key.
In response to the next prompt, supply the current name for the FairCom Server and press enter. If the FairCom Server name has not been changed (see Basic Keywords for details on SERVER_NAME), simply press Enter to use the default name.

Authentication File - This utility supports the use of an encoded password file. Encoded password files keep user IDs and passwords from plain view when the utility is used within a script file. They are created with the ctcmdset utility. The plain text form of the file should be:

; User Id
USERID ADMIN
; User Password
PASSWD <pass>

Use the -1 option to specify the name of the encoded file.

Menus

After finishing these steps, the main menus for the FairCom Server Administrator Utility will be displayed, allowing access to the following groups of operations:

1. User Operations

2. Group Definitions

3. File Security

4. Monitor Clients

5. Server Information (IOPERFORMANCE)

6. Server Configuration (SystemConfiguration)

7. Stop Server

8. Quiesce the Server

9. Monitor Server Activity

10. Change Server Settings

The following sections detail the areas controlled by the Administrator Utility. The steps necessary for each operation may vary slightly in different environments.

Note: While using the Administration Utility, press the question mark key (‘?’) at any prompt to access Help.

Server Administration API

The FairCom Server Administrator Function API allows developers to maintain system administrator necessities from within C programs. Use this set of functions to maintain users, groups, and file security. Also included are functions to monitor the Server.

The command line version of FairCom’s system administrator program, sa_admin, can be used to perform operations from shell scripts. This example program demonstrates how developers can implement system administrator capabilities within their own programs.

It is possible to perform administration functions, such as adding users, changing passwords, defining user groups, from a C function API. The FairCom DB module ctiadm.c contains the following functions, which are detailed in the function reference section:

NINT SA_USERS(COUNT action, saUSRINFO *usrinfo);

NINT SA_GROUP(COUNT action, saGRPINFO *grpinfo);

NINT SA_FILES(COUNT action, saFILINFO *filinfo);

NINT SA_LOGON(pTEXT admnuid, pTEXT admnpwd, pTEXT filepwd, pTEXT servername);

NINT SA_LOGOF();

For more information on the User, Group, and File security options, see “Security and Encryption (File Security and Encryption, /doc/ctreeplus/FileSecurityandEncryption.htm)” in this manual and the “Controlling FairCom Server Access” section of the FairCom Server Administrator’s Guide.

ctstop - Server Stop Utility

Operational Model:

Client

Usage

ctstop [ -auto ] [ <AdminId> <AdminPassword> <ServerName> [ delay ] ]

This utility shuts down a FairCom Server.

Passing the -auto switch to ctstop without specifying a ServerName, AdminPassword, or AdminID shuts down a FairCom Server with the defaults shown below:

ctstop -AUTO ADMIN ADMIN FAIRCOMS

The ctstop utility supports passing in ServerName, AdminPassword, and AdminID when the -auto switch is used.

The optional delay value is the number of seconds to wait before shutting down, with a default of no delay and a maximum of 60 seconds.

Note: If the Replication Agent is running on a server, replication should be stopped before trying to stop the server. If replication is running when you attempt to stop the server, you will see error 792 (External server shutdown disabled).

Authentication File

This utility supports the use of an encoded password file. Encoded password files keep user IDs and passwords from plain view when the utility is used within a script file. They are created with the ctcmdset utility. The plain text form of the file should be:

; User Id
USERID ADMIN
; User Password
PASSWD <pass>

Use the -1 option to specify the name of the encoded file.

sa_admin - Command-line security administration utility

The command-line version of the system administrator program, sa_admin, can be used to perform many user operations directly from shell scripts.

Operational Model:

Client

Usage:

sa_admin [-a<adminuserid>] [-p<adminpassword>] [-f<filepassword>] [-s<servername>] [-1 <auth_filename>] <option>

option is one of the following:

Options Users

-oua Add a user account
-oud Change user account description
-oue Change user account extended settings
-oug Add a user to a group
-oul List user accounts
-oum Change user account memory limit
-oup Change user account password
-our Delete a user account
-ous Show user account information
-oux Remove a user from a group

Options Group

-ofg Change file group
-oga Add a group
-ogd Change group description
-ogl List groups
-ogm Change group memory limit
-ogr Delete a group
-ogs Show group information
-oug Add a user to a group
-oux Remove a user from a group

Options File

-ofg Change file group
-ofl List files matching filename
-oflp List file permissions mask
-ofo Change file owner
-ofp Change file password
-ofs Change file permissions

Wildcard specifiers with sa_admin

sa_admin, -ofp, -ofs, -ofg, and -ofo options support specifying filenames with wildcard characters. When one of these options specifies a filename that includes ? or * characters, the utility retrieves a list of files matching the filename wildcard specifier and executes the specified command for each file.

Retrieve a List of Filenames from the server with sa_admin

-ofl (list files) is used to list the files on the FairCom Server system matching the specified filename including wildcard characters.

sa_admin Support for Encrypted Password Files

; User Id
USERID ADMIN
; User Password
PASSWD <pass>

Use the -1 option to specify the name of the encoded file.

The encrypted password file name is specified using the command-line option:

-1 <filename>

Using -ofs to Clear the Permission Mask

Sometimes it is desirable to make the permission mask empty instead of just adding more permissions to allow all access. For example, consider a file with this permission mask, which has permissions for "owner" and "world":

Permission mask = 0x2783e = {

owner: read write def delete

world: read write def delete nopass

}

To remove all permissions for "owner" and "world" you can use the -ofs option as follows:

sa_admin -aadmin -pADMIN -f"" -sFAIRCOMS -ofs ./ctreeSQL.dbs/admin_permTest.dat -worldall -ownerall

-worldall removes all permissions for "world" and -ownerall removes all permissions for "owner." The resulting permission mask will be:

Permission mask = 0x421 = {

}

To clear permissions for world, user, and group, use the following:

sa_admin -aadmin -pADMIN -f"" -sFAIRCOMS -ofs ./ctreeSQL.dbs/admin_permTest.dat -worldall -ownerall -groupall

Display File Permissions

C:\> sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oflp "vcust*"

Permission mask for file vcusti: OPF_READ OPF_WRITE OPF_DEF OPF_DELETE GPF_READ GPF_WRITE WPF_READ

Permission mask for file vcusti.2: OPF_READ OPF_WRITE OPF_DEF OPF_DELETE GPF_READ GPF_WRITE WPF_READ

Permission mask for file vcusti.ndx: OPF_READ OPF_WRITE OPF_DEF OPF_DELETE GPF_READ GPF_WRITE WPF_READ

Administrator Options

-a System administrator User ID.
-p System administrator password.
-f Optional server system file password.
-s Optional server name.

Note: There is no space between the switch and its parameter.

User Options

The following options, all beginning with -ou, allow changes to user information. Additional group and file options are described below.

Note: To use any optional entry, you must use all the previous entries even if they would otherwise be optional. For example, to add a user with the -oua option and specify a group, you must also enter the userid, desc, and password.

Option User Add

-oua <userid> [-d <desc>] [-w <password>] [-g <group>] [-m <memory>[<rule>]]
[-b <begdat>] [-e <enddat>] [-l <loglimit>] [-r <rsmlogon>] [-t <mstlogon>]

userid: User id (required). The ID is limited to alphanumeric characters, hyphens, underscores, and periods. A hyphen cannot be the first character in a name.
-d desc: Optional user description
-w password: Optional user password
-g group: Optional user group
-m memory: Optional user memory limit.
- rule: Optional user memory rule. Used only with memory. The optional <rule> is A for absolute, D for default, or G for guideline (example -m 10485760a specifies an absolute memory limit of 10 MB). NULL for Default.
-b begdat: Optional starting validity date. Specify as mm/dd/yyyy. NULL for Default.
-e enddat: Optional ending validity date. Specify as mm/dd/yyyy. NULL for Default.
-l loglimit: Optional maximum invalid logon attempts. NULL for Default.
-r rsmlogon is the logon block period in minutes. Specifying a value of “block” (e.g., -r block) blocks the account indefinitely (until it is unblocked by an administrator, and specifying a value of “unblock” (e.g., -r unblock) unblocks the account immediately.
-t mstlogon is the interval in minutes during which the user must logon at least once, otherwise the account is blocked.
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oua user1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oua user2 -d "description" -w password -ggroup1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oua user3 -m 5000000g -b 01/31/2022 -e 02/20/2022

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oua user4 -l 10 -r unblock -t 15
Example for blocking users:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oue user1 -r block
Example for unblocking users:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oue user1 -r unblock

Option User Remove

-our userid

userid: User id (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -our user1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -our user2

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -our user3

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -our user4

Option User List

-oul

Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oul

Option User Change Password

-oup userid password

userid: User id (required)
password: New password (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oup user1 mynewpassword

Option User Add user to Group

-oug userid group

userid: User id (required)
group: Group name (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user1 group1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user2 group1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user3 group2

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user4 group2

Option User (Group) Extract - Remove a user from a group

-oux userid group

userid: User id (required)
group: Group name (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oux user1 group1

Option User Change Description

-oud userid desc

userid: User id (required)
desc: New user description
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oud user1 "My new description for user1"

Option User Memory

-oum userid -m memory rule

userid: User id (required)
memory: New memory limit. This can be a number of bytes or ‘D’ for default or left NULL for no limit
rule: Optional user memory rule. Used only with memory. This may be ‘A’ for Absolute, ‘G’ for Guideline, ‘D’ for Default, or NULL for Default
Example:
change user1 to 5 MiB as a Guideline

change user2 to unlimited

change user3 to default limit

change user4 to 2 MiB absolute fixed limit

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oum user1 -m 5000000g

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oum user2

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oum user3 -m d

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oum user4 -m 2000000a

Option User Change Extended Settings

-oue <userid> [-b <begdat>] [-e <enddat>] [-l <loglimit>] [-r <rsmlogon>] [-t <mstlogon>]

userid: User id (required)
-b begdat: Optional starting validity date. Specify as mm/dd/yyyy. NULL for Default
-e enddat: Optional ending validity date. Specify as mm/dd/yyyy. NULL for Default
-l loglimit: Optional maximum invalid logon attempts. 0 for Default. -1 to disable invalid logon check.
-t mstlogon: Optional must logon period, e.g., how often the user must log on to remain active. The interval in minutes during which the user must logon at least once, otherwise the account is blocked. Specify as number of minutes. NULL for Default. -1 to disable must logon period.
-r rsmlogon: Optional logon timeout remaining. If a user has been denied access to the FairCom Server due to excessive invalid logon attempts, you can adjust the remaining user lockout time here. Specify as number of minutes. NULL to leave unchanged. Specifying a value of “block” (e.g., -r block) blocks the account indefinitely (until it is unblocked by an administrator), and specifying a value of “unblock” (e.g., -r unblock) unblocks the account immediately.
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oue user1 -b 01/31/2022 -e 02/20/2022 -l 10 -r 1 -t 15

Option User Show

-ous userid

userid: User id (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ous user1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ous user2

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ous user3

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ous user4

Group Options

The following options, all beginning with -og, allow changes to group information. Additional user and file options are described elsewhere.

Note: To use any optional entry, you must use all the previous entries. For example, to specify a rule when adding a group with the -oga option, you must also enter the desc and memory options for the group.

Option Group Add

-oga <groupid> [-d <desc>] [-m <memory>][<rule>]]

groupid: Group id (required)
-d desc: Optional group description
memory is the memory limit and the optional <rule> is A for absolute, D for default, or G for guideline (example -m 10485760a specifies an absolute memory limit of 10 MB).
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oga group1
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oga group2 -d "Description for group2" -m 10485760a

Option File Group

-ofg filename groupid

filename: File name (required)
groupid: File group id (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofg "C:\FairCom\data\db1.dbs\owner1_table2.dat" OWNER1_RWD

Option Group Remove

-ogr groupid

groupid: Group id (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogr group1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogr group2

Option Groups List

-ogl

Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogl

Option Group Change Description

-ogd groupid desc

groupid: Group id (required)
desc: New group description
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogd group1 -d "group1 description"

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogd group2 -d "group2 description"

Option Group Memory

-ogm groupid [-m <memory>[<rule>]]

groupid: Group id (required)
-m memory: New memory limit. memory is the memory limit
- <rule> (optional) is A for absolute, D for default, or G for guideline (example -m 10485760a specifies an absolute memory limit of 10 MB).
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogm group1 -m 5000000g

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogm group2 -m d

Option Group Show

-ogs groupid

groupid: Group id (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogs group1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ogs group2

Option User Add user to Group

-oug userid group

userid: User id (required)
group: Group name (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user1 group1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user2 group1

Option User (Group) Extract - Remove a user from a group

-oux userid group

userid: User id (required)
group: Group name (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oux user1 group1

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user3 group2

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -oug user4 group2

File Options

The following options, all beginning with -of, allow changes to file information. Additional user and group options are described elsewhere.

Option File Password

-ofp filename password

filename: File name (required)
password: File password (required)
Example:
Set new password:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofp "C:\FairCom\data\db1.dbs\owner1_table2.dat" mynewpassword

Remove password:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofp "C:\FairCom\data\db1.dbs\owner1_table2.dat"

Option File Security (permissions)

-ofs <filename> <permission> ...

-ofs +|-<permission> ...

filename: File name (required)
permission: File permission mask.
To set a permission, set the byte at the corresponding offset to a value of ‘+’.
To reset a specified permission, set the corresponding byte to ‘-’.
For example, the string “+++++-----+++++” sets all OWNER and WORLD permissions, and clears all GROUP permissions.
This field is interpreted as a 15-byte permission mask containing owner, group, and world permissions:

(offset)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

----OWNER---- ----GROUP---- -----WORLD----

r w f d p r w f d p r w f d p

r = Read w = Write f = define d = Delete p = noPass

permission can also be one of the following:

ownerall, ownerread, ownerwrite, ownerdefine, ownerdelete, ownernopass,

groupall, groupread, groupwrite, groupdefine, groupdelete, groupnopass,

worldall, worldread, worldwrite, worlddefine, worlddelete, worldnopass

Options are evaluated left to right. For example, specifying -groupwrite +groupwrite has the effect of adding the groupwrite permission, and specifying +worldall -worldread turns on all world permissions except read permission.

Example - remove all file permissions:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofs "C:\FairCom\data\db1.dbs\owner1_table2.dat" -ownerall -groupall -worldall

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofs "C:\FairCom\data\db1.dbs\owner1_table2.dat" ---------------
Example - set all file permissions:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofs "C:\FairCom\data\db1.dbs\owner1_table2.dat" +ownerall +groupall +worldall

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofs "C:\FairCom\data\db1.dbs\owner1_table2.dat" +++++++++++++++
Example - set typical file permissions: owner ALL, group RWDP, world R:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofs "C:\FairCom\data\db1.dbs\owner1_table2.dat" +ownerall +groupall -groupdefine -worldall +worldread

sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofs "C:\FairCom\data\db1.dbs\owner1_table2.dat" +++++++-+++----

Option File Group

-ofg filename groupid

filename: File name (required)
groupid: File group id (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofg "C:\FairCom\data\db1.dbs\owner1_table2.dat" OWNER1_RWD

Option File Owner

-ofo filename owner

filename: File name (required)
owner: File owner (required)
Example:
sa_admin -aADMIN -pADMIN -f"" -sFAIRCOMS -ofo "C:\FairCom\data\db1.dbs\owner1_table2.dat" OWNER1

Examples of -ofs usage:

-ofs <filename> <permmask> is the same as current usage:

-ofs test.dat ++++++++++-----

-ofs <filename> <permission> ... sets the file permissions to the specified permissions. The following command sets all owner and group permissions and resets all world permissions:

-ofs test.dat ownerall groupall

-ofs <filename> +|- <permission> ... adds/removes specified permissions to/from current file permissions. The following command adds the worldread permission to the current file permissions and removes the groupwrite permission from the current file permissions:

-ofs test.dat +worldread -groupwrite

ctdump - Schedule Backup Utility

Note: For a complete discussion of dynamic dumps, see Backups and Data Integrity in the FairCom Server Administrator's Guide.

Operational Model:

Client

ctdump schedules a dynamic dump on the fly. The backup definition script may be located either on the server, or passed in from the client.

Command Syntax

ctdump [-s svn] [-u uid] [-p upw] [-t script] [-b bufsiz] [-n] [-c] [-o backup_filename] [-x]

Options

-s svn - c-tree Server name
-u uid - User name
-p upw - User password
-t script - Dump script name
-b bufsiz - Use buffer size of bufsiz bytes
-c - Send dump script from client
-m - Minimize progress notifications
-n - Send progress notifications to client
-o backup_filename - Write dump stream from server to file on client
-x - Write dump stream from server to stdout

For options available when scripting a dynamic dump, see Dynamic Dump Options.

The following demonstrates example usage of this utility:

ctdump ADMIN ADMIN thescript FAIRCOMS

Secure Authentication File

; User Id
USERID ADMIN
; User Password
PASSWD <pass>

Use the -1 option to specify the name of the encoded file.

Dump and Restore Version Compatibility

The ctrdmp utility is used to restore a Dynamic Dump and the ctfdmp utility can be used to roll forward. Occasionally an update to the FairCom Database Engine may cause an incompatibility between versions. For this reason you must use the ctrdmp from the same release from which the dump was created. It is important to save a copy of the ctrdmp utility that is compatible with each dump file. ctfdmp, ctldmp, and ctrdmp utilities display the FairCom DB version used to compile them when they are run.

Compression and Encryption

The ctdump.exe utility can accept compression and encryption utility inputs.

Windows backup compressed with 7z compression utility:

ctdump -s FAIRCOMS -u admin -p ADMIN -t dump.txt -c -x | 7z a -si backup.7z

Unix backup compressed and AES encrypted with standard utilities:

./ctdump -s FAIRCOMS -u admin -p ADMIN -t dump.txt -c -x | gzip|openssl enc -aes-256-cbc -salt >backup.gz.aes

Tip! Unix/Linux Alternative

This method also works with server-side dynamic dump backups prior to this new support and can be implemented on Unix systems using named pipes (mkfifo), and sending the dump to the named pipe.

Create a named pipe:

mkfifo testpipe

The pipe blocks until both sides are in use. So we start our compression command:

cat testpipe | gzip > backup.gz &

Now we create our dynamic dump, with "testpipe" as the dump file. (This example uses the client output, however it should work the same from the server):

ctdump -s FAIRCOMQ -u admin -p ADMIN -t dump.txt -c -o testpipe

c-tree(tm) Version 11.8.0.61131(Build-180802) Dynamic Backup Utility

Reading dump stream from server with buffer size of 100000

Dynamic Dump has been successfully written to the file testpipe.

The backup.gz file contains the dump file.

The same approach can be used to compress and encrypt (and potentially other operations provided by system utilities):

cat testpipe|gzip|openssl enc -aes-256-cbc -salt >backup.gz.aes

To extract the dump:

cat backup.gz.aes | openssl enc -aes-256-cbc -d |gzip -d >backup.ctree

Use ctrdmp to extract the files from backup.ctree.

Error Codes

The following error codes are related to dynamic dump operations:

Error Name	Error Code	Explanation
FUNK_ERR	13	Cannot determine file type. Possibly a c-tree V4.3 file?
READ_ERR	36	Failed to read file, either a corrupted or non-tree file.
TCOL_ERR	537	Transaction log collision. Two sets of transaction logs in the same directory?
FCPY_ERR	796	Immediate dump restore file copy failed.
DRST_ERR	797	Immediate dump restore failed.

ctrdmp - Backup Restore or System Rollback

Used to restore backups created with ctdump.

Operational Model:

Standalone

Usage:

ctrdmp [ dumpscript ] [ -x ]

dumpscript - The name of a valid dynamic dump restore script
-x - Read dump stream from stdin

A successful ctrdmp completion always writes the following message to CTSTATUS.FCS:

DR: Successful Dump Restore Termination

A failed ctrdmp writes the following message to CTSTATUS.FCS when ctrdmp terminates normally:

DR: Dump Restore Error Termination...: <cterr>

where <cterr> is the error code.

When the -x option is specified, the !DUMP keyword in the dump script is ignored and the dump stream is read from standard input.

This might be combined with the ctdump output redirection to pipeline a backup and restore operation:

ctdump -s FAIRCOMS -u admin -p ADMIN -t test6.dmp -c -x| ctrdmp test6.dmp -x

Note: If an error occurs during the restore phase, no backup exists on disk.

If encrypted files are being restored and input redirection is used, ctrdmp is not able to prompt for the master password during the recovery phase of the restore. In this case, an alternate means of providing the master password is required, such as using the CTREE_MASTER_KEY_FILE environment variable.

If for some reason ctrdmp terminates prematurely (for example, a fatal error causes ctrdmp to terminate abnormally), the “Dump Restore Error Termination...” message might not be written to CTSTATUS.FCS. In that case, ctrdmp might have written error messages to standard output or to CTSTATUS.FCS before terminating that helps explain the reason for ctrdmp terminating prematurely.

Note: A 32-bit ctrdmp could fail with error 75 if run on transaction logs created by a 64-bit FairCom Server, which might support more than 2048 connections.

The ctfdmp, ctldmp, and ctrdmp utilities display the FairCom DB version used to compile them when they are run.

Dump and Restore Version Compatibility

The ctrdmp (ctrdmp - Backup Restore or System Rollback, Backup Restore Utility) utility is used to restore a Dynamic Dump and the ctfdmp utility can be used to roll forward. Occasionally an update to the FairCom Database Engine may cause an incompatibility between versions. For this reason you must use the ctrdmp from the same release from which the dump was created. It is important to save a copy of the ctrdmp utility that is compatible with each dump file.

Restore Recovery Options

The ctrdmp utility now supports the RECOVER_DETAILS and RECOVER_MEMLOG transaction recovery options (the same options that FairCom Server supports).

If you specify !RECOVER_DETAILS YES in your dump restore script, ctrdmp will log progress messages to the file CTSTATUS.FCS as it performs its automatic recovery.

Environment Variable for Advanced Encryption Password

If this utility has advanced encryption enabled, it can read an encrypted password file instead of being prompted to enter the master password. To enable this, set the environment variable CTREE_MASTER_KEY_FILE to the name of the encrypted master password file.

ctrdmp supports options to help analyze recovery behavior. The following options behave like their corresponding server configuration options:

!DIAGNOSTICS TRAN_RECOVERY logs detailed recovery steps to RECOVERY.FCS.
!RECOVER_DETAILS YES logs additional recovery progress messages to CTSTATUS.FCS.

See also

Maintaining Database Integrity in the c-tree Server Administrator's Guide

In This Section

Rollback to New Restore Points with ctrdmp

In V11 and later, ctrdmp is able to rollback to a Restore Point. Restore Points permit server clients to establish quiet spots in the transaction log where there are no active transactions.

Prior to the V11 modifications, ctrdmp could either perform a dynamic dump recovery or rollback to a specified date and time. ctrdmp has been extended such that, as an alternative to specifying a date and time, the rollback script can provide the name of a Restore Point file.

A typical ctrdmp script file used for a rollback looks like:

!ROLLBACK

!DATE MM/DD/YYYY

!TIME HH:MM:SS

....

Now the script can be composed as follows:

!RP <Restore Point File Name>

....

The Restore Point File Name generated by the server is either of the following:

RSTPNT_NO_CHK.YYYYMMDD_HHMMSS.FCS for a Lightweight Restore Point
RSTPNT_CHKPNT.YYYYMMDD_HHMMSS.FCS for a Checkpoint Restore Point

Note that, as with the !ROLLBACK script keyword, the !RP keyword must be the first entry in the script file.

See also

ctrdmp - Dynamic Dump Recovery or System Rollback (ctrdmp - Backup Restore or System Rollback, Backup Restore Utility)
ctfdmp - Forward Dump Utility

ctfdmp - Forward Roll Utility

Operational Model:

Standalone

ctfdmp

Used to restore data to a given time following a ctrdmp restore.

ctfdmp takes an optional !RP <name> argument to set the point in time to stop the forward roll. This argument is also used by the ctrdmp script option, as described in Rollback to New Restore Points with ctrdmp. To incrementally roll forward from there, rename the previous RSTPNT_CHKPNT*.FCS to S0000001.FCS (the start point for the ctfdmp), and supply a new !RP and transaction logs. More information on the !RP <name> can be found here: ctfdmp - Forward Roll Utility

Note: The ! prefix needs to be escaped when using the Unix Bash shell.

Environment Variable for Advanced Encryption Password

ctfdmp accepts !RECOVER_DETAILS for additional progress notifications diagnostic output.

See also

Maintaining Database Integrity in the c-tree Server Administrator's Guide
ctrdmp - Dynamic Dump Recovery or System Rollback (ctrdmp - Backup Restore or System Rollback, Backup Restore Utility)
Rollback to New Restore Points with ctrdmp

ctldmp - Transaction Log Dump

Operational Model:

Standalone

Usage:

ctldmp [ DATE mm/dd/yyyy ] [ TIME hh:mm:ss ] [ LOG number ] [ scanerr ]

Performs a partial dump of the transaction logs to assist the developer in problem resolution.

ctldmp displays the file ID component header values that are included in file open, close, and file ID reassignment log entries.

The scanerr option causes the utility to scan the transaction log for the next valid log entry if an invalid log entry is read.

Example output

LOGPOS:01-00015ae2x #0:000015 P005ea054b5-00x F0022-600 T27 U11 A0000x L116 042 OPNTRAN

0000000000000000000000000000000000005313b5a544545545445244500000000000000000000

10000000000000000000000000000000a0006429440e462b3414544e63300000000000000000000

....................................V4.9.T.^DFRKSTATEDT.FCS....................

fileid=10 (0x0000000a)

servid=957494358 (0x39123456)

timeid=1587565748 (0x5ea054b4)

See also

Maintaining Database Integrity in the c-tree Server Administrator's Guide
Options for Transaction Log Dump in the c-tree Server Administrator's Guide

ctldmp option to create transaction start files from checkpoints in transaction log files

The ctldmp utility allows you to create transaction log start files for the transaction logs that it scans. Specify the csf ("create start files") option to use this feature.

The start files are named S<lognumber>_<sequencenumber>.FCA, where lognumber is the transaction log number and sequencenumber is the checkpoint number in that log (starting from 1).

These start files can be renamed to S0000000.FCS and S0000001.FCS so that FairCom Server can use the checkpoint positions as starting points for automatic recovery.

Example:

# ctldmp log 1254 csf

getlogfil

LOGPOS:1254-002473fax #0:690034176 P0051a057f3-00x F0000-000 T26 U03 A0000x L139672 042 CHKPNT

0000e0007fa0a000a0009200920073000000000071001200e000000000005100000000000000000

300054001cb100002000020042002c00000030008a20f0003400000040004b20100000000000000

........q............""...""..r<..........x.... ..............T................

Created start file S0001254_0001.FCA for log position 0x002473fa

Sat May 25 01:19:31 2013

LOGPOS:1254-0026a624x #0:690034177 P0051a057f3-00x F0000-000 T26 U17 A0000x L10684 042 CHKPNT

ffffe000f720a000a0009200920092000000000092001200e000000000007200000000000000000

bfff6400a3400000200002004200820000003000c200f0003400000040008300100000000000000

.........s$..........""...""...""...........""... ..............x#.............

Created start file S0001254_0002.FCA for log position 0x0026a624

Sat May 25 01:19:31 2013

ctheader - Header Update Utiilty

Operational Model

Client (ctheader.exe)
Standalone (ctheader.standalone.exe)

Description

Updates various header fields, allowing for operations such as enabling/disabling particular file features or repairing files with damaged headers.

WARNING: Improper use of this utility can render your file unusable. It should be used rarely and carefully.

Usage

ctheader [-s server] [-u uid][-p pwd] -f filename -m mode -v val

-s: server name (default: FAIRCOMS)

-u: username (default: admin)

-p: password (default: ADMIN)

-f: filename

-m: mode (string). One of the following names:

ctSERNUMhdr - Set serial number to the specified value
ctTSTAMPhdr - Set timestamp to the specified value
ctNUMENThdr - Set number of entries in file to the specified value
ctTIMEIDhdr - Set creation time stamp used as part of unique ID to the specified value
ctLOGEOFhdr - Set logical end of file to the specified value
ctPHYEOFhdr - Set physical end of file to the specified value
ctDELSTKhdr - Set delete stack pointer to the specified value
ctUNIQIDhdr - Set unique file ID to the specified value
ctALIGNhdr - Set file alignment to the specified value
ctFLAVORhdr - Set file byte ordering to the specified value
ctXFLSEThdr - Set extended file mode to the lower-order two bytes of the value parameter
ctXFL_ONhdr - Turn on the lower-order two bytes of value in the extended file mode
ctXFLOFFhdr - Turn off the lower-order two bytes of value in the extended file mode
ctEXTSIZhdr - Set file extension size to the specified value
ctSUSSRLSEGhdr - Suspend the serial segment handling for data file (YES or NO)
ctREPLICATEhdr - Enable replication for this file (YES or NO)
ctMAXPARTMBRhdr - Set maximum partitions for this host to the specified value and auto purge as needed
ctIFIL3hdr - Turn On / Off DEF_MASK_IFIL3 - IFIL resource format change
ctMXFILZhdr - Set maximum file size to the specified value
ctLXTSIZhdr - Set large file extent size to the specified value
ctGUIxIDXENThdr - Turn on/off DEF_MASK_PRTENT (YES/NO)
ctInsertOnlyhdr - Turn on/off ctInsertOnly (YES/NO)
ctAugmentedFxdhdr - Turn on/off ctAugmentedFxd (YES/NO)
ctADD2ENDhdr - Turn on/off ctADD2END (YES/NO)

-v: Update header value (integer in decimal format)

Example

Enable replication:

ctheader -f myfile.dat -m ctREPLICATEhdr -v 1

Performance

The following items can be considered for optimizing FairCom Server throughput and are intended for use by advanced users only. “Optimizing Transaction Processing” can make disaster recovery difficult or impossible!

I/O caching

If the machine where the FairCom Server resides has sufficient memory, and the size of the files controlled by the FairCom Server are relatively large, increasing DAT_MEMORY and IDX_MEMORY could increase performance. In general, the larger the data and index cache sizes, the better the performance for heavy use situations. The FairCom Server uses a hashed caching algorithm, so there is no need for concern with having the cache sizes set too large.

SetOperationState

SetOperationState() sets operation modes for special performance related functionality and tests the operational states for critical events. With proper use, this can potentially reduce the number of requests between a client and Server, decreasing network traffic and user response times. This function is intended for advanced users.

Batch Operations

DoBatch() can enhance performance for reading keyed sequential data by decreasing the amount of network traffic. Instead of making individual function calls for each record in a related group, DoBatch() can retrieve many records in one call. However, many low-level packet transmissions may be required to retrieve a large block of records, so be sure to use DoBatch() wisely. If just one or two records are necessary, it may be quicker to perform individual retrievals rather than retrieving all the records that match a generic target.

ISAM Functions

It is more effective to use ISAM functions with client/server based applications to minimize the network traffic. Using low-level functions with client/server based applications introduces a large volume of unnecessary network traffic.

ctDUPCHANEL File Mode

In selected environments, the FairCom Server can use the ctDUPCHANEL file mode to provide an additional file handle into heavily used files, enhancing throughput. See the ctDUPCHANEL discussion in File Modes.

Fastest Server

A commonly asked question is which FairCom Server offers the fastest response times. The answer is the performance of the FairCom Server is closely tied to the host hardware and the communication protocol, as discussed in the next section. The faster the CPU and disk I/O subsystem, the faster the FairCom Server responds. The internal performance differences for the FairCom Server across platforms are negligible. The decision of which hardware platform to choose for the FairCom Server should be based on the optimum hardware specifications. The order of priority for the hardware specifications are:

Fastest Disk I/O Subsystem
Fastest CPU
Fastest and most supported RAM

Communication Protocol

The FairCom Server supports many different operating system/hardware combinations in addition to many different communication protocols. Typically, the largest I/O bottleneck with the client/server model is the communication protocol used to communicate between the FairCom Server process and the client process. Choosing the best suited communication protocol for the FairCom Server plays a crucial role in the client side response times. Due to the variables involved in providing response times, (record size, quantity of records, number of users, network traffic, speed of the network cards...), it is impossible to provide an absolute guideline for which protocol to use. The best way to determine the best protocol for the target platform is by conducting time trials with the available protocols. However, as a general guideline, the protocols that work on the local machine will typically be the fastest (i.e., shared memory).

Optimizing Transaction Processing - ADVANCED

The FairCom transaction processing logic offers three modes of operation:

No transaction control at all
ctPREIMG - partial transaction control (atomicity only)
ctTRNLOG - full transaction control (atomicity and automatic recovery)

Transaction processing is covered in detail in Data Integrity.

In This Section

Transaction Control Background

Transaction Options

Transaction Control Background

No transaction control

With no transaction control defined for a data file, the read/write access to the file will be very quick. However, no guarantee for data integrity will be available through atomicity or automatic recovery.

ctPREIMG

If just the ctPREIMG file mode is used, the database access will still be fast, and some data integrity will be provided through atomicity. With atomicity only, changes can be made on an all-or-nothing basis, but no automatic recovery is provided.

ctTRNLOG

If your application files have been setup with the ctTRNLOG file mode, all benefits of transaction processing are available to you, including atomicity and automatic recovery. Automatic recovery is only available with ctTRNLOG, because ctTRNLOG is the only mode where all changes to the database are written immediately to transaction logs. The presence of this transaction log, (history of changes to the files), allows the Server to guarantee the integrity of these files in case of a catastrophic event, such as a power failure. Recovering files without a ctTRNLOG file mode from a catastrophic event entails rebuilding the files.

Note: File rebuilding cannot recover data that was not flushed to disk prior to the catastrophic event.

Transaction Options

SUPPRESS_LOG_FLUSH

Full transaction processing offers the best possible data integrity, but at the expense of performance. Using SUPPRESS_LOG_FLUSH reduces overhead with transaction processing log flushes, at the expense of the automatic recovery. This keyword would only be used with the PREIMAGE_DUMP keyword described below.

PREIMAGE_DUMP

Although automatic recovery is not available to ctPREIMG files, it is possible to perform periodic dynamic backups. By using PREIMAGE_DUMP, it is possible to promote ctPREIMG files to full ctTRNLOG files during the dynamic dump process. The promotion to a ctTRNLOG file means that a full transaction log is maintained during the dump process. This process guarantees that any changes made to the files during the backup are saved in these specially maintained transaction logs. The ability to dynamically backup user data files minimizes the loss of the automatic recovery feature with this mode.

Transaction Log Flush Delay

Increasing the amount of time before the transaction logs are flushed can increase the number of transactions sent to disk with each flush. This activity is controlled with the COMMIT_DELAY keyword. See COMMIT_DELAY in the FairCom Server Administrator’s Guide for additional information.

Resource Records Detected

Several common API calls request resource information, such as DODA and alternative collating sequences, from the FairCom Server. Ordinarily, these requests are only made once after a file is opened. However, an application that frequently opens and closes a file repeats these requests. A file which has explicitly disabled resources, file mode on create includes ctDISABLERES, is automatically detected so the information is not requested, reducing network traffic.

Automatic DODA requests to process record information can be reduced or eliminated, independent of ctDISABLERES, by turning on the USERPRF_NDATA user profile bit, explicitly disabling automatic record transformations in heterogeneous environments. Disabling automatic record transformation is only desirable when performing your own record level byte flipping or your record structures contain no numeric data, (i.e., LONG, FLOAT, . . .). See InitISAMXtd for additional information on user profile bits.

FairCom Server Cache Subsystem

The FairCom Server internal data and index file caching schemes provide a balanced approach to minimizing disk activity while ensuring data integrity in environments with high transaction volume. Data integrity, transaction control, and concurrency issues are but a few of the many internal complexities involved in the cache subsystem. To properly put the benefits of the caching subsystem into perspective, one must understand all the options, facts and trade-offs involved in supporting an advanced cache in environments with high transaction volumes.

This section clarifies questions raised about the FairCom Server cache subsystem and describes features affecting cache management:

Changing the number of operations performed in one pass of the cache
Adjusting the amount of a large record to keep cached.
Disabling cache for a file.
Dedicating cache to a file and limiting dedicated cache.
Loading files to cache at open.
Avoiding file flush at server shutdown.

Allocating Memory for Data/Index Caches

In general, FairCom Server cache is easy to configure. By placing the following keywords in the FairCom Server configuration, the Server Administrator defines the memory size for both data and index caches. The values represent bytes. This example sets each cache to 20MB:

IDX_MEMORY 20000000

DAT_MEMORY 20000000

Because the default value is very low (250,000 bytes), it is strongly recommended to specify an increased cache size. The FairCom Server keeps the most recently accessed data in memory using a “most-recently-used” scheme to determine which pages remain in memory and which do not. FairCom Server cache-management techniques allow the server administrator to set the cache sizes as high as reasonably possible, without defeating the purpose of enhancing performance.

Cached Data - Writing to Disk

In general, the purpose of the cache is to prevent reading and writing data to/from the disk. The data managed in the cache accesses the disk for a number of reasons:

File Extension Size: If any index or data file has been defined to have a “file extension size” (as defined in the IFIL or IIDX structures), these files are defined to require disk writes. Specifying an extension size tells FairCom DB to extend the file by the given number of bytes each time it runs out of space. To secure this extension, c‑tree forces the disk write, thus ensuring the operating systems file allocation table (FAT) registers this new extended size. The best way to ensure maximum performance is to define a ZERO (0) FILE EXTENSION SIZE for all files. A value of zero suppresses this operating system overhead.
File Header Updates: There are a number of situations where the server updates the control header portion of a c-tree file. Updating the number of active records in the file is a good example. Typically, a file header update does not necessitate a disk write, but some situations cause a required disk write. Perhaps the best example is when adding a new root node to an index file, which requires a file header write. These types of circumstances are very infrequent and should offer no apparent overhead to the overall performance of the system.
Not Enough Cache Space: Of course, if the demand on the cache exceeds its defined limit, data not currently being used will be written to disk. Cached index pages are analyzed using a “most-recently-used” strategy to determine which pages remain in cache.

Starting with V11, a new caching strategy is used for the data cache (index cache is still “most-recently-used”). The data cache gets the next page from one of several lists of pages set up at startup time. All of the cache pages are included in the lists; each page is on only one list. The number of such parallel page lists is specified by the DATA_LRU_LISTS configuration entry.

Middle Portion of Variable-length Records: c-tree does not store the middle portion of variable-length records in cache, only the first and last pages. This prevents large blocks of data from consuming the cache and also alleviates the management of a large number of cache pages for any one particular record. There is increased overhead when managing a large volume of individual cache pages for any one particular record. The cache pages for consecutive segments of a record (where a segment fills a cache page) are completely independent of each other - they are not stored in consecutive memory. I/O is performed separately for each cache page, therefore if a large number of cache pages exist; it is likely to be slower than faster.
File Open and Close Operations: Any action to open or close a file causes disk I/O. Besides the obvious reasons, a number of internal operations take place during these critical operations.
Transaction Control: To manage transactions, there are a number of transaction processing operations that require disk flush activity. Securing transaction logs, issuing checkpoints, and handling transaction cache ageing are a few examples of disk activity that support automatic recovery to ensure data integrity.

Write-Through Cache

Our goal is to offer the most efficient caching scheme while assuring complete data integrity. FairCom does not categorize its cache as what some have called a “write-through” cache. The file-size extension issue discussed above is one reason. Given a zero file size extension, all “adds” to the database should go directly to cache, with no disk activity. The ctWRITETHRU file mode can be used to turn on write-through caching behavior for a file.

Does c-tree cache everything?

Not everything is cached, for example:

As mentioned, the middle portions of variable-length records are not cached.
As defined below, a file may be specified “not to be cached”.
Data “not recently used” when the cache is full will generate disk activity.

For example, when writing/reading a 100K record, do you try to cache it, thus wiping out the utility of the cache? No! Because we only retain the first and last page of a variable-length record in the cache, this problem does not exist.

Size Limits on What to Cache

The approach used for variable-length records alleviates any concern for size limits on what you try to cache.

Caching Variable-Length Records When the Entire Record Does Not in Cache (NO_CACHE)

For a file that has mostly records of length greater than two times the PAGE_SIZE, NO_CACHE might be beneficial because the middle portion of the record is not cached. With a page size of 32KB, this means records up to 64KB in size will be cached (since the first and last page of a variable length record are cached). An example of when to use NO_CACHE is writing variable-length journal files that seldom need to be read, the NO_CACHE could be a good choice for this type of file.

Advanced Cache Settings

While the default cache settings should be considered efficient, it is possible to adjust the following settings to further tune performance to the specific application needs.

The topics in this section explain the cache settings available for the cache subsystem.

Prime Cache at Server Startup

This feature is new with V13.

FairCom Server technology supports configuration options that are used to prime the server's data and index caches when the server starts up. The functionality is similar to cache priming configuration options PRIME_CACHE, PRIME_CACHE_BY_KEY, and PRIME_INDEX, except that those options take effect only when an application opens the files.

This feature relies on atomic operations. If atomic operation support is off at compile time, the feature will be disabled.

The new configuration options are specified in JSON format, such as in the following example:

subsystem cache prime_cache_at_startup {

"allow_connections": true,

"thread_count": 8,

"prime_cache": [

{"file": "filename1.dat", "size": "1 gb"},

{"file": "filename2.dat"}

"prime_cache_by_key": [

{"file": "filename1.dat", "size" : "1 gb", "index": 1, "reverse": false},

{"file": "filename2.dat", "size" : "1 gb", "index": 1}

"prime_index": [

{"file": "filename1.dat", "size": "1 gb", "index": 1},

{"file": "filename2.dat", "size": "1 gb", "index": 1}

]

}

Supported attributes:

allow_connections is a boolean value that indicates if connections to the server are allowed while the cache priming is performed at server startup. Defaults to false.
thread_count is an integer value that sets the number of threads to use to prime the cache. It defaults to 4. Maximum is 255.
prime_cache is an array of names of data files that are to be read into the cache in physical order, and an optional size indicating the maximum number of bytes to read into data cache for the file. The file name is a string. The file name can include wildcard characters (for example, *.dat). The wildcard characterdoes not recurse into subdirectories. The size can include a suffix, such as kb, mb, gb, tb (for example, "1 gb"). If size is not specified, the entire file is read into the data cache.
prime_cache_by_key is an array of names of data files that are to be read into the cache in the order of the specified index. The index number is a positive value, where 1 means the first index. If the index number is omitted, the first index is used. Size has the same definition as for prime_cache. Reverse is a boolean indicating whether the key traversal is in ascending or descending key order.
prime_index is an array of names of index files that are to be read into the index cache. Size and index have the same meaning as for prime_cache_by_key.

In addition to the specified size limits, the data and index cache sizes limit the cache priming as follows:

prime_cache and prime_cache_by_key stop if the data cache becomes full.

prime_index stops if the index cache becomes full.

The SUBSYSTEM CACHE PRIME_CACHE_AT_STARTUP configuration option can be specified in a server settings file. If it is used in a settings file, the option cannot also be used in the configuration file. In that case, the subsystem is considered blocked and the configuration file options are ignored. Likewise, if this subsystem is specified more than once in the configuration file, only the first occurrence of the subsystem block takes effect. The other occurrences are ignored.

The subsystem can be commented out by placing a semicolon at the start of the SUBSYSTEM line. However, individual lines within the subsystem block cannot be commented out. Attempting to do so will cause a syntax error.

By default, the server logs information about the files that it loads into cache to CTSTATUS.FCS. The configuration option CTSTATUS_MASK PRIME_CACHE_AT_STARTUP can be used to disable this logging.

Priming Cache

The FairCom DB database engine optionally maintains a list of data files and the number of bytes of data cache to be primed at file open. When priming cache, FairCom DB reads the specified number of bytes for the given data file into data cache when physically opening the data file.

Data files are added to the priming list with configuration entries of the form:

PRIME_CACHE <data file name>#<bytes primed>

The <bytes primed> parameter accepts scaling factors (for example 2GB).

Example: The following keyword instructs the Server to read the first 100,000 bytes of data records from customer.dat into the data cache at file open:

PRIME_CACHE customer.dat#100000

A dedicated thread performs cache priming, permitting the file open call to return without waiting for the priming to be completed.

Use PRIME_CACHE with the SPECIAL_CACHE_FILE keyword to load dedicated cache pages at file open.

To prime index files, use configuration entries of the form:

PRIME_INDEX <index file name>#<bytes primed>[#<member no>]

If the optional <member no> is omitted, all index members are primed. If an index member number is specified, only that member is primed.

For example, the following keyword instructs the Server to read the first 100,000 bytes of index nodes in customer.idx into the index buffer space at file open:

PRIME_INDEX customer.idx#100000

The nodes are read first for the host index, and then for each member until the entire index is primed or the specified number of bytes has been primed.

The following example restricts the priming to the first member (the index after the host index):

PRIME_INDEX customer.idx#100000#1

The <data file name> or <index file name> can be a wild card specification using a ‘?’ for a single character and a ‘*’ for zero or more characters.

FairCom DB also supports priming the data cache in forward AND reverse order by index.

PRIME_CACHE_BY_KEY <data_file_name>#<data_record_bytes_to_prime>#<index_number>

For example

PRIME_CACHE_BY_KEY mark.dat#100000000#-1

Primes up to 100,000,000 bytes from mark.dat reading by the first index in reverse key order.

To use the prime cache file list for a bound server, use ctFILELIST() calls of the following form, where filename points to the file name, bytes is a LONG holding the number of bytes of index buffers to prime at file open, and member is a LONG holding the index member number. A negative value for member indicates the entire index.

ctFILELIST(filename,&bytes,ctPRICACHElist,ctADDfilelist);

ctFILELIST(filename,&bytes,&member,ctPRIINDEXlist,ctADDfilelist);

This function must be called prior to the opening of the file for which the cache is to be primed. The cache priming is performed by a dedicated thread launched by the Server at the end of the file open. This permits the open to return without waiting for the priming to be accomplished.

Scanner Cache

Note: The LRU algorithm is no longer used for the data cache.

The index cache pages are managed using a least-recently-used (LRU) scheme such that the page that has not been touched the longest will be the page selected for re-assignment when a page is required to satisfy a current request. For indexes, this simple strategy works very well since traversing an index tree frequently requires access to the same, important high level nodes (such as the root node) and/or to the “current” leaf node.

For data files, the LRU scheme is not always effective. In particular, when data file access or updates involve many different records with little or no revisiting of records once they have been processed, the LRU scheme can result in many cache pages to be assigned to recently accessed records. But, at least from the user’s perspective, there is little chance of revisiting these pages.

Consider simply traversing a large set of records using NXTREC()/NXTVREC(): it would be possible to exhaust the cache with the contents of these records; but if no access beyond the primary access is required, these cache pages may be wasted. Even when we consider adding a large number of records, especially to the end of a file, this presents the same problem.

Using the advanced FairCom DB header mode, ctSCANCACHEhdr, allows an alternative caching strategy. The scanner cache strategy uses all the standard cache routines except when requiring a cache page (that is, FairCom DB has not found the data block it needs already in cache), it uses one of two dedicated cache pages.

Example

The following call will enable the scanner mode of the file, specified by the file number datno, to use the scan cache strategy:

PUTHDR(datno, YES, ctSCANCACHEhdr);

The scanner only goes to the LRU list if it has not yet dedicated both cache pages. Once dedicated, these pages may be referenced by other users, but they are not placed back on the LRU until either the file is closed or a scanner mode is toggled off.

Note: The strategy only applies to the calling connection's access to datno; each connection controls its own data cache strategies for the files it opens.

The significance of this strategy is that a large set of data record operations can be performed with only two cache pages with little or no decrease in performance compared to no restrictions on cache usage. Moreover, such restricted use may help other aspects of system performance since the bulk of data cache can be used for other files and/or other users. A large traversal of the data file will not replace other cache pages.

Two dedicated cache pages were chosen since two pages are always sufficient to hold the beginning and end of a record image which is typically all that is stored in the cache. A short record can always fit in one or two cache pages, and a long record has its interior (i.e., that portion which does not fit within the two pages) read or written directly (regardless of scanner mode unless MPAGE_CACHE has been enabled), bypassing the cache.

Particular Cases

SPECIAL_CACHE_PERCENT

The previous dedicated cache strategy uses the keyword SPECIAL_CACHE_PERCENT to specify the maximum percentage of the data cache that can be dedicated. This percentage defaults to 50%. If the number of dedicated cache pages is already at this limit, then the scan cache approach will not take effect until the total number of dedicated cache pages falls below the limit.

SPECIAL_CACHE_FILE

The SPECIAL_CACHE_FILE list of files with dedicated cache does not affect the scan cache strategy: a user can call PUTHDR() for any data file whether or not it is on the SPECIAL_CACHE_FILE list. The SNAPSHOT statistics for special cache pages include dedicated scan cache pages.

Disabling File Cache

In some cases, it might be beneficial to define that a certain file NOT be cached.

For example, if a file contains very large variable-length records (BLOBS), it might be more efficient to bypass the cache and rely solely on the operating system's cache support. The FairCom Server does not store the full variable-length record in cache, but retains the first and last page of the variable-length record. This prevents large blocks of data from consuming the cache and also alleviates the management of a large number of cache pages for any one particular record.

To disable cache for a given file, use the following server configuration keyword:

NO_CACHE <data file name>

Note: <data file name> may include a wildcard pattern using ‘?’ for a single character and ‘*’ for zero or more characters. The Server Administrator can specify multiple NO_CACHE configuration entries.

Caching can only be turned off for entire superfiles (i.e., the superfile host), not for individual superfile members. Index files require the use of index cache pages and must be cached.

Limiting Special Cache Space

To specify the percentage of the overall data cache space that may be dedicated to individual files, use a configuration entry of the form:

SPECIAL_CACHE_PERCENT <percentage>

For example, the following keyword would permit up to 10% of the total data cache pages to be assigned to files on the special cache file list:

SPECIAL_CACHE_PERCENT 10

To disable any special cache, enter -1 for the percentage. The percentage defaults to 50% and the maximum amount that can be specified with the keyword is 90%.

For a bound server, the dspllmt member of the ctINIT structure may be used to override the 50% default. For a single-user, standalone application, the ct_dxspllmt must be set to the percentage before the system is initialized (which may require a call to RegisterCtree() if ctNOGLOBALS is defined).

Avoiding File Flush at Server Shutdown

With the FairCom cache system, it is possible for users to have extremely large cache sizes (e.g., 2GB). With caches this large, it may be possible for a non-tranlog file to never be written to disk during its entire life cycle (tranlog files are written to disk by background flushing). The c‑tree Server flushes files to disk when it is shut down. In the case of a “scratch” or “temp” file, it may not be necessary to flush the file to disk. If the file is large, this will cause extra time for the server to shut down, and you might not want to suffer this extra shutdown time.

To handle this situation, the server keyword NO_SHUTDOWN_FLUSH allows the server to skip flushing non-transaction controlled files during server shutdown. Non-transaction controlled files can be specified as shown below for this treatment. Notice that such a file will be corrupted after shutdown (if file flushing was indeed skipped). For a server, configuration entries are of the form:

NO_SHUTDOWN_FLUSH <file name>

Note that <file name> may specify a wildcard pattern.

FairCom provides two keywords to flush updates for non-tranlog files to the file system as soon as possible in the background:

NONTRAN_DATA_FLUSH_SEC IMMEDIATE

NONTRAN_INDEX_FLUSH_SEC IMMEDIATE

For non-tranlog files, cache writes occur only for background flush, page replacement, explicit flush request, or file close.

NO_SHUTDOWN_FLUSH <file name> applies only to non-tranlog files. Furthermore, it only applies when NONTRAN_DATA_FLUSH_SEC IMMEDIATE and NONTRAN_INDEX_FLUSH_SEC IMMEDIATE are not in ctsrv.cfg. These settings are present by default.

To use the skip file flush at shutdown file list for a bound server, use calls of the following form, where filname points to the file name, bytes and member are LONG, but the values are ignored:

ctFILELIST(filname,&bytes,&member,ctNO_FLUSHlist,ctADDfilelist);

Multi-Page Cache

The c-tree data cache uses the following approach to cache data record images:

If the data record fits entirely within one or two cache pages (PAGE_SIZE bytes per cache page), then the entire record is stored in the cache.
If the data record image covers more than two cache pages, then the first and last segments of the record are store in the cache, but the middle portion is read from or written to the disk. These direct I/O’s are efficient operations since they are aligned and are for a multiple of the cache page size.

The nature of this approach can be modified. Set the Server keyword MPAGE_CACHE to a value, N, greater than zero, and records that fall within N+2 cache pages will be stored entirely within the cache. The default value is zero, which causes the cache to behave as described above.

Note: Setting MPAGE_CACHE greater than zero does NOT ensure faster system operation. It is more likely to be slower than faster. It does cause more of a record to be in cache, but there is increased overhead managing each individual cache page. The cache pages for consecutive segments of a record (where a segment fills a cache page) are completely independent of each other. They are not stored in consecutive memory and I/O is performed separately for each cache page. This configuration option should only be used for special circumstances with careful, realistic testing.

Note: Even a record smaller than a single cache page may require two cache pages because the record position is generally not aligned on a cache page boundary.

For a bound server, the MPAGE_CACHE configuration parameter is stored in the mpagche member of the ctINIT structure. For Standalone applications, the configuration parameter is stored in the “global” ctmpagcache integer variable. When ctNOGLOBALS is defined, the application instance must be registered before ctmpagcache can be reset. A client cannot affect this value on the Server.

Buffer Run Length

The Server keyword, BUFFER_RUNLENGTH, which defaults to 10, determines how many consecutive write operations are performed while walking a list of buffer/cache pages before the mutex controlling access to the list is released and then reacquired. Releasing the mutex permits other threads to acquire control of the list. If this value is set to a negative value, it is ignored.

For bound server applications, the brunlen member of the ctINIT structure is used to override the default.

Other Notes

Use both the PRIME_CACHE and SPECIAL_CACHE_FILE keywords for a data file to load dedicated cache pages at file open.

The ctFILELIST() processing, used to store and retrieve cache parameters, ignores the mirrored portion of a file name. Only the primary file is cached.

If multi-byte file names are supported (e.g., Unicode), the configuration information (or direct call to ctFILELIST()) must have the file names in UTF8 form (i.e., a NULL terminated string).

SystemConfiguration Cache/Buffer Statistics

SystemConfiguration() returns nine values referenced with the following constants used as subscripts in the output array of LONG values an application can use to capture system-wide cache and buffer statistics, (cache pages hold data record images and buffers hold index nodes), allowing an application to track the use of these resources.

SystemConfiguration Return Value	Description
cfgCACHE_PAGES	Available cache pages
cfgCACHE_PAGES_INUSE	Cache pages in use
cfgCACHE_PAGES_MXUSE	Maximum cache pages used
cfgCACHE_PAGES_DED	Available dedicated cache pages
cfgCACHE_PAGES_DEDINUSE	Dedicated cache pages in use
cfgCACHE_PAGES_DEDMXUSE	Maximum dedicated cache pages used
cfgBUFFER_PAGES	Available index buffers
cfgBUFFER_PAGES_INUSE	Index buffers in use
cfgBUFFER_PAGES_MXUSE	Maximum index buffers used

Note: The dedicated cache pages are a subset of the regular cache pages.

See the following example for the use of this feature.

SystemConfiguration() Example

LONG ctcfg[ctCFGLMT];

SystemConfiguration(ctcfg);

printf("\nAvailable Dedicated Cache Pages - %ld",

ctcfg[cfgCACHE_PAGES_DED]);

File Modes

The FairCom Server supports all of the file modes described in this guide (see Data and Index Files and Multi-User Concepts). In addition, the following modes are available:

Record Locking Mode

Although these modes are specified when opening or creating a file, they affect how the FairCom Server handles record locks. If specified at file creation, these mode values become a permanent part of the file attributes.

ctCHECKLOCK

If added at file open or create, FairCom DB requires a record lock before a record can be updated. If a lock is not obtained, the error code DADV_ERR (57) is returned. Do not apply this file mode to index files.

ctCHECKREAD

As above, except that it applies to record reads instead of updates. Obtain at least a read lock on a record before it can be read, otherwise the function will return error code DADV_ERR (57). Do not apply this file mode to index files.

Note: ctCHECKLOCK and ctCHECKREAD are intended as debugging aids to help developers to identify code locations where locks are not being acquired.

ctCHECKREAD is not compatible with ctSUSPEND because ctSUSPEND will disable record lock acquisition without releasing existing locks, so any record read would then trigger the ctCHECKREAD error condition.

ctCHECKLOCK and ctCHECKREAD should not be applied to index files.

I/O Management

ctDUPCHANEL

In some environments where FairCom uses native operating system threads, it may be possible to allow FairCom DB to use more than one file handle or I/O channel for a given file or index. If the operating environment supports this, it can significantly improve the file access performance of FairCom DB. For instance, if two users read from and write to the same file at the same time, the FairCom Server can improve the performance of the system by using multiple handles into the same file, up to #define NUMCHANEL, presently set at 2.

Once established using the ctDUPCHANEL file mode, the operation is transparent to the client. FairCom Server manages it automatically.

Specify ctDUPCHANEL for files that have heavy simultaneously use by multiple users. The benefits may decrease if a large number of files are open with this mode. In addition, with a large number of files and a restricted number of file handles available from the operating environment, you may not see an improvement.

Deployment Options

The following developer deployment options allow developers to ensure the FairCom Server operates as the application requires.

The Tamper-Proof Server Configuration File or settings file, ctsrvr.set by default, enforces developer-specified configuration settings. Because entries in the settings file cannot be overridden, users cannot alter important aspects of the Server configuration. A special activation key from FairCom can make the settings file mandatory, forcing the FairCom Server to operate with your specific settings. The settings file is encrypted to prevent unauthorized modifications.
Some configuration options are specific to the settings file. This allows the developer to encrypt the FAIRCOM.FCS, to specify a validation key for the FAIRCOM.FCS file, and to specify a default password for the ADMIN User ID.
FairCom Servers detect when a disk volume is getting full. The DISK_FULL_LIMIT keyword activates a disk space threshold mechanism to check the space available on a drive when a file is about to be expanded, preventing system-level ‘Disk Full’ errors. This allows the application to evaluate and respond to the situation before it becomes critical.
The FairCom Server optionally maintains the System Event Log in the form of a c‑tree data file with a record for each recordable system event. Making the System Event Log as FairCom DB files allows client applications to read the file and make reports. Unlike the CTSTATUS.FCS file, the system log files can be encrypted so entries cannot be added, deleted, or modified with a simple text editor. Vendors can add application specific entries to the log.
Whether or not you are migrating to the FairCom Server, see Migrating Your Application Between Operational Models for information critical to the different operational models, including the client/Server model.

Tamper-Resistant Server Configuration File

FairCom DB supports an optional tamper-resistant version of the server configuration file (ctsrvr.cfg) referred to as the "settings file," ctsrvr.set. Because entries in the settings file cannot be overridden, users cannot alter important aspects of the server configuration. A special activation key from FairCom can make the settings file mandatory, forcing FairCom DB to operate with only your specific settings. The settings file is encrypted to prevent unauthorized modifications.

The following keywords are not allowed in ctsrvr.set:

LOCAL_DIRECTORY, MULTILINE_STATUS_LOG_MESSAGE, FIPS_ENCRYPTION

Contact your FairCom office for additional information about this security feature as it requires a custom build of FairCom DB.

Encryption of FAIRCOM.FCS

FAIRCOM.FCS contains the user, group and hashed passwords. With the release of V10, the FairCom DB user information file FAIRCOM.FCS is encrypted by default as additional security for the user information stored in this file. Passwords are not directly stored in this file, only secure hashes.

FairCom DB supports the configuration keyword ADMIN_ENCRYPT YES as a configuration option to enable this additional security for prior lines. Adding ADMIN_ENCRYPT YES to the configuration file enables encryption of a new FAIRCOM.FCS file if it does not already exist. While only secured hashes of passwords are stored in this file, it is recommended to enable this additional encryption if possible.

If FAIRCOM.FCS already exists, the only way to encrypt it is to delete the existing file and create a new unencrypted file. To do this:

Ensure you have all User, Group, and File entries written down so they can be reentered.
Shut down FairCom DB.
Delete the existing FAIRCOM.FCS file.
Add ADMIN_ENCRYPT YES to the configuration file.
Start FairCom DB.
Re-enter your User, Group, and File data.

Disk Full Detection

The Disk Full feature offers three levels of control over disk full checks:

The DISK_FULL_LIMIT keyword provides checking on all files.
The volume-specific limit overrides the system-wide limit.
The file-specific check overrides the system-wide and volume-specific checks.

If extending the size of the file would leave less than the specified threshold, then the write operation causing the file extension fails, returning SAVL_ERR (583).

Note the following issues related to this feature:

FairCom Servers that do not support this feature ignore the DISK_FULL_LIMIT keyword. This feature is currently limited to Windows and Unix platforms. (However, this changes over time, so check with your nearest FairCom office for current supported platforms.)
In non-server models, test the disk full condition by setting the global ULONG ct_disklmt to the threshold value.
In the bound server model, set the disk full threshold in the disklmt member of the configuration structure.
If a file is mirrored, the check is performed only for the primary file.

File-by-file Disk Full Checks

To disable disk full checks for a particular file:

Set the ctNOSPACHK bit in the x8mode member of the XCREblk structure.
Create the file using the 8-byte extended creates (e.g., CreateDataFileXtd8()).

To enable file-by-file checking with a file-specific disk full threshold:

Set the dskful member of the XCREblk structure to the desired limit in bytes.
Create the file using the 8-byte extended creates (e.g., CreateDataFileXtd8()).

Volume Disk Full Checks

The FairCom Server keyword DISK_FULL_VOLUME takes as its argument a concatenated volume name and limit. A path separator must occur between the volume name and the threshold limit, which may be zero.

In Unix this is in the form /name/<limit>. The following example places a disk full threshold of one million bytes on the volume /home:

DISK_FULL_VOLUME /home/1000000

From Windows, the form of the argument is <DRIVE>:\<limit>. The following example places a 1MB threshold on drive E:

DISK_FULL_VOLUME e:\1048576

DISK_FULL_ACTION (SUBSYSTEM)

FairCom Server supports a configuration option to call an external script when a specified DISK_FULL_LIMIT or DISK_FULL_VOLUME limit is reached. This allows server administrator to configure an external process to run when a disk-full limit is reached, for example, the process could write information to a log or notify the administrator.

To configure an external process to run when the space on the specified volume is below the specified limit, set the DISK_FULL_ACTION configuration option in ctsrvr.cfg with the run action.

The supported syntax is:

SUBSYSTEM EVENT DISK_FULL_ACTION {

volume VOLUME

limit LIMIT

run EXE [OPTIONS]

freq FREQUENCY

maxruntime MAXRUNTIME

}

The volume, limit, and run parameters are required. The freq and maxruntime are optional.

VOLUME is the name of the disk volume to check: a drive name (for example, C:\) on Windows systems, or a directory name (for example, /users/faircom/data) on Unix/Linux systems.
LIMIT is the threshold of available space that triggers the running of the script. The KB, MB, GB, TB, and PB suffixes can be used for this value (for example: "limit 100 GB" means 100 gigabytes).
EXE is the name of an executable, batch file, or Unix shell script to run, and OPTIONS are optional command-line options to pass to the external program.
FREQUENCY indicates the frequency at which this condition is checked. The default is to check the condition every 60 seconds. Minimum frequency is 1; maximum frequency is 32,000,000.
MAXRUNTIME indicates the maximum amount of time, in seconds, that the external script is permitted to run. If the external script has not terminated after the specified amount of time, the server terminates the script. The default is 60 seconds. A setting of 0 means the script will never be forced to terminate.

Note: When setting a MAXRUNTIME value, consider that no other DISK_FULL_ACTION checks will take place until the script completes, regardless of the FREQ setting. This is of particular importance if more than one DISK_FULL_ACTION is created.

In V12 and later, disk full monitoring keywords have been added to the default ctsrvr.cfg. These keywords are present in this config file, but they have been commented out for so they are there if you need them, but they are NOT enabled.

; SUBSYSTEM EVENT DISK_FULL_ACTION {

volume VOLUME

limit LIMIT

run EXE [OPTIONS]

freq FREQUENCY

maxruntime MAXRUNTIME

}

Example DISK_FULL_ACTION with external process action

The following configuration option causes the batch file diskfull.bat to be run when writing to a c-tree data or index file detects that the space on C:\ has dropped below 1 GB. This disk-full condition is checked once every 60 seconds.

SUBSYSTEM EVENT DISK_FULL_ACTION {

volume C:\

limit 1 GB

run diskfull.bat

freq 60

maxruntime 120

}

Reading parameters sent from the server to the external process

FairCom Server passes a set of ASCII key/value pairs to the external process using standard input, where the input format is key=value, one per line. The keys that FairCom Server passes to the external process are:

volume - The VOLUME setting from the DISK_FULL_ACTION configuration option.
limit - The disk-full limit in bytes that was specified for that volume.
available - The amount of disk space in bytes that is currently available on that volume.

Writing parameters from the external process to the server

The external process can optionally pass ASCII key/value pairs back to the server by writing them to standard output in the format key=value, one entry per line. FairCom Server recognizes the following keys:

action=continue - (indicating to continue FairCom Server operation; this is the default)

action=shutdown - (indicating to shut down FairCom Server)
comments=MESSAGE - A user-defined message that will be written to CTSTATUS.FCS.

Note: FairCom Server reads only the first 1024 bytes of the output data returned by the script. All data after the first 1024 bytes of output data is ignored.

If the external script's output is not in the supported format, a message is written to CTSTATUS.FCS. Example:

DISK_FULL_ACTION: Command diskfull.bat output does not match format var=value: line 3 column 8

Unlimited MAXRUNTIME script

An unlimited script runtime can be set with MAXRUNTIME 0. When configured as unlimited and a script is run due to the LIMIT and RUN options, the script will never be forced to terminate.

Multi-CPU Performance

A cache-line is the smallest amount of memory a processor will retrieve and store in its highest speed cache. Typical cache-line sizes are 16, 32 or 64 bytes. On multiple CPU systems, when a memory word is updated, any other processor with a cache-line containing the updated word must eliminate or refresh the cache-line. When the other processors are not actually using the particular word that was updated, this is called false-sharing.

Many of the Server’s high use objects (e.g., semaphores, mutexes, transaction related arrays) are structured and/or allocated in a manner sensitive to the false sharing problem. This minimizes performance loss from false-sharing. Semaphores and mutexes are allocated to ensure each semaphore exists in its own cache-line and an update in one semaphore’s state by a CPU will not cause other CPUs to synchronize cache.

The cache-line size defaults to 16 and may be overridden with a configuration entry of the form: CACHE_LINE <size>. Set this value to the appropriate value for your CPU, as described above, to maximize performance.

Server System Event Log

The FairCom Server optionally maintains the System Event Log in the form of a c‑tree data file with a record for each recordable system event. Making the System Event Log as FairCom DB files allows client applications to read the file and make reports. Unlike the CTSTATUS.FCS file, the system log files can be encrypted so entries cannot be added, deleted, or modified with a simple text editor. Vendors can add application specific entries to the log.

The System Event Log contents are controlled by SYSLOG configuration keywords in the Server configuration. They are entered as pairs in the form: SYSLOG <keyword>. As many of these pairs as desired may be used. The current SYSLOG keywords are:

SYSLOG Keyword	Description
ADMIN_API	Only allow users in the ADMIN group to use SystemLog to create vendor-defined entries in the log.
CTSTATUS	Log each entry to CTSTATUS.FCS in the System Event Log, except for those entries which occur before or after the system logging monitor is in operation.
DISABLE_API	Do not allow SystemLog calls for user defined entries.
DYNAMIC_DUMP	Log the beginning and end of each dynamic dump and a status result for each file dumped.
ENCRYPT	Encrypt the SYSLOG files.
USER_INFO	Log all logons, logoffs, and changes to user logon profiles.
NONE	Used in a settings file to eliminate additional SYSLOG entries in a configuration file.
LOGFAIL_PURGE	Causes an automatic purge of the oldest entries in the log if the system cannot add a record to SYSLOGDT.FCS. All the entries occurring on the oldest day are deleted unless there are only entries for the current day in which case no entries are purged. After a successful purge, an attempt is made to add the new entry that triggered the automatic purge. If the add succeeds, the system log operation continues in its usual fashion.
LOGFAIL_CTSTATUS	If there is no automatic purge, or it fails, and the configuration includes LOGFAIL_CTSTATUS, the log entries are rerouted to CTSTATUS.FCS. This disables SYSLOG CTSTATUS.
LOGFAIL_TERMINATE	If there is no automatic purge or it fails, and if there is no rerouting to CTSTATUS.FCS, either the system log will stop operation, or if LOGFAIL_TERMINATE is in the configuration file, the FairCom Server will shutdown. USE LOGFAIL_TERMINATE WITH CAUTION!
SQL_STATEMENTS	Logs detailed SQL statement information. This information includes connection information, improved timing, and logging the statement before it is actually executed for a detailed audit trail of all SQL operations.

Retrieving System Event Log Entries

The System Event Log consists of two files: SYSLOGDT.FCS and SYSLOGIX.FCS. Retrieving event log entries is easily done using the FairCom DB search functions such as FirstInVSet() or GetGTEVRecord(). SYSLOGIX.FCS contains two indexes over the system log data based on the following fields in the SYSLOGrec structure:

evclass/event/seqnm
date/time/seqnm

Search the first index for entries with a given event class code or evclass/event combination. Use the second index to read the entries in time order. Open the log files, SYSLOGDT.FCS and SYSLOGIX.FCS, from a client application with the call below, which returns a negative on error or the filno of the log file on success:

filno = OpenFileWithResource(-1,"SYSLOGDT.FCS",ctSHARED);

To determine how many entries are in the system log, use a call of the form below, where filno is returned from the OpenFileWithResource() as noted above. Use filno + 1 to reference the event class based index. Use filno + 2 to reference the time based index. For example:

NbrOfKeyEntries(filno + 1);

SYSLOGDT.FCS is a FairCom DB variable-length data file with a record for each auitable system event. Each entry in SYSLOGDT.FCS is a variable-length record using the SYSLOGrec structure defined in ctport.h and shown below:

typedef struct ctslog {

LONG evclass; /* overall type of entry */

LONG event; /* the particular event code */

LONG date; /* date measured in days: r-tree compatible */

LONG time; /* seconds past midnight */

LONG rsvrd; /* for future use */

LONG seqnm; /* sequence number */

LONG error; /* uerr_cod at time of entry */

TEXT userid[SYSLOGidz]; /* logon user ID */

TEXT nodnam[SYSLOGidz]; /* logon node name */

UCOUNT vlen; /* length of variable region */

TEXT vfld[2]; /* beginning of variable region */

} SYSLOGrec, ctMEM * pSYSLOGrec;

Overlay this structure on a buffer capable of holding an entire record. If the buffer is SYSLOGmax bytes, it will hold any entry in the system log. The maximum length for the variable-length portion of the log entry is given by the constant SYSLOGvar, which defaults to 8100. The fixed length user ID and node name fields are SYSLOGidz bytes, which defaults to 32. These defaults are set in ctport.h.

Example

SYSLOG SQL_STATEMENTS events can be logged for auditing and viewed with the ctalog utility. The output is similar to the following.

Class = 16 (SQL)

Event = 1 (SQL statement)

Date = 09/24/2020

Time = 17:40:11

Sequence number = 37

Error code = -20005

User ID = 'admin'

Node name = 'isql'

Variable-length information:

---------------------------------------------------

{"timestamp":"Tue Sep 24 17:40:27 2020","ipaddr":"127.0.0.1","db":"CTREESQL","user":"admin","thread":29,"statement":"select * from missingtable"}

---------------------------------------------------

Adding System Event Log Entries

To add your own entries to the log, use the SystemLog() function defined in the Function Description pages that follow. Only the system log thread of the FairCom Server can write to the system log files. A client opening the files is given read-only permission. Attempts to write directly to the log return error SWRT_ERR (458).

System Event Log evclass and event Codes

The evclass codes defined in ctport.h are listed below. It is anticipated that additional evclass codes will be defined, with appropriate event codes, as additional types of FairCom Server operations are added to the system log facility. Developers can define evclass codes with values of 0x402 and above without interfering with future FairCom-defined codes. The developer of the calling application assigns the developer-defined event codes.

Symbolic Constant	Value	Explanation
ctSYSLOGuser	0x001	User logon information
ctSYSLOGddmp	0x002	Dynamic dump information
ctSYSLOGstat	0x003	CTSTATUS.FCS entries
ctSYSLOGdelfil	0x005	File delete information
ctSYSLOGsql	0x008	SQL info
ctSYSLOGpurge	0x400	Special purge request
ctSYSLOGapi	0x401	Beginning of vendor defined event classes

The user logon info event codes are below. On the ADDUSER, CHGUSER, and DELUSER entries, the variable buffer portion of the entry contains the user ID and a brief description of the type of activity. The actual details of the user profile information are not in the log.

Symbolic Constant	Value	Explanation
ctSYSLOGuserLOGON	0x001	Successful logon
ctSYSLOGuserLOGOFF	0x002	Logoff
ctSYSLOGuserLOGFAIL	0x003	Failed logon
ctSYSLOGuserADDUSER	0x004	Add new user
ctSYSLOGuserCHGUSER	0x005	Modify user profile
ctSYSLOGuserDELUSER	0x006	Delete user
ctSYSLOGuserSQLLOGON	0x007	Successful SQL logon
ctSYSLOGuserSQLLOGOFF	0x008	SQL logoff
ctSYSLOGuserSQLLOGFAIL	0x009	failed SQL logon

The dynamic dump event codes are below. The TRAN, CLEAN, and DIRTY log entries contain the name of the file dumped in the vfld portion of the log record.

Symbolic Constant	Value	Explanation
ctSYSLOGddmpBEGIN	0x001	Begin dynamic dump
ctSYSLOGddmpTRAN	0x002	Transaction controlled file
ctSYSLOGddmpCLEAN	0x003	Clean non-transaction file
ctSYSLOGddmpDIRTY	0x004	Dirty non-transaction file
ctSYSLOGddmpEND	0x010	End dynamic dump

There are no event codes for CTSTATUS.FCS entries. They are all event zero. The file delete event codes are:

Symbolic Constant	Value (LONG)	Explanation and variable contents
ctSYSLOGdelfRSTR	0x0001	Restore of deleted file. (12-byte File ID, Original Name, Copy Name)
ctSYSLOGdelfTRAN	0x0002	Transaction-dependent delete. (12-byte File ID, Original Name, Copy Name)
ctSYSLOGdelfNOTRAN	0x0003	Non-transaction-dependent delete. (12-byte File ID, Original Name)

Restore point event codes

Symbolic Constant	Value (LONG)	Explanation and variable contents
ctSYSLOGrstpntCREATE	0x001	Create restore point
ctSYSLOGrstpntRECOVERY	0x002	Recovery results
ctSYSLOGrstpntTRANBAK	0x003	TRANBAK results
ctSYSLOGrstpntNOKEEP	0x004	Create RP but no keep

SQL statement log event codes

Symbolic Constant	Value (LONG)	Explanation and variable contents
ctSYSLOGsqlSTMT	0x0001	SQL Statement

Create a List of Arbitrary Names in Server Configuration File

FairCom DB allows a list of arbitrary names to be kept in the server's configuration file (ctsrvr.cfg). This name list is a user-defined, general-purpose name list that can be used by client applications.

A typical use for this feature is to keep a list of data or index files that require special processing by the client application. For example, an application may introduce system queue and notification processing only for data files that are named in an APP_NAME_LIST. A list may also contain user definitions of relationships between different data files.

Application Name List Usage

The configuration keyword APP_NAME_LIST permits the server configuration file to create a list of arbitrary names that can be retrieved by clients. The names are any text string not containing spaces. If spaces are desired, the text string can be placed in quotes, but the quotes will be part of the text string on retrieval.

As many APP_NAME_LIST entries as required may be created in the configuration file. For example: a list of file names may be created and then the names of the files can be retrieved by the client application. Further, by convention, one could use a delimiter to add parameters to the end of the string. Assuming, for the sake of example, that a pound sign (#) is used as a delimiter, then some entries could be added as follows:

APP_NAME_LIST first.dat#parm11#parm12#parm13

APP_NAME_LIST second.dat#parm21#parm22#parm23#parm24

It is up to the client application to determine how to parse the APP_NAME_LIST string that is returned in its entirety by calls to the c-tree GetSymbolicNames() API function.

Application Name List Restrictions

The text string used in APP_NAME_LIST entries have the following restrictions:

The text string cannot contain a vertical bar (|) character; and
The string cannot exceed the maximum file name length (255 characters, or 512 characters when multi-byte characters are enabled).

While any delimiter can be used in the APP_NAME_LIST text string, except for the vertical bar character (|), FairCom recommends the pound character (#). Also, if the server is likely to be used by applications from multiple vendors, FairCom further recommends that the first parameter of an APP_NAME_LIST text string be considered a sub-list name so that different applications can select only the items that are relevant to them. For example:

APP_NAME_LIST first.dat#LIST_1#parm12#parm13

APP_NAME_LIST second.dat#LIST_1#parm22#parm23#parm24

APP_NAME_LIST first.dat#LIST_2#parm32

Use the c-tree function GetSymbolicNames() to retrieve the APP_NAME_LIST strings. The first APP_NAME_LIST text string is retrieved with the following call:

GetSymbolicNames(-1, buffer, buflen, FIRST_ITEM);

If GetSymbolicNames() is successful, it returns NO_ERROR and the first APP_NAME_LIST in the server’s configuration file is placed in buffer. If buflen is too small, GetSymbolicNames() returns error VBSZ_ERR (153). If no APP_NAME_LIST entry is found, GetSymbolicNames() returns error INOT_ERR (101).

The second and any subsequent APP_NAME_LIST text strings are returned by consecutive calls of the form:

GetSymbolicNames(-1, buffer, buflen, NEXT_ITEM);

When APP_NAME_LIST strings in the server's configuration file are exhausted, GetSymbolicNames() returns INOT_ERR (101). There is no way, without repeated calls to GetSymbolicNames(), to know in advance how many APP_NAME_LIST entries exist. Calling GetSymbolicNames() with the FIRST_ITEM mode always attempts to return the first item in the list, resetting the client position to the beginning of the list.

Application Name List Example

The following sample function displays all APP_NAME_LIST strings that exist in a server's configuration file:

void DisplayAppNameList(void)

{

TEXT buffer[MAX_NAME + 1];

NINT count = 0;

if (GetSymbolicNames(-1, buffer, sizeof(buffer), FIRST_ITEM))

printf("No APP_NAME_LIST present.\n");

else do {

printf("%d: %s\n", (count+1), buffer);

count++;

} while (!GetSymbolicNames(-1, buffer, sizeof(buffer), NEXT_ITEM));

}

Server Internals

This section provides answers to the advanced questions that are sometimes asked about FairCom DB internal details.

FairCom Server Threads

When listing active clients in ctadmn, the first user is usually number 5 or 6. This is because FairCom DB starts various threads upon start up. The following threads are created by FairCom DB:

Description of Server Threads

Main (ctsrvr): The main c-tree Server thread. It initializes the database engine and creates the administrative threads.
Delete Node (ctdnode): Makes empty index nodes available for reuse. When an index node becomes empty, the thread that emptied the node adds an entry to the delete node queue. The delete node thread reads entries from the delete node queue, prunes the empty nodes from the index tree and adds them to a list of nodes that are available for reuse.
Checkpoint (ctqchkp): Periodically records the current transaction state of the c-tree Server in the c-tree Server’s transaction log files. The checkpoint thread writes a checkpoint entry to the transaction logs each time a specified amount of data has been written to the transaction log files. The most recent checkpoint entry in the transaction logs provides a starting point for the c-tree Server’s automatic recovery procedure if the c-tree Server process terminates abnormally. If you are not using transaction control, the checkpoint thread is idle. The frequency at which the checkpoint thread writes checkpoints can be tuned for optimal performance.
Communication Listener: Listens for connection requests from c-tree client processes. There is one communication listener thread for each communication protocol your c-tree Server is configured to support. For example, the c-treeSQL Server for Windows has four communication listener threads if both the TCP/IP and shared memory communication protocols are enabled for both ISAM and SQL connections.
SYSLOG (ctsyslog): Logs messages to the System Log Data file. The logging of these messages are triggered by the SYSLOG c-tree Server configuration options and some uses of the SNAPSHOT API, such as the collecting of c-tree function timings.
Deleted Space Reclamation (ctrspac): Makes deleted space in recovered variable-length c-tree data files and deleted c-tree superfile members available for reuse. This thread also coalesces adjacent blocks of deleted space in variable-length files into larger regions. The thread reads and processes entries stored in a permanent queue file named D0000001.FCS. Improves performance and disk space usage.
Dynamic Dump (idyndump): The c-tree Server Dynamic Dump backup thread. A dynamic dump thread is created for each backup that is scheduled, so more than one dynamic dump thread can exist, but only one dynamic dump thread at a time performs a backup.
Idle Flush (Idle transaction buffer flush / Idle non-transaction buffer flush): Flushes updated cache pages to disk when the database engine is idle. One thread flushes updated pages for transaction-controlled files and one thread flushes updated pages for non-transaction controlled-files.

FairCom DB Trapped Unix Signals

FairCom DB Unix servers trap the following signals:

SIGINT
SIGQUIT
SIGTERM
SIGABRT
SIGPWR
SIGPIPE
SIGHUP
SIGALRM

Client/Server ctntio Communications Errors (Formerly VDP Errors)

FairCom DB performs rigorous error checking and logging during the course of everyday operation. Because of the depth of error checking that is performed, warnings and error messages are logged in even the most benign situations.

When a communication error such as ARQS_ERR (127, send failure) or ARSP_ERR (128, receive failure) occurs, the FairCom DB logs an entry in the status log file, CTSTATUS.FCS.

Wed Nov 26 14:58:59 2008

- User# 00013 ctntio: read error - O13 bytes=0 pErr=127

|ADMIN|ctreplr 192.168.1.110|: 161

This is not a serious situation unless the client application is also getting errors such as 127, 128, or other general connection failure errors. To ensure the errors are not serious, try to reconcile the ctntio errors in the log with the client events that triggered them. Since these errors do not usually happen frequently and user and node names are provided, it should be easy to determine which event caused this situation.

The context of the ctntio error is that a server thread receives a notification that a message is available, however, when the server performs the read operation, nothing is returned.

This can be caused by (but not limited to):

Clients or processes aborting without calling CloseISAM() or StopUser(), or end users turning their machines off without properly logging out of the application. A client application crash effectively results in this same condition.
A faulty network connection or related network hardware failure such as a network adapter, cabling, switch, or router. Check the the server machine, client machines, and all connections between them.
An overworked network transport layer that is timing out and doing retries.
Other processes on the network consuming the majority of the CPU time and not permitting the communications layer bound to FairCom DB sufficient time to handle incoming messages.

Faulty network connections or problems in related network hardware can be investigated by your network managers. This may require considerable investigation to discover the culprit, which could be just a single network card in one machine.

By raising the priority of the FairCom Server, you can easily experiment with the problem in CPU consumption. To raise the priority of FairCom DB, go into the Windows task Manager and find ctsrvr. Right click on it and raise the priority to REAL TIME. If this eliminates or significantly reduces the errors you need to look at either reducing the processing load on the machine or increasing its processing power by moving to a faster machine, possibly with multiple processors.

To avoid these errors, ensure FairCom DB's host machine is not burdened beyond its capacity. A more powerful machine or limiting the number and types of applications on the machine can improve performance and limit communication level errors. Also, ensure no specific application is over-using resources on the host machine. If appropriate in the Server's operating environment, increasing the priority of the process can eliminate or reduce ctntio errors. This should be done cautiously as it will affect other applications running on the same machine.

The error messages in the server status log can be turned off, however, unless they are an inconvenience, this is not recommended. The messages serve as a general health check on the state of your network and application logic and may be an early warning of more serious network and/or system problems. Should you wish to disable the messages, The CTSTATUS_MASK VDP_ERROR keyword can be added to the ctsrvr.cfg server configuration file. Restart FairCom DB to enable the configuration changes.

Note: Legacy c-tree I/O errors were previously noted as VDP (Virtual Device Protocol) errors in the past. These have largely been replaced with ctntio errors.

If java.nav (formerly JTDB) connects to the SQL_PORT instead of the SERVER_PORT (or SERVER_NAME), that java.nav client will get error 128.

Diagnostic Tools

cttrap - Communications Trap Playback utility

Client application to "play back" a TRAP_COMM log file made by using the DIAGNOSTICS TRAP_COMM server keyword to "record" the communication traffic coming in to a server.

Operational Model:

Client

Usage

cttrap [-display | -displayonly] [-p <password_file>] [-t <trapcomm_log_file_name>] [-s <servername>] [-v] [-x]

where:

-display - Displays log file contents and replays the function calls
-displayonly - Displays log file contents without replaying the function calls
-p - Specifies text file containing user name and password list
-v - Enables verbose output
-x - Disables inter-thread ordering for fast replay

Description

When activated, the DIAGNOSTICS TRAP_COMM keyword instructs the FairCom Server to log incoming communications packets to TRAPCOMM.FCS prior to execution. This log can be played back using the cttrap utility and a debug Server to observe the results of the client requests, allowing the developer to exactly duplicate and repeat client activities.

cttrap is a multi-threaded client application that "plays back" a TRAPCOMM.FCS log file. Whenever a multi-threaded client library is created, cttrap will also be generated.

DIAGNOSTICS TRAP_COMM Keyword

When the DIAGNOSTICS TRAP_COMM keyword is active, the trap file, TRAPCOMM.FCS, is created in the server directory by default. To prepend a path onto the trap file name (to route it to a separate disk or directory), add an entry of the form DIAGNOSTIC_STR <trap file path>. For example, if the trap file is located in /bigdisk/TRAPCOMM.FCS, the configuration file needs to contain the following:

DIAGNOSTIC_STR /bigdisk/

Example:

C:\> cttrap -t TRAPCOMM.FCS -displayonly

c-tree(tm) Version 12.0.0.103(Build-190918)

Client Communication Replay Utility

seqnum taskid function filno long size outlen mode filename

3 24 INTISAM 0 0 0 0 0

5 24 ctSETCOMPRESS 3 1 1 0 55

7 24 SYSCFG 0 0 0 1024 0

9 24 TRANBEG 50 0 0 0 0

89 24 OPNRFIL 5 -1 0 2064 0 qadfridx.dat

...

255 25 SETNODE 0 0 0 0 0

257 25 SYSCFG 0 0 0 1024 0

259 25 STPUSR 0 0 0 0 0

cttrap is processing log file \TRAPCOMM.FCS for FAIRCOMS [upto 256 clients]

Starting Time Stamp: 1568924640 Server Flavor: LOW_HIGH

Using a Password File

To allow cttrap to connect to a password-protected server when reading from a TRAPCOMM.FCS file, use the -p <password_file> option to specify the name of a text file containing a list of user names and passwords, one per line. For example, the file pass.txt might contain the entries:

ADMIN ADMIN

myusername mypswd

Then cttrap can be run as follows:

cttrap TRAPCOMM.FCS -p pass.txt FAIRCOMS

The passwords must match the passwords on the server where the playback occurs, not necessarily where the playback was recorded.

Up to 16 user name/password entries can be specified in the cttrap password file. If you wish to support more entries, change #define MAXUPW in cttrap.c and recompile the utility.

History

Before V11.9, prior usage was as follows:

cttrap <Trap Comm Log File Name> [-p <password_file>][-x] [<Server>]

Where:

<TrapCommLogFileName> - the name of the file to be played back.
<ServerName> - the optional name of the server.
-p <password_file> - use a text file containing a list of user names and passwords, one per line (see Using a Password File, below).
-x - skip serialization. cttrap serializes all the recorded operations and enforces the original call ordering when allowing multiple client threads to replay the original operation. The ‑x option allows for faster replay, although it increases the likelihood of causing errors or differences in the final state of the data that did not exist in the original.

DIAGNOSTICS LOWL_FILE_IO Keyword

The DIAGNOSTICS LOWL_FILE_IO Server keyword logs low-level system errors into the Server status file, CTSTATUS.FCS. Although client applications have access to system errors through sysiocod, it is useful to see these errors logged on the Server side. For example, an end-user has problems opening a file. The end-user copied the data file from a CD-ROM onto the hard disk leaving the file marked “read-only”. When the user attempted to open these files, open errors were generated within the application. Adding the DIAGNOSTICS LOWL_FILE_IO keyword directed the Server to log the “not authorized” operating system error to CTSTATUS.FCS. This pointed the user to the “read-only” issue.

Diagnostic Dump of Server File-lock Table

The FairCom Server supports a diagnostic dump of its internal lock table through the LockDump() function, short name ctLOKDMP(). This option allows developers to debug lock related issues. It should be used with discretion since dumping the lock table may impact performance. To limit this impact, only members of the ADMIN group may call LockDump(), unless the FairCom Server was started with the DIAGNOSTICS LOCK_DUMP keyword, in which case any user may call the function.

LockDump() supports a number of modes allowing dumps based on files or users. The sample ctlokdmp demonstrates the options dumping all locks either by file or by user. The sample output below shows the “All Users” dump. See LockDump for a detailed breakdown of the options.

Example LockDump Output

=================================================

All Users Lock Dump at Tue Apr 18 08:56:49 2000

----------------

User #10>

vcusti

00000abax write/1

00000afcx write/1

00000a99x write/1

00000adbx write/1

00000b1dx write/1

User lock count: 5

----------------

=================================================

DIAGNOSTICS USER_FUNCTIONS keyword added

By placing the keyword DIAGNOSTICS USER_FUNCTIONS in the ctsrvr.cfg file, a table of “called” functions will be written to the CTSTATUS.FCS file for each user who “ever” logged on to the server. In order to activate this feature, a special diagnostics server must be generated by activating the following #define in ctopt2.h:

#define ctDIAG_FNC_MON 10

The value (defaults to 10) represents the number of “last-functions” or “depth-of-history” you wish keep track of. Raising this value will cause additional global memory to be needed, for the history for each user (server defaults to supporting up to 1024 users) is tracked using global memory buffers.

A function has been added to ctdosv.c named ‘ctSHowDiagFncMon’. This is the function that will be called when the server shuts down (called in ctcatend()) or has an exception (Win32 only: called in Win32 exception handler). Each users buffer is circular, which means the output must be inspected for the sequence number in order to see the exact order of the calls. Also, the same buffer is used for all user of a particular user number. In other words, it is possible for one user to be, say USER #45, and then log off, then another user logon and use slot number #45. Therefore, the function output presented in this table for a particular user number might be two separate users. Look for the log off sequence in the output.

In addition to tracking the FairCom DB calls made by each user, this feature will also output a table showing the total number of times each FairCom DB function has been called. A user might use this feature to inspect that only the functions that were expected to be called, were, in fact, the ones that were called.

The following tables will be written to the CTSTATUS.FCS file:

=================================

== START USER FUNCTION RECAP ==

---------

Client#07 func#077-GETMAP seq# 2936 cnt# 265 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#050-GETALTSEQ seq# 2938 cnt# 795 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#076-GETSEG seq# 2940 cnt# 795 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#050-GETALTSEQ seq# 2942 cnt# 795 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#076-GETSEG seq# 2944 cnt# 795 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#050-GETALTSEQ seq# 2946 cnt# 795 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#076-GETSEG seq# 2948 cnt# 795 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#017-CLRFIL seq# 2950 cnt# 265 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#010-STPUSR seq# 2952 cnt# 271 on:Thu Jun 22

07:23:38 2000 vcusti

Client#07 func#196-OPNRFIL seq# 2933 cnt# 273 on:Thu Jun 22

07:23:38 2000 vcusti

---------

Client#10 func#196-OPNRFIL seq# 2372 cnt# 273 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#077-GETMAP seq# 2373 cnt# 265 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#050-GETALTSEQ seq# 2374 cnt# 795 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#076-GETSEG seq# 2375 cnt# 795 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#050-GETALTSEQ seq# 2376 cnt# 795 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#076-GETSEG seq# 2377 cnt# 795 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#050-GETALTSEQ seq# 2378 cnt# 795 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#076-GETSEG seq# 2379 cnt# 795 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#017-CLRFIL seq# 2380 cnt# 265 on:Thu Jun 22

07:23:34 2000 vcusti

Client#10 func#010-STPUSR seq# 2381 cnt# 271 on:Thu Jun 22

07:23:34 2000 vcusti

---------

== END USER FUNCTION RECAP ==

=================================

== START GLOBAL FUNCTION RECAP ==

Function#001-PUTFIL was called (0) times.

Function#002-UPDCURI was called (0) times.

Function#003-DELFIL was called (0) times.

Function#004-LKISAM was called (0) times.

Function#005-DELREC was called (0) times.

Function#006-ALCSET was called (0) times.

Function#007-CHGSET was called (0) times.

Function#008-DELVREC was called (0) times.

Function#009-CLISAM was called (0) times.

Function#010-STPUSR was called (271) times.

...

Function#017-CLRFIL was called (265) times.

...

Function#050-GETALTSEQ was called (795) times.

Function#051-SETDEFBLK was called (2) times.

Function#052-MIDSET was called (0) times.

...

Function#076-GETSEG was called (795) times.

Function#077-GETMAP was called (265) times.

Function#078-SECURITY was called (0) times.

...

Function#111-INTISAM was called (272) times.

...

Function#196-OPNRFIL was called (273) times.

Function#197-TMPNAME was called (0) times.

...

Function#229-SESSVAL was called (0) times.

Function#230-SESSINC was called (0) times.

== END GLOBAL FUNCTION RECAP ==

=================================

Dynamic CRC Check support for SPX and TCP/IP

To help detect communication problems related to network hardware failures, support for a low-level CRC check has been added to the SPX and TCP/IP protocols. Activate this feature for the FairCom Server with the keyword:

DIAGNOSTICS LOWL_CRC_ON

When activated, the Server performs a CRC check at the low-level entry point on/off the network.

This option is designed to help detect hardware problems. This CRC option ensures that the exact data sent by the client to the server (or visa versa) has not been corrupted by the network. FairCom received a number of reports of error 8502, which were a result of a bad network card. This CRC ability is designed to help diagnose this type of problem quickly.

Note: By activating this keyword, the TCP/IP and SPX protocols become incompatible with standard clients. In order for a client application to take advantage of this feature, the application developer must compile the FairCom DB libraries with the following define: #define ctFeatLOWL_CRC