Compatibility & Migration Guidelines

• Tables created by ISAM or low-level APIs are not affected and have none of the support documented here.
• Tables created at the CTDB level or above (SQL, JSON, etc.) use the default string encoding (see ctdbSetDefaultStringEncoding) for all columns unless explicitly set. The server keyword DEFAULT_STRING_ENCODING provides a global default value.
• Automatic charset conversion requires both a session charset and a field string encoding. Only affects columns that have a defined Field Encoding.
• ctdbSetFieldStringEncoding() may set the string encoding to NULL, creating a table with an undefined string encoding and disabling all conversions on this field. ctdbSetSessionCharset() may set the session charset to NULL, disabling all conversions on the session. This may be used to revert to prior behavior with an unspecified string encoding.
• Legacy tables lacking explicit encoding metadata do not support automatic conversion. It is recommended to use ctdbSetFieldStringEncoding() and ctdbAlterTable(CTDB_ALTER_NORMAL) to explicitly set field-level string encodings on existing tables. This is a metadata-only operation, so it is independent of table size. However, it does not validate that existing data actually uses the specified encoding.
• The application must ensure that the provided string data uses an encoding that matches ctdbGetSessionCharset(). See the OS documentation for setlocale() if this depends on the local system encoding.

Limitations

• Conditional expressions using charsets that are not 100% US-ASCII-compatible cannot properly support expressions that require any non-encoded symbols. (e.g., SHIFT-JIS overline symbol replaces ASCII tilde (value 0x7F), which is the bitwise NOT operator in the conditional expression grammar ).

Runtime Conversion Semantics

• On Read: Storage → Working. The CTDB client receives text in the session’s working charset from ctdbGetFieldAsString().
• On Write: Working → Storage. Data sent via ctdbSetFieldAs… setters is converted to the field’s storage charset before persistence.
• Validation: Invalid code unit sequences cause a conversion error.  Explicit validation can be controlled using ctdbSetBehavior() and the following options:
  • CTDB_BEHAVE_CHECK_UTF8_ENCODING
  • CTDB_BEHAVE_CTJSON_CHECK_VALUE
  • CTDB_BEHAVE_STRICT_UTF8_CHARSET
  • CTDB_BEHAVE_UNICODE_VALIDATION 

Error Handling

• CTDBRET codes: 
  • 0 (success)
  • CTDBRET_INVCHARSET
  • CTDBRET_CHARSETCONV
  • CTDBRET_INDEXRANGE
  • CTDBRET_NOMEMORY
• Extended diagnostics: 
  • Calls failing with CTDBRET_CHARSETCONV should provide last-error context retrievable from session/table handles (error string, failing bytes).
  • All conversions are to/from UTF16, which generally involve 2 conversion steps. Thus, a conversion from Shift-JIS to Latin-1 might fail with the message “Invalid UTF16->ISO_8859-1:1987 bytes 0x3053”. The problem data is the Shift-JIS character in the source data that maps to the UTF16 value 0x3053, which is not representable in Latin-1.

Usage Examples

Set working charset and default storage encoding

CTHANDLE ses = ctdbAllocSession(...);
CTDBRET st = 0;

st = ctdbSetSessionCharset(ses, "UTF-8");
if (st) goto fail;

st = ctdbSetDefaultStringEncoding(ses, "LATIN1");
if (st) goto fail;

// Create a table: all new string fields default to LATIN1 unless overridden.

fail:
ctdbCloseSession(ses);

Override per-field encoding before create/alter

CTHANDLE tbl = ctdbOpenTable(ses, "customers", CTOPEN_EXCLUSIVE);

// Using index
ctdbSetEncodingByField(tbl, 2 /* city */, "UTF-8");

// Or using field handle
CTHANDLE fld = ctdbGetField(tbl, 3 /* notes field */);
ctdbSetFieldEncoding(fld, "LATIN-1");

// Apply create/alter to persist metadata on per-field encodings.

Read/write with transparent conversion

// Session working charset = UTF-8
// Field storage charset ("name") = LATIN1

char buf[256];
size_t outlen = sizeof(buf);

// Read: storage (LATIN1) -> working (UTF-8)
CTDBRET st = ctdbGetFieldAsString(rec, F_NAME, buf, &outlen);

// Write: working (UTF-8) -> storage (LATIN1)
st = ctdbSetFieldAsString(rec, F_NAME, "André");