rerunIntegrationTableTransformSteps

Rerun transform steps after updating them

The "rerunIntegrationTableTransformSteps" action causes the server to rerun an integration table's transform steps. You can limit its scope to specific records or all existing records. The "scope" property is required.

This action is valuable when transform steps are added or altered after records have been inserted. Benefits include:

  • Experimentation: Observe the behavior of various transform steps on your dataset.
  • Bug Resolution: Easily correct a bug in a transform step and re-run the steps to reprocess the original data with the corrected transform, negating the need to undo any prior damage.
  • Data Correction: Fix problematic data in the source_payload field.
  • Resilience: If an interruption occurs during the re-running of transform steps (e.g., power failure), you can simply restart the process.

To ensure this action runs successfully, you should preserve the original value of the source_payload field.

  • In an Integration Table, the source_payload field contains the original event data from connectors, MQTT clients, SQL inserts, the JSON DB "insertRecords" action, etc.
  • Before inserting the record, the server runs transform steps that read the source_payload field and write to other fields in the table and perhaps other tables.
  • Typically, a transform step does not overwrite the original data in the source_payload field; otherwise, it would destroy the original data, change the historical record, break the audit trail, and potentially prevent the "rerunIntegrationTableTransformSteps" and "testIntegrationTableTransformSteps" actions from running successfully.
  • Preserving the original data in the source_payload field allows you to rerun transform steps as many times as you want.
  • If data in the source_payload field has an incorrect value, you can create transform steps to fix it. The first transform step can back up the value of the source_payload field by copying it into a property in the metadata field. The next transform step can correct the problem and update the original value in the source_payload field. You can test the fix using the "testIntegrationTableTransformSteps" action. You can apply the fix using the "rerunIntegrationTableTransformSteps" action.

Note Updating a large number of existing records may take significant time and may slow the intake of newly inserted records.

This action is not atomic. If it fails part way through, some records will be updated and some will not. In this case, you must rerun this action to finish the process. Thus, it is important that the source_payload field contains good data so the transform steps will run properly.

Use the "rerunTransformScope" property (required) to define the scope of this action:

  • "allRecords" returns immediately after starting a background thread that runs the transform steps on all records in the specified table.
  • "stop" stops the table's background thread that reruns transform steps on all its records. If there is no background thread, the action does nothing and returns an error.
  • "firstRecord" returns the first record's fields before and after being modified by the transform steps. It runs synchronously and does not persist any changes.
  • "lastRecord" returns the last record's fields before and after being modified by the transform steps. It runs synchronously and does not persist any changes.
  • "specificRecords" returns the specified record's fields before and after being modified by the transform steps. It runs synchronously and does not persist any changes.

Limitations on the "allRecords" scope:

  • The server runs one "allRecords" scope at a time. This applies to both the "rerunIntegrationTableTransformSteps" and  "testIntegrationTableTransformSteps" actions.
  • While running transform steps on all records in an integration table, you cannot alter the structure of the table.

 

Request Examples

Minimal example stops rerunning transform steps on existing records

The following example stops a background thread that is currently rerunning transform steps on records in the myTable1 table. It defaults to using the database and owner specified in the current JSON action session (see "createSession" and "alterSession"). It returns an error if a background thread is not currently rerunning transform steps on the records in myTable1.

{
  "action": "rerunIntegrationTableTransformSteps",
  "params": {
    "tableName": "myTable1",
    "rerunTransformScope": "stop"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
 
 

Example running transform steps on the first record

The following minimal example reruns the transform steps on the earliest inserted record in myTable1 that has not been purged by the retention policy. It uses the database and owner specified in the current JSON action session (see "createSession" and "alterSession").

{
  "api": "hub",
  "action": "rerunIntegrationTableTransformSteps",
  "params": {
    "tableName": "myTable1",
    "rerunTransformScope": "firstRecord"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
 
 

Example running transform steps on the last record inserted

The following example reruns the transform steps on the most recently inserted record in myTable1. It uses fully qualified table names that specify the database and owner of the table.

{
  "api": "hub",
  "action": "rerunIntegrationTableTransformSteps",
  "params": {
    "databaseName": "myDb1",
    "ownerName": "myOwner1",
    "tableName": "myTable1",


    "rerunTransformScope": "lastRecord"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
 
 

Example running transform steps on all records

The following example reruns the transform steps assigned to myTable1 on all existing records.

{
  "api": "hub",
  "action": "rerunIntegrationTableTransformSteps",
  "params": {
    "tableName": "myTable1",
    "rerunTransformScope": "allRecords"
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
 
 

Example running transform steps on specific records

The following example reruns the transform steps on the records in myTable1 with the specified identifiers.

{
  "api": "hub",
  "action": "rerunIntegrationTableTransformSteps",
  "params": {
    "tableName": "myTable1",


    "rerunTransformScope": "specificRecords",
    "ids": [1,3]
  },
  "authToken": "replaceWithAuthTokenFromCreateSession"
}
 
 

 

Properties 

Request properties ("params")

Property  Description Default Type Limits (inclusive)

databaseName

The "databaseName" property specifies the database that contains an object, such as a table or code package. If it is set to null or is omitted, it defaults to the default database of the JSON Action session, see "createSession" and the "defaultDatabaseName" property. 

You specify this property when you want to use a different database instead of the default. 

This property is useful because objects, such as tables and code packages, can have the same name in multiple databases. This feature allows you to create multiple environments in the same server and reuse the same JSON actions in each environment. For example, you can create "dev", "test", "stage", and "prod" databases on the same server and use the "defaultDatabaseName" or "databaseName" properties to specify the desired environment.

It is an error to set "databaseName" to the empty string "".

If no default database is specified during "createSession", the server sets the "defaultDatabaseName" to the "defaultDatabaseName" value specified in the services.json file.

 Defaults to the session's "defaultDatabaseName" property string 1 to 64 bytes

ids

The "ids" property specifies 0 or more ids. Each identifier in the array uniquely specifies a table row, indicating which records the action affects. It is required if the "primaryKeys" property is "null" or not specified.

  • The "ids" property is mutually exclusive with the "primaryKeys" property meaning it is required when "primaryKeys" is omitted or an error is returned if both have values.
  • It is typically an array of integers ("ids": [1,3,5]).
  • It can be an array of an array of strings ("ids": ["9555444333222111","9555444333222112", "9555444333222113"]).
    • A string "id" supports numbers larger than 9,007,199,254,740,991.
    • This is the largest number supported by many programming languages and JSON parser implementations that use IEEE double-precision floats to hold numbers.
  • It can be the primary key value of another field in the table making it useful when your table is created by another API, such as SQL, that allows any field in the table to be the primary key.
    • If your table does not have an "id" field but uses a "vin" field as the primary key, you can use vin values to look up records ("ids": [ "4Y1SL65848Z411439", "1HGBH41JXMN109186" ]).
  • If your table uses more than one field as the primary key, you must use the "primaryKeys" property to look up records.

Tip The "getRecordsByIds" action uses a primary key index to look up records. A primary key index must be a unique, non-null index without conditional filtering. For best performance and maximum simplicity, create tables using the JSON DB API because it automatically creates an auto increment "id" field that is indexed as a primary key.

Optional with default of "null".

Required when "primaryKeys" is omitted

 array 0 or more ids

ownerName

The "ownerName" property specifies the account that owns an object, such as a table or code package. See "createSession" and the "defaultOwnerName" property for more details. 

You specify this property when you want to use a different account instead of the default. Your session's account must have the appropriate privileges to access the code package. 

This property is useful because objects, such as tables and code packages, can have the same name in the same database as long as different accounts own each object. This feature allows you to create duplicate objects for different users on the same server and reuse the same JSON actions on those objects. For example, an administrator can copy objects from a production environment to her account so she can troubleshoot an issue using the same JSON actions, JavaScript, and SQL code.

It is an error to set "ownerName" to the empty string "".

If no default owner is specified during "createSession", the server sets the "defaultOwnerName" to the "defaultOwnerName" value specified in the services.json file.

 Optional with default of the session's "defaultOwnerName" property string 1 to 64 bytes

rerunTransformScope

The "rerunTransformScope" property defines the scope of this action. It has one of the following values:

  • "allRecords" returns immediately and asynchronously creates a background thread that runs the transform steps on all records in that table.
    • You can stop this action by calling the "rerunIntegrationTableTransformSteps" action with "rerunTransformScope": "stop".
    • The server regularly updates the "transformStatus" and the "transformPercentComplete" properties, which are returned by the "describeIntegrationTables" action.
    • If an integration table's background thread is already rerunning the transform steps, the action returns an error indicating the "rerunIntegrationTableTransformSteps" action is already running on the integration table. The user can decide if he wants to wait or stop the action.
  • "stop" stops the table's background thread that reruns transform steps on all its records. If the background thread is not running, the action does nothing and returns an error.
  • "firstRecord" immediately and synchronously runs the transform steps on the first record in the table.
    • This option does not interfere with a background thread that is currently rerunning the transform steps.
  • "lastRecord" immediately and synchronously runs the transform steps on the last record in the table.
    • This option does not interfere with a background thread that is currently rerunning the transform steps.
  • "specificRecords" immediately and synchronously runs the transform steps on the specified records in the table.
    • Each record has an "id" field containing an integer number that uniquely identifies it.
    • The action uses the "ids" property to specify the identifiers of one or more records to be transformed.
    • This option does not interfere with a background thread that is currently rerunning the transform steps.
 Required - No default value string enum

"allRecords"

"firstRecord"

"lastRecord"

"specifiedRecords"

"stop"

tableName

The "tableName" property contains the unique, user-defined name of a table.

See table name in System limits for the table naming requirements and limitations.

 

"params": {
  "tableName": "ctreeTable"
}
 Required - No default value string 1 to 64 bytes

 

 

Use Cases

There are a variety of times when you want to run transform steps on existing records.

  • A bug in a transform step may cause records to contain bad data. After you fix the bug, you may want to apply the fix to existing records.
  • If you enhance transform steps, you may want to apply the enhancement to existing records.
  • If you alter table fields, you may want to rerun the transform steps on existing records to take advantage of new field capabilities. For example, you may change an integer field to a double so it can store floating point numbers.
  • If you disable the transform steps, you may later want to enable them and apply the changes to existing records.
  • A device sometimes produces bad data. After you fix that data, you may want to run the transform steps on existing records to reprocess data changes.

 

Update existing records

  • Updating existing records cannot be undone, but it can be stopped.
  • The action runs asynchronously on all records; thus, it returns immediately.
  • It starts with the first record in the table and proceeds to the last.
  • After running this action, you can view or query the initial records in the table to verify it is working as expected. If not, you can stop the action.
  • You can stop the action when the specified table is currently processing existing records by calling this action with "rerunTransformScope": "stop". This stops the action from processing the remaining records, but it cannot undo the changes it has already performed.

 

Tips

  • Before running this action, use the "testIntegrationTableTransformSteps" action to verify the transform steps work. This is important because this action modifies thousands of records a second and these changes cannot be undone.
  • To get the status of the "rerunIntegrationTableTransformSteps" action, run the "describeIntegrationTables" action on the integration table. When this action is actively processing existing records, it returns "transformStatus": "running" and a "transformPercentComplete" property set to an integer between 1 and 99.
  • Avoid running the "alterIntegrationTable" action while this action is running because it may change what subsequent transform steps do.
  • The "logTransformOverwrites" property, when true, causes the action to log attempts by JavaScript to overwrite a field value.

 

Was this article helpful?

Give feedback about this article

Related Articles