Overview

Incremental loading is the process of loading all data from the data source the first time that a table is run, and then only needing to bring in records that have been added or modified since the last time that the data refresh happened. This way only a small number of transactions will need to be brought in during the data load instead of having to reload all data every time. This article will cover how to configure incremental loading for tables.

Details

To configure a table for incremental loading, you will first check out the Cosmos data model. Once the data model has been checked out you will navigate to the Source Tables section in the data model within the Cosmos Portal.

Enabling or Disabling Incremental Loading

Select the table that you would like to enable or disable incremental loading for and expand the Fields section. At the bottom of the Fields section you will see an Incremental toggle. Turning this on will enable incremental loading and turning it off will disable incremental loading.

Specifying the Key for the Table

If any source tables feeding into a table in the Data Model section have incremental loading enabled, the Data Model table will be incrementally loaded for that table as well. 

For incremental loading to be successful in the reporting database, Cosmos needs to know what the primary key is for the Data Model table. A primary key is a field, or combination of fields, which ensures that every record in the table is unique.

This is used when the data is loaded in the following manner:

• New data is brought in from the data source. This will include any records that have been created or modified since the last time that the data refresh was run
• Cosmos will use the primary key fields specified by the user to know what field(s) make every record unique
  • For any records brought in from the data source where a primary key already exists, an UPDATE operation is done to update the existing record with the new values from the data source
  • For any records brought in from the data source where the primary key doesn't already exist, an INSERT operation is done to insert the new record into the Data Model table

To specify the Key Fields for the table, edit the field and check the Key Field box.