Overview
The Staging Area within Data Management serves as a single place to select the tables and fields that should be brought into Cosmos from your data source(s). This article will discuss the concepts around the Staging Area and address the significant role that it plays within the over Data Management lifecycle.
Details
Within the architecture of modern data analytics tools, it is commonly noted as a best practice to first bring the raw data from data sources into a staging area, where it can then be transformed and consolidated with other data to prepare it for use by the business.
The Staging Area within Cosmos acts as an initial place for all the raw data to be stored. When a business user selects a table and field to be imported from a data source, for example, a corresponding table and field will be created in the organization's Cosmos tenant and the next time a pipeline is run, this new data will be brought in from the data source and stored in this staging area.
Downstream activities, such as field transformations and mapping this source data into the reporting tables that the business will use for running their reports, will then extract the data from this Staging Area, move the data along the Cosmos pipelines, and place the cleaned-up version of the staging data into the reporting tables.
The diagram below shows the Staging Area and how it plays into the rest of the Cosmos data architecture.
In the case of some data sources, such as Business Central, the data from the data source is also incrementally loaded into the Staging Area when the pipelines are run. This means that instead of needing to reload millions of transactions each time the data refresh occurs, only the records in the data source that are new or modified will need to be retrieved. This ensures that data refreshes are fast and have a minimal impact on resources allocated to the data sources.
Comments
0 comments
Article is closed for comments.