Generic Import RDM Code Load

Prerequisites

Area Prerequisite

Reltio Tenant

Configure Reltio tenant details, refer to Configure Connection.
Notification Channels At least one notification channel must be configured among Email ID, MS Teams and Slack. For more information, see Adding Alert Notification.
GIF Files Have the GIF file sets ready as per the layout defined in the GIF Data Interface Documents.

Running the Pipeline

Run the pipeline MDM_Load_Refresh_Reltio_Views_Customer_Master which creates/replaces one time Reltio Views used for inbound connector.

Generic Import Code Load

Overview

Generic code layout is defined with single file with canonical code, external codes, dependency and locales. this file is transformed into 4 different tables in Canonical schema and pushed into Reltio RDM tenant as per the configuration. See the process flow diagram below.

Create Task Group for RDM Code Load

To load the RDM Generic Source Codes to Reltio RDM tenant, you need to first create a Task Group in IDP OA platform to execute the loading process. Follow the below steps to create a task group in IDP OA.

  1. Using WinSCP or s3 browser, connect to the IDP default s3 bucket and go to the folder <bucket_name>/templates/product.

  2. Find the template MDM_Load_RDM_Generic_<version>.json and download the latest version template to the local machine.

  3. Open the pipeline template MDM_Load_RDM_Generic_<version>.json in any text editor and replace all the occurrences of below place holders.

    Placeholder Replaceable String
    <RELTIO_CONN_NAME> Reltio Connection name configured in Entity Collection. For example, RELTIO_MDM

    Note:  The search strings given above are case-sensitive, do not enclose with any character, replace as it is.

  4. Login to IDP OA platform and under the Data Management section, click Data Pipeline.

  5. On the Landing Page, click Data Pipeline tile to open the Task Group Pipeline Flow.

  6. Click Task Group from Template, select the latest downloaded template MDM_Load_RDM_Generic_<version>.json and then click OPEN.

  7. The pipeline task group for MDM_Load_RDM_Generic will be created. This task group is used for executing the data load process.

Load RDM Codes to Reltio RDM Tenant

Follow the below steps to load RDM Codes to Reltio RDM tenant:

  1. From the IDP OA platform, open the task group MDM_Load_RDM_Generic and then click Tasks tab.

The below figure shows the list of Tasks and their Task Plugins that are displayed under Tasks tab.

To load the RDM Codes to Reltio, RUN the task group MDM_Load_RDM_Generic. The function of each task is described below:

Note:  You can also load the RDM codes by executing the MDM_Load_Reltio_Generic_Customer_Multi_Source task group.

MDM_GenericImport Files

This task uses the s3Connector task plug-in to load both the initial and delta data files from s3 bucket to Landing and Staging tables using SCD Type 1 Full Refresh Method. Perform the below steps to execute this task:

  1. Using WinSCP or S3 browser, login to S3 bucket and navigate to the root folder (<S3_Bucket>/<Client_Folder>/input/MDM_Generic_Import*.*).

  2. Place the GIF files inside the root folder/input folder.

  3. From IDP OA platform, open the task group MDM_Load_RDM_Generic and navigate to Tasks tab.

  4. Click RUN to execute the MDM_GenericImport Files task.

  5. Once the task is successfully executed, it will load the data present in each flat file to separate Landing tables. Before each load, previous data present in the Landing tables will be truncated.

  6. The data present in the Landing tables will then be loaded to Staging tables.

  7. The Staging process pulls data from the Landing tables, cleanses the data if required, and then populates the appropriate Staging tables.

  8. For each load, data in Staging table is always either inserted or updated.

  9. Below are the genetic data file patterns that will be executed in the MDM_GenericImport Files task.

FileName Patterns
Generic_Code_Import.*

Set Current Date

By running this task, you can set the current date before you load the data using pythonExecutor plugin to identify the incremental data as of Last run date to Current time stamp.

Stage Codes to Canonical

In this task, the deformalized Generic Codes that are staged from file gets transformed into canonical schema as a normalized form that helps to generate RDM JSON for plug-in.

Task Description
Generic Codes to Canonical Moves data from Staging Generic_Code_Import_table to R_CODE_ITEMS and R_EXTERNAL_ITEMS,R_DEPENDENCY_ITEMS and R_LOCALE_ITEMS table.

The task group status will be ExecutionInProgress while the data is being extracted and status will be displayed as ExecutionCompletedonce the extract process is done.

Set Last Run Date

When you RUN this task, it uses the sqlExecutor plugin to set the load processing time as the last_run_date in the log table which helps to identify the incremental data.

Code - Canonical to Reltio

This task loads the Code from Canonical to reltio using the reltioInboundConnector plugin.

Code Reprocess

This task is RUN to restart or retry the load process when an error occurs during the initial load process.

Codes can be invalidated due to below reasons:

Codes which falls under these rules will be invalidated and ingested into odp_core_staging.Generic_Invalid_codes table for review.

  • when LOOKUPTYPE is null

  • when DATASOURCE is null

  • when SOURCECODE is null

  • when DATASOURCE,SOURCECODE,LOOKUPTYPE combination is duplicated with different CANONICAL_CODE

  • DATASOURCE, SOURCECODE, LOOKUPTYPE combination is duplicated with different CANONICAL_DESC

  • DATASOURCE, SOURCECODE, LOOKUPTYPE combination is duplicated with different CANONICAL_CODE

  • DATASOURCE, SOURCECODE, LOOKUPTYPE combination is duplicated with different SOURCEDESC

select * from ODP_CORE_STAGING.Generic_Invalid_codes; to review the codes that are invalidated.

Troubleshooting

  1. Incorrect codes are pushed into the ODP_CORE_STAGING.Generic_Invalid_codes table. after the load, delivery have to check this table and make sure there is no incorrect code present.

  2. In case of pipeline failure in any task, fix the error and restart the task from the failed task till the end.

Data Interface Document

For more information on DID forGeneric Import Code Load, refer Data Interface Documents.