DDDMD Data Load

Prerequisites

The following are the prerequisite conditions for DDD Data Load:

  1. To configure Reltio tenant details, see Configure Connection.

  2. To configure notification channels such as email id or MS teams channel or slack channel, see Adding Alert Notification. Configure at least one notification channel and Subscribe to the task group pipeline.

  3. Keep the DDDMD data files and code file ready. Code files should be in generic code import supported layout. see to Codes sheet in this DDD DID.

  4. Keep MDM_Extract pipeline executed once and make sure RDM data tables are created.

  5. Create a folder with name as DDDMD/input under s3 default root folder.

  6. Task group MDM_Load_Refresh_Reltio_Views_Customer_Master should be imported and successfully executed atleast once per OA/IDP MDM release.

DDDMD Data Load Overview

DDDMD data is loaded to Landing and Staging tables using SCD Type1 Full Refresh.

DDDMD Data from Staging History table is transformed as intermediate views. The data from these views are loaded to canonical tables using generic procedure.

DDDMD RDM data codes are loaded till Reltio RDM tenant.

Note:   

To avoid the mapping issues while extracting the RDM codes from two different tenants in the same IDP environment, each RDM_SOURCEMAPPINGS table name is configured with a suffix specifying the Reltio connection name as IDPSchema.MV_@ReltioConnectionName_RDM_SOURCEMAPPINGS. The RDM_SRCMAPPINGS view references from Staging RDM table for which, this process manages the transaction and sync it from Land to Stage.

Import Pipeline Template

To Import Pipeline Template

  1. Connect to IDP default s3 bucket and go to the folder bucket_name>/templates/product.

  2. Download the MDM_Load_Reltio_DDDMD<version>.json file template to local windows folder.

    Note:   

    If there are multiple files prefixed MDM_Generic_RDM_Code_Import_* with different versions, download the latest version.

  3. Open the downloaded template json file with any text editor to find and replace the below placeholders with appropriate values.

    Placeholder Replaceable String
    <RELTIO_CONN_NAME> Reltio Connection name configured in Entity Collection. For example, RELTIO_MDM
    <S3_CONNECTION> s3 connector name example: s3cfonnecor

    Note:   

    The search strings given above are case-sensitive, do not enclose with any character, replace as it is.

  4. Login to IDP platform with valid credentials.

  5. Go to Data Pipeline app, click Task Group from Template and import the template using updated template file.

    Note:   

    Process creates the pipeline task group after the successful import. This has to be used for execution.

Operational Steps

The following are the operational steps:

  1. Keep the data files ready as per the layout defined in the DDDMD DID.

  2. Login to IDP console and open the MDM_Load_Reltio_DDDMD task group.

  3. Subscribe MDM_Load_Reltio_DDDMD task group, see Subscribe Global Task Group.

  4. Using Winscp or S3 browser, login to default S3 bucket path mentioned and go to the root folder. Place the data files inside the <s3 root folder>/DDDMD/input folder.

  5. Once files are placed, click Run in IDP platform to trigger the pipeline task.

  6. Click Show Execution Logs which is next to Run button for verifying the task status.

  7. Monitor jobs:

    1. Both Data pipeline Jobs or API call jobs used to post data to Reltio can be monitored through spring cloud data flow (SCDF).

    2. To monitor, Login to IDP portal, click Data Pipeline in Landing Page.

    3. Identify the Task Group/Pipeline name for which monitoring is required (For example. OK_US_Data_Load). Click Show Execution Logs.

    4. Spring cloud data flow page opens in a new tab. Click Jobs.

Troubleshooting

  • Mapping failure issues has to be fixed in the staging to canonical configurations. Make sure that Reltio attribute and data type is matching with it.

  • In case of failure in any task, fix the error and restart the task from the failed task till the end. If unable to identify or fix the error, contact MDM support team.

See DID of DDDMD data load to Reltio in Data Interface Documents.