IMS Factory Codes Extract

Prerequisite

The following are the prerequisite conditions for IMS Factory Codes Extract:

  1. Keep the XPO, DDD data files ready.

  2. IDP_S3_STAGE_NAME

  3. Create a folder with name as IMSFactory/codes/input/ under s3 default root folder.

IMS Factory Codes Extract Overview

Generate the XPO, DDD, DDDMD related codes into a generic Import code layout.

Delivery can use the Extracted codes file and can work with client to map to the right canonical code and load it using GIF code import pipeline.

XPO, DDD, INCLUDE_DDDMD data files are loaded to Landing and Staging tables using SCD Type1 Full Refresh.

Extract the codes either with Full Refresh or Incremental option.

Import Pipeline Template

To Import Pipeline Template

  1. Connect to IDP default s3 bucket and go to the folder bucket_name>/templates/product.

  2. Download the MDM_IMSFactory_Codes_Extract<version>.json file template to local windows folder.

    Note:   

    If there are multiple files prefixed MDM_IMSFactory_Codes_Extract_* with different versions, download the latest version.

  3. Login to IDP platform with valid credentials.

  4. Go to Data Pipeline app, click Task Group from Template and import the template using updated template file.

    Note:   

    Process creates the pipeline task group after the successful import. This has to be used for execution.

Operational Steps

The following are the operational steps:

  1. Keep the data files ready as per the layout defined in the XPO, DDD.

  2. Login to IDP console and open the MDM_Load_Reltio_DDD task group.

  3. Open the task File_Extract and configure the parameters <IDP_S3_STAGE_NAME>, <EXTRACT_S3_BASE_FOLDER>, <EXTRACT_FILE_NAME_PREFIX>.

  4. Using Winscp or S3 browser, login to default S3 bucket path mentioned and go to the root folder. Place the data files inside the <s3 root folder>/IMSFactory/codes/input folder.

  5. Once files are placed, click Run in IDP platform to trigger the pipeline task.

  6. Click Show Execution Logs which is next to Run button for verifying the task status.

  7. Verify if the file got generated in S3 in the configured path. Open the file and see if it is same as GIF code file.

  8. Monitor jobs:

    1. Both Data pipeline Jobs or API call jobs used to post data to Reltio can be monitored through spring cloud data flow (SCDF).

    2. To monitor, Login to IDP portal, click Data Pipeline in Landing Page.

    3. Identify the Task Group/Pipeline name for which monitoring is required (For example. OK_US_Data_Load). Click Show Execution Logs.

    4. Spring cloud data flow page opens in a new tab. Click Jobs.

  9. The content of extracted file can be modified to replace the canonical codes as per client requirement.

  10. Import GIF code import template and rename the Data pipeline as IMSFactory code import, keep the input file after following step 9. Sync the columns in the first task, Also change the source name to IMSFACTORY in pipeline and save the pipeline. Run the IMSFactory code import.

    Note:   

    For more information on the GIF code import pipeline, please refer to the GIF code Import GIF Pipeline.

Troubleshooting

Mapping failure issues has to be fixed in the staging to canonical configurations. Make sure that Reltio attribute and data type is matching with it.

In case of failure in any task, fix the error and restart the task from the failed task till the end. If unable to identify or fix the error, contact MDM support team.