MDM Reltio Extract

Prerequisites

The following are the prerequisite conditions for MDM Reltio Extract:

  1. Add resolveRelationEdgeTypes in OA reltio connection properties under options property as highlighted below. If already done then ignore this step.

    For Example:

    auth_host=auth.reltio.com;auth_path=/oauth/token;client_id=cmVsdGlvX3VpOm1ha2l0YQ==;svc_host=test.reltio.com;export_svc_path=/jobs/export;export_status_svc_path=/reltio;tenant_id=z2wSGtOLkuXY8T9;

    fileFormat=json;distributed=true;taskPartsCount=6;options=parallelExecution,resolveRelationEdgeTypes,resolveMergedEntities;partSize=100mb;s3Connection=s3_connector;s3Bucket=oaidp-dev-usv-iqviadev-odp;s3Folder=mdm/plugins;esbConnection=esb_default;rdmConnection=mdm_reltio_dev2_abcrdm

  2. If resolveRelationEdgeTypes is enabled for the first time for a reltio tenant then, latest task group Reltio_MDM_Extract should be run in full refresh (IDL) mode.

Setup Entities and Relations to be Ignored during Extract

It is recommended to exclude Location entity type and has address relation type as they are not used. Address attributes are taken from Entities Address reference attributes itself. Hence these are redundant.

  1. Go to IDP portal home page and under Business Unit, select IT Support.

  2. Click Entity Collection.

  3. Click of ODP.Admin.ReltioIgnoreObjects as shown in the image below.

  4. Create two entries as shown below. Replace ReltioConnectionName with Reltio Connection Name configured in connection strings.

Import Pipeline Template

To Import Pipeline Template

  1. Connect to IDP default s3 bucket and go to the folder <bucket_name>/templates/product.

  2. Download Reltio MDM Extract_<version>.json file to local folder. If there are multiple files, download the latest version.

  3. Open the downloaded template json file in a text editor and replace the following placeholders with appropriate values.

    Placeholder Replaceable String
    __SOURCE__ Ideally it can be any source name, recommended to put OV
    __RELTIO_CONNECTION_NAME__ Reltio connection name configured in connection string.
    __COUNTRY__ Put a country name as US or if it'sa region put NA or EU.

    Note:   

    It is recommended to use separate Reltio users for Reltio connection and RDM connection in configuration settings.

  4. Login to IDP platform with valid credentials.

  5. Go to Data Pipeline, click Task Group from Template and import the template using updated template file.

  6. After importing the template, it creates a task group named Reltio MDM Extract.

  7. Click on the task group and search for a task named Reltio Outbound.

  8. Select task and click Sync Objects and click Save.

  9. Select entities, click Sync Attributes and click Save.

    Note:   

    Do not change any other properties.

  10. Do the above step again for relations, merges and RDM.

  11. Now Extract pipeline is all set for execution.

Note:   

Make sure the user configured in Reltio connection strings has ROLE_ANALYTICS role.

Troubleshooting

The following are the troubleshooting details:

  • During IDL extract from Reltio to IDP, if Reltio has huge amount of data, there is a chance that Reltio's extract job is not finished within 24 hrs. Currently IDP has a limit of long running job cannot run more than 24 hrs. Although IDP job is abandoned due to the long running job limit, Reltio still runs the extraction in the S3 bucket. There are few ways to resolute the failure.

    1. Create a ticket with Reltio if there is a need to increase the Resources in case of the tasks are WAITING_FOR_RESOURCES to run in parallel. To identify, please use the transaction: {{ReltioEnvUrl}}/jobs/{{TenantId}}/tasks and view the tasks running. If any old running jobs are present, ask Reltio to kill it or use (tasks/_stop or tasks/_force_stop)

    2. If the above is done, please rerun the job once again and verify if still the issue persists.

  • If the job still fails, then proceed to the next steps for successful completion.

    1. Wait for Reltio to finish the job. When the job finishes, {{ReltioEnvUrl}}/jobs/{{TenantId}}/tasks can show about the COMPLETION status OR the folder where it is getting extracted can have an empty _SUCCESS file.

    2. Once the job finishes by Reltio, please run the stored procedure choosing the Reltio types to process one by one / Please choose whichever failed in the process to run.

      Change the parameter RELTIO_CONN_NAME (Same Connection which is used to extract the data) & S3_INPUT_FOLDER (Folder name where Reltio extraction is finished )before running the pipeline.

      To run the different profiles:

      • call ODP_CORE_LOG.MDM_RELTIO_DATA_EXTRACT_LOAD('<Reltio Connection Name>','S3 input folder', 'entities' );

      • call ODP_CORE_LOG.MDM_RELTIO_DATA_EXTRACT_LOAD('<Reltio Connection Name>','S3 input folder', 'relations' );

      • call ODP_CORE_LOG.MDM_RELTIO_DATA_EXTRACT_LOAD('<Reltio Connection Name>','S3 input folder', 'merges' );

      • call ODP_CORE_LOG.MDM_RELTIO_DATA_EXTRACT_LOAD('<Reltio Connection Name>','S3 input folder', 'rdm' );

    3. After the stored procedure is done (it can take time based on the amount of data), please run the Outbound Staging process to push the data to Staging tables to finish the steps.