Duplicate Detection for Leads (preview)

I was working in one of my environments when I noticed a new item on the sitemap in the Dynamics 365 Sales Hub app. The sitemap item is called ‘Duplicate Detection(preview)’ which is a little confusing since we already have duplicate detection in Dynamics 365. More reason for me to explain what this feature is. This is not replacing the current duplicate detection settings and rules in the application, and I would recommend keeping that turned on regardless of using the new feature or not. One of the main differences with the duplicate detection that comes with Dynamics 365 and this preview feature is that the preview feature will only look for duplicate leads. Another difference is that it works a little different from the existing duplicate detection functionality.

How does it work?

The existing duplicate detection uses duplicate detection rules which are created for each table and we can configure the columns that need to be included. For example for contacts you would use the email fields or the mobile fields so that if a contact is entered with an email address or mobile phone number that is associated with an existing contact record, the duplicate detection window will alert the user of the duplicate. The way records are compared in the backend is by using MatchCodes. To learn more take a look at this article from Microsoft.

The preview feature works a little bit different as it uses an AI model which has rules based on a combination of fuzzy logic and match codes. An example of how fuzzy logic can flag a lead as a duplicate is when leads have a similar name and the same company name, or when they have a similar name and the same email domain. We can’t configure which rules the AI model uses to find duplicates, the rules that will be used are shown below:

• The same email address: Leads are shown as duplicates when they have the same email address, regardless of data in other fields
• The same phone number: Leads are shown as duplicates when they have the same phone number, regardless of data in other fields
• Similar lead name and company name: If leads have a similar name and company they are shown as duplicates. The AI Model can identify the duplicate leads even when the name and company name sounds similar. For example, there is a lead in the system with the name Jack Murphy and another lead with the name Jack Merphy. These will be flagged as duplicates because even though the spelling of the names is different, they sound similar.
• Similar lead name and email domain: If leads have a similar name and company they are shown as duplicates. The AI Model can identify the duplicate leads even when the name sounds similar. For example there is a lead in the system with the name Jack Murphy with email address jack@abc.com and another lead is entered for Jack Merphy with email address jack.merphy@abc.com. These will be flagged as duplicates because even though the spelling of the names is different, they sound similar and the domain of the email address is the same.

User In-App Experience

When users selects a lead from any view in the Sales Hub and the lead has duplicates associated to it, the user will see the ‘View duplicates’ button appear on the command bar, as shown on the image below.

Keep in mind if there are no duplicates for the lead that is selected, the button will not be visible. Similar behavior happens when a lead record is opened. If the lead has duplicates flagged for it, a warning massage shows on the toolbar with a button allowing the user to view any possible duplicates for the lead. Clicking the view duplicates from the view and clicking ‘View possible duplicates’ from the lead form takes the user to a screen where the duplicate record(s) are listed. It shows the current selected lead on the left side of the screen and the duplicated lead(s) on the right. There can only be a total of 3 leads on this screen, if there are more than 2 duplicates you’ll need to click the arrow to move to the next page in order to view the other duplicates. You’ll notice that the field(s) in the duplicate lead(s) with similar data are shown in bold on the screen, allowing users to see why the lead(s) was flagged as a duplicate.

When the current lead is selected on this screen, the user has the option to delete the record by clicking the delete button. When any of the duplicates are selected, the user also has the option to ‘Detach’ the duplicate from the current lead. Detaching a record means it will no longer be flagged as a duplicate for the current lead record. If the lead is flagged as a duplicate for another lead record, you’ll have to perform the same step to remove the duplicate flag for that lead.

Enable Duplicate Detection for Leads

If you want to take a look yourself at this preview feature you will need to turn it on first. Keep in mind that Dataverse search needs to be enabled in the organization as the AI model to detect duplicate leads uses the Dataverse search for better results. 

You can enable this feature by navigating to ‘App Settings’ in the Sales Hub app. You’ll see ‘Duplicate Detection(preview)’ below the ‘Data Improvement’ section on the sitemap. By clicking ‘Duplicate Detection(preview)’ the Duplicate Detection window will load, from where you can enable the preview. Let me know what you think!

I hope you found this article informative! Be sure to check in again next week for a new article or subscribe here to never miss another post!