Smart contact deduplication

Context

The DBF Automobiles database centralizes more than 500,000 contact records. Over time, many duplicates crept in: the same name with a slight variation, a shared address, a common business phone number... These duplicates disrupt day-to-day usage, especially in CRM and marketing.

A simple method was already in place (direct comparison of emails or phone numbers), but it proved insufficient. It became necessary to move to a smarter approach.

Problem

How can duplicates be reliably identified when the information is not strictly identical? And how can this be done at scale, while keeping control over the final decisions?

The requirements:

• Flexible yet relevant detection
• A trainable and interpretable model
• The ability to process several hundred thousand rows
• The possibility of integration into internal tools

Research

After some exploration, it became clear that the topic belonged to record linkage, a branch of data science that aims to link different entries that actually correspond to the same entity.

I quickly found the Dedupe library, which ticked every box: open-source, well documented, and based on supervised machine learning. Exactly what I needed to start a first version.

Prototyping

A first prototype was built on a 5,000-row sample. The goal: train the model to recognize duplicates using manually labeled pairs of contacts.

I developed a Python script with a Flask API to wrap the training and make testing easier. An annotation interface was also designed to streamline manual validation.

To improve the quality of the suggestions, several features were chosen as comparison criteria:

• Last name
• Phone numbers (mobile, business, home)
• Personal and business emails
• Street
• City
• An amplified feature combining street + city

I also added a button to dynamically enrich the model while it trains, which considerably improved its relevance across iterations.

Implementation

Once the model was sufficiently trained, I built a second script to run it on the production database, connecting to PostgreSQL. I proceeded in increasing batches (5,000, 10,000, up to the full database) to progressively validate performance.

The similarity threshold was set to 0.10, as it gave the best results during testing: enough duplicates identified, without generating too many false positives.

Result

The system in place now makes it possible to:

• Intelligently spot duplicates, even with partial information
• Retrain the model at any time, iteratively
• Easily integrate with the company's internal tools

This project let me concretely explore the application of machine learning to a real business case, while putting in place a durable and easily maintainable solution.

---