Deduplication overview

Field data is never perfectly tidy. The same school can be registered twice under slightly different names, a teacher can be entered once by each of two monitors, and a single visit can be recorded more than once. Deduplication is the tool that finds these suspected duplicates so a reviewer can confirm and tidy them.

The system looks for suspected duplicate:

• Schools
• Teachers
• School visits
• Teacher visits
• Pupil assessments

It does not delete or merge anything on its own. It flags candidates, and a person decides what happens to each one. This page explains the two summary screens (country and project) and how to move down to the district where the actual review happens.

Suspected, not confirmed

A “suspected duplicate” is a guess made by the matching logic. Every flagged pair still has to be reviewed by a person before any merge or deletion. See Reviewing duplicate candidates.

The country summary

At country level the Configuration dropdown (the gear icon, shown to administrators) has a Deduplication System group with a Country Deduplication System link. This opens the country Deduplication System page, an administrator screen.

It shows one table, grouped by project, with these columns:

Column	What it shows
Project	The project, shown by its region name (links through to that region’s page)
Status	Completed, Started or Not Started
Last Evaluated	How long ago the project was last evaluated (for example “3 days ago”), or “Unknown”
Suspected Duplicates Schools	Count of suspected duplicate schools, shown as a colour-coded badge
Suspected Duplicates Teachers	Count of suspected duplicate teachers, shown as a colour-coded badge
(unlabelled)	A Deduplication System link into that project’s deduplication page

The count badges are colour-coded so problems stand out: green for none, orange for a handful, red for many. A status of Not Started means the project has not been fully evaluated yet (it shows whenever at least one of the project’s districts is still unevaluated).

This page is read-only. You use it to see which projects need attention, then follow a row’s Deduplication System link to drill in (the Project column itself links to the region’s page).

The project summary

Inside a project, open the Configuration gear (senior managers and above) and choose Deduplication System from the Data column. This opens the project Deduplication System page, headed Deduplication.

Here the table is broken down by district, with a richer set of suspected counts:

Column	What it shows
District	The district
Suspected Schools	Suspected duplicate schools
Suspected Teachers	Suspected duplicate teachers
Suspected School Visits	Suspected duplicate school visits
Suspected Teacher Visits	Suspected duplicate teacher visits
Suspected Pupils Assessments	Suspected duplicate pupil assessments
Last Evaluation	How long ago the district was last evaluated, coloured by recency
Deduplicate	A link through to that district’s deduplication page

Counts read “Unknown” or “Not Evaluated” for any district that has not yet been evaluated.

The word for a district

In most of the dashboard the level below a project is shown using your country’s own term, but in this table the header always reads District. This guide always calls it a district.

Drilling from country to district

The work always flows downwards through these three screens:

1. Country Deduplication System: see which projects have suspected duplicates, then click a project.
2. Project Deduplication System: see which districts hold the suspected duplicates, then click a district’s Deduplicate link. Administrators also get an Evaluate for Duplicates button here that evaluates the whole project at once.
3. District Deduplication System: this is where you run an evaluation for one district and review candidates.

The district page is where the review happens. Those steps are covered next.

Next: Evaluating a district