Why Your CIs Keep Duplicating Despite Discovery: Understanding the IRE

Discovery runs every night. The scans complete without blocking errors. And yet the same machine shows up three times in the CMDB, under three slightly different names, with three different last discovered dates.

The usual reflex is to blame Discovery, then launch a manual merge campaign. Six weeks later the duplicates are back. The problem is almost never in the collection step. It sits in what happens immediately after, when the collected data reaches the CMDB.

That gateway has a name: the Identification and Reconciliation Engine, or IRE. It alone decides whether an incoming record matches an existing CI or warrants a new one. Understanding its rules changes the nature of the work: you stop cleaning up symptoms and start closing the doors duplicates come through.

What Discovery does, and what it does not

Discovery collects. It queries a host over SSH, WMI or SNMP, runs its probes and sensors, and produces a set of attributes. At no point does it compare that result against what already exists in the CMDB.

That comparison is the IRE's job. Discovery hands it a JSON payload, the IRE evaluates it against the identification rules of the target class, then decides: update an existing CI, create a new one, or flag an ambiguity.

The consequence is direct. A duplicate that appears after a scan is not a collection defect, it is an engine verdict. The IRE looked at the data and concluded, quite logically given what it had, that it did not know this machine.

How the IRE decides a CI already exists

Every CI class has an identification rule. You can see it in CI Class Manager under the Identification Rule tab, or as a list under Configuration then Identification/Reconciliation then CI Identifiers.

A rule holds one or more identifier entries. Each entry carries:

• a numeric priority,

• one or more criterion attributes, the attributes used as a key,

• a flag stating whether the entry identifies the CI on its own or through a relation to another CI.

The IRE evaluates those entries in increasing priority order. On cmdb_ci_hardware, the first entry combines serial number and serial number type, the next one uses serial number alone, and so on. As soon as an entry returns a single match, the CI is updated and the remaining entries are skipped. If no entry matches after the full evaluation, a new CI is created.

A duplicate is a payload that went through every identifier entry of its class without ever matching.

The other half of the engine: reconciliation

Identification decides whether the CI exists. Reconciliation decides what it contains. Reconciliation rules state which data source is allowed to write which attribute, on which table. Data source precedence rules break the tie when two authorised sources target the same attribute.

These rules do not create duplicates, but they produce the other classic symptom: an attribute whose value flips on every cycle because two sources are fighting over it. If you see that behaviour, look at reconciliation, not identification.

First cause: writes that bypass the IRE

Identification rules only apply to methods that go through the engine. Going through the IRE: Discovery, Service Mapping, Service Graph Connectors, imports run through IntegrationHub ETL, and the platform identification APIs.

Not going through the IRE: a GlideRecord inserting straight into a CMDB table, a plain transform map targeting cmdb_ci_server, a manual import, a one-off migration script.

// Direct write: no identifier entry is ever evaluated
var ci = new GlideRecord('cmdb_ci_server');
ci.initialize();
ci.name = 'srv-app-01';
ci.insert(); // the CI is created even if srv-app-01 already exists

A CI created this way lands in the table with only the attributes the script filled in, and with no record in sys_object_source, the table that links a CI to its data sources and that is only populated when the IRE runs. If the class criterion attributes are empty, no later scan will be able to tie the real machine to that record. Discovery sends its payload, the IRE finds nothing, and creates its own CI next to it.

This is the most common cause on platforms inherited after a migration, and the easiest one to close.

Second cause: an identification attribute that is empty or invalid

If the payload carries none of the criterion attributes needed by at least one identifier entry, the IRE has nothing to compare. It returns the MISSING_MATCHING_ATTRIBUTES error.

The most common case involves the serial number, and it deserves a detour, because the platform does not trust every collected value. Discovered serial numbers are stored in the cmdb_serial_number table with a valid flag. A value is marked invalid when it is empty, when it consists of a single repeated character, or when it appears in the invalid serial number table dscy_invalid_serial, which lists the generic values shipped by some vendors and hypervisors.

The field scenario is always the same. A fleet of virtual machines reports a generic or empty value. The serial number entry cannot do its job. The IRE falls through to the next entry, usually the name. The day the name changes, because the machine is renamed, moves domain or comes back in uppercase, a new CI appears.

Beware the opposite reflex: making the serial_number field unique at table level is a bad idea documented by ServiceNow, because that field is unique neither by design nor in real estates.

Third cause: dependent CIs cut off from their parent

Some classes cannot identify themselves. A database, an application instance, a running process: their identity only makes sense attached to a host. Their identification rule contains related entries that require the relation to the parent to be present in the payload.

If the payload arrives without that parent, or with a parent the IRE cannot identify either, the item fails with MISSING_DEPENDENCY. Depending on the case, the engine keeps the item as a partial payload in the cmdb_ire_partial_payloads table, waiting for a source to complete the information, then merges the two.

The trap sits upstream. If the parent host is already duplicated, each host duplicate gets its own copy of the databases, instances and processes it carries. A single badly identified server can therefore produce a dozen duplicate CIs. This is why a de-duplication effort always starts with infrastructure classes, never with application layers.

Fourth cause: several matches instead of one

The opposite case exists too. The payload matches several existing CIs, for instance because two records share the same valid serial number. The IRE does not arbitrate on its own: it groups the CIs involved into a de-duplication task, on the reconcile_duplicate_task table.

These tasks are created as data flows through the engine, not by a nightly job. A growing de-duplication queue is therefore a real-time signal, not an audit report. A platform where nobody watches that queue quietly accumulates pairs of CIs the IRE has already detected.

For remediation, the platform provides a wizard that merges one task at a time, and a de-duplication dashboard in CMDB Workspace, with templates that apply the same settings to a batch of tasks in the same class.

Diagnose before you clean up

Merging CIs without identifying the cause guarantees they come back on the next cycle. The useful order of work is as follows.

1. Take two duplicate CIs and compare their criterion attributes, not their names or IP addresses.

2. Open the sys_object_source related list on each one. A CI with no known source never went through the IRE: the cause is upstream, in a script or an import.

3. Check cmdb_serial_number for the machine involved, and cross-check against dscy_invalid_serial.

4. Replay a representative payload in Identification Simulation, which builds the payload from the class criterion attributes and shows the engine decision without writing anything.

5. Read the errors the engine returns. The most informative are MISSING_MATCHING_ATTRIBUTES, REQUIRED_ATTRIBUTE_EMPTY, MISSING_DEPENDENCY and INSERT_NOT_ALLOWED_FOR_SOURCE.

{
  "items": [
    {
      "className": "cmdb_ci_linux_server",
      "values": {
        "name": "srv-app-01",
        "serial_number": "VMware-42 0c 5f",
        "ip_address": "10.20.30.40"
      }
    }
  ]
}

A payload this simple is enough to learn whether the target class identifies on name, on serial number, or on a combination of both.

What to put in place

Closing direct writes is the first measure, and the only one with a lasting effect. Every CMDB feed must use a method that goes through the engine, and legacy migration scripts must be rewritten or switched off.

Next, document criterion attributes class by class before opening a new data source. A source that cannot provide the key attributes of its class will produce duplicates from the very first import, however well the integration is built.

Finally, treat de-duplication tasks as a work queue with an owner, not as an annual purge. And stay careful with the rules shipped out of the box: adding an attribute to an identifier entry makes identification stricter, which increases the number of creations. Any rule change should be tested in Identification Simulation before it reaches production.

The question to ask is not why Discovery creates duplicates, but which attribute the engine is missing to recognise this machine. The answer almost always fits in one column name.