Els's blog

Wednesday, June 21, 2006

USN Rollback

Lately, the only thing I've talked about is Vista. And although there is a lot more, today I would like to discuss something else: USN Rollback.

When talking about backup and restore of Active Directory, a question that I get regularly is: can we restore a Domain Controller from an image?
At first sight you might say: why not?
Restoring a system state backup to get Active Directory back to a previous state, or restoring a disk image you took a few days ago: the end result will be the same.

And that would be true in an environment with only 1 domain controller.
But in most situations there will be multiple DCs and then you will encounter a serious, yet very difficult to troubleshoot, problem: USN Rollback.

When you restore a DC by performing a system state restore, AD will change the database Invocation ID. The Invocation ID identifies the version of the database. After a correct system state restore this ID is reset before AD starts. (Event ID 1587 of source Replication will be logged in the Directory Services event log.)
For the other DCs this indicates that the database version has changed and in response they reset their high-water marks and update the restored DC with changes that occurred after the backup.

If you restore a DC from an image on the other hand, the Invocation ID will not be reset and noone will be aware of the fact that one of the DCs actually rolled back to a previous state. This will lead to inconsistencies in the database, since the restored DC will not get any updates that were taken since the backup and never will.
Why is this?

AD replication is triggered by comparing USNs (Update Sequence Numbers). Every DC has a USN table for every attribute. In that table it keeps track of the highest USN it received for this attribute from a given replication partner. If the current USN on the replication partner is higher than the one in his list, the DC knows that a change has been made and that it still has to copy these changes.

Imagine the following scenario: You have a domain containing 3 domain controllers DC1, DC2 and DC3.
  1. You create 10 new users on DC1 corresponding to USN 1 through 10. These user accounts replicate to DC2 and DC3.
  2. At this moment an image is created on DC1.
  3. You make more changes to AD: you reset the passwords on the 10 user accounts (USNs 11 to 20), you create 10 computer accounts (USNs 21 to 30), you create 10 security groups (USNs 31 to 40). All these changes replicate to DC2 and DC3.
  4. Then DC1 encounters a hardware or software failure. The image that was created is used to restore DC1. DC1 now uses a database that has a record of USNs 1 to 10 when AD starts.
  5. DC1 maintains its original invocation ID and DC2 and DC3 maintain their original up-to-dateness vector of USN 40 for DC1.
As a result DC1 will never receive the changes for USNs 11 to 40 and its database will be inconsistent (resulting in logon failures and other problems).
A second problem arises when new objects are now created on DC1.
If you create a new user, that account will be given the next unused USN on DC1: USN 11.
Since DC2 and DC3 think that they already have every update from DC1 up until USN 40 they will not do anything and the new changes are not replicated.
All changes created on DC1 with USNs 11 to 40 will never be available on DC2 and DC3.

In case of a proper system state restore these problems do not arise, since the reset of the Invocation ID informs the replication partners of the restore and causes them to reset their up-to-dateness vector for the restored DC.

A USN rollback can be very difficult to detect, since no errors are logged in the Event Viewer (unless you are running W2003 SP1 or hotfix 875495, then Event ID 2095 is logged).

One way to detect a USN rollback is running the following command on a DC and its replication partners: repadmin /showutdvec.
If the USN for the DC on the replication partners is higher than the one the DC has for itself and no replication errors are reported, you are probably dealing with USN rollback!

Tomorrow: Recovering from USN rollback!

13 Comments:

Post a Comment

<< Home