Exchange 2010 Backup Product Support Matrix

Well, the #1 recent article here is my Exchange 2007 and Exchange 2010 Backup how to. The reason, of course, is that Exchange 2010 has been out since November 2009 and select few have yet announced or released Exchange 2010 support, and many companies are still trying to find how to backup Exchange 2010 in their production environment.

While I have not used many of these yet, I am hoping to try to demonstrate them all and update this post as new information comes. However, if you have made a significant investment in any of the below (or others, email me!) and are waiting for support, the best thing you can do is ask your vendor. The more input received, the more important you are making it for them!

Here is a chart of my findings thus far:

Vendor	Product	Exchange 2010 Support	Release Date	Version
Commvault	Commvault	Non-DAG in 8.0, DAG in 9.0	9.0 - 2nd 1/2 of 2010	8/9
Symantec	NetBackup	Yes	2/4/2010	7.0
Symantec	Backup Exec 2010	Yes	2/4/2010	2010
CA	Arcserve	Expected	Unknown	v14
EMC	Networker	Expected	May/June 2010
EMC	DataDomain	Unknown	?
EMC	Avamar	Expected	Q2 2010
i365	eVault	Expected	Q2 2010
Microsoft	Data Protection Manager 2010	Expected	?	2010
Microsoft	Windows Server Backup	Yes	11/09/2009	Windows 2008

By the way, if you know of ANY updates, feel free to comment, or email me directly at me@chrislehr.com. If I said Unknown above it's more likely that your web site didn't make this information readily accessible. If I receive emails from the appropriate domain names, I will post it as official, more so if you can provide a link!

Exchange 2010 - Recovery Scenario #2 - Recover from a DAG member loss

In this scenario, I have a 3 three server DAG, and I use Windows Server backup to backup my Exchange 2010 Active database. On the server with the active copy, I hit the virtual power button. The Exchange services fail over to another server in the DAG right away.

The Microsoft documentation:

Recover a DAG member Exchange Server

Remove the copy of the Database in a DAG:

This will warn that it cannot communicate with the server. That is expected.

Then, you can remove the server from the DAG:

Reinstall Windows 2008 R2 from DVD (remember, DAG requires Enterprise!)

Reset computer account in Domain (Right click Reset in AD Users and Computers)

Name and IP the server, confirm the date/time is correct (since in a DAG, I also needed to IP my DAG network)

Install Exchange Pre-Requisites

Install Exchange 2010 using:

setup /m:recoverserver

If you skipped the DAG removal steps above, setup will fail with:

Once setup succeeds, you need to reboot the server (at this point, I would also patch as needed - at the time of this writing 2008 R2 and Exchange 2010 RC have no additional patches)

Since I have a DAG, I am able to re-add it to the database availability group and allow the database to reseed. If you were in a single server environment, this is where your backup would come into play. This might be scenario #3

Add the recovered server back to the DAG


Add a mailbox database copy to the recovered server.

Assuming you have a real DAG and you might have 300GB of data to re-seed by adding a copy, this is where the Windows Server Backup may play part. You may be able to restore the Exchange data to an alternate location, and before you add the database copy, move the restored EDB to the folder path for the database. This would allow you to skip time consuming reseeding, as long as your restored EDB was the most recent backup taken of the database.

Optionally, you can activate the database on the recovered server as well.

Exchange 2010 - Recovery Scenario #1 - Mailbox or items

In this post I wrote about how you can now backup Exchange 2007 SP2 and Exchange 2010 mailbox databases with Windows Server backup.

Since them, word of Exchange 2010's release has come, and with that, the question of "when will my backup vendor provide updates that are compatible with Exchange 2010" and "what can I do in the meantime"

One of the easiest solutions is using Windows Server backup, and then allow your existing backups to do file level backups of that data. The question once this is in place, of course is how do I recover from that?

So I intend to cover three scenarios:

1. Single Item or Mailbox recovery - accidental delete, assuming you passed or misconfigured deleted item and deleted mailbox retention.
   
2. Loss of a DAG member - How to recover from losing a single member of a DAG.
   
3. Entire Server recovery - Building/site failure, need to return to service and restore data from backups. (this will be single server from backup)

Additionally, I am using a database that is in a DAG for this, but am writing it as if it was standalone, as the #2 and #3 scenarios would be addressed by the Database Availability Group.

So, on to scenario #1 - I have disabled my mailbox in the EMC, and run the below powershell to force the database clean:

Clean-MailboxDatabase geodb1

Now I see my mailbox under "Disconnected Mailbox" in a normal scenario, this is what Exchange 2003 and up has offered, where I could right click my mailbox and choose to re-connect it to my user account:

Of course, I want to go to backups so I reset my database to have a 0 day deleted item mailbox retention and refreshed this screen and my mailbox was no more!

Do note, the settings in the above screenshot are NOT recommended. Default is 30 days, and I recommend leaving it there or higher!

Next, we must recover the data "to an alternate location" using Windows Server Backup
Choose your Backup Date, then your recovery type should be "Applications"

Choose Exchange:

(I included the show details, which is the store GUID)

I chose here to recover to another location (Note: c:\RDB1 is NOT where my RDB's EDB/logs/anything are)

Do note that "this option will copy just the application data" - there are additional steps after this!

Finally, launch the recovery.

Once completed, you will have the file structure of the database in the path specified:

Now that we have our data files, the recovery is similar to Exchange 2007 SCR's database portability.

Run ESEUTIL /R from the log file directory

Then we can run:
Eseutil /mh geodb1.edb

And determine the DB is healthy:

Next few steps edited on 11/3/2009 for missing content:

Now we can create our new Recovery mailbox database using:
new-MailboxDatabase -Recovery -Name rdb1 -Server exch2010 -EDBFilePath "c:\rdb1\rdb1.edb" -LogFolderPath "c:\rdb1\"

Then we need to allow Restores:
Set-MailboxDatabase -AllowFileRestore:True

Copy the EDB file to the path EDBFilePath of the RDB1 Database, renamed it appropriately, and then it should mount successfully (NOTE: Logs didn't need to be copied since ESEUTIL /R replayed them into the EDB, however if you do copy them into place, Exchange will see they are replayed and move on)

Once mounted, we can use

get-MailboxStatistics -database rdb1

to see that the data is there:>

Now, in the Exchange documentation, it states that:

Restore-Mailbox -Identity chris -RecoveryDatabase rdb1

would recover the data into the mailbox. The problem is, we don't have a mailbox with that GUID any more. If I re-enable a new mailbox for chris, he will get a new mailbox GUID.

Enable-Mailbox chris
Restore-Mailbox chris -RecoveryDatabase rdb1

I get:

This makes sense, it cannot match GUID's and stops - more on this in a second.

However, you are able to run a recovery operation (similar to Export-Mailbox in Exchange 2007)

Restore-Mailbox -RecoveryMailbox chris -Identity chris -RecoveryDatabase rdb1 -TargetFolder "Recovery"

And the results, all of the content in a subfolder named "Recovery"

I attempted a few other things to see if I could restore directly into the mailbox, but was not able to find any luck.

Important to note - if I was recovering for a user that missed their deleted item retention time, I can use the restore-mailbox to specify by subject, dates, folders and more. Because I mail disabled the user, I am not able to restore directly.

Best Practices for Active Directory Schema changes

Part of my job is to extend AD Schemas to support new versions for products like Exchange and OCS, and this is part of what I do prior to Schema changes for customers as well as internally.

First off, a quick review of AD schema, and what it is and the function it performs. The Schema is essentially the "database" that AD resides in, so when we say things like "extending the schema" we mean the same thing any SQL DBA would mean - we are adding additional objects attributes to AD. These new additions allow for features in products that were not previously there to store their settings in Active Directory.

Some of the recent Schema extensions you will see:

• Exchange 2007 SP2 requires schema extension.
  
• Exchange 2010 requires schema extension.
  
• OCS 2007 R1 or R2 require schema extension.
  

Additionally, while not an extension, these best practices also apply before raising your forest or domain functional levels.

Step One - Determine your Schema Master FSMO role holder

1. On any domain controller, click Start, click Run, type Ntdsutil in the Open box, and then click OK.
   
2. Type roles, and then press ENTER.
   
3. Type connections, and then press ENTER.
   
4. Type connect to server <servername>, where <servername> is the name of the server you want to use, and then press ENTER.
5. Type q to return to the fsmo maintenance prompt.
   
6. At the FSMO maintenance: prompt, type Select operation target, and then press ENTER again.
   
7. At the select operation target: prompt, type List roles for connected server, and then press ENTER again.
   
8. This will display all 5 FSMO roles. The one that has Schema is the one we need to back up.
   
9. Type q 3 times to exit the Ntdsutil prompt.
   

Step Two - Ensure you have your DSRM password

1. Most of the time, even if this is known, it has not been changed in a long time and is likely due.
   
2. Follow instructions to reset DSRM password from KB322672
   
3. This allows your backup to be authoritatively restored in the case you need to. Without this password being correct, your backup may not be usable.
   

Step Three - Take a system state backup (or two)

1. I recommend taking an ntbackup.exe (Windows 2003) or Windows Server Backup (Windows 2008) if you are more comfortable with Microsoft restore procedures.
   
2. I recommend taking another backup using whatever third party vendor product you typically use, if you are more comfortable with their restore procedures.
   
3. I usually recommend taking BOTH of the above for the Schema Master FSMO role holder.
   

While I have YET to run into any issues or problems with Schema extensions, if I ever did, I know I want a really good backup or two!

Backing up Exchange 2010 and Exchange 2007 SP2 using Windows Server Backup

2/17/2010 Update

Hi there - this was written in August 2009, 3 months before Exchange 2010 released, and these screen shots are Windows 2008, not Windows 2008 R2. We are now almost 3 mos PAST RTM of Exchange 2010 and there are starting to be other options for backup solutions. I am trying to keep a running list of these here:

http://chrislehr.com/2010/02/exchange-2010-backup-product-support.htm

Now, on to the main article:

In both Exchange 2007 SP2 on Windows 2008 and Exchange 2010, Microsoft has enabled Windows Server Backup to allow VSS backups of the Exchange database. I hope to shed some light on how to configure these backups, for both one off backups as well as scheduled daily backups.

First off - how to take a ONE off backup.

Launch Windows Server Backup on your mailbox role, and click the "Backup Once�" action.

Not having a schedule at this point, only "Different Options" is a choice

Only full server or custom is a choice. I am OK with Full server, but I will go with Custom for this.

Now, we can see that there is not much granularity to selections. Since this is VSS based, it has to be by disk. You can also see there is no "Information Store" to choose. Select ALL volumes that host Exchange (this is why I am OK with Full Server above)

For location - if I choose local drive, I will have only the DVD as an option. You cannot back up a drive to a drive that it is backing up. (this is the one down side of VSS in my opinion. It was nice to exclude e:\backups and save them there)

Specify location:

If you specify a location already used, you will receive this message.

This is nice because unlike old scheduled ntbackup.exe BKF files, we won't have an ever-growing backup set that is not being watched.

Something to note here - when you do this as a scheduled job, it needs to be a local disk. A network share will not suffice. I have used an iSCSI SAN device

This is an IMPORTANT step. If you choose copy, your tlogs won't be flushed, and your databases will not register as backed up.

Confirm the settings (not pictured screen) and then click BACKUP and you can watch progress of the VSS backup. Here the shadow copy is produced.

Exchange 2010 consistency check being run:

This backup process flushed transaction logs in Exchange 2010, marked the databases housed on Exchange 2010 as backed up. The backup set is larger than my actual Exchange data would be since I am backing up all the binaries on C every time.

This is a GREAT tool and I am very glad Microsoft listened to the need for an included backup utility.

Impact of ESX snapshot backups on Microsoft database servers

Up until Update 2, ESX 3.5 uses a VMWare tool called the Sync Manager as part of the snapshot backup process. The Sync manager quiesces the file system (pauses all incoming I/O requests and dumps dirty data to disk) before the snapshot backup is taken. This allows the backup to file-system consistent.

If you were to take a snapshot without pausing the I/O requests and then restore the snapshot, the virtual machine will start up for the first time thinking that it is recovering from a power failure type of crash. This is because the recovered system will not be able to find the I/O requests that were stored in the memory (RAM) at the time the snapshot was taken.

What does this have to do with database servers? For servers housing databases (Active Directory, Exchange, or SQL servers), stopping I/O requests with the Sync manager halts incoming requests to the database without notifying the database of what is happening. The database is waiting on that information to arrive � so when the data doesn't come in when expected, errors are logged to the event log and in some cases, the databases become corrupt. I happened to find this out on an active directory domain controller, and the sequence of errors looks like this:

First, you'll see event ID 1 logged with source LGTO_Sync. This is the Sync driver starting to do its work quiescing the file system.

On domain controllers, AD requests will begin to fail. The description will differ based on the request, but the Event ID stays the same.

For domain controllers running DNS, dynamic updates will fail as well:

For DHCP servers, you will see this:

On Exchange servers, you'll see Autodiscover errors:

If you are seeing these errors, stop using the sync manager now. Eventually you will corrupt your database.

Workaround
Stop quiescing guest database servers before taking snapshots, and start adding snapshots of virtual machine memory to your backups. Most backup applications allow you to do this. If yours doesn't, you can script it using vimsh.

Example:
vimsh -n -e "vmsvc/createsnapshot [VmId] [snapshotName] [snapshotDescription] [includeMemory]"

vimsh -n -e "vmsvc/createsnapshot XXXX FIRST_SNAPSHOT MY_FIRST_SNAPSHOT_1"

By taking a snapshot of the guest machine's memory, you are creating a full snapshot. When you restore, you restore the memory on top of the file system. When the machine starts, it will be able to access all the necessary information in memory to start normally - no crash.

Resolution
To combat the Sync Manager problem, ESX has released update2, which includes an ESX VSS tool that integrates with Microsoft VSS. It works by using the windows operating system to hold I/O requests, eliminating the need for the sync driver. When the operating system is in charge of halting its own I/O activity, the databases are notified that a backup is taking place. The databases can then pause their own processing of requests, and no errors occur.

This update is relatively new, and many third-party backup applications do not support update 2 yet, which is why I have offered the workaround here.

One last note about Microsoft domain controllers and Vmware snapshot backups
In 2006 (revised in Dec. 2008), Microsoft released KB 888794 (http://support.microsoft.com/kb/888794/en-us), which states that

"Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent. "

In reality, the BURFLAGS registry referred to in Microsoft KB290762 (http://support.microsoft.com/kb/290762) can be set so that the virtual DCs are nonauthoritative, and an existing domain controller can be set to authoritative. This will allow the USN to be overwritten by the authoritative domain controller, and no USN rollback will occur.

Performing an Exchange 2007 mailbox restore using Backup Exec 12.5

1. Create the recovery storage group using the Exchange Management Console's toolbox. Confirm you will have enough disk space to do the restore to the drives it chooses!
   
2. If you run the "restore wizard" in Backup Exec 12.5, it doesn't prompt for redirection options. However, Exchange 2007 by default will find and use a RSG for you. If there is not an RSG, the job will fail.
   
3. I used the non wizard in the screenshots below to create a Restore job, selecting the Exchange Storage group and Database to be restored, and choose the RSG redirection as well as unchecking the "commit after restore" (the restore will work either way, I just like manually mounting stores) Remember, you do NOT need to restore all databases, only the one you want data from!
   

   
   

   
   

   
   

4. Mount your RSG database
   
   

   
   

   
   

5. Launch the Exchange Recovery Toolbox again, and it's now time to "Merge or copy contents"
   
   

   
   

6. Here is where I found a very ODD issue. As you can see above, the DMNotify mailbox is in the High limit store, and this is the one I restored from. However, When I go to merge data, the Staff and High limit DB's are flip flopped. So even though Backup Exec 12.5 restored the data, when I go to mount the databases, the high limit is 2gb (empty) and the staff is 17GB (the size of my data restored)
   

   
   

   
   

   
   

7. However, I was still able to merge the mailbox I wanted out by choosing the Staff DB
   
   

   
   

   
   

   
   

   
   

8. Mail data was all successfully restored!