Exchange and the loss of single instance storage

Great post up on the Exchange Team blog about SIS. I have talked about this to many customers, and it's nice to have a link to refer people to for an explanation.

http://msexchangeteam.com/archive/2010/02/22/454051.aspx

In short, single instance storage was invented when disks were expensive and massive amounts of storage cost companies a lot more money. It worked great on that scale. As Exchange gained in popularity, and those users gained data, the disk planning for Exchange became more important, as the random I/O that SIS wrote took more time to read in than a sequential I/O. By moving to a sequential read/write model, they were able to vastly improve IOPS. So now, disk space is cheap and plentiful. Server level 2TB disks are here/coming soon, and I have a few customers with plans to do DAG's on 2TB SATA's.

Normalizing Phone Numbers to E.164 format in Excel

Recently, I had the need to import some users for a large company. In order to populate as much of their Active Directory as possible, they wanted their phone numbers to be in a standardized format. Both Microsoft and Cisco have standardized on E.164 (additional information here) as a numbering standard, which basically starts with + [country code] + phone number.

This particular customer is US based only, so all the numbers in their spreadsheet had a US country code of 1. If I had a multinational organization, some additional coding would need to be done to account for other country codes.

My major need was to simply re-input all the different numbering standards the various internal organizations had inputted their phone numbers as. In other words, normalization. This helps to set up AD for later integration of OCS, or other VoIP systems, as well as Exchange 2007 or Exchange 2010 UM.

Either way, the Excel formula I was using here was the following:
=CONCATENATE("+1",SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE((SUBSTITUTE(A2,"(","")),")","")," ",""),"-",""),".",""),"x",";ext="))

In logical order..

1. Replace ( with null
   
2. Replace ) with null
   
3. Replace space with null
   
4. Replace hyphen with null
   
5. Replace period with null
   
6. Replace x with ";ext=" (which is the E164 standard for non-DID numbers)
   
7. Concatenate the +1 country code
   

Here again without the horrible color:
=CONCATENATE("+1",SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE((SUBSTITUTE(A2,"(","")),")","")," ",""),"-",""),".",""),"x",";ext="))

The end result, computer readable phone numbers!

Exchange 2010 High Availability and Disaster Recovery Video

Made this video for Simpler-Webb regarding Exchange 2010 Database Availability Groups and the differences between Exchange 2007 HA/DR planning and Exchange 2010 HA/DR planning - it's a little goofy and embarrassing, but have at it!

Exchange CAS and the DMZ

This is really just to promote this posting on the MS Exchange team blog, which highlights planning for Client Access Roles and why MS no longer supports the "front end" in the DMZ. I have had several customers that have configured Exchange 2003 and prior with their CAS in their DMZ, and while it makes for an overcomplicated firewall configuration it is supported, however, support for this ended with Exchange 2003, and Microsoft finally wrote up some very good talking points on why this is no longer supported in Exchange 2007 and 2010.

Exchange Remote Connectivity Analyzer out of beta!

More info at the MS exchange team blog!

Shawn and Brad work hard everyday on the RCA tool, taking input from Exchange administrators, consultants, and the product development and support teams to build a tool that will help anyone work through Exchange connectivity issues. Great tool, great guys, and they are really responsive. The newest updates (specifically the CAPTCHA) is a big win for everyone!

https://www.testexchangeconnectivity.com/

I also really like the ability to manually specify server settings. Great work guys!

Exchange 2007 SCR How to - Part 2

This is a continuation from Part 1 where we configured the SCR replication.

Failing over to the target SCR server

In this example, I am not having an ACTUAL failure. I am choosing to dismount the DB on the source server. I typically also will check the "do not mount the database store on startup" just so that if I do get to stopping/starting any services later that I don't accidentally remount the DB that I am attempting to fail over. And once I successfully fail over, I delete the now "old" SG and database from the EMC as well as the EDB file and transaction logs that were associated. I like to keep out of date data tidy like this. If you disagree, at the minimum, you should move these files into a well labeled folder so you know exactly what/when the files were from so that 6 months later (or wherever your comfort level is) you can do housecleaning in an educated manner. In order to ensure we have live data, I sent myself an email just before dismounting the database. You can see behind it the Get-StorageGroupCopyStatus showing the status as Healthy, with the hard coded 50 log replaylaglength.

First, we dismount the active database (in a real failure, something did this for you)

Dismount-Database OECU-EXCH1\ExecDB

Once the source database is dismounted, we can begin the SCR activation process. The first step here is to create a new SG and a new DB. DO NOT USE the folders/files/paths of your SCR data! For example, create a new SG named "RecoverySG" and a new DB named "RecoveryDB" and have their paths be new/unused folders and paths. (this is the part I mentioned above is not clear enough in the technet article)Run powershell as admin so it can do file operations!

New-StorageGroup -Name RecoverySG -LogFolderPath 'D:\Exchange Logs\RecoverySG' -SystemFolderPath 'D:\Exchange Logs\RecoverySG'

New-MailboxDatabase -name RecoveryDB -StorageGroup RecoverySG -EdbFilePath 'D:\Exchange Databases\RecoveryDB\REcoveryDB.edb'

Notice again - those are NEW and empty paths and files. If you attempt to use your SCR data at this point, you will have undesirable results! Now we run the restore command. This checks the status of the log shipping, and will attempt to copy missing log if files if needed. It also disables the original SCR and makes the database viable for mounting.

Restore-StorageGroupCopy "EXCHANGESOURCE\ExecSG" -standbymachine EXCHANGETARGET

Now is a good time for a quick reminder on what SCR is and how it does log shipping. SCR essentially just copies log files to the target, and when a backup occurs, the target will also replay those log files. So if we skip the next step, we risk bringing up a database that essentially is only as up to date as the last backup. If that was just before this test, it may not be a big deal, or may not be noticed (especially in lab where you don't have live mail flow, etc) So now we need to run ESEUTIL /R to replay the log files. This is done by running eseutil from the location of the log files like so:

eseutil /r E00

The /r Exx is the prefix for that databases logs (you can check by looking in the log folder directory for that storage group)

This should replay the logs and bring the database to a clean shut down state. You can confirm this by running eseutil /mh and specify the EDB file. The database state should be Clean Shutdown.

The below command updates the new RecoverySG's paths to match the paths of our SCR Database. The "configurationOnly" flag here tells it NOT to move the existing file, but to just change the configuration.

Move-StorageGroupPath "EXCHANGETARGET\RecoverySG" -SystemFolderPath "D:\Exchange Logs\ExecSG" -LogFolderPath "D:\Exchange Logs\ExecSG" -ConfigurationOnly

Now we need to do the same thing for the Database - point the "new" DB at our "recovered" data.

Move-DatabasePath "EXCHANGETARGET\RecoveryDB" -EdbFilePath "D:\Exchange Databases\ExecDB.edb" -ConfigurationOnly

Now we need to set the database to allow file restore. This is what will allow this database to be mounted.

Set-MailboxDatabase "EXCHANGETARGET\RecoveryDB" -AllowFileRestore:$true

If you skip the above step, when you attempt to mount the database, you will receive an error that appears to be permissions related.

Mount-Database "EXCHANGETARGET\RecoveryDB"

This is where most admins breathe a sigh of relief, but we aren't done - we need to move user s to this DB. Well, not really. What this really does is updates these user's AD objects to have their Exchange server and homeMDB at their new location in the RecoveryDB.

Get-Mailbox -Database "EXCHANGESOURCE\ExecDB" where {$_.ObjectClass -NotMatch '(SystemAttendantMailboxExOleDbSystemMailbox)'} | Move-Mailbox -ConfigurationOnly -TargetDatabase "EXCHANGETARGET\RecoveryDB"

The "where" clause in the middle of this is to prevent you from moving the System mailboxes that are unique to each DB. When you run this command - assuming your SCR target is in a different AD site - keep in mind you will need to sync AD to have your users in the main site start coming back online. You can trigger this in AD Sites and Services or various other ways, or just wait for replication to occur. At this point the users and database should all be online and client access to their data should be restored. If any HT servers in your organization had mail delivered for these users during the outage, it would deliver now. I recommend using OWA to test data as you may have client connectivity issues to troubleshoot as well with Outlook or Outlook Anywhere.

We can see the test email that I sent just prior to dismounting the database on the SCR source, so we know the logs replayed correctly. If your data is "out of date" then the likely thing that did not work is your ESEUTIL /R to replay the logs. You may be able to dismount the database and replay these again, but if you leave the DB mounted for a while and new messages flow in, your log sequence will likely be corrupted or broken. If you do this, you likely will need to go to restoring your DB from the previous night and then attempting to replay the SCR log files again.

Reseeding back to the original source server

Reseeding back is the exact same process, but your source and target are flip flopped. So you re-seed your "live" data that is on your DR server back to your main server. First, clean up the old DB/SG and the files/folders under it on your target server. Then, you can choose to rename/modify any of the DB or SG names or paths to your liking. (This can be skipped if wanted, but needs to be done before you configure SCR) Then, you can repeat the creation of an SCR replica and reseed the data back. Once data is seeded and healthy on the target, you repeat the failover process to "fail back" Once you fail back, clean up all the SG/DB paths/names once again on the DR server. Don't forget to recreate the SCR seed to your DR location after!

Exchange 2007 SCR How to - Part 1

All of this document is based on database portability offered in Exchange 2007 SP1 known as Standby Continuous Replication, or SCR. Microsoft's article is here: http://technet.microsoft.com/en-us/library/bb738132.aspx and one of the most overlooked items I felt in this document is this bit:

I I will get into more details on this below.

Before getting started
Storage group and database paths must match on the source and target. So D:\Exchange Databases\ for the EDBFilePath must be valid on both servers. Due to this, I recommend creating the folder/path structure on the source and target as you go and name everything really smartly so you know what logs you are looking at when you are in explorer. I typically use the two cmdlets below to make sure I have the info I need:

Get-StorageGroup -server EXCHANGESOURCE ft name,logfolderpath,systemfolderpath

Get-MailboxDatabase -server EXCHANGESOURCE ft name,edbfilepath

The System Folder path is recommended to be in the same folder as the log file path for uniformity as well as to reduce the risk of missing it in this step of having to type the default location of C:\Program Files\Microsoft\Exchange Server\Mailbox\Storage Group.

Also, only one database per storage group is supported for SCR log shipping to work and replay logs on the SCR target.

I recommend putting each .edb file into a separate sub folder as well because you later (in eseutil) need to specify the database directory (not path to EDB) to replay logs, and it�s a little intimidating if you have 4-5 edb files in the same directory.

Seeding the databases

On EXCHANGESOURCE, enable SCR:

Enable-StorageGroupCopy -StandbyMachine EXCHANGETARGET -Identity "EXCHANGESOURCE\ExecSG" -ReplayLagTime 0.1:0:00 -TruncationLagTime 0.2:0:0

Those day/time formats are in day.hour:minutes:seconds format - leaving one or two zeroes does not matter for hour/day/time.

ReplayLagTime is the time that the target server will wait to replay a log file into the EDB. Above, it is set to 1 hour. If not specified, the default is 24 hours. While this may work - it can mean replaying a lot of logs, so a lower setting is preferred. There is a hard coded lag of 50 files here. This means you will always see the ReplayLagTime as at least 50 when running Get-StorageGroupCopyStatus

TruncationLagTime is the time that the SCR target will delay deleting a replayed log. This is helpful if there was ever an incident where restoring from backup had to be performed, and the replayed log files from an SCR source could be used to shorten the gap between backup and moment of failure. The Microsoft default for this is 0, however.

You may receive this warning:
WARNING: ExecSG copy is enabled but seeding is needed for the copy. The storage group copy is temporarily suspended. Please resume the copy after seeding the database.

Get-StorageGroupCopyStatus -StandbyMachine EXCHANGETARGET

Will show the SCR replication status, including copy queue length and a suspended status because the DB is not there yet. If not suspended, suspend with:

Suspend-StorageGroupCopy -Identity "EXCHANGESOURCE\ExecSG" -StandbyMachine EXCHANGETARGET

Now, on EXCHANGETARGET, we can seed the database.
Run EMS as administrator, or these will error with "Access to the path (edbfilelocation)\temp-seeding is denied"

Update-StorageGroupCopy -Identity "EXCHANGESOURCE\Executive Staff Storage Group" -StandbyMachine EXCHANGETARGET

This will then seed the data:

If you receive the following error:
Database Seeding Error: Error returned from an ESE function call (0xc7ff1004), error code (0x0).

You need to enable and allow Windows Powershell as a program in Windows firewall.

Once the seeding is completed, the suspend operation should automatically resume. If it does not, you can manually do this with:

Resume-StorageGroupCopy -Identity "EXCHANGESOURCE\ExecSG" -StandbyMachine EXCHANGETARGET

Confirming that the Database seed is healthy

From the SCR source:
Get-StorageGroupCopyStatus -StandbyMachine EXCHANGETARGET

From the SCR target:
Get-StorageGroupCopyStatus -server EXCHANGESOURCE -StandbyMachine EXCHANGETARGET

This outputs something like:

Name		SummaryCopyStatus		CopyQueueLength	ReplayQueueLength	LastInspectedLogTime
ExecSG		Healthy		0	1187	10/6/2009

Obviously, "Healthy" is what you want to see here. If there are NotConfigured, they are either not configured, OR you left the -standbymachine off! If you have errors, check your application event logs, ensure the folder structure is correct and read the next step below.

CopyQueueLength is the number of transaction logs waiting to be shipped. If this number is commonly growing, your WAN connection may not have sufficient bandwidth.

ReplayQueueLength is the number of logs in the SCR target's log directory waiting to be replayed. This number will increase continually until a full backup is taken on the SCR source, at which point the SCR target "replays" these logs and commits them to the EDB on the target server. It is important to know there is a hard coded lag of 50 log files that cannot be changed.

Last InspectedLogTime shows the data and time of the last log inspected on the SCR target. The time usually is � in powershell, so run something like:
Get-StorageGroupCopyStatus -StandbyMachine EXCHANGETARGET ft name, LastInspectedLogTime

Additionally, from the SCR target, you can run test-ReplicationHealth to troubleshoot any issues with SCR. This cmdlet does not work from the source server and errors that LCR (local continuous replication) is not configured. It also accepts a -verbose argument which displays a lot more detail.

Continue reading Part 2 which includes failover and failback as well.

Windows 2008 R2 and Exchange 2007 SP2

The MS Exchange team today announced that they do not support Exchange 2007 SP2 on Windows 2008 R2. Good to know before you have to reload the OS!

http://msexchangeteam.com/archive/2009/09/21/452567.aspx

Edit: They decided to reverse this!

http://msexchangeteam.com/archive/2009/11/04/453026.aspx

Creating an RPC directory on an additional IIS7 Web site for Exchange 2007 Outlook Anywhere

Usually when I get stumped, and I find the fix, I blog about it. It's usually a really good way to capture information that I myself was unable to find (and you would be surprised how often I personally refer back to my own blog entries)

Today, I had a pretty interesting challenge. A customer had an internal FQDN that they did not own on the Internet, and could not get an SSL cert issued for their AD FQDN name on their SAN certificate. Now, I have not run into this issue in a while - the last time was on a Windows 2003 (IIS6) server - this time was a Windows 2008 (IIS7) configuration.

The actual issue of splitting the certificates is a pretty well known and well documented procedure. We used the "default web site" for internal CAS, and secured it with an Enterprise CA signed certificate. I created a "External OWA" site for external CAS, and assigned the Digicert SAN cert to that site. I should also note, the External OWA site listened on a different LAN IP that was ONLY used for the NAT entry with TCP/443 (HTTPS) enabled to it.

Using Powershell, I was able to run (not exact commands, some of these will prompt you for additional required attributes)

• New-OWAVirtualDirectory -WebSiteName "External OWA"
• New-ActivesyncVirtualDirectory -WebSiteName "External OWA"
• New-OABVirtualDirectory -WebSiteName "External OWA"
• New-WebServicesVirtualDirectory -WebSiteName "External OWA"

This got me 95% of the way to having a working CAS server. The only issue - I was missing the /rpc and /rpcwithcert directories that Outlook Anywhere and RPC over HTTPS rely upon. There is no Powershell command for this, as it's not really an Exchange component.

Now, the last time I had to splut internal/external was on Windows 2003, you would back up the virtual directory in question to an XML file, then import it on the other site. This is no longer an option in IIS7.

I also admit, I do not know IIS7 very well, and I attempted to manually recreate the directories by investigating settings and mimicking them. I got pretty close, but the Exchange Remote Connectivity Analyzer was still reporting issues.

Google and Bing really didn't turn up much (rpc, iis7, windows 2008, exchange 2007, are all pretty common search words)

I eventually found this blog entry written by Saurabh Singh related to RPC over HTTP as it related to a TS Gateway issue that he ran into.

And awesomely - it WORKED.

So opening up the applicationhost.config file, I was able to build up the virtual directories and all their settings identically on a second non-default web site. Ran an IISreset, and then re-enabled Outlook Anywhere and everything worked!

Been busy, here's the brief updates!

Been a few weeks since my last post, lamentably. However, there have been some noteable posts on other blogs that I feel I would really be missing if I didn't mention here as well.

First off, The Exchange team released an Exchange supportability matrix that shows you what versions are supported/required for migrations. A great link to bookmark if you work with Exchange often.

Second, the OCS team released a planning document on OCS certificates. I still have not had the time to get through it, but it is on my reading list. I am hoping to gain a better understanding of how to approach the PKI needs of an OCS R2 implementation.

Third, Technet has been a resource to many IT Pros for years, and this year there is a sweet 25% off deal that Keith Comb's blogged about.

Now, what have I been so busy with? Work mostly. We have several large customers who we are doing Exchange 2010 designs for, as well as several Exchange 2007 implementations and migrations running concurrently. Add to that a 4 day weekend for labor day (where I managed to stay off email!) and it makes for not enough time to get it all done and blog (or read the guide above!)

Need to find more time! Off for now, preparing for a Pilot group migration for a customer in Alaska tonight!

Exchange 2007 SP2 downloadable CHM help file available

Just saw this over at Exchange Server Share, and I absolutely LOVE having this on my laptop at all times. If you work with Exchange at all, it's pretty much TFM ;)

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=85c5ef71-6a1f-4eb0-9ff3-3feee6057e96

Best Practices for Active Directory Schema changes

Part of my job is to extend AD Schemas to support new versions for products like Exchange and OCS, and this is part of what I do prior to Schema changes for customers as well as internally.

First off, a quick review of AD schema, and what it is and the function it performs. The Schema is essentially the "database" that AD resides in, so when we say things like "extending the schema" we mean the same thing any SQL DBA would mean - we are adding additional objects attributes to AD. These new additions allow for features in products that were not previously there to store their settings in Active Directory.

Some of the recent Schema extensions you will see:

• Exchange 2007 SP2 requires schema extension.
  
• Exchange 2010 requires schema extension.
  
• OCS 2007 R1 or R2 require schema extension.
  

Additionally, while not an extension, these best practices also apply before raising your forest or domain functional levels.

Step One - Determine your Schema Master FSMO role holder

1. On any domain controller, click Start, click Run, type Ntdsutil in the Open box, and then click OK.
   
2. Type roles, and then press ENTER.
   
3. Type connections, and then press ENTER.
   
4. Type connect to server <servername>, where <servername> is the name of the server you want to use, and then press ENTER.
5. Type q to return to the fsmo maintenance prompt.
   
6. At the FSMO maintenance: prompt, type Select operation target, and then press ENTER again.
   
7. At the select operation target: prompt, type List roles for connected server, and then press ENTER again.
   
8. This will display all 5 FSMO roles. The one that has Schema is the one we need to back up.
   
9. Type q 3 times to exit the Ntdsutil prompt.
   

Step Two - Ensure you have your DSRM password

1. Most of the time, even if this is known, it has not been changed in a long time and is likely due.
   
2. Follow instructions to reset DSRM password from KB322672
   
3. This allows your backup to be authoritatively restored in the case you need to. Without this password being correct, your backup may not be usable.
   

Step Three - Take a system state backup (or two)

1. I recommend taking an ntbackup.exe (Windows 2003) or Windows Server Backup (Windows 2008) if you are more comfortable with Microsoft restore procedures.
   
2. I recommend taking another backup using whatever third party vendor product you typically use, if you are more comfortable with their restore procedures.
   
3. I usually recommend taking BOTH of the above for the Schema Master FSMO role holder.
   

While I have YET to run into any issues or problems with Schema extensions, if I ever did, I know I want a really good backup or two!

Backing up Exchange 2010 and Exchange 2007 SP2 using Windows Server Backup

2/17/2010 Update

Hi there - this was written in August 2009, 3 months before Exchange 2010 released, and these screen shots are Windows 2008, not Windows 2008 R2. We are now almost 3 mos PAST RTM of Exchange 2010 and there are starting to be other options for backup solutions. I am trying to keep a running list of these here:

http://chrislehr.com/2010/02/exchange-2010-backup-product-support.htm

Now, on to the main article:

In both Exchange 2007 SP2 on Windows 2008 and Exchange 2010, Microsoft has enabled Windows Server Backup to allow VSS backups of the Exchange database. I hope to shed some light on how to configure these backups, for both one off backups as well as scheduled daily backups.

First off - how to take a ONE off backup.

Launch Windows Server Backup on your mailbox role, and click the "Backup Once�" action.

Not having a schedule at this point, only "Different Options" is a choice

Only full server or custom is a choice. I am OK with Full server, but I will go with Custom for this.

Now, we can see that there is not much granularity to selections. Since this is VSS based, it has to be by disk. You can also see there is no "Information Store" to choose. Select ALL volumes that host Exchange (this is why I am OK with Full Server above)

For location - if I choose local drive, I will have only the DVD as an option. You cannot back up a drive to a drive that it is backing up. (this is the one down side of VSS in my opinion. It was nice to exclude e:\backups and save them there)

Specify location:

If you specify a location already used, you will receive this message.

This is nice because unlike old scheduled ntbackup.exe BKF files, we won't have an ever-growing backup set that is not being watched.

Something to note here - when you do this as a scheduled job, it needs to be a local disk. A network share will not suffice. I have used an iSCSI SAN device

This is an IMPORTANT step. If you choose copy, your tlogs won't be flushed, and your databases will not register as backed up.

Confirm the settings (not pictured screen) and then click BACKUP and you can watch progress of the VSS backup. Here the shadow copy is produced.

Exchange 2010 consistency check being run:

This backup process flushed transaction logs in Exchange 2010, marked the databases housed on Exchange 2010 as backed up. The backup set is larger than my actual Exchange data would be since I am backing up all the binaries on C every time.

This is a GREAT tool and I am very glad Microsoft listened to the need for an included backup utility.

Exchange 2007 SP2 released!

Saw this came out last night!

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=4c4bd2a3-5e50-42b0-8bbb-2cc9afe3216a

Major features:

1. ntbackup replacement via Windows Server Backup
2. Allows for Exchange 2010 co-existence

Keep in mind this is a schema upgrade, so make sure you take a proper backup of your Schema master.

Outlook for Mac OS coming soon, Entourage EWS out now!

The previously known as Entourage EWS beta has evolved into Outlook for the Mac, coming Sept 15th, 2009.

Lots more info at:
http://www.microsoft.com/presspass/press/2009/aug09/08-13MacOutlookPR.mspx

If not paying for things is your bag, the Entourage EWS client is now publically available at:
http://www.microsoft.com/mac/downloads.mspx?link_name=downloads

I have a customer that just went SBS 2008 and have a lot of MacOS machines, and I am hoping this solves a few of their issues.

If going to Exchange 2007 - Windows 2003 or Windows 2008?

I had a customer ask this very question, and I replied with this. Feel free to comment and add any pros/cons you can think of.

Pros of deploying Exchange 2007 on Windows 2008:

Quicker deployment time � Win 2003 requires IE7, IE8, SP2, and about 55 patches to get it current. This speeds initial deployment, but more importantly in a crisis when you need to redeploy, you won�t need to wait as long for patching procedures. Additionally to this point, 2008 keeps the OS CD installed to a hidden directory unlike 2003 that will prompt for a CD when you add/remove features like IIS. Windows 2008 also has command line deployment for most Exchange pre-req�s so that spin up time is pretty quick.
Longer solution shelf life � Windows 2003 support lifecycle policy is to end support 2 years after the last Service Pack. SP2 for 2003 released in March of 2007. So unless we see SP3, at this point, Windows 2003 is already out of support. We typically attempt to deploy solutions that last the 3-5 years most organizations amortize their hardware.
Less known bugs/exploits available � granted, with newer OS, there is an inherent risk of a newly found exploit being found, but there is a LOT more know about 2003 code now.
Options to change after are limited � if you go 2003 now, and later want to move to 2008, there is no in place upgrade of the OS on an Exchange 2007 server � it would be decomm a server, rebuild as 2008 and repurpose. Far easier to start off at the end point of OS.

Cons of deploying Exchange 2007 on Windows 2008:
Cost - Your first Windows 2008 server means you need to migrate all of your CAL's to Windows 2008. For smaller organizations, or anyone with their CAL's on a Software Assurance plan, this is VERY easy. If this is an unexpected expense, Windows 2003 may be your only path.
Newest stuff - If you are of the "Wait for SP1" mentality when it comes to Microsoft products, you likely will have a hard time with this decision.
Missing skillset - If you have never run Windows 2008, there is a slight learning curve. And I do mean slight - most things are VERY similar, but if admin skills are a concern, sticking to what you know may be appealing.


Again, PLEASE comment, I know this is controversial for some folks.

Unlimited TotalItemSize during 2007 to 2010 mailbox moves

Neat. While a mailbox is moving, the TotalItemSize it unlimited!

This was odd to see while trying to figure out the progress of a move. I suppose if this was a spambox accepting thousands of messages, this could be a true statement, but odd to see it when the finite size could be calculated.

A better way to check move progress is to filter on Mailbox Replication -> Mailbox Move in your application event log.

Exchange 2007 SP2 Windows backups information released

Sorry, its been almost a month since my last post. More on that in a bit.

The MS Exchange team just posted some info on how Windows backup will be able to back up Exchange 2007 in the upcoming SP2. This is a must read. Finally, an integrated way to take an Exchange backup!

http://msexchangeteam.com/archive/2009/05/13/451311.aspx

Using ADModify to ensure inheritable permissions are set on user objects

Most Exchange migrations I might find 1-2 of these that can be addressed as one off issues, but today I had customer that had a lot more than usual.

I searched for a good adsiedit, ldp, or other query - no good, its an ACL, not an AD user object property, I did find some .NET examples that I did not want to venture down that road.

And then, I found admondify.net:
http://technet.microsoft.com/en-us/library/aa996216(EXCHG.65).aspx

This requires a PSS call (or google) to download. Not an install, just an unzip and use utility.

Click Modify Attributes

Select domain, select a DC, Hit the big green arrow

Then select the root or OU you want and enter a custom LDAP query of (objectClass=*) and select add to list

Then Ctrl-a or control select user objects and hit next.

Then you can go to the account tab (normally would be security->advanced in ADUC) and finally select the "allow inheritable permissions to propagate to this object" checkbox and go.

Quick, easy, and best of all, whatever you do generates an XML log file that you can also use to "undo" your changes!

Exchange 2007 SP1 rollup 7 released today

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=2074fefd-fa1a-4c3e-bf72-94585e454150

As seen at http://www.bink.nu/

Update Rollup 7 for Exchange Server 2007 Service Pack 1 (SP1) resolves issues that were found in Exchange Server 2007 SP1 since the software was released. This update rollup is highly recommended for all Exchange Server 2007 SP1 customers.

For a list of changes that are included in this update rollup, see KB960384.

This update rollup does not apply to Exchange Server 2007 Release To Manufacturing (RTM). For a list of update rollups applicable to Exchange Server 2007 RTM, refer to the section Update rollups for Exchange Server 2007 RTM in the Knowledge Base article KB937052.

MS exchange team blog article:

http://msexchangeteam.com/archive/2009/03/18/450863.aspx

Exchange 2007 HT and CAS in NLB - Unicast versus Multicast

I have implemented about 4 NLB clusters for HT, CAS, or both now, and every time the rule of thumb has been, use what works.

From the MS Exchange page on IPv6 (also applies to IPv4) we know that:

• Unicast address A packet is delivered to one interface.
  
• Multicast address A packet is delivered to multiple interfaces.
  

More searching turned up this very helpful article describing the differences:

http://vittoriop77.blogspot.com/2006/03/nlb-unicast-vs-multicast.html

So, in one implementation, the customer had a Proventia M60 firewall, and unicast ended up working and performing more reliably. In another using an Juniper SSG firewall, unicast performed poorly and multicast ended up being the fix.

I searched the Exchange implementation guide and CHM thoroughly and didn't find any particular guidance, so I think at this point trial and error is the best bet. Learn what works in your config and go with it.

Restoring Exchange using Microsoft DPM 2007 SP1

This article will walk you step-by-step through performing a restore on Microsoft Exchange 2007 using Data Protection Manager 2007 SP1.

To recover Exchange data, you must first have an active recovery storage group mounted. To do this, open the Exchange Management Console (EMC), go to tools, and then click "Troubleshooting Assistant." You must first enter the server and user information and click next.

Next, you must select a task. Choose "Create a recovery storage group."

Now you must select the recovery storage group that contains the mailbox you want to recover. For this example, we will choose the executive storage group. Click next.

The next screen asks you to name the recovery storage group and verify the location of the exchange data and logs. When done, click "Create Recovery Storage Group."

The last screen will show you whether or not the creation was successful:

Next, recover the database to the recovery storage group. Log onto your DPM server, and go to the Recovery tab of the DPM 2007 console. Expand the Exchange server, and click on the storage group from which you want to restore the mailbox database:

Right-click on the database in the right-hand side of the screen and click "Recover�" Click Next on the review screen.

Select "Recover to Recovery Storage Group."

Specify the Exchange server, the Storage Group Name (Recovery Storage Group), and the database (Executive Database):

Select to mount the databases after they are recovered:

Click Next, review your recovery settings, and then click "Recover."

Now to recover a single mailbox, you will need to open the Exchange Management Shell on your exchange server. The following commands are examples of different mailbox recovery scenarios:

• To recover a single mailbox from the RSG database to the active mailbox 'username', run the following command in the Exchange Management Shell:
  

Restore-Mailbox -identity 'UserDisplayName' -RSGDatabase 'RSG\mailbox
database'

• Use the following command to restore a mailbox in a Recovery Storage Group into a folder in the same active mailbox or a different active mailbox:
  

  Restore-Mailbox -RSGMailbox 'SourceUsername' -RSGDatabase 'RSG\Mailbox Database' -id 'DestinationMailbox' -TargetFolder 'FolderName'
  

• To recover email in a particular date range into a folder in the same active mailbox or a different active mailbox, use this command:

Restore-Mailbox -RSGMailbox 'SourceUsername' -RSGDatabase 'RSG\mailbox database' -id 'DestinationUsername' -TargetFolder 'FolderName' -StartDate 'mm/dd/yy' -EndDate 'mm/dd/yy'

A note about "single mailbox restores" in DPM 2007: If you use DPM 2007 to restore a single mailbox, DPM still restores the entire mailbox database to your Exchange server, and you still have to use powershell to pull the mailbox data out of the database. Personally, I don't use this feature. It seems repetitive to me. But if you do use it, make sure you have enough space to restore the whole database.

Exchange 2007 and machines with underscores in their name

I somehow completely forgot this won't work. Luckily there is a workaround.

Rename the machine :( Easier said than done for some situations.

Exchange 2007 SP1 RU6 released yesterday

The Exchange team RTW'd SP1 RU6 yesterday.

The KB Article and Download are now available

Some important fixes included in this rollup include: (from the Exchange Team blog )

1. Fix for a security issue which has been assigned a severity rating of critical. More information about the issue can be found in the Microsoft Security Bulletin http://www.microsoft.com/technet/security/bulletin/MS09-003.mspx
   
2. Fix to allow Internet Explorer 8 to be used for Outlook Web Access (OWA) 2007. This does not include the OWA 2007 S/MIME control. We are still working on some changes in the control to make it work better with Internet Explorer 8. We will be releasing an updated version of the S/MIME control in a future rollup. Users using the S/MIME control should continue to use Internet Explorer 7.From the installation perspective, a reminder that the rollup installer will overwrite any OWA script files if required to ensure proper operation of OWA. If you have customized the logon.aspx page or other similar OWA pages, you will need to redo any customization after installation of the rollup.

Also of note from the Exchange Team blog post, there is a new Technet forum just for Exchange updates:

http://social.technet.microsoft.com/Forums/en-US/exchangesoftwareupdate/threads/

Exchange 2007 SP1 rollup 6 expected out Feb 10th.

This update coincides with patch Tuesday and will be a critical rated release for the security update. Also should make IE8 work with OWA 2007. Here's what it looks like for me:

More detail here:

http://msexchangeteam.com/archive/2009/02/06/450583.aspx

Exchange 2007 with Entourage 2008 will use EWS!

Finally!

Entourage users have been forced to use the "mostly" backwards compatible from Exchange 2003 WebDAV to do Exchange connectivity. In Entourage 2008 Beta, this changes to Exchange Web services.

From the Exchange team blog:

The Office for Mac folks over in the Mac Business Unit have announced availability of the public beta of Entourage for Exchange Web Services.

Exchange Web Services is a modern, robust protocol that enables client applications to communicate more efficiently with the server running Microsoft Exchange Server 2007 SP1 with Update Rollup 4 or later.

Besides saying goodbye to WebDAV, used by earlier versions of Entourage to access Exchange, Entourage users can now synchronize Tasks, Notes, and Categories with Exchange Server. Recipients can be resolved against the Global Address List (GAL) using EWS/HTTP, and improved AutoDiscover functionality to keep account settings up-to-date after setup.

Exchange Named properties limits and risk mitigation

Huge thanks to my co-worker, Matt Bastek (ill link a email or URL when he gets me one) for a lot of the leg work on this one in hunting it down. I merely played the role of reporter here.

Lately, we have heard a few customers reporting Event ID 9667 in their Application event log stating a failure to create a named property.

And reporting that some users were no longer able to receive email. The NDR sent to another internal user was:

#550 5.2.0 STOREDRV.Deliver: The Microsoft Exchange Information Store service reported an error. The following information should help identify the cause of this error: "MapiExceptionNamedPropsQuotaExceeded:16.18969:3D010000, 17.27161:00000000AC000000000000000000000000000000, 255.23226:00000000, 255.27962:7A000000, 255.27962:56000000, 255.17082:00090480, 0.16993:80030400, 4.21921:00090480, 255.27962:FA000000, 255.1494:00000000, 255.26426:56000000, 4.6363:0F010480, 2.31229:00000000, 4.6363:0F010480, 2.22787:00000000, 2.22787:00000000, 2.22787:00000000, 4.5415:00090480, 4.7867:00090480, 4.4475:00090480, 4.4603:00090480, 4.5323:00090480, 255.1750:00090480, 0.26849:00090480, 255.21817:00090480, 0.24529:00090480, 4.18385:00090480". ##

We investigated, and found out that there is a named properties table that is written to with additional headers from any internet messages. You can add these headers yourself, but often it is third party applications that are scanning, routing, marking emails with their stamp.

Here is the Microsoft article we found on the issue:
http://technet.microsoft.com/en-us/library/bb851495.aspx

Typically these are harmless and do not pose a risk. For many applications they are relied upon.

The issue we found with mail stopping delivery was due to the named properties database hitting its maximum limit as defined by the registry. This is 8192 rows by default, but is user configurable at HKLM\System\CurrentControlSet\Services\MSExchangeIS\SERVERNAME\STORENAME-GUID\

DWORD Value "NonMAPI Named Props Quota" Based on our findings this has a hard limit of 32767.

From here, we learn the default maximum header size is 64kb.
http://technet.microsoft.com/en-us/library/bb124345.aspx

64kb is ABOUT 2300 unique named properties. Knowing this, I sent a 61k message, hoping to fly under the radar. The email never came in and sat in the queue. I downed the VM and bumped RAM from 1024 to 2500 at this point.

In order to monitor the named properties rows, you need to enable the perfmon dll for Exchange.
http://support.microsoft.com/?kbid=254606

Then you can view/monitor your row count:

The below screen was the result of a single email on a test domain with 10 new headers.

I sent approximately 10k of random x-headers for this jump from 1200 to 1602

Here is the email - looks harmless to ANY user:

Unless you look at the headers:

So I sent a few emails to get the job done.

Now, it was about here I noticed I didn't get all my emails! So I checked the queue viewer and found that it was queuing up.

OK, great, maybe there was another layer of protection here. I started trying smaller snippets of header after this. Smaller gains in more emails, but I still only sent a handful.

Now, this is a VM with 2GB of RAM, so a memory issue isn't shocking. After waiting and submitting a few smaller emails as well, the works started flowing and they all went through:

And finally, I sent the final blow. All told, I only sent 25 emails for a total of around 120kb of data.

And what happened?

Not much, unfortunately.

In the event log, a warning, then a failure:

After that, I tried sending:

OWA to OWA internally - worked.
SMTP in without additional headers - worked.
SMTP in with additional headers - worked. Errors as shown above, but no non delivery of messages.

No service really denied. All aspects of email continue to work, but your event log begins to flood with those 9667 event id's These events are apparently pretty harmless.

Possible Risks:

• If your NamedProperties were full and you tried to add a new SMTP based service relying on new X-headers, and it might not function right.
• If you had existing NamedProperties applications (anti virus, anti spam, sharepoint, OCS, etc) these might have issues functioning correctly.
• Your application event log could be filled with the warnings and subsequent errors.
• Eventually, you could begin to have email denied to recipients in that database. While I was not able to reproduce this (this is a zero load Exchange server) the entire project here started because a customer running Exchange 2007 SP1 RU4 complained about this.
  

According to Microsoft, there are two fixes.

1. Up the registry entry at HKLM\System\CurrentControlSet\Services\MSExchangeIS\SERVERNAME\STORENAME-GUID\ (effective until you hit the hard limit of 32767)
2. Create a new DB and move mailboxes to the new store. Time costly option for little gain other than quiet event logs.

Details on both of these are here:
http://technet.microsoft.com/en-us/library/bb851493.aspx

Overall, I am not sure if this is really a risk, and if it is, I would love to know how.

Please, let me know.