Thursday, July 28, 2011

Exchange MCM Update

I'm happy to say that I definitely will be entering into the next Exchange MCM rotation in September.  My three week training begins September 12, 2010 and ends October 1st.

I'm really excited to be entering this exclusive program.  If I pass (and survive) the course I will be ExtraTeam's fourth MCM on staff, along with Mike Sneeringer, Tom Pacyk, and Keif Machado.  I have a lot of studying to do before September!

Tuesday, July 19, 2011

Fixing Time Errors on VMware vSphere and ESX Hosts

Time synchronization across a Windows domain is very important.  If a member server's clock varies more than 5 minutes from other domain servers, Kerberos tickets will fail.  This causes random authentication errors for users and/or applications which are sometimes difficult to troubleshoot.

Normally, time is synchronized in a Windows domain using the domain hierarchy.  The domain controller holding the PDC Emulator FSMO role is normally configured to get time from an authoritative NTP time source, and syncs time with all the other DCs in the domain.  The domain clients in each site sync time from the DCs in their local site, maintaining a relatively close synchronization of time across the domain.

Virtual machines are no different than physical computers and normally sync time using the same domain hierarchy.  Lately, however, I've seen VMs running on VMware vSphere boot up with random time differences from the domain.  I've seen this problem with three different clients lately, so I figured this might be a pervasive enough issue to blog about.

The trouble happens when the VMware vSphere, ESX or ESXi host does not have an accurate source of time, or time "drifts" due to an inaccurate system clock module.  vSphere and ESX hosts run a proprietary operating system and are not domain member servers, therefore they do not participate in domain hierarchy time synchronization. 

Most companies that use VMware hosts use vCenter to manage these hosts and their VMs.  Often, the servers that run vCenter are domain member computers and administrators think that since the vCenter syncs time with the domain, the hosts and VMs do, too.  Not true.  You need to configure the vSphere or ESX hosts to sync time from an accurate time source, otherwise the VM guests may start up with the wrong time - this can happen even if time synchronization between the virtual machine and the ESX server in VMware Tools is not enabled.

Here's how to configure your vSphere or ESX hosts to get time from an authoritative source.
  • Logon to vCenter and select your vSphere or ESX host.
  • Click the Configuration tab and then Time Configuration under the Software heading.  Notice that the time on the vSphere host does not match the domain time shown on the Windows client running vCenter .

  • Click Properties in the top left of the Configuration tab.  This opens the Time Configuration window.

  • Click the Options button and add a new NTP server that is the accurate source of time.  I recommend using the PDC emulator, since it should already be configured as an authoritative time source. 

  •  Select the checkbox to Restart NTP service to apply changes and click OK twice to close the Time Configuration window.  You will see that the vSphere/ESX host now has the correct time and is configured to use as its time server.

You may need to restart the VM guests running on that VMware host to have them sync time with the domain.  The Windows Time service will not correct the time on the VMs if it varies too much from domain time.  All domain computers sync time when they start up on the domain, regardless of how far out of sync they were.

I have not seen this type of behavior with Hyper-V, only vSphere, ESX and ESXi hosts.

Monday, July 18, 2011

How to Configure Exchange 2010 SP1 Federation

Exchange federation allows different Exchange organizations to share free/busy information with each other.  It does this without having to configure a one- or two-way trust between the organizations.

Federation is accomplished using the Microsoft Federated Gateway server, a free cloud-based service offered by Microsoft.  The Microsoft Federated Gateway (MFG) server acts as a trust broker between federated organizations, similar to the way a trusted root CA works for certificates.  All organizations that use federation must configure a one-time federation trust with the MFG, and orgs that share free/busy information must have an Organization Relationship with the other org(s) they want to share with.  Organization Relationships (sometimes called sharing policies) can be one-way, meaning that CompanyABC can share free/busy info with CompanyXYZ, but not necessarily the other way around.  Usually, each org will have a reciprocal Organization Relationship with the other org so they can see each other's calendar data.

There are a number of articles that explain how to configure federation, but all of them are for Exchange 2007 or Exchange 2010 RTM.  Exchange Server 2010 SP1 simplifies federation configuration, primarily by eliminating the requirement for a trusted-CA certificate and providing most of the federation configuration from the Exchange Management Console (EMC).

Microsoft also changed the Microsoft Federation Gateway servers in Exchange 2010 SP1.  The RTM version uses what Microsoft calls the "consumer instance" of MFG and requires a trusted certificate for federation.  SP1 uses the same Microsoft Online Services MFG used by the Business Productivity Online Suite (BPOS) and Office365, Microsoft's cloud offerings.  This new Online Services MFG uses self-signed certificates for federation (recommended), but can also still use trusted third-party certs.

If you are using the new SP1 MFG and try to create an organization relationship with an external org that uses the RTM MFG, you will see the following warning:

The token issuers for the domain (uri:WindowsLiveID) don't match the issuer for your organization (urn:federation:MicrosoftOnline).

The following guide explains how to configure federation between two Exchange 2010 SP1 organizations.

Note: This article assumes there is a working autodiscover record for the partner organization.  Federation uses autodiscover to automatically configure the Organization Relationship for the remote org.  If autodiscover is not working, you will need to enter that information manually.

Create a new Federation Trust
  • Open the Exchange Management Console (EMC) and select the Organization Configuration node.
  • In the Actions pane, select New Federation Trust.  The New Federation Trust wizard will run.
  • Click New to form the new trust with the Microsoft Federation Gateway.  The wizard will create a new self-signed certificate called Exchange Delegation Federation with the subject name of Federation.  The Federation and SMTP services will be assigned to this certificate, but it will not change the default SMTP certificate.  The Microsoft File Distribution service will automatically copy and install this self-signed certificate to all of your Exchange 2010 client access servers.
  • Click Finish to close the wizard.

Create Domain Proof Records
Domain Proof records are TXT records created in your domain's external DNS zone.  The purpose of these TXT records is to prove the identity of your domain for the trust with the MFG server.  Exchange SP1 requires that you have at least two TXT records, one dedicated for domain delegation (typically, and another for each SMTP domains you use for users (for example,

Run the following cmdlets from the Exchange Management Shell (EMS) to generate the domain proof values:

Get-FederatedDomainProof -DomainName
Get-FederatedDomainProof -DomainName

Repeat the second cmdlet for additional SMTP domains you want to federate, if any.

Each cmdlet will generate a unique Proof value, based on a hash using the Exchange Delegation Federation self-signed certificate.  If the MFG can read the domain proof value in an external DNS record and it matches the calculated value, it proves domain ownership and validates the trust.

You must create one TXT record in external DNS for each of the Proof values.  How you do this depends on your external DNS management platform.  Here's how that looks for Microsoft DNS:

And here's how it may look in a managed DNS web GUI:

Remember, these TXT records should be entered in your external DNS, not internal.  You may need to wait a bit for the new TXT records to propagate across the Internet.  You will be unable to manage the federated domains until the MFG servers can access the domain proof TXT records.

Manage the Federated Domains
Once the domain proof TXT records have propagated you can add the federated domains to the Federation Trust.  But before we can add the federated domains, we must first add the new namespace to the Accepted Domains on the hub transport configuration.
  • Back in the EMC navigate to Hub Transport in the Organization Configuration node.
  • Click the Accepted Domains tab and click New Accepted Domain in the Actions pane.
  • Enter Exchange Federated Delegation for the Name and enter for the Accepted Domain, then click New.  This new authoritative accepted domain will never be used by users - it is only used by the federated trust.

  • Click the Organization Configuration node and select the Microsoft Federation Gateway trust under the Federation Trust tab.
  • Click Manage Federation in the Actions pane.  You will see the current federation certificate status.  You can click Show distribution state to check that the federation certificate is installed on all Exchange 2010 client access servers. 
  • Click Next to bring up the Manage Federated Domains window. 
  • Click Add and select the Microsoft Federated Trust accepted domain you created earlier.  I recommend adding just the Microsoft Federated Trust first, which creates the delegation namespace on the MFG server, the unique Application Identifier (AppID) and Application URI.  Then go back and add the SMTP domain(s) you want to federate (i.e.,
  • Click Next and Manage to configure Microsoft Federated Trust.  When the configuration is successful you will see the federation trust has an Application Identifier and Application URI.

  • If the TXT records you created earlier are incorrect or have not propagated yet to the MFG server, you will get the following error:
Proof of domain ownership has failed. Make sure that the TXT record for the specified domain is available in DNS. The format of the TXT record should be " IN TXT hash-value" where "" is the domain you want to configure for Federation and "hash-value" is the proof value generated with "Get-FederatedDomainProof -DomainName".
The proof of domain ownership is not valid or is missing.
  • Once you have configured the original Microsoft Federated Trust you can repeat these steps to add your other accepted domains.  You can only add accepted domains that you have created domain proof TXT records for.

Create Organization Relationships
Now that the federated trust has been created and then validated by the MFG, you can create organization relationships.  These are the federation sharing policies that determine what is shared with whom.
  • Click the Organization Relationships tab on the Organization Configuration node in the EMC.
  • Click New Organization Relationship in the Actions pane.  The New Organization Relationship wizard will start.
  • Enter a name, such as Share with CompanyXYZ.
  • Select the Enable free/busy information access checkbox and specify the free busy data access level you wish to share using the dropdown box.
  • You may select a security group for which this relationship should apply.  If you do not select a security group the settings will apply for all users.

  • Click Next to enter the External Organization details.
  • Enter the domain you want to federate with (i.e.,, then click Next and New.  Exchange will create a new organization relationship using the data results from the Get-FederationInformation cmdlet.  If the external domain does not have a valid federation trust with the MFG or autodiscover record, you will see an error:
Federation information could not be received from the external organization.
  • When the organization relationship has been successfully configured you will see it listed under the Organization Relationships tab.  Sharing Enabled and Calendar enabled will show as True.

Testing and Troubleshooting
Use the following command to query for TXT records in DNS:

nslookup -q=txt [DNS server name to query]

Use the following cmdlets to get or test Exchange federation configuration information:
  • Get-FederatedOrganizationIdentifier - gets the Microsoft Exchange Server 2010 organization's federated organization identifier and related details, such as federated domains, organization contact, and status.  The Enabled attribute will show as False until the MFG has validated the trust using the domain proof TXT records in external DNS.
  • Get-FederationInformation - gets federation information, including federated domain names and target URLs, from an external Exchange organization.  It does this using the autodiscover record of the external domain.  This cmdlet will not work until you have a valid Federated Trust configured.
  • Get-FederationTrust - displays the federation trusts configured for the organization.  Use with Format-List to display the ApplicationIdentifier and ApplicationUri attributes, details about the federation certificates. and token information.
  • Get-OrganizationRelationship - gets settings for a relationship that has been created for free/busy information access or secure e-mail delivery using federated delivery.
  • Test-OrganizationRelationship - verify that the organization relationship is properly configured and functioning as expected for a given user.
  • Test-FederationTrust - runs the following series of tests to ensure that federation is working as expected:
    • A connection to the Microsoft Federation Gateway is established.  This test ensures that communication between the local Exchange server and the Microsoft Federation Gateway is working correctly.
    • Certificates are checked to ensure they're valid and can be used with the Microsoft Federation Gateway.
    • A security token is requested from the Microsoft Federation Gateway.  This test ensures that a token can be properly retrieved and used.

    Note that you must run the Test-FederationTrust cmdlet from either an Exchange Server 2010 Hub Transport or Client Access server.

If you're federating with a mixed-mode Exchange organization with Exchange 2003 users (as in a migration scenario) you will need to populate the TargetSharingEpr attribute of the Organization Relationship with that domain.  If you don't populate this value the free/busy information for Exchange 2003 users will be unavailable.  Populate the TargetSharingEpr value  in both organizations with the following cmdlet:
Set-OrganizationRelationship "CompanyABC" -TargetSharingEpr
Replace with the FQDN used by the external organization's Exchange Web Services (EWS) ExternalURL.  For example, run the following cmdlet in CompanyABC:
Get-WebServicesVirtualDirectory -Server ex2010 | fl ExternalUrl

ExternalUrl :

CompanyXYZ should set Organization Relationship's TargetSharingEpr for CompanyABC to

Continuing the example, run the same cmdlet in CompanyXYZ:

Get-WebServicesVirtualDirectory -Server exchange01 | fl ExternalUrl

ExternalUrl :

CompanyABC should set Organization Relationship's TargetSharingEpr for CompanyXYZ to

Thursday, July 7, 2011

Fix for DCOM 10009 Errors in Exchange 2010 SP1

You may notice DistributedCOM 10009 errors in the Windows Server 2008 R2 System Event Log whenever you run any of the following Exchange 2010 SP1 cmdlets:
  • Get-OWAVirtualDirectory
  • Get-WebServicesVirtualDirectory
  • Get-ActiveSyncVirtualDirectory

The DCOM 10009 error reads as follows:
Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          7/1/2011 10:16:11 AM
Event ID:      10009
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
DCOM was unable to communicate with the computer using any of the configured protocols.
This happens because of an security context error when invoking an RPC call to the remote CAS server.  The fix is to direct the RPC Runtime to ignore delegation failures.  This can be done by configuring the registry on both the source and target machines, but is more easily done using Group Policy.
To configure Ignore Delegation Failures manually:
  • Run REGEDIT on the source computer
  • Navigate to HKLM\Software\Policies\Microsoft\Windows NT\Rpc
  • Create a new DWORD value called IgnoreDelegationFailure with the value of 1
  • Restart the computer
  • Repeat for each Exchange 2010 SP1 Client Access Server
 To configure this setting using Group Policy:
  • Open the Group Policy Management Console
  • Edit the Group Policy Object (GPO) that applies to the Exchange 2010 SP1 servers.  I usually use the Default Domain Policy.
  • Navigate to Computer Configuration > Policies > Administrative Templates > System > Remote Procedure Call
  • Double-click Ignore Delegation Failure.
  • Enable the policy and set the Ignoring Delegation Failure setting to ON.
  • Restart the Exchange 2010 SP1 Client Access Servers
This DCOM 10009 error does not seem to affect Windows Server 2008 servers, only Windows Server 2008 R2.