Wash the "unclean". Fixing mailbox enabled cloud-only user accounts in Office 365 for hybrid migrations.

Thursday, January 18, 2018
I've come across the following scenario in a number of migration projects. The customer creates a new tenant with or without AAD Connect with the intention of "kicking the tires" in Office 365. They create several cloud-only accounts and assign them licenses. Since these cloud-only accounts don't have an ExchangeGUID tying them back to on-prem Exchange, they are given a new mailbox in Exchange Online.

Later, the customer decides to perform a hybrid migration of all or some mailboxes to EXO. AAD Connect is configured with the Exchange Hybrid setting and Azure AD is able to "soft-match" the cloud users to the on-prem user accounts. Now we have the same user object with two mailboxes, one on-premises and the other in the cloud. This is bad. Mail flow will be affected and we will be unable to migrate the user mailbox to EXO because a mailbox already exists. To fix it, we need to delete the cloud mailbox so that Exchange hybrid can create a new (empty) mailbox to migrate to when we're ready.

If we unlicense the user in Office 365 it will soft-delete the cloud mailbox, but when we license it again Office 365 will automatically reconnect the soft-deleted mailbox back to the user account. It does this because the PreviousRecipientTypeDetails property on user account is set to "UserMailbox". When this property is set, the user account is "unclean" and Exchange Online will always want to reconnect the soft-deleted mailbox. Even if we manage to permanently delete the cloud mailbox, EXO will create another new cloud mailbox when the user is re-licensed.

Previously, the only way to correct this was to permanently delete the cloud user account using Remove-MsolUser and Remove-MsolUser -RemoveFromRecycleBin remote PowerShell cmdlets, and then sync the object back from on-prem AD. The problem with this is that it's cumbersome to do and, more importantly, also deletes any prior SharePoint and OneDrive data that the cloud user may have accumulated.

A normal synced user in a hybrid scenario before migration to EXO

In the coming days, Microsoft is rolling out a new switch for the the Set-User remote PS cmdlet which addresses this issue. Running Set-User -PermanentlyClearPreviousMailboxInfo in remote PowerShell will clear the PreviousRecipientTypeDetails property and return the user account to a "clean" condition. This will prevent EXO from creating a cloud mailbox, as shown above. The new command disconnects the cloud mailbox without affecting other data stored in Office 365, such as SharePoint or OneDrive data.

Note that this will permanently delete the cloud mailbox, so you should have the user save any important emails first or export the mailbox to a PST.

Read more ...

Email Address Internationalization (EAI) support in Office 365

Thursday, December 28, 2017
Today, Microsoft announced that Office 365 will enable Email Address Internationalization (EAI) support in Q1 2018. This may have important implications for hybrid customers and those organizations involved with mergers and acquisitions. Here is the announcement:
"Out of 7.6 billion people in the world, only 360 million are native English speakers.

Although email has been the earliest and most widely-adopted platform for modern electronic communication, email addresses have only supported a limited subset of Latin characters (mostly due to historical reasons). People who don't read or speak English have been forced to use email addresses containing characters not used in their own language.

Many people have been working together to try to fix this situation. In fact, new email standards to support email address internationalization (RFCs: 6530, 6531,6532, 6533) were published in 2012. However, changes in standards are difficult to implement in the world of technology due to the millions of legacy systems that are still out there.

Microsoft is pleased to announce that we're joining the effort to adopt the new standards. Office 365 will enable Email Address Internationalization (EAI) support in Q1 2018. As a first step, Office 365 users will be able to send messages to and receive messages from internationalized email addresses. Admins can also use internationalized email addresses in other Office 365 features (for example, mail flow rules that look for EAI addresses, or outbound connectors to Internationalized Domain Name (IDN) domains). But please note that this new release will not support adding EAI addresses for Office 365 users, or IDN domains for Office 365 organizations.  We will continue to evaluate these features as the standards are more widely adopted. We will also keep you posted on the plan to release this to Exchange Enterprise version."
As the article mentions, a change like this is difficult to implement due to millions of legacy systems out there - including Exchange Server on-premises, although I wouldn't call that "legacy". It's impossible to say how every email system will handle email addresses with "non-ASCII" character sets, but I suspect many will bounce emails as undeliverable.

Examples of EAI email addresses include:

  • Latin alphabet (with diacritics): Pelé@example.com
  • Greek alphabet: δοκιμή@παράδειγμα.δοκιμή
  • Traditional Chinese characters: 我買@屋企.香港
  • Japanese characters: 甲斐@黒川.日本
  • Cyrillic characters: чебурашка@ящик-с-апельсинами.рф
  • Hindi email address: संपर्क@डाटामेल.भारत

EAI support in Office 365 won’t matter for hybrid customers if Exchange Server on-premises doesn’t support it, since AD on-prem is the source of authority for email addresses. Microsoft says plans are being made to release EAI support for Exchange Server on-premises, but there are no release dates for this yet.

Organizations involved with mergers and acquisitions will need to be aware of EAI email addresses, especially in tenant-to-tenant migrations where EAI addresses are used. For example, if Contoso (an Exchange hybrid organization) acquires Fabrikam (an O365 tenant with EAI addresses), careful planning and reconfiguration of Fabrikam email addresses may be required.

It's also important to note that the EAI RFCs also permit the use of non-ASCII characters in email headers. This may have effects on various SMTP gateways used and may happen even if your own organization doesn't use EAI email addresses. You should ensure that your gateways are always updated. As usual, careful planning and testing is required.

Read more ...

Hub Transport fails when NIC is not set to register in DNS

Friday, December 22, 2017
In a previous post I wrote that email delivery fails after installing Exchange 2016 CU8. Fellow MCSM and MVP Brian Ricks reminded me of an old bug in Exchange Server that causes Hub Transport services to fail if a server is not configured to register this connection's addresses in DNS.

This is the default setting for all Windows servers, but in some cases, it's not appropriate. For example, Edge Transport servers are not domain-joined and will log event ID 8015 - "The system failed to register host (A or AAAA) resource records (RRs) for network adapter with settings: <NIC details>":

The problem is that the Hub Transport service relies on the NIC being configured to register this connection's address in DNS for it to use the Windows Server settings for name resolution. If you uncheck that box to prevent the DNS Client Events warning, Hub Transport no longer uses the Windows DNS settings and doesn't have any name resolution.

The solution is either to check the box to register DNS (which will fail and cause the warning) or manually configure the Transport Service to use specific DNS settings. You can do that with the following cmdlet on the server experiencing mail delivery issues:
Set-TransportService <edgetransportserver> -ExternalDNSServers <external DNS server> -InternalDNSServers <internal DNS server>
You can configure multiple DNS servers by entering them in comma delimited format. For example:
Set-TransportService mailgate -ExternalDNSServers, -InternalDNSServers,
Once configured (either by selecting the DNS registration checkbox or through the cmdlet) you'll need to restart the Microsoft Exchange Transport Service.
Restart-Service MSExchangeTransportService
I don't believe that the Exchange Transport Service should rely on this checkbox being set, so I've filed a bug on this to have the dependency removed or at least document it.

Read more ...

Email delivery fails after installing Exchange 2016 CU8

Thursday, December 21, 2017
Please see Hub Transport fails when NIC is not set to register in DNS for an update to this article.

After installing Exchange 2016 CU8 on my Edge Transport server in a hybrid environment, I found that emails from on-prem users could not be delivered to EOP. They would queue with the following error: 
LED=451 4.4.0 DNS query failed. The error was: DNS query failed with error ErrorRetry -> DnsQueryFailed: ErrorRetry

For troubleshooting I confirmed that name resolution works from the Edge Transport server using NSLOOKUP. I also was able to connect to the tenant SMTP namespace using Telnet on TCP 25 so I know that name resolution is working fine from the server.

Eventually I found that I had to configure the external DNS server in Set-TransportService and restart the transport service on the Edge Transport server to get emails to deliver.
Set-TransportService <edgetransportserver> -ExternalDNSServers -InternalDNSServers <internal DNS server>
When I revert the change, emails start queuing again. I've reported this as a bug to Microsoft.

Read more ...

Free Exchange Hybrid webinar is now available on-demand!

Friday, December 15, 2017
For those of you who missed my free live webinar, "Exchange Hybrid for the Long Haul - Critical Information to Know" it's now available on-demand, brought to you by Enow Software.

The webinar answered many questions including:

  • What is Hybrid Mode and what are its advantages?
  • When is it appropriate?
  • Is AAD Connect required?
  • Which authentication method is preferred?
  • How do you optimize your network?

Dozens of additional questions arise when looking at Hybrid environments. If you're currently running a hybrid environment or looking into migrating in the near future, 'Hybrid for the Long Haul' will provide answers, solutions, and countless tips and tricks accumulated over hundreds of deployments. Watch the webinar below:

Read more ...

Secure AAD Connect! New build 1.1.654.0 and AdSyncConfig.psm1 module is available

Wednesday, December 13, 2017
If you attended my webinar yesterday on "Exchange Hybrid for the Long Haul - Critical Information to Know" you learned how important it is to keep AAD Connect up-to-date.

Yesterday Microsoft released AAD Connect build 1.1.654.0 which addresses a security vulnerability that allows lower level administrators to gain elevated privileges by resetting the password of the on-prem MSOL_xxxxxx account used by AAD Connect for synchronization. To be honest, this is no more risky than allowing junior admins from resetting the password of any highly privileged account or being able to add themselves to a highly privileged group, like Domain or Enterprise Admins. You should always secure and monitor any highly privileged account or group in your organization.

The AAD Connect Version History explains the security issue that build 1.1.654.0 addresses. It also gets the EXPTA award for the most poorly written release notes ever. Spelling errors, grammar errors, syntax errors, oh my. But I digress...

Microsoft also released a PowerShell script (module, really), AdSyncConfig.psm1, which configures the new recommended permissions on the MSOL_xxxxxx account for those organizations who need to secure the account immediately, but can't upgrade to the latest version of AAD Connect at this time. The script is available on the TechNet Gallery.

The AdSyncConfig module is not quite as straight forward as written, though, so I wanted to document here how to use it. The first problem is that the module is not digitally signed. That means that you'll get the warning, "The publisher of AdSyncConfig.psm1 couldn't be verified" when you download it with Internet Explorer. Edge doesn't complain.

It also means that you will probably have to change your execution policy in PowerShell to run it. The recommended execution policy of RemoteSigned will prevent the module from being imported since it's not digitally signed.

If you can't upgrade to the latest build immediately, here are all the correct steps to use the AdSyncConfig.psm1 module to secure the MSOL_xxxxxx account immediately. There is no downtime associated with these steps.

  • Download AdSyncConfig.psm1 to your AAD Connect computer and open an elevated PowerShell console.
  • Run Get-ExecutionPolicy to view the current policy. Usually it's RemoteSigned or Restricted.
  • Run Set-ExecutionPolicy -ExecutionPolicy Unrestricted in order to import the unsigned module.
  • Run Import-Module AdSyncConfig.psm1 to import the module.
  • Run $Account = Get-ADUser -Filter 'Name -like "msol_*"' to store the account information for the MSOL_xxxxxx account.
  • Run Set-ADSyncRestrictedPermissions -ObjectDN $Account.DistinguishedName -Credential (Get-Credential) to reset the permissions on the MSOL account to the new recommended settings. Be sure to use domain\username format. 
  • Run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned to reset the execution policy.
  • Run Restart-Service adsync to restart the Microsoft Azure AD Sync service.
  • Run Start-ADSyncSyncCycle -PolicyType Delta to force a new delta sync and confirm that synchronization is working properly.

Once you install AAD Connect build 1.1.654.0 or run the steps for the module above, you will see that the MSOL_xxxxxx account has updated permissions for SELF.

Read more ...

Join me for an Exchange Hybrid webinar December 12th!

Thursday, December 7, 2017

I'll be partnering with my friends at ENow Software for a webinar, "Hybrid for the Long Haul - Critical Information to Know"

Join me and ENow Software on Tuesday, December 12th at 12:00pm PST as we discuss Hybrid deployments, their nuances, and various optimizations during our invite-only Webinar.

  • What is Hybrid Mode and what are its advantages?
  • When is it appropriate?
  • Is AAD Connect required?
  • Which authentication method is preferred?
  • How do you optimize your network?

Dozens of questions arise when looking at Hybrid environments. If you're currently running a hybrid environment or looking into migrating in the near future, 'Hybrid for the Long Haul' will provide answers, solutions, and countless tips and tricks accumulated over hundreds of deployments.

Register Here

Read more ...