Wednesday, January 27, 2016

Unable to Download Offline Address Book

The other day I was troubleshooting an issue where users were not able to download the Offline Address Book (OAB). After some troubleshooting steps, I ran the Outlook AutoConfiguration Test and noticed that the URL for the OAB was not listed:

Looking at the properties of the OAB, I noticed that Global Web Distribution was disabled and Virtual Directories was empty:

So what do these two properties do exactly?
  • GlobalWebDistributionEnabled specifies whether distribution occurs to all virtual directories in the organization. If the value of the GlobalWebDistributionEnabled parameter is $True, distribution occurs to all virtual directories in the organization, and we cannot add values to the VirtualDirectories parameter;
  • VirtualDirectories specifies the array of OABVirtualDirectory objects. If this parameter is specified, Version4 of the OAB must be generated.

Because, for some strange reason, none of these properties was set, there was no virtual directory for users to download the OAB from.

Remember that in Exchange 2013:
  • OAB files are not stored locally on the CAS;
  • OAB files are generated and stored in the Organization Mailbox first and later copied to the %ExchangeInstallPath%\ClientAccess\OAB\ folder on the Mailbox server responsible for generating the OAB;
  • CAS 2013 proxies all OAB download requests to the appropriate Mailbox server (the one above);
  • Outlook receives OAB URL from Autodiscover and reaches designated CAS 2013 through OAB URL.

So, without OAB URL users simply do not know where to go to get the OAB! Depending on the environment, we now have two options. The first is to specify which virtual directory(ies) to use:

Or simply enable Global Web Distribution:

After resetting IIS (or recycling the OAB application pool), if we run the Outlook AutoConfiguration Test again we should see the OAB URL:

To further test the OAB, beside manually downloading it in Outlook, you can write down the OAB URL, paste it into the address box in Windows Internet Explorer, add /OAB.xml to the end of the path, and then press Enter. The following is an example of what the URL and response resembles:

Outlook 2016 New Sync Slider Options

With Outlook 2013 we saw the introduction of the Sync Slider, which helps limit the size of OST’s by controlling how much recent mail was synchronized locally on the device. For Exchange accounts, this was 12 months by default, with the option to go down to 1 month or up to all mail.

In Outlook 2016, Microsoft added more sync options that allow for better storage management by only retaining this much mail on the device:
• 3 days;
• 1 or 2 weeks;
• 1, 3, 6, 12 or 24 months;
• All.

As before, this setting can also be adjusted using Group Policy.

Restricting Outlook Clients to Cached Mode

Some organizations prefer all Outlook clients to connect to their Exchange servers using Cached Exchange Mode (also known as offline mode) because clients in Online Mode can generate significantly more disk I/O traffic. Besides using Group Policy to control this from the client side, in Exchange 2007, 2010, 2013 and 2016 we can force all clients to use Cached Mode at the server end by running the following cmdlet:
Get-Mailbox | Set-CASMailbox -MAPIBlockOutlookNonCachedMode $True

The Set-CASMailbox cmdlet is used to configure client access settings on a mailbox. For example, we can configure settings for ActiveSync, Outlook, Outlook on the web, POP3, and IMAP4. The MAPIBlockOutlookNonCachedMode parameters, which is available only in on-premises Exchange servers, controls access to the mailbox by using Outlook in online or offline mode. Valid values for this parameter are:
  • True: only Outlook clients that are configured to use Cached mode are allowed to access the mailbox;
  • False: the state of the Cached mode setting is not checked before Outlook clients are allowed to access the mailbox (both online and cached mode are allowed). This is the default value.

However, be careful with this setting! If there are Outlook clients currently configured in Online mode and you set this parameter to True, they will receive the following error message and will not be able to connect until they change their Outlook profile to Cached mode:

Friday, December 4, 2015

Prevent Users from Changing Photo in OWA

Using Outlook Web App (OWA, or Outlook on the Web as it is now known), users can change/update their photo using two methods:
1.    By clicking on the Display Name (Exchange 2013) or user photo (Exchange 2016) in the main OWA window and then Change:
2.    Using the options menu and then account followed by Edit information... in Exchange 2013, or “My account” section:


Some companies do not like the idea of users updating their pictures themselves. There are already many posts out there on how to prevent users from changing their photo in OWA. However, some posts only say to change the SetPhotoEnabled parameter to False for all OWA virtual directories:
Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -SetPhotoEnabled $False
If we now click on our display name or photo, the Change button is now gone:


The problem is that this method might only prevents users from clicking their photo to change their photo! It is possible that users can still change it through the options menu:

The reason why this might happen is because this setting only applies to mailboxes if they do not have an OWA Mailbox Policy applied to them! If they do, then we also need to update this policy, whether it is the Default policy or not. In this case I am updating all policies and applying the default one just to my account for testing:
Get-OWAMailboxPolicy | Set-OWAMailboxPolicy -SetPhotoEnabled $False
Set-CASMailbox nuno -OWAMailboxPolicy Default

Now the change button is finally gone:

Wednesday, December 2, 2015

Exchange Web Services Editor

Using PowerShell, Exchange administrators can develop scripts to do almost everything. However, there are occasions where a script that uses Exchange Web Services (EWS) is required, may that be for an administrative task or to develop an entire application.
I have developed a few EWS scripts but I admit I am no expert on the subject. As such, sometimes I rely on a great tool that not many people are aware of, the EWS Editor. This tool has 3 main goals:
1.      "Demonstrate the Exchange Web Services Managed API functionality and simplicity to developers through its source code;
2.      Demonstrate the Exchange Web Services SOAP traffic used to perform actions initiated through an explorer user interface;
3.      Assist non-developers in debugging and understanding Exchange stores by exploring items, folders, and their properties in depth."
Each release of EWS Editor includes the distribution of the EWS Managed API it was built for. EWSEditor uses .NET Framework version 4.5 which can be downloaded here: .NET Framework 4.5. The latest version of EWS Editor (from May 2015) uses .EWS Managed API 2.2, which requires a minimal of .NET 3.5. EWS Editor can be downloaded from CodePlex.
EWS Editor is, in some ways, similar to MFCMapi, another great tool. It does not require installation and unlike MFCMapi it does not require Outlook profiles in order to access mailbox items as it does everything over EWS.
We can check mailbox items and their properties:

We can look into particular folders, its emails, attachments, properties (notice how all the property names are easy to read), etc.:

We can test and get information regarding AutoDiscover:

The tool comes with a multitude of EWS Posts examples that we can use to see how they are built or even post them against Exchange and analyze the response:

We can also check and/or set Out-Of-Office messages for users (assuming we have the right permissions to do so):

It even comes with a handy Distribution Group expansion tool that allows us to see exactly who is a member of a particular group:
Another great tool is the Debug Log Viewer where we can see all the EWS requests and responses for all the actions we do using this tool (in this case we can see the response from the Messaging group expansion above):

There are many, many other things we can use this tool for:

Sunday, November 29, 2015

Identifying IIS Worker Process in Exchange 2013/2016

It might come a time when troubleshooting performance issues where we need to identify exactly what a particular IIS Worker Process is in an Exchange 2013 environment:

First thing we need to do is get the Process ID (or PID). To do this, right-click on one of the columns and select PID:

We can now see that that PID for the IIS Worker Process we are trying to identify is 10300:

Now open a Command line or PowerShell console, navigate to Windows\System32\Inetsrv and then run the command “appcmd list wp” or “.\appcmd list wp” if you are using PowerShell:

We now know that the IIS Worker Process with the PID of 10300 is the Exchange OWA Application Pool.

Tuesday, October 27, 2015

Message Tracking Logs Missing

The other day someone asked me for help regarding Message Tracking Logs missing. The problem was that the person had the message tracking logs configured to save 60 days’ worth of data but he could not find a particular email that was supposedly delivered before those 60 days.
The first step was to verify that 60 days of logs were actually being saved:
Get-TransportService | Select MessageTrackingLogMax*
MessageTrackingLogMaxAge:           60 Days
MessageTrackingLogMaxDirectorySize: 1000 MB
MessageTrackingLogMaxFileSize:      10MB

Going to the location where the logs are saved (by default %ExchangeInstallPath%TransportRoles\Logs\MessageTracking), there were indeed 60 days’ worth of logs, so why couldn’t we find the email?! Also, the overall folder size was over 1GB in size... But how can that be if we specified to only use 1000MB?! Let us go back a bit first...
The naming convention for message tracking log files in Exchange 2013 is MSGTRKyyyymmdd-nnnn.log, MSGTRKMAyyyymmdd-nnnn.log, MSGTRKMDyyyymmdd-nnnn.log and MSGTRKMSyyyymmdd-nnnn.log . The different logs are used by the following services:
  • MSGTRK: these logs are associated with the Transport service;
  • MSGTRKMA: these logs are associated with the approvals and rejections used by moderated transport;
  • MSGTRKMD: these logs are associated with messages delivered to mailboxes by the Mailbox Transport Delivery service;
  • MSGTRKMS: these logs are associated with messages sent from mailboxes by the Mailbox Transport Submission service.
The placeholders in the log file names represent the following information:
  • The placeholder yyyymmdd is the coordinated universal time (UTC) date on which the log file was created. Yyyy = year, mm = month and dd = day;
  • The placeholder nnnn is an instance number that starts at the value of 1 daily for each message tracking log file name prefix.
Information is written to each log file until the file size reaches its maximum specified value (MessageTrackingLogMaxFileSize) for each log file. Then, a new log file that has an incremented instance number is opened. This process is repeated throughout the day. The log file rotation functionality deletes the oldest log files when either of the following conditions is true:
  • A log file reaches its maximum specified age;
  • The message tracking log directory reaches its maximum specified size.
Now here comes the important part! The maximum size of the message tracking log directory is calculated as the total size of all log files that have the same name prefix. Other files that do not follow the name prefix convention are not counted in the total directory size calculation.
On Exchange 2013 Mailbox servers, the maximum size of the message tracking log directory is three times the specified value. Although the message tracking log files that are generated by the four different services have four different name prefixes, the amount and frequency of data written to the MSGTRKMA log files is negligible compared to the three other log file prefixes.
Going back to the initial issue, the problem was that there were less than 60 days’ worth of MSGTRKMD logs as the combined log files size has met the 1000 MB limit...
So, the bottom line is that, as with Exchange 2010, if you want to keep X amount of days of message tracking logs, ensure you set MessageTrackingLogMaxDirectorySize to a high enough value.

How to Bypass the Clutter in Exchange Online

Clutter is a feature in Office 365 designed to help users focus on the most important messages in their Inbox by moving lower priority messages out of their way and into a new Clutter folder. Exchange Online keeps track of the emails users read and the ones they don't read. Once it is turned it on, Clutter is automatic. As new email comes in, it takes messages users are most likely to ignore and puts them into the Clutter folder. The more users use it, the better it gets.

For organizations, it is important to ensure that certain messages do not end up in users’ Clutter folders. These could be messages from the CEO, from the HR or Finance departments, from the ServiceDesk, etc. In order to ensure this is the case, we need to create a transport rule that makes sure that these messages bypass the clutter folder.

We can do this by using PowerShell to create a rule similar to the following for example:
New-TransportRule -Name “Bypass Clutter” –From -SetHeaderName “X-MS-Exchange-Organization-BypassClutter” -SetHeaderValue “true”

Please note that both “X-MS-Exchange-Organization-BypassClutter” and “true” are case sensitive!

If we prefer to use the UI, the transport rule will look like this:

Office 365 Support and Recovery Assistance tool

If you are experiencing problems with Outlook or are having trouble installing Office apps, Microsoft's new Office 365 Support and Recovery Assistant tool (aka SaRA) can help diagnose and fix many common user or client side issues.

The tool performs a series of diagnostics tests to help identify the root cause of issues, such as verifying users’ credentials, licenses, updates to Outlook clients, and whether Outlook servers are reachable. Depending on the test results, it can offer to automatically fix problems for users or provide instruction on recommended solutions. All the diagnostics results are saved in a log file for users to share with their Outlook admin or support engineers for further investigation.
Each time we run SaRa, it automatically gets updated to its latest version, so it can troubleshoot any new Outlook problems.

SaRA, still in the pre-release (beta) stage, can be downloaded from this link:

Friday, September 25, 2015

How to determine which store worker process is responsible for which mailbox database?

As we all know by now, the Exchange Store service in Exchange 2013 has been rewritten in such a way that each database now runs under its own process, thus preventing store issues to affect all databases in the server. Managed Store is the new name for the rewritten Information Store process (store.exe). It is now written in C#, designed to enable a more granular management of resources (additional I/O reduction, for example) and is even more integrated with the Exchange Replication service (MSExchangeRepl.exe), in order to provide a higher level of availability.

The database engine continues to be ESE, but the mailbox database schema itself has changed in order to provide many optimizations.

The Managed Store is composed of two processes. The first one is the Store Worker Process (Microsoft.Exchange.Store.Worker.exe) that is similar to the old store.exe process. The difference is, as already mentioned, that there is one Store Worker Process for each database. This means that if one of these processes fails, only the database it is responsible for will be affected, while all the other databases will remain operational.

The second one is the Store Service Process (Microsoft.Exchange.Store.Service.exe) that controls all store worker processes. For example, when a database is mounted, the store service process will start a new store worker process for that particular database. On the other hand, when a database is dismounted, it will terminate the store worker process responsible for that database.

The question that sometimes arises is “how do we determine which store worker process is responsible for which mailbox database?” To show how to do this, I am going to use a test server where I have 2 mailbox databases, and therefore two Microsoft.Exchange.Store.Worker.exe):

First thing we need to do is get the Process ID (or PID). To do this, right-click on one of the columns and select PID:
We can now see that that PID for the store worker we are trying to identify is 3308:
Now open an Exchange Management Shell console and run the following cmdlet:
Get-MailboxDatabase -Status | Sort Name | FT Name, WorkerProcessID

We now know that the store worker process with the PID of 3308 is responsible for the mailbox database DB01.

Thursday, September 24, 2015

How to Generate Exchange OAB Multiple Times a Day

The generation of the Offline Address Book (OAB) back in Exchange 2010 was based on a schedule set on the OAB’s properties. In Exchange 2013 (at least in CU9 at the time of writing this) and in Exchange 2016 "beta" we still see this property but it is no longer used:
Instead, Exchange Server 2013 OAB Generation takes place according to OABGeneratorWorkCycle and OABGeneratorWorkCycleCheckpoint properties configured at the Mailbox Server. According to TechNet:
  • The OABGeneratorWorkCycle parameter specifies the time span in which the OAB generation on the Mailbox server will be processed. The default value is 8 hours;
  • The OABGeneratorWorkCycleCheckpoint parameter specifies the time span at which to run OAB generation on the Mailbox server. The default value is 1 hour.
Maybe it is because English is not my first language, but it took me a while to understand exactly what the above means... Eventually I got to the conclusion that what the above text is trying to tell us is that, by default, Exchange updates the OAB every 8 hours. If I look at my environment, I can see this is indeed the case:
For organizations where recipients change at a considerable rate, this might not be ideal. Let us say that we want to want to update the OAB every 2 hours instead. To achieve this, we simply run the following cmdlet:
Set-MailboxServer “server” -OABGeneratorWorkCycle 00.02:00:00 -OABGeneratorWorkCycleCheckpoint 00:30:00
After a few hours, we can check the Application log and look for Event ID 17002 to ensure the OAB is actually being generated every two hours (Event ID 17001 shows when the generation started while 17002 when it completed):
Don’t forget that to set these properties across all your Mailbox servers that might be generating the OAB. The server hosting the arbitration mailbox with Persisted Capability “OrganizationCapabilityOABGen” is the one responsible for the OAB generation. For a non-DAG environment, you can use the following cmdlet to identify the OAB Generation server:
Get-Mailbox -Arbitration | Where {$_.PersistedCapabilities -match "oab"} | FT Name, ServerName

For a DAG environment, use the following one instead:
Get-Mailbox -Arbitration | Where {$_.PersistedCapabilities -match "oab"} | FT Name, Database

And then check which server currently has that database mounted:
Get-MailboxDatabaseCopyStatus “database”

Friday, August 28, 2015

Unable to see multiple Organizational Units in Exchange 2013 EAC

On a new Exchange 2013 CU8 environment that I am currently working on, we experienced an issue with the Exchange Admin Center (EAC): when creating a new mailbox for a new user and trying to select the OU where the user should be create, we were unable to see some OUs.
It turns out that, by default, the EAC only displays 500 OUs to optimize EAC’s performance. However, in this environment there are over 900...
The solution was to edit the web.config file for the ECP. On a CAS server, this is located by default at C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\ecp, while on a Mailbox or multi-role server it is at
C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\ecp.
Once you locate the file, open it in Notepad with Admin rights, locate the <appsettings> section and add the following key:
<add key="GetListDefaultResultSize" value="1000">

Obviously, if you have more than 1000 OUs, simply increase the value.

When this is done, go to IIS and recycle the MSExchangeECPAppPool application pool. Reopen the EAC and you will now be able to see all OUs in your organization. Just do not forget to change this setting across all your servers otherwise the behaviour might be inconsistent.

Thursday, July 16, 2015

Fatal error TooManyMissingItemsPermanentException has occurred

During a recent transition from Exchange 2010 to 2013, a user’s mailbox failed to get migrated:

[PS] C:\>Get-MoveRequestStatistics “nuno mota”

DisplayName          StatusDetail
-----------          ------------
Nuno Mota            FailedOther


Ok, “FailedOther” does not provide much information... Let us get a report of the actual move request to try to find out exactly what happened:

[PS] C:\>Get-MoveRequestStatistics “nuno mota” -IncludeReport | FL


7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"Tania   resume", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"Netting Off entry 2011 March  28", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"Data for weekly report -  11/March/2011", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"FW: 2011 JAN ANALYSIS  revised based on ultimate parent name", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"PAYMENT RECONCILIATION  as on 16/2/2011", Folder:"Outbox"
7/13/2015 10:40:23 AM [server1] A missing item was encountered: Missing Item (IPM.Note.EnterpriseVault.Shortcut) Subject:"PORTFOLIO CAL  ACC _EOP  LIST CALCULATION   as on 07/01/2011", Folder:"Outbox"
7/13/2015 10:40:24 AM [server1] Mailbox contents verification complete: 139 folders, 254510 items, 2.829 GB (3,037,189,613 bytes).
7/13/2015 10:40:25 AM [server1] Stage: FinalIncrementalSync. Percent complete: 95.
7/13/2015 10:40:25 AM [server1] Fatal error TooManyMissingItemsPermanentException has occurred.
Ok, so from the output above we can have a clearer view of what is happening. At the end of the report we see a TooManyMissingItemsPermanentException error is preventing Exchange from completing the move request, and the reason for that is all the EnterpriveVault stubs that seem to be missing from the mailbox’s Outbox folder.
When we create a mailbox move request, or batch, we can use the BadItemLimit parameter to specify the maximum number of bad items that are allowed before the request fails. A bad item is a corrupt item in the source mailbox that cannot be copied to the target mailbox. However, also included in the bad item limit are missing items. Missing items are items in the source mailbox that cannot be found in the target mailbox when the request is ready to complete.
So, if we are comfortable ignoring this error and possibly leaving a few bad items behind, we can either increase the BadItemLimit by running:

Set-MoveRequest “nuno mota” –BadItemLimit 50
Resume-MoveRequest “nuno mota”

Valid input for BadItemLimit is an integer or the value unlimited. The default value is 0, which means the request will fail if any bad items are detected. If you set this value to 51 or higher, you also have to use the AcceptLargeDataLoss switch, otherwise the cmdlet will fail.

If too many bad items are detected, consider using the New-MailboxRepairRequest cmdlet to attempt to fix corrupted items in the source mailbox, and try the request again.

Alternatively, we can update the ContentVerificationMissingItemThreshold setting in the MsExchangeMailboxReplication.exe.config file which in Exchange 2013 is located by default at C:\Program Files\Microsoft\Exchange Server\V15\Bin.

If you opt for changing this setting, you have to restart the restart Microsoft Exchange Mailbox Replication service and then resume the move request as above.

Monday, July 13, 2015

Exchange ActiveSync v16 is coming!

The last major version of Exchange ActiveSync (EAS) was v14 which came as part of Exchange 2010 (EAS v14.1 came with Exchange 2010 SP1). Almost 6 years after it is now time for an update to EAS and the release of a new version, v16.
At the time of writing this tip, it is unclear if EAS v16 will be part of Exchange 2016... Like with other features, Microsoft will be releasing it first to Office 365 and then eventually enable it in the on-premises Exchange 2016 builds...
This new version of EAS is planned to contain three new major capabilities:
  1. Improved calendar reliability by reworking the calendar workflow. This will most likely go unnoticed by most end users but will, hopefully, help with those appointment nightmares some Exchange admins experience with 3rd-party mail clients;
  2. Calendar attachments. While currently calendar items synchronized with EAS cannot include attachments such as agendas, presentations or spreadsheets, in version 16 these will also sync;
  3. The drafts folder cannot be synced with the current version of EAS, but it will be in EAS v16. This means we can start composing an email on our EAS device and continue editing it on our desktop, for example, or vice-versa.
To check which ActiveSync protocol versions your Exchange Online mailbox supports, go to and run the Exchange ActiveSync test from the Office 365 tab. From the test’s output, look for MS-ASProtocolVersions:
Like with previous EAS version changes, v16 will require the client to support it. It seems that iOS 9 will at least support the calendar features when it is released this fall.
As always, please note that the information provided here might change when EAS v16 is released.