How to Determine Continuous Replication Mode (Block Mode or File Mode)?

In Exchange 2007 and 2010, Continuous Replication operates by shipping copies of the logs created by the active database copy to the passive database copies. With Exchange 2010 SP1, this is known as Continuous Replication - File Mode as the log file is only copied once it is full (1MB). But SP1 introduces a new form of continuous replication known as Continuous Replication - Block Mode. In block mode, when an update is written to the active database log file it is immediately copied to the passive mailbox copies, thus reducing the latency between the time a change is made on the active copy and the time that same change is replicated to a passive copy. This way, if a failure occurs on the active copy, the passive copies will have been updated with most or all of the latest updates.

However, Block Mode is only active when continuous replication is up-to-date in file mode. The Log Copier component monitors the copy and replay queue lengths of databases as transaction logs are generated and takes care of transitioning into and out of block mode automatically.

To determine if continuous replication is operating in block mode or file mode, use the following cmdlet:

Get-Counter -ComputerName <<DAG_Member_Name>> -Counter “\MSExchange Replication(*)\Continuous replication - block mode Active”

The output will be something similar to:

Timestamp                 CounterSamples

---------                 --------------

04/09/2012 11:39:46       \\MBX1\\msexchange replication(mdb31)\continuous replication - block mode active : 1

                          \\ MBX1\\msexchange replication(mdb32)\continuous replication - block mode active : 1

                          \\ MBX1\\msexchange replication(mdb33)\continuous replication - block mode active : 0

Here, the “1” means that block mode is active while a “0” means it is not. However, note that your active databases will always show “0”, we are just interested in the passive copies!

Mailboxes Quarantined due to Troubleshoot-DatabaseSpace.ps1 Script

Exchange 2010 SP1 added a new script called Troubleshoot-DatabaseSpace.ps1 that is used to detect excessive growth of database and log drive volumes. This script can be run manually by administrators but if you have Microsoft System Center Operations Manager (SCOM) 2007 it is run automatically every 15 minutes.

This script performs the following actions:

1. Track log generation rate for the highest log generators (mailboxes) per database. This helps determine which users are logging too heavily and potentially causing space issues;

2. Track available disk space for both database and log files. If either of these is within a configurable threshold of being full (25% by default), further action must is taken;

3. Track log generation rate. If it appears that the disk is going to run out of space within the value specified by the HourThreshold parameter (12 hours by default and based on the log generation rate), further action is taken;

4. If all of the above conditions are fulfilled, the script determines the list of top 25 users who accessed the database during the last 1 hour period. The script then quarantines the top high-usage mailboxes for 6 hours, during which they will not have access to e-mail;

5. If the troubleshooter is unsuccessful at dropping the log generation rate to below the threshold level, it will write out events that translate into health model alerts. At this point, the script removes the database from provisioning by running the Set-MailboxDatabase cmdlet with the ExcludeFromProvisioning parameter set to $True against the specified database;

When mailboxes are quarantined, you will see entries in the Registry of the mailbox server hosting that database/mailbox in:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\"server_name"\Private-"DB.Guid"\"Mailbox_Guid"

To check the events logged by this script go to the mailbox server you want to check and then Event Viewer -> Application and Services Logs -> Microsoft-Exchange-Troubleshooters/Operational. Note that if you run the script manually through the Shell it will not produce any output - you have to check the Event Viewer.

If you run the script manually without any parameters or if you have SCOM running this script automatically, then it will use default values specified in the StoreTSConstants.ps1 script located in the same folder. This is where you can customize the 25% threshold, for example, if it is not ideal for your environment:

   # The percentage of disk space for the EDB file at which we should start quarantining users.

   $PercentEdbFreeSpaceDefaultThreshold = 25

   # The percentage of disk space for the logs at which we should start quarantining users.

   $PercentLogFreeSpaceDefaultThreshold = 25

   # The percentage of disk space for the EDB file at which we are at alert levels.

   $PercentEdbFreeSpaceAlertThreshold = 16

   # The percentage of disk space for the EDB file at which we are at critical levels.

   $PercentEdbFreeSpaceCriticalThreshold = 8

   #The number of hours we can wait before running out of space.

   $HourDefaultThreshold = 12

In order for mailboxes to be quarantined, the –Quarantine parameter must be passed to the script, which the SCOM monitor uses by default. Because the Exchange Management Pack is sealed, you can’t change this...

So, if 25% is too high for your environment, you can change the value by updating the StoreTSConstants.ps1 script across all your mailbox servers or, ultimately, simply disable the SCOM monitors that run this script:

• KHI: Failed to execute Troubleshoot-DatabaseSpace.ps1

• KHI:The database copy is low on database volume space and continues to grow. The volume has reached critical levels 8% free.

• KHI:The database copy is low on database volume space and continues to grow. The volume has reached error levels under 16% free.

• KHI:The database copy is low on database volume space and continues to grow. The volume is under 25% free.

Resubmit Messages in Queues

There might come a time where you experience a problem with you Hub/Edge Transport servers, Mailbox servers or e-mail gateway and messages are stuck in Queues. Once you resolve the issue, you can either wait a few minutes for Exchange to resubmit the e-mails or you can manually resubmit them to the Submission queue for the categorizer to reprocess as long as they have the following status:

·         Mailbox delivery queues or remote delivery queues that have the status of Retry. The messages in the queues must not be in the Suspended state;

·         Messages in the Unreachable queue that aren't in the Suspended state;

·         Messages in the poison message queue.

 

 

To manually resubmit messages, you can use the following methods (examples below):

·         Use the Retry-Queue cmdlet with the -Resubmit parameter;

·         Export the messages to .eml message files and resubmit them by placing them in the Replay directory;

·         Use Queue Viewer or the Resume-Message cmdlet to resubmit the messages in the poison message queue. You can’t use the Retry-Queue with the -Resubmit parameter cmdlet to resubmit messages in the Poison Queues.

 

 

By using the –Resubmit parameter, messages are forced to be resubmitted through the Categorizer for a new delivery attempt. If you do not use the –Resubmit parameter, the delivery queue will try to connect to the next hop immediately without resubmitting the messages through the Categorizer.

 

 

Let’s look at a few examples:

1. To export a copy of a message (so you can put it into the Replay directory) that has an InternalMessageID of 1234 that's located in the remote delivery queue for the domain MSExchange.org on the server HUB01 to the path C:\MSExchange Export\export.eml:

Export-Message HUB01\MSExchange.org\1234 -Path “C:\MSExchange Export\export.eml”

2. To resubmit all the messages in the 62306 delivery queue of the HUB01 server, use the following cmdlet (you can’t use the EMC):

Retry-Queue “HUB01\62306” -Resubmit $True

3. To resubmit all the messages in all delivery queues of server HUB01 that have a status of Retry (again, you can’t use the EMC):

Retry-Queue –Server “HUB01” -Filter {Status -eq "Retry"} -Resubmit $True

4. To resubmit all messages located in the Unreachable queue of server HUB01, use the following cmdlet (again, you can’t use the EMC):

Retry-Queue “HUB01\Unreachable” -Resubmit $True

5. Let’s force a connection attempt for all queues that are holding messages for the domain msexhange.org, have a status of Retry and are located on the server HUB01:

Retry-Queue –Server “HUB01” -Filter {NextHopDomain -eq “msexchange.org” -and Status -eq “Retry”}

 

6. To resubmit messages in the Poison Queues, you have to resume the messages. As I mentioned previously, the Poison Queue cannot be resubmitted by using the Retry-Queue cmdlet with the -Resubmit parameter.

Remember that the Poison Queue contains messages that are determined to be harmful to the Exchange system after a server failure. They may also be victims of a poorly written agent that crashed the Exchange server while it was processing them. If you're unsure of the safety of the messages in the poison message queue, you should export them to files so you can examine them.

 

Using the Exchange Management Console to resume messages in the Poison Queue:

1.       In the EMC click Toolbox;

2.       Open the Queue Viewer tool;

3.       In Queue Viewer, click the Queues tab;

4.       Click the Poison Queue and select View Messages;

5.       Select one or more messages from the list, right-click them and select Resume.

 

To use the Shell you must first determine the identity of the message to be resumed. This example will show the identity of all messages in the poison queue:

Get-Message -Queue “Poison” | Select Identity

 

Now you can use the identity to resume the message. For example, let’s resume the message with an identity value of 123:

Resume-Message 123