Shadow Redundancy

Exchange Server 2010 introduces the shadow redundancy feature to provide redundancy for messages for the entire time they’re in transit. The solution involves a technique similar to the transport dumpster. With shadow redundancy, the deletion of a message from the transport databases is delayed until the transport server verifies that all of the next hops for that message have completed delivery. If any of the next hops fail before reporting back successful delivery, the message is resubmitted for delivery to that next hop.

Shadow redundancy provides the following benefits:

  • It eliminates the reliance on the state of any specific Hub Transport or Edge Transport server. As long as redundant message paths exist in your routing topology, any transport server becomes disposable. If a transport server fails, you can remove it from production without emptying its queues or losing messages. If you want to upgrade a Hub Transport or Edge Transport server, you can bring that server offline at any time without the risk of losing messages.
  • It eliminates the need for storage hardware redundancy for transport servers.
  • It consumes less bandwidth than creating duplicate copies of messages on multiple servers. The only additional network traffic generated with shadow redundancy is the exchange of discard status between transport servers. Discard status is the information each transport server maintains.
  • It indicates when a message is ready to be discarded from the transport database.
  • It provides resilience and simplifies recovery from a transport server failure.

Shadow redundancy is implemented by extending the SMTP service. The service extensions allow SMTP hosts to negotiate shadow redundancy support and exchange discard status for shadow messages.

However, this functionality doesn’t cover the load-balancing of messages received from non-Exchange sources, such as external mail servers, third-party anti-spam or antivirus solutions, any internal mail servers outside your Exchange organization, line-of-business (LOB) applications, and POP-based or IMAP-based e-mail clients.

Customers who have upgraded from Exchange 2007 to Exchange 2010 and use dedicated Receive connectors for the purpose of relaying messages from sources such as line-of-business (LOB).

Applications may see a significant decrease in SMTP throughput. This decrease in throughput is because of the default delayed acknowledgement time-out of 30 seconds configured for a Receive connector. To increase the SMTP throughput for the relay Receive connector, we recommend you either lower the time-out value for the delayed acknowledgement attribute or disable it completely. Whether you should lower or disable the time-out value depends on the amount of messages that go through the relay Receive connector. A good approach is to first lower the value and then verify whether SMTP throughput still suffers and, if it does, then disable the feature completely.

This example lowers the time-out value for a Receive connector named “SMTP Application relay” from 30 to 15 seconds.

Set-ReceiveConnector “SMTP Application relay” -MaxAcknowledgementDelay 00:00:15

Restart of the Exchange Transport Service.

Exchange 2010 tarpits by default. Have you tried lowering the tarpit value to see if it alleviates the issue?

A tarpit also known as Teergrube, the German word for tarpit is a service on a computer system (usually a server) that purposely delays incoming connections. The technique was developed as a defense against a computer worm, and the idea is that network abuses such as spamming or broad scanning are less effective, and therefore less attractive, if they take too long.

You may want to disable SMTP tarpitting for specific trusted/internal hosts to avoid delays in sending mail – for example, if the hosts need to send bulk mail. In such scenarios, you should create a dedicated Receive Connector for trusted/internal hosts, specify the IP addresses or ranges of those hosts in the RemoteIPRanges parameter of the Receive Connector and disable SMTP tarpitting.

Get-ReceiveConnector | Select name,tarpitinterval

Solution: Create another receive connector and allow only the IP address of the machine that runs the application to use it? Stick with anonymous SMTP sessions and disable the TarpitInterval on just that connector.

Get-ReceiveConnector | Set-ReceiveConnector -tarpitinterval 00:00

What you need to check,

Was it single recipient messages or did each message have a large number of recipients in any of the To/Cc/Bcc fields?

Exchange 2010 SP1 introduces two new throttling cmdlets: get-ThrottlingPolicyAssociation and set-ThrottlingPolicyAssociation. In Exchange 2010 RTM, in order to determine which throttling policy was associated with a given user, you would use something like:

Get-Mailbox JohnDoe | fl ThrottlingPolicy

To assign a non-default throttling policy to a user you would call

Set-Mailbox JohnDoe -ThrottlingPolicy Foo

One thing that should stand out with these two CMDlets is that policies could only be associated with mailbox accounts. Why would you ever want throttling policies associated with anything else? I’m glad you asked. There are at least two valid scenarios to consider:

Scenario 1 – A machine account: Let’s say you write a web site that uses Exchange Impersonation to call into EWS and perform actions on behalf of a user that logged onto your web site. Further assuming that your web site is configured to run as Network Service, the EWS call will come from the *machine* account (such as MyDomain/MyMachine$). When such a call is encountered by EWS, it tries to determine which throttling policy to apply to the machine account. Given that set-Mailbox is not applicable to Active Directory Computer objects, and given that a computer is not *in* any of the organizations defined in the Active Directory, the throttling framework must use the fallback policy for the computer account. Given that the fallback policy is hardcoded within the Exchange binaries, you have no control over reducing or increasing its policy values.

Scenario 2 – A cross forest contact: In cross forest scenarios, you have the option of creating a linked mailbox in the Exchange forest or a linked contact. If an account from user forest A is given Exchange Impersonation rights and needs custom throttling values defined, your only option in Exchange 2010 RTM is to use a linked mailbox so that you can assign a non-default throttling policy using the set-Mailbox cmdlets. When a cross forest account calls into Exchange via EWS, the user is authenticated via the user forest (via the trust), but the Active Directory object that Exchange uses to gather “Exchange” information is contained in the Exchange forest. If that Exchange object is a linked contact, the throttling framework must use the fallback policy to throttle the call, which as mentioned above cannot be modified since it is hardcoded.

To cover these scenarios, we added the get/set-ThrottlingPolicyAssociation cmdlets which operate on “virtual” ThrottlingPolicyAssociation objects. By virtual, I mean that there is no ThrottlingPolicyAssociation class in the Active Directory schema. The association represents the link between some “account” and its throttling policy. And what is an “account”? Well, it could be a mailbox, a computer object or a contact.

The Computer account option with Set-ThrottlingPolicyAssociation is if you had a web site using EWS that uses Exchange impersonation to call into EWS and perform actions on behalf of the user logged into your web page. The EWS calls would come from the machine account to Exchange in that case.  It is described here and unfortunately won’t work for a LOB Application without authentication;

If you can’t use SMTP authentication, then Exchange isn’t going to be able to apply a personalized throttling policy as it won’t know who you really are and will fall back to a default policy. When it comes to POP or IMAP clients sending message directly via SMTP the MaxMessageRate limit should give a transient error to the client if they go over their budget, then the client should try to connect later and send more messages.

If the emails are coming from a service running on a machine as NETWORK SERVICE (including a web server) then you can apply a throttling policy to a computer object. What’s more likely though is this is a client app run as whatever user happens to be logged on, in which scenario I don’t believe you’ll be able to throttle using Exchange. I don’t believe you can throttle by client IP address either, which is a bit annoying I have to admit.

If you do fall into the category of an app running as NETWORK SERVICE, you can create a new throttling policy and apply it to Active Directory computer objects. The value you’ll want to throttle on is probably MessageRateLimit, which is messages per minute. If more messages than the throttling policy are submitted, the app will receive a transient error. The documentation is not very specific on whether it will actually accept the message and delay delivery or temporarily reject it, but your apps really should have some mechanism to handle transient errors anyway so that shouldn’t be an issue. The commands to set this up would be New-ThrottlingPolicy -Name "Naughty Apps" -MessageRateLimit 50 and then Set-ThrottlingPolicyAssociation -Identity COMPUTERNAME$ -ThrottlingPolicy "Naughty Apps".

Do you have protocol logging enabled on the hub servers?

With the log we can verify whether it is or isnt to multiple recipients or a single recipient. Looking more closely into the SmtpReceive log We can identify whether emails were being delivered one-by-one, rather than emails with many recipients in the To/Cc/Bcc fields.

You may notice it waits for an acknowledgment before the next message is sent.  For example, Starting at 08:10:28 in the SmtpReceive log, each individual message sent logged the following:

  • 250 2.0.0 Resetting,
  • MAIL FROM:<>,
  • xxxxxDate/time,receiving message
  • 250 2.1.0 Sender OK,
  • RCPT TO:<>
  • Recipient OK,
  • DATA,
  • 354 Start mail input; end with <CRLF>.<CRLF>,
  • Tarpit for ‘0.00:00:16.765’ due to ‘DelayedAck’,Delivered”
  • 250 2.6.0 <> Queued mail for delivery,
  • RSET,

Exchange Protocol Logging

The following options are available for the protocol logs of all Send connectors and Receive connectors on the Exchange server:

  • Specify the location of the protocol log files. The default locations are:
    • Front End Transport service on Mailbox servers
      • Receive connectors   %ExchangeInstallPath%TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpReceive
      • Send connectors   %ExchangeInstallPath%TransportRoles\Logs\FrontEnd\ProtocolLog\SmtpSend
    • Transport service on Mailbox servers
      • Receive connectors   %ExchangeInstallPath%TransportRoles\Logs\Hub\ProtocolLog\SmtpReceive
      • Send connectors   %ExchangeInstallPath%TransportRoles\Logs\Hub\ProtocolLog\SmtpSend
    • Mailbox Transport Delivery service on Mailbox servers (Receive connectors)   %ExchangeInstallPath%TransportRoles\Logs\Mailbox\ProtocolLog\SmtpReceive\Delivery
    • Mailbox Transport Submission service on Mailbox servers (Send connectors)   %ExchangeInstallPath%TransportRoles\Logs\Mailbox\ProtocolLog\SmtpSend\SubmissionNoteProtocol logging for side effect messages that are submitted after messages are delivered to mailboxes occurs in %ExchangeInstallPath%TransportRoles\Logs\Mailbox\ProtocolLog\SmtpSend\Delivery. For example, a message that’s delivered to a mailbox triggers an Inbox rule that redirects the message to another recipient.
    • Transport service on Edge Transport servers
      • Receive connectors   %ExchangeInstallPath%TransportRoles\Logs\Edge\ProtocolLog\SmtpReceive
      • Send connectors   %ExchangeInstallPath%TransportRoles\Logs\Edge\ProtocolLog\SmtpSend
  • Specify a maximum size for the protocol log files. The default size is 10 megabytes (MB).
  • Specify a maximum size for the protocol log files. The default size is 250 MB.
  • Specify a maximum age for the protocol log files. The default age is 30 days.

Verify whether Protocol Logging is Enabled

Write-Host “Send Connectors:” -ForegroundColor yellow;

Get-SendConnector | Format-List Name,ProtocolLoggingLevel;

Write-Host “Receive Connectors:” -ForegroundColor yellow;

Get-ReceiveConnector | Format-List Name,TransportRole,ProtocolLoggingLevel;

Write-Host “Mailbox Transport Delivery service:” -ForegroundColor yellow;

Get-MailboxTransportService | Format-List *ProtocolLoggingLevel;

Write-Host “Front End Transport service:” -ForegroundColor yellow;

Get-FrontEndTransportService | Format-List *ProtocolLoggingLevel;

Write-Host “Transport service and Mailbox Transport Submission service:” -ForegroundColor yellow;

Get-TransportService | Format-List *ProtocolLoggingLevel

Configure Protocol Logging

Set-MailboxTransportService Mailbox01 -MailboxDeliveryConnectorProtocolLoggingLevel Verbose

Set-TransportService Mailbox01 -ReceiveProtocolLogPath “D:\Hub SMTP Receive Log” -ReceiveProtocolLogMaxFileSize 20MB -ReceiveProtocolLogMaxDirectorySize 400MB -ReceiveProtocolLogMaxAge 45.00:00:00 -SendProtocolLogPath “D:\Hub SMTP Send Log” -SendProtocolLogMaxFileSize 20MB -SendProtocolLogMaxDirectorySize 400MB -SendProtocolLogMaxAge 45.00:00:00

Message Throttling Policy Considerations

When relaying application server SMTP traffic through an Exchange transport server, there are Receive connector specific message throttling policy options that may need to be adjusted, so the overall SMTP throughput doesn’t suffer.
For example, the MessageRateLimit parameter specifies the maximum number of messages that a Receive connector accepts from a single IP address per minute. On Hub Transport servers, this parameter is set to a value of Unlimited, which means SMTP throughput won’t be affected.
But, for an Edge Transport server, it’s set to accept 600 messages per minute. Depending on the relay application server SMTP traffic in your specific environment, you may need to raise this limit.

This example raises the message rate limit for a Receive connector named “SMTP Application relay” from 600 to 2000.

Set-ReceiveConnector “SMTP Application relay” -MessageRateLimit 2000

Another Receive connector specific option that can have an impact on overall SMTP throughput from a relay application server is expressed in the value of the MessageRateSource parameter. With this parameter, you specify how the message submission rate is calculated. It can be set to None, IPAddress, User, or All. By default, the parameter is set to IPAddress, which means the message submission rate is calculated for sending hosts. If this parameter has a negative impact on SMTP throughput from your relay application servers, you should consider setting the value to None.

This example disables the MessageRateSource parameter for a Receive connector named “SMTP Application relay”.

Set-ReceiveConnector “SMTP Application relay” -MessageRateSource None

If you’re planning to use a dedicated transport server for relay application server SMTP traffic, you should also consider increasing the maximum number of connections that a Receive connector will serve at the same time from a single IP address. This is done using the MaxInboundConnectionPercentagePerSource parameter. The value for this parameter is expressed as the percentage of available remaining connections on a Receive connector. By default, the value is set to 2 percent.

This example changes the value of MaxInboundConnectionPercentagePerSource for a Receive connector named “SMTP Application relay” from 2 to 30 percent.

 Set-ReceiveConnector “SMTP Application relay” – MaxInboundConnectionPercentagePerSource 30
Advise: Use IIS SMTP instead to keep the traffic off Exchange. In my experience Exchange is not very good at handling bulk email, it is like taking a Ferrari to do the weekly food shop – you can do it, but it isn’t the appropriate tool for the job. Simon Butler, Exchange MVP
The likelihood of the Sending IP being blacklisted is higher.
An incident where the settings didn’t work
“They suggested keep running the relay off of a 2003 server……so that’s what we’re doing for now.”
Another way of analyzing the delay is taking a look at the header of the sent Mail.

Leave a Reply

Your email address will not be published. Required fields are marked *