Tips N Tricks Videos
newest 100 tips n tricks videos / tips n tricks widgets / media rss:

We have released the final version of Update Rollup 4 for Exchange Server 2007 SP1 (KB952580) The update is live at: http://www.microsoft.com/downloads/details.aspx?FamilyId=8B492ED2-EA92-412F-A852-3AA1C58D9499 v2 in the file name i.e. Exchange2007-KB952580-v2-x64-EN.msp to account for the accidental release of v1 via Microsoft Update on 9th September. This version of rollup will install over the v1 mentioned above if you have installed a v1 when it was accidentally released. The rollup is available via the download center link starting today. It will be available via Microsoft Update starting with 10/14/2008. Installation notes There are a few possible installation issues that we would like you to be aware of: Exchange 2007 managed services might time out during certificate revocation checks When installing a Rollup, we recommend you use the same account that you used to install Exchange Server. If you are using a different account, that account needs to have Local Administrator rights as well as rights to read Active Directory on Exchange object as well as server level (as the update needs to determine which roles are installed on the server). Not having required permissions can lead to OWA not being updated correctly and displaying a blank page after update has completed. If you have modified the logon.aspx file, it will not be patched by the Update Rollup installer. As a result Outlook Web Access may not be updated correctly and it may display a blank page after the update has finished. In order to avoid this problem, rename the logon.aspx file before applying the update rollup. After you apply an update rollup package, you must re-create Outlook Web Access customization in logon.aspx. - Nino Bilic Share this post :

As mentioned in my previous post on the subject of Autodiscover Service and Outlook Providers - I wanted to follow-up with the discussion of modifying the providers. What is the impact when we change the Outlook provider configuration? In what scenarios the Outlook provider should be modified? Relatively few changes will need to be made in the Outlook Providers. Please also note that depending on the modification that was made, Autodiscover might stop working or prevent the client to connect to the Exchange server, so this should not be done lightly! The cmdlet Set-OutlookProvider allows modifying related settings. As we can see in the table below, the parameters Server and CertPrincName only apply to Outlook EXPR provider - Outlook Anywhere clients. By default both values are set to $null. Parameter Optional Description Type Identity N If RPC is used (Outlook Anywhere not selected in GUI), protocol is EXCH. If RPC/HTTP is selected, protocol is EXPR. String Server Y The value here specifies the name of the mail server to use for RPC/HTTP. String CertPrincName Y This value is only used for EXPR types. It specifies the SSL certificate principal name required when connecting externally from the Exchange topology and using SSL. For example, if SERVER were specified as "fourthcoffee.com" and CERTPRINCNAME were left blank the default value of CERTPRINCNAME would be "msstd:fourthcoffee.com". String TTL Y The value here specifies the time to live in hours that these settings are valid for. After that time has elapsed (from the time the settings were retrieved), the settings should be rediscovered via Autodiscover again. A value of 0 indicates that no rediscovery will be required. If no value is specified, the default will be a TTL of 1 hour. Integer We will consider a few scenarios and the impact if you the change Outlook provider configuration: Scenario 1: Multiple AD sites where both CAS servers are Internet-Facing and Outlook Anywhere is enabled. Each Client Access Server has its own certificate installed. The User1 mailbox is located on a Mailbox server on AD site 1, and User2 mailbox on a Mailbox server on AD site 2. Both Outlook clients are on the Internet, thus they will connect through Outlook Anywhere. Note: Autodiscover is configured properly on the Internet as Autodiscover.fourthcoffee.com. When the User1 connect to Autodiscover.fourthcoffee.com server, the Autodiscover service will identify the request comes from an Outlook client and then will return both InternalURLs and ExternalURLs. In this scenario we will explain the importance of not changing Outlook providers. As the parameters Server and CertPrincName are $null. The Service Discovery will return to the client the best CAS for Outlook Anywhere, in this case mail1.fourthcoffee.com. The same behavior will happen when the User2 connects to Autodiscover.fourthcoffee.com. The Service Discovery will return to the client the best CAS for Outlook Anywhere, in this case mail2.fourthcoffee.com. As the parameters Server and CertPrincName are set to $null, they will be populated with the same value as ExternalHostName. Remember that the Outlook provider is a global setting in Active Directory. What would happen if you have modified the parameter Server to mail1.fourthcoffee.com? Set-OutlookProvider EXPR -Server mail1.fourthcoffee.com This setting will force all Outlook Anywhere clients, User1 and User2 to connect to the same CAS server mail1.fourthcoffee.com no matter where the user mailbox is located, preventing the Service Discovery to provide the best CAS. Another issue could result if you decided to change Outlook Anywhere ExternalHostName to Outlookanywhere.fourthcoffee.com. The setting on the EXPR Outlook provider set to mail1.fourthcoffee.com will prevent Outlook Anywhere to connect since the mail1.fourthcoffee.com is not longer available. Scenario 2: Consider the same scenario as the above, however a wildcard certificate was deployed across the Client Access Servers - *.fourthcoffee.com. No change was made to the ExternalUrls and Outlook Anywhere ExternalHostName is set to mail1.fourthcoffee.com. As the parameters Server and CertPrincName are $null. The Service Discovery will return to the client the best CAS for Outlook Anywhere, in this case mail1.fourthcoffee.com, and will configure the Certificate Principal Name to msstd:mail1.fourthcoffee.com. Given that the Certificate Principal Name setting does not match to the wild certificate installed on the CAS, it is required to modify the parameter CertPrincName. Set-OutlookProvider EXPR -CertPrincName *.fouthcoffee.com With this new setting the Service will always return to the Outlook client the CertPrincName set in the EXPR provider. See: When Outlook Anywhere clients connect to Exchange 2007 and a wildcard certificated are deployed across Exchange Client Access servers. Scenario 3: Once the Outlook 2007 client has successfully created a profile, it will update by default every hour according to the parameter TTL set. This configuration can be modified. Set-OutlookProvider -Identity msExchAutoDiscoverConfig -TTL 2 See: Duration that the auto-discovery settings are valid for the Outlook Provider. I hope you have found this useful! - Vandy Rodrigues Share this post :

For a brief period of time on 9/9 (Tuesday), a pre-release version of Update Rollup 4 for Exchange Server 2007 Service Pack 1 (KB952580) was inadvertently made available to Microsoft Update, the Microsoft Update Catalog, and WSUS servers for download. While we quickly identified and removed the update from Microsoft Update within a short period of time, some servers using these distribution methods might have detected, downloaded and/or installed this version of the update. An issue exists with this pre-release version of the Rollup 4 with regard to the Exchange Web Service (EWS) that creates the potential for a continuous crashing cycle. The final version of this update will be released shortly and we recommend customers who have not already installed the update to wait to do so. For those who have already installed the update, we recommend uninstalling this version. Once a new the final version is made available, all Exchange 2007 SP1 Installations with pre-release version of Rollup 4 installed will be offered the final version which will install over the pre-release. We apologize for any inconvenience and are working to make sure this does not happen again. While at this time we are not aware of many customers being affected by this, we wanted to let you know in case you are currently testing what you downloaded 2 days ago. Identification of Rollup version via Add/Remove Programs: To identify the version of the Rollup that you have installed, press the Click here for support information link on the rollup, under your Add or Remove Programs applet in Control Panel: Identification of the rollup version via the Registry: To identify if the rollup is installed via the registry, the DisplayVersion value shown below will be 1. The newer version of the rollup to be released will consist of a 2 or higher. HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersionUninstallKB952954 Name: DisplayVersion Type: REG_SZ Data: 1 Identification of crashes caused by this problem: If you are experiencing crashes caused by this problem, you will see the following event being logged on your server: Log Name: Application Source: MSExchange Common Event ID: 4999 Task Category: General Level: Error Keywords: Classic Description: Watson report about to be sent to dw20.exe for process id: 9364, with parameters: E12IIS, RTL-AMD64, 08.01.0240.006, WS, M.E.Services, M.E.S.C.PerfCounterReader.GetCPUPercent, System.InvalidOperationException, d177, 08.01.0311.001. ErrorReportingEnabled: True Event Xml: Event xmlns= http://schemas.microsoft.com/win/2004/08/events/event System Provider Name= MSExchange Common / EventID Qualifiers= 16388 4999 /EventID Level 2 /Level Task 1 /Task Keywords 0x80000000000000 /Keywords TimeCreated SystemTime= 2008-09-11T00:49:11.000Z / EventRecordID 1262 /EventRecordID Channel Application /Channel Computer test.contoso.com /Computer Security / /System EventData Data 9364 /Data Data E12IIS /Data Data RTL-AMD64 /Data Data 08.01.0240.006 /Data Data WS /Data Data M.E.Services /Data Data M.E.S.C.PerfCounterReader.GetCPUPercent /Data Data System.InvalidOperationException /Data Data d177 /Data Data 08.01.0311.001 /Data Data True /Data /EventData /Event - Scott Roberts Share this post :

We've recently encountered a small handful of customers who reported mailflow problems shortly after running Exchange 2007's /prepareAD, or after they installed their first Exchange 2007 role (which auto-launches the pre-requisite prepareAD process). Through determining root cause, we found that this problem might affect a larger set of our customers that have (or have ever attempted) e-mail domain name sharing and use the "forward all mail with unresolved recipients to host" option. What happens? Some of the Exchange 2000/2003 mailflow problem symptoms include: - Messages eventually accumulate in deferred delivery queues - mostly on bridgeheads. - In some cases, message tracking shows some messages routing back and forth a small number of times between the same Bridgeheads and mailbox servers. Why does this happen? The e-mail domain name that users primarily use was ambiguously nonauthoritative. /PrepareAD does not expect this condition when enumerating recipient policies, and attempts to "fix" the Exchange 2000/2003 mis-configuration by making e-mail domain(s) consistently non-authoritative on recipient policies. A few other things occur behind the scenes, but eventually mail will queue-up. If you only have a handful of recipient policies, here's how to determine whether or not you are at risk of a mailflow outage: First, navigate in Exchange 2000/2003's system manager, and pull-up the recipient policies. Here is a screenshot of the recipient policies in the Contoso lab organization. Since we know that "contoso.com" is present on multiple recipient policies, we need to check each occurrence for the authoritative setting. In the site01 policy, we can see that there is an occurrence of contoso.com. When you double-click (or edit) the contoso.com entry, the popup-dialog box on the right indicates that it is authoritative. I've also included an ldifde output to illustrate how the GUI maps to the raw data from Active Directory. In the site08 policy, we can see that contoso.com doesn't have the checkbox, and thus is nonauthoritative. Since this setting differs from the instance of contoso.com within the site01 policy, prepareAD would detect this mis-configuration, and subsequently converts any authoritative occurrences of contoso.com to non-authoritative. The recipient policy change(s) would eventually cause all Exchange servers in the organization to remove contoso.com from their own metabases, and transport on such servers are in an "in-between" state until IIS is restarted. Notice that the screenshots show recipient policies which are appropriately-named for this blog. However, in real-world environments, they are rarely named descriptively. Another challenge is that not all of the policy objects (such as pure mailbox-manager policies) will have visible e-mail settings, in which case you would need to temporarily enable the "e-mail addresses" property sheet to expose the addresses written to those policies when they were created (which could have been years ago). Lastly, an admin would be prone to error if he/she were to check hundreds of policies in a large environment. Therefore, Marc Nivens took our requirements and wrote a read-only script to assist you in detecting ambiguously authoritative e-mail domain names. Seeing that you might not have any Exchange 2007 servers in the org yet, all you need is a machine that has Powershell 1.0 installed, and you can copy the script to run it. (Do note that you will need to change your execution policy. If you don't feel comfortable changing your execution policy, you can always author your own script by creating a .ps1 file with notepad and paste-in the contents of our code.) After invoking the script in this lab environment, the contoso.com domain (among others) was found to be ambiguously authoritative: What do you do? If you have not yet executed prepareAD: As you've seen above, the corrective action is to check to see if there's an e-mail domain name that exists on multiple recipient policies. If so, double-click on each occurrence of that domain to make sure the checkbox for "This organization is responsible..." is either consistently checked or unchecked across all occurrences of that domain. If you intended for all occurrences of the e-mail domain name to be nonauthoritative, you need to uncheck the checkbox from all occurrences of that domain. On the other hand, if you intended for all occurrences of the e-mail domain name to be authoritative, you need to check the checkbox from all occurrences of that domain. Any changes to the "This organization is responsible..." checkbox (i.e. msexchnonauthoritativedomains) do not affect address generation, and thus will not produce the pop-up dialog requesting to update all recipient e-mail addresses. You can use the attached PowerShell script for detection, but the corrective steps are manual. The PowerShell script may be used again to double-check your corrections after AD has replicated. If you have already executed prepareAD and are experiencing a mailflow issue with one of your e-mail domain names: Restart IIS on all servers in the Exchange organization. This is because the metabase on all servers exists in an in-between state that doesn't fully recognize the route change. In some cases, restarting IIS will alleviate the mailflow outage, but it doesn't necessarily resolve the issue if you truly intended for the domain to be authoritative. Additionally, Exchange System Manager will not accurately reflect the nonauthoritative state of the e-mail domain names. This is due to /prepareAD having populated a lowercase value of "smtp" into msexchnonauthoritativedomains. If the gatewayproxy attribute has the uppercase "SMTP" then ESM doesn't see the case match, and will misleadingly render the domain as "authoritative" in the GUI. The chart below illustrates the before-and-after changes: Policy name msexchnonauthoritativedomains ESM GUI "This Exchange organization is responsible..." Default policy (authoritative for contoso.com) not set checkmark (grayed-out) Site08 policy (nonauthoritative for contoso.com) smtp:@contoso.com no checkmark Site01 policy (authoritative for contoso.com) not set checkmark Additionally, the metabase shows a domain key for contoso.com with routeaction=32. During /PrepareAD execution, the recipient policies are changed at the same time the following entries are logged to the file ExchangeSetup.log: [9/2/2008 12:24:32 PM] [2] [WARNING] The SMTP address template 'SMTP:@contoso.com' is invalid because it references a domain that is not an accepted domain. [9/2/2008 12:24:32 PM] [2] [WARNING] Found 'contoso.com' in NonAuthoritativeDomains but did not expect it. Run Set-EmailAddressPolicy to correct this. After SP1 /prepareAD succeeds, the changes (in red) are apparent: Policy name msexchnonauthoritativedomains ESM GUI "This Exchange organization is responsible..." Default policy smtp:@contoso.com checkmark (grayed-out) Site08 policy smtp:@contoso.com checkmark Site01 policy smtp:@contoso.com checkmark At this point, if you dump the IIS metabase on any Exchange server in the organization, you will no longer see the local route; contoso.com actually gets removed from the metabases. And since gatewayproxy is capitalized whilst msexchnonauthoritativedomains is not, the ESM UI will mis-lead administrators to believe the e-mail domain names are authoritative. Long-term solution if you have already run /PrepareAD: You will need to edit your recipient policies to re-write the authoritative/nonauthoritative status for each occurrence of that shared e-mail domain name, and restart IIS on all servers after AD replication has completed. If you wish to make the domain authoritative, you will need to uncheck the checkbox next to "This organization is responsible...", hit apply, and re-check it, and hit "apply" to correct any lowercasing issues in the raw data to force ESM to reflect an accurate checkbox. For reference, article 321721 mentions two methods for namespace sharing and recommends against configuring SMTP virtual servers having been configured with "forward all unresolved mail to this host". What if I already ran /prepareAD and didn't notice a problem? Check your ExchangeSetup.log on the machine where you ran /prepareAD. If you skipped /prepareAD, you need to open the exchangesetup.log file on your first Exchange 2007 server role). Search for the existence of "in NonAuthoritativeDomains but did not expect it" (without quotes) and make a note of the referenced e-mail domain(s) preceding occurrences of that text string. As long as you're aware of the ESM GUI display issue described above, you can ignore this entire post if: - The referenced e-mail domain name isn't used, or planned to be used. - The referenced e-mail domain name is used, but you intend for it to be an internal relay domain. (/prepareAD will have copied that domain name to the list of accepted domains as an internal relay domain) - You have already migrated resources from Exchange 2000/2003 to E12. - You do not find the text string. In conclusion, if an e-mail domain name is intended to be non-authoritative, then make sure it's non-authoritative on ALL recipient policies (even mailbox-manager-only policies). The opposite also holds true: If an e-mail domain name is intended to be authoritative, make sure it's consistently authoritative on all policies. So any e-mail domain name that is ambiguously non-authoritative (i.e. authoritative on one policy and non-authoritative on the other) will cause prepareAD to modify the authoritative setting (msexchnonauthoritativedomains) on recipient policies. We are trying to work on a BPA pre-requisite block setup, as well as a normal (healthcheck) BPA rule to detect this setting in an Exchange 2000/2003 environment since the ambiguously authoritative domains have also caused routing issues outside of any Exchange 2007 operations. Hope this has been helpful. We are working on a better solution for this problem but in the mean time, you can use the script to help you identify this situation. You can download the script from the following link. Note: this script is not officially supported by Microsoft. Please see the script for more details. File: chkpolicyconflict - Vincent Yim Share this post :

Many companies and users consider mobile access to Exchange data an essential feature. Exchange ActiveSync (EAS) is very popular as it allows this access and many devices have licensed and implemented EAS (including Windows Mobile). Some companies use remote servers to access Exchange data and push it out to their mobile clients that aren't EAS enabled. Of course these mobile access options can be a little concerning when you think about the security implications. Evaluating these devices (and servers) to make sure they comply with data protection policies is a necessary step for a lot of companies that want to protect their messaging data. This post will details some of the options available to companies that want to limit access to a specific set of supported devices. The first question we usually get is, "How do you stop unapproved devices and servers from accessing Exchange data?" In general, I hear people talk about three ways to block devices: Use an ISAPI filter (Not recommended) Set policies that only the devices you care about can implement (Better) Block the devices at the firewall (Recommended!) Let me describe each method in a bit more detail. Custom ISAPI filter:Since creating a custom ISAPI filter is both time consuming (you have to write custom code) and not a best practice, I'm not going to talk too much about it except mentioning that it is a possible solution. More details can be found here for those interested in exploring this option. Policies as a blocking agent:Using policies is a very easy way to do this (by unchecking the box titled "Allow non-provisionable devices" (image below) and then setting a policy that the particular device doesn't support. A list of which policies are enabled with which version of the server (and thus which generation of clients) is listed below. You may need to test a device to see which version of the policies are implemented as it varies by licensee). The challenge with this method is that many of today's devices are upgradable and thus may implement a policy in the future while still not providing the security you want (for example, if you want to make sure the device and storage memory are encrypted by at least 128-bit encryption (WM 6+ uses AES 128-bit encryption for storage card encryption but another device might simply do 40-bit encryption). Because of this discrepancy, it is important to understand what devices are connecting to your network, and which ones can connect, so that you can decide if your organization is ok with this level of control. Using ISA Server 2006 to block unwanted clients: Finally, you can block at the firewall. This is by far the best solution and is really easy to implement with ISA Server. Below, I list out, and describe, the steps of how to configure your ISA server so that you can block by device type (using the User-Agent string of the device to identify and block it), and by server type (using the IP address range of the server). Blocking by User-Agent String in ISA Server 2006: Blocking a particular device from accessing EAS is easy to do if you filter by the device's User-Agent string at the firewall. You can create a rule for each device type you want to block. In ISA Server you should already have an EAS rule set up (if not follow the wizard that the "Publish Exchange Web Client Access" takes you through). Simply right click on that rule (pictured below) and then select "Configure HTTP". From here, you'll get the below dialog to appear. Make sure you select the last tab named "Signatures". When you hit OK you will go back to the main ISA Server screen but at the top it will ask you if you want to apply or discard the changes you made. (see below) When you accept the change you are done. Some mobile access methods work though an Internet service as opposed to directly from the device itself. This method might be of concern to certain organizations because they have no idea what devices are connecting to their servers, how many devices are using the service or what control (policies) they have on the phone (in many cases, none at all). These services usually have users enter their login and password and then save that on a remote server. The remote server then logs into OWA for that user, scrapes out the information and then pushes the data down to the user's device. In these cases, user credentials and data have left the organization and there is no control over it, or even knowledge of where it is. From the mail ISA screen, click on the "Create Access Rule" link. (shown below) This will bring up a wizard that will take you through the process. In this case we will block the fictitious servers of "Bad Site". On the first page you will Name the rule; let's say we call it "Block Bad Site". On the next screen we will be asked if we want this to be an Allow or Deny rule. Since we want to keep these servers from accessing our Exchange Server, we will select Deny. The next screen will ask us what traffic we want to block. This is where the wording can be a bit tricky if you're not familiar with ISA Server. In this case you want to choose "All outbound traffic". You now need to define who this rule applies to. Select "Add" if the group you want isn't already listed (for this example I'll assume nothing has been created or set up yet). You will get a dialog that lets you pick who you want to apply their access rule to; select New and choose Computer Set as we are going to specify a range of IP addresses. Now we need to name this new computer set (we'll call it Bad Site Servers for this example). You can also add a description in the bottom of the dialog if you want. Then click "Add" and select "Address Range". You now can specify the address range of the server. If the service you are blocking has more than one range of IP addresses, you will enter more than one of these (we'll go through an example of two). You need to enter a name and the starting/ending IP addresses (shown below is just an example). The description is optional. Then click OK. You can repeat this process (clicking on Add and adding another address range) as many times as there are Address ranges. Below you can see our example has two IP address ranges. When all of them are entered, click OK. You will now notice that in the "Add Network Entities" list (under Computer Sets) you have the set you just defined (Bad Site Servers in this case). Select the new Computer Set that you defined, click Add, then click Close. Your access rule now applies to the computers you specified (in this case the Computer Set of Bad Site Servers). You can add additional sources if you wish or click Next to move on. You now need to define the destination for this traffic. In this case, ISA Server (localhost) is the destination of this traffic. Select Add to bring up your options to make this selection. Under Network, you can select Localhost, then click Add and then click Close. In this case Localhost is our only destination so we can select Next. Now select who this applies to add "All Users" and then click Next. You're done creating the access rule. Just click Finish to take you back to the main ISA Server view. When you hit OK you will go back to the main ISA Server screen but at the top it will ask you if you want to apply or discard the changes you made. (see below) When you accept the change you are done. The following dialog will appear once your changes have been applied. Click OK. You're done. Below you can see that there is now a new firewall policy rule that you created to block the server you didn't want to access your site. Note: In general, "Deny" rules should precede any "Allow" rules, although there are exceptions to this. You will need to be familiar with ISA Policies Best Practices to understand the fine points of ISA rule definition. And there you have it; how you can block by device type (User-agent string) and how you can block by server (IP address). New devices and services come online all the time so it's tough to have a comprehensive list but some of the more common User-Agents are: Symbian devices: "Symbian" http://www.developershome.com/wap/detection/detection.asp?page=userAgentHeader Motorola: "mot-" http://www.developershome.com/wap/detection/detection.asp?page=userAgentHeader Samsung: "sec-" or "samsung" http://www.developershome.com/wap/detection/detection.asp?page=userAgentHeader LG: "lg-" http://www.developershome.com/wap/detection/detection.asp?page=userAgentHeader Siemens: "sie-" http://www.developershome.com/wap/detection/detection.asp?page=userAgentHeader Nokia Devices: "Nokia" http://discussion.forum.nokia.com/forum/showthread.php?t=83267 BlackBerry Devices: "BlackBerry" http://na.blackberry.com/eng/developers/resources/journals/mar_2007/profile.jsp Apple Devices: "Appl" (no e on this one as Device ID starts with Appl so this covers all cases) Note: if you just want to block only the iPod Touch, or only the iPhone, you can just block on "iPod" or just "iPhone". http://forums.macrumors.com/showthread.php?t=361166 Some of the common Servers that try and access Exchange Server are below (with links to their docs that list their server IP address ranges): BlackBerry Internet Service: http://www.blackberry.com/btsc/articles/644/KB11036_f.SAL_Public.html Good Mobile Messaging (GoodLink): http://www.goodlink.com/documentation/GoodAdminGuide_exchange.pdf For those that want more info on HTTP filters in ISA server, there are a lot more detail here. - Adam Glick Share this post :

Just wanted to post a quick note that yesterday we have released a new Scalable Networking Pack (SNP) hotfix rollup package for Windows Server 2003. This is all in relation to a blog post on the subject that we have posted a while ago. Mike has posted an update about this new rollup hotfix on his blog, you can check it out here: http://blogs.technet.com/mikelag/archive/2008/08/28/scalable-networking-pack-rollup-released.aspx - Nino Bilic Share this post :

Just wanted to drop a quick note that we have now released a version of Exchange 2003 Exchange System Manager (ESM) that runs on Windows Vista. this has been a frequent request so - now it is here! You can download the package and related release notes here: http://www.microsoft.com/downloads/details.aspx?familyid=3403d74e-8942-421b-8738-b3664559e46f&displaylang=en - Nino Bilic Share this post :

This August 2008, Microsoft has released an update to the Outlook Time Zone Update Tool. This new version (Version 3.0) offers, amongst others, better support for Israel and Brazil with regards to time zone handling and rebasing of calendars. The Exchange Calendar Update Configuration Tool utilizes the Time Zone Data Update Tool for Microsoft Office Outlook for rebasing operations, which is why version 2.0 of the Exchange Calendar Update Configuration Tool included the Time Zone Data Update Tool for Microsoft Office Outlook, in a redistributable format, as part of the package. A new version of the Exchange Calendar Update Configuration Tool was not released along with the release wave of DST updates in Aug 2008, but instead the redistributable version of the Time Zone Data Update Tool for Microsoft Office Outlook V3.0 was made available for download. The OutlookTimeZoneMoveLibRedist.exe is a redistributable installer that installs the assembly file Tzmovelib.dll in C:Program FilesMsExTmz. Third-party calendar rebasing tools can use the assembly file to programmatically update calendars that are displayed incorrectly because of daylight saving time. With this assembly, developers can use the same APIs that Outlook and Exchange Server use in their calendar rebasing tools. The below steps will allow you to utilize the added support in the Time Zone Data Update Tool for Microsoft Office Outlook V3.0, from the Exchange Calendar Update Configuration Tool V2.0: Install the Exchange Calendar Update Configuration Tool from here: http://support.microsoft.com/kb/941018 Go to the control panel, and using Add/Remove programs (found in Programs and Features in Windows Vista) and remove the Time Zone Data Update Engine for Microsoft Office Outlook. Install the Time Zone Data Update Tool for Microsoft Office Outlook V3.0 - redist package http://www.microsoft.com/downloads/details.aspx?FamilyID=ee0af8fd-bbb7-44de-be4d-f33cb1b59563&DisplayLang=en - This download contains the following file: OutlookTimeZoneMoveLibRedist.exe Rebase as appropriate Links: Exchange Calendar Update Configuration Tool: http://support.microsoft.com/kb/941018 Outlook Time Zone Update Tool V3.0 - redist package Outlook tool: http://www.microsoft.com/downloads/details.aspx?FamilyID=ee0af8fd-bbb7-44de-be4d-f33cb1b59563&DisplayLang=en Time Zone Data Update Tool for Microsoft Office Outlook: http://support.microsoft.com/kb/931667/ - The Exchange Team Share this post :

We have always recommended that customers prepare formal server build documentation for their Exchange environments. Until now, we've not provided any formal guidance around what that documentation should look like. We now have build documentation templates and instructions for preparing a build automation DVD. You can use these templates as a starting point for formally documenting your Exchange server builds. Preparing a build automation DVD can help streamline the installation of Exchange 2007 on both Windows Server 2003 and Windows Server 2008. The templates and build automation guidance can be found here: http://technet.microsoft.com/en-us/library/cc533547(EXCHG.80).aspx. Thanks to Ross Smith IV, with Exchange Center of Excellence, for his extensive help with this release. Thanks,Tom Di Nardo Share this post :

Lately, we've been getting a lot of questions related to Exchange 2007 memory consumption and why Exchange uses so much of the paging file. Well, there are some valid reasons why and some misconceptions that need to be corrected. Comparing memory usage between Exchange 2007 and Exchange 2003First, let's start with why Store.exe uses so much RAM. If we take a step back in time to the Exchange 2003 era, this blog was quite active on how to tune memory on an Exchange 2003 server. I am not going to go in to specifics or great detail, but one thing to call out is that on the 32-bit architecture, we were limited to addressing 4GB of virtual memory on any given server. So essentially, any 32bit program could address up to 4GB of virtual memory. This address space is typically split so that applications could address 2GB of memory and 2GB would be for the Windows Executive. By adding the /3GB switch in the boot.ini, applications could now access up to a 3GB virtual address space and lower the Executive down to addressing only 1GB, essentially halving the memory that is can be addressed or is available for kernel drivers, paged/non-paged pool memory, PTE's etc. The larger the load that you put on a server has the potential to exhaust important resources on the server which eventually causes a failure or server outages. Any type of memory leak in these areas could be detrimental to the stability of the server. With the numerous posts on this blog regarding the /3GB, /USERVA and /PAE switches, what drivers to update, how to tweak memory to allow Exchange to run optimally, etc., it was always a juggling act to try to keep Exchange servers online. Cluster servers were especially touchy as http.sys will start rejecting HTTP connections when non-paged pool hit a certain level, thus causing cluster failovers. Due to these constraints of the 32-bit architecture, we needed a way to balance I/O based applications on the server. Essentially, there are two types of I/O for the Windows operating system, Buffered and Unbuffered I/O. For Buffered I/O, only portions of the working set can be contained in memory; those portions not in use are retained in the page file. For Unbuffered I/O, the entire working set can be contained in memory. On any given system, we needed to maintain a balance so that each would get the right resources that they need. This is one of the main reasons why we needed to put a maximum on the ESE database cache to leave us enough room for the system cache.In most implementations, the default of 900MB of database cache was used which was sufficient for most deployments, but some implementations had the need for more cache increasing it to a maximum of 1.2GB of memory on systems of 2GB or more of physical RAM. This maintained the balance that was necessary so that the system cache could perform efficiently.So you may ask, why did I go through all of the above to explain how Exchange 2003 worked? Well, one of the aspects of Exchange 2007 that people ask about is the difference between Exchange 2007 memory usage to Exchange 2003. Indeed, the memory usage is much higher,and there are very good reasons for this. Let me explain further.Exchange 2007 Design ConsiderationsDuring the design phase of Exchange 2007, we needed to find a way to scale out servers further than Exchange 2003 did due to the rising demands of email usage in companies. For many companies, email is their life line and if an exchange server should go down, they could be losing money at an astronomical rate. One of the performance bottlenecks in Exchange 2003 was disk latencies which was mostly due to the memory constraints of the 32-bit platform. Ultimately, -, we had to read more from disk than what was available in memory. This required a well performing disk subsystem to handle the sustained I/O necessary to respond to client requests in a timely manner. In many cases where disks were the underlying problem, a re-design of the disk subsystem was necessary to allow the current amount of IOPS that were being generated on the server to be handled by the disks. This could range from adding more spindles to the disk arrays, rebuilding different RAID solutions to provide enough sustained throughput based on current user load.Why we learned to love the 64-bit PlatformWith the 64-bit architecture being available, we made a conscious decision that Exchange 2007 would only be supported in production on 64-bit hardware. This was mainly done to overcome the limitations that the 32bit platform had. The design goal here was to reduce cost of Exchange deployments in which storage turned out to be the number one driver of cost and support overhead. By moving to the 64-bit platform, we are able to reduce our I/O requirements thus bringing us back to easier balancing of I/O and capacity requirements. Now, one of the results of being able to address more memory is the capability to cache more memory for each application. To allow this to happen for Exchange, the ESE Database Cache Size limit was removed to allow Exchange the ability cache more pages in memory. Accessing pages in memory is extremely fast, so the more data we cache in memory, the less we actually have to read andwrite from the disk. When following our best practice guidance around storage group deployment and memory sizing, Exchange 2007 reduces the amount of I/O required overall. This gave us rather huge performance gains. For more information regarding I/O improvements on the 64-bit platform for Exchange, see the following blog post http://msexchangeteam.com/archive/2006/09/08/428860.aspx. With no limitation in Database Cache for Exchange 2007, the memory usage for the store process will naturally be much higher compared to what Exchange 2003 used due to the many benefits discussed earlier. Memory allocation for the ESE cache is dynamic, so Jet will use all available memory on a system and return it as needed. For example, if a server had, let's say, 16GB of physical RAM installed, the database cache could consume approximately 14GB of memory, roughly 2GB less than the total amount of physical RAM in the server, leaving enough memory for the system cache or other applications running on the server. The algorithm is relatively simple for the ESE Database Cache. The DBA (Dynamic Buffer Allocation) will grow the cache by comparing the amount of I/O the databases are doing with the amount of I/O the system is doing in terms of hard page faults (MemoryTransition pages repurposed/sec). So if the database is doing more I/O, and Exchange is hitting hard page faults, we will increase the cache. If the database is doing less I/O than the hard page faults, then we will decrease the cache. The goal is to keep the ESE cache in balance with the disk cache so we don't end up paging ourselves to death. In some cases, on larger servers, Online maintenance during the night could cause up to a 4GB swing in memory for the Database cache alone due to the aggressive nature of how we touch pages during this process. (It may also be prudent to stagger your OLM schedules to assist with this.) Each server configuration is different, so it is important to have a monitoring solution in place or a Performance monitor log to track memory usage over time if memory pressure is occurring. If memory usage gets to a point where some driver or application attempt to allocate a large block of contiguous memory, and there is not one available, then the OS may decide to start trimming working sets which is a really bad thing performance wise. I've discussed this on my own blog http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx and possible remediation's to work around those type scenarios if you fall in to this bucket.Other Server Memory ConsiderationsSo we've talked a lot about memory usage for ESE database cache, but this cache alone is not the only significant consumer of memory on an Exchange server, it is just the majority. From an Exchange perspective, we have other processes going on such as Content Indexing, log shipping in continuous replication and underlying .NET applications that Exchange makes use of now. Most of Exchange has been rewritten in managed code using the .NET framework except for Store, DAV, and DSAccess. Managed code offers some significant advantages over non-managed code such as the ability to compile applications in real-time so that you don't have to worry about running your application on different architectures or platforms and the ability to manage memory efficiently. .NET applications use the Common Language Runtime (CLR) to allow easier coding in different languages because they share the same runtime. Any code that you develop with a language compiler that targets the runtime is called managed code as mentioned here: http://msdn2.microsoft.com/en-us/library/ddk909ch.aspx.Reasons for Memory PagingOne of the primary reasons for memory paging in Exchange 2007 is Content Searching, which uses the system cache since this is buffered. Performing more searches increases the file cache you have which in turn reduces the amount of IOPS that are generated. Content Indexing itself does not make use of the file cache as this is designed to work on a memory structure that is not affected by the file cache. Master merges of index data does use a small amount of the file cache, but it very short lived. Database I/O is actually unbuffered, so this not affected by the system cache. This is why the ESE cache doesn't continuously grow (ESE cache I/O is less than the system cache I/O). It's being balanced out by the content searching I/O (and potentially other applications that use buffered memory I/O) and thus the system cache. CCR log shipping is also unbuffered while log files are being copied over SMB, so this is also not affected by the system cache.Now couple all of this memory management with other 3rd party applications and drivers that are installed on the server which are all competing for the same resources, you can see that this is vastly different than what Exchange 2003 used to be. There is actually no comparison at all.How Exchange makes use of the Paging File So this brings up the second part of this document where I will now talk about the paging file. So you may ask, if I have so much memory in the server, why is a paging file needed at all? Our system requirements talk about needing one for getting a memory dump on the server, but what does Exchange really use it for? I get this very question asked very frequently. In certain cases, it is true that Windows 2003 does not require a paging file. It is however completely incorrect that Exchange does not need a paging file for normal runtime operations when running on top of that OS. There is always going to be a need to page out unused or rarely used memory to the page file so that we can make maximum use of RAM. For example, there could be a large amount of VM pages used during system startup that are not useful under main load. Unexpected transient conditions could cause large upward swings in memory that could cause us to spill pages of memory in to the paging file. There are enough processes outside of the cache manager that consume a significant amount of memory in Exchange 2007 (e.g. ESE and all processes that utilize the CLR) that there is a need for a substantially sized paging file as the system balances the memory footprint of the competing processes. Without a paging file (or with one that is too small) there is a very high chance that applications will encounter OOM (out of memory) conditions and both the performance and the functionality of the server will suffer and may lead to a possible blue screen.The cost of having a 32GB paging file on a server with 32GB of physical RAM on disk is pretty minimal, so having a properly configured paging file (RAM+10MB) will be able to sustain any unexpected growths in memory where we need more commit charge to handle current load. The other benefit is that if we should get ourselves in to a working set trimming issue, we would at least have the ability to flush these pages out to the paging file and then page them back in when a particular page needs to be accessed again. Excessive working set trimming on an Exchange 2007 server is not normal and should be investigated. Again, check out my blog post on working set trimming problems and why this is bad for performance.Why does Exchange allocate so much memory in the paging file?Another question that generally comes up is why does Exchange allocate so much memory in the paging file? Depending on the amount of memory that is consumed for the store process, we will need to at least allocate enough memory in the paging file to match the current working set that is in RAM on the server. If a system wide working set trim should occur, we would be able to page the current pages in the store working set out to the backing store or paging file. It is important to monitor MemoryPages/sec on the server to detect if this working set trimming problem is causing any performance related problems on the server. You should be able to see by adding ProcessWorking Sets a sharp decline in this value along with a sharp increase in MemoryPages/sec. With the other processes on the server possibly using parts of the system or file cache, there is always a balancing act trying to compete for memory on any given server. The good news is that Windows 2008 has made some significant changes in memory management to help prevent these system wide working set trims and to manage memory more efficiently. With server applications having the need for using more RAM as technology advances, it is imperative that memory management works effectively to prevent any negative performance problems on any given server. For more information on some of these memory management changes in Windows 2008, see http://www.microsoft.com/whdc/system/sysinternals/memmgt.mspx. I hope this helps clarify some of the reasons why Exchange uses more memory overall and provide clarity for these most common questions.A special thanks goes to Andrew Goodsell, Matt Gossage, and Ross Smith IV for their invaluable information in this area and Nick Basile for tech reviewing this document.Mike LagaseSupport Escalation Engineer Share this post :

A little trick from Jeffrey Snover over on PowerShell team blog: Speeding Up PowerShell Startup. Running the script mentioned by Jeffrey speeds up PowerShell startup times. Yes, hard to believe at first, but after having run this on a few servers, as Jeffrey says - the reaction is "Wow!" Paste the following in notepad and save it as Update-Gac.ps1 (or whatever you want to call it): Set-Alias ngen @(dir (join-path ${env:windir} "Microsoft.NETFramework64") ngen.exe -recurse |sort -descending lastwritetime)[0].fullName[appdomain]::currentdomain.getassemblies() | %{ngen $_.location} Note: On x86 systems; replace Framework64 in the second line of this script with Framework. [Optional] Close all open windows [Optional] Start the Exchange Management Shell and note the time it takes to start up Run the script: .Update-Gac.ps1 (or whatever you saved it as) Quit all open windows, start the Shell. Notice the difference? As Jeffrey says in his blog post, the PowerShell team had a problem in Windows PowerShell v1, that caused the binaries not to be NGENed. What's NGEN?Managed code exists in Common Intermediate Language (CIL). The Just In Time (JIT) compiler takes that and converts it into native code for a particular processor/platform - a process also referred to as "jitter". If you don't want this translation to happen at runtime, you can use NGEN.exe - the Native Image Generator - to convert managed code from the CIL to native code. Native code does not need to be compiled at run time, and thereby avoids jitter. So how does speeding up PowerShell help Exchange?The Shell is built on Windows PowerShell technology, and under the hood, the Exchange Management Console also executes PowerShell cmdlets. If you're running Exchange Server 2007 on Windows Server 2003, you're likely to see performance gains in shell start-up times after running this script. On one of my Exchange 2007 servers, EMS now starts up in as little as 2-3 seconds. Disclaimer: Your mileage may vary. Feel free to leave feedback about results you're seeing with this. - Bharat Suneja Share this post :

Background This is a follow up to my recent post: A study of Exchange 2007 SP1 Hub throughput with different message sizes . Please refer to this post for full background information and test details. I promised in that post that I would add results for smaller message sizes and include data for SATA disks -- what we can call commodity storage'. Columns 1 and 2 of the Table below show the results of 2 more scenarios, using same high end SCSI storage as in the previous post, adding message sizes of 57 KB and 80KB to the posted measurements. Columns 3 and 4 were taken on a newer, more powerful server but with less expensive 'commodity storage' (SATA drives). Server Configuration 1. Ultra SCSI storage server: 2 processors x 2 core, 2.2 GHz, 800 MHz FSB, 1MB L2 cache per processor, 4 GB RAM, 400 MHz memory, Ultra 3 SCSI disk controller ("entry level") with 128 MB Write-Back Cache, 3 x Ultra320 Universal SCSI 15K RPM disk.Optimized E2K7 transport database queue configuration: 1 disk for DB logs 1 disk for DB queue file 1 disk for OS and other transport logs: Message Tracking, Connectivity, Agent logs, etc. 2. SATA disks server: 2 processors x 2 core, 2.33 GHz, 1.33 GHz FSB, 4MB L2 cache per processor, 16 GB RAM, 333 MHz memory, SATA interface without Battery Backed Write-Back Cache, 4 SATA disks 7200 RPM . Optimized E2K7 transport database queue configuration: 1 disk for DB logs 2 disk for DB queue file, Raid 0 configuration 1 disk for OS and other transport logs: Message Tracking, Connectivity, Agent logs, etc. Disabled: Enable advanced performance disk policy Disabled: Enabling write caching to disk disk policy Hub Storage Ultra SCSI storage SATA disks Limiting Resource CPUBound CPUBound IOBound IOBound Message Size 57KB 80KB 57KB 80KB SMTP Receive Throughput (msg/sec) 138.17 118.87 48.85 40.36 Aggregate Queue length (MAX) 358 409 44 64 Queue size in MB (MAX) 20.41 32.72 2.51 5.12 %CPU 84.60 84.15 17.65 17.81 Msg Cost (MCyc/msg) 50.93 59.29 30.36 37.30 Msg Cost (MCyc/ByteOfMsg) 893.51 741.13 532.63 466.25 Disk Writes/sec (log) 146.14 165.08 48.50 51.51 Disk Writes/sec (queue) 91.15 124.78 75.00 116.00 Disk Writes/msg (log) 1.06 1.39 0.99 1.28 Disk Writes/msg (queue) 0.66 1.05 1.54 2.87 Avg msec/write (log) 0 0 11 13 Avg msec/write (queue) 0 0 97 109 Avg disk Queue length (log) 0.14 0.17 0.64 0.74 Avg disk Queue length (queue) 0.18 0.33 3.90 6.57 Disk Reads/sec (log) 0.00 0.00 0.00 0.00 Disk reads/sec(queue) 0.00 0.00 0.00 0.00 Analysis of Results The first 2 columns are an extension of the previous results, using the default 128 MB transport DB cache size.Quoting the previous blog post: "storage is key for transport performance, all the above data only applies to a Hub server with at least an "entry level" SCSI controller with 128 MB of BBWC (battery backed write-back cache) that optimizes the IO pattern transport performs on steady state flow: continuous writes with very few or no reads."The last 2 columns present the results on the new hardware. Notice the high disk latencies and the appearance of a disk queue without having a BBWC (battery backed write-back cache). Also notice the smaller MCyc/msg cost, this is because the new machine's cycles are much more powerful than the cycles on the older machines thanks to higher FSB, and more L2 cache. New disclaimer: On the SATA disk machines, the Write Caching disk policies checkmarks Enable advanced performance and Enable advanced performance have been both disabled.Yes, I tested briefly with these policies enabled. Setting Enable advanced performance, raises the SATA disks performance to the level of the SCSI storage, resulting in even better throughput in the test because it's a faster machine! But it is not safe to enable it on production machines, because without a real hardware BBWBC controller data can be lost during hardware failures. See Windows Confidential - The Power of Bugs (April 2007 issue of TechNet Magazine) for a through explanation. Elías KaplanSDET II, Exchange Shared Share this post :

Enterprise Communications Support (formerly known as Exchange Messaging Support) has recently seen an increase in support incidents around the -519 Jet_errLogSequenceEnd issue. In this blog I will explain this issue in detail, why it may occur, how to recognize it, and what to do next. Exchange has had a robust data engine that provides an ACID database from the early days; the transaction-based JET database engine ensures all database changes are Atomic, Consistent, Isolated and Durable. We accomplish this with the use of transaction log files. In Exchange Server 2000 and 2003, we introduced the concept of Storage Groups. Storage Groups can contain up to 5 databases, but share one set of common Transaction Logs. To differentiate between Storage Groups, all Transaction Log file names contain a prefix, followed by a 5 digit hexadecimal number, like so: Enn00001.log Where nn is the number of the Storage Group, typically 01, 02, and so on. As transactions (changes such as new e-mail, mail being read, tasks created, views built, etc) occur Exchange records them in sequentially numbered logs. This allows us to recover from certain database problems by knowing exactly which log comes next in a replay sequence. In Exchange 2000 and 2003, the transaction log file name length is 8 characters long. Since 3 of the characters form the file name prefix, 5 remain. Thus, the largest possible hexadecimal number we can represent is EnnFFFFF.log The actual largest number ESE in 2000/2003 will allow is FFFF0. That number in decimal is 1,048,560, representing the maximum number of log files we can write sequentially before running out. With over one million 5 MB log files available, in some cases, it can take as long as a few years to hit this condition. When the transaction log sequence is exhausted, the Microsoft Jet database engine returns error -519, JET_errLogSequenceEnd to the Information Store. Depending on which version of Exchange 2000 or 2003 you are on, this error will result in slightly different symptoms. These symptoms are described in the following Knowledge Base articles: 830408 Store databases are dismounted without warning or users cannot log on to their mailboxes in Exchange Server 2003 or in Exchange 2000 Server http://support.microsoft.com/default.aspx?scid=kb;EN-US;830408 896001 An event is not logged in the Application log before the last available transaction log in the sequence is used in Exchange 2000 Server http://support.microsoft.com/default.aspx?scid=kb;EN-US;896001 In the latest revisions for Exchange 2000 and 2003, we added some fixes to warn you of this impending problem and prevent it from occurring. Store will warn you when you are nearing the end of the log sequence with the following event: Event Type: Warning Event Source: ESE Event Category: Logging/Recovery Event ID: 514 Description: Information Store (2748) SG2: Log sequence numbers for this instance have almost been completely consumed. To begin renumbering from generation 1, the instance must be shutdown cleanly and all log files must be deleted. Backups will be invalidated. If you have databases in Storage Groups that have been online contiguously for several years, you should be monitoring your transaction log sequence and the Application Log for this event. When you see this, it's time to reset the transaction log sequence. Notice that if you miss the ESE 514 warning your databases will dismount and generate the following events: Event ID: 1159Event Type: ErrorEvent Source: MSExchangeISEvent Category: GeneralDescription: Database error 0xfffffdf9 occurred in function JTAB_BASE::EcEscrowUpdate while accessing the database "First Storage GroupMailbox Store (SERVER)". Event ID: 9518Event Type: ErrorEvent Source: MSExchangeISEvent Category: GeneralDescription: Error 0xfffffddc starting Storage Group Path_of_Storage_Group on the Microsoft Exchange Information Store. Storage Group - Initialization of Jet failed. OK, great, what the heck does 0xfffffddc mean? Glad you asked! You can look up Exchange error codes here: Microsoft Exchange Server Error Code Look-up Note that error translates to Jet_errLogSequenceEndDatabasesConsistent: Err 0xfffffddc -# for hex 0xfffffddc / decimal -548 JET_errLogSequenceEndDatabasesConsistent esent98.h# /* databases have been recovered, but all possible log# generations in the current sequence are used; delete all# log files and the checkpoint file and backup the databases# before continuing */ If you attempt to mount databases in this condition you will discover another cause for this common error in the Exchange System Manager: An internal processing error has occurred. Try restarting the Exchange System Manager or the Microsoft Exchange Information Store service, or both.ID no: c1041724Exchange System Manager Let's Get Fixed! Now that I've explained the issue and how it will be reported in the event logs, here's how to correct this problem: KB 830408 has a workaround section that describes how to reset transaction log sequence manually. However, we have included this functionality in the Microsoft Exchange Troubleshooting Assistant "Reset log generation number task". We talk about that here: MSExchangeIS 9518 (0xfffffddc): Exceeded the Maximum Number of Transaction Logs Available for this Storage Group Microsoft recommends using the Troubleshooting Assistant (ExTRA) because it automates the process of verifying the health of your databases and resetting the transaction log sequence. Download the Microsoft Exchange Troubleshooting Assistant v1.0 and follow these instructions to reset transaction log sequence: http://technet.microsoft.com/en-us/library/aa998611(EXCHG.80).aspx We also recommend monitoring all your Exchange 2000 and 2003 Storage Groups' transaction log sequences to avoid a potential temporary outage of service. Monitor your application logs for the ESE 514 events. Now, while it is theoretically possible to hit this condition in Exchange 2007, it will probably take a really long time. For Exchange Server 2007 we decreased the log file size to 1 MB but increased the transaction log file name length to 11 digits, meaning we can go up to EnnFFFFFFFF.log Due to the way ESE math works our upward limit is actually 7fffffec log files, but that is still a HUGE number. After figuring in the change in size to 1 MB, that's still about 409 times the number of logs we could generate in 2000/2003. Read more about this improvement in the following TechNet Magazine article located here. Corbin MeekEnterprise Communications Support Share this post :

Hello from the Sound Department from Jan McLaughlin on Vimeo. Shozu auto-crossposts cellphone-created videos to Vimeo for me. Very nice-looking, and a different kind of community than the YouTube, to which Shozu also automatically uploads cellphone videos on my behalf. The downside? Tougher to keep track of overall hits. I don't personally care that much, but marketers and advertisers do. My guess is that in social media terms, responses count more than hits. I like what Hugh McLeod has to say about social objects. What are the social objects of a film-in-progress? What are "City Island" social objects? Perhaps audio, video & stills form a part of those objects. I would love to get Jeff Pulver to do some stories about "City Island" blogging efforts. My guess is there are 'concerns' among the beautiful intensity. Of course there are. It's the bleeding edge of the digital now. A Pulver breakfast @ "City Island" could rock for everyone. Insert brief PRAYER. [Vimeo] Tags: Faux Press Art Metaphrasty

Introduction We have been working on a problem that surfaced with the release of Exchange 2007 Rollup 5. A number of customers reported that some of their Exchange 2007 managed services did not start automatically after Rollup application, however they would start manually. In most of these cases the computer in question was not connected to the Internet. Investigation showed that the problem was the timing of the Windows Service Control Manager (SCM) and validation of all of the certificates associated with a service within the SCM timeout. In the case of computers that were connected to the Internet, the problem seemed to be network latency, as the problem in those cases can happen only intermittently. Why this happens To sum it up: the problem did not manifest itself until Exchange 2007 managed binaries were signed with two certificates (because the original one was expiring). For Exchange 2007 RTM this occurred when RU5 was released. Because now .Net Framework Common Language Runtime (CLR) attempted to validate two certificates by connecting to http://crl.microsoft.com, the process took longer, to the point where SCM timeout would pass and services would fail to start up automatically. In environments with limited or no Internet connectivity, this might fail every time. In other cases, this might fail intermittently based on current network state and load. Workarounds KB article 944752 Exchange 2007 managed code services do not start after you install an update rollup for Exchange 2007 is being revised to only contain the recommended workaround, which is to modify the services configuration files. The modification will to prevent the CLR from going onto the Internet in the first place. This is accomplished by adding a section to the managed executable configuration file as outlined in Bypassing the Authenticode Signature Check on Startup (MSDN .NET Security Blog, Shawn Farkas). Security concerns The first question that people usually ask about is if making the modification to .config files compromises server security. In case of Exchange Server 2007, this is not relaxing security at all. From a security point of view, what it's doing is saying assume that the Authenticode signature is invalid. Let's say that you have these three assemblies: An assembly without an Authenticode signature. One with an invalid Authenticode signature One with a valid signature that has this option set in the Configuration file All three behave exactly the same. The CLR will load them, and not give Publisher evidence to the assembly. Since the assembly doesn't get the Publisher evidence, any trust decisions that were being made based upon the validity of the signature will no longer apply - so if the assembly is only trusted due to its signature it will lose its trust status due to the config switch. Exchange 2007 assemblies however, do not use this mechanism as the only one to determine if the assembly should be run or not, so disabling it does not compromise Exchange. To make this .config file modification The resolution to this problem involves editing the configuration files associated with the Exchange Services to use a switch which was added to CLR 2.0 SP1 (which is by default present in .NET framework V3.5.) You can update .NET Framework 2.0 or 3.0 by installing the fix from http://support.microsoft.com/kb/942027/. That hotfix contains the fix outlined in http://support.microsoft.com/default.aspx/kb/936707. Before continuing with this procedure, save a copy of your existing configuration files to a safe location. In the event of an error in the configuration file, the applicable service will fail to start. Create configuration files for all managed code Exchange 2007 services to resolve this issue. Please note that we do not suggest going to create/modify the .config files unless your server is actually impacted by this problem. To create an application configuration file that contains this configuration setting, follow these steps: 1. Create a file, and then name the file the ApplicationName .exe.config file. 2. In a text editor, open the file that you created in step 1. 3. Add the following code to the file: configuration runtime generatePublisherEvidence enabled= false / /runtime /configuration 4. Save the changes to the file to the applicable directory (see below for a list of files and locations - the .config files should be saved into the same folder where the affected executable is). If the configuration file already exists for a specific service, just add the generatePublisherEvidence enabled= false / line to the runtime options section in the file. Exchange 2007 Services/Apps which come with a .config file that might need to be updated: BinEdgeTransport.exe BinExBPA.exe BinExBPACmd.exe BinExTRA.exe BinMicrosoft.Exchange.Cluster.ReplayService.exe BinMicrosoft.Exchange.EdgeSyncSvc.exe BinMicrosoft.Exchange.Monitoring.exe BinMicrosoft.Exchange.Search.ExSearch.exe BinMicrosoft.Exchange.ServiceHost.exe BinMSExchangeMailboxAssistants.exe BinMSExchangeMailSubmission.exe BinMSExchangeTransportLogSearch.exe ClientAccessPopImapMicrosoft.Exchange.Imap4.Exe ClientAccessPopImapMicrosoft.Exchange.Pop3.Exe Exchange 2007 Services for which you need to create a new .config file (unless it was already created for another reason): BinMicrosoft.Exchange.AntispamUpdateSvc.exe BinMsExchangeFDS.exe BinMSExchangeTransport.exe Troubleshooting If a service fails to start after modifying or creating the config files, the most likely reason is an XML syntax error or an incorrect value. In both of these cases the service will fail to start and you'll get an error similar to this example from the Exchange 2007 Edge Transport Service: Event Type: Error Event Source: MSExchangeTransport Event Category: Process Event ID: 14004 Description: The worker process has failed to load application configuration file: System.Configuration.ConfigurationErrorsException: Configuration system failed to initialize --- System.Configuration.ConfigurationErrorsException: The 'generatePublisherEvidence' start tag on line 4 does not match the end tag of 'runtime'. Line 5, position 6. (C:Program FilesMicrosoftExchange ServerBinedgetransport.exe.config line 5) --- System.Xml.XmlException: The 'generatePublisherEvidence' start tag on line 4 does not match the end tag of 'runtime'. Line 5, position 6. at System.Xml.XmlTextReaderImpl.Throw(Exception e) at System.Xml.XmlTextReaderImpl.ThrowTagMismatch(NodeData startTag) at System.Xml.XmlTextReaderImpl.ParseEndElement() at System.Xml.XmlTextReaderImpl.ParseElementContent() at System.Xml.XmlTextReaderImpl.Skip() at System.Configuration.XmlUtil.StrictSkipToNextElement(ExceptionAction action) at System.Configuration.BaseConfigurationRecord.ScanSectionsRecursive(XmlUtil xmlUtil, String parentConfigKey, Boolean inLocation, String locationSubPath, OverrideModeSetting overrideMode, Boolean skipInChildApps) at System.Configuration.BaseConfigurationRecord.ScanSections(XmlUtil xmlUtil) at System.Configuration.BaseConfigurationRecord.InitConfigFromFile() --- End of inner exception stack trace --- at System.Configuration.ConfigurationSchemaErrors.ThrowIfErrors(Boolean ignoreLocal) at System.Configuration.BaseConfigurationRecord.ThrowIfParseErrors(ConfigurationSchemaErrors schemaErrors) at System.Configuration.ClientConfigurationSystem.EnsureInit(String configKey) --- End of inner exception stack trace --- at System.Configuration.ConfigurationManager.GetSection(String sectionName) at System.Configuration.ConfigurationManager.get_AppSettings() at Microsoft.Exchange.Transport.TransportAppConfig.GetConfigBool(String label, Boolean defaultValue) at Microsoft.Exchange.Transport.TransportAppConfig.ResourceManagerConfig.Load() at Microsoft.Exchange.Transport.TransportAppConfig.Load() at Microsoft.Exchange.Transport.Main.Program.Run(String[] args) Additional information We're continuing to investigate this problem and will provide more information as it becomes available. - Ed Beck, Nino Bilic Share this post :

Recently I came across a situation where it was reported that Connection Filtering stopped working (IPs on the Blocklist and RBLs were no longer being blocked). The solution led me to write this blog to clarify some confusion about when connection filtering is applied and how configuration settings are applied when the agents are installed on a Hub server. Let's begin by looking at the online documentation regarding Connection Filtering: By default, connection filtering is enabled on the Edge Transport server for inbound messages that come from the Internet but are not authenticated. These messages are handled as external messages. You can disable the filter in individual computer configurations by using the Exchange Management Console or the Exchange Management Shell. When connection filtering is enabled on a computer, the Connection Filter agent filters all messages that come through all Receive connectors on that computer. As noted earlier in this topic, only messages that come from external sources are filtered. External sources are defined as non-authenticated sources. These are considered anonymous Internet sources. http://technet.microsoft.com/en-us/library/bb123943(EXCHG.80).aspxv-61marf From this explanation we see 4 things: That Connection Filtering is installed on Edge by default (as are all the other AS agents) Enabled for inbound (ExternalMail) by default For connections that have not authenticated Connection Filtering (and all AS agents) can be disabled/enabled on individual computers So in the scenario (where connection filtering was no longer blocking) we checked: Get-Transportagent which showed the Connection Filtering agent enabled Get-IPBlocklistconfig which showed True for both Enabled and ExternalMailEnabled (False for InternalMailEnabled - default setting) Get-IpBlocklistentry which contained IPs that should be blocked Confirmed that ActiveDirectory correctly reflected that the agent and config were enabled Agent Logs did not show activity related to the IPs that should be blocked The missing piece was in understanding that connection filtering is a combination of how the Agents are enabled (noted above) and what rights the connecting SMTP session is granted. Examining the SMTP receive log files indicated that the session was granted all the rights possible (including ByPassAntiSpam) which only occurs with Externally Secured Authentication. So here's the way it works: When the AS Filter components Enabled and ExternalMailEnabled parameters are set to true, any mail that comes in from an SMTP Session anonymously or via a Partner may be scanned. If the AS Filter components Enabled and InternalMailEnabled parameters are set to true, any mail from an authenticated session may be scanned. Note: Authenticated partner sessions are not considered Internal. So to recap: The following 5 points should be considered when determining whether an AS agent executes against a particular SMTP session. 1) The agent itself must be enabled. i.e. The Connection Filtering agent. Use Get-TransportAgent to determine which agents are installed and enabled/disabled. 2) The Anti-Spam config must be enabled. i.e. Get-IPBlockListconfig | fl enabled 3) Consider whether the Anti-spam component is set for ExternalMailEnabled and/or InternalMailEnabled Default settings IPAllowListConfig: 4) Anonymous and Partner SMTP Sessions are governed by the ExternalMailEnabled parameter. Authenticated sessions (including connectors that are configured for External Authoritative) are governed by the InternalMailEnabled parameter. 5) What permissions does the submitting client have? i.e. All Exchange Servers and Externally Secured sessions get the Bypass Anti-spam privilege (this cannot be removed). Even when ExternalMailEnabled is true and the SMTP session is anonymous, if NT AuthorityAnonymous Logon has the Bypass Anti-Spam associated with the receive connector, mail will not be checked. Now to dispel some other misunderstandings with regard to Configuration controls IPAllowlistconfig or IPBlocklistconfig command default settings are below. However, if InternalMailEnabled is set to True... ...no action is taken on trusted servers in the Exchange Organization. For grins, I decided to test this in my lab sending mail from one Hub to another. The sending server passed the X-EXPS Auth command which would be the auth used for Exchange Servers . In the debug tracing you could see that the IP was checked against the IPBlocklist, but not rejected because the Exchange Servers group is granted ByPassAntispam permissions on the connector. Configuration Misunderstandings when Anti-Spam Agents are installed on Hub servers Anti-spam Agents are installed per server by running install-antispamagents.ps1 script. After running the script you will have Organization level and Server level controls. There are two Anti-Spam Tabs added to the Exchange Management Console, one at the Org level and another at the ServerHub level. Organization level settings in the Exchange Management Console: Server level setting in the Exchange Management Console: Get-TransportAgent cmdlet is a per Transport server configuration setting. This example has 3 of the agents disabled. So this will only affect the Hub this is configured on: The cmdlet, Set- Transport server, -AntispamAgentsEnabled, is a bit confusing at first. The default value is True when you run the script to install the agents on a Hub. When set to False, it does not disable the AS agents. It simply hides the Anti-Spam tab at the Server level for that particular Hub server in the Exchange Management Console (may require restart of the msExchangeTransport and close / reopen the console). The overlooked 'internalSmtpServers' list Imagine this scenario: Mail with valid SPF records is rejected by your SenderID Agent. The SPF shows these IP addresses: contoso.com text = v=spf1 ip4:192.168.50.2 ip4:192.168.50.3 ip4:192.168.50.4 -all The rejected Message headers are: Received: From senderserver.contoso.com (192.168.50.3) by hosting1.company.com (192.168.2.3) Received: From hosting1.company.com (192.168.51.3) by mail02.yourcompany.com (192.168.75.6) From: sender@contoso.com Since Exchange has to pick an IP to compare to the SPF records, which one does it pick? To determine this, Exchange starts with the last received: From header in the mail message and looks for a match in the internalSmtpServers list moving up the received: From headers until a match is NOT found. In the example above, Received: From hosting1.company.com (192.168.51.3) will be the first IP match attempted. The reason the mail was rejected above was that IP was not in the internalSmtpServers list. Adding it then returns a match so the next Recevied:From header is now examined and that IP is not only the last external IP (not in the list of internalSmtpServers) but also on the Sender's SPF records (192.168.50.3) and the mail passes SenderID Agent. In some scenarios mail is filtered through a hosted service provider that provides services such as Anti-Spam, Anti-Virus. By failing to add the hosted service provider IP addresses to the internalSmtpServers list, it's possible that all inbound mail will cease. Upon investigation you find the following in your Agent Log: Agent : Connection Filtering Agent Event : OnEndOfHeaders Action : RejectMessage SmtpResponse : 550 5.7.1 External client does not have permissions to submit to this server Reason : LocalBlockList ReasonData : machine-generated entry Machine generated entries are those added by the Sender Reputation Agent. You can get a quick look with the following cmdlet: PS get-IPBlockListEntry | {where $_.IsMachineGenerated} Remember, the internalSmtpServers determines what the 'last external IP' to be used by the AS agents. If incoming mail is filtered through an appliance or hosted service it's imperative that the ip address(s) of those servers be listed here. When the AS agents are installed but the InternalSmtpServers is not populated, Event 1022 is logged: Anti-spam agents are enabled and the list of internal SMTP servers is empty. Please use the set-TransportConfig task to populate this list. Troubleshooting connection filtering Determine if the connecting server authenticated by examining the SMTP protocol receive logs What permissions were ultimately granted to the session (get-adpermission for the receive connector Exchange Extended rights on the user) Check the IPAllowlistconfig or IPBlocklistconfig for how they are enabled Check the IPAllowlistentry and / or IPBlocklistentry Check the individual server settings with Get-Transportagent - Dave Forrest Share this post :

It's been a while since I've been thinking of writing a blog post about various aspects of Outlook Anywhere that people have been asking questions about. Somehow, I keep getting myself caught up in one thing or another, and have consequently delayed writing this blog post by almost 4 months. Ugh. Better late than never I figure. Given how long this blog post is overdue, I plan to cover a lot of topics, from frequently asked questions to common misconceptions to problems with Outlook Anywhere to suggested solutions for different problems. How does Outlook Anywhere work? I won't cover details on the cmdlets that enable and change settings for Outlook Anywhere. There is already a bunch of documentation on it. Instead, let's do a slightly deeper dive than the cmdlet documentation provides. The values that you provide to Outlook Anywhere settings can be classified into 2 types of properties - client facing and server facing. Examples of client facing properties are ClientAuthenticationMethod, External Host Name. Examples of Server facing properties are IISAuthenticationMethods, SSLOffloading. Client facing properties are picked up by Autodiscover and supplied to Outlook to configure client access to the Outlook Anywhere service. Server facing properties are picked up by a servicelet called RpcHttpConfigurator which runs as part of the Microsoft Exchange Service Host service. This servicelet runs every 15 mins by default, but the interval can be adjusted by changing the value of the HKEY_LOCAL_MACHINESystemCurrentControlSetServicesMSExchangeServiceHostRpcHttpConfiguratorPeriodicPollingMinutes regkey. Note that setting this value to 0 turns off the RpcHttpConfigurator. When the RpcHttpConfigurator runs, it picks up the IISAuthenticationMethods and SSLOffloading values from the AD and stamps it on the rpc vdir settings in the IIS metabase - overwriting any previously set value. This means that if you manually change the settings on this vdir, you should expect to be run over pretty shortly by the RpcHttpConfigurator (unless you have set the reg key to 0). Ok, so that's just part of what the servicelet does. Outlook Anywhere depends on the RPC/HTTP Windows component to do the marshalling and unmarshalling of the RPC packets from the client to the CAS server. A client side RPC component is responsible for marshalling every RPC packet into an HTTP tunnel and sending it over to the rpc vdir on the CAS server. RPCProxy is an ISAPI extension that unmarshals the RPC packet, retrieves the RPC endpoint that the client wants to talk to and forwards the packet to the endpoint. But imagine if you were able to connect to any server in the organization if you were able to auth against an IIS box running RPCProxy. By the weakest link theory, all you'd need to do would be hack into a single IIS server and you'd have free access to all servers in the org. Ouch ! To alleviate this problem, RPCProxy only allows connections to be made to servers and ports that are in a trusted list. This list is maintained through the HKEY_LOCAL_MACHINESoftwareMicrosoftRpcRpcProxyValidPorts regkey and contains all the servers/ports that RPCProxy is allowed to talk to. So, the other part of what the RpcHttpConfigurator servicelet does it that is queries the AD for all mailbox servers and stamps them in the ValidPorts regkey allowing access to ports 6001, 6002, 6004 for both FQDN and Netbios access. So, you will typically see something like mbx1:6001-6002;mbx1:6004;mbx1.contoso.com:6001-6002;mbx1.contoso.com:6004 as the value for the key. As new mailbox servers are added to the org, they will be picked up when the servicelet runs and be added to the key. Again, if you manually change this regkey, you should expect to be bulldozed by the servicelet. Note that the ValidPorts key is only used by RPCProxy as a filter to disallow communication with unlisted server ports. It is not used to determine which server to send requests to. For the same reason, the order in which servers are listed in this key does not matter. I just thought I'd clarify this since I was recently told that there was confusion on what this key accomplished. Ok, simple enough, now that all the configuration is done, how does Outlook Anywhere actually establish its connections. The following diagram may help: As you see above, the client specifies the VIP of the Load balancer (or direct CAS FQDN if the CAS is exposed to the Internet) as the HTTP endpoint and the mailbox server as the RPC endpoint. The query string is somewhat like this: http://nlb.contoso.com/rpc/rpcproxy.dll?mbx1.contoso.com:6001 This tells the RPCProxy on CAS1 that the client is trying to connect to server mbx1.contoso.com on port 6001. RPCProxy looks up the ValidPorts key and if mbx1.contoso.com:6001 is listed there, it allows the connection to go through. The blue and red arrows above represent the 2 different connections spawned by the RPC/HTTP client component to represent a single RPC connection. This is done because HTTP connections are half duplex (i.e. they either allow you to send information or receive information, not both at the same time). In the case of RPC, connections need to be long lived and full duplex, so the RPC_IN_DATA connection acts as the sending half duplex connection, while the RPC_OUT_DATA connection acts as the receiving half duplex connection. Since HTTP requires that each connection be given a max length, each of these connections are 1GB long and are terminated when this limit is reached. Each of these connections is tagged with a client session id. When the RPC server component receives the RPC_IN_DATA and RPC_OUT_DATA with the same client session id, it knows that for any request received on the RPC_IN_DATA connection, it must reply back on the RPC_OUT_DATA connection. That's magic. Ok, so you already know this, but I'll reiterate - the mailbox server has 3 ports that are used for RPC/HTTP: port 6001 is used for Mail connections, port 6002 is used for directory referral, port 6004 is used for proxying directory connections to AD. The Referral Service running on port 6002 and DSProxy running on port 6004 are part of the same mad.exe process, and the Referral Service just refers clients back to DSProxy to establish their Directory connections. If you Ctrl+Right Click the Outlook icon and click on Connection Status, it will tell you what connections exist (Mail vs. Directory), what server they are going to and what protocol they are using (HTTPS vs. TCP(direct Exchange RPC connection)). I have conveniently omitted any discussion around certificates, since that can take up another few blog posts. As some would say, that is beyond the scope of this article and is left as an exercise to the reader. How do I know Outlook Anywhere is working? Simple... when no one is complaining! Seriously though, it is preferable is to run diagnostics on Outlook Anywhere before subjecting it to thousands of users. The one tool that works pretty well in most cases is rpcping. Yes, it has a lot of parameters and is confusing, but it does provide pretty good diagnostic information and as long as you have the KB open, you can figure out where problems lie. Start by pinging just the RPCProxy by using the -E option. Once that works, move onto testing the mailbox server endpoints by removing the -E and adding -e 6001 instead. Similarly for 6002, 6004. A typical command line would be something like this. Refer to http://support.microsoft.com/kb/831051 for usage details rpcping -t ncacn_http -o RpcProxy=cas1.contoso.com -P user,domain,password -H 1 -F 3 -a connect -u 9 -v 3 -s mailbox.contoso.com -I user,domain,password -e 6004 How does Outlook Anywhere not work? Unfortunately, there are some cases where Outlook Anywhere does not work without requiring manual tweaks. This is the part I wish I had blogged about earlier. I'm sure there are poor folks out there that have hit these issues and wasted their time figuring out what I had already learned... DSProxy and IPv6 As of E12 SP1, Outlook Anywhere on Windows 2008 requires that IPv6 be manually turned off on the CAS server. This is because the DSProxy component that listens on port 6004 (mad.exe) for directory connections does not listen on the IPv6 stack. If you do a netstat -ano | findstr 6004, you will see only 1 LISTENING entry - the one that corresponds to the IPv4 stack. Contrast this with ports 6001 and 6002 that have 2 entries. (As most of you already know, if you are running your Mailbox role on the same machine as a DC, lsass.exe not mad.exe listens on port 6004, so this problem will not surface since lsass.exe listens on both protocol stacks.) How do you turn off IPv6 ? It depends on whether you are running CAS and Mailbox on the same server or different ones. If you're in a multi-server scenario where the RPCProxy is not on the same server as the Mailbox, then you need to do the following: Unselect IPv6 from the properties of your NIC (on the RPC-over-HTTP Proxy machine); that will force the RPC-over-HTTP Proxy to use IPv4 to talk to Exchange and everything will be fine. In most cases, this step suffices. If it does not, continue with steps 2 and 3. Under the regkey HKLMSYSTEMCurrentControlSetServicesTcpip6Parameters, add a 32 bit DWORD with the name Disabled Components and value 0xFF Reboot the machine If you're in a single-server scenario where the RPCProxy and Mailbox are on the same machine, then the above does not work since the loopback interface still uses IPv6. In this case, you need to make the following changes in the system32driversetchosts file: Comment out the line :::1 localhost Add the following two lines: IPv4 address hostname of the computer IPv4 address FQDN of the computer Thanks to Kevin Reeuwijk and others for finding and reporting the issue and solution. A fix (make DSProxy listen on the IPv6 stack) is on the way and is expected to be available in Exchange 2007 SP1 RU4 in Q3 2008. DSProxy and Split RPC_IN_DATA, RPC_OUT_DATA connections In the diagram above, you will notice that I have used a Source IP Loadbalancing layer. This ensures that the RPC_IN_DATA and RPC_OUT_DATA connections coming from a single Outlook instance are always affinitized to the same CAS server. However, there are some legitimate scenarios where Source IP affinity is not viable for customers. A typical example is when a large number of end users are behind NAT devices causing all connections to end up with the same IP and hence the same CAS server... yay load balancing! Outlook Anywhere does not support cookies, so cookie based Load balancing cannot be used either. The only way of spreading load across the server farm is to use with no affinity or SSL-ID based affinity. However, this has the problem that the RPC_IN_DATA and RPC_OUT_DATA connections could (and most likely would) end up on different CAS servers as shown in the diagram below: If you've been reading closely, you'll remember my earlier mention that the RPC server component is well aware of client session IDs and can reply on RPC_OUT_DATA for any requests on RPC_IN_DATA. And if that's the case, we should still be fine since Outlook always specifies the mailbox server as it's RPC endpoint. Well, almost. We are fine for ports 6001 and 6002 which are real RPC end points. The issue is with port 6004 where DSProxy pretends to be an RPC endpoint, but is just a proxy as the name implies. DSProxy only proxies client connections through to the DC. In the example above, RPC_IN_DATA is proxied to DC1 while RPC_OUT_DATA is proxied to DC2. The DCs are the real RPC endpoints. However, now that the 2 connections have been split, neither of the DCs is aware of the other connection and requests sent on RPC_IN_DATA are lost in oblivion. We call this split connectivity and it is a problem surfaced by SSL-ID or no affinity load balancing. While I would recommend not using these configurations if avoidable, it is clear as described earlier that these may be the only alternatives. Think hard if this is the case since the workaround that I am describing below will be tedious to maintain. The goal of these steps is to eliminate the possibility of split connectivity by (1) having clients bypass DSProxy wherever possible and (2) constrain DSProxy to talking to a single DC for any requests to DSProxy. First off, you need to avoid using DSProxy wherever possible. Normally, the Referral Service running on port 6002 refers clients to DSProxy on port 6004. By setting the following regkey, you instruct Referral Service to not send clients to DSProxy, but instead give them a referral to a DC for directory connections. So, instead of client connections going from Client to RPCProxy to DSProxy to DC, the path would be from Client to RPCProxy to DC. Note that the client is not directly connecting to the DC, so it is not required to publish the DCs to the internet or open any new firewall ports. See KB http://support.microsoft.com/kb/872897 for details: On the Mailbox servers: a DWORD entry needs to be created on each Mailbox server named Do Not Refer HTTP to DSProxy at HKLMSystemCCSServicesMSExchangeSAParameters and the value set to 1 Next, as indicated earlier, the RPCProxy will block access to the DC servers unless there servers are included in the ValidPorts regkey. So, set the following on the Client Access Servers The ValidPorts setting at HKLMSoftwareMicrosoftRPCRPCProxy needs setting so that the entries referring to 6004 point to DC servers in addition to the mailbox server. The PeriodicPollingMinutes key at HKEY_LOCAL_MACHINESystemCurrentControlSetServicesMSExchangeServiceHostRpcHttpConfigurator needs setting to zero to prevent RpcHttpConfigurator from updating the Valid Ports key automatically. Finally, you need to make sure that the DCs are listening on port 6004: On the Global Catalog servers: a REG_MULTI_SZ entry needs to be created on each GC named NSPI interface protocol sequences at HKLMSystemCCSServicesNTDSParameters and the value set to ncacn_http:6004 These fixes will make sure that all directory connections bypass DSProxy and terminate at the DCs, thereby allowing the DC RPC server side component to receive both the RPC_IN_DATA and RPC_OUT_DATA connections. There is 1 last thing to deal with in this SSL-ID load balanced configuration. Outlook profile creation hard codes a call to DSProxy on 6004. Which means that we can get split connectivity during profile creation. To deal with this minimal volume of traffic, there is 1 final regkey that should be set on the mailbox servers: On the Mailbox Servers - set the HKLMSystemCCSServicesMSExchangeSA Parameters key NSPI Target Server to the