Exchange Management Console login issue with quota request exceeded - or "I'm giving her all (the Kerberos requests) she's got, Captain!"

I've been having issues with some of my Exchange servers being unable to open the Exchange Management Console.  When attempting to make the initial connection to the Exchange Management service on the specified server the following error is given:

The following error occurred while attempting to connect to the specified Exchange server 'xxxxx':
The attempt to connect to <server FQDN>/Powershell using "Kerberos" authentication failed:  Connecting to remote server failed with the following error message : The WS-Management service cannot process the request.  The system load quota of 1000 requests per 2 seconds has been exceeded.  Send future requests at a slower rate or raise the system quota.  The next request from this user will not be approved for at least 124275504 milliseconds.

Thanks to Jason Shave for his elegantly simple solution to this convuluted error:

The server had recently received a new SSL certificate using the Exchange 2010 certificate provisioning and assignment process in the GUI. Unfortunately the IIS service hadn't been restarted yet and the URL used for remote PowerShell was using a certificate which wasn't trusted or valid anymore.

A quick "IISRESET" on the server resulted in my fix.

You can see Jason's original post here regarding this issue.

Down-under Terminal Server connection woes 4 - or "You must *feel* the Force (packets) flowing through you!"

(The following is meant as humor, but it does describe the solution to this issue.)

While I was working on the terminal server issues described previously, I checked the Netgear DEVG2020 router the ISP had provided to us.  I have to say, the Netgear DEVG2020 is probably one of the most functional routers ever designed...

"What?  I can't say that?  Not passing packets on multiple connections, huh?  It was causing the terminal server multiple connection issue?  Really?  Well, OK, let me try it again."

The Netgear DEVG2020.  The router you need for security & performance...

"Not that either?  Multiple VPN connections were being blocked as well?  Oh, all multiple connections of certain types?  Issues with business applications as well?  Well..."

The Netgear DEVG2020.  A router based on industry standards for internet connectivity and....

"Again, really?!  Yeah, it has a customized firmware specific to each ISP?  Completely unsuitable for business-critical internet access?  Replaced as quickly as possible, huh?  OK."

I'm sorry, it looks like I won't be writing a review for the Netgear DEVG2020 this week as it was apparently the cause of almost every networking issue I've seen for the past few weeks.  It was replaced this past weekend by the ISP and all previously discussed issues have been fixed:

1. Unable to connect to the terminal server twice in a row
2. Unable to connect to the VPN twice in a row
3. Extremely slow Windows desktop logins
4. Business application connection issues

This explains why Microsoft was unable to see the packets for the second connections in the NetMon traces.  Turns out it wasn't Sonicwall blocking the packets, it was the Netgear DEVG2020 that was dropping the connections.

Doing more internet research, it appears there are many cases of issues with this specific router from various ISP's globally.  So the moral of the story is folks, if you happen to own a Netgear DEVG2020 router somewhere in your wide-area network, you may want to consider replacing it if at all possible.

"What a piece of junk!" - Netgear DEVG2020

Down-under Terminal Server connection woes 3 - or "I left my (packet) back in San Francisco"

Microsoft came back with an update after examining network traces from the client and the server.  Seems the initial 'scaling' connection works fine but when a second connection is attempted, it fails with no explanation.  The Wireshark trace was inconclusive so a NetMon trace was created and sent to Microsoft for analysis.

The NetMon packet trace confirmed what I already suspected, the client packets were not making it to the server (duh?).  Microsoft recommends taking a workstation outside the firewall and testing.  This would eliminate the firewall from the potential issues.

Sonicwall came back and recommended we change the VPN connection timeout (TCP & UDP) on the firewall.  This of course, had no effect as it is not the VPN connection having the issue.  The workstations in Australia can access the rest of the network fine.  It's just when they try to make a second connection to the terminal server that things go south.

The idea I'm floating around right now is this may have nothing to do with the Sonicwall after all, it could be related to the gateway device at this location provided by the ISP.  It has firewall capabilities and may be causing an issue with the Sonicwall.

I need a user connect to the wireless on the gateway device (thereby eliminating the firewall from the equation) and see if he is able to work correctly.  This would seem to point back to the firewall being an issue - unless the wireless side of the gateway doesn't go through the same processing as the LAN side of the gateway, where the Sonicwall is.

Things are continuing to develop - unfortunately the picture isn't any clearer...

(to be continued....)

Down-under Terminal Server connection woes 2 - or "What's love (of encryption) got to do with it?"

What's in a name?  What's in a service pack?  Two (almost) identical questions when it comes to the world of Microsoft - for all Microsoft updates are not the same.

In my testing of the Windows 2008 R2 SP1 terminal server issue described in the previous post, it came upon me to test to a different terminal server - but one still located in the same subnet as the problematic one.  This revealed something very interesting.

If I used a Windows 2008 R2 (base) terminal server, I could not re-produce the issue from the same client that I was getting with Windows 2008 R2 SP1.

So, this puts us back to a very interesting question then.  What is it about the SP1 install that causes our client to only be able to connect once and only once?  This was the question that I was going to have to resolve.

(to be continued again...)

Down-under Terminal Server connection woes - or "Throw another protocol error on the Barbie!"

Recently, I've had to deal with a rather bizarre terminal server issue.  At this one location, no computer could connect to a specific terminal server twice.  The computers could connect fine one time.  But, if the user logged off or disconnected, and then tried to connect to the terminal server again, the following error was displayed each time:

Your Remote Desktop sessions has ended.  The connection to the remote computer was lost, possibly due to network connectivity problems.  Try connecting to the remote computer again.  If the problem continues, contact your network administrator or technical support

Of course, you would think 'orphaned session' or a terminal server setting, but that was not the case.  No limits were set and I could see the users' sessions disconnecting just fine from the server.  The first connection would work fine until the user logged off the terminal server or was disconnected.  Once that happened, the user could not sign in again with the above error.

But here's the trick:  If I rebooted the server or the firewall at the location, the users could connect again - but again, only once, then another reboot would be required.

So after confirming the usual suspects like DNS, AD account status, and VPN tunnels were all active and working normally, I decided the issue had to be something deeper.  I found the following error in the Terminal Server's System Event Log:

"Event 56, TermDD - The Terminal Server security layer detected an error in the protocol stream and has disconnected the client."

This little Event ID led down a real rabbit-hole of blog posts, forum discussions, and random Microsoft KB articles.  Let me give you some of the highlights:

• Reduce the encryption level of the terminal server to Low & use "RDP Encryption"
• Set the RDP encryption algorithm to balance network & memory usage
• Enable 'keep alive' on the terminal server
• Disable TCP Chimney Offload, Receive-Side Scaling State (RSS), and NetDMA
• Confirm RDC client version is the latest on all clients
• Use "ERR.EXE" to analyze the last word byte of the above error (B50000D0 in this case)

No one online seemed to have the final solution and none of the suggestions helped me.  I put everything back the way it was, pulled my head away from the wall, and decided to just get down and dirty with a Wireshark trace.  Hopefully the trace would help figure out exactly what was happening with these failed connections.  Running a quick client trace gave me some errors but nothing definite.  Wireshark did report some checksum errors and this "dissector bug":

"Dissector bug, protocol T.124 proto.c:3478 failed assertion (guint)hfindex < gpa_hfinfo.len) unregistered hf!"

The checksum errors led me down the hardware stack to the network cards, turning off the "checksum offload" at the IP & TCP levels on the virtual host & virtual server.  This cleared up some of the Checksum errors in Wireshark but still the same terminal server error persisted.

I was still not convinced that the Sonicwall at the location wasn't to blame for all this.  After all, we had other network issues with a business application at that same location which had still not been fixed.

Bruised and beaten, I elected to open support tickets with Sonicwall and Microsoft and begin working this issue from each end with them....

(To be continued...)