Thursday, May 31, 2007

Fix for Non-Responsive WSUS Clients

WSUS 2.0 Behavior of WUAU
The WSUS 2.0 version of WUAU is version 5.8.0.2607. This version has an issue that causes the CPU to race to 98-100% utilization and causes the machine to become unresponsive during the WUAU detection cycle. The cycle normally runs for about 2 minutes.


The Windows Update Automatic Update service (WUAUSERV) runs under the SVCHOST process, along with many other processes. If you look in Task Manager you may see 5-8 processes named SVCHOST.EXE. To determine which process is running WUAU, enter TASKLIST /SVC at the CMD prompt. Output will look like this:

Image Name PID Services
========================= ====== ============================================
System Idle Process 0 N/A
System 4 N/A
smss.exe 864 N/A
csrss.exe 928 N/A
winlogon.exe 952 N/A
services.exe 1004 Eventlog, PlugPlay
lsass.exe 1016 Netlogon, PolicyAgent, ProtectedStorage,
SamSs
svchost.exe 1200 DcomLaunch, TermService
svchost.exe 1268 RpcSs
svchost.exe 1912 AppMgmt, AudioSrv, CryptSvc, Dhcp, dmserver,
ERSvc, EventSystem, helpsvc, lanmanserver,
lanmanworkstation, Netman, Nla, RasMan,
Schedule, seclogon, SENS, SharedAccess,
ShellHWDetection, srservice, TapiSrv,
Themes, W32Time, winmgmt, wuauserv, WZCSVC
svchost.exe 2040 Dnscache
svchost.exe 480 Alerter, LmHosts, RemoteRegistry, SSDPSRV,
WebClient
spoolsv.exe 664 Spooler
FrameworkService.exe 1568 McAfeeFramework
mcshield.exe 1668 McShield


In this example you can see that PID 1912 is running wuauserv (Windows Update Automatic Update Service).

WSUS 3.0 Behavior of WUAU
The fix for this behavior is to install the new Windows Update Agent 3.0 (WUAU version 7.0.6000.374) and the hotfix KB927891. Both the new agent and the hotfix must be installed to correct the bug. After both updates are installed and the computer is rebooted, SVCHOST will still take 96-99% of the CPU, but the computer will remain responsive. Occasionally, I've seen it perform a little sluggishly, but nothing like it was before the updates. Microsoft says they made "deep architectural improvements" to the WUAU process, which provide greater time slicing in the SVCHOST process. This allows other processes to run while WUAU is performing a detection cycle. This fix has worked for all my clients.


Links to WSUS 3.0 Standalone Agent
http://download.windowsupdate.com/v7/windowsupdate/redist/standalone/WindowsUpdateAgent30-x86.exe
http://download.windowsupdate.com/v7/windowsupdate/redist/standalone/WindowsUpdateAgent30-x64.exe
http://download.windowsupdate.com/v7/windowsupdate/redist/standalone/WindowsUpdateAgent30-ia64.exe

Required KB 927891 Hotfix
http://support.microsoft.com/?kbid=927891

Wednesday, May 30, 2007

That's better than some hybrids!

A 2006 study found that the average American walks about 900 miles per year. Another study found that Americans drink an average of 22 gallons of beer per year. That means, on average, Americans get 41 miles per gallon.
 

Getting Ready for Tech-Ed 2007!



It's that time again -- I'm off to Tech-Ed in Orlando! This will be my fourth Tech-Ed and the second time in Orlando. I'm hoping to blog about it daily while I'm there. I'll be attending mainly Exchange and Windows Server seminars, but I'll also see some Systems Center, Sharepoint, WSUS and security sessions as well. My schedule builder has as many as 8 simultaneous sessions booked, so I think I'll be busy enough.


As usual, I'll be packing a suitcase inside a suitcase so I'll have one to fill with swag and t-shirts. Parties are already booked for Tuesday and Wednesday, and of course there will be the Tech-Ed Attendee Party at Islands of Adventure on Thursday. Fun times! Here's a link to the IOA Unofficial Guide with lots of good information about the park.

Tuesday, May 29, 2007

Handy Way to Lookup KB Articles

Sometimes it seems I live in TechNet. Here's a registry entry you can use to easily look up a KB article from the Address bar in Internet Explorer:
  1. [HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\SearchUrl\KB]
  2. @="http://support.microsoft.com/?kbid=%s"

Save the two lines above (wrapped for clarity) in a REG file and double-click it to import it. Now just enter KB 834544 in the IE address bar to bring up the article.

Friday, May 25, 2007

"Do not deliver before" Behavior Doesn't Work as Expected in Outlook Cached Mode

Users sometimes schedule delivery of an email for a later date or time.

For Outlook 2000, there were two different options: Deferred Send (Outlook handles it) and Deferred Delivery (the Exchange server handles it). If a message is configured for Deferred Send, it will stay in the user's Outbox until the scheduled time. Outlook then submits the message to the Exchange Information Store and the message is delivered. If a message is configured for Deferred Delivery, Outlook will immediately submit the message to the Exchange Information Store and the Exchange server will hold the message until the scheduled time. With Deferred Send, Outlook must be running to send the message and the user can edit or remove the message before it's delivered. With Deferred Delivery, Outlook does not have to be running and the user cannot edit or remove the message before it's sent.

Microsoft merged the two features together for Outlook 2003/2007. The only option available in these versions is "Do not deliver before". If this is configured the message will stay in the user's Outbox, but Outlook does not need to be running to deliver it. By keeping it in the Outbox, the user is able to edit or remove the message before it's sent. However, if the user is configured for Exchange Cached Mode, Outlook MUST be running for delivery of message to occur. http://support.microsoft.com/?kbid=918824 says this behavior "by design".

On a side note, the message will show as Received in the Inbox at the delayed send time (say, 8:00am today). When you open the message, the Sent time will be the time the sender clicked the Send button (say, 5:00pm yesterday). This prevents a user from scheduling an email the night before saying, "I'm in the office this morning like you requested, but I'm going home now."

Monday, May 14, 2007

How to Debug Windows Memory Dumps

From time to time, we're faced with the dreaded BSOD, or bugcheck, on a Windows machine. The procedures below guide you through the steps necessary to analyze and debug dump files.

For a downloadable copy of these procedures, click here: How%20To%20Debug%20Memory%20Dumps.doc

  • Download and install the Microsoft Debugging Tools from http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx

  • Go to Start All Programs Debugging Tools For Windows WinDbg

  • Click on File Symbol File Path, enter:
    SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
    and click OK.

  • Click File Save Workspace so that your symbols path is saved for future use.

  • Now locate your memory dumps. Small memory dumps are usually located in %systemroot%\minidump and Kernel memory dumps are located in %systemroot%\MEMORY.DMP.

  • Go to File Open Crash Dump and load the file. You may get a message to save base workspace information. If so, choose No. Now you will get a debugging screen. It may take a little bit to run, since the symbols are downloaded as they are needed. Then you will see information such as:

Microsoft (R) Windows Debugger Version 6.7.0005.0
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [\\hoem02\c$\windows\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2003 Kernel Version 3790 MP (4 procs) Free x86 compatible
Product: Server, suite: TerminalServer SingleUserTS
Built by: 3790.srv03_gdr.050225-1827
Kernel base = 0xe0b49000 PsLoadedModuleList = 0xe0be66a8
Debug session time: Wed May 9 02:01:49.965 2007 (GMT-7)
System Uptime: 6 days 22:51:23.840
Loading Kernel Symbols
......................................................................................................
Loading User Symbols
PEB is paged out (Peb.Ldr = 7ffff00c). Type ".hh dbgerr001" for details
Loading unloaded module list
..
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck A, {4, 2, 0, e0b6136d}

Probably caused by : volsnap.sys ( volsnap!VspWriteVolumePhase35+3a )

Followup: MachineOwner
---------

  • So far, we can tell that the bugcheck was caused by volsnap.sys, which is the Microsoft volume shadow copy driver. Use !analyze -v to get detailed debugging information. The most useful information is at the top of the analysis:

*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)

An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses.

If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: e0b6136d, address which referenced memory

  • From here, we can tell that volsnap.sys tried to read memory from an IRQL that was too high. This is usually caused by a bad driver, in this case, volsnap.sys.

  • Next, let's find out what process was calling volsnap.sys. Enter !thread in the kd> command line input box and look for the line that begins with Owning Process:

2: kd> !thread
THREAD faa03658 Cid 0568.1954 Teb: 7ffac000 Win32Thread: 00000000 RUNNING on processor 2
Not impersonating
DeviceMap e1003978
Owning Process fc1913b0 Image: cvd.exe
Wait Start TickCount 38443765 Ticks: 0

  • Now enter !process fc1913b0 0 (the hex number of the Owning Process), a space and the number 0.

2: d> !process fc1913b0 0
PROCESS fc1913b0 SessionId: 0 Cid: 0568 Peb: 7ffff000 ParentCid: 0218
DirBase: dd4a3000 ObjectTable: e141a910 HandleCount: 475.
Image: cvd.exe

  • We can now tell that the cvd.exe process (used by Commvault) called the volsnap.sys driver. Since volsnap.sys is a Microsoft driver, a quick check on TechNet reveals that there is an updated VSS package available for our server (http://support.microsoft.com/kb/887827) which addresses the problem.

Note: Writing debugging information must be configured on the machine prior to the BSOD in order to get a memory dump. This is done in the Advanced tab of system properties. Set it to "Kernel memory dump" in order to get the process information.

Wednesday, May 9, 2007

Exchange 2007 Update Rollup 2 Available

Last month I wrote about the availability of Exchange 2007 Update Rollup 1. As I mentioned, the Exchange team was expecting to produce these rollups every 6-8 weeks, but Exchange 2007 Update Rollup 2 follows Update Rollup 1 by only 19 days.

Microsoft has changed the product service strategy for Exchange 2007. Details can be read at http http://support.microsoft.com/?kbid=937194, but the short story is that Microsoft will no longer produce traditional updates that you download and patch Exchange. The Update Rollups are more like service packs, where it's a complete reinstallation of the product with the patches incorporated. The web article above explains why this is an improvement and provides a more stable environment.

Now that we've all sipped the Kool-Aid, let me explain what this means in reality. Microsoft released Security Update for Exchange Server 2003 SP2 (KB931832), which is a critical security update that affects Exchange 2000, 2003 and 2007. The updates for E2K and E2K3 are 3.7MB. The Update Rollup 2 for E2K7 is 27-57MB, an increase of 730%-1,550% for the same patch. The size varies, depending on the roles installed. Keep in mind that you're going to have to download, distribute and deploy this to each E2K7 server, so plan your deployment and bandwidth accordingly.

A few other items of note:
  1. Deployment on an E2K3 server takes 2 minutes, stops and starts Exchange related services. Deployment on an E2K7 server takes 9 minutes and requires a server restart.
    Computers running only the E2K7 management tools require the same 27-57MB update as a full Exchange server. In most environments, administrators load the Exchange management tools on their DCs to facilitate user provisioning. You must now be aware of DC unavailability caused by this issue.
  2. Updates that would normally not need a system restart in previous versions will need one in E2K7.
  3. You will no longer have the option of not deploying a specific update, since all Update Rollups are cumulative.

Security Update for Exchange Server 2003 SP2 (KB931832) can be downloaded via Windows Update or from here.

Friday, May 4, 2007

High CPU Utilization on WSUS Clients

High CPU Utilization on WSUS Clients

Clients who uses WSUS may have noticed that when the client checks for updates the CPU climbs to 100% utilization and the computer becomes unresponsive. Microsoft released a two-tier fix for this.

First, install the hotfix listed in KB927891. There are specific versions for x86, x64 and Itanium platforms, as well as Server 2003 and XP.

Next, install the new WSUS 3.0 Agent software. This new agent will work with WSUS 2.0 or 3.0 servers.

After you install the new software you will still see the CPU peg at ~100% while the client checks for updates, but the computer will still be responsive. This is the result of "deep architectural performance optimizations" in the client software. My testing has shown substantial improvements in performance and shorter update times.