Friday, June 3, 2011

Analysing a Crash Dump-Basics

One of the worst experiences we can have while working on Windows is the BSOD or Blue Screen of Death. Its also a difficult assignment for the support professionals to investigate these issues. Debugging crash dumps requires extensive knowledge about Memory Management and other Windows System internal concepts.


However there is something that pretty much every one can do to analyse a crash dump in its simplest form. If you are lucky enough then most of the issues can be solved by this simple step.


All that you need is install the Debugging tools for Windows.


You can download this tool from the following location (for Windows XP) : http://msdl.microsoft.com/download/symbols/debuggers/dbg_x86_6.11.1.404.msi

Once Installed, the first thing you need to do is add the Microsoft symbol server path. Follow the below steps


Go to "ALL Programs"-->Debugging Tools for Windows and then select WinDbg.

Select the File Menu-> and then Select Symbol File Path





Add the path : SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols and then hit OK





Now, the configuration part is over now you need to open the crash dump in the debugging tool.


Go to File menu and then select Open Crash dump--> Point to the location where the crash dump is saved in explorer and select the file. By default system crash dumps will be saved under %SystemRoot%\Minidump folder.






Once the dump file is loaded, you need to wait for some time for the debugger to download the symbol files. Once the symbol files are downloaded the debugger will look something similar to below screen shot




Just click on the ! analyze -v link and debugger will let you know the possible driver that has caused the crash. In the example below, the crash was caused by igxpmp32.sys file which is the file related to Intel Graphics driver.





BSOD issues are often caused by faulty or outdated drivers and you can solve these issues by looking for an updated driver in the corresponding vendor site.

Monday, May 30, 2011

Auditing feature in Windows

Thought of sharing my experience around auditing feature available in Windows, which helped me solve a critical issue that affected many users.

The issue was that the users started reporting that every day they were receiving an automatic update notification to install a specific patch. The user installs the patch and the next day he again gets notification to install the same patch.

The patch that was repeatedly being offered was MS10-026 (KB977816). This patch updates the following files.






On the problem machines I observed that the file L3codeca.acm was getting reverted back to the previous version (1.9.0.305).The scenario is that the user installs the patch, the file gets updated to 1.9.0.306 and then after some time, something is replacing the file with the older 1.9.0.305 version. The subsequent scan by the automatic update agent detects the required file version at a lower level and offers the patch to the machine. This loop continued and users had every reason to get frustrated.


Now, how to determine who is replacing the file l3codeca.acm with the previous version?


I decided to use the Audit feature available in Windows XP that allows an administrator to audit user access to files, folders, printers, registry etc.

Steps to Enable Auditing for the file l3codeca.acm

You can enable auditing through the group policy console.

1. Start Run, type gpedit.msc and press Enter.
2. Under Computer Configuration, look in Windows Settings Security Settings Local Policies Audit Policy, and enable Audit object access on both Success and Failure.

Once we have enabled auditing through policy, we can specify the file that needs to be audited by the following steps

1. In Windows Explorer, locate the file or folder you want to audit (C:\WINDOWS\system32\ l3codeca.acm).
2. Right-click the file and then click Properties.
3. Click the Security tab, and then click Advanced.
4. Click the Auditing tab, and then click Add.
5. In the Enter the object name to select box, type the name of the user or group whose access you want to audit. You can browse the computer for names by clicking Advanced, and then clicking Find Now in the Select User or Group dialog box.
6. Click OK.
7. Select the Successful or Failed check boxes for the actions you want to audit, and then click OK.
8. Click OK, and then click OK.

Okay, now we have everything ready to isolate the process/user trying to change the file.

After few days I got the event logs from the problem machine.
I analyzed the Security event log to search for the accesses to the file l3codeca.acm and following is what I found. At around 10:49 PM the process msiexec has accessed the file in question.





After analyzing the Application event logs to identify the application that triggered msiexec process, it was observed that at around the same time during which the above object access failure occurred, an msi instance of the application Audio_codec was triggered.




From the above analysis, it was pretty clear that the application Audio_codec is the culprit.

From all these data, it turned out that once the automatic update installs the patch and update the l3codeca.acm file, the Audio_Codec application does an msi repair when it finds the file to be not at the expected version, and reverts the file back to an older version. The next automatic update scan will detect the file to be at a vulnerable version and re offers the patch. This cycle continues in a loop.

The users were requested to uninstall the Audio_codec application and this solved thier annoying issue.

Automatic Updates Registry & Policies !

This time I thought of putting together the various registry entries and group policies available for Automatic Updates.

Configure Automatic Updates by Using Group Policy

Configure Automatic Updates
The settings for this policy enable you to configure how Automatic Updates works. You must specify that Automatic Updates download updates from the WSUS server rather than from Windows Update.
To configure the behavior of Automatic Updates
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Configure Automatic Updates.

Click Enabled and select one of the following options:

Notify for download and notify for install. This option notifies a logged-on administrative user prior to the download and prior to the installation of the updates.
Auto download and notify for install. This option automatically begins downloading updates and then notifies a logged-on administrative user prior to installing the updates.
Auto download and schedule the install. If Automatic Updates is configured to perform a scheduled installation, you must also set the day and time for the recurring scheduled installation.
Allow local admin to choose setting. With this option, the local administrators are allowed to use Automatic Updates in Control Panel to select a configuration option of their choice. For example, they can choose their own scheduled installation time. Local administrators are not allowed to disable Automatic Updates.
Click OK.


Specify Intranet Microsoft Update Service Location
The settings for this policy enable you to configure a WSUS server that Automatic Updates will contact for updates. You must enable this policy in order for Automatic Updates to download updates from the WSUS server.

Enter the WSUS server HTTP(S) URL twice, so that the server specified for updates is also used for reporting client events. For example, type http(s)://servername in both boxes. Both URLs are required.

To redirect Automatic Updates to a WSUS server
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Specify Intranet Microsoft update service location.
Click Enabled and type the HTTP(S) URL of the same WSUS server in the Set the intranet update service for detecting updates box and in the Set the intranet statistics server box. For example, type http(s)://servername in both boxes.
Click OK.

Enable Client-side Targeting
This policy enables client computers to self-populate computer groups that exist on the WSUS server.

If the status is set to Enabled, the specified computer group information is sent to WSUS, which uses it to determine which updates should be deployed to this computer. This setting is only capable of indicating to the WSUS server which group the client computer should use. You must actually create the group on the WSUS server.

If the status is set to Disabled or Not Configured, no computer group information will be sent to WSUS.

To enable client-side targeting
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Enable client-side targeting.
Click Enabled and type the name of the computer group in the box.
Click OK.

Reschedule Automatic Update Scheduled Installations


This policy specifies the amount of time for Automatic Updates to wait, following system startup, before proceeding with a scheduled installation that was missed previously.

If the status is set to Enabled, a scheduled installation that did not take place earlier will occur the specified number of minutes after the computer is next started.

If the status is set to Disabled, a missed scheduled installation will occur with the next scheduled installation.

If the status is set to Not Configured, a missed scheduled installation will occur one minute after the computer is next started.

This policy applies only when Automatic Updates is configured to perform scheduled installations of updates. If the Configure Automatic Updates policy is disabled, this policy has no effect.

To reschedule Automatic Update scheduled installation

In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Reschedule Automatic Update scheduled installations, click Enable, and type a value in minutes.
Click OK.

No Auto-restart for Scheduled Automatic Update Installation Options


This policy specifies that to complete a scheduled installation, Automatic Updates will wait for the computer to be restarted by any user who is logged on, instead of causing the computer to restart automatically.

If the status is set to Enabled, Automatic Updates will not restart a computer automatically during a scheduled installation if a user is logged on to the computer. Instead, Automatic Updates will notify the user to restart the computer in order to complete the installation.

Be aware that Automatic Updates will not be able to detect future updates until the restart occurs.

If the status is set to Disabled or Not Configured, Automatic Updates will notify the user that the computer will automatically restart in 5 minutes to complete the installation.

This policy applies only when Automatic Updates is configured to perform scheduled installations of updates. If the Configure Automatic Updates policy is disabled, this policy has no effect.

To inhibit auto-restart for scheduled Automatic Update installation options
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click No auto-restart for scheduled Automatic Update installation options, and set the option.
Click OK.

Automatic Update Detection Frequency

This policy specifies the hours that Windows will use to determine how long to wait before checking for available updates. The exact wait time is determined by using the hours specified here, minus 0 to 20 percent of the hours specified. For example, if this policy is used to specify a 20-hour detection frequency, then all WSUS clients to which this policy is applied will check for updates anywhere between 16 and 20 hours.

If the status is set to Enabled, Automatic Updates will check for available updates at the specified interval.

If the status is set to Disabled or Not Configured, Automatic Updates will check for available updates at the default interval of 22 hours.

To set Automatic Update detection frequency
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Automatic Update detection frequency, and set the option.
Click OK.

Allow Automatic Update Immediate Installation


This policy specifies whether Automatic Updates should automatically install certain updates that neither interrupt Windows services nor restart Windows.

If the status is set to Enabled, Automatic Updates will immediately install these updates after they have been downloaded and are ready to install.

If the status is set to Disabled, such updates will not be installed immediately.

To allow Automatic Update immediate installation
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Allow Automatic Update immediate installation, and set the option.
Click OK.

Delay Restart for Scheduled Installations


This policy specifies the amount of time for Automatic Updates to wait before proceeding with a scheduled restart.

If the status is set to Enabled, a scheduled restart will occur the specified number of minutes after the installation is finished.

If the status is set to Disabled or Not Configured, the default wait time is five minutes.

To delay restart for scheduled installations
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Delay restart for scheduled installations, and set the option.
Click OK.

Re-prompt for Restart with Scheduled Installations
This policy specifies the amount of time for Automatic Updates to wait before prompting the user again for a scheduled restart.

If the status is set to Enabled, a scheduled restart will occur the specified number of minutes after the previous prompt for restart was postponed.

If the status is set to Disabled or Not Configured, the default interval is 10 minutes.

To re-prompt for restart with scheduled installations
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Re-prompt for restart with scheduled installations, and set the option.
Click OK.

Allow Non-administrators to Receive Update Notifications


This policy specifies whether logged-on non-administrative users will receive update notifications based on the configuration settings for Automatic Updates. If Automatic Updates is configured, by policy or locally, to notify the user either before downloading or only before installation, these notifications will be offered to any non-administrator who logs onto the computer.

If the status is set to Enabled, Automatic Updates will include non-administrators when determining which logged-on user should receive notification.

If the status is set to Disabled or Not Configured, Automatic Updates will notify only logged-on administrators.

To allow non-administrators to receive update notifications
In Group Policy Object Editor, expand Computer Configuration, expand Administrative Templates, expand Windows Components, and then click Windows Update.
In the details pane, click Allow non-administrators to receive update notifications, and set the option.
Click OK.

Remove Links and Access to Windows Update


If this setting is enabled, Automatic Updates receives updates from the WSUS server. Users who have this policy set cannot get updates from a Windows Update Web site that you have not approved. If this policy is not enabled, the Windows Update icon remains on the Start menu for local administrators to visit the Windows Update Web site. Local administrative users can use it to install unapproved software from the public Windows Update Web site. This happens even if you have specified that Automatic Updates must get approved updates from your WSUS server.

To remove links and access to Windows Update
In Group Policy Object Editor, expand User Configuration, expand Administrative Templates, and then click Start Menu and Taskbar.
In the details pane, click Remove links and access to Windows Update, and set the option.
Click OK.

Configuring Automatic Updates by editing the registry



The Automatic Updates configuration information is stored under HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU

Value name: NoAutoUpdate
Value data: 0 or 1

0: Automatic Updates is enabled (default).
1: Automatic Updates is disabled.
Registry Value Type: Reg_DWORD

Value name: AUOptions
Value data: 1 to 4
1: Keep my computer up to date has been disabled in Automatic Updates.
2: Notify of download and installation.
3: Automatically download and notify of installation.
4: Automatically download and scheduled installation.
Registry Value Type: Reg_DWORD

Value name: ScheduledInstallDay
Value data: 0 to 7
0: Every day.
1 through 7: The days of the week from Sunday (1) to Saturday (7).
Registry Value Type: Reg_DWORD

Value name: ScheduledInstallTime
Value data: n, where n equals the time of day in a 24-hour format (0-23).
Registry Value Type: Reg_DWORD

Value name: UseWUServer
Value data: Set this value to 1 to configure Automatic Updates to use a server that is running Software Update Services instead of Windows Update.
Registry Value Type: Reg_DWORD

Value name: RescheduleWaitTime
Value data: m, where m equals the time to wait between the time Automatic Updates starts and the time it begins installations where the scheduled times have passed. The time is set in minutes from 1 to 60, representing 1 minute to 60 minutes)
Registry Value Type: Reg_DWORD


Value name: NoAutoRebootWithLoggedOnUsers
Value data: Reg_DWORD: 0 (false) or 1 (true). If set to 1, Automatic Updates does not automatically restart a computer while users are logged on.
Registry Value Type: Reg_DWORD

Value name:DisableWindowsUpdateAccess
Stores configuration data for the policy setting Remove access to use all Windows Update features
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\WindowsUpdate Registry Value Type: Reg_DWORD
DisableWindowsUpdateAccess
Value data: 1 to Disable Access

Value name : NoWindowsUpdate
Stores configuration data for the policy setting Remove links and access to Windows Update.
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer
Registry Value Type: Reg_DWORD

Slow Boot / Logon Performance

Some of the Microsoft blogs related troubleshooting Boot / Logon Performance


http://blogs.technet.com/b/instan/archive/2008/10/21/time-travel-and-factors-that-increase-client-startup-or-login-time.aspx

http://blogs.technet.com/b/instan/archive/2008/04/17/troubleshooting-the-intermittent-slow-logon-or-slow-startup.aspx

http://blogs.technet.com/b/askperf/archive/2008/10/14/help-i-m-stuck-at-applying-computer-settings.aspx


http://blogs.technet.com/b/askds/archive/2009/09/23/so-you-have-a-slow-logon-part-1.aspx


http://blogs.technet.com/b/askds/archive/2009/09/24/so-you-have-a-slow-logon-part-2.aspx


221833 How to enable user environment debug logging in retail builds of Windows
http://support.microsoft.com/kb/221833


http://blogs.technet.com/b/askds/archive/2008/11/11/understanding-how-to-read-a-userenv-log-part-1.aspx


http://blogs.technet.com/b/askds/archive/2008/11/11/understanding-how-to-read-a-userenv-log-part-2.aspx

The case of mysterious reboots

I got the chance to work on an interesting case some time back and thought its worth for a blog.

The issue reported to me was that machines in Czech republic site were continuosly rebooting after they were migrated to our company managed domain.This issue was observed only in this country , and the same process was succesfully implemented for many other sites round the globe.The major difference is Czech republic site is using native MUI pack, which after uninstalling, the reboot issue disappeared. Okay, thats pretty much a description of what the issue was around.

I started my investigation and connected remotely to a problem machine. After waiting for some time I saw the issue and got the following message.





The error was related to services.exe crash and once I cliked Ok, the machine gets rebooted. If I cancel the error, machine would apparently turn into an unusable state which I guess is expected because some exception has occured in the memory. As there are many services that comes under the umbrella of "Services.exe" process my next effort was to identify the real culprit causing this issue.





The first thought that came to my mind was to use Process Explorer from sysinternals. Process Explorer lets you see the threads that are executing with an Process. In this way I could see the threads that are running when services.exe is crashing.



I compared the threads that were running when the machine was normal and suspected that it could be the ESENT.dll and Userenv.dll that could be the ones responsible for the issue. The group policy engine is contained within the Userenv.dll which runs inside winlogon.exe. Hence all my focus now turned to Group policy client side extensions. I checked the event log files for errors related to group policy and found the following

Userenv 1085 The Group Policy Client Side extension Security failed to execute
A further look at the Security policy events showed the following

The security policies were failing with the following error

Event : Security policies are propagated with warning. 0x428 : An exception occurred in the service when handling the control request.

Hence it was now clear that an exception occured while processing some security polices and this caused the services.exe to crash. So every time there was a group policy refresh on the machine, it rebooted

I then tried to enable verbose logging for security client side extension in an effort to check the policies that were getting applied. You can enable verbose logging by adding the DWORD value
ExtensionDebugLevel = REG_DWORD 0x2 under HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\GpExtensions\{827d319e-6eac-11d2-a4ea-00c04f79f83a}\
and the log will be created under %windir%\security\logs\winlogon.log


But I was not able to get much clue from the winlogon.log and the only thing I observed was that on a machine without MUI, I could see information on all the policies that were applied and on the problem machine I found only half of them and it got stuck in the middle of processing.
A further search for the 0X428 error lead to the following links :



The case of screen saver timeout resetting to 10 minutes

One of the most amazing feautures I have seen in Process Monitor tool is its capability to capture system boot time events. Here I am going to write about a case where I have used this utility to find the root cause of an interesting issue.

The issue was that the screensaver timeout setting was resetting to a value of 10 minutes every time a user reboots his machine. The user would manually change the timeout value to a higher one only to find it reverted back to 10 minutes after a reboot. Users were annoyed and the case was assigned to me for investigation to find the root cause of this behavior.

I started my investigation suspecting that it could be a group policy setting that doesn’t allow the setting to persist a reboot. The fact that even a logoff would revert back the setting only accentuated my suspicion. I verified the resultant set of policies applied on the machine (using the command gpresult /z >gpresult.txt) and exported the output to a text file for offline analysis. However I couldn’t find any entries related to screensaver settings in the group policy log. I also observed that a gpupdate /force command was not reverting the setting which forced me to leave group policy and I knew I had a bigger task ahead.

The registry key that determines the group policy timeout value is HKEY_CURRENT_USER\Control Panel\Desktop\ScreenSaveTimeOut.

So something is changing the registry key value from a user set value to a value of 600 (10 minutes).The question is who is responsible for this change and how to catch the culprit it. :-)

I decided to use Procmon’s Boot logging capability which captures all the events during a system boot. I launched Procmon , set it to capture boot time events. I also set the timeout value to 100 minutes.

Procmon’s boot log is huge in size and has data about thousands of events. However it has a very efficient filtering mechanism and all I had to do was set a filter to display events about the above registry key.


I found the following very interesting entry in the Procmon log.

It was evident from the Process Monitor boot capture, that the process P95Tray.exe sets the registry HKEY_CURRENT_USER\Control Panel\Desktop\ScreenSaveTimeOut to a value of 600 (10 Minutes) every time a log off or reboot happens.


P95Tray.exe is related to the CheckPoint encryption software and who would imagine an encryption software would reset the screensaver setting :-). This process was configured to run as a system startup item.It was also further confirmed that once we disable the Pointsec Tray from the startup list the screensavertimeout setting doesn't revert back to 10 minutes.

That pretty much explains how powerful a tool can be and hope you have enjoyed reading :-)