What Installation Authors Need to Know About COM Extraction

By Josh Stechnij

COM extraction was originally created to be a tool for populating Windows Installer COM/registry table data without having to manually determine and populate data by hand from a given COM server. Windows Installer provides the Class, Registry, Typelib (deprecated), and ProgId tables to automatically install COM server registration data in a way that allows for consistent rollback on failed installations and also provides a way for Windows Installer to hook into application usage to verify installation consistency (install on demand/self-resiliency). Over the years since COM extraction was first provided, InstallShield has updated COM extraction functionality in an attempt to provide more accurate extraction and useful functionality. This white paper aims to provide some background on how COM extraction works in InstallShield and some challenges that have come up when attempting to use COM extraction.

COM Extraction Methods Employed in InstallShield

In order to extract COM information from a COM server, some method needs to be used to capture COM data from the file. While static analysis of the file (type library and registration scripts) could be used, this is limited in functionality since it can only capture declarative data that is contained in obvious places in the file. More flexible methods could be used through dynamic capture of registration data as it is written to the machine while a file is being registered. Since the introduction of COM extraction in InstallShield, a few different dynamic methods have been used in an attempt to provide the most accurate extraction.

API Hooking

The original support for COM extraction was implemented through the use of API hooks. When COM extraction was started for a given file, APIs in Kernel32 and Advapi32 were hooked by COM extraction to monitor registry modifications on the system. API hooking provided a relatively clean approach to capturing registry changes such that the system was affected minimally by the registration process. One downside to this approach was that the registration itself was not insulated from the system. If a given file was already registered on the machine prior to running COM extraction, this would alter how most COM registration proceeded (no changes to the system would typically occur since the registration took note that its information was already present on the machine). To work around this issue, the file to be registered is first unregistered. Then, the capture through API hooking is enabled and the file registration is started.

Capture of in-process COM servers such as DLLs, OCXs, etc. is a fairly simple process of hooking APIs in the COM extraction process. However, capturing registration information from out-of-process servers in the form of EXEs requires the use of a helper DLL that is injected into the process to capture. At the time this method of COM extraction was created, injecting DLLs into an arbitrary process was rather simple. Over time, with the introduction of more heavily secured operating systems such as Windows XP SP2, Windows 2003 Server, and Windows Vista, DLL injection became increasing difficult and less reliable. In addition, API hooking also became less reliable after COM servers started building with Visual Studio 2005 (likely due to changes in the registration implementations).

Registry Redirection

As API hooking became less reliable with changes in OS and development tools, a new way of capturing COM information was needed. Starting with Windows 2000, the operating system allows for redirection of all registry accesses to private keys through the use of the RegOverridePredefKey API. This API is expressly provided for the purpose of software installations to allow for capturing system registry changes.

Starting with InstallShield 12 (which provided mainstream support for Windows Vista), COM extraction defaulted to using registry redirection as a means of capturing COM registration data during COM extraction. This method provided a way to present a “clean” view of a system to the file being registered, eliminating the need to unregister the file prior to registering. This method resolved most outstanding issues occurring with API hooking that had cropped up over the years. In addition, this method has the advantage that the machine state prior to and after capturing registry information will not change. As with most solutions, though, some tradeoffs appeared. Since registry redirection presents a view of the registry that is available in the redirected keys (which tend to be completely empty), COM servers that depended on existing registry information would no longer capture correctly. Some workarounds to this were available, such as writing dummy ATL registry keys needed to register some components dependent on ATL. However, no generic solution to missing registry dependencies is available, and components that have such dependencies will not correctly have all COM information extracted.

Kernel Mode Registry Filtering

In order to resolve the issues brought about by registry redirection COM extraction while maintaining a similar level of system isolation, and to provide some additional support of .NET assemblies, InstallShield 2012 introduced a new method of extracting COM information. On Windows Vista and newer versions of Windows, InstallShield will use kernel mode registry filtering to capture registry data for COM extraction.

Kernel mode registry filtering is a Windows mechanism for intercepting system-wide registry accesses through a kernel mode driver. This support was first provided on a limited basis with Windows XP. Windows Vista improved registry filtering such that it was generally usable to capture registry information from any process running on a system. Registry filtering is used by utilities like Sysinternals’ Process Monitor to able to capture registry accesses across processes on a system.

Through registry filtering, the COM extraction provided in InstallShield 2012 and newer versions results in a hybrid approach that combines aspects of the API hook and registry redirection methods. A COM server being registered with register filtering COM extraction will have a view of the actual system registry like API hooking. The captured data is completely redirected by the filter driver to a private InstallShield registry key, similar to the registry redirection method. The end result is no DLL injection is required and a registering component has complete access to any registry dependencies it may have.

An additional benefit of registry filtering is being able to capture registry modifications from any process, not just a process that is registering COM data. This now allows for capturing registry changes for .NET COM interop made by regasm. Since creating a registry file with regasm does not include code-level registry changes made by an assembly, the registry filtering can now be used to capture registry changes made by registering an assembly for COM interop with regasm. In addition, C++/CLI assemblies that “register” can now also be captured correctly (previous COM extraction methods resulted in failures in the assembly or CLR that caused the extraction to fail).

Specifying Exclusions for COM Extraction

To prevent InstallShield from extracting undesired COM data using any of the available extraction methods (either at build or design time), the Filters.xml file contains a list of registry keys that need to be excluded. Filters.xml can be customized as needed. It is available in the Support folder of an installed copy of InstallShield (typically located in C:\ Program Files\InstallShield\).

The element in the Filters.xml file contains a subelement for each registry key that the COM extraction process should ignore. By default, the element has subelements for common system registry keys that are required.

Following is a sample of a registry subelement that blocks changes to an InprocServer32 registry key, all of its values, and all of its subkeys:

Following is a sample of a registry subelement that blocks changes to only the default value of an InprocServer32 registry key:

Following is a sample of a registry subelement that blocks changes to only the ThreadingModel value name for an InprocServer32 registry key: