Python 2.4 Extensions w/ the MS Toolkit Compiler

This document describes the process of building Python 2.4 binary extensions with the (free) Microsoft Visual C++ Toolkit Compiler, which is an optimising native command-line compiler for Microsoft Windows.  The page is intended for use by extension developers who would like to provide binary Python packages for use by Microsoft Windows end-users, and assumes quite some familiarity with development tools.  It also tends to assume that you are using distutils for extension setup.  It is not really targeted at end-users.

Note that this document represents the findings of a very limited set of tests with what is essentially an unsupported approach to building extensions.  You should expect to find bugs.  No warranty is made that the approach outlined is safe or appropriate.

Note also that the author of the document (Mike Fletcher) has abandoned Windows as a development platform (in some part because of the difficulties in getting a working compiler to support the platform), and is thus no longer actively maintaining this page.  While some people are still reporting successes, others are having difficulties with building certain projects.  The Toolkit approach is, in short, a bit dicey as a platform for serious development.

Getting the Packages

The Microsoft Toolkit Compiler doesn't come out-of-the-box with everything you need to compile extensions.  There are 5 packages you will need to acquire and install in order to begin making Python 2.4 extensions.

• Microsoft Visual C++ Toolkit Compiler (a.k.a. MS Toolkit Compiler)
  
• The core compiler and linker, these are the command-line tools which do the actual work.  You want the 2003 version of the compiler (version 7.1) if you can find it... which has become so hard I can't actually do it any more from Microsoft's website.
  
• Microsoft Platform SDK (Internet Explorer (or Mozilla + Prefbar) required to download)
  
• Provides the headers and libraries required for linking to Microsoft system services

  Why not just use the compiler in the Microsoft platform SDK?  It turns out that this compiler is a non-optimising compiler, the same as is included in MSVC Standard Edition (the Professional Version is the one with the optimising compiler).  Reports of late suggest that the platform SDK no longer contains a compiler, that it is now the Visual Studio Express package that has the non-optimising compiler.
  

• .NET Runtime (Get from Windows Update)
• Required by the .NET Framework
  
• .NET Framework SDK Version 1.1
Provides the core msvcrt.lib for msvcr71.dll against which to link your extensions.  This is critically important, as without it you are going to wind up linking against the wrong run-time and will see crashes whenever a core object such as a file is shared across run-time barriers.

The 2.0 beta also works according to reports, and may be required to build Numpy 23.6 (whereas 23.1 seems to work without it).  I (Mike) have never been able to get 2.0 to work, and would advise using the 1.1 version which is known to work properly.


• Python 2.4 (Python.org version)
• This version is built with Visual Studio 7.1, and so does not work with extensions compiled with Visual Studio 6.0.  The Toolkit Compiler is apparently the same as that found in Visual Studio 7.1

You will need to install all of these packages in order to do any compilation.

Configuration

Now we need to set up a build environment which points to all the various executables, headers and library files we will be using.  The Toolkit compiler does not AFAIK use the registry for storing any directory information (though it does store basic installation keys).  Since the normal mechanism for setting up the Toolkit is to run the vcvars.bat file, we'll copy that file and make changes to it which set up the various extra paths we need.  Here's a sample batch file, vc7.bat:

rem vc7.bat, copied to the command path for the machine
@echo off

Set PATH=C:\Program Files\Microsoft Visual C++ Toolkit 2003\bin;C:\Program Files\Microsoft SDK\Bin;c:\bin\lang\py24;%PATH%
Set INCLUDE=C:\Program Files\Microsoft Visual C++ Toolkit 2003\include;C:\Program Files\Microsoft SDK\include;c:\bin\lang\py24\include;%INCLUDE%
Set LIB=C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\lib;C:\Program Files\Microsoft Visual C++ Toolkit 2003\lib;C:\Program Files\Microsoft SDK\Lib;c:\bin\lang\py24\libs;%LIB%

echo Visit http://msdn.microsoft.com/visualc/using/documentation/default.aspx for
echo complete compiler documentation.

The path variable needs to point to %Toolkit%/bin, %PlatformSDK%/bin, and the Python 2.4 installation directory (that last is optional, but makes things easier if you're working on a machine which normally has a different Python on the path).

The include path needs to point to %Toolkit%/include, %PlatformSDK%/include, and %Python24%/include.

For the lib path, the "Microsoft Visual Studio .NET 2003" directory referenced above is created as a side-effect of one of our installs and includes the all-important msvcr71.lib file, we also need to add the %Toolkit%/lib, %PlatformSDK%/lib and %Python24%/libs directories to the path.

There may still be required utilities/libs/header directories that this configuration doesn't include.  If you find one, please let me know so I can add them to this page.

Now, when we want to use the Toolkit compiler, we run vc7 at the command prompt and our build environment is configured.  Now we need to teach Python's distutils how to work with this compiler.

Distutils Modifications

Distutils doesn't know how to deal with the MS Toolkit Compiler, so we need to modify it.  At the moment we're doing this by actually modifying the %Python24%/lib/distutils/msvccompiler.py file.  Eventually the changes could be wrapped into distutils, either as a sub-class of msvccompiler, or by altering the base file.

The differences are quite minor, however, different versions are required for the 2.4 alpha msvccompiler and 2.4 final.  Here are the 2.4 final diffs:

*** msvccompiler-final-original.py	Mon Nov 29 08:58:32 2004
--- msvccompiler.py	Fri Dec  3 15:31:54 2004
***************
*** 126,132 ****
          self.set_macro("FrameworkDir", net, "installroot")
          try:
              if version > 7.0:
!                 self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1")
              else:
                  self.set_macro("FrameworkSDKDir", net, "sdkinstallroot")
          except KeyError, exc: #
--- 126,138 ----
          self.set_macro("FrameworkDir", net, "installroot")
          try:
              if version > 7.0:
!                 try:
!                     self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1")
!                 except KeyError:
!                     # likely using free Command-line compiler with free SDK
!                     freeSDK = r"SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\63DADB24-DC99-45EB-A748-EC93AB8A7497"
!                     # following should raise key error if not available...
!                     self.set_macro( "FrameworkSDKDir", freeSDK, 'install dir' )
              else:
                  self.set_macro("FrameworkSDKDir", net, "sdkinstallroot")
          except KeyError, exc: #
***************
*** 571,576 ****
--- 577,595 ----
                          "You must at least run the Visual Studio GUI once "
                          "so that these entries are created.")
                  break
+         if self.__version >= 7:
+             # well, what if we've already set the environment variables?
+             map = {
+                 'library dirs': 'lib',
+                 'path dirs': 'path',
+                 'include dirs': 'include',
+             }
+             path = map.get( path )
+             if os.environ.get( path ) is not None:
+                 return string.split(
+                     os.environ.get( path ),
+                     os.pathsep,
+                 )
          return []
  
      def set_path_env_var(self, name):

The first change relies on the GUID of the 1.1 .NET Framework SDK.  Obviously that's not going to work if you install a different version of the SDK.  If someone has a better approach, let me know. 

Martin Bless reports that for his XP SP2 SDK the GUID is: B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05 rather than that in the patch.

The second change should be fairly robust, in fact, it should even allow for using Visual Studio 6 from the command line if you haven't yet run the IDE (a limitation of the current code) but have run vcvars.bat.

You can either manually edit the file or use the unix-style patch utility to integrate the patch (the patches above were made against the msvccompiler.py file in Python-2.4a1 and Python-2.4 (final)).

Armin Rigo reports that he needs to use 'library' instead of "library dirs" in the second half of the patch.  My own system doesn't seem to require this, so I've left the patch as-is for now.  My reading of the code is that path will always have ' dirs' appended to it.

From another report: it's possible to replace the distutils.msvccompiler.read_values with a simple function that simply returns the values from the shell environment, (with only the toolkit installed you will always get the values from the environment), but I don't recommend that approach, as at some point you may install MSVC and find everything messed up.

Compiling your Extension

At this point, you should be able to run vc7.bat from the command prompt, then run python setup.py bdist_wininst and have the MS Toolkit Compiler be used to build your extension for Python 2.4 and turn it into a binary redistributable for MS Windows end-users.

If you succeed, fail, have suggestions for enhancements, or need clarification, let me know and I'll try to integrate the feedback into this document.  You will notice that there are a large number of warnings printed about soon-to-be obsolete compiler flags.  At some point someone will need to track those down and see if we can eliminate them from distutils.

Success/Failure Matrix

This table is to collect the set of successes/failures encountered in attempting to build various extensions with the setup outlined here:

Other Resources

• SAPDB Wiki Page
• Fairly detailed look at the various elements and features needed, not much focus on how to set up the environment, and lots of notes for missing things that don't seem to relate to Python extension writing/porting
• Building Python with the MSToolkit Compiler
• Post to Python-dev regarding how to build Python itself (as distinct from extension modules) using the Toolkit compiler
  
• TCL Wiki Page
• Minimal instructions on how to build TCL with the MS Toolkit Compiler
• MingW and Cygwin
  
• Free (libre) software for use in compiling extensions for Win32

Copyright 2004 Mike Fletcher -- All trademarks property of their respective owners