Python 2.4 Extensions w/ the MS Toolkit Compiler

This document describes the process of building Python 2.4 binary extensions with the (free) Microsoft Visual C++ Toolkit Compiler, which is an optimising native command-line compiler for Microsoft Windows.  The page is intended for use by extension developers who would like to provide binary Python packages for use by Microsoft Windows end-users, and assumes quite some familiarity with development tools.  It also tends to assume that you are using distutils for extension setup.  It is not really targeted at end-users.

Note that this document represents the findings of a very limited set of tests with what is essentially an unsupported approach to building extensions.  You should expect to find bugs.  No warranty is made that the approach outlined is safe or appropriate.

Note also that the author of the document (Mike Fletcher) has abandoned Windows as a development platform (in some part because of the difficulties in getting a working compiler to support the platform), and is thus no longer actively maintaining this page.  While some people are still reporting successes, others are having difficulties with building certain projects.  The Toolkit approach is, in short, a bit dicey as a platform for serious development.

Getting the Packages

The Microsoft Toolkit Compiler doesn't come out-of-the-box with everything you need to compile extensions.  There are 5 packages you will need to acquire and install in order to begin making Python 2.4 extensions.

You will need to install all of these packages in order to do any compilation.

Configuration

Now we need to set up a build environment which points to all the various executables, headers and library files we will be using.  The Toolkit compiler does not AFAIK use the registry for storing any directory information (though it does store basic installation keys).  Since the normal mechanism for setting up the Toolkit is to run the vcvars.bat file, we'll copy that file and make changes to it which set up the various extra paths we need.  Here's a sample batch file, vc7.bat:

rem vc7.bat, copied to the command path for the machine
@echo off

Set PATH=C:\Program Files\Microsoft Visual C++ Toolkit 2003\bin;C:\Program Files\Microsoft SDK\Bin;c:\bin\lang\py24;%PATH%
Set INCLUDE=C:\Program Files\Microsoft Visual C++ Toolkit 2003\include;C:\Program Files\Microsoft SDK\include;c:\bin\lang\py24\include;%INCLUDE%
Set LIB=C:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\lib;C:\Program Files\Microsoft Visual C++ Toolkit 2003\lib;C:\Program Files\Microsoft SDK\Lib;c:\bin\lang\py24\libs;%LIB%

echo Visit http://msdn.microsoft.com/visualc/using/documentation/default.aspx for
echo complete compiler documentation.

The path variable needs to point to %Toolkit%/bin, %PlatformSDK%/bin, and the Python 2.4 installation directory (that last is optional, but makes things easier if you're working on a machine which normally has a different Python on the path).

The include path needs to point to %Toolkit%/include, %PlatformSDK%/include, and %Python24%/include.

For the lib path, the "Microsoft Visual Studio .NET 2003" directory referenced above is created as a side-effect of one of our installs and includes the all-important msvcr71.lib file, we also need to add the %Toolkit%/lib, %PlatformSDK%/lib and %Python24%/libs directories to the path.

There may still be required utilities/libs/header directories that this configuration doesn't include.  If you find one, please let me know so I can add them to this page.

Now, when we want to use the Toolkit compiler, we run vc7 at the command prompt and our build environment is configured.  Now we need to teach Python's distutils how to work with this compiler.

Distutils Modifications

Distutils doesn't know how to deal with the MS Toolkit Compiler, so we need to modify it.  At the moment we're doing this by actually modifying the %Python24%/lib/distutils/msvccompiler.py file.  Eventually the changes could be wrapped into distutils, either as a sub-class of msvccompiler, or by altering the base file.

The differences are quite minor, however, different versions are required for the 2.4 alpha msvccompiler and 2.4 final.  Here are the 2.4 final diffs:

*** msvccompiler-final-original.py	Mon Nov 29 08:58:32 2004
--- msvccompiler.py Fri Dec 3 15:31:54 2004
***************
*** 126,132 ****
self.set_macro("FrameworkDir", net, "installroot")
try:
if version > 7.0:
! self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1")
else:
self.set_macro("FrameworkSDKDir", net, "sdkinstallroot")
except KeyError, exc: #
--- 126,138 ----
self.set_macro("FrameworkDir", net, "installroot")
try:
if version > 7.0:
! try:
! self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1")
! except KeyError:
! # likely using free Command-line compiler with free SDK
! freeSDK = r"SOFTWARE\Microsoft\MicrosoftSDK\InstalledSDKs\63DADB24-DC99-45EB-A748-EC93AB8A7497"
! # following should raise key error if not available...
! self.set_macro( "FrameworkSDKDir", freeSDK, 'install dir' )
else:
self.set_macro("FrameworkSDKDir", net, "sdkinstallroot")
except KeyError, exc: #
***************
*** 571,576 ****
--- 577,595 ----
"You must at least run the Visual Studio GUI once "
"so that these entries are created.")
break
+ if self.__version >= 7:
+ # well, what if we've already set the environment variables?
+ map = {
+ 'library dirs': 'lib',
+ 'path dirs': 'path',
+ 'include dirs': 'include',
+ }
+ path = map.get( path )
+ if os.environ.get( path ) is not None:
+ return string.split(
+ os.environ.get( path ),
+ os.pathsep,
+ )
return []

def set_path_env_var(self, name):

The first change relies on the GUID of the 1.1 .NET Framework SDK.  Obviously that's not going to work if you install a different version of the SDK.  If someone has a better approach, let me know. 

Martin Bless reports that for his XP SP2 SDK the GUID is: B44C7E10-89BD-4A32-A3BF-D9D0BC4C9A05 rather than that in the patch.

The second change should be fairly robust, in fact, it should even allow for using Visual Studio 6 from the command line if you haven't yet run the IDE (a limitation of the current code) but have run vcvars.bat.

You can either manually edit the file or use the unix-style patch utility to integrate the patch (the patches above were made against the msvccompiler.py file in Python-2.4a1 and Python-2.4 (final)).

Armin Rigo reports that he needs to use 'library' instead of "library dirs" in the second half of the patch.  My own system doesn't seem to require this, so I've left the patch as-is for now.  My reading of the code is that path will always have ' dirs' appended to it.

From another report: it's possible to replace the distutils.msvccompiler.read_values with a simple function that simply returns the values from the shell environment, (with only the toolkit installed you will always get the values from the environment), but I don't recommend that approach, as at some point you may install MSVC and find everything messed up.

Compiling your Extension

At this point, you should be able to run vc7.bat from the command prompt, then run python setup.py bdist_wininst and have the MS Toolkit Compiler be used to build your extension for Python 2.4 and turn it into a binary redistributable for MS Windows end-users.

If you succeed, fail, have suggestions for enhancements, or need clarification, let me know and I'll try to integrate the feedback into this document.  You will notice that there are a large number of warnings printed about soon-to-be obsolete compiler flags.  At some point someone will need to track those down and see if we can eliminate them from distutils.

Success/Failure Matrix

This table is to collect the set of successes/failures encountered in attempting to build various extensions with the setup outlined here:

Package
Success?
Notes
Numpy 23.1 Yes
There are some ominous warnings from the lapack_lite module about undefined functions, but the core library seems to work fine with PyOpenGL and OpenGLContext.
Numpy 23.6 Yes (w/ changes) Ming-Yu reports success when compiling with the .Net SDK 2.0 beta1, while I can't seem to get compilation to work with the setup described above.  Looks, again, like the lapack_lite module is missing something, but in this case, the entire lapack.lib module.
mxTextTools-nr 2.1
Yes
Only non-recursive rewrite tested
mxExperimental 0.9.0
No
Requires lib.exe, which isn't part of the above setup.  See the SAPDB wiki for description of workarounds that may eventually allow compilation to work.  Probably want to eliminate use of lib.exe eventually, as it seems distutils could do the (trivial) work itself. (Report from Steve Holden)
mxBase, mxODBC (2.0.6, 2.0.7 resp.)
Yes
Report from Steve Holden
OpenGLContext 2.0.0b2 Accellerators
Yes

PyOpenGL 2.0.2.01 (Swig 1.3.23 version), and 2.0.1.09 (Swig 1.3.13)
Yes
2.0.2 tested with Python 2.4 final.  2.0.1 tested with Python 2.4 alpha and beta.
Psyco 1.3
Yes (w/ changes)
Armin reports success building Pysco with the above setup, modulo the noted changes to the distutils patch to remove " libs" from the directory name mapping (eliminating the need for the mapping in the process if I'm not mistaken).
PyTables
Yes
Needs HDF5 dependency recompiled as well, apparently. (Report from Francesc Altet).
PyGreSQL 3.6-pre040918 (with patches)
Yes
Against PostgreSQL 7.4.6 libpq.dll (created w/ toolkit)
PyPgSQL 2.4
Yes
Against PostgreSQL 7.4.6 libpq.dll (created w/ toolkit)
PyCrypto 2.0
Yes
Report from Jody Burns
MySQL-python 1.0.0
Yes
Compiled with MySQL 4.0.20d-win.  Required copying c:\mysql\lib\opt\zlib.lib to c:\mysql\lib\opt\z.lib . Report from Y.H. Rhiu.  Steve Holden also reports success.
Sendkeys (unspecified version)
Yes
Report from Dan Milgram.
InformixDB 2.0, 2.1 Yes Report from Carsten Haese
Thuban Yes Report from Didrik Pinte
ctypes 0.9.9.3 Yes
OpenGL-ctypes (CVS) Yes
4suite Yes Report from Mike Starovoytov
Zope 2.9.0 Yes (w/ alternate approach) Report from Patrick Decat; created registry keys instead of modifying the distutils scripts.
pypcap-1.0 Yes Report from Dug Song.  Didn't use vc7.bat, used settings in cygwin shell
pyevent-0.3 Yes Report from Dug Song.  Didn't use vc7.bat, used settings in cygwin shell

Other Resources