top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Python: Running code from source that includes extension modules

+2 votes
726 views

I have started looking into distutils because I need to write an extension module in C (for performance reasons) and distutils seems to be the most straight-forward way.

I have had success building a C file into a Python extension module using "python setup.py build" but I am wondering what the recommended way for using that module during development is. While writing Python code I am used to just run the code from the source directory. But the built extension modules .so of course does not just end up on sys.path magically.

So how do I run my code so it will find the built extension module? Do I pass the output directory on the command line manually or is there some other solution? I would like to still be able to run the code from the source directory as I am using PyCharm to edit and debug the code.

posted Oct 2, 2013 by Luv Kumar

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button
I'm running OS X 10.8 and Python 3.2, sorry I didn't mention it. But I assume the differences to Linux are minimal.

The current directory is included in sys.path, otherwise I wouldn't be able to import modules in the same directory. But the problem is that the built extension module is in a subdirectory of the "build" directory:

$ find -name '*.so'
./build/lib.macosx-10.8-x86_64-3.2/_foo.so

And so I can't import it without manually adding that directory to sys.path. I'm convinced, someone on this list can shout at me, telling me that I got it completely backwards and that there's a straightforward and intuitive way to develop extension modules!

2 Answers

+1 vote
 
Best answer

You can run

 python setup.py build_ext -i

That will build your extension module and install it right into your package structure.

BTW, if you use Cython instead of plain C, you can use pyximport to get on-the-fly extension module builds during development.

answer Oct 2, 2013 by Ahmed Patel
This is really very much what I was looking for! I've set up PyCharm to run this command (by configuring it as an "external tool", maybe there's a simpler way), before running the actual application. o/

> BTW, if you use Cython instead of plain C, you can use pyximport to get on-the-fly extension module builds during development.

I will look into that too, that sounds very convenient. But am I right, that to use Cython the non-Python code needs to be written in the Cython language, which means I can't just copy&past C code into it? For my current project, this is exactly what I do, because the C code I use already existed.
It's better than that. Don't copy/paste your code. Just declare it in Cython and you can call straight into the existing C functions cutting out most of the boilerplate involved in making C code accessible to
Python: http://docs.cython.org/src/userguide/external_C_code.html

You'll sometimes need a short Cython wrapper function to convert from Python types to corresponding C types. But this is about 5 lines of easy to read Cython code vs maybe 30 lines of hard to follow C code.

Having written CPython extension modules both by hand and using Cython I strongly recommend to use Cython.
+1 vote

Doesn't Python on Linux (I assume that since you mentioned the module's .so) support having current-dir '.' in $PYTHONPATH? Works fine on Windows.

Check with "python -v script.py | grep ".

answer Oct 2, 2013 by Ahmed Patel
Similar Questions
+4 votes

I had developed extension for firefox. In that i used

var cookieManager = Cc["@mozilla.org/cookiemanager;1"] .getService(Ci.nsICookieManager2); 
var count = cookieManager.getCookiesFromHost("example.com");

to read cookie. Noted in Access Specific cookie.

Now i tried same in private window but i can't able to read a single cookie info.

Kindly suggest me to read a cookie info in main.js, private window.

+4 votes

I just started rewritting my project from python 2 to python 3. I noticed that there are these new parameter and return value annotations. I have docstrings everywhere in my project, but I plan to convert many of them into annotations. The question is: what kind of auto documenting system should I use for this? Previously I have used Sphinx. Is it okay to use that for python 3 source code? Is there a better alternative?

+1 vote

I'd like to have the option to download the source code as text/plain from the docs.python.org pages.

For example: when I'm a docs page, such as:
http://docs.python.org/2/library/string.html

and I click the source code link I'm taken to a Mercurial page:
http://hg.python.org/cpython/file/2.7/Lib/string.py

but over there there's no way to get a clean text/plain version of the code because the line numbers are included.

A link to the text/plain version on that page would be nice!

+1 vote

Attempting to import modules from Shogun to python from a non-standard python directory i.e. from my /home/xxx directory. Is there a way on ubuntu to selectively some modules, scripts, data from one directory and others modules, scripts from another directory. In other words, is there a file(s) that provide pointers to where these modules are located.

0 votes

I'm trying to write a function that programmatically obtains and returns the exact location of all first-level modules for all installed packages.

For example, if the packages named 'django' and 'django-debug-toolbar' are installed, I'd like this function to return something like:

installed_modules()
/Users/my_user/.virtualenvs/my_venv/lib/python2.6/site-packages/django
/Users/my_user/.virtualenvs/my_venv/src/debug_toolbar

That is, this function needs to consider all installed packages, including those that have been installed in "edit" mode (i.e. in the src/ folder). Note also that the main module for the 'django-debug-toolbar' is in fact named 'debug_toolbar'.

So far the closest I've been to retrieving the list of first-level modules is as follows:

 import os
 import pkg_resources
 import setuptools

 pkgs = set()

 for dist in pkg_resources.working_set:
 if os.path.isdir(dist.location):
 for pkg in setuptools.find_packages(dist.location):
 if '.' not in pkg:
 pkgs.add(pkg)

The idea is then to loop through that list of modules, import them and get their exact locations by fetching their __file__ attribute values.

However, this feels very hackish and I don't think it's actually quite correct either. I'm sure there must be a better way. If possible I'd also like to avoid having to use setuptools.

Does anyone have any tips on how to achieve this?

...