python extending and embedding · bodenseo Extending and Embedding SWIG S implified Wrapper and I nterface G enerator Swig wandelt vollautomatisch C/C++ Code in Skripting-Sprachen

Einführung in Python II

© Bernd Klein, Bodenseo

© bodenseo, Bernd Klein, 07.05.15 Folie:1

bodenseo Extending and Embedding

ExtendingExtendingandandEmbeddingEmbeddingPythonPython



Embedding and Extending

C/C++

Python

Python

C/C++

Embedding Python in C/C++C/C++ access to the Python interpreter.

Extending Python with C/C++,Python access to C/C++



Grundsätzliche Vorgehensweise

Grundsätzlich gibt es zwei Vorgehensweisen, ein Programm zu schreiben, das Python mit einer anderen Sprache kombiniert:● In einem Python-Programm soll eine andere Sprache (z.B. C)

ausgeführt werden.● In einem Programm in einer anderen Sprache (z.B. C) soll ein

Python-Script ausgeführt werden.



C/C++ in Python

Python

C / C++





ctypes_Modul

Dynamisch ladbare Bibliotheken können mit dem Modul ctypes in Python eingebunden werden, also z.B. DLL-Dateien (Dynamic Link Library) unter Windows oder SO-Dateien (Shared Object) unter Linux/Unix.

Eine dynamische Bibliothek ist nicht notwendigerweise aus einem C oder C++ Programm erstellt worden.

Dynamische Bibliotheken sind sprachunabhängige Schnittstellen zwischen verschiedenen Programmen. Voraussetzung ist nur, dass es sich um eine kompilierbare Sprache handelt.



Beispiel

import ctypes # unter Windows:# libc = ctypes.CDLL("MSVCRT") # unter Linux:libc = ctypes.CDLL("libc.so.6")libc.printf(b"Hallo Welt\n")#libc.printf("Hallo Welt\n") Python2x = libc.rand();print(x)x = libc.putchar(48);

Muss groß geschrieben werden und ohne .dll



Einbinden einer eigenen Libraryint fak(int n) { int i; int fac; for (i = 1, fac = 1; i <= n; i++) fac *= i; return fac; }

int komb(int k, int n) {/* Anzahl: k Objekte werden aus n Objekten ohne Berücksichtigung der Reihenfolge ausgewählt*/ int anzahl; anzahl = fak(n) / (fak(k) * fak(n - k)); return anzahl;}



so-Library: Erzeugen u. benutzen

Erzeugen der Library:gcc -c -fPIC fakultaet.c -o fak.ogcc -shared -Wl,-soname,fakultaet.so -o fakultaet.so fak.o

Python-Skript, dass die Library benutzt:

import ctypesperm = ctypes.cdll.LoadLibrary("./fakultaet.so") perm = ctypes.CDLL("./fakultaet.so") + Python2x = perm.fak(5)print(x)y = perm.komb(3,4)print(y)y = perm.komb(6,49)print(y)

Aufruf ergibt:$ python ctypes_fakulaet.py1204Floating point exception





Numerisch stabile Version

int komb2(int k, int n) { /* numerisch stabile Version: erg = (n - (k-1))! / k! */ int i; float erg;

for (i=0, erg=1; i<k; i++) { erg *= (n - i) / (float) (1 + i); } return erg;}



Proper Wrapper Functions

Wrappers serve as „glue“ languages between languages.We need them to return results in a Python friendly form.

// factorial.c

int fact(int n) { if (n <= 1) return 1; else return n*fact(n-1);}

PyObject *wrap_fact(PyObject *self, PyObject *args) { int n, result; if (!PyArg_ParseTuple(args, "i:fact", &n)) return NULL; result = fact(n); return Py_BuildValue("i", result);}



Wrapper Function

PyArg_ParseTuple(args, "i:fact", &n)

parses the parameters of a function that takes only positional parameters into local variables. Returns true on success; on failure, it returns false and raises the appropriate exception.

The args argument must be a tuple object containing an argument list passed from Python to a C function. "i:fact" is the format stringThe following arguments are addresses of variables whose type is determined by the format string, in our case only &n

The format string "i:fact" defines that n must be an integer (the i in front of the colon) defines the error message, which will be used instead of the default error message.



Examples of PyArg_ParseTuple

int ok;int i, j;long k, l;char *s;

No arguments:ok = PyArg_ParseTuple(args, ""); /* Python call: f() */

A string:ok = PyArg_ParseTuple(args, "s", &s); /* Possible Python call: f('whoops!') */

Two longs and a string:ok = PyArg_ParseTuple(args, "lls", &k, &l, &s); /* Possible Python call: f(1, 2, 'three') */





Examples of PyArg_ParseTuple

int ok;int i, j;char *s;int size;char *file;char *mode = "r";int bufsize = 0;

A pair of ints and a string, whose size is also returned:ok = PyArg_ParseTuple(args,"(ii)s#", &i, &j, &s, &size); /* Possible Python call: f((1, 2), 'three') */

A string, and optionally another string and an integer: ok = PyArg_ParseTuple(args, "s|si", &file, &mode, &bufsize); /* Possible Python calls: f('spam') f('spam', 'w') f('spam', 'wb', 100000) */



Py_BuildValue

Py_BuildValue("i", result);

PyObject* Py_BuildValue(const char *format, ...)

Return value: New reference.

Create a new value based on a format string and a sequence of values. Returns the value or NULL in the case of an error; an exception will be raised if NULL is returned.

Py_BuildValue does not always build a tuple. It builds a tuple only if its format string contains two or more format units. If the format string is empty, it returns None; if it contains exactly one format unit, it returns whatever object is described by that format unit. To force it to return a tuple of size 0 or one, parenthesize the format string.



Module Initialization

The initialization function is called, when an extension module is imported.New methods are registered with the Python interpreter.

Initialization method for our example module „factorial“:

Methods Tables:

static PyMethodDef factorialMethods[] = { { "fact", wrap_fact, 1 }, { NULL, NULL }};

Initialization Function:

void initfactorial() { PyObject *m; m = Py_InitModule("factorial", factorialMethods);}



Compiling for Dynamic Loading

gcc -fPIC -c -I/usr/include/python2.7 -I/usr/lib/python2.7/config factorial.c wrapper.c

gcc -shared factorial.o wrapper.o -o factorial.so

Importing Module in Python:

>>> import factorial>>> factorial.fact(10)3628800>>> factorial.fact(6)720>>>

If a module is called example the so file has to be called example.so or examplemodule.so!





Exercise

Change the factorial module so that factorial.fact(n) will return a tuple (n, fact(n)),

i.e.

>>> import factorial>>> factorial.fact(6)(6, 720)>>>

Solution:All we have to do is change the last line of the wrap_fact() function from return Py_BuildValue("i", n, result);intoreturn Py_BuildValue("(n, i)", n, result);



Problem and Solution

We saw that➔ It is hard to understand and cumbersome to write extensions

manually➔ It will be difficult to maintain

We need tools to alleviate, i.e. to automate, the extension process!

Extension tools:

Boost.PythonctypespyfortPyrexSwig



SWIG

Simplified Wrapper and Interface Generator

Swig wandelt vollautomatisch C/C++ Code in Skripting-Sprachen.

C/C++

SWIG

Python Tcl/Tk PerlMATLAB ...



Simple Example: example.c#include <time.h>double My_variable = 3.0; int fact(int n) { if (n <= 1) return 1; else return n*fact(n-1);} int my_mod(int x, int y) { return (x%y);} char *get_time() { time_t ltime; time(&ltime); return ctime(&ltime);





SWIG Interface FileThe example.i file:

/* example.i */%module example%{/* header files here or function declarations like below: */extern double My_variable;extern int fact(int n);extern int my_mod(int x, int y);extern char *get_time();%}

extern double My_variable;extern int fact(int n);extern int my_mod(int x, int y);extern char *get_time();

Module name



Building the Python Interface

The example.i file:

extern double My_variable;extern int fact(int n);extern int my_mod(int x, int y);extern char *get_time();

Building the Python Interface with SWIG:

swig -python example.ior in case of C++:swig -c++ -python example.i

This creates the following files:example_wrap.cexample.py



Simple Example: Compiling C Prog

Compiling:

bernd@venus $ gcc -fPIC -c example.c example_wrap.c -I/usr/include/python3.4

Creating the so file:bernd@venus $ ld -shared example.o example_wrap.o -o _example.so

Attention: Underscore is important! NO file example.so should be in the directory!!!



Simple Example: Using in Python

>>> import example>>> example.fact(5)120>>> example.my_mod(7,3)1>>> example.get_time()'Tue Jan 13 22:52:26 2015\n'

accessing the global variable My_variable:

>>> example.cvar.My_variable3.0>>>





A C++ Simple Example

// factorial.i

%module factorial%{extern int fact(int n);%}#define VERSION 0.1

extern int fact(int n);

// factorial.cpp

int fact(int n) { if (n <= 1) { return 1; } return n * fact(n - 1);}

$ swig -c++ -python factorial.i$ g++ -fPIC -c factorial.cpp factorial_wrap.cxx -I/usr/include/python3.4$ g++ -shared factorial.o factorial_wrap.o -o _factorial.so



Using the New Factorial Module

>>> import factorial>>> factorial.VERSION0.1>>> factorial.fact(5)120>>>



SWIG Transformations

C/C++ Python

C/C++ functions Python functions

Global Variables Attributes of the Python object „cvar“

C constants Python variables

int, long, short Python integers

float, double Python floats

char, char* Python strings

void None



Constants and enum in CDeclarations with #defineconst enumare turned into Python variables:

see example constants_enums

%module ultimate%{#define VERSION "1.0"

enum days {Mon,Tue,Wed,Thu,Fri,Sat,Sun};%}#define VERSION "1.0"enum days {Mon,Tue,Wed,Thu,Fri,Sat,Sun};%constant int SPECIAL = 42;%constant const char *question = "The Ultimate ...";





Using the „ultimate“ ModuleHelp on module ultimate:

NAME ultimate

DESCRIPTION # This file was automatically generated by SWIG (http://www.swig.org). # Version 2.0.11 # # Do not make changes to this file unless you know what you are doing--modify # the SWIG interface file instead.

DATA Fri = 4 Mon = 0 SPECIAL = 42 Sat = 5 Sun = 6 Thu = 3 Tue = 1 VERSION = '1.0' Wed = 2 question = 'The Ultimate Question is the actual question behind the Ul...



//boxes.i %module boxes %inline %{ struct Box { double length; double breadth; double height; }; %}

struct Example

Creating the module:

$ swig -c++ -python boxes.i$ g++ -fPIC -c boxes_wrap.cxx -I/usr/include/python3.4$ g++ -shared boxes_wrap.o -o _boxes.so

Using the module in Python3:>>> import boxes>>> b = boxes.Box()>>> b.length = 20>>> b.width = 8>>> b.height = 17>>>

SWIG creates a set of accessor functions for the definition of a structure or union.

The accessor functions generated by SWIG use a pointer to an object and allow access to the individual members.

For example, the declaration :



Internal Representation in SWIGWe can find the following getter, setter, constructors and destructor definitions in boxes_wrap.cxx:

static PyMethodDef SwigMethods[] = { { (char *)"SWIG_PyInstanceMethod_New",

(PyCFunction)SWIG_PyInstanceMethod_New, METH_O, NULL}, { (char *)"Box_length_set", _wrap_Box_length_set, METH_VARARGS, NULL}, { (char *)"Box_length_get", _wrap_Box_length_get, METH_VARARGS, NULL}, { (char *)"Box_breadth_set", _wrap_Box_breadth_set, METH_VARARGS, NULL}, { (char *)"Box_breadth_get", _wrap_Box_breadth_get, METH_VARARGS, NULL}, { (char *)"Box_height_set", _wrap_Box_height_set, METH_VARARGS, NULL}, { (char *)"Box_height_get", _wrap_Box_height_get, METH_VARARGS, NULL}, { (char *)"new_Box", _wrap_new_Box, METH_VARARGS, NULL}, { (char *)"delete_Box", _wrap_delete_Box, METH_VARARGS, NULL}, { (char *)"Box_swigregister", Box_swigregister, METH_VARARGS, NULL}, { NULL, NULL, 0, NULL }

};



//boxes.i %module boxes %inline %{ struct Box { double length; double breadth; double height; };

Box *newBox(double l, double b, double h) {

Box *box = new Box; box->length = l; box->breadth = b; box->height = h; return box; }

void deleteBox(Box *box) { delete box; } %}

struct Example II

Creating the module:

$ swig -c++ -python boxes.i$ g++ -fPIC -c boxes_wrap.cxx -I/usr/include/python3.4$ g++ -shared boxes_wrap.o -o _boxes.so

Using the module in Python3:>>> import boxes>>> x = boxes.newBox(3,2,5)>>> type(x)<class 'boxes.Box'>>>> x.height5.0>>> x.height = 7>>> x.height7.0>>> y = boxes.Box()>>> y.length0.0





Exercise

Structures involving character strings can also be defined. They will be dynamically allocated using malloc() and they are NULL-terminated ASCII strings. When such a member is modified, the previously contents will be released, and the new contents allocated.

Write a struct for a

person with the

components

firstname

surname and

height



Solution

%module persons %inline %{ struct Person {

char *firstname; char *surname;

float height; }; %}

>>> from persons import Person>>> x = Person()>>> from persons import Person>>> x = Person()>>> x.firstname = "Marc">>> x.firstname = 3Traceback (most recent call last):...TypeError: in method 'Person_firstname_set', argument 2 of type 'char *'>>> x.surname = "Steiner">>> x.height = 176.9



Typemaps

A typemap is a code generation rule that is attached to a specific C datatype.

We can e.g. convert integers from Python to C defining the following typemap: (see:typemap_simple1)

%module example%typemap(in) long {

$1 = (long) PyLong_AsLong($input);printf("Received an integer :

%ld\n",$1);}

%inline %{long fact(long n) { if (n <= 1) { return 1;} return n * fact(n – 1);}%}



Using the „example“ module>>> import example>>> example.fact(10)Received an integer : 103628800>>>

In the previous example, the typemap is applied to all occurrences of the „long“ datatype. We can refine this by supplying an optional parameter name. For example:

%typemap(in) long n {$1 = (long) PyLong_AsLong($input);

if ($1 <= 0) PyErr_SetString(PyExc_ValueError,"Expected a positive value."); return NULL; }}

The typemap code will only be applied to arguments which exactly match „long n“ (not „long m“ e.g.)





Using the Module

>>> import example>>> example.fact(10)3628800>>>>>> example.fact(0)Traceback (most recent call last): File "<stdin>", line 1, in <module>ValueError: Expected a positive value.>>>>>> example.fact(-1)Traceback (most recent call last): File "<stdin>", line 1, in <module>ValueError: Expected a positive value.>>>



typemaps.i libray

void add(double a, double b, double *result) { *result = a + b;}

A human reader understands, that this function is storing a „return“ value in the double *result parameter. However, SWIG has no way to know that this is the underlying behavior. One way to deal with this problem is to use the typemaps.i library.

The corresonding code looks like this:

%module example%include "typemaps.i"

%apply double *OUTPUT { double *result };%inline %{extern void add(double a, double b, double *result);%}

See example „typemaps1“

using the module:>>> import example>>> example.add(3,4)7.0



typemaps.i libray, II

The typemap stays in effect for all other occurrences of the type and name. For example, we can define further similar functions for our previous example:

%module functions%include "typemaps.i"

%apply double *OUTPUT { double *result };

%inline %{extern void add(double a, double b, double *result);extern void sub(double a, double b, double *result);extern void mul(double a, double b, double *result);%}

See example „typemaps2“



Pointers

%module pointers FILE *fopen(const char *filename, const char *mode); int fputs(const char *, FILE *); int fclose(FILE *);

Using the pointers module:

>>> import pointers>>> fh = pointers.fopen("Blabla.txt", "w")>>> pointers.fputs("This text will be written to file", fh)1>>> pointers.fclose(fh)0>>>

Let's check the content of „Blabla.txt“:$ more Blabla.txt This text will be written to file





Returning Lists%module Foo

%typemap(out) double *bar %{ $result = PyList_New(2); // size of list for (int i = 0; i < 2; ++i) { PyList_SetItem($result, i, PyFloat_FromDouble($1[i])); } delete $1; // important to avoid a memory leak%}

%inline %{double *bar() { double *toReturn = new double[2]; toReturn[0] = 42; toReturn[1] = 7; return toReturn;}%}



Converting Python list to a char **See code in /python_list_to_char_array

Using the module pl2c:$ pythonPython 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> import pl2c>>> pl2c.print_args(["Dave","Mike","Mary","Jane","John"])list size = 5argv[0] = Daveargv[1] = Mikeargv[2] = Maryargv[3] = Janeargv[4] = John5>>>



Creating and Using the Module foo

swig -c++ -python foo_list.i

g++ -fPIC -c foo_list.cpp foo_list_wrap.cxx -I/usr/include/python3.4

g++ -shared foo_list.o foo_list_wrap.o -o _Foo.so

Using Foo in Python3:

>>> import Foo>>> Foo.bar()[2.0, 4.0]>>> type(Foo.bar())<class 'list'>>>>



Classes and Swig

10:17:59We will write a class clock in C++ with a Python interfacefor a digital clock.

It should implement the following methods:set (hours, minutes, seconds = 9);tick();

The method tick() should “tick” for a second, i.e. the time should be advanced by one second:11:59:59 → 12:00:00





Clock Class: Clock.h and Clock.i// Clock.h

class Clock {protected:

int hours;int minutes;int seconds;

public: Clock( int h, int min, int sec); void set( int h, int min, int sec); int getHours() { return hours; }; int getMinutes() { return minutes; }; int getSeconds() { return seconds; };

void tick();};

// Clock.i%module Clock%{#include "Clock.h" %}

%include "Clock.h"



Clock Class Used in Python

>>> import Clock>>> x = Clock.Clock(11,8,23)>>> x.getMinutes()8>>> x.getHours()11>>>

What about having a method, which returns the time in hours, minutes and seconds?

i.e.>>> x.getTime()11, 8, 23



Clock Class with getTime()%module Clock%include "typemaps.i"%apply int *OUTPUT { int &h, int &min, int &sec }; %{#include "Clock.h" %}

class Clock { // … // include code as before void getTime(int &h, int &min, int &sec); // ….};

using the module:

>>> import Clock>>> x = Clock.Clock(11,8,23)>>> x.getTime()[11, 8, 23]>>> t = x.getTime()>>> type(t)<class 'list'>>>>



Python in C/C++

C / C++

Python





Python in C-Programme einbetten

Zuerst sicherstellen, dass die Header-Dateien und statische Library von Python installiert sind:

dpkg-query -l | grep python.*-dev

Wenn nicht, dann

sudo aptitude install python2.5-devodersudo aptitude install python2.5-devbzw.sudo aptitude install python-dev



Kleines Beispiel

Das folgende C-Programm bindet den Python-Interpretor ein:

#include <python2.6/Python.h>

main( ) { Py_Initialize( ); PyRun_SimpleString("a = 2 * 3"); PyRun_SimpleString("print a"); }

Kompiliert und gelinkt wird das Programm wie folgt:gcc -lpython2.6 -lm -L/usr/lib/python2.6/config minimum.c



Aufruf des Interpretors als Shell


main(int argc, char** argv ) { printf("embed-simple\n"); Py_Initialize(); Py_Main(argc, argv);}

Ruft man das kompilierte Programm auf, erhält man:$ ./a.outembed-simplePython 2.6.4 (r264:75706, Nov 2 2009, 14:54:49) [GCC 4.4.1] on linux2Type "help", "copyright", "credits" or "license" for more information.>>>



Einlesen einer Int-Variablen


main( ) { PyObject * module; PyObject * dict; PyObject * obj; long lval;

Py_Initialize( ); PyRun_SimpleString("abc = 2 * 3");

module = PyImport_AddModule("__main__"); dict = PyModule_GetDict(module); obj = PyMapping_GetItemString(dict, "abc"); if (obj != NULL){ lval = PyInt_AsLong(obj); } else { printf("Object not found\n"); } /* if */printf("Wert von abc: %d\n", (int) lval);

Die Variable abc von Python wird als Integer in C eingelesen.Namensraum in Python:__main__





int, string und float einlesen#include <python2.6/Python.h>

main( ) { PyObject * module; PyObject * dict; PyObject * obj; long lval; float x; char* str;

Py_Initialize( ); PyRun_SimpleString("abc = 2*3; xyz = 34/7.0; name=\"Ede\""); module = PyImport_AddModule("__main__"); dict = PyModule_GetDict(module); obj = PyMapping_GetItemString(dict, "abc"); lval = (obj != NULL) ? PyInt_AsLong(obj) : 0; obj = PyMapping_GetItemString(dict, "name"); str = (obj != NULL) ? PyString_AsString(obj) : "undefined"; obj = PyMapping_GetItemString(dict, "xyz"); x = (obj != NULL) ? PyFloat_AsDouble(obj) : 0; printf("1st: %d, 2nd: %s, 3rd: %f\n", (int) lval, str, x);}

Documents

python extending and embedding · bodenseo Extending and Embedding SWIG S implified Wrapper and I nterface G enerator Swig wandelt vollautomatisch C/C++ Code in Skripting-Sprachen