Good ways to pass 2D variable length arrays to/from Python and C?

Question

With some hacking about I got data going back and forth between Python and C using python c-types. It seems a little messy, so I was hoping someone with more experience could tell me if I'm doing something incorrectly, or making this more difficult than it needs to be.

Python Code:

from ctypes import cdll, POINTER, byref, c_double
lib = cdll.LoadLibrary('./test.so')

def alloc_arrays(length, python_list):
    # Create array of float*
    data = ((POINTER(c_double)) * length)()
    out_data = ((POINTER(c_double)) * length)()
    for i in xrange(length):
        # Create arrays of float
        data[i] = (c_double * 3)()
        out_data[i] = (c_double * 3)()
        # Set Values
        for j in xrange(3):
            data[i][j] = python_list[i][j]
            out_data[i][j] = 0.0
    return data, out_data

if __name__ == "__main__":
    a = [[1.0, 11.0, 21.0],
         [2.0, 12.0, 22.0],
         [3.0, 13.0, 23.0],
         [4.0, 14.0, 24.0],
         [5.0, 15.0, 25.0],
         [6.0, 16.0, 26.0],
         [7.0, 17.0, 27.0]]
    in_data, out_data = alloc_arrays(len(a), a)
    out_len = lib.smain(byref(in_data), len(a), out_data)

    print "out_len", out_len
    clean_out = [None, None, None]
    clean_out = [clean_out[:] for i in xrange(out_len)]
    for i in xrange(out_len):
        for j in xrange(3):
            clean_out[i][j] = out_data[i][j]
    print "Out:", clean_out

C code (test.c):

int smain(double* in_points[3], int in_len, double* out_points[3]){
    int i;
    int j;
    printf("%s", "\n");
    for(i = 0; i < in_len; i++){
        for(j = 0; j < 3; j++){
            printf("%g, ", *in_points[i][j]);
        }
        printf("%s", "\n");
    }
    printf("%s", "\n");

    //*out_points = malloc(len*sizeof(float[3]));
    int out_len = in_len-2; //Randomly chosen shorter length
    for(i = 0; i < out_len; i++){
        for(j = 0; j < 3; j++){
            //Random function just to see I can do this
            *out_points[i][j] = i*j;
        }
    }
    return out_len;
}

To build I use:

gcc -c -fPIC test.c
gcc -shared -o test.so test.o

The main things I'm wondering about are if there are better ways to create empty 2D vectors to pass into C, and if there are beter ways to cast variables to/from ctypes to python types. I don't like the alloc_arrays function having to do the conversion, and I don't like having my for loops read the c_types 2D list (just realized I did it a little messy too).

EDIT: Just want to be clear that this code works as expected, I just think there have to be some ways to improve it.

If I try to compile your c code with `gcc -std=c99 -shared -o test.so -fPIC test.c` I get `invalid type argument of unary ‘*’ (have ‘double’)` error messages for `printf("%g, ", *in_points[i][j]);` and `*out_points[i][j] = i*j;` What could be the reason? — Framester, Nov 06 '13 at 16:43

score 0 · Accepted Answer · edited May 23 '17 at 12:04

This isn't an answer regarding your exact code and ctypes, but I wanted to contribute...

I just got into cython recently. I am not even intermediate yet but it seems a whole lot more friendly for writing extensions, and there are a bunch of examples out there about using numpy arrays.

I have found cython to be really great so far because it lets you remain in a python-like syntax, and also mix python freely. It handles quite a lot of heavy lifting for you in terms of helping convert the code to C in wrappers, which is done under the hood. And there are no intermediate dependencies in the produced .so. Just a pure extension.

If you were to use numpy in your python code, then passing them into your extensions would be effortless since it would not have to do any conversions. numpy by itself is already a big speed up on the python side, but doing extra custom stuff would allow you to just pass the numpy pointer over onto the cython extension side.

Good ways to pass 2D variable length arrays to/from Python and C?

1 Answers1