[ARROW-2787] [Python] Memory Issue passing table from python to c++ via cython

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Component/s: Documentation, Python
• Labels: cython pull-request-available
• Environment: clang6
• External issue URL: https://github.com/apache/arrow/issues/19169
• Assignee: Antoine Pitrou
• Reporter: Joseph Toth
• Created: 03/Jul/18 16:25
• Updated: 11/Jan/23 07:23
• Resolved: 17/Jul/18 13:56

Description

I wanted to create a simple example of reading a table in Python and pass it to C+, but I'm doing something wrong or there is a memory issue. When the table gets to C+ and I print out column names it also prints out a lot of junk and what looks like pydocs. Let me know if you need any more info. Thanks!

demo.py

import numpy
from psy.automl import cyth
import pandas as pd
from absl import app

def main(argv):
  sup = pd.DataFrame({
  'int': [1, 2],
  'str': ['a', 'b']
  })
  table = pa.Table.from_pandas(sup)
  cyth.c_t(table)

cyth.pyx

import pandas as pd
import pyarrow as pa
from pyarrow.lib cimport *

cdef extern from "cyth.h" namespace "psy":
 void t(shared_ptr[CTable])

def c_t(obj):
 # These print work
 # for i in range(obj.num_columns):
 # print(obj.column(i).name
  cdef shared_ptr[CTable] tbl = pyarrow_unwrap_table(obj)
  t(tbl)

cyth.h

#include <iostream>
#include <string>
#include "arrow/api.h"
#include "arrow/python/api.h"
#include "Python.h"

namespace psy {

void t(std::shared_ptr<arrow::Table> pytable) {

// This works
  std::cout << "NUM" << pytable->num_columns();

// This prints a lot of garbage
  for(int i = 0; i < pytable->num_columns(); i++) {
  std::cout << pytable->column(i)->name();
  }
 }
}

 

Issue Links

• supercedes: ARROW-2867 [Python] Incorrect example for Cython usage
• links to: GitHub Pull Request #2276