[ARROW-2872] [Python] Add pytest mark to opt into TensorFlow-related unit tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/19243
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 17/Jul/18 20:47
• Updated: 11/Jan/23 07:23
• Resolved: 17/Jul/18 23:17

Description

After pulling in ARROW-1744, I hit this error:

pyarrow/tests/test_plasma_tf_op.py::test_plasma_tf_op FAILED                   [ 82%]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stdout >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
TensorFlow version: 1.8.0
Compiling Plasma TensorFlow Op...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+++ dirname /home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh
++ cd /home/wesm/code/arrow/python/pyarrow/tensorflow
++ pwd
+ PYARROW_TENSORFLOW_DIR=/home/wesm/code/arrow/python/pyarrow/tensorflow
++ python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))'
+ TF_CFLAGS='-I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=0'
++ python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))'
+ TF_LFLAGS='-L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow -ltensorflow_framework'
++ uname
+ '[' Linux == Darwin ']'
+ NDEBUG=-DNDEBUG
++ pkg-config --cflags --libs plasma arrow arrow-python
+ g++ -std=c++11 -g -shared /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc -o /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.so -DNDEBUG -I/home/wesm/local/include -I/home/wesm/miniconda/envs/arrow-dev/include/python3.6m -I/home/wesm/local/include -L/home/wesm/local/lib -lplasma -larrow_python -larrow -fPIC -I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=0 -L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow -ltensorflow_framework -O2
/home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc:33:10: fatal error: arrow/adapters/tensorflow/convert.h: No such file or directory
 #include "arrow/adapters/tensorflow/convert.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

use_gpu = False

    @pytest.mark.plasma
    def test_plasma_tf_op(use_gpu=False):
        import pyarrow.plasma as plasma
    
>       plasma.build_plasma_tensorflow_op()

pyarrow/tests/test_plasma_tf_op.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/plasma.py:56: in build_plasma_tensorflow_op
    subprocess.check_call(["bash", script_path])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

popenargs = (['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh'],)
kwargs = {}, retcode = 1
cmd = ['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the call function.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']' returned non-zero exit status 1.

../../../miniconda/envs/arrow-dev/lib/python3.6/subprocess.py:291: CalledProcessError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/wesm/miniconda/envs/arrow-dev/lib/python3.6/subprocess.py(291)check_call()
-> raise CalledProcessError(retcode, cmd)

I misdiagnosed the problem originally (which was my stale Arrow install), but I think it would be good to have a separate pytest mark for TensorFlow-related unit tests since they are also time consuming. The unit test in question now has both the plasma and tensorflow marks

Issue Links

• links to: GitHub Pull Request #2279