Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2872

[Python] Add pytest mark to opt into TensorFlow-related unit tests

    XMLWordPrintableJSON

    Details

      Description

      After pulling in ARROW-1744, I hit this error:

      pyarrow/tests/test_plasma_tf_op.py::test_plasma_tf_op FAILED                   [ 82%]
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stdout >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      TensorFlow version: 1.8.0
      Compiling Plasma TensorFlow Op...
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      +++ dirname /home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh
      ++ cd /home/wesm/code/arrow/python/pyarrow/tensorflow
      ++ pwd
      + PYARROW_TENSORFLOW_DIR=/home/wesm/code/arrow/python/pyarrow/tensorflow
      ++ python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))'
      + TF_CFLAGS='-I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=0'
      ++ python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))'
      + TF_LFLAGS='-L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow -ltensorflow_framework'
      ++ uname
      + '[' Linux == Darwin ']'
      + NDEBUG=-DNDEBUG
      ++ pkg-config --cflags --libs plasma arrow arrow-python
      + g++ -std=c++11 -g -shared /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc -o /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.so -DNDEBUG -I/home/wesm/local/include -I/home/wesm/miniconda/envs/arrow-dev/include/python3.6m -I/home/wesm/local/include -L/home/wesm/local/lib -lplasma -larrow_python -larrow -fPIC -I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=0 -L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow -ltensorflow_framework -O2
      /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc:33:10: fatal error: arrow/adapters/tensorflow/convert.h: No such file or directory
       #include "arrow/adapters/tensorflow/convert.h"
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      compilation terminated.
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      
      use_gpu = False
      
          @pytest.mark.plasma
          def test_plasma_tf_op(use_gpu=False):
              import pyarrow.plasma as plasma
          
      >       plasma.build_plasma_tensorflow_op()
      
      pyarrow/tests/test_plasma_tf_op.py:89: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      pyarrow/plasma.py:56: in build_plasma_tensorflow_op
          subprocess.check_call(["bash", script_path])
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
      popenargs = (['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh'],)
      kwargs = {}, retcode = 1
      cmd = ['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']
      
          def check_call(*popenargs, **kwargs):
              """Run command with arguments.  Wait for command to complete.  If
              the exit code was zero then return, otherwise raise
              CalledProcessError.  The CalledProcessError object will have the
              return code in the returncode attribute.
          
              The arguments are the same as for the call function.  Example:
          
              check_call(["ls", "-l"])
              """
              retcode = call(*popenargs, **kwargs)
              if retcode:
                  cmd = kwargs.get("args")
                  if cmd is None:
                      cmd = popenargs[0]
      >           raise CalledProcessError(retcode, cmd)
      E           subprocess.CalledProcessError: Command '['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']' returned non-zero exit status 1.
      
      ../../../miniconda/envs/arrow-dev/lib/python3.6/subprocess.py:291: CalledProcessError
      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
      > /home/wesm/miniconda/envs/arrow-dev/lib/python3.6/subprocess.py(291)check_call()
      -> raise CalledProcessError(retcode, cmd)
      

      I misdiagnosed the problem originally (which was my stale Arrow install), but I think it would be good to have a separate pytest mark for TensorFlow-related unit tests since they are also time consuming. The unit test in question now has both the plasma and tensorflow marks

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wesm Wes McKinney
                Reporter:
                wesm Wes McKinney
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h