Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7200

[C++][Flight] Running Arrow Flight benchmark on two hosts doesn't work

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.15.0, 0.15.1
    • 0.16.0
    • Benchmarking, C++, FlightRPC
    • AWS EC2
      Instance type: t3a.xlarge
      AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
      Number of instances: 2
      They are capable of pinging each other.

    Description

      I was trying to evaluate the performance of Apache Arrow Flight on two hosts (one as the client and the other one as the server), using [the official benchmark|https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc].

      Flags I used to build the project were:

       

      -DARROW_FLIGHT=ON
      -DCMAKE_BUILD_TYPE=Debug
      -DARROW_BUILD_BENCHMARKS=ON
      

       

      The branch I used was maint-0.15.x since there was a build error on the master branch. (The build error on master only existed in the environment where I set up two hosts: AWS. On my local environment (macOS) the build was successful on the master branch. I don't think this build error is relevant to the issue since there is no difference in the cpp source code.)

      On the host acting as the server, I ran 

      ./arrow-flight-perf-server

      On the host acting as the client, I ran 

      ./arrow-flight-benchmark --server_host ip-172-31-11-18

      It gives the following error:

      Failed with error: << IOError: gRPC returned unavailable error, with message: Connect Failed. Detail: Unavailable

       

       If I ran 

      ./arrow-flight-benchmark --server_host ip-172-31-11-17

      the error will be different:

      IOError: Server was not available after 10 attempts

      This is understandable since this host doesn't exist at all.

      This indicates that Flight is able to find the existing host (ip-172-31-11-18), but the communication somehow didn't succeed.

      The benchmark works fine if I run it with the localhost, either by not specifying the server_host flag or running the server in another process on the same host.

      I am not sure if the problem is in the environment or in the code itself. Could someone please give me some hint on how to resolve the problem?

      Attachments

        1. Screen Shot 2019-11-19 at 14.41.40.png
          342 kB
          Chengxin Ma
        2. Screen Shot 2019-11-18 at 16.00.38.png
          1.16 MB
          Chengxin Ma

        Issue Links

          Activity

            People

              cxma Chengxin Ma
              cxma Chengxin Ma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m