Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4740

Add option to use hdfsPread() instead of hdfsRead() for HDFS hedged reads

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      In order to use HDFS hedged reads, the hdfsPread API must be used instead of the hdfsRead() call. We should add an option to use hdfsPread().

        Issue Links

          Activity

          Hide
          laszlog Laszlo Gaal added a comment -

          Hi Matthew Jacobs, I see there's an accepted code review for this issue at https://gerrit.cloudera.org/#/c/5635/; may I resolve the issue? Thanks a lot!

          Show
          laszlog Laszlo Gaal added a comment - Hi Matthew Jacobs , I see there's an accepted code review for this issue at https://gerrit.cloudera.org/#/c/5635/ ; may I resolve the issue? Thanks a lot!
          Hide
          mjacobs Matthew Jacobs added a comment -

          commit 889494004e429a7dab4e693e737ca2a6dd7799b9
          Author: Matthew Jacobs <mj@cloudera.com>
          Date: Fri Jan 6 14:49:31 2017 -0800

          IMPALA-4740: Add option to use hdfsPread() for HDFS hedged reads

          In order to use HDFS hedged reads, the hdfsPread API must be
          used instead of the hdfsRead() call. Adds a flag to use
          hdfsPread: --use_hdfs_pread

          Testing:

          • Running existing tests with this flag enabled.
          • Cluster testing with HDFS hedged reads enabled via the HDFS
            client config.
          • Manually tested setting the 'max_chunk_size' to a small
            value to force multiple iterations of the while loop which
            would only normally happen on S3. Tested reading lineitem
            was OK.

          Change-Id: Iecc8b12aa20cbfe08f4ef6a08a191e49709d9525
          Reviewed-on: http://gerrit.cloudera.org:8080/5635
          Reviewed-by: Matthew Jacobs <mj@cloudera.com>
          Tested-by: Impala Public Jenkins

          Show
          mjacobs Matthew Jacobs added a comment - commit 889494004e429a7dab4e693e737ca2a6dd7799b9 Author: Matthew Jacobs <mj@cloudera.com> Date: Fri Jan 6 14:49:31 2017 -0800 IMPALA-4740 : Add option to use hdfsPread() for HDFS hedged reads In order to use HDFS hedged reads, the hdfsPread API must be used instead of the hdfsRead() call. Adds a flag to use hdfsPread: --use_hdfs_pread Testing: Running existing tests with this flag enabled. Cluster testing with HDFS hedged reads enabled via the HDFS client config. Manually tested setting the 'max_chunk_size' to a small value to force multiple iterations of the while loop which would only normally happen on S3. Tested reading lineitem was OK. Change-Id: Iecc8b12aa20cbfe08f4ef6a08a191e49709d9525 Reviewed-on: http://gerrit.cloudera.org:8080/5635 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins

            People

            • Assignee:
              mjacobs Matthew Jacobs
              Reporter:
              mjacobs Matthew Jacobs
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development