Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8581

Switch hdfs commandline calls in dataload and tests to use "-d" on s3 where applicable

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: Impala 3.3.0
    • Fix Version/s: Not Applicable
    • Component/s: Infrastructure
    • Labels:
    • Epic Color:
      ghx-label-9

      Description

      When doing a put or copyFromLocal via the HDFS commandline, it puts the file into a temporary file and then moves it into place. The "-d" option skips the temporary file and copies the file directly into place. On S3, this avoids an extra copy and can also avoid consistency issues. We should look into using the "-d" option across our tests. Our usages of the HDFS commandline generally don't require atomicity of this file copy, so using "-d" is unlikely to impact HDFS tests.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                joemcdonnell Joe McDonnell
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: