Hive
  1. Hive
  2. HIVE-6200

Hive custom SerDe cannot load DLL added by "ADD FILE" command

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      When custom SerDe need to load a DLL file added using "ADD FILE" command in HIVE, the loading fail with exception like "java.lang.UnsatisfiedLinkError:C:\tmp\admin2_6996@headnode0_201401100431_resources\hello.dll: Access is denied".
      The reason is when FileSystem creating local copy of the file, the permission of local file is set to default as "666". DLL file need "execute" permission to be loaded successfully.
      Similar scenario also happens when Hadoop localize files in distributed cache. The solution in Hadoop is to add "execute" permission to the file after localizationl.

      1. HIVE-6200.1.patch
        1 kB
        Shuaishuai Nie
      2. HIVE-6200.2.patch
        0.9 kB
        Shuaishuai Nie
      3. HIVE-6200.3.patch
        1 kB
        Shuaishuai Nie

        Activity

        Hide
        Thejas M Nair added a comment -

        As this seems to be required only on windows, how about adding a check if it is windows before changing the permission ?

        Show
        Thejas M Nair added a comment - As this seems to be required only on windows, how about adding a check if it is windows before changing the permission ?
        Hide
        Shuaishuai Nie added a comment -

        Hi Thejas M Nair. I think this problem is not limited to Windows. On both Windows and Linux, when creating the copy file in local FS, Hadoop will set the permission of the file to default permission (hard coded to 666). In the implementation of distributed cache in Hadoop, it uses the same way to add "execute" permission for the file regardless of the OS at TrackerDistributedCacheManager.downloadCacheObject().

        Show
        Shuaishuai Nie added a comment - Hi Thejas M Nair . I think this problem is not limited to Windows. On both Windows and Linux, when creating the copy file in local FS, Hadoop will set the permission of the file to default permission (hard coded to 666). In the implementation of distributed cache in Hadoop, it uses the same way to add "execute" permission for the file regardless of the OS at TrackerDistributedCacheManager.downloadCacheObject().
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12623035/HIVE-6200.1.patch

        ERROR: -1 due to 1 failed/errored test(s), 4925 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/918/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/918/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12623035

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12623035/HIVE-6200.1.patch ERROR: -1 due to 1 failed/errored test(s), 4925 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/918/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/918/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12623035
        Hide
        Thejas M Nair added a comment -

        +1

        Show
        Thejas M Nair added a comment - +1
        Hide
        Shuaishuai Nie added a comment -

        Validated the failed test in HIVE QA result. It is not related to the change made in this JIRA.

        Show
        Shuaishuai Nie added a comment - Validated the failed test in HIVE QA result. It is not related to the change made in this JIRA.
        Hide
        shanyu zhao added a comment -

        +1

        Show
        shanyu zhao added a comment - +1
        Hide
        Ashutosh Chauhan added a comment -

        Changing of permissions should be moved in downloadResource() method, such that we change permissions only when files are downloaded. Because, int that case we are guaranteed to successfully complete chmod operation. Otherwise, we may try to change permissions of local files (which may not be allowed since we may not own them).

        Also, patch needs a rebase.

        Show
        Ashutosh Chauhan added a comment - Changing of permissions should be moved in downloadResource() method, such that we change permissions only when files are downloaded. Because, int that case we are guaranteed to successfully complete chmod operation. Otherwise, we may try to change permissions of local files (which may not be allowed since we may not own them). Also, patch needs a rebase.
        Hide
        Shuaishuai Nie added a comment -

        Thanks for the commend Ashutosh Chauhan. Updated the patch

        Show
        Shuaishuai Nie added a comment - Thanks for the commend Ashutosh Chauhan . Updated the patch
        Hide
        Ashutosh Chauhan added a comment -

        +1

        Show
        Ashutosh Chauhan added a comment - +1
        Hide
        Alan Gates added a comment -

        Build failed with

        [ERROR] COMPILATION ERROR :
        [INFO] -------------------------------------------------------------
        [ERROR] /grid/0/hortonal/ptest/working_dir/working/apache-github-source/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java:[788,9] cannot find symbol
        symbol  : variable FileUtil
        location: class org.apache.hadoop.hive.ql.session.SessionState
        [INFO] 1 error
        [INFO] -------------------------------------------------------------
        [INFO] ------------------------------------------------------------------------
        [INFO] Reactor Summary:
        [INFO]
        [INFO] Hive .............................................. SUCCESS [52.644s]
        [INFO] Hive Ant Utilities ................................ SUCCESS [8.526s]
        [INFO] Hive Shims Common ................................. SUCCESS [18.758s]
        [INFO] Hive Shims 0.20 ................................... SUCCESS [20.058s]
        [INFO] Hive Shims Secure Common .......................... SUCCESS [4.372s]
        [INFO] Hive Shims 0.20S .................................. SUCCESS [1.443s]
        [INFO] Hive Shims 0.23 ................................... SUCCESS [48.812s]
        [INFO] Hive Shims ........................................ SUCCESS [0.227s]
        [INFO] Hive Common ....................................... SUCCESS [2.576s]
        [INFO] Hive Serde ........................................ SUCCESS [3.588s]
        [INFO] Hive Metastore .................................... SUCCESS [19.850s]
        [INFO] Hive Query Language ............................... FAILURE [23.695s]
        
        Show
        Alan Gates added a comment - Build failed with [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /grid/0/hortonal/ptest/working_dir/working/apache-github-source/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java:[788,9] cannot find symbol symbol : variable FileUtil location: class org.apache.hadoop.hive.ql.session.SessionState [INFO] 1 error [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hive .............................................. SUCCESS [52.644s] [INFO] Hive Ant Utilities ................................ SUCCESS [8.526s] [INFO] Hive Shims Common ................................. SUCCESS [18.758s] [INFO] Hive Shims 0.20 ................................... SUCCESS [20.058s] [INFO] Hive Shims Secure Common .......................... SUCCESS [4.372s] [INFO] Hive Shims 0.20S .................................. SUCCESS [1.443s] [INFO] Hive Shims 0.23 ................................... SUCCESS [48.812s] [INFO] Hive Shims ........................................ SUCCESS [0.227s] [INFO] Hive Common ....................................... SUCCESS [2.576s] [INFO] Hive Serde ........................................ SUCCESS [3.588s] [INFO] Hive Metastore .................................... SUCCESS [19.850s] [INFO] Hive Query Language ............................... FAILURE [23.695s]
        Hide
        Shuaishuai Nie added a comment -

        Sorry, HIVE-6200.2.patch is the wrong incomplete patch. Update with the new one HIVE-6200.3.patch

        Show
        Shuaishuai Nie added a comment - Sorry, HIVE-6200 .2.patch is the wrong incomplete patch. Update with the new one HIVE-6200 .3.patch
        Hide
        Hive QA added a comment -

        Overall: -1 at least one tests failed

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12636997/HIVE-6200.3.patch

        ERROR: -1 due to 1 failed/errored test(s), 5491 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part
        

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1983/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1983/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 1 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12636997

        Show
        Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12636997/HIVE-6200.3.patch ERROR: -1 due to 1 failed/errored test(s), 5491 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_dyn_part Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1983/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1983/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12636997
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Shuaishuai!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Shuaishuai!

          People

          • Assignee:
            Shuaishuai Nie
            Reporter:
            Shuaishuai Nie
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development