Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11003

org.apache.hadoop.util.Shell should not take a dependency on binaries being deployed when used as a library

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Major
        • Resolution: Duplicate
        • Affects Version/s: None
        • Fix Version/s: None
        • Component/s: util
        • Labels:
          None
        • Environment:

          Windows

        • Tags:
          windows, winutils

          Description

          HIVE-7845 shows how an exception is being thrown when org.apache.hadoop.util.Shell is being used as a library, not as part of a deployed Hadoop environment.

          13:20:00 [ERROR pool-2-thread-4 Shell.getWinUtilsPath] Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
               at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:324)
               at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:339)
               at org.apache.hadoop.util.Shell.<clinit>(Shell.java:332)
               at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:918)
               at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:228)

          There are similar native dependencies (eg. NativeIO and hadoop.dll) that handle lack of binaries with fallback to non-native code paths.

            Issue Links

            duplicates

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10775 Shell operations to fail with meaningful errors on windows if winutils.exe not found

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved
            is related to

            Bug - A problem which impairs or prevents the functions of the product. HADOOP-10051 winutil.exe is not included in hadoop bin tarball

            • Major - Major loss of function.
            • Open

            Improvement - An improvement or enhancement to an existing feature or task. HADOOP-10775 Shell operations to fail with meaningful errors on windows if winutils.exe not found

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Resolved
            relates to

            Bug - A problem which impairs or prevents the functions of the product. SPARK-2356 Exception: Could not locate executable null\bin\winutils.exe in the Hadoop

            • Critical - Crashes, loss of data, severe memory leak.
            • Open

            Bug - A problem which impairs or prevents the functions of the product. HIVE-7845 "Failed to locate the winutils binary" when loading JDBC driver on Windows

            • Major - Major loss of function.
            • Patch Available

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              it's not actually failing, but logging @ error. Note that winutils is needed not just server-side, but if you try to set file permissions or do symlinks in the file:// filesystem, so depending on what you want to do client-side it will be needed.

              What could be done is maybe lower the stack trace level to info, and at error print a message that includes a wiki URL to discuss the topic,

              Show
              stevel@apache.org Steve Loughran added a comment - it's not actually failing, but logging @ error. Note that winutils is needed not just server-side, but if you try to set file permissions or do symlinks in the file:// filesystem, so depending on what you want to do client-side it will be needed. What could be done is maybe lower the stack trace level to info, and at error print a message that includes a wiki URL to discuss the topic,
              Hide
              Permalink
              cnauroth Chris Nauroth added a comment -

              This feels like the behavior should be analogous to what we do in NativeCodeLoader: log once at warn level. The full stack trace doesn't have much value here IMO.

              Show
              cnauroth Chris Nauroth added a comment - This feels like the behavior should be analogous to what we do in NativeCodeLoader : log once at warn level. The full stack trace doesn't have much value here IMO.
              Hide
              Permalink
              aw Allen Wittenauer added a comment -

              One of the things I keep meaning to do is to wrap that warn with a way to turn it off because on systems that don't have native libs, it's not useful. So keep that in mind too....

              Show
              aw Allen Wittenauer added a comment - One of the things I keep meaning to do is to wrap that warn with a way to turn it off because on systems that don't have native libs, it's not useful. So keep that in mind too....
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              ...I'm starting to think we may want a policy option here

              • warn: as today
              • fail: refuse to come out to play
              • ignore: don't complain
              • no-native: don't even look
              Show
              stevel@apache.org Steve Loughran added a comment - ...I'm starting to think we may want a policy option here warn: as today fail: refuse to come out to play ignore: don't complain no-native: don't even look
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              What about having a special log category just for "no native libs"; Shell & NativeCodeLoader could share it?

              Show
              stevel@apache.org Steve Loughran added a comment - What about having a special log category just for "no native libs"; Shell & NativeCodeLoader could share it?
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              In HADOOP-10775 we're not logging until the path is needed, at which point an exception will be raised with meaningful text.

              Show
              stevel@apache.org Steve Loughran added a comment - In HADOOP-10775 we're not logging until the path is needed, at which point an exception will be raised with meaningful text.
              Hide
              Permalink
              stevel@apache.org Steve Loughran added a comment -

              Closing as fixed. There's still the dependency, but no warning unless/until the winutils path is actually needed

              Show
              stevel@apache.org Steve Loughran added a comment - Closing as fixed. There's still the dependency, but no warning unless/until the winutils path is actually needed

                People

                • Assignee:
                  stevel@apache.org Steve Loughran
                  Reporter:
                  rusanu Remus Rusanu
                • Votes:
                  3 Vote for this issue
                  Watchers:
                  8 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development