Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
      None
    • Environment:

      Solaris
      Illumos
      AIX
      ... likely others

      Description

      hadoop-functions.sh uses xargs -P in the ssh handler. -P is a GNU extension and is not available on all operating systems. We should add some detection for support and perform an appropriate action.

      1. HADOOP-11406.002.patch
        4 kB
        Sean Busbey
      2. HADOOP-11406.002.patch
        4 kB
        Kengo Seki
      3. HADOOP-11406.001.patch
        1 kB
        Kengo Seki

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/210/)
          HADOOP-11406. xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #210 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/210/ ) HADOOP-11406 . xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2140/)
          HADOOP-11406. xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2140 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2140/ ) HADOOP-11406 . xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #942 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/942/)
          HADOOP-11406. xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #942 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/942/ ) HADOOP-11406 . xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #212 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/212/)
          HADOOP-11406. xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84)

          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #212 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/212/ ) HADOOP-11406 . xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84) hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/200/)
          HADOOP-11406. xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84)

          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #200 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/200/ ) HADOOP-11406 . xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84) hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7917 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7917/)
          HADOOP-11406. xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84)

          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7917 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7917/ ) HADOOP-11406 . xargs -P is not portable (Kengo Seki via aw) (aw: rev 5504a261f829cf2e7b70970246bf5a55c172be84) hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          Hide
          aw Allen Wittenauer added a comment -

          +1 committed

          thanks!

          Show
          aw Allen Wittenauer added a comment - +1 committed thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings.
          +1 shellcheck 0m 5s There were no new shellcheck (v0.3.3) issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
              0m 24s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735118/HADOOP-11406.002.patch
          Optional Tests shellcheck
          git revision trunk / 446d515
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6813/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings. +1 shellcheck 0m 5s There were no new shellcheck (v0.3.3) issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace.     0m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735118/HADOOP-11406.002.patch Optional Tests shellcheck git revision trunk / 446d515 Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6813/console This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          reattaching last patch to get another QA run now that shellcheck is in place.

          Show
          busbey Sean Busbey added a comment - reattaching last patch to get another QA run now that shellcheck is in place.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 0m 0s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings.
          0 shellcheck 0m 15s Shellcheck was not available.
              0m 25s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12728457/HADOOP-11406.002.patch
          Optional Tests shellcheck
          git revision trunk / 7f07c4d
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6197/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 0m 0s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 release audit 0m 15s The applied patch does not increase the total number of release audit warnings. 0 shellcheck 0m 15s Shellcheck was not available.     0m 25s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12728457/HADOOP-11406.002.patch Optional Tests shellcheck git revision trunk / 7f07c4d Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6197/console This message was automatically generated.
          Hide
          sekikn Kengo Seki added a comment -

          Sorry for no updating for a long time. I'm attaching the revised patch.

          Show
          sekikn Kengo Seki added a comment - Sorry for no updating for a long time. I'm attaching the revised patch.
          Hide
          sekikn Kengo Seki added a comment -

          Thanks Allen Wittenauer, it sounds better. I'll do as follows:

          • in hadoop-functions.sh, move the for loop into another function as the default.
          • put the paralell xargs one into hadoop-user-functions.sh.example as a example to override above.
          Show
          sekikn Kengo Seki added a comment - Thanks Allen Wittenauer , it sounds better. I'll do as follows: in hadoop-functions.sh, move the for loop into another function as the default. put the paralell xargs one into hadoop-user-functions.sh.example as a example to override above.
          Hide
          aw Allen Wittenauer added a comment -

          So I've been thinking about this the past few days. Moving from xargs to a for loop means a pretty significant drop in performance. I'm sort of leaning towards keeping using the portable as the default and putting the paralell xargs one as an example in the hadoop-user-functions.sh. One thing that should probably be done in order to facilitate this is to move the actual loop into another function so that it makes it easy to replace.

          Thoughts?

          Show
          aw Allen Wittenauer added a comment - So I've been thinking about this the past few days. Moving from xargs to a for loop means a pretty significant drop in performance. I'm sort of leaning towards keeping using the portable as the default and putting the paralell xargs one as an example in the hadoop-user-functions.sh. One thing that should probably be done in order to facilitate this is to move the actual loop into another function so that it makes it easy to replace. Thoughts?
          Hide
          sekikn Kengo Seki added a comment -

          More precisely, a parallelized for loop.

          Show
          sekikn Kengo Seki added a comment - More precisely, a parallelized for loop.
          Hide
          sekikn Kengo Seki added a comment -

          Allen Wittenauer Brahma Reddy Battula According to POSIX standard, it seems difficult to know whether the error cause is lack of -P support or not, from status code or error message. So how about using for-loop instead of xargs? I attach a patch for review.

          Show
          sekikn Kengo Seki added a comment - Allen Wittenauer Brahma Reddy Battula According to POSIX standard, it seems difficult to know whether the error cause is lack of -P support or not, from status code or error message. So how about using for-loop instead of xargs? I attach a patch for review.
          Hide
          aw Allen Wittenauer added a comment -

          Possible actions might include:

          • Just throw an error and offer a suggested workaround (pdsh)
          • single step through the slave hosts, ignoring parallelism
          Show
          aw Allen Wittenauer added a comment - Possible actions might include: Just throw an error and offer a suggested workaround (pdsh) single step through the slave hosts, ignoring parallelism

            People

            • Assignee:
              sekikn Kengo Seki
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development