Hadoop YARN
  1. Hadoop YARN
  2. YARN-3172

MR-279: Write a simple Java application

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently for isolation purposes, many simple java applications run in cluster with 1 map only job. (eg. Oozie). This is not really required with nextgen hadoop (mrv2) and non-MR apps are first class and easy to write.
      A simple hadoop java app can be written which runs in the cluster in the user space.

        Issue Links

          Activity

          Hide
          Devaraj K added a comment -

          Thanks Allen Wittenauer for looking into the patch and trying to commit it. I will rebase and update the patch.

          Show
          Devaraj K added a comment - Thanks Allen Wittenauer for looking into the patch and trying to commit it. I will rebase and update the patch.
          Hide
          Allen Wittenauer added a comment -

          OK, looks like it needs to get rebased because this was before the 900th re-arrangement of the dir structure.

          Show
          Allen Wittenauer added a comment - OK, looks like it needs to get rebased because this was before the 900th re-arrangement of the dir structure.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 37s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 5m 27s There were no new checkstyle issues.
          0 shellcheck 5m 27s Shellcheck was not available.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 0s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
              39m 47s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12523089/MAPREDUCE-2720.patch
          Optional Tests javadoc javac unit findbugs checkstyle shellcheck
          git revision trunk / 8847777
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7505/artifact/patchprocess/whitespace.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7505/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7505/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 37s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 whitespace 0m 0s The patch has 8 line(s) that end in whitespace. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 5m 27s There were no new checkstyle issues. 0 shellcheck 5m 27s Shellcheck was not available. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 0s The patch does not introduce any new Findbugs (version 2.0.3) warnings.     39m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12523089/MAPREDUCE-2720.patch Optional Tests javadoc javac unit findbugs checkstyle shellcheck git revision trunk / 8847777 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7505/artifact/patchprocess/whitespace.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7505/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7505/console This message was automatically generated.
          Hide
          Allen Wittenauer added a comment - - edited

          Welp, I'm committing this to trunk if test-patch says it is still good to go.

          Show
          Allen Wittenauer added a comment - - edited Welp, I'm committing this to trunk if test-patch says it is still good to go.
          Hide
          Allen Wittenauer added a comment - - edited

          This has been open for about 4 years with a lot of people watching. It still applies.

          Do we feel that the distributed shell example is good enough? Would it benefit the community to also have this committed along side of it? Or does Slider, et. al., fill the original need here?

          We should make a decision in the next week: commit or close.

          Show
          Allen Wittenauer added a comment - - edited This has been open for about 4 years with a lot of people watching. It still applies. Do we feel that the distributed shell example is good enough? Would it benefit the community to also have this committed along side of it? Or does Slider, et. al., fill the original need here? We should make a decision in the next week: commit or close.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523089/MAPREDUCE-2720.patch
          against trunk revision 7c6b654.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6590//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6590//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523089/MAPREDUCE-2720.patch against trunk revision 7c6b654. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6590//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6590//console This message is automatically generated.
          Hide
          Sharad Agarwal added a comment -

          correction above:
          ..make sense to use the MAPREDUCE-2719. ..

          Show
          Sharad Agarwal added a comment - correction above: ..make sense to use the MAPREDUCE-2719 . ..
          Hide
          Sharad Agarwal added a comment -

          On thinking more on this, I think this is a special case of a shell command application. So instead of having a parallel implementation, it makes sense to use the MAPREDUCE-2720. The only downside is that shell command application forks another process to run the command which can be avoided in a java app. But that seems a very minor downside.

          Show
          Sharad Agarwal added a comment - On thinking more on this, I think this is a special case of a shell command application. So instead of having a parallel implementation, it makes sense to use the MAPREDUCE-2720 . The only downside is that shell command application forks another process to run the command which can be avoided in a java app. But that seems a very minor downside.
          Hide
          Devaraj K added a comment -

          The above test failures are not related to the patch.

          Show
          Devaraj K added a comment - The above test failures are not related to the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523089/MAPREDUCE-2720.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
          org.apache.hadoop.yarn.server.TestDiskFailures
          org.apache.hadoop.yarn.server.TestContainerManagerSecurity
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.v2.TestMRJobs
          org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
          org.apache.hadoop.mapred.TestMiniMRBringup
          org.apache.hadoop.mapred.TestMiniMRChildTask
          org.apache.hadoop.mapred.TestReduceFetch
          org.apache.hadoop.mapred.TestClusterMRNotification
          org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.mapreduce.TestChild
          org.apache.hadoop.mapred.TestMiniMRClientCluster
          org.apache.hadoop.ipc.TestSocketFactory
          org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
          org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
          org.apache.hadoop.mapred.TestClientRedirect
          org.apache.hadoop.mapred.TestLazyOutput
          org.apache.hadoop.mapred.TestJobCleanup
          org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.mapreduce.v2.TestRMNMInfo
          org.apache.hadoop.mapred.TestClusterMapReduceTestCase
          org.apache.hadoop.mapreduce.v2.TestNonExistentJob
          org.apache.hadoop.mapred.TestJobSysDirWithDFS
          org.apache.hadoop.mapreduce.v2.TestUberAM
          org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
          org.apache.hadoop.mapred.TestJobName
          org.apache.hadoop.mapreduce.security.TestJHSSecurity

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2245//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2245//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523089/MAPREDUCE-2720.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell org.apache.hadoop.yarn.server.TestDiskFailures org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMiniMRBringup org.apache.hadoop.mapred.TestMiniMRChildTask org.apache.hadoop.mapred.TestReduceFetch org.apache.hadoop.mapred.TestClusterMRNotification org.apache.hadoop.mapred.TestReduceFetchFromPartialMem org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.mapreduce.TestChild org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.ipc.TestSocketFactory org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService org.apache.hadoop.mapreduce.v2.TestMROldApiJobs org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapred.TestLazyOutput org.apache.hadoop.mapred.TestJobCleanup org.apache.hadoop.mapreduce.TestMapReduceLazyOutput org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapreduce.v2.TestRMNMInfo org.apache.hadoop.mapred.TestClusterMapReduceTestCase org.apache.hadoop.mapreduce.v2.TestNonExistentJob org.apache.hadoop.mapred.TestJobSysDirWithDFS org.apache.hadoop.mapreduce.v2.TestUberAM org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser org.apache.hadoop.mapred.TestJobName org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2245//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2245//console This message is automatically generated.
          Hide
          Devaraj K added a comment -

          I have attached first level patch for review. Please give your comments/suggestions on the patch.

          Show
          Devaraj K added a comment - I have attached first level patch for review. Please give your comments/suggestions on the patch.
          Hide
          Sharad Agarwal added a comment -

          This issue can share lot of functionality from MAPREDUCE-2719 if option #2 is chosen from http://bit.ly/pZkCjo

          Show
          Sharad Agarwal added a comment - This issue can share lot of functionality from MAPREDUCE-2719 if option #2 is chosen from http://bit.ly/pZkCjo

            People

            • Assignee:
              Devaraj K
              Reporter:
              Sharad Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:

                Development