Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None
    • Target Version/s:

      Description

      Clean up a bunch of existing javac warnings in hadoop-streaming module.

        Issue Links

          Activity

          Hide
          Harsh J added a comment -

          Patch that cleans all warnings of hadoop-streaming sub-module save for usage of MiniMRCluster and DistributedCache and one instance of StreamJob's non usual constructor cause a job reference is still required there.

          Show
          Harsh J added a comment - Patch that cleans all warnings of hadoop-streaming sub-module save for usage of MiniMRCluster and DistributedCache and one instance of StreamJob's non usual constructor cause a job reference is still required there.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523700/0008-MapReduce-Streaming-Cleanup.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 25 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/876//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523700/0008-MapReduce-Streaming-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 25 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/876//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          A few things:

          1. Looks like even Hadoop-Common QA does not pick up hadoop-tools changes.
          2. The eclipse's auto unused import fixer removed RecRecord imports which caused TestIO not to compile. I fixed those manually in this new patch.
          3. TestStreaming had a change in original patch that caused it to lose info about the running jobs and such. I tweaked the change to use a non ToolRunner approach, and TestStreamingCounters/Combiner now work nicely again - minus deprecation warnings.

          Since Hadoop QA bot can't run the tests, here is the test result manually run, with patch applied:

          cd hadoop-tools/hadoop-streaming; mvn clean test

          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.streaming.TestAutoInputFormat
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.663 sec
          Running org.apache.hadoop.streaming.TestClassWithNoPackage
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.261 sec
          Running org.apache.hadoop.streaming.TestDumpTypedBytes
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.21 sec
          Running org.apache.hadoop.streaming.TestFileArgs
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.49 sec
          Running org.apache.hadoop.streaming.TestGzipInput
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.188 sec
          Running org.apache.hadoop.streaming.TestLoadTypedBytes
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.486 sec
          Running org.apache.hadoop.streaming.TestMRFramework
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.265 sec
          Running org.apache.hadoop.streaming.TestMultipleArchiveFiles
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.727 sec
          Running org.apache.hadoop.streaming.TestMultipleCachefiles
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.324 sec
          Running org.apache.hadoop.streaming.TestRawBytesStreaming
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.07 sec
          Running org.apache.hadoop.streaming.TestStreamAggregate
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.062 sec
          Running org.apache.hadoop.streaming.TestStreamDataProtocol
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.058 sec
          Running org.apache.hadoop.streaming.TestStreaming
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.165 sec
          Running org.apache.hadoop.streaming.TestStreamingBackground
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.09 sec
          Running org.apache.hadoop.streaming.TestStreamingBadRecords
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.769 sec
          Running org.apache.hadoop.streaming.TestStreamingCombiner
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.21 sec
          Running org.apache.hadoop.streaming.TestStreamingCounters
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.191 sec
          Running org.apache.hadoop.streaming.TestStreamingExitStatus
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.031 sec
          Running org.apache.hadoop.streaming.TestStreamingFailure
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.788 sec
          Running org.apache.hadoop.streaming.TestStreamingKeyValue
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.324 sec
          Running org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.626 sec
          Running org.apache.hadoop.streaming.TestStreamingSeparator
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.068 sec
          Running org.apache.hadoop.streaming.TestStreamingStderr
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.579 sec
          Running org.apache.hadoop.streaming.TestStreamingTaskLog
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.659 sec
          Running org.apache.hadoop.streaming.TestStreamJob
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.761 sec
          Running org.apache.hadoop.streaming.TestStreamReduceNone
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec
          Running org.apache.hadoop.streaming.TestStreamXmlMultipleRecords
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.238 sec
          Running org.apache.hadoop.streaming.TestStreamXmlRecordReader
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.152 sec
          Running org.apache.hadoop.streaming.TestSymLink
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.418 sec
          Running org.apache.hadoop.streaming.TestTypedBytesStreaming
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec
          Running org.apache.hadoop.streaming.TestUnconsumedInput
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.139 sec
          Running org.apache.hadoop.typedbytes.TestIO
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.166 sec
          Running org.apache.hadoop.typedbytes.TestTypedBytesWritable
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec
          
          Results :
          
          Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
          
          Show
          Harsh J added a comment - A few things: Looks like even Hadoop-Common QA does not pick up hadoop-tools changes. The eclipse's auto unused import fixer removed RecRecord imports which caused TestIO not to compile. I fixed those manually in this new patch. TestStreaming had a change in original patch that caused it to lose info about the running jobs and such. I tweaked the change to use a non ToolRunner approach, and TestStreamingCounters/Combiner now work nicely again - minus deprecation warnings. Since Hadoop QA bot can't run the tests, here is the test result manually run, with patch applied: cd hadoop-tools/hadoop-streaming; mvn clean test ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.streaming.TestAutoInputFormat Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.663 sec Running org.apache.hadoop.streaming.TestClassWithNoPackage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.261 sec Running org.apache.hadoop.streaming.TestDumpTypedBytes Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.21 sec Running org.apache.hadoop.streaming.TestFileArgs Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.49 sec Running org.apache.hadoop.streaming.TestGzipInput Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.188 sec Running org.apache.hadoop.streaming.TestLoadTypedBytes Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.486 sec Running org.apache.hadoop.streaming.TestMRFramework Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.265 sec Running org.apache.hadoop.streaming.TestMultipleArchiveFiles Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.727 sec Running org.apache.hadoop.streaming.TestMultipleCachefiles Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.324 sec Running org.apache.hadoop.streaming.TestRawBytesStreaming Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.07 sec Running org.apache.hadoop.streaming.TestStreamAggregate Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.062 sec Running org.apache.hadoop.streaming.TestStreamDataProtocol Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.058 sec Running org.apache.hadoop.streaming.TestStreaming Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.165 sec Running org.apache.hadoop.streaming.TestStreamingBackground Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.09 sec Running org.apache.hadoop.streaming.TestStreamingBadRecords Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.769 sec Running org.apache.hadoop.streaming.TestStreamingCombiner Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.21 sec Running org.apache.hadoop.streaming.TestStreamingCounters Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.191 sec Running org.apache.hadoop.streaming.TestStreamingExitStatus Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.031 sec Running org.apache.hadoop.streaming.TestStreamingFailure Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.788 sec Running org.apache.hadoop.streaming.TestStreamingKeyValue Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.324 sec Running org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.626 sec Running org.apache.hadoop.streaming.TestStreamingSeparator Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.068 sec Running org.apache.hadoop.streaming.TestStreamingStderr Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.579 sec Running org.apache.hadoop.streaming.TestStreamingTaskLog Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.659 sec Running org.apache.hadoop.streaming.TestStreamJob Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.761 sec Running org.apache.hadoop.streaming.TestStreamReduceNone Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec Running org.apache.hadoop.streaming.TestStreamXmlMultipleRecords Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.238 sec Running org.apache.hadoop.streaming.TestStreamXmlRecordReader Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.152 sec Running org.apache.hadoop.streaming.TestSymLink Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.418 sec Running org.apache.hadoop.streaming.TestTypedBytesStreaming Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec Running org.apache.hadoop.streaming.TestUnconsumedInput Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.139 sec Running org.apache.hadoop.typedbytes.TestIO Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.166 sec Running org.apache.hadoop.typedbytes.TestTypedBytesWritable Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec Results : Tests run: 59, Failures: 0, Errors: 0, Skipped: 0
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12523715/0008-MapReduce-Streaming-Cleanup.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 25 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/878//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12523715/0008-MapReduce-Streaming-Cleanup.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 25 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/878//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Patches were too broad and have gone stale. Will address these forms of issue over separate, smaller and more divided JIRAs in future.

          Closing out parent JIRA MAPREDUCE-4172, and hence closing out this.

          Show
          Harsh J added a comment - Patches were too broad and have gone stale. Will address these forms of issue over separate, smaller and more divided JIRAs in future. Closing out parent JIRA MAPREDUCE-4172 , and hence closing out this.

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development