Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6823

FileOutputFormat to support configurable PathOutputCommitter factory

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-alpha2
    • Fix Version/s: None
    • Component/s: mrv2
    • Labels:
      None
    • Environment:

      Targeting S3 as the output of work

    • Target Version/s:

      Description

      In HADOOP-13786 I'm adding a custom subclass for FileOutputFormat, one which can talk direct to the S3A Filesystem for more efficient operations, better failure modes, and, most critically, as part of HADOOP-13345, atomic commit of output. The normal committer relies on directory rename() being atomic for this; for S3 we don't have that luxury.

      To support a custom committer, we need to be able to tell FileOutputFormat (and implicitly, all subclasses which don't have their own custom committer), to use our new S3AOutputCommitter.

      I propose:

      1. FileOutputFormat takes a factory to create committers.
      2. The factory to take a URI and TaskAttemptContext and return a committer
      3. the default implementation always returns a FileOutputCommitter
      4. A configuration option allows a new factory to be named
      5. An S3AOutputCommitterFactory to return a FileOutputCommitter or new S3AOutputCommitter depending upon the URI of the destination.

      Note that MRv1 already supports configurable committers; this is only the V2 API

      1. HADOOP-13786-HADOOP-13345-001.patch
        126 kB
        Steve Loughran
      2. MAPREDUCE-6823-002.patch
        43 kB
        Steve Loughran
      3. MAPREDUCE-6823-002.patch
        43 kB
        Steve Loughran
      4. MAPREDUCE-6823-004.patch
        46 kB
        Steve Loughran

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is the initial HADOOP-13786 001 PoC patch, to give the MR bit of the code some testing too. It adds a new factory for FileOutputFormatter to use when creating committers; the default one returns FileOutputCommitter instances as normal; a special S3a one in hadoop-aws to handle S3a specific operations.

          Now, the other way to do this (given the need to keep the s3a code in the s3a module) would be to allow a notion of a new algorithm, one which relayed to an implementation of an interface. That would hand a problem not addressed here: how to address subclasses of FileOutputFormat with custom subclasses of FileOutputCommitter, and make it easier to add committers for other non-FS-destinations, namely the other object stores.

          However, it would be a more significant change to FileOutputCommitter; I could go that way, but it'd need support before I started.

          Show
          stevel@apache.org Steve Loughran added a comment - This is the initial HADOOP-13786 001 PoC patch, to give the MR bit of the code some testing too. It adds a new factory for FileOutputFormatter to use when creating committers; the default one returns FileOutputCommitter instances as normal; a special S3a one in hadoop-aws to handle S3a specific operations. Now, the other way to do this (given the need to keep the s3a code in the s3a module) would be to allow a notion of a new algorithm, one which relayed to an implementation of an interface. That would hand a problem not addressed here: how to address subclasses of FileOutputFormat with custom subclasses of FileOutputCommitter , and make it easier to add committers for other non-FS-destinations, namely the other object stores. However, it would be a more significant change to FileOutputCommitter ; I could go that way, but it'd need support before I started.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 1m 45s Maven dependency ordering for branch
          +1 mvninstall 7m 39s HADOOP-13345 passed
          +1 compile 7m 39s HADOOP-13345 passed
          +1 checkstyle 1m 30s HADOOP-13345 passed
          +1 mvnsite 0m 57s HADOOP-13345 passed
          +1 mvneclipse 0m 30s HADOOP-13345 passed
          +1 findbugs 1m 16s HADOOP-13345 passed
          +1 javadoc 0m 36s HADOOP-13345 passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 0m 43s the patch passed
          +1 compile 7m 17s the patch passed
          -1 javac 7m 17s root generated 2 new + 716 unchanged - 0 fixed = 718 total (was 716)
          -1 checkstyle 1m 35s root: The patch generated 48 new + 66 unchanged - 1 fixed = 114 total (was 67)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          -1 whitespace 0m 0s The patch has 43 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 50s the patch passed
          -1 javadoc 0m 16s hadoop-aws in the patch failed.
          +1 unit 2m 52s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 0m 38s hadoop-aws in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          57m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843640/HADOOP-13786-HADOOP-13345-001.patch
          JIRA Issue MAPREDUCE-6823
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 313569e92b73 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision HADOOP-13345 / c7885de
          Default Java 1.8.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 1m 45s Maven dependency ordering for branch +1 mvninstall 7m 39s HADOOP-13345 passed +1 compile 7m 39s HADOOP-13345 passed +1 checkstyle 1m 30s HADOOP-13345 passed +1 mvnsite 0m 57s HADOOP-13345 passed +1 mvneclipse 0m 30s HADOOP-13345 passed +1 findbugs 1m 16s HADOOP-13345 passed +1 javadoc 0m 36s HADOOP-13345 passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 7m 17s the patch passed -1 javac 7m 17s root generated 2 new + 716 unchanged - 0 fixed = 718 total (was 716) -1 checkstyle 1m 35s root: The patch generated 48 new + 66 unchanged - 1 fixed = 114 total (was 67) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 34s the patch passed -1 whitespace 0m 0s The patch has 43 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 50s the patch passed -1 javadoc 0m 16s hadoop-aws in the patch failed. +1 unit 2m 52s hadoop-mapreduce-client-core in the patch passed. +1 unit 0m 38s hadoop-aws in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 57m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843640/HADOOP-13786-HADOOP-13345-001.patch JIRA Issue MAPREDUCE-6823 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 313569e92b73 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision HADOOP-13345 / c7885de Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-aws.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6848/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Cancelling this PoC; redesigning. In order to support existing subclasses of FOF (e.g. the Parquet one); we'll have to come in lower.

          I propose adding a new algorithm, "3", which really means "plug in a new committer of classname X", with another property to define that classname. We can then add an s3 committer which supports this new protocol.

          This does mean that we will need to define a committer plugin...that we can declare as unstable/limited private, and implement the s3a one

          Show
          stevel@apache.org Steve Loughran added a comment - Cancelling this PoC; redesigning. In order to support existing subclasses of FOF (e.g. the Parquet one); we'll have to come in lower. I propose adding a new algorithm, "3", which really means "plug in a new committer of classname X", with another property to define that classname. We can then add an s3 committer which supports this new protocol. This does mean that we will need to define a committer plugin...that we can declare as unstable/limited private, and implement the s3a one
          Hide
          stevel@apache.org Steve Loughran added a comment -

          *2017/06/23 update* no, that's just messy. Best to find when those committers are used and allow them to be more generic. Example: all the parquet one does is add an optional schema summary file. If you don't want that, any FOF committer can be used

          Resubmitting the original patch, as it stands, from HADOOP-13786

          Show
          stevel@apache.org Steve Loughran added a comment - * 2017/06/23 update * no, that's just messy. Best to find when those committers are used and allow them to be more generic. Example: all the parquet one does is add an optional schema summary file. If you don't want that, any FOF committer can be used Resubmitting the original patch, as it stands, from HADOOP-13786
          Hide
          fabbri Aaron Fabbri added a comment -

          Hey Steve Loughran. I will rebase this patch on trunk unless you've already done it. Just let me know.

          Show
          fabbri Aaron Fabbri added a comment - Hey Steve Loughran . I will rebase this patch on trunk unless you've already done it. Just let me know.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          no, I've been doing work on this in my HADOOP-13786 branch, working with Ewan Higgs pointing out where it doesn't work. I'll need to pull out what's in that branch, src and test alike

          Show
          stevel@apache.org Steve Loughran added a comment - no, I've been doing work on this in my HADOOP-13786 branch, working with Ewan Higgs pointing out where it doesn't work. I'll need to pull out what's in that branch , src and test alike
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 002

          This is the mapreduce part of the latest HADOOP-13786 patch. It adds

          • the notion of a committer factory which is given the job configuration and destinatioon path, and returns the committer to use for this job.
          1. The ability to declare a committer factory for org.apache.hadoop.mapreduce.lib.output.FileOutputFormat in the option mapreduce.outputcommitter.factory.class
          2. The ability to register the default committer to use for different file formats (searched for if mapreduce.outputcommitter.factory.class is unset. This lets you define different committers for s3a, wasb, etc
          3. the default committer factory for creating, FileOutputCommitter instances
          4. Another committer factory for creating a named instance of PathOutputCommitter (see MAPREDUCE-6956), though the property mapreduce.outputcommitter.named.classname
          5. Static methods to make this straightforward to use
          6. FileOutputFormat wired up to use this factory mechanism.
          7. Tests for all this

          As a result of this, you can now define a committer factory/committer class for file output on an explicit per-job or implicit per-destination basis. As these are subclasses of PathOutputCommitter, they don't have to go near the complexity of FileOutputFormat and its existing ls/rename/merge strategy, which is high performance and reliable for "real" filesystems, but not for object stores or other destinations.
          Because the factories are used in FileOutputFormat, all file output formats which don't provide their own committer will get the feature too. Those which do provide their own committer (ParquetOutputFormat) don't pick up this new feature. It's left to applications to sort that out by calling the PathOutputCommitterFactory directly (which I have been doing downstream)

          The per-dest logic is used in HADOOP-13786 for committer factories, most interesting being the "DynamicCommitterFactory" which chooses the actual committer based on the settings of the destination bucket. Thats why the complexity of config -> factory instance -> committer instance is important: it offloads the more complex decision making to the factories. The per-fs-schema logic allows us to declare different factories for different filesystems, and the factories making the final call

          mapreduce.outputcommitter.factory.scheme.s3a  = org.apache.hadoop.fs.s3a.commit.DynamicCommitterFactory
          

          At the same time, jobs can override this with their own decisions, especially with a simple factory which just instantiates a committer class from the propery mapreduce.outputcommitter.named.classname. This gives the MRv2 output formats the same feature that MRv1 has long had: the ability to declare a new output committer for file outputs

          mapreduce.outputcommitter.factory.class = org.apache.hadoop.mapreduce.lib.output.NamedCommitterFactory
          mapreduce.outputcommitter.named.classname = YOUR-COMMITTER-CLASSNAME
          

          This does not have any impact on normal commit operations, as the default factory creates FileOutputCommitters, as before. It merely offers the ability to change committers & makes it easy to do this for specific filesystems.

          This has now been tested all the way through Spark; with a change to its parquet commit logic (SPARK-22217) and a special PathOutputCommitter, this feature lets us write data direct to S3 through Hadoop MR and spark, with ORC, Parquet and other formats. It is also lined up to support any other committers/filesystem committers people write.

          Testing: unit tests here, functional ITests in hadoop-aws, downstream tests elsewhere.

          Things to consider/refine

          • Config option names?
          • Best way to document. I've got an empty option in mapred-default.xml, otherwise its all in the javadocs.
          • Should the factory code all go into its own package, e.g. org.apache.hadoop.mapreduce.lib.output.factory?
          Show
          stevel@apache.org Steve Loughran added a comment - Patch 002 This is the mapreduce part of the latest HADOOP-13786 patch. It adds the notion of a committer factory which is given the job configuration and destinatioon path, and returns the committer to use for this job. The ability to declare a committer factory for org.apache.hadoop.mapreduce.lib.output.FileOutputFormat in the option mapreduce.outputcommitter.factory.class The ability to register the default committer to use for different file formats (searched for if mapreduce.outputcommitter.factory.class is unset. This lets you define different committers for s3a, wasb, etc the default committer factory for creating, FileOutputCommitter instances Another committer factory for creating a named instance of PathOutputCommitter (see MAPREDUCE-6956 ), though the property mapreduce.outputcommitter.named.classname Static methods to make this straightforward to use FileOutputFormat wired up to use this factory mechanism. Tests for all this As a result of this, you can now define a committer factory/committer class for file output on an explicit per-job or implicit per-destination basis. As these are subclasses of PathOutputCommitter, they don't have to go near the complexity of FileOutputFormat and its existing ls/rename/merge strategy, which is high performance and reliable for "real" filesystems, but not for object stores or other destinations. Because the factories are used in FileOutputFormat, all file output formats which don't provide their own committer will get the feature too. Those which do provide their own committer (ParquetOutputFormat) don't pick up this new feature. It's left to applications to sort that out by calling the PathOutputCommitterFactory directly (which I have been doing downstream) The per-dest logic is used in HADOOP-13786 for committer factories, most interesting being the "DynamicCommitterFactory" which chooses the actual committer based on the settings of the destination bucket. Thats why the complexity of config -> factory instance -> committer instance is important: it offloads the more complex decision making to the factories. The per-fs-schema logic allows us to declare different factories for different filesystems, and the factories making the final call mapreduce.outputcommitter.factory.scheme.s3a = org.apache.hadoop.fs.s3a.commit.DynamicCommitterFactory At the same time, jobs can override this with their own decisions, especially with a simple factory which just instantiates a committer class from the propery mapreduce.outputcommitter.named.classname. This gives the MRv2 output formats the same feature that MRv1 has long had: the ability to declare a new output committer for file outputs mapreduce.outputcommitter.factory.class = org.apache.hadoop.mapreduce.lib.output.NamedCommitterFactory mapreduce.outputcommitter.named.classname = YOUR-COMMITTER-CLASSNAME This does not have any impact on normal commit operations, as the default factory creates FileOutputCommitters, as before. It merely offers the ability to change committers & makes it easy to do this for specific filesystems. This has now been tested all the way through Spark; with a change to its parquet commit logic ( SPARK-22217 ) and a special PathOutputCommitter, this feature lets us write data direct to S3 through Hadoop MR and spark, with ORC, Parquet and other formats. It is also lined up to support any other committers/filesystem committers people write. Testing: unit tests here, functional ITests in hadoop-aws, downstream tests elsewhere. Things to consider/refine Config option names? Best way to document. I've got an empty option in mapred-default.xml, otherwise its all in the javadocs. Should the factory code all go into its own package, e.g. org.apache.hadoop.mapreduce.lib.output.factory ?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 5m 22s Docker failed to build yetus/hadoop:0de40f0.



          Subsystem Report/Notes
          JIRA Issue MAPREDUCE-6823
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892051/MAPREDUCE-6823-002.patch
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7186/console
          Powered by Apache Yetus 0.5.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 5m 22s Docker failed to build yetus/hadoop:0de40f0. Subsystem Report/Notes JIRA Issue MAPREDUCE-6823 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892051/MAPREDUCE-6823-002.patch Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7186/console Powered by Apache Yetus 0.5.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 15m 35s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 15m 37s trunk passed
          +1 compile 0m 31s trunk passed
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 0m 35s trunk passed
          +1 findbugs 0m 56s trunk passed
          +1 javadoc 0m 24s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 25s the patch passed
          -1 javac 0m 25s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 2 new + 161 unchanged - 0 fixed = 163 total (was 161)
          -1 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 13 new + 44 unchanged - 1 fixed = 57 total (was 45)
          +1 mvnsite 0m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 1s the patch passed
          -1 javadoc 0m 20s hadoop-mapreduce-client-core in the patch failed.
                Other Tests
          +1 unit 2m 55s hadoop-mapreduce-client-core in the patch passed.
          +1 asflicense 0m 13s The patch does not generate ASF License warnings.
          40m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0de40f0
          JIRA Issue MAPREDUCE-6823
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892405/MAPREDUCE-6823-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux fc5bf22a8744 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9fcc3a1
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/console
          Powered by Apache Yetus 0.5.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 15m 35s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 37s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 35s trunk passed +1 findbugs 0m 56s trunk passed +1 javadoc 0m 24s trunk passed       Patch Compile Tests +1 mvninstall 0m 25s the patch passed +1 compile 0m 25s the patch passed -1 javac 0m 25s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 2 new + 161 unchanged - 0 fixed = 163 total (was 161) -1 checkstyle 0m 17s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: The patch generated 13 new + 44 unchanged - 1 fixed = 57 total (was 45) +1 mvnsite 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 1s the patch passed -1 javadoc 0m 20s hadoop-mapreduce-client-core in the patch failed.       Other Tests +1 unit 2m 55s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 13s The patch does not generate ASF License warnings. 40m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:0de40f0 JIRA Issue MAPREDUCE-6823 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892405/MAPREDUCE-6823-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux fc5bf22a8744 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9fcc3a1 Default Java 1.8.0_144 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/artifact/patchprocess/patch-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7187/console Powered by Apache Yetus 0.5.0 http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 004; in sync with the S3A patch being revieed at HADOOP-14971/HADOOP-15003

          dealing with the various checkstyle issues. This code has been tested all the way through spark now, though it's not completely hidden, particularly in the case of Parquet, where Spark has some hard expectations about the type of committer there. All is working (testable, because the _SUCCESS file generated by the new committers contains some summary data)

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 004; in sync with the S3A patch being revieed at HADOOP-14971 / HADOOP-15003 dealing with the various checkstyle issues. This code has been tested all the way through spark now, though it's not completely hidden, particularly in the case of Parquet, where Spark has some hard expectations about the type of committer there. All is working (testable, because the _SUCCESS file generated by the new committers contains some summary data)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 4s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
                trunk Compile Tests
          0 mvndep 1m 0s Maven dependency ordering for branch
          +1 mvninstall 23m 53s trunk passed
          +1 compile 2m 54s trunk passed
          +1 checkstyle 0m 52s trunk passed
          +1 mvnsite 1m 49s trunk passed
          +1 shadedclient 15m 5s branch has no errors when building and testing our client artifacts.
          +1 findbugs 2m 27s trunk passed
          +1 javadoc 1m 5s trunk passed
                Patch Compile Tests
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 39s the patch passed
          +1 compile 2m 44s the patch passed
          -1 javac 2m 44s hadoop-mapreduce-project_hadoop-mapreduce-client generated 2 new + 333 unchanged - 0 fixed = 335 total (was 333)
          -0 checkstyle 0m 44s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 3 new + 45 unchanged - 3 fixed = 48 total (was 48)
          +1 mvnsite 1m 30s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 shadedclient 13m 14s patch has no errors when building and testing our client artifacts.
          +1 findbugs 3m 6s the patch passed
          +1 javadoc 1m 7s the patch passed
                Other Tests
          +1 unit 3m 59s hadoop-mapreduce-client-core in the patch passed.
          +1 unit 11m 51s hadoop-mapreduce-client-app in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          99m 47s



          Subsystem Report/Notes
          Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639
          JIRA Issue MAPREDUCE-6823
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896218/MAPREDUCE-6823-004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
          uname Linux 6d985f4fa1eb 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/patchprocess/precommit/personality/provided.sh
          git revision trunk / dcd99c4
          maven version: Apache Maven 3.3.9
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/artifact/out/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/testReport/
          Max. process+thread count 508 (vs. ulimit of 5000)
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/console
          Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 4s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.       trunk Compile Tests 0 mvndep 1m 0s Maven dependency ordering for branch +1 mvninstall 23m 53s trunk passed +1 compile 2m 54s trunk passed +1 checkstyle 0m 52s trunk passed +1 mvnsite 1m 49s trunk passed +1 shadedclient 15m 5s branch has no errors when building and testing our client artifacts. +1 findbugs 2m 27s trunk passed +1 javadoc 1m 5s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 2m 44s the patch passed -1 javac 2m 44s hadoop-mapreduce-project_hadoop-mapreduce-client generated 2 new + 333 unchanged - 0 fixed = 335 total (was 333) -0 checkstyle 0m 44s hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 3 new + 45 unchanged - 3 fixed = 48 total (was 48) +1 mvnsite 1m 30s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 shadedclient 13m 14s patch has no errors when building and testing our client artifacts. +1 findbugs 3m 6s the patch passed +1 javadoc 1m 7s the patch passed       Other Tests +1 unit 3m 59s hadoop-mapreduce-client-core in the patch passed. +1 unit 11m 51s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 99m 47s Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 JIRA Issue MAPREDUCE-6823 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12896218/MAPREDUCE-6823-004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml uname Linux 6d985f4fa1eb 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / dcd99c4 maven version: Apache Maven 3.3.9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/artifact/out/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/testReport/ Max. process+thread count 508 (vs. ulimit of 5000) modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7220/console Powered by Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development