Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-1218

Do the dist-layout-stitching for Ozone after the test-compile phase

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      HDDS-1135 fixed the order of the maven goal executions to include all the required jar files in the distribution package.

      It turned out that the suggested compile phase is too early for the dist-layout-stitching.

      Int case of test-compile execution the shaded ozone datanode service plugin is not created (it's created at the package phase) but the hadoop-ozone/dist tries to use (copy it from the source):

      The error (from Yetus) is:

      [INFO] --- exec-maven-plugin:1.3.1:exec (dist) @ hadoop-ozone-dist ---
      cp: cannot stat '/testptch/hadoop/hadoop-ozone/objectstore-service/target/hadoop-ozone-objectstore-service-0.4.0-SNAPSHOT-plugin.jar': No such file or directory
      
      Current directory /testptch/hadoop/hadoop-ozone/dist/target
      
      $ rm -rf ozone-0.4.0-SNAPSHOT
      $ mkdir ozone-0.4.0-SNAPSHOT
      $ cd ozone-0.4.0-SNAPSHOT
      $ cp -p /testptch/hadoop/LICENSE.txt .
      $ cp -p /testptch/hadoop/NOTICE.txt .
      $ cp -p /testptch/hadoop/README.txt .
      $ mkdir -p ./share/hadoop/mapreduce
      $ mkdir -p ./share/hadoop/ozone
      $ mkdir -p ./share/hadoop/hdds
      $ mkdir -p ./share/hadoop/yarn
      $ mkdir -p ./share/hadoop/hdfs
      $ mkdir -p ./share/hadoop/common
      $ mkdir -p ./share/ozone/web
      $ mkdir -p ./bin
      $ mkdir -p ./sbin
      $ mkdir -p ./etc
      $ mkdir -p ./libexec
      $ cp -r /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/conf etc/hadoop
      $ cp /testptch/hadoop/hadoop-ozone/dist/src/main/conf/om-audit-log4j2.properties etc/hadoop
      $ cp /testptch/hadoop/hadoop-ozone/dist/src/main/conf/dn-audit-log4j2.properties etc/hadoop
      $ cp /testptch/hadoop/hadoop-ozone/dist/src/main/conf/scm-audit-log4j2.properties etc/hadoop
      $ cp /testptch/hadoop/hadoop-ozone/dist/src/main/conf/ozone-site.xml etc/hadoop
      $ cp -f /testptch/hadoop/hadoop-ozone/dist/src/main/conf/log4j.properties etc/hadoop
      $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop bin/
      $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd bin/
      $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/ozone bin/
      $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh libexec/
      $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.cmd libexec/
      $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh libexec/
      $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/ozone-config.sh libexec/
      $ cp -r /testptch/hadoop/hadoop-ozone/common/src/main/shellprofile.d libexec/
      $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh sbin/
      $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/workers.sh sbin/
      $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/start-ozone.sh sbin/
      $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/stop-ozone.sh sbin/
      $ mkdir -p ./share/hadoop/ozoneplugin
      $ cp /testptch/hadoop/hadoop-ozone/objectstore-service/target/hadoop-ozone-objectstore-service-0.4.0-SNAPSHOT-plugin.jar ./share/hadoop/ozoneplugin/hadoop-ozone-datanode-plugin-0.4.0-SNAPSHOT.jar
      
      Failed!
      
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO] 
      [INFO] Apache Hadoop Ozone ................................ SUCCESS [  5.471 s]
      [INFO] Apache Hadoop Ozone Common ......................... SUCCESS [ 13.453 s]
      [INFO] Apache Hadoop Ozone Client ......................... SUCCESS [  6.271 s]
      [INFO] Apache Hadoop Ozone Manager Server ................. SUCCESS [  8.081 s]
      [INFO] Apache Hadoop Ozone Object Store REST Service ...... SUCCESS [  5.535 s]
      [INFO] Apache Hadoop Ozone S3 Gateway ..................... SUCCESS [  7.966 s]
      [INFO] Apache Hadoop Ozone Integration Tests .............. SUCCESS [  7.708 s]
      [INFO] Apache Hadoop Ozone FileSystem ..................... SUCCESS [  7.121 s]
      [INFO] Apache Hadoop Ozone FileSystem Single Jar Library .. SUCCESS [  3.041 s]
      [INFO] Apache Hadoop Ozone FileSystem Legacy Jar Library .. SUCCESS [  3.217 s]
      [INFO] Apache Hadoop Ozone Tools .......................... SUCCESS [  7.720 s]
      [INFO] Apache Hadoop Ozone Datanode ....................... SUCCESS [  1.912 s]
      [INFO] Apache Hadoop Ozone Distribution ................... FAILURE [  2.260 s]
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 01:21 min
      [INFO] Finished at: 2019-03-05T10:22:14+00:00
      [INFO] Final Memory: 71M/1852M
      [INFO] ------------------------------------------------------------------------
      [WARNING] The requested profile "hdds" could not be activated because it does not exist.
      [WARNING] The requested profile "native" could not be activated because it does not exist.
      [WARNING] The requested profile "yarn-ui" could not be activated because it does not exist.
      [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (dist) on project hadoop-ozone-dist: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]
      [ERROR] 
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR] 
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
      [ERROR] 
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR]   mvn <goals> -rf :hadoop-ozone-dist
      

      One fix could be to move the dist-layout stitching to a later phase (but before the tar).

        Attachments

        1. HDDS-1218.002.patch
          0.4 kB
          Marton Elek

          Activity

            People

            • Assignee:
              elek Marton Elek
              Reporter:
              elek Marton Elek
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: