Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Build Infrastructure
    • Labels:
    1. HIVE-5611.patch
      20 kB
      Szehon Ho
    2. HIVE-5611.3.patch
      14 kB
      Brock Noland
    3. HIVE-5611.2.patch
      14 kB
      Szehon Ho
    4. HIVE-5611.1.patch
      15 kB
      Szehon Ho

      Issue Links

        Activity

        Hide
        Brock Noland added a comment -

        FYI I added instructions on how to generate the tarball here:

        https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ

        Show
        Brock Noland added a comment - FYI I added instructions on how to generate the tarball here: https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ
        Hide
        Brock Noland added a comment -

        I opened two follow issues:

        HIVE-5740 - Tar files should extract to the directory of the same name minus tar.gz

        HIVE-5741 - Hcatalog needs to be added to the binary tar

        Show
        Brock Noland added a comment - I opened two follow issues: HIVE-5740 - Tar files should extract to the directory of the same name minus tar.gz HIVE-5741 - Hcatalog needs to be added to the binary tar
        Hide
        Brock Noland added a comment -

        On commit I made a very small change so the tars extract in directories.

        Show
        Brock Noland added a comment - On commit I made a very small change so the tars extract in directories.
        Hide
        Brock Noland added a comment -

        Thanks Szehon for the contribution! I have committed this to trunk.

        Show
        Brock Noland added a comment - Thanks Szehon for the contribution! I have committed this to trunk.
        Hide
        Brock Noland added a comment -

        Thank you! We'll have to do some followup work for hcatalog but this good for now.

        +1

        Show
        Brock Noland added a comment - Thank you! We'll have to do some followup work for hcatalog but this good for now. +1
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12611707/HIVE-5611.2.patch

        SUCCESS: +1 4547 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/121/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/121/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12611707

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12611707/HIVE-5611.2.patch SUCCESS: +1 4547 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/121/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/121/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12611707
        Hide
        Szehon Ho added a comment -

        Incorporated review feedback in HIVE-5611.2.patch

        Show
        Szehon Ho added a comment - Incorporated review feedback in HIVE-5611 .2.patch
        Hide
        Szehon Ho added a comment -

        Also I have created HIVE-5729 and HIVE-5730 , to track the beeline bundling issues found during manually testing the bin.

        Show
        Szehon Ho added a comment - Also I have created HIVE-5729 and HIVE-5730 , to track the beeline bundling issues found during manually testing the bin.
        Hide
        Szehon Ho added a comment -
        Show
        Szehon Ho added a comment - Done: https://reviews.apache.org/r/15187/
        Hide
        Brock Noland added a comment -

        Can you put up a RB item for this patch? https://reviews.apache.org

        Show
        Brock Noland added a comment - Can you put up a RB item for this patch? https://reviews.apache.org
        Hide
        Szehon Ho added a comment -

        Fixed up the mvn src assembly descriptor to copy the source directory structure.

        Did not end up using <moduleSet> because the directory structure those put in is different than the source tree (maven gives option to either prepends the project's module-name, or prepends nothing)

        Show
        Szehon Ho added a comment - Fixed up the mvn src assembly descriptor to copy the source directory structure. Did not end up using <moduleSet> because the directory structure those put in is different than the source tree (maven gives option to either prepends the project's module-name, or prepends nothing)
        Hide
        Brock Noland added a comment -

        Thejas M Nair

        The ant binary tarball had an hcatalog directory and all hcatalog stuff was located there. I believe part of the reason was that with the ant build the hive and hcatalog builds were somewhat separate. The two builds are very much integrated at this point, therefore I'd like to eliminate this hcatalog directory and include the stuff that was under bin/ under the hive bin/ etc.

        Do you see an issue with this or are you opposed to this?

        Show
        Brock Noland added a comment - Thejas M Nair The ant binary tarball had an hcatalog directory and all hcatalog stuff was located there. I believe part of the reason was that with the ant build the hive and hcatalog builds were somewhat separate. The two builds are very much integrated at this point, therefore I'd like to eliminate this hcatalog directory and include the stuff that was under bin/ under the hive bin/ etc. Do you see an issue with this or are you opposed to this?
        Hide
        Szehon Ho added a comment -

        Sure, I'll see if I can dumb down the src assemly descriptor to copy over the source tree, I guess those are non-issues then.

        Show
        Szehon Ho added a comment - Sure, I'll see if I can dumb down the src assemly descriptor to copy over the source tree, I guess those are non-issues then.
        Hide
        Brock Noland added a comment -

        Great! I will take a look at this today. A couple notes below.

        In src assembly, the prefixes for the .java files are different. This is because in ant, they copied over the project name.

        The ant based "src" tarball hive released are bad and should be completely ignored. The source assembly should exactly match a checkout of the source tree.

        The java-docs are not included in src assembly, as they are not generated.

        I don't think javadocs should be included in the src assembly anyway, as I said above it should match a checkout of the source tree exactly.

        Show
        Brock Noland added a comment - Great! I will take a look at this today. A couple notes below. In src assembly, the prefixes for the .java files are different. This is because in ant, they copied over the project name. The ant based "src" tarball hive released are bad and should be completely ignored. The source assembly should exactly match a checkout of the source tree. The java-docs are not included in src assembly, as they are not generated. I don't think javadocs should be included in the src assembly anyway, as I said above it should match a checkout of the source tree exactly.
        Hide
        Szehon Ho added a comment -

        I have a partial patch that supports basic Hive functionalities. This patch converts all the Ant tar/bin steps into Mvn assembly src/bin descriptors. All the non-jar files should now be in the Mvn tarball, with same directory location as in Ant tarballs, with directory location exactly the same. But some jars are not yet complete and need further investigation. In particular, the Hcatalog sub-project had many custom Ant steps in /hcatalog/build.xml in creating its unique directory structure, I think it will be a big effort to replicate the same in Mvn.

        Issues (non-blockers) that I encountered between Ant and Mvn assemblies:
        1. In src assembly, the prefixes for the .java files are different. This is because in ant, they copied over the project name. In mvn, I am using the <moduleSet>, and module names are slightly different than folder name. I don't believe this would be a big issue.
        2. The java-docs are not included in src assembly, as they are not generated. This is already tracked in HIVE-5717.

        Issues in running Hive so far (very basic testing), there might be more:
        3. The version number is not displaying correctly on starting Beeline, because MANIFEST is not included in jar built by Beeline project. It is not related to assembly creation.
        4. There is background NPE in starting SQLCompletor, because the resource file "sql-keyword.properties" has been moved to src/main/resources from org/apache/hive/beeline. Again it is not related to assembly creation.

        HCatalog (must fix)
        To add the hcatalog jars in the correct directory location would take a big effort, I might not be able to complete this task myself, I would appreciate if someone could help take a look!

        Show
        Szehon Ho added a comment - I have a partial patch that supports basic Hive functionalities. This patch converts all the Ant tar/bin steps into Mvn assembly src/bin descriptors. All the non-jar files should now be in the Mvn tarball, with same directory location as in Ant tarballs, with directory location exactly the same. But some jars are not yet complete and need further investigation. In particular, the Hcatalog sub-project had many custom Ant steps in /hcatalog/build.xml in creating its unique directory structure, I think it will be a big effort to replicate the same in Mvn. Issues (non-blockers) that I encountered between Ant and Mvn assemblies: 1. In src assembly, the prefixes for the .java files are different. This is because in ant, they copied over the project name. In mvn, I am using the <moduleSet>, and module names are slightly different than folder name. I don't believe this would be a big issue. 2. The java-docs are not included in src assembly, as they are not generated. This is already tracked in HIVE-5717 . Issues in running Hive so far (very basic testing), there might be more: 3. The version number is not displaying correctly on starting Beeline, because MANIFEST is not included in jar built by Beeline project. It is not related to assembly creation. 4. There is background NPE in starting SQLCompletor, because the resource file "sql-keyword.properties" has been moved to src/main/resources from org/apache/hive/beeline. Again it is not related to assembly creation. HCatalog (must fix) To add the hcatalog jars in the correct directory location would take a big effort, I might not be able to complete this task myself, I would appreciate if someone could help take a look!

          People

          • Assignee:
            Szehon Ho
            Reporter:
            Brock Noland
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development