HCatalog
  1. HCatalog
  2. HCATALOG-7

Ivy fetch of CDH Hadoop versions does not work properly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1
    • Component/s: None
    • Labels:
      None

      Description

      When checking out hcatalog for the first time on a system and doing a load, the build fails with the error:

      [ivy:retrieve] downloading http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hadoop-0.20.3-CDH3-SNAPSHOT/hadoop-0.20.3-CDH3-SNAPSHOT.tar.gz ...
      [ivy:retrieve] [SUCCESSFUL ] hadoop#core;0.20.3-CDH3-SNAPSHOT!hadoop.tar.gz(source) (160129ms)
      [ivy:retrieve] :: resolution report :: resolve 6848ms :: artifacts dl 220906ms
      ---------------------------------------------------------------------

        modules artifacts
      conf number search dwnlded evicted number dwnlded

      ---------------------------------------------------------------------

      default 2 1 0 0 2 1

      ---------------------------------------------------------------------
      [ivy:retrieve]
      [ivy:retrieve] :: problems summary ::
      [ivy:retrieve] :::: WARNINGS
      [ivy:retrieve] [FAILED ] hadoop#core;0.20.1!hadoop.tar.gz(source): Not in GZIP format (60771ms)
      [ivy:retrieve] [FAILED ] hadoop#core;0.20.1!hadoop.tar.gz(source): (0ms)
      [ivy:retrieve] ==== hadoop-source: tried
      [ivy:retrieve] http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz
      [ivy:retrieve] ==== apache-snapshot: tried
      [ivy:retrieve] https://repository.apache.org/content/repositories/snapshots/hadoop/core/0.20.1/hadoop-0.20.1.tar.gz
      [ivy:retrieve] ==== maven2: tried
      [ivy:retrieve] http://repo1.maven.org/maven2/hadoop/core/0.20.1/core-0.20.1.tar.gz
      [ivy:retrieve] ==== datanucleus-repo: tried
      [ivy:retrieve] http://www.datanucleus.org/downloads/maven2/hadoop/core/0.20.1/core-0.20.1.tar.gz
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve] :: FAILED DOWNLOADS ::
      [ivy:retrieve] :: ^ see resolution messages for details ^ ::
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve] :: hadoop#core;0.20.1!hadoop.tar.gz(source)
      [ivy:retrieve] ::::::::::::::::::::::::::::::::::::::::::::::
      [ivy:retrieve]
      [ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

      The download appears to be fetching the wrong file or corrupting the file in download. Once the proper version of the file is fetched and put in the ivy cache all works just fine.

      1. HCAT-7.patch
        0.6 kB
        Alan Gates

        Activity

        Hide
        Alan Gates added a comment -

        Hmm, there doesn't seem to be a "submit patch" option in our workflow. But this patch is ready for review. Jakob and Ashutosh, could you check if this resolves the issues for your systems? If you already have the corrupted file in your ant cache you'll have to clean it (~/.ant/cache).

        Show
        Alan Gates added a comment - Hmm, there doesn't seem to be a "submit patch" option in our workflow. But this patch is ready for review. Jakob and Ashutosh, could you check if this resolves the issues for your systems? If you already have the corrupted file in your ant cache you'll have to clean it (~/.ant/cache).
        Hide
        Jakob Homan added a comment - - edited

        I tried running ant without the patch and this time had problems with:

        [FAILED ] hadoop#core;0.20.1!hadoop.tar.gz(source): Not in GZIP format (13800ms)

        After applying the patch, I was able to build, clean, build docs, delete ivy cache, repeat above, etc. all without problems. +1. Also, props on adding the svn external.

        Show
        Jakob Homan added a comment - - edited I tried running ant without the patch and this time had problems with: [FAILED ] hadoop#core;0.20.1!hadoop.tar.gz(source): Not in GZIP format (13800ms) After applying the patch, I was able to build, clean, build docs, delete ivy cache, repeat above, etc. all without problems. +1. Also, props on adding the svn external.
        Hide
        Jakob Homan added a comment -

        I should mention this is on OSX 10.6 with ant 1.8.

        Show
        Jakob Homan added a comment - I should mention this is on OSX 10.6 with ant 1.8.
        Hide
        Ashutosh Chauhan added a comment -

        +1 After applying the patch, problem went away.

        Show
        Ashutosh Chauhan added a comment - +1 After applying the patch, problem went away.
        Hide
        Alan Gates added a comment -

        Patch checked in. Thanks Jakob for a taking a look.

        Show
        Alan Gates added a comment - Patch checked in. Thanks Jakob for a taking a look.
        Hide
        Mark Butler added a comment -

        I do not believe the patch solved the problem, I am still seeing it for HCatalog-0.2.

        I think this is HIVE-2580 so the version of Ivy is the real culprit see -
        https://issues.apache.org/jira/browse/HIVE-2580

        Here is what I think is happening: Hive-0.8.0 uses Ivy 2.1 but HCatalog uses Ivy 2.2.

        Hive 0.8.0 only retrieves the hadoop sources if the files hive-0.8.0/build/hadoopcore/hadoop-0.20.1.installed and hive-0.8.0/build/hadoopcore/hadoop-0.20.3-CDH3-SNAPSHOT.installed are not present.

        So if you have built Hive by itself (with Ivy 2.1), it will all be ok. But if you build with HCatalog (with Ivy 2.2) it may fail, unless Hive has already been built successfully, in which case it will skip the step retrieving the Hadoop sources because Hive detects the .installed files.

        Show
        Mark Butler added a comment - I do not believe the patch solved the problem, I am still seeing it for HCatalog-0.2. I think this is HIVE-2580 so the version of Ivy is the real culprit see - https://issues.apache.org/jira/browse/HIVE-2580 Here is what I think is happening: Hive-0.8.0 uses Ivy 2.1 but HCatalog uses Ivy 2.2. Hive 0.8.0 only retrieves the hadoop sources if the files hive-0.8.0/build/hadoopcore/hadoop-0.20.1.installed and hive-0.8.0/build/hadoopcore/hadoop-0.20.3-CDH3-SNAPSHOT.installed are not present. So if you have built Hive by itself (with Ivy 2.1), it will all be ok. But if you build with HCatalog (with Ivy 2.2) it may fail, unless Hive has already been built successfully, in which case it will skip the step retrieving the Hadoop sources because Hive detects the .installed files.
        Hide
        Travis Crawford added a comment -

        Hey Mark - see HCATALOG-186 where we downgraded HCatalog to Ivy version 2.1.

        You may need to blow away your local ivy cache though if builds are still failing, so the tarballs are downloaded correctly.

        Show
        Travis Crawford added a comment - Hey Mark - see HCATALOG-186 where we downgraded HCatalog to Ivy version 2.1. You may need to blow away your local ivy cache though if builds are still failing, so the tarballs are downloaded correctly.

          People

          • Assignee:
            Alan Gates
            Reporter:
            Alan Gates
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development