Pig
  1. Pig
  2. PIG-2349

Ant build repeats ivy-buildJar several times

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.11
    • Fix Version/s: 0.11
    • Component/s: build
    • Labels:
      None

      Description

      The ivy-buildjar step is repeated 3 times in one build.
      I think we can use an <uptodate> task to avoid checking the libs every time, speeding up the build and reducing screen spam.

      Thoughts?

      1. PIG-2349.3.patch
        6 kB
        Gianmarco De Francisci Morales
      2. PIG-2349.2.patch
        5 kB
        Gianmarco De Francisci Morales
      3. PIG-2349.1.patch
        5 kB
        Gianmarco De Francisci Morales

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        Anything reducing build time is a welcome change.

        Show
        Ashutosh Chauhan added a comment - Anything reducing build time is a welcome change.
        Hide
        Gianmarco De Francisci Morales added a comment -

        Attaching PIG-2349.1.patch

        First attempt to reduce build time.
        Now Ivy:resolve gets called only once, for all configurations.
        This is slower the first time, but much faster afterwards.
        It cuts my common-case build time by half.

        Comments?

        Show
        Gianmarco De Francisci Morales added a comment - Attaching PIG-2349 .1.patch First attempt to reduce build time. Now Ivy:resolve gets called only once, for all configurations. This is slower the first time, but much faster afterwards. It cuts my common-case build time by half. Comments?
        Hide
        Daniel Dai added a comment -

        +1

        Show
        Daniel Dai added a comment - +1
        Hide
        Gianmarco De Francisci Morales added a comment -

        Rebased, finalized and committed PIG-2349.2.patch to trunk.

        Show
        Gianmarco De Francisci Morales added a comment - Rebased, finalized and committed PIG-2349 .2.patch to trunk.
        Hide
        Daniel Dai added a comment -

        Seems this patch break 23 tests. I guess the dependency of ivy cache in 23 tests is the cause. To reproduce it, run any MiniCluster test in 23, such as:
        ant -Dtestcase=TestKeyTypeDiscoveryVisitor -Dhadoopversion=23 test

        I have to temporarily revert this patch until we fix the issue.

        Show
        Daniel Dai added a comment - Seems this patch break 23 tests. I guess the dependency of ivy cache in 23 tests is the cause. To reproduce it, run any MiniCluster test in 23, such as: ant -Dtestcase=TestKeyTypeDiscoveryVisitor -Dhadoopversion=23 test I have to temporarily revert this patch until we fix the issue.
        Hide
        Daniel Dai added a comment -

        To be more specific, Pig puts ivy cache into a text file mrapp-generated-classpath, MiniCluster will search for this file and load dependent jars. I guess this process is broken by the patch.

        Show
        Daniel Dai added a comment - To be more specific, Pig puts ivy cache into a text file mrapp-generated-classpath, MiniCluster will search for this file and load dependent jars. I guess this process is broken by the patch.
        Hide
        Gianmarco De Francisci Morales added a comment -

        Thanks for reporting the problem Daniel.
        I will fix it and test it with hadoop23.

        Show
        Gianmarco De Francisci Morales added a comment - Thanks for reporting the problem Daniel. I will fix it and test it with hadoop23.
        Hide
        Gianmarco De Francisci Morales added a comment -

        PIG-2349.3.patch should fix the issue with hadoop23.
        For the records, the path generated after the buggy patch was a superset of the original one, so the fault was caused by some jar that should not have been in the classpath.
        I fixed it restricting the classpath to the test ivy configuration.

        test-commit passes also for hadoop23.
        I have a failure for TestBuiltin on hadoop23 but it seems unrelated as it fails also in trunk.

        Show
        Gianmarco De Francisci Morales added a comment - PIG-2349 .3.patch should fix the issue with hadoop23. For the records, the path generated after the buggy patch was a superset of the original one, so the fault was caused by some jar that should not have been in the classpath. I fixed it restricting the classpath to the test ivy configuration. test-commit passes also for hadoop23. I have a failure for TestBuiltin on hadoop23 but it seems unrelated as it fails also in trunk.
        Hide
        Gianmarco De Francisci Morales added a comment -

        Forgot to grant license to ASF.

        Show
        Gianmarco De Francisci Morales added a comment - Forgot to grant license to ASF.
        Hide
        Dmitriy V. Ryaboy added a comment -

        TestBuiltin passes in trunk. Do you mean that it fails with hadoop23? Is that related to PIG-2468 ?

        Show
        Dmitriy V. Ryaboy added a comment - TestBuiltin passes in trunk. Do you mean that it fails with hadoop23? Is that related to PIG-2468 ?
        Hide
        Gianmarco De Francisci Morales added a comment -

        Yes, I meant it fails with hadoop23.
        I reverted to before PIG-2468 and TestBuiltin passes, so it is most likely caused by it.

        Show
        Gianmarco De Francisci Morales added a comment - Yes, I meant it fails with hadoop23. I reverted to before PIG-2468 and TestBuiltin passes, so it is most likely caused by it.
        Hide
        Daniel Dai added a comment -

        +1. Both hadoop23/hadoop20 runs fine for me.

        Show
        Daniel Dai added a comment - +1. Both hadoop23/hadoop20 runs fine for me.
        Hide
        Gianmarco De Francisci Morales added a comment -

        Patch committed to trunk.
        Thanks!

        Show
        Gianmarco De Francisci Morales added a comment - Patch committed to trunk. Thanks!

          People

          • Assignee:
            Gianmarco De Francisci Morales
            Reporter:
            Gianmarco De Francisci Morales
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development