Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 1.0.0
    • Component/s: general
    • Labels:
      None

      Description

      We may consider to include zookeeper 3.4.6 in bigtop 0.9:

      1) 12 blocker issues were resolved in zookeeper 3.4.6:

      project = ZOOKEEPER AND issuetype = Bug AND status in (Resolved, Closed) AND priority = Blocker AND fixVersion = 3.4.6 ORDER BY summary DESC

      2) 16 critical issues were resolved in zookeeper 3.4.6:

      project = ZOOKEEPER AND issuetype = Bug AND status in (Resolved, Closed) AND priority = Critical AND fixVersion = 3.4.6 ORDER BY summary DESC

      1. BIGTOP-1406.1.patch
        32 kB
        Olaf Flebbe
      2. BIGTOP-1468.2.diff
        26 kB
        Olaf Flebbe

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        If I am not mistaken 3.4.6 was pretty screwed up - if I am not mistaken, we'd better skip to until the consequent release.

        Show
        cos Konstantin Boudnik added a comment - If I am not mistaken 3.4.6 was pretty screwed up - if I am not mistaken, we'd better skip to until the consequent release.
        Hide
        rvs Roman Shaposhnik added a comment -

        Let me ping a few ZK PMC members to find out what the good vintage is.

        Show
        rvs Roman Shaposhnik added a comment - Let me ping a few ZK PMC members to find out what the good vintage is.
        Hide
        phunt Patrick Hunt added a comment -

        I'm not aware of it being "screwed up" (not sure what that means actually) but Flavio Junqueira was the release manager and could provide more insight than I could.

        Show
        phunt Patrick Hunt added a comment - I'm not aware of it being "screwed up" (not sure what that means actually) but Flavio Junqueira was the release manager and could provide more insight than I could.
        Hide
        cos Konstantin Boudnik added a comment -

        Sorry Patrick Hunt: perhaps 'screwed-up' is a too strong of an expression. But I remember that one of the downstreams was considering moving to 3.4.6 and then deciding against it. I want to say HBase, but am not sure now. I guess I should've add more info into my comment. I will try to recall my line of thoughts.

        Show
        cos Konstantin Boudnik added a comment - Sorry Patrick Hunt : perhaps 'screwed-up' is a too strong of an expression. But I remember that one of the downstreams was considering moving to 3.4.6 and then deciding against it. I want to say HBase, but am not sure now. I guess I should've add more info into my comment. I will try to recall my line of thoughts.
        Hide
        phunt Patrick Hunt added a comment -

        Screwed up might be accurate, I just don't know what it's referring to so it's hard to speculate.

        Show
        phunt Patrick Hunt added a comment - Screwed up might be accurate, I just don't know what it's referring to so it's hard to speculate.
        Hide
        cos Konstantin Boudnik added a comment -

        Are we doing this or shall I move it to 1.1.0?

        Show
        cos Konstantin Boudnik added a comment - Are we doing this or shall I move it to 1.1.0?
        Hide
        gkesavan Giridharan Kesavan added a comment -

        I can help with upgrading zookeeper to 3.4.6 in bigtop. As a matter of fact HDP stack released zk-3.4.6 a while back.

        Show
        gkesavan Giridharan Kesavan added a comment - I can help with upgrading zookeeper to 3.4.6 in bigtop. As a matter of fact HDP stack released zk-3.4.6 a while back.
        Hide
        cos Konstantin Boudnik added a comment -

        That'd be great! Please do, thank you!

        Show
        cos Konstantin Boudnik added a comment - That'd be great! Please do, thank you!
        Hide
        cos Konstantin Boudnik added a comment -

        Any progress here? Shall this ticket be assigned to Giridharan Kesavan?

        Show
        cos Konstantin Boudnik added a comment - Any progress here? Shall this ticket be assigned to Giridharan Kesavan ?
        Hide
        oflebbe Olaf Flebbe added a comment -

        I take this for 1.0.0

        I did a look into the HDP changes relative to 3.4.6

        There relevant changes for BIGTOP (excluding Windows specific fixes) seems:
        ZOOKEEPER-1897
        ZOOKEEPER-2064
        ZOOKEEPER-1506
        ZOOKEEPER-1926 (not important since AFAIK we do not use unit tests in BIGTOP)

        Show
        oflebbe Olaf Flebbe added a comment - I take this for 1.0.0 I did a look into the HDP changes relative to 3.4.6 There relevant changes for BIGTOP (excluding Windows specific fixes) seems: ZOOKEEPER-1897 ZOOKEEPER-2064 ZOOKEEPER-1506 ZOOKEEPER-1926 (not important since AFAIK we do not use unit tests in BIGTOP)
        Hide
        oflebbe Olaf Flebbe added a comment -

        Work In Progress

        Show
        oflebbe Olaf Flebbe added a comment - Work In Progress
        Hide
        cos Konstantin Boudnik added a comment -

        Do we have to patch ZK release?

        Show
        cos Konstantin Boudnik added a comment - Do we have to patch ZK release?
        Hide
        oflebbe Olaf Flebbe added a comment -

        Yes, since zookeeper 3.4.6 is rather old a couple of issues surfaced:

        Ressource Leaks, recovering from DNS changes, A buggy commandline handling. Will leave out the Unit test fixes for now.
        All three are rather small patches, and are officially merged into upcoming release. Only ZOOKEEPER-1506 (DNS failover) is a little bit longer and is not officially applied on zookeeper-3.4 branch, since it has to be backported. I use the Hortonworks patch from the HDP release, giving them the credits .

        We either have to patch these issues or use an unreleased version.

        Besides, our 3.4.5 contains a patch ZOOKEEPER-1911 for packaging (will reuse it). I have stability issues (spurious connection breakdown) with this release on production loads, which seem to be fixed by 3.4.6 . If you look into Clouders ZOOKEEPER Release you see tons of patches which have already more or less gone into 3.4.6.

        Show
        oflebbe Olaf Flebbe added a comment - Yes, since zookeeper 3.4.6 is rather old a couple of issues surfaced: Ressource Leaks, recovering from DNS changes, A buggy commandline handling. Will leave out the Unit test fixes for now. All three are rather small patches, and are officially merged into upcoming release. Only ZOOKEEPER-1506 (DNS failover) is a little bit longer and is not officially applied on zookeeper-3.4 branch, since it has to be backported. I use the Hortonworks patch from the HDP release, giving them the credits . We either have to patch these issues or use an unreleased version. Besides, our 3.4.5 contains a patch ZOOKEEPER-1911 for packaging (will reuse it). I have stability issues (spurious connection breakdown) with this release on production loads, which seem to be fixed by 3.4.6 . If you look into Clouders ZOOKEEPER Release you see tons of patches which have already more or less gone into 3.4.6.
        Hide
        cos Konstantin Boudnik added a comment -

        Thanks for the explanation Olaf. My only concern here - and I have expressed it a multiple times elsewhere - is stepping on this way of patching official Apache releases and, essentially, getting into the business of supporting our own version of said components.

        If the community at large feel that's an ok solution - I won't stand in the way but I really think we need to make a conscious decision on this.

        Show
        cos Konstantin Boudnik added a comment - Thanks for the explanation Olaf. My only concern here - and I have expressed it a multiple times elsewhere - is stepping on this way of patching official Apache releases and, essentially, getting into the business of supporting our own version of said components. If the community at large feel that's an ok solution - I won't stand in the way but I really think we need to make a conscious decision on this.
        Hide
        oflebbe Olaf Flebbe added a comment -

        I like to be BIGTOP an unchanged vanilla apache releases sandbox for playing with technology, too. No intentions to get into support business.

        Your concern in first comment was that zookeeper-3.4.6 is too unstable to use.

        I think this concern is not well founded, since HDP is relying on 3.4.6 and is only patching 3 unix specific runtime issues, where 2 of them are really easy to understand and already commited upstream. ZOOKEEPER-1506 stands out, since it did not found its way into official upstream repository and seems only necessary for enterprise grade infrastructure.

        In the light of your concerns going into support business, I will leave ZOOKEEPER-1506 out.

        Show
        oflebbe Olaf Flebbe added a comment - I like to be BIGTOP an unchanged vanilla apache releases sandbox for playing with technology, too. No intentions to get into support business. Your concern in first comment was that zookeeper-3.4.6 is too unstable to use. I think this concern is not well founded, since HDP is relying on 3.4.6 and is only patching 3 unix specific runtime issues, where 2 of them are really easy to understand and already commited upstream. ZOOKEEPER-1506 stands out, since it did not found its way into official upstream repository and seems only necessary for enterprise grade infrastructure. In the light of your concerns going into support business, I will leave ZOOKEEPER-1506 out.
        Hide
        oflebbe Olaf Flebbe added a comment -

        I forgot to mention ZOOKEEPER-2064 , already on 3.4 branch, too.

        Konstantin Boudnik I would be perfectly happy to use zookeeper-3.4.7, if it would be released. Maybe you can ask the project lead on the situation?

        Show
        oflebbe Olaf Flebbe added a comment - I forgot to mention ZOOKEEPER-2064 , already on 3.4 branch, too. Konstantin Boudnik I would be perfectly happy to use zookeeper-3.4.7, if it would be released. Maybe you can ask the project lead on the situation?
        Hide
        oflebbe Olaf Flebbe added a comment -

        Updated patch and tested.

        Show
        oflebbe Olaf Flebbe added a comment - Updated patch and tested.
        Hide
        cos Konstantin Boudnik added a comment -

        As you have seen, ZK guys replied about their release plans. How about this: let's not use the patching as a general rule, but considering how many components in the stack are using ZK let's patch it. Once 3.4.7 is out - we'll update the component build. Do you agree? If so - +1 and let's commit it. Also, could you please open a ticket for 3.4.7 transition?

        Thanks!

        Show
        cos Konstantin Boudnik added a comment - As you have seen, ZK guys replied about their release plans. How about this: let's not use the patching as a general rule, but considering how many components in the stack are using ZK let's patch it. Once 3.4.7 is out - we'll update the component build. Do you agree? If so - +1 and let's commit it. Also, could you please open a ticket for 3.4.7 transition? Thanks!
        Hide
        oflebbe Olaf Flebbe added a comment -

        Yep, I agree... patch committed.

        Show
        oflebbe Olaf Flebbe added a comment - Yep, I agree... patch committed.
        Hide
        evans_ye Evans Ye added a comment -

        Hi Olaf Flebbe I got patch file missing when building zookeeper-rpm during my test run CI setup
        The debian build is ok though.

        Patch #1 (patch1-ZOOKEEPER-1911.diff):
        + /usr/bin/cat /ws/build/zookeeper/rpm/SOURCES/patch1-ZOOKEEPER-1911.diff
        + /usr/bin/patch -p1 --fuzz=0
        patching file src/contrib/rest/build.xml
        + echo 'Patch #2 (patch2-ZOOKEEPER-1926.diff):'
        Patch #2 (patch2-ZOOKEEPER-1926.diff):
        + /usr/bin/cat /ws/build/zookeeper/rpm/SOURCES/patch2-ZOOKEEPER-1926.diff
        + /usr/bin/patch -p1 --fuzz=0
        patching file src/java/systest/org/apache/zookeeper/test/system/BaseSysTest.java
        patching file src/java/systest/org/apache/zookeeper/test/system/QuorumPeerInstance.java
        patching file src/java/test/org/apache/zookeeper/server/quorum/LearnerTest.java
        patching file src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java
        + echo 'Patch #3 (patch3-ZOOKEEPER-2064.diff):'
        Patch #3 (patch3-ZOOKEEPER-2064.diff):
        + /usr/bin/cat /ws/build/zookeeper/rpm/SOURCES/patch3-ZOOKEEPER-2064.diff
        + /usr/bin/patch -p1 --fuzz=0
        patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/Log4JSource.java
        patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/RandomAccessFileReader.java
        patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/servlets/NumEvents.java
        patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/servlets/StaticContent.java
        patching file src/contrib/rest/src/java/org/apache/zookeeper/server/jersey/cfg/RestCfg.java
        patching file src/java/systest/org/apache/zookeeper/test/system/GenerateLoad.java
        patching file src/java/systest/org/apache/zookeeper/test/system/QuorumPeerInstance.java
        + exit 0
        Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.hCtSLT
        + umask 022
        + cd /ws/build/zookeeper/rpm//BUILD
        + cd zookeeper-3.4.6
        + cp /ws/build/zookeeper/rpm//SOURCES/ZOOKEEPER-1911.patch ./
        cp: cannot stat '/ws/build/zookeeper/rpm//SOURCES/ZOOKEEPER-1911.patch': No such file or directory
        error: Bad exit status from /var/tmp/rpm-tmp.hCtSLT (%build)
        
            Bad exit status from /var/tmp/rpm-tmp.hCtSLT (%build)
        
        RPM build errors:
        :zookeeper-rpm FAILED
        
        FAILURE: Build failed with an exception.
        
        * Where:
        Script '/ws/packages.gradle' line: 423
        
        * What went wrong:
        Execution failed for task ':zookeeper-rpm'.
        > Process 'command 'rpmbuild'' finished with non-zero exit value 1
        
        * Try:
        Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
        
        BUILD FAILED
        
        Total time: 24.567 secs
        Build step 'Execute shell' marked build as failure
        Finished: FAILURE
        

        Is this intended since zookeeper is in sort of transition to 3.4.7?

        Show
        evans_ye Evans Ye added a comment - Hi Olaf Flebbe I got patch file missing when building zookeeper-rpm during my test run CI setup The debian build is ok though. Patch #1 (patch1-ZOOKEEPER-1911.diff): + /usr/bin/cat /ws/build/zookeeper/rpm/SOURCES/patch1-ZOOKEEPER-1911.diff + /usr/bin/patch -p1 --fuzz=0 patching file src/contrib/ rest /build.xml + echo 'Patch #2 (patch2-ZOOKEEPER-1926.diff):' Patch #2 (patch2-ZOOKEEPER-1926.diff): + /usr/bin/cat /ws/build/zookeeper/rpm/SOURCES/patch2-ZOOKEEPER-1926.diff + /usr/bin/patch -p1 --fuzz=0 patching file src/java/systest/org/apache/zookeeper/test/system/BaseSysTest.java patching file src/java/systest/org/apache/zookeeper/test/system/QuorumPeerInstance.java patching file src/java/test/org/apache/zookeeper/server/quorum/LearnerTest.java patching file src/java/test/org/apache/zookeeper/server/quorum/Zab1_0Test.java + echo 'Patch #3 (patch3-ZOOKEEPER-2064.diff):' Patch #3 (patch3-ZOOKEEPER-2064.diff): + /usr/bin/cat /ws/build/zookeeper/rpm/SOURCES/patch3-ZOOKEEPER-2064.diff + /usr/bin/patch -p1 --fuzz=0 patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/Log4JSource.java patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/RandomAccessFileReader.java patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/servlets/NumEvents.java patching file src/contrib/loggraph/src/java/org/apache/zookeeper/graph/servlets/StaticContent.java patching file src/contrib/ rest /src/java/org/apache/zookeeper/server/jersey/cfg/RestCfg.java patching file src/java/systest/org/apache/zookeeper/test/system/GenerateLoad.java patching file src/java/systest/org/apache/zookeeper/test/system/QuorumPeerInstance.java + exit 0 Executing(%build): /bin/sh -e / var /tmp/rpm-tmp.hCtSLT + umask 022 + cd /ws/build/zookeeper/rpm //BUILD + cd zookeeper-3.4.6 + cp /ws/build/zookeeper/rpm //SOURCES/ZOOKEEPER-1911.patch ./ cp: cannot stat '/ws/build/zookeeper/rpm //SOURCES/ZOOKEEPER-1911.patch': No such file or directory error: Bad exit status from / var /tmp/rpm-tmp.hCtSLT (%build) Bad exit status from / var /tmp/rpm-tmp.hCtSLT (%build) RPM build errors: :zookeeper-rpm FAILED FAILURE: Build failed with an exception. * Where: Script '/ws/packages.gradle' line: 423 * What went wrong: Execution failed for task ':zookeeper-rpm'. > Process 'command 'rpmbuild'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 24.567 secs Build step 'Execute shell' marked build as failure Finished: FAILURE Is this intended since zookeeper is in sort of transition to 3.4.7?
        Hide
        cos Konstantin Boudnik added a comment -

        Transition or not - build should work, IMO.

        Show
        cos Konstantin Boudnik added a comment - Transition or not - build should work, IMO.
        Hide
        warwithin YoungWoo Kim added a comment -

        I got a same error. Filed a JIRA for it, BIGTOP-1839.

        Show
        warwithin YoungWoo Kim added a comment - I got a same error. Filed a JIRA for it, BIGTOP-1839 .

          People

          • Assignee:
            oflebbe Olaf Flebbe
            Reporter:
            rguo Guo Ruijing
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development