Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1421

Phoenix build is missing some variables

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:

      Description

      After deploying Phoenix 4.0.1, HBase 0.98.5 and Hadoop 2.4.1 using BigTop build, phoenix is not working is is missing those variables.

      1) Phoenix-Core need to be in HBase classpath. I have added it manually in my hbase-env.sh file: export HBASE_CLASSPATH=/usr/lib/phoenix/lib/phoenix-core-4.0.1.jar

      2) Phoenix command lines need PHOENIX_LIB_DIR to be configured. I just run it before any command line: export PHOENIX_LIB_DIR=/usr/lib/phoenix/

      1. BIGTOP-1421.patch
        4 kB
        Andrew Purtell
      2. BIGTOP-1421.patch
        5 kB
        Andrew Purtell

        Issue Links

          Activity

          Hide
          apurtell Andrew Purtell added a comment -

          How about creating these envvars with a defaults file?

          I tested the debian build.

          Show
          apurtell Andrew Purtell added a comment - How about creating these envvars with a defaults file? I tested the debian build.
          Hide
          plinnell Peter Linnell added a comment -

          +1

          A defaults file sounds like an interesting idea to test.

          Show
          plinnell Peter Linnell added a comment - +1 A defaults file sounds like an interesting idea to test.
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Thanks for the patch. Just tried without it and had to add everything manually, will retry the .deb with this patch and BIGTOP-1420

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Thanks for the patch. Just tried without it and had to add everything manually, will retry the .deb with this patch and BIGTOP-1420
          Hide
          rvs Roman Shaposhnik added a comment -

          +1

          Question for Konstantin Boudnik should we be including this in 0.8.0?

          Show
          rvs Roman Shaposhnik added a comment - +1 Question for Konstantin Boudnik should we be including this in 0.8.0?
          Hide
          cos Konstantin Boudnik added a comment -

          If it breaking the stuff - sure.

          Show
          cos Konstantin Boudnik added a comment - If it breaking the stuff - sure.
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Project will still build without this patch, but Phoenix will not run correctly. So I recommend to push that into 0.8.

          I'm building 0.8+BIGTOP-1420+BIGTOP-1421 on Debian 7.6. Should be done in the next hour or 2. Will keep you posted shortly.

          I found another issue but probably not a blocker. Will open another JIRA for that. Need to do more tests first.

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Project will still build without this patch, but Phoenix will not run correctly. So I recommend to push that into 0.8. I'm building 0.8+ BIGTOP-1420 + BIGTOP-1421 on Debian 7.6. Should be done in the next hour or 2. Will keep you posted shortly. I found another issue but probably not a blocker. Will open another JIRA for that. Need to do more tests first.
          Hide
          cos Konstantin Boudnik added a comment -

          but Phoenix will not run correctly

          Yeah, which exactly constitutes "breaking the stuff" IMO :0

          Show
          cos Konstantin Boudnik added a comment - but Phoenix will not run correctly Yeah, which exactly constitutes "breaking the stuff" IMO :0
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          I tried to build on Debian with the 2 patchs with no success. Fine for ZK, Hadoop and HBase, but not for Phoenix:

          [ERROR] COMPILATION ERROR : 
          [INFO] -------------------------------------------------------------
          [ERROR] /home/jmspaggi/bigtop/output/phoenix/phoenix-4.1.0/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexQosCompat.java:[23,35] cannot find symbol
            symbol:   class RpcControllerFactory
            location: package org.apache.hadoop.hbase.ipc
          [ERROR] /home/jmspaggi/bigtop/output/phoenix/phoenix-4.1.0/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexQosRpcControllerFactory.java:[26,35] cannot find symbol
            symbol:   class DelegatingPayloadCarryingRpcController
            location: package org.apache.hadoop.hbase.ipc
          

          It went weel this morning on CentOS but without the patchs. I will try to figure why and will retry with CentOS and the 2 patches.

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - I tried to build on Debian with the 2 patchs with no success. Fine for ZK, Hadoop and HBase, but not for Phoenix: [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /home/jmspaggi/bigtop/output/phoenix/phoenix-4.1.0/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexQosCompat.java:[23,35] cannot find symbol symbol: class RpcControllerFactory location: package org.apache.hadoop.hbase.ipc [ERROR] /home/jmspaggi/bigtop/output/phoenix/phoenix-4.1.0/phoenix-core/src/main/java/org/apache/phoenix/hbase/index/IndexQosRpcControllerFactory.java:[26,35] cannot find symbol symbol: class DelegatingPayloadCarryingRpcController location: package org.apache.hadoop.hbase.ipc It went weel this morning on CentOS but without the patchs. I will try to figure why and will retry with CentOS and the 2 patches.
          Hide
          cos Konstantin Boudnik added a comment - - edited

          Is there a possibility that Phoenix doesn't compile against HBase 0.98.5?

          Show
          cos Konstantin Boudnik added a comment - - edited Is there a possibility that Phoenix doesn't compile against HBase 0.98.5?
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          We have been able to get Phoenix running fine with HBase 0.98.5 using BigTop 0.8 and small minor modifications (-D options, etc.).

          With 1420 and 1421 I'm not able to build Phoenix

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - We have been able to get Phoenix running fine with HBase 0.98.5 using BigTop 0.8 and small minor modifications (-D options, etc.). With 1420 and 1421 I'm not able to build Phoenix
          Hide
          apurtell Andrew Purtell added a comment -

          You need also 1432!

          Show
          apurtell Andrew Purtell added a comment - You need also 1432!
          Hide
          apurtell Andrew Purtell added a comment -

          Apply in order: 1432, 1420, 1421.

          1432 bumps HBase up to >= 0.98.4 (specifically, 0.98.5) which is required for 4.1.0 as explained on that issue.

          1420 then bumps Phoenix to 4.1.0

          1421 then attempts something useful with defaults files.

          Show
          apurtell Andrew Purtell added a comment - Apply in order: 1432, 1420, 1421. 1432 bumps HBase up to >= 0.98.4 (specifically, 0.98.5) which is required for 4.1.0 as explained on that issue. 1420 then bumps Phoenix to 4.1.0 1421 then attempts something useful with defaults files.
          Hide
          apurtell Andrew Purtell added a comment -

          Here's an updated patch that modifies the Phoenix package.

          • Adds a defaults file in common src
          • Updates RPM and DEB builds to pick up the defaults file
          • Updates install_phoenix.sh in common src to provide a new version independent symlink for phoenix-core, which is what needs to go onto the regionserver classpath
          Show
          apurtell Andrew Purtell added a comment - Here's an updated patch that modifies the Phoenix package. Adds a defaults file in common src Updates RPM and DEB builds to pick up the defaults file Updates install_phoenix.sh in common src to provide a new version independent symlink for phoenix-core, which is what needs to go onto the regionserver classpath
          Hide
          apurtell Andrew Purtell added a comment -

          Tested the DEB build. Passes. Defaults file ends up in the right location. New version independent symlink looks good:

          $ ls -l /usr/lib/phoenix/
          total 34328
          drwxr-xr-x 2 apurtell apurtell     4096 Sep  7 19:38 bin
          drwxr-xr-x 2 apurtell apurtell     4096 Sep  7 19:44 lib
          -rw-r--r-- 1 apurtell apurtell 32523055 Sep  7 19:38 phoenix-4.1.0-client.jar
          lrwxrwxrwx 1 apurtell apurtell       24 Sep  7 19:38 phoenix-client.jar -> phoenix-4.1.0-client.jar
          -rw-r--r-- 1 apurtell apurtell  2617104 Sep  7 19:38 phoenix-core-4.1.0.jar
          lrwxrwxrwx 1 apurtell apurtell       22 Sep  7 19:38 phoenix-core.jar -> phoenix-core-4.1.0.jar
          
          Show
          apurtell Andrew Purtell added a comment - Tested the DEB build. Passes. Defaults file ends up in the right location. New version independent symlink looks good: $ ls -l /usr/lib/phoenix/ total 34328 drwxr-xr-x 2 apurtell apurtell 4096 Sep 7 19:38 bin drwxr-xr-x 2 apurtell apurtell 4096 Sep 7 19:44 lib -rw-r--r-- 1 apurtell apurtell 32523055 Sep 7 19:38 phoenix-4.1.0-client.jar lrwxrwxrwx 1 apurtell apurtell 24 Sep 7 19:38 phoenix-client.jar -> phoenix-4.1.0-client.jar -rw-r--r-- 1 apurtell apurtell 2617104 Sep 7 19:38 phoenix-core-4.1.0.jar lrwxrwxrwx 1 apurtell apurtell 22 Sep 7 19:38 phoenix-core.jar -> phoenix-core-4.1.0.jar
          Hide
          cos Konstantin Boudnik added a comment -

          +1.
          Please push to the master - I will merge the release branch later.

          Show
          cos Konstantin Boudnik added a comment - +1. Please push to the master - I will merge the release branch later.
          Hide
          apurtell Andrew Purtell added a comment -

          Pushed to master

          Show
          apurtell Andrew Purtell added a comment - Pushed to master
          Hide
          cos Konstantin Boudnik added a comment -

          Fixed in the master and merged to the branch-0.8

          Show
          cos Konstantin Boudnik added a comment - Fixed in the master and merged to the branch-0.8
          Hide
          cos Konstantin Boudnik added a comment -

          Can I ask committers to resolve the tickets and mark them against correct versions and components, once they are committed? Thanks!

          Show
          cos Konstantin Boudnik added a comment - Can I ask committers to resolve the tickets and mark them against correct versions and components, once they are committed? Thanks!
          Hide
          apurtell Andrew Purtell added a comment -

          I left it alone because you specifically stated you would merge to branch

          Show
          apurtell Andrew Purtell added a comment - I left it alone because you specifically stated you would merge to branch
          Hide
          cos Konstantin Boudnik added a comment -

          Ah, got it - thanks!

          Show
          cos Konstantin Boudnik added a comment - Ah, got it - thanks!
          Hide
          cos Konstantin Boudnik added a comment -

          The fix evidently broke all RPM bulds as seen from
          http://bigtop01.cloudera.org:8080/job/Bigtop-0.8.0/5/label=fedora18/console

          can it be fixed ASAP?

          Show
          cos Konstantin Boudnik added a comment - The fix evidently broke all RPM bulds as seen from http://bigtop01.cloudera.org:8080/job/Bigtop-0.8.0/5/label=fedora18/console can it be fixed ASAP?
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Confirmed. Not able to build with the branch. Strange because I was ableto build with the previous branch + those 3 patchs.
          Seems that it misses /mnt/jenkins/workspace/Bigtop-0.8.0/label/fedora18/build/phoenix/rpm/BUILDROOT/phoenix-4.1.0.5-1.fc18.x86_64/etc/default/phoenix

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Confirmed. Not able to build with the branch. Strange because I was ableto build with the previous branch + those 3 patchs. Seems that it misses /mnt/jenkins/workspace/Bigtop-0.8.0/label/fedora18/build/phoenix/rpm/BUILDROOT/phoenix-4.1.0.5-1.fc18.x86_64/etc/default/phoenix
          Hide
          cos Konstantin Boudnik added a comment -

          bigtop-packages/src/common/phoenix is identical between the master and the branch.
          The root cause of the issue is here

            -%__install -m 0644 %{SOURCE5} $RPM_BUILD_ROOT/etc/default/%{name}
          +%__install -m 0644 %{SOURCE3} $RPM_BUILD_ROOT/etc/default/%{name}
          
          Show
          cos Konstantin Boudnik added a comment - bigtop-packages/src/common/phoenix is identical between the master and the branch. The root cause of the issue is here -%__install -m 0644 %{SOURCE5} $RPM_BUILD_ROOT/etc/ default /%{name} +%__install -m 0644 %{SOURCE3} $RPM_BUILD_ROOT/etc/ default /%{name}
          Hide
          cos Konstantin Boudnik added a comment -

          I have fixed it in BIGTOP-1444

          Show
          cos Konstantin Boudnik added a comment - I have fixed it in BIGTOP-1444
          Hide
          apurtell Andrew Purtell added a comment -

          My bad. Not currently able to test RPM builds. Let me set up some VMs for next time so that won't happen again.

          Show
          apurtell Andrew Purtell added a comment - My bad. Not currently able to test RPM builds. Let me set up some VMs for next time so that won't happen again.
          Hide
          jmspaggi Jean-Marc Spaggiari added a comment -

          Sound like we are also missing phoenix-hadoop-compat jar. Not 100% sure for now. Will retest. If that's the case should I open a new JIRA or re-open this one?

          Show
          jmspaggi Jean-Marc Spaggiari added a comment - Sound like we are also missing phoenix-hadoop-compat jar. Not 100% sure for now. Will retest. If that's the case should I open a new JIRA or re-open this one?
          Hide
          cos Konstantin Boudnik added a comment -

          Just open the new one and mark it for 0.8.0, so I can put it into the RC

          Show
          cos Konstantin Boudnik added a comment - Just open the new one and mark it for 0.8.0, so I can put it into the RC

            People

            • Assignee:
              apurtell Andrew Purtell
              Reporter:
              jmspaggi Jean-Marc Spaggiari
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development