Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: general
    • Labels:
      None

      Description

      After BIGTOP-1281, we should bump the shipped version of Apache Phoenix to the 4.x release line, currently 4.0.0.

      Phoenix 4.x is meant to run on HBase 0.98. Phoenix 3.x is meant to run on HBase 0.96. Phoenix 2.x is meant to run on HBase 0.94.

      1. BIGTOP-1285.1.patch
        18 kB
        YoungWoo Kim
      2. BIGTOP-1285.2.patch
        22 kB
        YoungWoo Kim
      3. BIGTOP-1285.3.patch
        21 kB
        YoungWoo Kim
      4. BIGTOP-1285.patch
        8 kB
        YoungWoo Kim

        Activity

        Hide
        warwithin YoungWoo Kim added a comment -

        Initial patch for Apache Phoenix 4.0.0

        Show
        warwithin YoungWoo Kim added a comment - Initial patch for Apache Phoenix 4.0.0
        Hide
        apurtell Andrew Purtell added a comment -

        lgtm, this has been tested? What happens when you 'mvn verify' the Phoenix smokes?

        Show
        apurtell Andrew Purtell added a comment - lgtm, this has been tested? What happens when you 'mvn verify' the Phoenix smokes?
        Hide
        warwithin YoungWoo Kim added a comment -

        Updated patch

        • Fix invalid symlinks
        • Update namespace for the tests
        Show
        warwithin YoungWoo Kim added a comment - Updated patch Fix invalid symlinks Update namespace for the tests
        Hide
        warwithin YoungWoo Kim added a comment -

        Hi Andrew,

        I attached an updated patch. I don't know much about Bigtop' smoke tests so any instructions I can follow?

        Show
        warwithin YoungWoo Kim added a comment - Hi Andrew, I attached an updated patch. I don't know much about Bigtop' smoke tests so any instructions I can follow?
        Hide
        warwithin YoungWoo Kim added a comment -

        Andrew,

        I'm looking into smoke test for Phoenix. It seems that the test classes on e2e changed.
        I Modified class name for smokes and did run smoke tests and then I got following errors:

        ......
        Running org.apache.bigtop.itest.phoenix.smoke.TestPhoenixSmoke
        14/05/23 15:58:42 TRACE shell.Shell: /bin/bash -s << __EOT__
        HBASE_CLASSPATH=/usr/lib/phoenix/phoenix-client.jar:/usr/lib/phoenix/phoenix-core-4.0.0-incubating-tests.jar hbase org.junit.runner.JUnitCore org.apache.phoenix.end2end.AlterTableIT
        __EOT__
        14/05/23 15:58:43 TRACE shell.Shell: return: 1
        14/05/23 15:58:43 TRACE shell.Shell: 
        <stdout>
        JUnit version 4.11
        Could not find class: org.apache.phoenix.end2end.AlterTableIT
        
        Time: 0.005
        
        OK (0 tests)
        
        </stdout>
        
        .....
        

        Hmm, I don't know why the junit can not pick up the classes.

        Show
        warwithin YoungWoo Kim added a comment - Andrew, I'm looking into smoke test for Phoenix. It seems that the test classes on e2e changed. I Modified class name for smokes and did run smoke tests and then I got following errors: ...... Running org.apache.bigtop.itest.phoenix.smoke.TestPhoenixSmoke 14/05/23 15:58:42 TRACE shell.Shell: /bin/bash -s << __EOT__ HBASE_CLASSPATH=/usr/lib/phoenix/phoenix-client.jar:/usr/lib/phoenix/phoenix-core-4.0.0-incubating-tests.jar hbase org.junit.runner.JUnitCore org.apache.phoenix.end2end.AlterTableIT __EOT__ 14/05/23 15:58:43 TRACE shell.Shell: return: 1 14/05/23 15:58:43 TRACE shell.Shell: <stdout> JUnit version 4.11 Could not find class: org.apache.phoenix.end2end.AlterTableIT Time: 0.005 OK (0 tests) </stdout> ..... Hmm, I don't know why the junit can not pick up the classes.
        Hide
        apurtell Andrew Purtell added a comment -

        As a quick sanity check, I built Phoenix 4.0 from a source checkout, and was able to run the unit test:

        $ HBASE_CLASSPATH=/var/tmp/phoenix-4.0.0-incubating/lib/phoenix-core-4.0.0-incubating.jar:/var/tmp/phoenix-4.0.0-incubating/lib/phoenix-core-4.0.0-incubating-tests.jar ./bin/hbase org.junit.runner.JUnitCore org.apache.phoenix.end2end.AlterTableIT
        
        JUnit version 4.11
        2014-05-23 14:15:24,598 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
        2014-05-23 14:15:24,822 INFO  [main] hbase.HBaseCommonTestingUtility: Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s)
        ...
        

        As I didn't have a client jar symlink I put the core jar from phoenix/lib/ on the classpath. I would check your environment and check that the installed result has a valid symlink. Hope that helps.

        Show
        apurtell Andrew Purtell added a comment - As a quick sanity check, I built Phoenix 4.0 from a source checkout, and was able to run the unit test: $ HBASE_CLASSPATH=/var/tmp/phoenix-4.0.0-incubating/lib/phoenix-core-4.0.0-incubating.jar:/var/tmp/phoenix-4.0.0-incubating/lib/phoenix-core-4.0.0-incubating-tests.jar ./bin/hbase org.junit.runner.JUnitCore org.apache.phoenix.end2end.AlterTableIT JUnit version 4.11 2014-05-23 14:15:24,598 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available 2014-05-23 14:15:24,822 INFO [main] hbase.HBaseCommonTestingUtility: Starting up minicluster with 1 master(s) and 1 regionserver(s) and 1 datanode(s) ... As I didn't have a client jar symlink I put the core jar from phoenix/lib/ on the classpath. I would check your environment and check that the installed result has a valid symlink. Hope that helps.
        Hide
        warwithin YoungWoo Kim added a comment -

        Updated patch:

        • Fix the invalid symlinks
        • Revise the smoke tests

        Verified with bigtop smoke tests. There is no failure:

        ...
        Results :
        
        Tests run: 73, Failures: 0, Errors: 0, Skipped: 0
        
        Show
        warwithin YoungWoo Kim added a comment - Updated patch: Fix the invalid symlinks Revise the smoke tests Verified with bigtop smoke tests. There is no failure: ... Results : Tests run: 73, Failures: 0, Errors: 0, Skipped: 0
        Hide
        warwithin YoungWoo Kim added a comment -

        Andrew,

        Can you review the patch(BIGTOP-1285.2.patch)?

        Show
        warwithin YoungWoo Kim added a comment - Andrew, Can you review the patch( BIGTOP-1285 .2.patch)?
        Hide
        apurtell Andrew Purtell added a comment -

        Patch looks good.

        Minor stuff.

        Why are we forcing a particular snappy version?

        +    -Dsnappy.version=1.0.4.1  \
        

        Have you tried installing the bigtop Phoenix package and its dependencies, launching processes, and confirming the sqlline shell works?

        -# Fix up sqlline.sh for Bigtop install layout
        -sed -i -e s#target## -e s#\$current_dir/log4j.properties#$ETC_DIR/conf/log4j.properties# $PREFIX/$BIN_DIR/sqlline.sh
        

        Phoenix is now a TLP, please drop 'incubator' in project URL references:

        +Source: http://phoenix.incubator.apache.org
        

        Otherwise looks great!

        Show
        apurtell Andrew Purtell added a comment - Patch looks good. Minor stuff. Why are we forcing a particular snappy version? + -Dsnappy.version=1.0.4.1 \ Have you tried installing the bigtop Phoenix package and its dependencies, launching processes, and confirming the sqlline shell works? -# Fix up sqlline.sh for Bigtop install layout -sed -i -e s#target## -e s#\$current_dir/log4j.properties#$ETC_DIR/conf/log4j.properties# $PREFIX/$BIN_DIR/sqlline.sh Phoenix is now a TLP, please drop 'incubator' in project URL references: +Source: http: //phoenix.incubator.apache.org Otherwise looks great!
        Hide
        warwithin YoungWoo Kim added a comment -

        Sorry for the late reply I've been very busy for a while.

        1. To pass the Phoenix itests, I need to specify the snappy version. without that, I got following test failures:

        Tests run: 73, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 4,152.158 sec <<< FAILURE!
        
        Results :
        
        Failed tests:   testHashJoin(org.apache.bigtop.itest.phoenix.smoke.TestPhoenixSmoke): org.apache.phoenix.end2end.HashJoinIT failed
          testTenantSpecificTablesDML(org.apache.bigtop.itest.phoenix.smoke.TestPhoenixSmoke): org.apache.phoenix.end2end.TenantSpecificTablesDMLIT failed
        
        Tests run: 73, Failures: 2, Errors: 0, Skipped: 0
        

        2. I did simple tests for the Phoenix client using included examples. It works fine for me. please check following outputs:

        # cd /usr/lib/phoenix/bin
        # export PHOENIX_LIB_DIR=/usr/lib/phoenix
        # ./sqlline.py localhost /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.sql
        Setting property: [isolation, TRANSACTION_READ_COMMITTED]
        Setting property: [run, /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.sql]
        issuing: !connect jdbc:phoenix:localhost none none org.apache.phoenix.jdbc.PhoenixDriver
        Connecting to jdbc:phoenix:localhost
        14/06/21 13:05:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        Connected to: Phoenix (version 4.0)
        Driver: org.apache.phoenix.jdbc.PhoenixDriver (version 4.0)
        Autocommit status: true
        Transaction isolation: TRANSACTION_READ_COMMITTED
        Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
        53/53 (100%) Done
        Done
        ...
        -- creates stock table with single row
        CREATE TABLE IF NOT EXISTS STOCK_SYMBOL (SYMBOL VARCHAR NOT NULL PRIMARY KEY, COMPANY VARCHAR);
        No rows affected (0.357 seconds)
        3/5          UPSERT INTO STOCK_SYMBOL VALUES ('CRM','SalesForce.com');
        1 row affected (0.076 seconds)
        4/5          SELECT * FROM STOCK_SYMBOL;
        +------------+------------+
        |   SYMBOL   |  COMPANY   |
        +------------+------------+
        | CRM        | SalesForce.com |
        +------------+------------+
        1 row selected (0.049 seconds)
        5/5
        Closing: org.apache.phoenix.jdbc.PhoenixConnection
        sqlline version 1.1.2
        
        # ./psql.py localhost /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.csv /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.sql
        14/06/21 13:11:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        csv columns from database.
        CSV Upsert complete. 9 rows upserted
        Time: 0.062 sec(s)
        
        no rows upserted
        Time: 0.134 sec(s)
        
        1 row upserted
        Time: 0.007 sec(s)
        
        SYMBOL     COMPANY
        ---------- ----------
        AAPL       APPLE Inc.
        CRM        SalesForce.com
        GOOG       Google
        HOG        Harlet-Davidson Inc.
        HPQ        Hewlett Packard
        INTC       Intel
        MSFT       Microsoft
        WAG        Walgreens
        WMT        Walmart
        Time: 0.02 sec(s)
        

        3. I'll update the Phoenix's project URL and then I'll upload a new patch.

        Show
        warwithin YoungWoo Kim added a comment - Sorry for the late reply I've been very busy for a while. 1. To pass the Phoenix itests, I need to specify the snappy version. without that, I got following test failures: Tests run: 73, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 4,152.158 sec <<< FAILURE! Results : Failed tests: testHashJoin(org.apache.bigtop.itest.phoenix.smoke.TestPhoenixSmoke): org.apache.phoenix.end2end.HashJoinIT failed testTenantSpecificTablesDML(org.apache.bigtop.itest.phoenix.smoke.TestPhoenixSmoke): org.apache.phoenix.end2end.TenantSpecificTablesDMLIT failed Tests run: 73, Failures: 2, Errors: 0, Skipped: 0 2. I did simple tests for the Phoenix client using included examples. It works fine for me. please check following outputs: # cd /usr/lib/phoenix/bin # export PHOENIX_LIB_DIR=/usr/lib/phoenix # ./sqlline.py localhost /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.sql Setting property: [isolation, TRANSACTION_READ_COMMITTED] Setting property: [run, /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.sql] issuing: !connect jdbc:phoenix:localhost none none org.apache.phoenix.jdbc.PhoenixDriver Connecting to jdbc:phoenix:localhost 14/06/21 13:05:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Connected to: Phoenix (version 4.0) Driver: org.apache.phoenix.jdbc.PhoenixDriver (version 4.0) Autocommit status: true Transaction isolation: TRANSACTION_READ_COMMITTED Building list of tables and columns for tab-completion (set fastconnect to true to skip)... 53/53 (100%) Done Done ... -- creates stock table with single row CREATE TABLE IF NOT EXISTS STOCK_SYMBOL (SYMBOL VARCHAR NOT NULL PRIMARY KEY, COMPANY VARCHAR); No rows affected (0.357 seconds) 3/5 UPSERT INTO STOCK_SYMBOL VALUES ('CRM','SalesForce.com'); 1 row affected (0.076 seconds) 4/5 SELECT * FROM STOCK_SYMBOL; +------------+------------+ | SYMBOL | COMPANY | +------------+------------+ | CRM | SalesForce.com | +------------+------------+ 1 row selected (0.049 seconds) 5/5 Closing: org.apache.phoenix.jdbc.PhoenixConnection sqlline version 1.1.2 # ./psql.py localhost /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.csv /usr/share/doc/phoenix-4.0.0/examples/STOCK_SYMBOL.sql 14/06/21 13:11:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable csv columns from database. CSV Upsert complete. 9 rows upserted Time: 0.062 sec(s) no rows upserted Time: 0.134 sec(s) 1 row upserted Time: 0.007 sec(s) SYMBOL COMPANY ---------- ---------- AAPL APPLE Inc. CRM SalesForce.com GOOG Google HOG Harlet-Davidson Inc. HPQ Hewlett Packard INTC Intel MSFT Microsoft WAG Walgreens WMT Walmart Time: 0.02 sec(s) 3. I'll update the Phoenix's project URL and then I'll upload a new patch.
        Hide
        warwithin YoungWoo Kim added a comment -

        BIGTOP-1285.3.patch addressed Andrew's review comments:

        • Update the Phoenix's project URL
        Show
        warwithin YoungWoo Kim added a comment - BIGTOP-1285 .3.patch addressed Andrew's review comments: Update the Phoenix's project URL
        Hide
        rvs Roman Shaposhnik added a comment -

        +1 and committed! Thanks for the patch!

        Show
        rvs Roman Shaposhnik added a comment - +1 and committed! Thanks for the patch!

          People

          • Assignee:
            warwithin YoungWoo Kim
            Reporter:
            apurtell Andrew Purtell
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development