Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: backlog
    • Fix Version/s: backlog
    • Component/s: general
    • Labels:
      None

      Description

      Bring Presto into Bigtop

        Issue Links

          Activity

          Hide
          jayunit100 jay vyas added a comment - - edited

          Hi Rob. I heard presto is quite powerful. but looking at the code, its not clear if its built to run in a polyglot environment or not.

          cons
          I've just been learning about it recently, but I dont see any deployment scripts etc (maybe im wrong)... For example, spark, hadoop , hbase and so on have a bin directory and so on where the shell scripts etc which are used to tie into bigtop are called.

          pros
          on the upside : it seems like its lightweight to get running, and is a good hedge/compliment (depending on your persepective) to other SQL type services bigtop has, and could give a great venue for people to learn more about presto in a production style setting.

          In any case, it will be good to evaluate presto in this JIRA and determine (1) if it is ready to fit into bigtop and (2) if so, how.

          if we do go through with it here are some details to help get started:

          • Looks like the common model is tarball deployment : http://prestodb.io/docs/current/installation/deployment.html
          • the discovery.uri i think is the way that things are coordinated ? IF so i think a pupet recipe similar to that of the spark/tachyon deployments we have. Having just worked on BIGTOP-1562, maybe i can provide some assistance with the puppet glueing.
          Show
          jayunit100 jay vyas added a comment - - edited Hi Rob. I heard presto is quite powerful. but looking at the code, its not clear if its built to run in a polyglot environment or not. cons I've just been learning about it recently, but I dont see any deployment scripts etc (maybe im wrong)... For example, spark , hadoop , hbase and so on have a bin directory and so on where the shell scripts etc which are used to tie into bigtop are called. pros on the upside : it seems like its lightweight to get running, and is a good hedge/compliment (depending on your persepective) to other SQL type services bigtop has, and could give a great venue for people to learn more about presto in a production style setting. In any case, it will be good to evaluate presto in this JIRA and determine (1) if it is ready to fit into bigtop and (2) if so, how. if we do go through with it here are some details to help get started: Looks like the common model is tarball deployment : http://prestodb.io/docs/current/installation/deployment.html the discovery.uri i think is the way that things are coordinated ? IF so i think a pupet recipe similar to that of the spark/tachyon deployments we have. Having just worked on BIGTOP-1562 , maybe i can provide some assistance with the puppet glueing.
          Hide
          rleidle Rob Leidle added a comment -

          Thanks Jay,

          With regards to the tarball would we want to build an RPM spec/debian packages around it or would we want to build up a new way of packaging & deploying? Perhaps with presto community taking back some changes so it is maintained.

          Show
          rleidle Rob Leidle added a comment - Thanks Jay, With regards to the tarball would we want to build an RPM spec/debian packages around it or would we want to build up a new way of packaging & deploying? Perhaps with presto community taking back some changes so it is maintained.
          Hide
          rvs Roman Shaposhnik added a comment -

          Rob Leidle what are the best references to Presto's deployment model and how it integrates with other members of Hadoop ecosystem. For example, does it integrate with things like HCatalog, etc?

          Show
          rvs Roman Shaposhnik added a comment - Rob Leidle what are the best references to Presto's deployment model and how it integrates with other members of Hadoop ecosystem. For example, does it integrate with things like HCatalog, etc?
          Hide
          rleidle Rob Leidle added a comment -

          Presto is:

          • Client of HDFS
          • Client of the Hive Metastore via the Hive Metastore Java libraries
            Some issues:
          • Trunk is compiling with Java 8, there are previous versions which use Java 7
          • Presto integrates with Apache Hadoop via this project: https://github.com/facebook/presto-hadoop-apache2, it is currently building against Hadoop 2.2.0 but would likely work if brought up to use the Bigtop Hadoop Version.

          The online documentation is actually quite good: http://prestodb.io/docs/current/

          Show
          rleidle Rob Leidle added a comment - Presto is: Client of HDFS Client of the Hive Metastore via the Hive Metastore Java libraries Some issues: Trunk is compiling with Java 8, there are previous versions which use Java 7 Presto integrates with Apache Hadoop via this project: https://github.com/facebook/presto-hadoop-apache2 , it is currently building against Hadoop 2.2.0 but would likely work if brought up to use the Bigtop Hadoop Version. The online documentation is actually quite good: http://prestodb.io/docs/current/
          Hide
          warwithin YoungWoo Kim added a comment -

          Rob Leidle Months ago I've deployed Presto on top of Bigtop stack. I've been using a home-brewing package of PrestoDB. I agree that It would be very nice to have it in Bigtop.
          I believe that scattered dependencies and Java 8 are important thing here. Most of all, It would be required to get consensus Java8 support along with Java7 in Bigtop community. I think Implementation itself – packaging, provision and etc are not a big deal at this time.

          Show
          warwithin YoungWoo Kim added a comment - Rob Leidle Months ago I've deployed Presto on top of Bigtop stack. I've been using a home-brewing package of PrestoDB. I agree that It would be very nice to have it in Bigtop. I believe that scattered dependencies and Java 8 are important thing here. Most of all, It would be required to get consensus Java8 support along with Java7 in Bigtop community. I think Implementation itself – packaging, provision and etc are not a big deal at this time.
          Hide
          jayunit100 jay vyas added a comment -

          YoungWoo Kim and Rob Leidle so, is someone going to maintain this and submit a patch?

          Show
          jayunit100 jay vyas added a comment - YoungWoo Kim and Rob Leidle so, is someone going to maintain this and submit a patch?
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          Patch, [^BIGTOP-1561.1.patch]:

          I revise my stuff to build Presto on Bigtop. Attached a WIP patch.

          Regarding build and Installation on single-node presto server:

          1) Build requirements - JDK8 and Maven 3.2.3 or higher:

          # export JAVA8_HOME=/usr/lib/jvm/java-1.8.0
          # export JAVA_HOME=$JAVA8_HOME
          
          

          2) Build

          # apply the patch...and then
          # ./gradlew presto-rpm
          
          ...
          
          # ls -als output/presto/*/*
             296 -rw-r--r--. 1 root root    300368 2015-02-17 15:38 output/presto/noarch/presto-doc-0.93-1.el6.noarch.rpm
            5112 -rw-r--r--. 1 root root   5233672 2015-02-17 15:38 output/presto/noarch/presto-jdbc-0.93-1.el6.noarch.rpm
          352456 -rw-r--r--. 1 root root 360906880 2015-02-17 15:38 output/presto/x86_64/presto-0.93-1.el6.x86_64.rpm
           10552 -rw-r--r--. 1 root root  10803264 2015-02-17 15:38 output/presto/x86_64/presto-cli-0.93-1.el6.x86_64.rpm
               4 -rw-r--r--. 1 root root      1648 2015-02-17 15:38 output/presto/x86_64/presto-debuginfo-0.93-1.el6.x86_64.rpm
               8 -rw-r--r--. 1 root root      4284 2015-02-17 15:38 output/presto/x86_64/presto-server-0.93-1.el6.x86_64.rpm
           45004 -rw-r--r--. 1 root root  46080608 2015-02-17 15:38 output/presto/x86_64/presto-verifier-0.93-1.el6.x86_64.rpm
          
          

          3) Install presto-server and presto-cli:

          
          

          4) Edit /etc/default/presto.

          JAVA8_HOME=/usr/lib/jvm/java-1.8.0
          
          export JAVA_HOME=${JAVA8_HOME}
          export PATH=${JAVA8_HOME}/bin:$PATH
          
          

          5) Edit /etc/presto/etc/config.properties:

          coordinator=true
          node-scheduler.include-coordinator=true
          http-server.http.port=8080
          task.max-memory=1GB
          discovery-server.enabled=true
          discovery.uri=http://HOSTNAME:8080
          
          

          HOSTNAME Should be replaced with your vm' or host' hostname (e.g., $ hostname -f)

          6) Start presto-server:

          # service presto-server start
          

          7) Run presto-cli like below:

          # presto
          
          presto:jmx> show catalogs;
           Catalog 
          ---------
           jmx     
           tpch    
          (2 rows)
          
          Query 20150217_065531_00012_czxxz, FINISHED, 1 node
          Splits: 2 total, 2 done (100.00%)
          0:00 [2 rows, 14B] [22 rows/s, 159B/s]
          
          presto:jmx> show schemas from jmx;
                 Schema       
          --------------------
           information_schema 
           jmx                
           sys                
          (3 rows)
          
          Query 20150217_065617_00015_czxxz, FINISHED, 1 node
          Splits: 2 total, 2 done (100.00%)
          0:00 [3 rows, 75B] [40 rows/s, 1004B/s]
          
          presto:jmx> use jmx.sys;
          presto:sys> show tables;
            Table  
          ---------
           catalog 
           node    
           query   
           task    
          (4 rows)
          
          Query 20150217_065633_00018_czxxz, FINISHED, 1 node
          Splits: 2 total, 2 done (100.00%)
          0:00 [4 rows, 144B] [66 rows/s, 2.33KB/s]
          
          presto:sys> select * from node;
           node_id |          http_uri          |   node_version   | is_active 
          ---------+----------------------------+------------------+-----------
           NODE_ID | http://172.27.199.181:8080 | presto-main:0.93 | true      
          (1 row)
          
          Query 20150217_065639_00019_czxxz, FINISHED, 1 node
          Splits: 2 total, 2 done (100.00%)
          0:00 [1 rows, 50B] [22 rows/s, 1.09KB/s]
          
          presto:sys> show schemas from tpch;
                 Schema       
          --------------------
           information_schema 
           sf1                
           sf100              
           sf1000             
           sf10000            
           sf100000           
           sf300              
           sf3000             
           sf30000            
           sys                
           tiny               
          (11 rows)
          
          Query 20150217_065711_00020_czxxz, FINISHED, 1 node
          Splits: 2 total, 2 done (100.00%)
          0:00 [11 rows, 163B] [247 rows/s, 3.59KB/s]
          
          presto:sys> use tpch.sf1;
          presto:sf1> show tables;
            Table   
          ----------
           customer 
           lineitem 
           nation   
           orders   
           part     
           partsupp 
           region   
           supplier 
          (8 rows)
          
          Query 20150217_065736_00023_czxxz, FINISHED, 1 node
          Splits: 2 total, 2 done (100.00%)
          0:00 [8 rows, 230B] [153 rows/s, 4.32KB/s]
          
          presto:sf1> select count(*) from customer;
           _col0  
          --------
           150000 
          (1 row)
          
          Query 20150217_065746_00024_czxxz, FINISHED, 1 node
          Splits: 9 total, 9 done (100.00%)
          0:04 [150K rows, 0B] [36.7K rows/s, 0B/s]
          
          presto:sf1> exit;
          
          

          Now, 'jmx' and 'tpch' are built-in catalogs for default.

          I'll update this patch and add instructions for integration Hadoop and Hive. also you can follow the official docs, https://prestodb.io/docs/current/connector/hive.html

          Show
          warwithin YoungWoo Kim added a comment - - edited Patch, [^BIGTOP-1561.1.patch] : I revise my stuff to build Presto on Bigtop. Attached a WIP patch. Regarding build and Installation on single-node presto server: 1) Build requirements - JDK8 and Maven 3.2.3 or higher: # export JAVA8_HOME=/usr/lib/jvm/java-1.8.0 # export JAVA_HOME=$JAVA8_HOME 2) Build # apply the patch...and then # ./gradlew presto-rpm ... # ls -als output/presto/*/* 296 -rw-r--r--. 1 root root 300368 2015-02-17 15:38 output/presto/noarch/presto-doc-0.93-1.el6.noarch.rpm 5112 -rw-r--r--. 1 root root 5233672 2015-02-17 15:38 output/presto/noarch/presto-jdbc-0.93-1.el6.noarch.rpm 352456 -rw-r--r--. 1 root root 360906880 2015-02-17 15:38 output/presto/x86_64/presto-0.93-1.el6.x86_64.rpm 10552 -rw-r--r--. 1 root root 10803264 2015-02-17 15:38 output/presto/x86_64/presto-cli-0.93-1.el6.x86_64.rpm 4 -rw-r--r--. 1 root root 1648 2015-02-17 15:38 output/presto/x86_64/presto-debuginfo-0.93-1.el6.x86_64.rpm 8 -rw-r--r--. 1 root root 4284 2015-02-17 15:38 output/presto/x86_64/presto-server-0.93-1.el6.x86_64.rpm 45004 -rw-r--r--. 1 root root 46080608 2015-02-17 15:38 output/presto/x86_64/presto-verifier-0.93-1.el6.x86_64.rpm 3) Install presto-server and presto-cli: 4) Edit /etc/default/presto. JAVA8_HOME=/usr/lib/jvm/java-1.8.0 export JAVA_HOME=${JAVA8_HOME} export PATH=${JAVA8_HOME}/bin:$PATH 5) Edit /etc/presto/etc/config.properties: coordinator=true node-scheduler.include-coordinator=true http-server.http.port=8080 task.max-memory=1GB discovery-server.enabled=true discovery.uri=http://HOSTNAME:8080 HOSTNAME Should be replaced with your vm' or host' hostname (e.g., $ hostname -f) 6) Start presto-server: # service presto-server start 7) Run presto-cli like below: # presto presto:jmx> show catalogs; Catalog --------- jmx tpch (2 rows) Query 20150217_065531_00012_czxxz, FINISHED, 1 node Splits: 2 total, 2 done (100.00%) 0:00 [2 rows, 14B] [22 rows/s, 159B/s] presto:jmx> show schemas from jmx; Schema -------------------- information_schema jmx sys (3 rows) Query 20150217_065617_00015_czxxz, FINISHED, 1 node Splits: 2 total, 2 done (100.00%) 0:00 [3 rows, 75B] [40 rows/s, 1004B/s] presto:jmx> use jmx.sys; presto:sys> show tables; Table --------- catalog node query task (4 rows) Query 20150217_065633_00018_czxxz, FINISHED, 1 node Splits: 2 total, 2 done (100.00%) 0:00 [4 rows, 144B] [66 rows/s, 2.33KB/s] presto:sys> select * from node; node_id | http_uri | node_version | is_active ---------+----------------------------+------------------+----------- NODE_ID | http://172.27.199.181:8080 | presto-main:0.93 | true (1 row) Query 20150217_065639_00019_czxxz, FINISHED, 1 node Splits: 2 total, 2 done (100.00%) 0:00 [1 rows, 50B] [22 rows/s, 1.09KB/s] presto:sys> show schemas from tpch; Schema -------------------- information_schema sf1 sf100 sf1000 sf10000 sf100000 sf300 sf3000 sf30000 sys tiny (11 rows) Query 20150217_065711_00020_czxxz, FINISHED, 1 node Splits: 2 total, 2 done (100.00%) 0:00 [11 rows, 163B] [247 rows/s, 3.59KB/s] presto:sys> use tpch.sf1; presto:sf1> show tables; Table ---------- customer lineitem nation orders part partsupp region supplier (8 rows) Query 20150217_065736_00023_czxxz, FINISHED, 1 node Splits: 2 total, 2 done (100.00%) 0:00 [8 rows, 230B] [153 rows/s, 4.32KB/s] presto:sf1> select count(*) from customer; _col0 -------- 150000 (1 row) Query 20150217_065746_00024_czxxz, FINISHED, 1 node Splits: 9 total, 9 done (100.00%) 0:04 [150K rows, 0B] [36.7K rows/s, 0B/s] presto:sf1> exit; Now, 'jmx' and 'tpch' are built-in catalogs for default. I'll update this patch and add instructions for integration Hadoop and Hive. also you can follow the official docs, https://prestodb.io/docs/current/connector/hive.html
          Hide
          oflebbe Olaf Flebbe added a comment -

          A '-1' for the need of a Java8 runtime for Bigtop now. If the need for JRE8/JDK8 can be isolated to this package – no objections.

          I would suggest to open up a new JIRA ignored to update maven for complete Bigtop toolchain. I did a try months ago and it broke Bigtop severely IIRC.

          Show
          oflebbe Olaf Flebbe added a comment - A '-1' for the need of a Java8 runtime for Bigtop now . If the need for JRE8/JDK8 can be isolated to this package – no objections. I would suggest to open up a new JIRA ignored to update maven for complete Bigtop toolchain. I did a try months ago and it broke Bigtop severely IIRC.
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          After a long time, I've updated on this. As Olaf Flebbe mentioned in a preceding comment, to include the Presto to Bigtop, we need updates on JDK8 issues.
          I've created a branch[1] for Presto at my Github. if someone is interested in Presto on Bigtop, that might be useful although for some reason there are ugly hacks and hardcoded blocks.

          I'll keep updates on this. Thanks!

          [1] https://github.com/youngwookim/bigtop/commits/BIGTOP-1561

          Show
          warwithin YoungWoo Kim added a comment - - edited After a long time, I've updated on this. As Olaf Flebbe mentioned in a preceding comment, to include the Presto to Bigtop, we need updates on JDK8 issues. I've created a branch [1] for Presto at my Github. if someone is interested in Presto on Bigtop, that might be useful although for some reason there are ugly hacks and hardcoded blocks. I'll keep updates on this. Thanks! [1] https://github.com/youngwookim/bigtop/commits/BIGTOP-1561
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user youngwookim opened a pull request:

          https://github.com/apache/bigtop/pull/32

          BIGTOP-1561. Add Presto as a component

          https://issues.apache.org/jira/browse/BIGTOP-1561

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/youngwookim/bigtop BIGTOP-1561

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/bigtop/pull/32.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #32


          commit 148c2f50a7237abf7b81abe6eefba6c47c69f098
          Author: Youngwoo Kim <ywkim@apache.org>
          Date: 2015-02-16T11:39:41Z

          BIGTOP-1561. Add Presto as a component


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user youngwookim opened a pull request: https://github.com/apache/bigtop/pull/32 BIGTOP-1561 . Add Presto as a component https://issues.apache.org/jira/browse/BIGTOP-1561 You can merge this pull request into a Git repository by running: $ git pull https://github.com/youngwookim/bigtop BIGTOP-1561 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/bigtop/pull/32.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #32 commit 148c2f50a7237abf7b81abe6eefba6c47c69f098 Author: Youngwoo Kim <ywkim@apache.org> Date: 2015-02-16T11:39:41Z BIGTOP-1561 . Add Presto as a component
          Hide
          tomzeng Tom Zeng added a comment - - edited

          YoungWoo Kim I was able to install the presto and jdk 1.8 manually with yum install. Looks like the services are not set up running, and also I don't see a presto cli command. Any instructions? Looks like this is pretty close. Our team is also working on this separately, I just noticed your patch yesterday.

          Show
          tomzeng Tom Zeng added a comment - - edited YoungWoo Kim I was able to install the presto and jdk 1.8 manually with yum install. Looks like the services are not set up running, and also I don't see a presto cli command. Any instructions? Looks like this is pretty close. Our team is also working on this separately, I just noticed your patch yesterday.
          Hide
          warwithin YoungWoo Kim added a comment -

          Tom Zeng, Thanks for your comment. Unfortunately, it looks like that current patch(PR) has gone stale. I'd like to update the patch but, don't have enough time to do. If you or your team would like to working on it, feel free to take this. Please let me know what do you think. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - Tom Zeng , Thanks for your comment. Unfortunately, it looks like that current patch(PR) has gone stale. I'd like to update the patch but, don't have enough time to do. If you or your team would like to working on it, feel free to take this. Please let me know what do you think. Thanks!
          Hide
          warwithin YoungWoo Kim added a comment -

          Tom Zeng, I've done quick rebase to master. Patch is here, https://patch-diff.githubusercontent.com/raw/apache/bigtop/pull/32.patch

          Feel free to let me know when you run into a problem. Thanks!

          Notes regarding instructions for Presto:
          1. Apply the patch:

          git am < patch.file
          

          2. Building Presto packages
          Make sure you have JDK8 on your build envs:

          export JAVA_HOME=/path/to/openjdk8...
          export PATH=$JAVA_HOME/bin:$PATH
          

          and then, './gradlew presto-rpm' should work.

          3. Installing and running Presto:

          # ls -als output/presto/*/*.rpm
           10416 -rw-r--r-- 1 vagrant vagrant  10663356 Sep 14 08:45 output/presto/noarch/presto-cli-0.115-1.el6.noarch.rpm
             340 -rw-r--r-- 1 vagrant vagrant    345488 Sep 14 08:45 output/presto/noarch/presto-doc-0.115-1.el6.noarch.rpm
            5256 -rw-r--r-- 1 vagrant vagrant   5379712 Sep 14 08:45 output/presto/noarch/presto-jdbc-0.115-1.el6.noarch.rpm
               8 -rw-r--r-- 1 vagrant vagrant      4384 Sep 14 08:45 output/presto/noarch/presto-server-0.115-1.el6.noarch.rpm
          453392 -rw-r--r-- 1 vagrant vagrant 464272636 Sep 14 08:46 output/presto/x86_64/presto-0.115-1.el6.x86_64.rpm
               4 -rw-r--r-- 1 vagrant vagrant      1672 Sep 14 08:46 output/presto/x86_64/presto-debuginfo-0.115-1.el6.x86_64.rpm
          
          # yum localinstall output/presto/*/*.rpm
          ......
          
          # rpm -qa | grep presto
          presto-debuginfo-0.115-1.el6.x86_64
          presto-0.115-1.el6.x86_64
          presto-cli-0.115-1.el6.noarch
          presto-doc-0.115-1.el6.noarch
          presto-server-0.115-1.el6.noarch
          presto-jdbc-0.115-1.el6.noarch
          
          # whereis presto
          presto: /usr/bin/presto /etc/presto /etc/presto.dist /usr/lib/presto
          

          Edit /etc/default/bigtop-utils:

          …..
          export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
          

          You should configure JAVA_HOME to Java8’s home directory.

          Edit /etc/presto/config.properties:

          ….
          discovery.uri=http://bigtop1.docker:8080
          

          Replace ‘bigtop1.docker’ to your hostname.

          # service presto-server start
          Starting Presto server (presto-server):                    [  OK  ]
          Started as 8440
          

          Running Presto CLI:

          $ presto
          
          presto:default> SHOW SCHEMAS FROM tpch;
                 Schema       
          --------------------
           information_schema 
           sf1                
           sf100              
           sf1000             
           sf10000            
           sf100000           
           sf300              
           sf3000             
           sf30000            
           tiny               
          (10 rows)
          
          
          presto:sf1> use tpch.sf1;
          presto:sf1> show tables;
            Table   
          ----------
           customer 
           lineitem 
           nation   
           orders   
           part     
           partsupp 
           region   
           supplier 
          (8 rows)
          
          presto:sf1> select * from customer limit 10;
           custkey |        name        |                address                | nationkey |      phone      | acctbal | mktsegment |                                                      comment                   
          ---------+--------------------+---------------------------------------+-----------+-----------------+---------+------------+--------------------------------------------------------------------------------
                 1 | Customer#000000001 | IVhzIApeRb ot,c,E                     |        15 | 25-989-741-2988 |  711.56 | BUILDING   | to the even, regular platelets. regular, ironic epitaphs nag e                 
                 2 | Customer#000000002 | XSTf4,NCwDVaWNe6tEgvwfmRchLXak        |        13 | 23-768-687-3665 |  121.65 | AUTOMOBILE | l accounts. blithely ironic theodolites integrate boldly: caref                
                 3 | Customer#000000003 | MG9kdTD2WBHm                          |         1 | 11-719-748-3364 | 7498.12 | AUTOMOBILE |  deposits eat slyly ironic, even instructions. express foxes detect slyly. blit
                 4 | Customer#000000004 | XxVSJsLAGtn                           |         4 | 14-128-190-5944 | 2866.83 | MACHINERY  |  requests. final, regular ideas sleep final accou                              
                 5 | Customer#000000005 | KvpyuHCplrB84WgAiGV6sYpZq7Tj          |         3 | 13-750-942-6364 |  794.47 | HOUSEHOLD  | n accounts will have to unwind. foxes cajole accor                             
                 6 | Customer#000000006 | sKZz0CsnMD7mp4Xd0YrBvx,LREYKUWAh yVn  |        20 | 30-114-968-4951 | 7638.57 | AUTOMOBILE | tions. even deposits boost according to the slyly bold packages. final accounts
                 7 | Customer#000000007 | TcGe5gaZNgVePxU5kRrvXBfkasDTea        |        18 | 28-190-982-9759 | 9561.95 | AUTOMOBILE | ainst the ironic, express theodolites. express, even pinto beans among the exp 
                 8 | Customer#000000008 | I0B10bB0AymmC, 0PrRYBCP1yGJ8xcBPmWhl5 |        17 | 27-147-574-9335 | 6819.74 | BUILDING   | among the slyly regular theodolites kindle blithely courts. carefully even theo
                 9 | Customer#000000009 | xKiAFTjUsCuxfeleNqefumTrjS            |         8 | 18-338-906-3675 | 8324.07 | FURNITURE  | r theodolites according to the requests wake thinly excuses: pending requests h
                10 | Customer#000000010 | 6LrEaV6KR6PLVcgl2ArL Q3rqzLzcT1 v2    |         5 | 15-741-346-9870 | 2753.54 | HOUSEHOLD  | es regular deposits haggle. fur                                                
          (10 rows)
          
          Query 20150916_092421_00025_7rhwr, FINISHED, 1 node
          Splits: 2 total, 2 done (100.00%)
          0:01 [851 rows, 0B] [591 rows/s, 0B/s]
          
          presto:sf1> 
          
          Show
          warwithin YoungWoo Kim added a comment - Tom Zeng , I've done quick rebase to master. Patch is here, https://patch-diff.githubusercontent.com/raw/apache/bigtop/pull/32.patch Feel free to let me know when you run into a problem. Thanks! Notes regarding instructions for Presto: 1. Apply the patch: git am < patch.file 2. Building Presto packages Make sure you have JDK8 on your build envs: export JAVA_HOME=/path/to/openjdk8... export PATH=$JAVA_HOME/bin:$PATH and then, './gradlew presto-rpm' should work. 3. Installing and running Presto: # ls -als output/presto/*/*.rpm 10416 -rw-r--r-- 1 vagrant vagrant 10663356 Sep 14 08:45 output/presto/noarch/presto-cli-0.115-1.el6.noarch.rpm 340 -rw-r--r-- 1 vagrant vagrant 345488 Sep 14 08:45 output/presto/noarch/presto-doc-0.115-1.el6.noarch.rpm 5256 -rw-r--r-- 1 vagrant vagrant 5379712 Sep 14 08:45 output/presto/noarch/presto-jdbc-0.115-1.el6.noarch.rpm 8 -rw-r--r-- 1 vagrant vagrant 4384 Sep 14 08:45 output/presto/noarch/presto-server-0.115-1.el6.noarch.rpm 453392 -rw-r--r-- 1 vagrant vagrant 464272636 Sep 14 08:46 output/presto/x86_64/presto-0.115-1.el6.x86_64.rpm 4 -rw-r--r-- 1 vagrant vagrant 1672 Sep 14 08:46 output/presto/x86_64/presto-debuginfo-0.115-1.el6.x86_64.rpm # yum localinstall output/presto/*/*.rpm ...... # rpm -qa | grep presto presto-debuginfo-0.115-1.el6.x86_64 presto-0.115-1.el6.x86_64 presto-cli-0.115-1.el6.noarch presto-doc-0.115-1.el6.noarch presto-server-0.115-1.el6.noarch presto-jdbc-0.115-1.el6.noarch # whereis presto presto: /usr/bin/presto /etc/presto /etc/presto.dist /usr/lib/presto Edit /etc/default/bigtop-utils: ….. export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk You should configure JAVA_HOME to Java8’s home directory. Edit /etc/presto/config.properties: …. discovery.uri=http://bigtop1.docker:8080 Replace ‘bigtop1.docker’ to your hostname. # service presto-server start Starting Presto server (presto-server): [ OK ] Started as 8440 Running Presto CLI: $ presto presto:default> SHOW SCHEMAS FROM tpch; Schema -------------------- information_schema sf1 sf100 sf1000 sf10000 sf100000 sf300 sf3000 sf30000 tiny (10 rows) presto:sf1> use tpch.sf1; presto:sf1> show tables; Table ---------- customer lineitem nation orders part partsupp region supplier (8 rows) presto:sf1> select * from customer limit 10; custkey | name | address | nationkey | phone | acctbal | mktsegment | comment ---------+--------------------+---------------------------------------+-----------+-----------------+---------+------------+-------------------------------------------------------------------------------- 1 | Customer#000000001 | IVhzIApeRb ot,c,E | 15 | 25-989-741-2988 | 711.56 | BUILDING | to the even, regular platelets. regular, ironic epitaphs nag e 2 | Customer#000000002 | XSTf4,NCwDVaWNe6tEgvwfmRchLXak | 13 | 23-768-687-3665 | 121.65 | AUTOMOBILE | l accounts. blithely ironic theodolites integrate boldly: caref 3 | Customer#000000003 | MG9kdTD2WBHm | 1 | 11-719-748-3364 | 7498.12 | AUTOMOBILE | deposits eat slyly ironic, even instructions. express foxes detect slyly. blit 4 | Customer#000000004 | XxVSJsLAGtn | 4 | 14-128-190-5944 | 2866.83 | MACHINERY | requests. final, regular ideas sleep final accou 5 | Customer#000000005 | KvpyuHCplrB84WgAiGV6sYpZq7Tj | 3 | 13-750-942-6364 | 794.47 | HOUSEHOLD | n accounts will have to unwind. foxes cajole accor 6 | Customer#000000006 | sKZz0CsnMD7mp4Xd0YrBvx,LREYKUWAh yVn | 20 | 30-114-968-4951 | 7638.57 | AUTOMOBILE | tions. even deposits boost according to the slyly bold packages. final accounts 7 | Customer#000000007 | TcGe5gaZNgVePxU5kRrvXBfkasDTea | 18 | 28-190-982-9759 | 9561.95 | AUTOMOBILE | ainst the ironic, express theodolites. express, even pinto beans among the exp 8 | Customer#000000008 | I0B10bB0AymmC, 0PrRYBCP1yGJ8xcBPmWhl5 | 17 | 27-147-574-9335 | 6819.74 | BUILDING | among the slyly regular theodolites kindle blithely courts. carefully even theo 9 | Customer#000000009 | xKiAFTjUsCuxfeleNqefumTrjS | 8 | 18-338-906-3675 | 8324.07 | FURNITURE | r theodolites according to the requests wake thinly excuses: pending requests h 10 | Customer#000000010 | 6LrEaV6KR6PLVcgl2ArL Q3rqzLzcT1 v2 | 5 | 15-741-346-9870 | 2753.54 | HOUSEHOLD | es regular deposits haggle. fur (10 rows) Query 20150916_092421_00025_7rhwr, FINISHED, 1 node Splits: 2 total, 2 done (100.00%) 0:01 [851 rows, 0B] [591 rows/s, 0B/s] presto:sf1>
          Hide
          tomzeng Tom Zeng added a comment -

          YoungWoo Kim thanks, will give it a try. Our version is based on yours with quite a bit new changes, will ask team to look at your new patch to see if anything we could incorporate before contributing back.

          Show
          tomzeng Tom Zeng added a comment - YoungWoo Kim thanks, will give it a try. Our version is based on yours with quite a bit new changes, will ask team to look at your new patch to see if anything we could incorporate before contributing back.
          Hide
          warwithin YoungWoo Kim added a comment -

          Sounds good! Thanks Tom Zeng

          Show
          warwithin YoungWoo Kim added a comment - Sounds good! Thanks Tom Zeng
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cawallin commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-183514054

          Presto has an RPM, it would be best to use that: https://github.com/facebook/presto/tree/master/presto-server-rpm.

          Show
          githubbot ASF GitHub Bot added a comment - Github user cawallin commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-183514054 Presto has an RPM, it would be best to use that: https://github.com/facebook/presto/tree/master/presto-server-rpm .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-183613836

          Does it also have deb?
          Does the layout and the build process for this is done according to the Bigtop expectation (also used by Hadoop distributions at large)?

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-183613836 Does it also have deb? Does the layout and the build process for this is done according to the Bigtop expectation (also used by Hadoop distributions at large)?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cawallin commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-185389704

          Oh, no; is there anywhere that the Bigtop RPM expectations are documented?

          Show
          githubbot ASF GitHub Bot added a comment - Github user cawallin commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-185389704 Oh, no; is there anywhere that the Bigtop RPM expectations are documented?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user c0s commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-185569957

          There's no explicit document on RPM's requirements, but all of our packaging code is following the same rules and informal guidelines, codified within the existing packaging code. Good examples for complex packages would be hadoop and hbase. Lesser complex stuff is ignite-hadoop and similar.

          We won't be able to accept only RPM packaging though: half of our supported OS are deb-based.

          Show
          githubbot ASF GitHub Bot added a comment - Github user c0s commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-185569957 There's no explicit document on RPM's requirements, but all of our packaging code is following the same rules and informal guidelines, codified within the existing packaging code. Good examples for complex packages would be hadoop and hbase. Lesser complex stuff is ignite-hadoop and similar. We won't be able to accept only RPM packaging though: half of our supported OS are deb-based.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-185595881

          @cawallin I did basic packaging for presto to use our internal deployment and at that time, there is no packages like rpm or deb on Presto. I'm aware of that you and your team (Teradata, right?) are doing great work in Presto community As of now, we have some prerequisites to integrate Presto into Bigtop stack. and... now, I believe it's a good starting point to discuss on that. Thanks!

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-185595881 @cawallin I did basic packaging for presto to use our internal deployment and at that time, there is no packages like rpm or deb on Presto. I'm aware of that you and your team (Teradata, right?) are doing great work in Presto community As of now, we have some prerequisites to integrate Presto into Bigtop stack. and... now, I believe it's a good starting point to discuss on that. Thanks!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cawallin commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-185824052

          Sure, thanks for the update, sorry I was a bit insistent before I'm not sure that we'll have room in our roadmap to work on deb packaging, but tools like presto-admin (https://github.com/prestodb/presto-admin) depend a little on how the RPM/init scripts work, and it'd be great if presto-admin worked for both RPMs. After looking more carefully through the code, I think it should work, though I made one comment about configurability (e.g. if you discover after installation that your /var mount is really small and want to change the log directory after the fact), and I'd have to test out the RPM with our presto-admin tests.

          Show
          githubbot ASF GitHub Bot added a comment - Github user cawallin commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-185824052 Sure, thanks for the update, sorry I was a bit insistent before I'm not sure that we'll have room in our roadmap to work on deb packaging, but tools like presto-admin ( https://github.com/prestodb/presto-admin ) depend a little on how the RPM/init scripts work, and it'd be great if presto-admin worked for both RPMs. After looking more carefully through the code, I think it should work, though I made one comment about configurability (e.g. if you discover after installation that your /var mount is really small and want to change the log directory after the fact), and I'd have to test out the RPM with our presto-admin tests.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cawallin commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/32#discussion_r53353145

          — Diff: bigtop-packages/src/common/presto/presto-server.svc —
          @@ -0,0 +1,74 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +TYPE="server"
          +DAEMON="presto-$

          {TYPE}"
          +DESC="Presto ${TYPE}

          "
          +EXEC_PATH="/usr/lib/presto/bin/launcher"
          +SVC_USER="presto"
          +WORKING_DIR="/usr/lib/presto"
          +DAEMON_FLAGS="--launcher-config /usr/lib/presto/bin/launcher.properties --jvm-config /etc/presto/jvm.config --config /etc/presto/config.properties --data-dir /var/lib/presto --pid-file /var/run/presto/$

          {DAEMON}.pid --launcher-log-file /var/log/presto/${DAEMON}

          .out --server-log-file /var/log/presto/$

          {DAEMON}

          .log"
          — End diff –

          Not sure if this is relevant for BigTop or not, but it's possible to set pretty much all of the log/data file locations in the Presto node.properties file (e.g. node.launcher-log-file), which wouldn't be picked up by this hard coding of locations.

          Show
          githubbot ASF GitHub Bot added a comment - Github user cawallin commented on a diff in the pull request: https://github.com/apache/bigtop/pull/32#discussion_r53353145 — Diff: bigtop-packages/src/common/presto/presto-server.svc — @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +TYPE="server" +DAEMON="presto-$ {TYPE}" +DESC="Presto ${TYPE} " +EXEC_PATH="/usr/lib/presto/bin/launcher" +SVC_USER="presto" +WORKING_DIR="/usr/lib/presto" +DAEMON_FLAGS="--launcher-config /usr/lib/presto/bin/launcher.properties --jvm-config /etc/presto/jvm.config --config /etc/presto/config.properties --data-dir /var/lib/presto --pid-file /var/run/presto/$ {DAEMON}.pid --launcher-log-file /var/log/presto/${DAEMON} .out --server-log-file /var/log/presto/$ {DAEMON} .log" — End diff – Not sure if this is relevant for BigTop or not, but it's possible to set pretty much all of the log/data file locations in the Presto node.properties file (e.g. node.launcher-log-file), which wouldn't be picked up by this hard coding of locations.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim commented on a diff in the pull request:

          https://github.com/apache/bigtop/pull/32#discussion_r53773010

          — Diff: bigtop-packages/src/common/presto/presto-server.svc —
          @@ -0,0 +1,74 @@
          +# Licensed to the Apache Software Foundation (ASF) under one or more
          +# contributor license agreements. See the NOTICE file distributed with
          +# this work for additional information regarding copyright ownership.
          +# The ASF licenses this file to You under the Apache License, Version 2.0
          +# (the "License"); you may not use this file except in compliance with
          +# the License. You may obtain a copy of the License at
          +#
          +# http://www.apache.org/licenses/LICENSE-2.0
          +#
          +# Unless required by applicable law or agreed to in writing, software
          +# distributed under the License is distributed on an "AS IS" BASIS,
          +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          +# See the License for the specific language governing permissions and
          +# limitations under the License.
          +
          +TYPE="server"
          +DAEMON="presto-$

          {TYPE}"
          +DESC="Presto ${TYPE}

          "
          +EXEC_PATH="/usr/lib/presto/bin/launcher"
          +SVC_USER="presto"
          +WORKING_DIR="/usr/lib/presto"
          +DAEMON_FLAGS="--launcher-config /usr/lib/presto/bin/launcher.properties --jvm-config /etc/presto/jvm.config --config /etc/presto/config.properties --data-dir /var/lib/presto --pid-file /var/run/presto/$

          {DAEMON}.pid --launcher-log-file /var/log/presto/${DAEMON}

          .out --server-log-file /var/log/presto/$

          {DAEMON}

          .log"
          — End diff –

          @cawallin Good point! Will take a look and update that. Thanks.

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim commented on a diff in the pull request: https://github.com/apache/bigtop/pull/32#discussion_r53773010 — Diff: bigtop-packages/src/common/presto/presto-server.svc — @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +TYPE="server" +DAEMON="presto-$ {TYPE}" +DESC="Presto ${TYPE} " +EXEC_PATH="/usr/lib/presto/bin/launcher" +SVC_USER="presto" +WORKING_DIR="/usr/lib/presto" +DAEMON_FLAGS="--launcher-config /usr/lib/presto/bin/launcher.properties --jvm-config /etc/presto/jvm.config --config /etc/presto/config.properties --data-dir /var/lib/presto --pid-file /var/run/presto/$ {DAEMON}.pid --launcher-log-file /var/log/presto/${DAEMON} .out --server-log-file /var/log/presto/$ {DAEMON} .log" — End diff – @cawallin Good point! Will take a look and update that. Thanks.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user youngwookim commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-187688594

          @cawallin I'll look into presto-server-rpm and presto-admin. I believe presto-admin is nice so it would be nice to have the compatibility between Bigtop's artifacts and presto-admin.

          First of all, packaging rpm on Bigtop is a bit different from presto-server-rpm. overall, we have a base package and sub packages. the spec file for Hadoop is a good example: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec

          I applied the rules to Presto's one. so there are base package 'presto' and the rest of the packages are sub package: 'presto-server' for server service, 'presto-cli' for Presto client, 'presto-jdbc' for JDBC driver and 'presto-doc' for generated documentation. usually, we have configurations and init script in the server package.

          Looks like presto-server-rpm contains the server package only. I wonder if you have a plan for cli, doc and etc.

          Show
          githubbot ASF GitHub Bot added a comment - Github user youngwookim commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-187688594 @cawallin I'll look into presto-server-rpm and presto-admin. I believe presto-admin is nice so it would be nice to have the compatibility between Bigtop's artifacts and presto-admin. First of all, packaging rpm on Bigtop is a bit different from presto-server-rpm. overall, we have a base package and sub packages. the spec file for Hadoop is a good example: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/rpm/hadoop/SPECS/hadoop.spec I applied the rules to Presto's one. so there are base package 'presto' and the rest of the packages are sub package: 'presto-server' for server service, 'presto-cli' for Presto client, 'presto-jdbc' for JDBC driver and 'presto-doc' for generated documentation. usually, we have configurations and init script in the server package. Looks like presto-server-rpm contains the server package only. I wonder if you have a plan for cli, doc and etc.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cawallin commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-188324478

          Thanks for the explanation! We aren't actively working on the RPM right now, but we have it in our backlog to look into adding the CLI/docs/verifier/benchmark driver to the RPM, though we'd probably have to refactor it to move to the packages/subpackages approach you have here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user cawallin commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-188324478 Thanks for the explanation! We aren't actively working on the RPM right now, but we have it in our backlog to look into adding the CLI/docs/verifier/benchmark driver to the RPM, though we'd probably have to refactor it to move to the packages/subpackages approach you have here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user spew commented on the pull request:

          https://github.com/apache/bigtop/pull/32#issuecomment-193497374

          On line 211 of install_presto.sh we reference the location of presto-env.sh without including the $CONF_FOLDER directory

          Show
          githubbot ASF GitHub Bot added a comment - Github user spew commented on the pull request: https://github.com/apache/bigtop/pull/32#issuecomment-193497374 On line 211 of install_presto.sh we reference the location of presto-env.sh without including the $CONF_FOLDER directory
          Hide
          warwithin YoungWoo Kim added a comment -

          For now I'm not working on presto integration for Bigtop. Jonathan Kelly, Tom Zeng Do you have some cycles on this? AFAIK, presto is a component of Amazon's solution stack. Anyway, feel free to take this If anyone is interested. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - For now I'm not working on presto integration for Bigtop. Jonathan Kelly , Tom Zeng Do you have some cycles on this? AFAIK, presto is a component of Amazon's solution stack. Anyway, feel free to take this If anyone is interested. Thanks!

            People

            • Assignee:
              Unassigned
              Reporter:
              rleidle Rob Leidle
            • Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development