[PHOENIX-3235] Tephra errors when trying to create a transactional table in Phoenix 4.8.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.8.0
• Fix Version/s: 4.8.1

Environment

Alpine Linux 3.4, OpenJDK 8 (JDK) in Docker 1.12.1 containers

• Assignee: Unassigned
• Reporter: Francis Chuang
• Created: 01/Sep/16 01:00
• Updated: 29/Sep/16 00:49
• Resolved: 29/Sep/16 00:49

Description

I've build a docker image to run HBase 1.2 with Phoenix 4.8.0 in fully distributed mode. There are 2 images, 1 containing HBase and Phoenix, and 1 with just the phoenix query server. Java is OpenJDK 8 (JDK flavour).

The docker images can be found in my branch here: https://github.com/F21/hbase-phoenix/tree/consolidate-images/

To build each image, simply go into the appropriate folder and do a `docker build . -t myimage`.

To run the master, these environment variables are required:

CLUSTER_NAME: hbase
HBASE_ROLE: master
HBASE_ZOOKEEPER_QUORUM: myzk
HDFS_CLUSTER_NAME: mycluster
DFS_NAMENODE_RPC_ADDRESS_NN1: nn1:8020
DFS_NAMENODE_RPC_ADDRESS_NN2: nn2:8020
DFS_NAMENODE_HTTP_ADDRESS_NN1: nn1:50070
DFS_NAMENODE_HTTP_ADDRESS_NN2: nn2:50070

To run the region server, these environment variables are required:

CLUSTER_NAME: hbase
HBASE_ROLE: regionserver
HBASE_ZOOKEEPER_QUORUM: myzk
HDFS_CLUSTER_NAME: mycluster
DFS_NAMENODE_RPC_ADDRESS_NN1: nn1:8020
DFS_NAMENODE_RPC_ADDRESS_NN2: nn2:8020
DFS_NAMENODE_HTTP_ADDRESS_NN1: nn1:50070
DFS_NAMENODE_HTTP_ADDRESS_NN2: nn2:50070

HBase, the transaction server starts up correctly along with the query server. I can connect to the query server using SquirrleSQL and the Phoenix tables were created correctly. I am also able to create non-transactional tables and run queries against them.

However, if I try to create a transactional table, I get an error message saying:

Error: Error -1 (00000) : Error while executing SQL "CREATE TABLE my_table (k BIGINT PRIMARY KEY, v VARCHAR) TRANSACTIONAL=true": Remote driver error: RuntimeException: java.lang.Exception: Thrift error for org.apache.tephra.distributed.TransactionServiceClient$2@660ae15c: Internal error processing startShort -> Exception: Thrift error for org.apache.tephra.distributed.TransactionServiceClient$2@660ae15c: Internal error processing startShort -> TApplicationException: Internal error processing startShort
SQLState:  00000
ErrorCode: -1

JPS confirms that the transaction manager is still running:

bash-4.3# jps
771 Jps
138 HMaster
190 TransactionServiceMain

In ` /tmp/tephra-/tephra-service--m9edd51-hmaster1.m9edd51.log`, it logs the following:

Wed Aug 31 23:12:33 UTC 2016 Starting tephra service on m9edd51-hmaster1.m9edd51
-f: file size (blocks)             unlimited
-t: cpu time (seconds)             unlimited
-d: data seg size (kb)             unlimited
-s: stack size (kb)                8192
-c: core file size (blocks)        unlimited
-m: resident set size (kb)         unlimited
-l: locked memory (kb)             64
-p: processes                      unlimited
-n: file descriptors               65536
-v: address space (kb)             unlimited
-w: locks                          unlimited
-e: scheduling priority            0
-r: real-time priority             0
Command:  /usr/lib/jvm/java-1.8-openjdk/bin/java -XX:+UseConcMarkSweepGC -cp /opt/hbase/bin/../lib/*:/opt/hbase/bin/../conf/:/opt/hbase/phoenix-client/target/*:/opt/hbase/conf:/usr/lib/jvm/java-1.8-openjdk/lib/tools.jar:/opt/hbase:/opt/hbase/lib/activation-1.1.jar:/opt/hbase/lib/antisamy-1.5.3.jar:/opt/hbase/lib/aopalliance-1.0.jar:/opt/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase/lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase/lib/api-util-1.0.0-M20.jar:/opt/hbase/lib/asm-3.1.jar:/opt/hbase/lib/avro-1.7.4.jar:/opt/hbase/lib/batik-css-1.8.jar:/opt/hbase/lib/batik-ext-1.8.jar:/opt/hbase/lib/batik-util-1.8.jar:/opt/hbase/lib/bsh-core-2.0b4.jar:/opt/hbase/lib/commons-beanutils-1.7.0.jar:/opt/hbase/lib/commons-beanutils-core-1.8.3.jar:/opt/hbase/lib/commons-cli-1.2.jar:/opt/hbase/lib/commons-codec-1.9.jar:/opt/hbase/lib/commons-collections-3.2.2.jar:/opt/hbase/lib/commons-compress-1.4.1.jar:/opt/hbase/lib/commons-configuration-1.6.jar:/opt/hbase/lib/commons-daemon-1.0.13.jar:/opt/hbase/lib/commons-digester-1.8.jar:/opt/hbase/lib/commons-el-1.0.jar:/opt/hbase/lib/commons-fileupload-1.3.1.jar:/opt/hbase/lib/commons-httpclient-3.1.jar:/opt/hbase/lib/commons-io-2.4.jar:/opt/hbase/lib/commons-lang-2.6.jar:/opt/hbase/lib/commons-logging-1.2.jar:/opt/hbase/lib/commons-math-2.2.jar:/opt/hbase/lib/commons-math3-3.1.1.jar:/opt/hbase/lib/commons-net-3.1.jar:/opt/hbase/lib/disruptor-3.3.0.jar:/opt/hbase/lib/esapi-2.1.0.1.jar:/opt/hbase/lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase/lib/guava-12.0.1.jar:/opt/hbase/lib/guice-3.0.jar:/opt/hbase/lib/guice-servlet-3.0.jar:/opt/hbase/lib/hadoop-annotations-2.5.1.jar:/opt/hbase/lib/hadoop-auth-2.5.1.jar:/opt/hbase/lib/hadoop-client-2.5.1.jar:/opt/hbase/lib/hadoop-common-2.5.1.jar:/opt/hbase/lib/hadoop-hdfs-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-api-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-client-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-common-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/opt/hbase/lib/hbase-annotations-1.2.2-tests.jar:/opt/hbase/lib/hbase-annotations-1.2.2.jar:/opt/hbase/lib/hbase-client-1.2.2.jar:/opt/hbase/lib/hbase-common-1.2.2-tests.jar:/opt/hbase/lib/hbase-common-1.2.2.jar:/opt/hbase/lib/hbase-examples-1.2.2.jar:/opt/hbase/lib/hbase-external-blockcache-1.2.2.jar:/opt/hbase/lib/hbase-hadoop-compat-1.2.2.jar:/opt/hbase/lib/hbase-hadoop2-compat-1.2.2.jar:/opt/hbase/lib/hbase-it-1.2.2-tests.jar:/opt/hbase/lib/hbase-it-1.2.2.jar:/opt/hbase/lib/hbase-prefix-tree-1.2.2.jar:/opt/hbase/lib/hbase-procedure-1.2.2.jar:/opt/hbase/lib/hbase-protocol-1.2.2.jar:/opt/hbase/lib/hbase-resource-bundle-1.2.2.jar:/opt/hbase/lib/hbase-rest-1.2.2.jar:/opt/hbase/lib/hbase-server-1.2.2-tests.jar:/opt/hbase/lib/hbase-server-1.2.2.jar:/opt/hbase/lib/hbase-shell-1.2.2.jar:/opt/hbase/lib/hbase-thrift-1.2.2.jar:/opt/hbase/lib/htrace-core-3.1.0-incubating.jar:/opt/hbase/lib/httpclient-4.2.5.jar:/opt/hbase/lib/httpcore-4.4.1.jar:/opt/hbase/lib/jackson-core-asl-1.9.13.jar:/opt/hbase/lib/jackson-jaxrs-1.9.13.jar:/opt/hbase/lib/jackson-mapper-asl-1.9.13.jar:/opt/hbase/lib/jackson-xc-1.9.13.jar:/opt/hbase/lib/jamon-runtime-2.4.1.jar:/opt/hbase/lib/jasper-compiler-5.5.23.jar:/opt/hbase/lib/jasper-runtime-5.5.23.jar:/opt/hbase/lib/java-xmlbuilder-0.4.jar:/opt/hbase/lib/javax.inject-1.jar:/opt/hbase/lib/jaxb-api-2.2.2.jar:/opt/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/lib/jcodings-1.0.8.jar:/opt/hbase/lib/jersey-client-1.9.jar:/opt/hbase/lib/jersey-core-1.9.jar:/opt/hbase/lib/jersey-guice-1.9.jar:/opt/hbase/lib/jersey-json-1.9.jar:/opt/hbase/lib/jersey-server-1.9.jar:/opt/hbase/lib/jets3t-0.9.0.jar:/opt/hbase/lib/jettison-1.3.3.jar:/opt/hbase/lib/jetty-6.1.26.jar:/opt/hbase/lib/jetty-sslengine-6.1.26.jar:/opt/hbase/lib/jetty-util-6.1.26.jar:/opt/hbase/lib/joni-2.1.2.jar:/opt/hbase/lib/jruby-complete-1.6.8.jar:/opt/hbase/lib/jsch-0.1.42.jar:/opt/hbase/lib/jsp-2.1-6.1.14.jar:/opt/hbase/lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/lib/jsr305-1.3.9.jar:/opt/hbase/lib/junit-4.12.jar:/opt/hbase/lib/leveldbjni-all-1.8.jar:/opt/hbase/lib/libthrift-0.9.3.jar:/opt/hbase/lib/log4j-1.2.17.jar:/opt/hbase/lib/metrics-core-2.2.0.jar:/opt/hbase/lib/nekohtml-1.9.16.jar:/opt/hbase/lib/netty-all-4.0.23.Final.jar:/opt/hbase/lib/paranamer-2.3.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-client.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-server.jar:/opt/hbase/lib/protobuf-java-2.5.0.jar:/opt/hbase/lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/lib/servlet-api-2.5.jar:/opt/hbase/lib/slf4j-api-1.7.7.jar:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar:/opt/hbase/lib/snappy-java-1.0.4.1.jar:/opt/hbase/lib/spymemcached-2.11.6.jar:/opt/hbase/lib/xalan-2.7.0.jar:/opt/hbase/lib/xml-apis-1.3.03.jar:/opt/hbase/lib/xml-apis-ext-1.3.04.jar:/opt/hbase/lib/xmlenc-0.52.jar:/opt/hbase/lib/xom-1.2.5.jar:/opt/hbase/lib/xz-1.0.jar:/opt/hbase/lib/zookeeper-3.4.6.jar:: org.apache.tephra.TransactionServiceMain
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
2016-08-31 23:12:34,307 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-08-31 23:12:34,307 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=m9edd51-hmaster1.m9edd51
2016-08-31 23:12:34,307 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_92-internal
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:java.class.path=/opt/hbase/bin/../lib/phoenix-4.8.0-HBase-1.2-server.jar:/opt/hbase/bin/../lib/phoenix-4.8.0-HBase-1.2-client.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/opt/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase/bin/../lib/httpcore-4.4.1.jar:/opt/hbase/bin/../lib/libthrift-0.9.3.jar:/opt/hbase/bin/../lib/hbase-hadoop2-compat-1.2.2.jar:/opt/hbase/bin/../lib/hadoop-hdfs-2.5.1.jar:/opt/hbase/bin/../lib/hbase-server-1.2.2-tests.jar:/opt/hbase/bin/../lib/commons-collections-3.2.2.jar:/opt/hbase/bin/../lib/commons-compress-1.4.1.jar:/opt/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/opt/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hbase/bin/../lib/jetty-util-6.1.26.jar:/opt/hbase/bin/../lib/hbase-annotations-1.2.2.jar:/opt/hbase/bin/../lib/jetty-6.1.26.jar:/opt/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/opt/hbase/bin/../lib/disruptor-3.3.0.jar:/opt/hbase/bin/../lib/commons-cli-1.2.jar:/opt/hbase/bin/../lib/hbase-thrift-1.2.2.jar:/opt/hbase/bin/../lib/xalan-2.7.0.jar:/opt/hbase/bin/../lib/hbase-shell-1.2.2.jar:/opt/hbase/bin/../lib/jackson-xc-1.9.13.jar:/opt/hbase/bin/../lib/activation-1.1.jar:/opt/hbase/bin/../lib/hadoop-yarn-server-common-2.5.1.jar:/opt/hbase/bin/../lib/jersey-server-1.9.jar:/opt/hbase/bin/../lib/commons-net-3.1.jar:/opt/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hbase/bin/../lib/batik-ext-1.8.jar:/opt/hbase/bin/../lib/jersey-guice-1.9.jar:/opt/hbase/bin/../lib/spymemcached-2.11.6.jar:/opt/hbase/bin/../lib/bsh-core-2.0b4.jar:/opt/hbase/bin/../lib/hbase-server-1.2.2.jar:/opt/hbase/bin/../lib/hbase-rest-1.2.2.jar:/opt/hbase/bin/../lib/guava-12.0.1.jar:/opt/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/opt/hbase/bin/../lib/xz-1.0.jar:/opt/hbase/bin/../lib/jsch-0.1.42.jar:/opt/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/opt/hbase/bin/../lib/hbase-examples-1.2.2.jar:/opt/hbase/bin/../lib/hbase-annotations-1.2.2-tests.jar:/opt/hbase/bin/../lib/hbase-resource-bundle-1.2.2.jar:/opt/hbase/bin/../lib/snappy-java-1.0.4.1.jar:/opt/hbase/bin/../lib/commons-daemon-1.0.13.jar:/opt/hbase/bin/../lib/commons-el-1.0.jar:/opt/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/bin/../lib/hadoop-client-2.5.1.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-core-2.5.1.jar:/opt/hbase/bin/../lib/servlet-api-2.5.jar:/opt/hbase/bin/../lib/httpclient-4.2.5.jar:/opt/hbase/bin/../lib/joni-2.1.2.jar:/opt/hbase/bin/../lib/hadoop-yarn-client-2.5.1.jar:/opt/hbase/bin/../lib/jaxb-api-2.2.2.jar:/opt/hbase/bin/../lib/hadoop-yarn-api-2.5.1.jar:/opt/hbase/bin/../lib/commons-digester-1.8.jar:/opt/hbase/bin/../lib/jersey-core-1.9.jar:/opt/hbase/bin/../lib/hbase-procedure-1.2.2.jar:/opt/hbase/bin/../lib/slf4j-api-1.7.7.jar:/opt/hbase/bin/../lib/commons-logging-1.2.jar:/opt/hbase/bin/../lib/hbase-it-1.2.2.jar:/opt/hbase/bin/../lib/guice-3.0.jar:/opt/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase/bin/../lib/api-util-1.0.0-M20.jar:/opt/hbase/bin/../lib/asm-3.1.jar:/opt/hbase/bin/../lib/hbase-hadoop-compat-1.2.2.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-common-2.5.1.jar:/opt/hbase/bin/../lib/hbase-common-1.2.2.jar:/opt/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/opt/hbase/bin/../lib/nekohtml-1.9.16.jar:/opt/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/opt/hbase/bin/../lib/hbase-protocol-1.2.2.jar:/opt/hbase/bin/../lib/esapi-2.1.0.1.jar:/opt/hbase/bin/../lib/junit-4.12.jar:/opt/hbase/bin/../lib/commons-fileupload-1.3.1.jar:/opt/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/opt/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/bin/../lib/hbase-it-1.2.2-tests.jar:/opt/hbase/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/hbase/bin/../lib/xom-1.2.5.jar:/opt/hbase/bin/../lib/avro-1.7.4.jar:/opt/hbase/bin/../lib/hadoop-common-2.5.1.jar:/opt/hbase/bin/../lib/leveldbjni-all-1.8.jar:/opt/hbase/bin/../lib/commons-io-2.4.jar:/opt/hbase/bin/../lib/batik-css-1.8.jar:/opt/hbase/bin/../lib/commons-codec-1.9.jar:/opt/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/bin/../lib/hadoop-yarn-common-2.5.1.jar:/opt/hbase/bin/../lib/javax.inject-1.jar:/opt/hbase/bin/../lib/commons-httpclient-3.1.jar:/opt/hbase/bin/../lib/commons-configuration-1.6.jar:/opt/hbase/bin/../lib/zookeeper-3.4.6.jar:/opt/hbase/bin/../lib/jets3t-0.9.0.jar:/opt/hbase/bin/../lib/guice-servlet-3.0.jar:/opt/hbase/bin/../lib/jersey-client-1.9.jar:/opt/hbase/bin/../lib/hbase-common-1.2.2-tests.jar:/opt/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/opt/hbase/bin/../lib/hadoop-mapreduce-client-app-2.5.1.jar:/opt/hbase/bin/../lib/aopalliance-1.0.jar:/opt/hbase/bin/../lib/antisamy-1.5.3.jar:/opt/hbase/bin/../lib/jsr305-1.3.9.jar:/opt/hbase/bin/../lib/metrics-core-2.2.0.jar:/opt/hbase/bin/../lib/xml-apis-1.3.03.jar:/opt/hbase/bin/../lib/protobuf-java-2.5.0.jar:/opt/hbase/bin/../lib/commons-math-2.2.jar:/opt/hbase/bin/../lib/jettison-1.3.3.jar:/opt/hbase/bin/../lib/netty-all-4.0.23.Final.jar:/opt/hbase/bin/../lib/paranamer-2.3.jar:/opt/hbase/bin/../lib/xml-apis-ext-1.3.04.jar:/opt/hbase/bin/../lib/log4j-1.2.17.jar:/opt/hbase/bin/../lib/jersey-json-1.9.jar:/opt/hbase/bin/../lib/jruby-complete-1.6.8.jar:/opt/hbase/bin/../lib/hbase-prefix-tree-1.2.2.jar:/opt/hbase/bin/../lib/commons-math3-3.1.1.jar:/opt/hbase/bin/../lib/jcodings-1.0.8.jar:/opt/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/opt/hbase/bin/../lib/commons-lang-2.6.jar:/opt/hbase/bin/../lib/hadoop-annotations-2.5.1.jar:/opt/hbase/bin/../lib/commons-beanutils-core-1.8.3.jar:/opt/hbase/bin/../lib/hbase-client-1.2.2.jar:/opt/hbase/bin/../lib/hadoop-auth-2.5.1.jar:/opt/hbase/bin/../lib/batik-util-1.8.jar:/opt/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/opt/hbase/bin/../lib/hbase-external-blockcache-1.2.2.jar:/opt/hbase/bin/../conf/:/opt/hbase/phoenix-client/target/*:/opt/hbase/conf:/usr/lib/jvm/java-1.8-openjdk/lib/tools.jar:/opt/hbase:/opt/hbase/lib/activation-1.1.jar:/opt/hbase/lib/antisamy-1.5.3.jar:/opt/hbase/lib/aopalliance-1.0.jar:/opt/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hbase/lib/api-asn1-api-1.0.0-M20.jar:/opt/hbase/lib/api-util-1.0.0-M20.jar:/opt/hbase/lib/asm-3.1.jar:/opt/hbase/lib/avro-1.7.4.jar:/opt/hbase/lib/batik-css-1.8.jar:/opt/hbase/lib/batik-ext-1.8.jar:/opt/hbase/lib/batik-util-1.8.jar:/opt/hbase/lib/bsh-core-2.0b4.jar:/opt/hbase/lib/commons-beanutils-1.7.0.jar:/opt/hbase/lib/commons-beanutils-core-1.8.3.jar:/opt/hbase/lib/commons-cli-1.2.jar:/opt/hbase/lib/commons-codec-1.9.jar:/opt/hbase/lib/commons-collections-3.2.2.jar:/opt/hbase/lib/commons-compress-1.4.1.jar:/opt/hbase/lib/commons-configuration-1.6.jar:/opt/hbase/lib/commons-daemon-1.0.13.jar:/opt/hbase/lib/commons-digester-1.8.jar:/opt/hbase/lib/commons-el-1.0.jar:/opt/hbase/lib/commons-fileupload-1.3.1.jar:/opt/hbase/lib/commons-httpclient-3.1.jar:/opt/hbase/lib/commons-io-2.4.jar:/opt/hbase/lib/commons-lang-2.6.jar:/opt/hbase/lib/commons-logging-1.2.jar:/opt/hbase/lib/commons-math-2.2.jar:/opt/hbase/lib/commons-math3-3.1.1.jar:/opt/hbase/lib/commons-net-3.1.jar:/opt/hbase/lib/disruptor-3.3.0.jar:/opt/hbase/lib/esapi-2.1.0.1.jar:/opt/hbase/lib/findbugs-annotations-1.3.9-1.jar:/opt/hbase/lib/guava-12.0.1.jar:/opt/hbase/lib/guice-3.0.jar:/opt/hbase/lib/guice-servlet-3.0.jar:/opt/hbase/lib/hadoop-annotations-2.5.1.jar:/opt/hbase/lib/hadoop-auth-2.5.1.jar:/opt/hbase/lib/hadoop-client-2.5.1.jar:/opt/hbase/lib/hadoop-common-2.5.1.jar:/opt/hbase/lib/hadoop-hdfs-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/opt/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-api-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-client-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-common-2.5.1.jar:/opt/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/opt/hbase/lib/hbase-annotations-1.2.2-tests.jar:/opt/hbase/lib/hbase-annotations-1.2.2.jar:/opt/hbase/lib/hbase-client-1.2.2.jar:/opt/hbase/lib/hbase-common-1.2.2-tests.jar:/opt/hbase/lib/hbase-common-1.2.2.jar:/opt/hbase/lib/hbase-examples-1.2.2.jar:/opt/hbase/lib/hbase-external-blockcache-1.2.2.jar:/opt/hbase/lib/hbase-hadoop-compat-1.2.2.jar:/opt/hbase/lib/hbase-hadoop2-compat-1.2.2.jar:/opt/hbase/lib/hbase-it-1.2.2-tests.jar:/opt/hbase/lib/hbase-it-1.2.2.jar:/opt/hbase/lib/hbase-prefix-tree-1.2.2.jar:/opt/hbase/lib/hbase-procedure-1.2.2.jar:/opt/hbase/lib/hbase-protocol-1.2.2.jar:/opt/hbase/lib/hbase-resource-bundle-1.2.2.jar:/opt/hbase/lib/hbase-rest-1.2.2.jar:/opt/hbase/lib/hbase-server-1.2.2-tests.jar:/opt/hbase/lib/hbase-server-1.2.2.jar:/opt/hbase/lib/hbase-shell-1.2.2.jar:/opt/hbase/lib/hbase-thrift-1.2.2.jar:/opt/hbase/lib/htrace-core-3.1.0-incubating.jar:/opt/hbase/lib/httpclient-4.2.5.jar:/opt/hbase/lib/httpcore-4.4.1.jar:/opt/hbase/lib/jackson-core-asl-1.9.13.jar:/opt/hbase/lib/jackson-jaxrs-1.9.13.jar:/opt/hbase/lib/jackson-mapper-asl-1.9.13.jar:/opt/hbase/lib/jackson-xc-1.9.13.jar:/opt/hbase/lib/jamon-runtime-2.4.1.jar:/opt/hbase/lib/jasper-compiler-5.5.23.jar:/opt/hbase/lib/jasper-runtime-5.5.23.jar:/opt/hbase/lib/java-xmlbuilder-0.4.jar:/opt/hbase/lib/javax.inject-1.jar:/opt/hbase/lib/jaxb-api-2.2.2.jar:/opt/hbase/lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/lib/jcodings-1.0.8.jar:/opt/hbase/lib/jersey-client-1.9.jar:/opt/hbase/lib/jersey-core-1.9.jar:/opt/hbase/lib/jersey-guice-1.9.jar:/opt/hbase/lib/jersey-json-1.9.jar:/opt/hbase/lib/jersey-server-1.9.jar:/opt/hbase/lib/jets3t-0.9.0.jar:/opt/hbase/lib/jettison-1.3.3.jar:/opt/hbase/lib/jetty-6.1.26.jar:/opt/hbase/lib/jetty-sslengine-6.1.26.jar:/opt/hbase/lib/jetty-util-6.1.26.jar:/opt/hbase/lib/joni-2.1.2.jar:/opt/hbase/lib/jruby-complete-1.6.8.jar:/opt/hbase/lib/jsch-0.1.42.jar:/opt/hbase/lib/jsp-2.1-6.1.14.jar:/opt/hbase/lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/lib/jsr305-1.3.9.jar:/opt/hbase/lib/junit-4.12.jar:/opt/hbase/lib/leveldbjni-all-1.8.jar:/opt/hbase/lib/libthrift-0.9.3.jar:/opt/hbase/lib/log4j-1.2.17.jar:/opt/hbase/lib/metrics-core-2.2.0.jar:/opt/hbase/lib/nekohtml-1.9.16.jar:/opt/hbase/lib/netty-all-4.0.23.Final.jar:/opt/hbase/lib/paranamer-2.3.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-client.jar:/opt/hbase/lib/phoenix-4.8.0-HBase-1.2-server.jar:/opt/hbase/lib/protobuf-java-2.5.0.jar:/opt/hbase/lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/lib/servlet-api-2.5.jar:/opt/hbase/lib/slf4j-api-1.7.7.jar:/opt/hbase/lib/slf4j-log4j12-1.7.5.jar:/opt/hbase/lib/snappy-java-1.0.4.1.jar:/opt/hbase/lib/spymemcached-2.11.6.jar:/opt/hbase/lib/xalan-2.7.0.jar:/opt/hbase/lib/xml-apis-1.3.03.jar:/opt/hbase/lib/xml-apis-ext-1.3.04.jar:/opt/hbase/lib/xmlenc-0.52.jar:/opt/hbase/lib/xom-1.2.5.jar:/opt/hbase/lib/xz-1.0.jar:/opt/hbase/lib/zookeeper-3.4.6.jar::
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=4.4.0-36-generic
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=hadoop
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/opt/hbase
2016-08-31 23:12:34,308 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/hbase
2016-08-31 23:12:34,309 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=m9edd51-zookeeper.m9edd51 sessionTimeout=90000 watcher=org.apache.tephra.zookeeper.TephraZKClientService$5@45c7e403
2016-08-31 23:12:34,335 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Opening socket connection to server m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-31 23:12:34,355 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Socket connection established to m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181, initiating session
2016-08-31 23:12:34,364 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Session establishment complete on server m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181, sessionid = 0x156e2dfb4500005, negotiated timeout = 40000
2016-08-31 23:12:34,385 INFO  [main] inmemory.InMemoryTransactionService: Configuring TransactionService, address: 0.0.0.0, port: 15165, threads: 20, io threads: 2, max read buffer (bytes): 16777216
2016-08-31 23:12:34,386 INFO  [main] tephra.TransactionServiceMain: Starting TransactionServiceMain
2016-08-31 23:12:34,395 INFO  [main] zookeeper.LeaderElection: Start leader election on m9edd51-zookeeper.m9edd51/tx.service/leader with guid 18d66d85-16a7-4d7b-ae90-794d46bb7088
2016-08-31 23:12:34,445 INFO  [leader-election-tx.service-leader] metrics.DefaultMetricsCollector: Configured metrics report to emit every 60 seconds
2016-08-31 23:12:34,590 INFO  [ThriftRPCServer] tephra.TransactionManager: Starting transaction manager.
2016-08-31 23:12:34,609 INFO  [DefaultMetricsCollector STARTING] metrics.DefaultMetricsCollector: Started metrics reporter
2016-08-31 23:12:34,686 WARN  [HDFSTransactionStateStorage STARTING] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-08-31 23:12:35,380 INFO  [HDFSTransactionStateStorage STARTING] persist.HDFSTransactionStateStorage: Using snapshot dir /tmp/tephra/snapshots
2016-08-31 23:12:35,412 INFO  [ThriftRPCServer] persist.HDFSTransactionStateStorage: Creating snapshot dir at /tmp/tephra/snapshots
2016-08-31 23:12:35,444 INFO  [ThriftRPCServer] persist.HDFSTransactionStateStorage: No snapshot files found in /tmp/tephra/snapshots
2016-08-31 23:12:35,446 INFO  [ThriftRPCServer] tephra.TransactionManager: Starting periodic timed-out transaction cleanup every 10 seconds with default timeout of 60 seconds.
2016-08-31 23:12:35,448 INFO  [ThriftRPCServer] tephra.TransactionManager: Starting periodic snapshot thread, frequency = 300 seconds, location = /tmp/tephra/snapshots
2016-08-31 23:12:35,448 INFO  [ThriftRPCServer] tephra.TransactionManager: Starting periodic Metrics Emitter thread, frequency = 10000
2016-08-31 23:12:35,457 INFO  [ThriftRPCServer] rpc.ThriftRPCServer: Starting RPC server for TTransactionServer
2016-08-31 23:12:35,457 INFO  [ThriftRPCServer] rpc.ThriftRPCServer: Running RPC server for TTransactionServer
2016-08-31 23:12:35,457 INFO  [ThriftRPCServer] server.TThreadedSelectorServerWithFix: Starting TThreadedSelectorServerWithFix
2016-08-31 23:12:35,541 INFO  [leader-election-tx.service-leader] distributed.TransactionService: Transaction Thrift Service started successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165
2016-08-31 23:13:34,595 INFO  [metrics-logger-reporter-1-thread-1] tephra-metrics: type=GAUGE, name=committed.size, value=0
2016-08-31 23:13:34,595 INFO  [metrics-logger-reporter-1-thread-1] tephra-metrics: type=GAUGE, name=committing.size, value=0
2016-08-31 23:13:34,595 INFO  [metrics-logger-reporter-1-thread-1] tephra-metrics: type=GAUGE, name=inprogress.size, value=0
2016-08-31 23:13:34,595 INFO  [metrics-logger-reporter-1-thread-1] tephra-metrics: type=GAUGE, name=invalid.size, value=0
2016-08-31 23:13:58,897 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x156e2dfb4500005, likely server has closed socket, closing socket connection and attempting reconnect
2016-08-31 23:13:58,999 INFO  [leader-election-tx.service-leader] zookeeper.LeaderElection: Disconnected from ZK: m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-08-31 23:13:58,999 INFO  [leader-election-tx.service-leader] zookeeper.LeaderElection: Stepping down as leader due to disconnect: m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-08-31 23:13:58,999 INFO  [leader-election-tx.service-leader] rpc.ThriftRPCServer: Request to stop RPC server for TTransactionServer
2016-08-31 23:13:59,000 INFO  [ThriftRPCServer] rpc.ThriftRPCServer: Done running RPC server for TTransactionServer
2016-08-31 23:13:59,003 INFO  [ThriftRPCServer] tephra.TransactionManager: Shutting down gracefully...
2016-08-31 23:13:59,003 INFO  [tx-clean-timeout] tephra.TransactionManager: Exiting thread tx-clean-timeout
2016-08-31 23:13:59,003 INFO  [tx-metrics] tephra.TransactionManager: Exiting thread tx-metrics
2016-08-31 23:13:59,003 INFO  [tx-snapshot] tephra.TransactionManager: Writing final snapshot prior to shutdown
2016-08-31 23:13:59,074 INFO  [tx-snapshot] tephra.TransactionManager: Exiting thread tx-snapshot
2016-08-31 23:13:59,178 INFO  [DefaultMetricsCollector STOPPING] metrics.DefaultMetricsCollector: Stopped metrics reporter
2016-08-31 23:13:59,178 INFO  [ThriftRPCServer] tephra.TransactionManager: Took 174.0 ms to stop
2016-08-31 23:13:59,178 INFO  [ThriftRPCServer] rpc.ThriftRPCServer: RPC server for TTransactionServer stopped.
2016-08-31 23:13:59,632 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Opening socket connection to server m9edd51-zookeeper.m9edd51/172.18.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-31 23:13:59,633 WARN  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Session 0x156e2dfb4500005 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-31 23:14:01,073 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Opening socket connection to server m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181. Will not attempt to authenticate using SASL (unknown error)
2016-08-31 23:14:01,073 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Socket connection established to m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181, initiating session
2016-08-31 23:14:01,075 INFO  [main-SendThread(m9edd51-zookeeper.m9edd51:2181)] zookeeper.ClientCnxn: Session establishment complete on server m9edd51-zookeeper.m9edd51/fe80:0:0:0:0:242:ac12:2:2181, sessionid = 0x156e2dfb4500005, negotiated timeout = 40000
2016-08-31 23:14:01,080 INFO  [leader-election-tx.service-leader] zookeeper.LeaderElection: Connected to ZK, running election: m9edd51-zookeeper.m9edd51 for /tx.service/leader
2016-08-31 23:14:01,115 INFO  [ThriftRPCServer] rpc.ThriftRPCServer: Starting RPC server for TTransactionServer
2016-08-31 23:14:01,115 INFO  [ThriftRPCServer] rpc.ThriftRPCServer: Running RPC server for TTransactionServer
2016-08-31 23:14:01,116 INFO  [ThriftRPCServer] server.TThreadedSelectorServerWithFix: Starting TThreadedSelectorServerWithFix
2016-08-31 23:14:01,122 INFO  [leader-election-tx.service-leader] distributed.TransactionService: Transaction Thrift Service started successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165
2016-08-31 23:18:05,517 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
        at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
        at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
        at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
        at com.sun.proxy.$Proxy17.startShort(Unknown Source)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
        at org.apache.thrift.server.Invocation.run(Invocation.java:18)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-08-31 23:18:05,625 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
        at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
        at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
        at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
        at com.sun.proxy.$Proxy17.startShort(Unknown Source)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
        at org.apache.thrift.server.Invocation.run(Invocation.java:18)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-08-31 23:18:06,032 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
        at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
        at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
        at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
        at com.sun.proxy.$Proxy17.startShort(Unknown Source)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
        at org.apache.thrift.server.Invocation.run(Invocation.java:18)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-08-31 23:18:07,635 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
        at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
        at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
        at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
        at com.sun.proxy.$Proxy17.startShort(Unknown Source)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
        at org.apache.thrift.server.Invocation.run(Invocation.java:18)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-08-31 23:18:14,038 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
        at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
        at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
        at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
        at com.sun.proxy.$Proxy17.startShort(Unknown Source)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
        at org.apache.thrift.server.Invocation.run(Invocation.java:18)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-08-31 23:18:39,640 ERROR [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error processing startShort
java.lang.IllegalStateException: Transaction Manager is not running.
        at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
        at org.apache.tephra.TransactionManager.ensureAvailable(TransactionManager.java:709)
        at org.apache.tephra.TransactionManager.startTx(TransactionManager.java:768)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:728)
        at org.apache.tephra.TransactionManager.startShort(TransactionManager.java:716)
        at org.apache.tephra.distributed.TransactionServiceThriftHandler.startShort(TransactionServiceThriftHandler.java:71)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServer.java:261)
        at com.sun.proxy.$Proxy17.startShort(Unknown Source)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:974)
        at org.apache.tephra.distributed.thrift.TTransactionServer$Processor$startShort.getResult(TTransactionServer.java:959)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
        at org.apache.thrift.server.Invocation.run(Invocation.java:18)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)