Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
v4.0.0
-
None
-
None
-
hive-2.3.9
kylin-4.0.0
spark-3.1.2-bin-hadoop3.2
zookeeper-3.7.0
Description
when I reduce hadoop version to hadoop-3.2.2 , run ./kylin.sh start
the command export :
[hadoop@node3 logs]$ ../bin/kylin.sh start
Retrieving hadoop conf dir...
...................................................[PASS]
KYLIN_HOME is set to /home/hadoop/kylin-4.0.0
Checking hive
...................................................[PASS]
Checking hadoop shell
...................................................[PASS]
Checking hdfs working dir
...................................................[PASS]Checking environment finished successfully. To check again, run 'bin/check-env.sh' manually.
Retrieving hive dependency...
Something wrong with Hive CLI or Beeline, please execute Hive CLI or Beeline CLI in terminal to find the root cause.
Then I set `set -exv` in the $KYLIN_HOME/bin/find-hive-dependency.sh
And the script print:
[hadoop@node3 bin]$ ./kylin.sh start Retrieving hadoop conf dir... ...................................................[PASS] KYLIN_HOME is set to /home/hadoop/kylin-4.0.0 Checking hive ...................................................[PASS] Checking hadoop shell ...................................................[PASS] Checking hdfs working dir ...................................................[PASS]Checking environment finished successfully. To check again, run 'bin/check-env.sh' manually. source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd -P)/../"}/bin/header.sh ++ source /home/hadoop/kylin-4.0.0/bin/header.sh #!/bin/bash# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ## source meverbose=${verbose:-""} +++ verbose=while getopts ":v" opt; do case $opt in v) echo "Turn on verbose mode." >&2 verbose=true ;; \?) echo "Invalid option: -$OPTARG" >&2 ;; esac done +++ getopts :v optif [[ "$dir" == "" ]] then dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set KYLIN_HOME with consideration for multiple instances that are on the same node KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"} export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd` dir="$KYLIN_HOME/bin" function quit { echo "$@" exit 1 } function verbose { if [[ -n "$verbose" ]]; then echo "$@" fi } function setColor() { echo -e "\033[$1m$2\033[0m" } # set JAVA if [[ "${JAVA}" == "" ]]; then if [[ -z "$JAVA_HOME" ]]; then JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print $2}'` if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java)))) else quit "Java 1.8 or above is required." fi [[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME" export JAVA_HOME fi export JAVA=$JAVA_HOME/bin/java [[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set JAVA_HOME correctly." verbose "java is ${JAVA}" fi fi +++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]## ${dir} assigned to $KYLIN_HOME/bin in header.sh source ${dir}/load-hive-conf.sh ++ source /home/hadoop/kylin-4.0.0/bin/load-hive-conf.sh #!/bin/bash# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. #source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd -P)/../"}/bin/header.sh +++ source /home/hadoop/kylin-4.0.0/bin/header.sh #!/bin/bash# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ## source meverbose=${verbose:-""} ++++ verbose=while getopts ":v" opt; do case $opt in v) echo "Turn on verbose mode." >&2 verbose=true ;; \?) echo "Invalid option: -$OPTARG" >&2 ;; esac done ++++ getopts :v optif [[ "$dir" == "" ]] then dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set KYLIN_HOME with consideration for multiple instances that are on the same node KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"} export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd` dir="$KYLIN_HOME/bin" function quit { echo "$@" exit 1 } function verbose { if [[ -n "$verbose" ]]; then echo "$@" fi } function setColor() { echo -e "\033[$1m$2\033[0m" } # set JAVA if [[ "${JAVA}" == "" ]]; then if [[ -z "$JAVA_HOME" ]]; then JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print $2}'` if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java)))) else quit "Java 1.8 or above is required." fi [[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME" export JAVA_HOME fi export JAVA=$JAVA_HOME/bin/java [[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set JAVA_HOME correctly." verbose "java is ${JAVA}" fi fi ++++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]# source me hive_conf_dir="${KYLIN_HOME}/conf/kylin_hive_conf.xml" +++ hive_conf_dir=/home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xmlnames=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<name>\(.*\)</name>|\1|p' | sed 's/ \+//g'`) +++ names=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<name>\(.*\)</name>|\1|p' | sed 's/ \+//g'`) sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<name>\(.*\)</name>|\1|p' | sed 's/ \+//g' ++++ sed -n 's|<name>\(.*\)</name>|\1|p' ++++ sed -n '/<!--/,/-->/!p' /home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xml ++++ sed 's/ \+//g' values=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'`) +++ values=(`sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g'`) sed -n '/<!--/,/-->/!p' ${hive_conf_dir} | sed -n 's|<value>\(.*\)</value>|\1|p' | sed 's/ \+//g' ++++ sed -n 's|<value>\(.*\)</value>|\1|p' ++++ sed -n '/<!--/,/-->/!p' /home/hadoop/kylin-4.0.0/conf/kylin_hive_conf.xml ++++ sed 's/ \+//g'len_names=${#names[@]} +++ len_names=9 len_values=${#values[@]} +++ len_values=9[[ $len_names == $len_values ]] || quit "Current hive conf file: ${hive_conf_dir} has inconsistent key value pairs." +++ [[ 9 == 9 ]]for((i=0;i<$len_names;i++)) do hive_conf_properties=${hive_conf_properties}" --hiveconf ${names[$i]}=${values[$i]} " done +++ (( i=0 )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true --hiveconf hive.auto.convert.join.noconditionaltask=true ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf mapreduce.job.split.metainfo.maxsize=-1 ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true --hiveconf hive.merge.mapfiles=false ' +++ (( i++ )) +++ (( i<9 )) +++ hive_conf_properties=' --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true --hiveconf hive.merge.mapfiles=false --hiveconf hive.merge.mapredfiles=false ' +++ (( i++ )) +++ (( i<9 ))echo Retrieving hive dependency... ++ echo Retrieving hive dependency... Retrieving hive dependency...client_mode=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client` bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.client +++ bash /home/hadoop/kylin-4.0.0/bin/get-properties.sh kylin.source.hive.client ++ client_mode= hive_env= ++ hive_env=if [ "${client_mode}" == "beeline" ] then beeline_shell=`$KYLIN_HOME/bin/get-properties.sh kylin.source.hive.beeline-shell` beeline_params=`bash ${KYLIN_HOME}/bin/get-properties.sh kylin.source.hive.beeline-params` hive_env=`${beeline_shell} ${hive_conf_properties} ${beeline_params} --outputformat=dsv -e "set;" 2>&1 | grep --text 'env:CLASSPATH' ` else source ${dir}/check-hive-usability.sh hive_env=`hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH'` fi ++ '[' '' == beeline ']' ++ source /home/hadoop/kylin-4.0.0/bin/check-hive-usability.sh #!/bin/bash# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. #source ${KYLIN_HOME:-"$(cd -P -- "$(dirname -- "$0")" && pwd -P)/../"}/bin/header.sh +++ source /home/hadoop/kylin-4.0.0/bin/header.sh #!/bin/bash# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ## source meverbose=${verbose:-""} ++++ verbose=while getopts ":v" opt; do case $opt in v) echo "Turn on verbose mode." >&2 verbose=true ;; \?) echo "Invalid option: -$OPTARG" >&2 ;; esac done ++++ getopts :v optif [[ "$dir" == "" ]] then dir=$(cd -P -- "$(dirname -- "$0")" && pwd -P) # set KYLIN_HOME with consideration for multiple instances that are on the same node KYLIN_HOME=${KYLIN_HOME:-"${dir}/../"} export KYLIN_HOME=`cd "$KYLIN_HOME"; pwd` dir="$KYLIN_HOME/bin" function quit { echo "$@" exit 1 } function verbose { if [[ -n "$verbose" ]]; then echo "$@" fi } function setColor() { echo -e "\033[$1m$2\033[0m" } # set JAVA if [[ "${JAVA}" == "" ]]; then if [[ -z "$JAVA_HOME" ]]; then JAVA_VERSION=`java -version 2>&1 | awk -F\" '/version/ {print $2}'` if [[ $JAVA_VERSION ]] && [[ "$JAVA_VERSION" > "1.8" ]]; then JAVA_HOME=$(dirname $(dirname $(readlink -f $(which java)))) else quit "Java 1.8 or above is required." fi [[ -z "$JAVA_HOME" ]] && quit "Please set JAVA_HOME" export JAVA_HOME fi export JAVA=$JAVA_HOME/bin/java [[ -e "${JAVA}" ]] || quit "${JAVA} does not exist. Please set JAVA_HOME correctly." verbose "java is ${JAVA}" fi fi ++++ [[ /home/hadoop/kylin-4.0.0/bin == '' ]]hive -e "select 1" > /dev/null 2>&1 & pid=$! +++ pid=25366((timeLeft = 60)) +++ (( timeLeft = 60 ))while ((timeLeft > 0)); do sleep 5 ps -p $pid > /dev/null || break ((timeLeft -= 5)) done +++ hive -e 'select 1' +++ (( timeLeft > 0 )) +++ sleep 5 +++ ps -p 25366 +++ breakif ps -p $pid > /dev/null then kill $pid sleep 5 # still alive, use kill -9 if ps -p $pid > /dev/null then kill -9 $pid fi quit "ERROR: Check hive's usability failed, please check the status of your cluster" fi +++ ps -p 25366 hive ${hive_conf_properties} -e set 2>&1 | grep 'env:CLASSPATH' +++ hive --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join=true --hiveconf hive.auto.convert.join.noconditionaltask=true --hiveconf hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.stats.autogather=true --hiveconf hive.merge.mapfiles=false --hiveconf hive.merge.mapredfiles=false -e set +++ grep env:CLASSPATH ++ hive_env=
I can't really tell what the problem is .
