[HIVE-18091] Failing tests of itests/qtest-spark and itests/hive-unit on branch-1

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0
• Component/s: Test, Testing Infrastructure
• Assignee: Daniel Voros
• Reporter: Daniel Voros
• Created: 17/Nov/17 09:47
• Updated: 30/Nov/17 12:00
• Resolved: 30/Nov/17 11:43

Description

Seen this while looking at ptest results for HIVE-17947 but is probably an older issue.

Tests under itests/qtest-spark and itests/hive-unit fail when trying to execute the download-spark plugin with:

[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hive Integration - Unit Tests 1.3.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-it-unit ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-unit ---
[INFO] Executing tasks

main:
     [exec] + /bin/pwd
     [exec] + BASE_DIR=./target
     [exec] + HIVE_ROOT=./target/../../../
     [exec] + DOWNLOAD_DIR=./../thirdparty
     [exec] + mkdir -p ./../thirdparty
     [exec] /home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit
     [exec] + download http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz spark
     [exec] + url=http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
     [exec] + finalName=spark
     [exec] ++ basename http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
     [exec] + tarName=spark-1.5.0-bin-hadoop2-without-hive.tgz
     [exec] + rm -rf ./target/spark
     [exec] + [[ ! -f ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz ]]
     [exec] + tar -zxf ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz -C ./target
     [exec] + mv ./target/spark-1.5.0-bin-hadoop2-without-hive ./target/spark
     [exec] + cp -f ./target/../../..//data/conf/spark/log4j.properties ./target/spark/conf/
     [exec] + sed '/package /d' /data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
     [exec] sed: can't read /data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java: No such file or directory
     [exec] + javac -cp /data/hiveptest/working/maven/org/apache/hive/hive-exec/1.3.0-SNAPSHOT/hive-exec-1.3.0-SNAPSHOT.jar /tmp/UDFExampleAdd.java -d /tmp
     [exec] + jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp UDFExampleAdd.class
     [exec] /tmp/UDFExampleAdd.class : no such file or directory
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.376s
[INFO] Finished at: Mon Nov 06 22:29:39 UTC 2017
[INFO] Final Memory: 18M/241M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (download-spark) on project hive-it-unit: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part ...<exec dir="/home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit" executable="bash" failonerror="true">... @ 4:141 in /home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit/target/antrun/build-main.xml

mvn antrun:run@download-spark passes when run locally, so I guess it might be an issue with the way wer're executing ptests.

Full list of classes with the same error:

TestAcidOnTez
TestAdminUser
TestAuthorizationPreEventListener
TestAuthzApiEmbedAuthorizerInEmbed
TestAuthzApiEmbedAuthorizerInRemote
TestBeeLineWithArgs
TestCLIAuthzSessionContext
TestClearDanglingScratchDir
TestClientSideAuthorizationProvider
TestCompactor
TestCreateUdfEntities
TestCustomAuthentication
TestDBTokenStore
TestDDLWithRemoteMetastoreSecondNamenode
TestDynamicSerDe
TestEmbeddedHiveMetaStore
TestEmbeddedThriftBinaryCLIService
TestFilterHooks
TestFolderPermissions
TestHS2AuthzContext
TestHS2AuthzSessionContext
TestHS2ClearDanglingScratchDir
TestHS2ImpersonationWithRemoteMS
TestHiveAuthorizerCheckInvocation
TestHiveAuthorizerShowFilters
TestHiveHistory
TestHiveMetaStoreTxns
TestHiveMetaStoreWithEnvironmentContext
TestHiveMetaTool
TestHiveServer2
TestHiveServer2SessionTimeout
TestHiveSessionImpl
TestHs2Hooks
TestJdbcDriver2
TestJdbcMetadataApiAuth
TestJdbcWithLocalClusterSpark
TestJdbcWithMiniHS2
TestJdbcWithMiniMr
TestJdbcWithSQLAuthUDFBlacklist
TestJdbcWithSQLAuthorization
TestLocationQueries
TestMTQueries
TestMarkPartition
TestMarkPartitionRemote
TestMetaStoreAuthorization
TestMetaStoreConnectionUrlHook
TestMetaStoreEndFunctionListener
TestMetaStoreEventListener
TestMetaStoreEventListenerOnlyOnCommit
TestMetaStoreInitListener
TestMetaStoreListenersError
TestMetaStoreMetrics
TestMetaStoreUtils
TestMetastoreAuthorizationProvider
TestMetastoreVersion
TestMiniHS2
TestMiniSparkOnYarnCliDriver
TestMultiAuthorizationPreEventListener
TestMultiSessionsHS2WithLocalClusterSpark
TestNoSaslAuth
TestOperationLoggingAPIWithMr
TestOperationLoggingAPIWithTez
TestPartitionNameWhitelistValidation
TestPasswordWithConfig
TestRemoteHiveMetaStore
TestRemoteHiveMetaStoreIpAddress
TestRemoteUGIHiveMetaStoreIpAddress
TestRetryingHMSHandler
TestSSL
TestSchemaTool
TestSemanticAnalyzerHookLoading
TestSerdeWithFieldComments
TestServerSpecificConfig
TestServiceDiscovery
TestSetUGIOnBothClientServer
TestSetUGIOnOnlyClient
TestSetUGIOnOnlyServer
TestSparkCliDriver
TestSparkNegativeCliDriver
TestStorageBasedClientSideAuthorizationProvider
TestStorageBasedMetastoreAuthorizationDrops
TestStorageBasedMetastoreAuthorizationProvider
TestStorageBasedMetastoreAuthorizationReads
TestThriftBinaryCLIService
TestThriftHttpCLIService
TestZooKeeperTokenStore

These are all the test classes of the two modules with the exception of TestHiveMetastore.