[IMPALA-11902] TmpFileMgrTest.TestCompressBufferManagementEncryptedRemoteUpload fails on Ozone

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: Impala 4.3.0
• Component/s: Backend
• Labels: broken-build
• Epic Color: ghx-label-7
• Assignee: Yida Wu
• Reporter: Joe McDonnell
• Created: 07/Feb/23 01:57
• Updated: 07/Feb/23 17:59

Description

In a recent run on Ozone, the  TmpFileMgrTest.TestCompressBufferManagementEncryptedRemoteUpload test failed with the following message:

[ RUN      ] TmpFileMgrTest.TestCompressBufferManagementEncryptedRemoteUpload
/data/jenkins/workspace/impala-asf-master-core-ozone/repos/Impala/be/src/runtime/tmp-file-mgr-test.cc:1315: Failure
Value of: io::DiskFileStatus::PERSISTED
  Actual: 4-byte object <01-00 00-00>
Expected: tmp_file->DiskFile()->GetFileStatus()
Which is: 4-byte object <00-00 00-00>
hdfsOpenFile(ofs://localhost:9862/impala/tmp/impala-scratch/844f2cbc90d7e11f_8c94fc058822a5af_0000000000000000_0000000000000000/impala-scratch-fc74b5be-ac85-420e-bfa1-df2550ca1120): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error:
FileNotFoundException: KEY_NOT_FOUND: Key:impala-scratch/844f2cbc90d7e11f_8c94fc058822a5af_0000000000000000_0000000000000000/impala-scratch-fc74b5be-ac85-420e-bfa1-df2550ca1120 not foundjava.io.FileNotFoundException: KEY_NOT_FOUND: Key:impala-scratch/844f2cbc90d7e11f_8c94fc058822a5af_0000000000000000_0000000000000000/impala-scratch-fc74b5be-ac85-420e-bfa1-df2550ca1120 not found
	at org.apache.hadoop.fs.ozone.BasicRootedOzoneClientAdapterImpl.readFile(BasicRootedOzoneClientAdapterImpl.java:358)
	at org.apache.hadoop.fs.ozone.BasicRootedOzoneFileSystem.open(BasicRootedOzoneFileSystem.java:211)
/data/jenkins/workspace/impala-asf-master-core-ozone/repos/Impala/be/src/runtime/tmp-file-mgr-test.cc:1328: Failure
Value of: status_.ok()
  Actual: false
Expected: true
Error: Disk I/O error on impala-ec2-centos79-m6i-4xlarge-ondemand-145c.vpc.cloudera.com:27000: Failed to open HDFS file ofs://localhost:9862/impala/tmp/impala-scratch/844f2cbc90d7e11f_8c94fc058822a5af_0000000000000000_0000000000000000/impala-scratch-fc74b5be-ac85-420e-bfa1-df2550ca1120
Error(2): No such file or directory
Root cause: FileNotFoundException: KEY_NOT_FOUND: Key:impala-scratch/844f2cbc90d7e11f_8c94fc058822a5af_0000000000000000_0000000000000000/impala-scratch-fc74b5be-ac85-420e-bfa1-df2550ca1120 not found

/data/jenkins/workspace/impala-asf-master-core-ozone/repos/Impala/be/src/runtime/tmp-file-mgr-test.cc:1329: Failure
Value of: memcmp(tmp.data(), DATA.data(), DATA.size())
  Actual: -116
Expected: 0
*** Check failure stack trace: ***
    @          0x3f6c69d  google::LogMessage::Fail()
    @          0x3f6e5d4  google::LogMessage::SendToLog()
    @          0x3f6c07c  google::LogMessage::Flush()
    @          0x3f6eaf9  google::LogMessageFatal::~LogMessageFatal()
    @          0x2197d6b  impala::TmpFileGroup::ReadAsync()
    @          0x2198f9b  impala::TmpFileGroup::Read()
    @          0x144a544  impala::TmpFileMgrTest::TestCompressBufferManagement()
    @          0x144ba57  impala::TmpFileMgrTest_TestCompressBufferManagementEncryptedRemoteUpload_Test::TestBody()
    @          0x4114a9d  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @          0x410df65  testing::Test::Run()
    @          0x410e0a8  testing::TestInfo::Run()
    @          0x410e185  testing::TestCase::Run()
    @          0x410e67c  testing::internal::UnitTestImpl::RunAllTests()
    @          0x410e723  testing::UnitTest::Run()
    @          0x10921d1  main
    @     0x7f0ccdbbf555  __libc_start_main
    @          0x1092037  (unknown)