[ORC-400] Support Hadoop3 in C++ lib

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 12/Sep/18 00:50
• Updated: 12/Sep/18 12:32

Description

After ORC-17, we now depends on hadoop c++ lib. When linking liborc into Impala-3.x which depends on hadoop3, we found the following errors:

../../../toolchain/openldap-2.4.25/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_nerr' is deprecated; use `strerror' or `strerror_r' instead
../../../toolchain/openldap-2.4.25/lib/libldap.a(os-ip.o):os-ip.c:function ldap_int_poll: warning: `sys_errlist' is deprecated; use `strerror' or `strerror_r' instead
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::read(void*, unsigned long, unsigned long): error: undefined reference to 'hdfs::Status::ToString() const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::URI()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::parse_from_string(std::string const&)'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::ConfigParser::ConfigParser()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::ConfigParser::LoadDefaultResources()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::ConfigParser::ValidateResources() const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::Options::Options()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::ConfigParser::get_options(hdfs::Options&) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::IoService::New()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::FileSystem::New(hdfs::IoService*&, std::string const&, hdfs::Options const&)'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_host(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_path(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::StatInfo::StatInfo()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_path(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::ConfigParser::~ConfigParser()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::has_port() const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_port() const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_host(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_host(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::Status::ToString() const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::str(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::Status::ToString() const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::ConfigParser::~ConfigParser()'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::Status::ToString() const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_path(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_host(bool) const'
../../../toolchain/orc-1.5.2-p1/lib/liborc.a(OrcHdfsFile.cc.o):OrcHdfsFile.cc:function orc::HdfsFileInputStream::HdfsFileInputStream(std::string): error: undefined reference to 'hdfs::URI::get_path(bool) const'
collect2: error: ld returned 1 exit status

The C++ liborc is built on hadoop2. We need to support hadoop3 as well.

Issue Links

• is related to: IMPALA-6772 Enable test_scanners_fuzz for ORC format