[HDFS-8707] Implement an async pure c++ HDFS client

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Component/s: hdfs-client
• Assignee: James Clampffer
• Reporter: Owen O'Malley
• Created: 01/Jul/15 21:32
• Updated: 26/Oct/18 14:04
• Resolved: 26/Oct/18 12:38

Description

As part of working on the C++ ORC reader at ORC-3, we need an HDFS pure C++ client that lets us do async io to HDFS. We want to start from the code that Haohui's been working on at https://github.com/haohui/libhdfspp .

Issue Links

• is related to: HDFS-9924 [umbrella] Nonblocking HDFS Access
• is related to: HDFS-9253 Refactor tests of libhdfs into a directory

Sub-Tasks

• Integrate the build infrastructure with hdfs-client: Resolved
• Import third_party libraries into the repository: Resolved
• Use std::chrono to implement the timer in the asio library: Resolved
• Initial implementation of a Hadoop RPC v9 client: Resolved
• Use Doxygen to generate documents for libhdfspp: Resolved
• Implement the continuation library for libhdfspp: Resolved
• Implement remote block reader in libhdfspp: Resolved
• Generate Hadoop RPC stubs from protobuf definitions: Resolved
• Implement a libhdfs(3) compatible API: Resolved
• Implement FileSystem and InputStream API for libhdfspp: Resolved
• SASL support for data transfer protocol in libhdfspp: Resolved
• Implement unit tests for remote block reader in libhdfspp: Resolved
• InputStream.PositionRead() should be aware of available DNs: Resolved
• Fix compilation issues on arch linux: Resolved
• Initialize protobuf fields in RemoteBlockReaderTest: Resolved
• RPC client should fail gracefully when the connection is timed out or reset: Resolved
• Retry reads on DN failure: Resolved
• InputStreamImpl::ReadBlockContinuation stores wrong pointers of buffers: Resolved
• Suppress false positives from Valgrind on uninitialized variables in tests: Resolved
• Config file reader / options classes for libhdfs++: Resolved
• Add logging system for libdhfs++: Resolved
• Move the implementation to the hdfs-native-client module: Resolved
• Refactor libhdfs into stateful/ephemeral objects: Resolved
• Simplify embedding libhdfspp into other projects: Resolved
• Add valgrind suppression for statically initialized library objects: Resolved
• libhdfs++ should respect NN retry configuration settings: Resolved
• InputStreamImpl should hold a shared_ptr of the BlockReader: Resolved
• Implement basic NN operations: Resolved
• Implement a unix-like cat utility: Resolved
• Import RapidXML 1.13 for libhdfspp: Resolved
• libhdfspp should use sizeof(int32_t) instead of sizeof(int) when parsing data: Resolved
• Allow the location of hadoop source tree resources to be passed to CMake during a build.: Resolved
• Create a generic function to synchronize async functions and methods.: Resolved
• libhdfspp fails to compile after HDFS-9207: Resolved
• Test libhdfs++ with existing libhdfs tests: Resolved
• Get libhdfs++ gmock tests running with CI: Resolved
• Implement reads with implicit offset state in libhdfs++: Resolved
• HDFS-8707 builds are failing with protobuf directories as undef: Resolved
• Build both static and dynamic libraries for libhdfspp: Resolved
• Clean up the RAT warnings in the HDFS-8707 branch.: Resolved
• Import the optional library into libhdfs++: Resolved
• Be able to build/test libhdfspp without Maven: Resolved
• Enable valgrind for libhdfspp unit tests: Resolved
• libhdfs++ Fix memory stomp in OpenFileForRead.: Resolved
• Fix protobuf runtime warnings: Resolved
• libhdfs++: suppress warnings from third-party libraries: Resolved
• Documentation needs to be exposed: Resolved
• No header files in mvn package: Resolved
• libhdfs++ Fix valgrind failures when using more than 1 io_service worker thread.: Resolved
• libhdfs++ Enable builds with no compiler optimizations: Resolved
• Enable CI infrasructure to use libhdfs++ hdfsRead: Resolved
• libhdfs++: move lib/proto/cpp_helpers to third-party since it won't have an ASF license: Resolved
• libhdfs++ Initialize BadNodeTracker in FileSystemImpl constructor: Resolved
• libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail: Resolved
• libhdfs++ deadlocks in Filesystem::New if NN conneciton fails: Resolved
• libhdfs++: implement HDFSConfiguration class: Resolved
• libhdfs++: load configuration from files: Resolved
• libhdfs++: pull Options from default configs by default: Resolved
• libhfds++: Allow seek to EOF: Resolved
• libhdfs++ Add runtime hooks to allow a client application to add low level monitoring and tests.: Resolved
• libhdfs++: Add a mechanism to retrieve human readable error messages through the C API: Resolved
• libhdfs++: Implement builder apis from C bindings: Resolved
• libhdfs++: Add additional type-safe getters to the Configuration class: Resolved
• libhdfs++: for consistency, include files should be in hdfspp: Resolved
• libhdfs++: Support async cancellation of read operations: Resolved
• libhdfs++: Fix inconsistencies with libhdfs C API: Resolved
• libhdfs++: potential segfault after teardown: Resolved
• libhdfs++: Add appropriate catch blocks for ASIO operations that throw: Resolved
• libhdfs++: Reimplement Status object as a normal struct: Resolved
• libhdfs++: Create examples of consuming libhdfs++: Resolved
• libhdfs++: Implement simple authentication: Resolved
• libhdfs++: GetLastError not returning meaningful messages after some failures: Resolved
• libhdfs++: RPC engine will attempt to close an asio socket before it's been opened: Resolved
• libhdfs++: Client fails to pass TokenProto from LocatedBlockProto to server when reading a block: Resolved
• libhfds++: ConfigurationLoader throws parse_exception on invalid input: Resolved
• libhdfs++: EACCES not setting errno correctly: Resolved
• libhdfs++: Cancel outstanding operations without calling shutdown: Resolved
• libhfds++: C++ exceptions should never escape the C API: Resolved
• libhdfs++: Add test suite to simulate network issues: Resolved
• libhdfs++: add hooks to facilitate fault testing: Resolved
• libhdfs++: find a URI parsing library: Resolved
• libhdfs++: Implement debug allocators: Resolved
• libhdfs++: Integrate logging with the C API: Resolved
• libhdfs++: Shutdown sockets to avoid "Connection reset by peer": Resolved
• libhdfs++: Fix race conditions in RPC layer: Resolved
• libhdfs++: Datanode protocol version mismatch: Resolved
• libhdfs++: File length doesn't always count the last block if it's being written to: Resolved
• libhdfs++: hdfsConnect hangs when given bad host or port: Resolved
• libhdfs++: DatanodeConnection::Cancel should not delete the underlying socket: Resolved
• hdfs-native-client fails to build with CMake 2.8.11 or earlier: Resolved
• libhdfs++: Add SASL authentication: Resolved
• libhdfs++: Get rid of lock in RpcConnectionImpl destructor: Resolved
• libhdfs++: HA namenode support: Resolved
• libhdfs++: Implement Cyrus SASL implementation in sasl_enigne.cc: Resolved
• libhdfspp: Move NameNodeOp to a separate file: Resolved
• libhdfs++: Implement GetPathInfo and ListDirectory: Resolved
• libhdfs++: Implement GetBlockLocations: Resolved
• libhdfs++: Implement GetFsStats: Resolved
• libhdfs++: Implement snapshot operations and GetFsStats: Resolved
• libhfds++: if HA is available, authentication doesn't get parsed in configs: Resolved
• libhdfs++: make error returning mechanism consistent across all hdfs operations: Resolved
• libhdfs++: Implement mkdirs, rmdir, rename, and remove: Resolved
• libhdfs++: Implement chmod and chown: Resolved
• libhdfs++: Add connect timeouts to async_connect calls: Resolved
• libhdfs++: hdfsGetBlockLocations doesn't null terminate ip address strings: Resolved
• libhdfs++: Silence compile warnings from URI parser: Resolved
• libhdfs++: Client Name Protobuf Error: Resolved
• libhdfs++: reorder directories in src/main/libhdfspp/examples, and add C++ version of cat tool: Resolved
• libhdfs++: Implement parallel find with wildcards tool: Resolved
• libhdfs++: return explicit error when non-secured client connects to secured server: Resolved
• libhdfs++: FileSystem should have a convenience no-args ctor: Resolved
• libhdfs++: Expose an InputStream interface for the apache ORC project: Resolved
• libhdfs++: In RPC engine replace vector with deque for pending requests: Resolved
• libhdfs++: Implement recursive directory generator: Resolved
• libhdfs++: synchronize access to working_directory and bytes_read_.: Resolved
• libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, hdfs_chmod and hdfs_find.: Resolved
• libhdfs++: Fix broken logic in HA retry policy: Resolved
• libhdfs++: Implement the rest of the tools: Resolved
• libhdfs++: Public API should expose configuration parser: Resolved
• libhdfs++: rationalize ioservice interactions: Resolved
• libhdfs++: Public API headers should not depend on internal implementation: Resolved
• libhdfs++: Make log levels consistent: Resolved
• libhdfs++: Fix object lifecycle issues in the BlockReader: Resolved
• libhdfs++: Make connection to HA clusters faster: Resolved
• libhdfs++: Don't retry if there is an authentication failure: Resolved
• libhdfs++: FileSystem needs to be able to cancel pending connections: Resolved
• Expose rack id in hdfsDNInfo: Resolved
• libhdfs++: Some refactoring to better organize files: Resolved
• libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails: Resolved
• libhdfs++: Log Datanode information when reading an HDFS block: Resolved
• libhdfs++: Fix race condition in ScopedResolver: Resolved
• libhdfs++: Log Datanode read size when reading an HDFS block: Resolved
• libhdfs++: Add a build option to skip building examples, tests, and tools: Resolved
• libhdfs++: RPC connection should handle authorization error call id: Resolved
• libhdfs++: Catch exceptions thrown by runtime hooks: Resolved
• libhdfs++: SASL events should be scoped closer to usage: Resolved
• libhdfs++: Get minidfscluster tests running under valgrind: Resolved
• libhdfs++: Authentication failure when first NN of kerberized HA cluster is standby: Resolved
• libhdfs++: Docker script fails while trying to download JDK 7.: Resolved
• libhdfs++: A few portability issues: Resolved
• libhdfs++: read with offset at EOF should return 0 bytes instead of error: Resolved
• libhdfs++: Fix compilation errors and warnings when compiling with Clang: Resolved
• libhdfs++: add Clang build and tests to the CI system: Resolved
• libhdfs++: Provide workaround to support cancel on filesystem connect until HDFS-11437 is resolved: Resolved
• libhdfs++: Make sure all steps in SaslProtocol end up calling AuthComplete: Resolved
• libhdfs++: Rebase 8707 branch onto an up to date version of trunk: Resolved
• libhdfs++: Add a synchronization interface for the GSSAPI: Resolved
• libhdfs++: PROTOC_IS_COMPATIBLE check fails if protobuf library is built from source: Resolved
• libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine: Resolved