[HDFS-14478] Add libhdfs APIs for openFile

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.5
• Component/s: hdfs-client, libhdfs, native
• Labels: pull-request-available
• Assignee: Sahil Takiar
• Reporter: Sahil Takiar
• Created: 07/May/19 20:13
• Updated: 29/Sep/22 14:23
• Resolved: 24/Apr/22 16:37

Description

HADOOP-15229 added a "FileSystem builder-based openFile() API" that allows specifying configuration values for opening files (similar to HADOOP-14365).

Support for openFile will be a little tricky as it is asynchronous and FutureDataInputStreamBuilder#build returns a CompletableFuture.

At a high level, the API for openFile could look something like this:

hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
                      int bufferSize, short replication, tSize blocksize);

hdfsOpenFileBuilder *hdfsOpenFileBuilderAlloc(hdfsFS fs,
        const char *path);

hdfsOpenFileBuilder *hdfsOpenFileBuilderMust(hdfsOpenFileBuilder *builder,
        const char *key, const char *value);

hdfsOpenFileBuilder *hdfsOpenFileBuilderOpt(hdfsOpenFileBuilder *builder,
        const char *key, const char *value);

hdfsOpenFileFuture *hdfsOpenFileBuilderBuild(hdfsOpenFileBuilder *builder);

void hdfsOpenFileBuilderFree(hdfsOpenFileBuilder *builder);

hdfsFile hdfsOpenFileFutureGet(hdfsOpenFileFuture *future);

hdfsFile hdfsOpenFileFutureGetWithTimeout(hdfsOpenFileFuture *future,
        int64_t timeout, javaConcurrentTimeUnit timeUnit);

int hdfsOpenFileFutureCancel(hdfsOpenFileFuture *future,
        int mayInterruptIfRunning);

void hdfsOpenFileFutureFree(hdfsOpenFileFuture *future);

Instead of exposing all the functionality of CompleteableFuture libhdfs would just expose the functionality of Future.

Issue Links

• depends upon: HADOOP-16759 Filesystem openFile() builder to take a FileStatus param
• is depended upon by: IMPALA-8523 Migrate hdfsOpen to builder-based openFile API
• is related to: HADOOP-15229 Add FileSystem builder-based openFile() API to match createFile(); S3A to implement S3 Select through this API.
• is related to: HDFS-14564 Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable
• relates to: HADOOP-16202 Enhance openFile() for better read performance against object stores
• links to: GitHub Pull Request #955
• links to: GitHub Pull Request #4166