[HDDS-2443] Python client/interface for Ozone

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Ozone Client
• Target Version/s: 1.5.0
• Assignee: Unassigned Assign to me
• Reporter: Li Cheng
• Created: 08/Nov/19 02:31
• Updated: 16/Dec/23 23:43

Description

This Jira will be used to track development for python client/interface of Ozone.

Original ideas: item#25 in https://cwiki.apache.org/confluence/display/HADOOP/Ozone+project+ideas+for+new+contributors

Ozone Client(Python) for Data Science Notebook such as Jupyter.

1. Size: Large
2. PyArrow: https://pypi.org/project/pyarrow/
3. Python -> libhdfs HDFS JNI library (HDFS, S3,...) -> Java client API Impala uses  libhdfs

Path to try:

1. s3 interface: Ozone s3 gateway(already supported) + AWS python client (boto3)
2. python native RPC
3. pyarrow + libhdfs, which use the Java client under the hood.
4. python + C interface of go / rust ozone library. I created POC go / rust clients earlier which can be improved if the libhdfs interface is not good enough. [By Marton Elek]

Sub-Tasks

• Make the pyarrow instruction under Ozone Recipes: Open
• Track pyarrow examples: Open
• Track pyarrow performance measurement: Open