[HDDS-6677] Not able to access the Ozone cluster running in Kuberenets through hdfs dfs -put/cat via hadoop client from external network

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.2.1
• Component/s: kubernetes, Ozone Datanode
• Assignee: Swaminathan Balachandran
• Reporter: Dileep
• Created: 01/May/22 06:52
• Updated: 21/Aug/23 16:54

Description

Ozone cluster is hosted in Kubernetes cluster with 3 datanode and 1 OM/SCM/S3G pod.

Cluster is created as per the steps mentioned in documentation and I am able to read/write files via hadoop client if its hosted with in Kubernetes cluster But I am not able to read/write the files using hadoop client which is hosted outside kuberenetes cluster. 

There is no documentation available which defines about connectivity to ozone cluster from outside network.

Getting below error. As per my initial investigation , hadoop client is trying to access directly the datanode pods but since no service has been defined with nodeport, it failed to communicate with it.

2022-04-28 18:25:06,135 WARN io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: Failed to write chunk 109611004723200005_chunk_1 into block conID: 5 locID: 109611004723200005 bcsId: 0 on the pipeline Pipeline[ Id: 2a523691-e3ee-4826-bf77-89a40a161194, Nodes: e837c443-7e06-4484-85be-8e36807030a1{ip: 10.42.4.0, host: 10.42.4.0, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}73608fb4-7dcb-426b-8e71-0c7b9bc31271{ip: 10.42.1.0, host: 10.42.1.0, ports: [REPLICATION=9886, RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859], networkLocation: /default-rack, certSerialId: null, persistedOpState: IN_SERVICE, persistedOpStateExpiryEpochSec: 0}, ReplicationConfig: RATIS/THREE, State:OPEN, leaderId:73608fb4-7dcb-426b-8e71-0c7b9bc31271, CreationTimestamp2022-04-28T18:05:45.383+05:30[Asia/Kolkata]]. The last committed block length is 0, uncommitted data length is 11 retry count 0

^C2022-04-28 18:25:25,616 WARN grpc.GrpcUtil: Timed out gracefully shutting down connection: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=41, target=10.42.4.0:9858}}.

 

 

 

Let me know the procedure to access the cluster from external network.

Issue Links

• relates to: HDDS-6438 Support Ozone to run as a service within Kubernetes