[ARROW-11062] [Java] When writing to flight stream, Spark's mapPartitions is not working

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 2.0.0
• Component/s: Java
• External issue URL: https://github.com/apache/arrow/issues/26976
• Assignee: Unassigned
• Reporter: Ravi Shankar
• Created: 29/Dec/20 09:34
• Updated: 11/Jan/23 08:17
• Resolved: 04/May/22 14:54

Description

Hi,

I have the following method:

 

val outRDD = myRdd.mapPartitions { it =>

val  l = Location.forGrpcInsecure("10.0.0.113", 12233);

val allocator = it.allocator.newChildAllocator("SparkFlightConnector", 0, Long.MaxValue)

       val client = FlightClient.builder(allocator, l).build();

       val desc = FlightDescriptor.path("wonderful")

       val stream = client.startPut(desc,it.root, new AsyncPutListener)

       it.foreach

{ root =>         // doPut on the populated VectorSchemaRoot         stream.putNext()       }

      stream.completed()

      // Need to call this, or exceptions from the server get swallowed

      stream.getResult

 

 //   println(it.next().contentToTSVString())

      client.close()

 

      Iterator.empty

      }.count

 

Following is the error:

 

Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V

at io.grpc.Metadata$Key.validateName(Metadata.java:742)

at io.grpc.Metadata$Key.<init>(Metadata.java:750)

at io.grpc.Metadata$Key.<init>(Metadata.java:668)

at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:959)

at io.grpc.Metadata$AsciiKey.<init>(Metadata.java:954)

at io.grpc.Metadata$Key.of(Metadata.java:705)

at io.grpc.Metadata$Key.of(Metadata.java:701)

at io.grpc.internal.GrpcUtil.<clinit>(GrpcUtil.java:80)

 

When I googles, its sayong some guava related jar issue. I did maven dependency tree and did not find anything wrong. Please help.

 

Best,

Ravion