Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
2.3.0
-
None
-
None
-
Spark version 2.3
JDK 1.8
MongoDB Version - 3.4 and 3.6Sample Data -
{"_id":"5b0d31f892549e10b61d962a","RSEG_MANDT":"800","RSEG_EBELN":"4500017749","RSEG_EBELP":"00020","RSEG_BELNR":"1000000001","RSEG_BUZEI":"000002","RSEG_GJAHR":"2013","RBKP_BUDAT":"2013-10-04","RSEG_MENGE":
{"$numberDecimal":"30.000"},"RSEG_LFBNR":"5000000472","RSEG_LFGJA":"2013","RSEG_LFPOS":"0002","NOT_ACCOUNT_MAINTENANCE":
{"$numberDecimal":"1.0000000000"},"RBKP_CPUTIMESTAMP":"2013-10-04T10:32:02.000Z","RBKP_WAERS":"USD","RSEG_BNKAN":
{"$numberDecimal":"0.00"},"RSEG_WRBTR":
{"$numberDecimal":"2340.00"},"RSEG_SHKZG":"S"}
Spark version 2.3 JDK 1.8 MongoDB Version - 3.4 and 3.6 Sample Data - {"_id":"5b0d31f892549e10b61d962a","RSEG_MANDT":"800","RSEG_EBELN":"4500017749","RSEG_EBELP":"00020","RSEG_BELNR":"1000000001","RSEG_BUZEI":"000002","RSEG_GJAHR":"2013","RBKP_BUDAT":"2013-10-04","RSEG_MENGE": {"$numberDecimal":"30.000"} ,"RSEG_LFBNR":"5000000472","RSEG_LFGJA":"2013","RSEG_LFPOS":"0002","NOT_ACCOUNT_MAINTENANCE": {"$numberDecimal":"1.0000000000"} ,"RBKP_CPUTIMESTAMP":"2013-10-04T10:32:02.000Z","RBKP_WAERS":"USD","RSEG_BNKAN": {"$numberDecimal":"0.00"} ,"RSEG_WRBTR": {"$numberDecimal":"2340.00"} ,"RSEG_SHKZG":"S"}
-
Important
Description
I tried to fetch data from MongoDB and got the following exception while converting the RDD to DF.
-----------------Code --------------
SparkSession spark = sparkSessionBuilder.appName("VendorProcessor")
.config("spark.mongodb.input.uri","myMongoUrl")
.config("spark.mongodb.output.uri","myMongoUrl")
.getOrCreate();
JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());
JavaMongoRDD<Document> rdd = MongoSpark.load(jsc);
Dataset<Row> rbkp = rdd.toDF();
------------------ OR ------------------------
DataFrameReader dfr = spark.read()
.format("com.mongodb.spark.sql")
.option("floatAsBigDecimal", "true");
Dataset<Row> rbkp = dfr.load();
--------------------