Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Update operation with bad records property fails with 3.1v.
[Steps to reproduce]:
0: jdbc:hive2://linux-221:22550/> set carbon.options.bad.records.action=force;
------------------------------------------+
| key | value |
------------------------------------------+
| carbon.options.bad.records.action | force |
------------------------------------------+
1 row selected (0.04 seconds)
0: jdbc:hive2://linux-221:22550/> create table t_carbn1(item_type_cd int, sell_price bigint, profit decimal(10,4), item_name string, update_time timestamp) stored a
---------
| Result |
---------
---------
No rows selected (2.117 seconds)
0: jdbc:hive2://linux-221:22550/> insert into t_carbn1 select 2, 100000,23.3,'Apple','2012-11-11 11:11:11';
INFO : Execution ID: 858
-------------
| Segment ID |
-------------
| 0 |
-------------
1 row selected (4.278 seconds)
0: jdbc:hive2://linux-221:22550/> update t_carbn1 set (item_type_cd) = (item_type_cd/1);
Error: org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.RuntimeException: Update operation failed. DataLoad failure
[Root cause]:
On update command, persist is called and with latest 3.1 spark changes, spark returns a cloned SparkSession from cacheManager with all specified configurations disabled. As now its using different sparkSession for 3.1 which is not initialized in CarbonEnv. So CarbonEnv.init is called where new CarbonSessionInfo is created with no sessionParams. So, the properties set were not accessible.
Spark creates cloned spark session based on following properties:
1. spark.sql.optimizer.canChangeCachedPlanOutputPartitioning
2. spark.sql.sources.bucketing.autoBucketedScan.enabled
3. spark.sql.adaptive.enabled