Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-389

Updates sent to diff partition for a given key with Global Index

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.5.1
    • Component/s: Index

      Description

      Updates sent to diff partition for a given key with Global Index should succeed by updating the record under original partition. As of now, it throws exception. 

      https://github.com/apache/incubator-hudi/issues/1021 

       

       

      error log:

      
      
       14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants java.util.stream.ReferencePipeline$Head@d02b1c7
       14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file system view for partition (2016/04/15)
       14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found in partition (2016/04/15) =0, Time taken =0
       14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
       14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding file-groups for partition :2016/04/15, #FileGroups=0
       14738 [Executor task launch worker-0] INFO com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load partition (2016/04/15) =0
       14754 [Executor task launch worker-0] ERROR com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType UPDATE for partition :0
       java.util.NoSuchElementException: No value present
       at com.uber.hoodie.common.util.Option.get(Option.java:112)
       at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
       at com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
       at com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
       at com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
       at com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
       at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
       at org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
       at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
       at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
       at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
       at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
       at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
       at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
       at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
       at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
       at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
       at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
       at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
       at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
       at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
       at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
       at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
       at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
       at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
       at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
       at org.apache.spark.scheduler.Task.run(Task.scala:99)
       at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
      
       

      Refer [here|https://github.com/apache/incubator-hudi/issues/1021] for some context. 

       

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shivnarayan sivabalan narayanan
                Reporter:
                shivnarayan sivabalan narayanan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 48h
                  48h
                  Remaining:
                  Remaining Estimate - 47h 40m
                  47h 40m
                  Logged:
                  Remaining Estimate - 47h 40m
                  20m