Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-474

Delta Streamer is not able to read the commit files

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.1
    • Component/s: DeltaStreamer
    • Labels:
      None

      Description

      DeltaStreamer is not to able to read the correct commit files under when job is deployed realtime.

      below is the stack trace: 

      
      ava.util.concurrent.ExecutionException:
       org.apache.hudi.exception.HoodieException: Could not read commit
       details from hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested      at
       java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)    at
       java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at
       org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService.waitForShutdown(AbstractDeltaStreamerService.java:72)      at
       org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:117)  at
       org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:297)  at
       sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at
       sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)   at
       sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)   at
       java.lang.reflect.Method.invoke(Method.java:498)        at
       org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:688)Caused
       by: org.apache.hudi.exception.HoodieException: Could not read commit
       details from hdfs:/user/hive/warehouse/hudi.db/tbltest/.hoodie/.aux/20191226153400.clean.requested      at
       org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:411)        at
       java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)     at
       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at
       java.lang.Thread.run(Thread.java:748)
       

       

        Attachments

          Activity

            People

            • Assignee:
              vbalaji Balaji Varadarajan
              Reporter:
              srkhan Shahida Khan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: