Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Cannot Reproduce
-
None
-
None
-
2
Description
When timeline server based markers are used for very large table with bulk_insert row writer (1TB or more), we are running into null pointer exception.
2022-06-14 13:27:13,135 WARN hudi.DataSourceUtils: Small Decimal Type found in current schema, auto set the value of hoodie.parquet.writelegacyformat.enabled to true 2022-06-14 13:27:13,562 WARN metadata.HoodieBackedTableMetadata: Metadata table was not found at path s3a://datasets-abcde/hudi/web_sales/.hoodie/metadata 2022-06-14 13:27:13,972 WARN metadata.HoodieBackedTableMetadata: Metadata table was not found at path s3a://datasets-abcde/hudi/web_sales/.hoodie/metadata Exception in thread "pool-37-thread-6" java.lang.NullPointerException200) / 200] at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-8" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-06-14 13:31:30,483 WARN impl.BulkDeleteRetryHandler: Bulk delete operation interrupted: delete: com.amazonaws.SdkClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$DeleteObjectsHandler Exception in thread "pool-37-thread-22" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-10" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-14" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-15" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-12" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-13" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-16" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-20" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Exception in thread "pool-37-thread-7" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-06-14 13:31:37,760 WARN scheduler.TaskSetManager: Lost task 172.0 in stage 91.0 (TID 11689) (ip-172-31-82-134.ec2.internal executor 52): org.apache.hudi.exception.HoodieRemoteException: Failed to create marker file ws_sold_date_sk=2451888/e9ac4775-437a-4284-8ca4-8c3f2265adf4-0_172-11689-0_20220614132711019.parquet.marker.CREATE Server Error at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:149) at org.apache.hudi.table.marker.WriteMarkers.create(WriteMarkers.java:64) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.createMarkerFile(HoodieRowCreateHandle.java:192) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.<init>(HoodieRowCreateHandle.java:99) at org.apache.hudi.io.storage.row.HoodieRowCreateHandleWithoutMetaFields.<init>(HoodieRowCreateHandleWithoutMetaFields.java:38) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.getRowCreateHandle(BulkInsertDataInternalWriterHelper.java:174) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.write(BulkInsertDataInternalWriterHelper.java:149) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:48) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:35) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:419) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:457) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.http.client.HttpResponseException: Server Error at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69) at org.apache.http.client.fluent.Response.handleResponse(Response.java:90) at org.apache.http.client.fluent.Response.returnContent(Response.java:97) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.executeRequestToTimelineServer(TimelineServerBasedWriteMarkers.java:180) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:145) ... 20 more Exception in thread "pool-37-thread-4" java.lang.NullPointerException at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.flushMarkersToFile(MarkerDirState.java:323) at org.apache.hudi.timeline.service.handlers.marker.MarkerDirState.processMarkerCreationRequests(MarkerDirState.java:219) at org.apache.hudi.timeline.service.handlers.marker.BatchedMarkerCreationRunnable.run(BatchedMarkerCreationRunnable.java:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2022-06-14 13:31:38,930 WARN scheduler.TaskSetManager: Lost task 79.0 in stage 91.0 (TID 11596) (ip-172-31-80-194.ec2.internal executor 37): org.apache.hudi.exception.HoodieRemoteException: Failed to create marker file ws_sold_date_sk=2451867/47e2cf7c-4f4b-4779-804e-57e990794050-0_79-11596-0_20220614132711019.parquet.marker.CREATE Server Error at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:149) at org.apache.hudi.table.marker.WriteMarkers.create(WriteMarkers.java:64) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.createMarkerFile(HoodieRowCreateHandle.java:192) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.<init>(HoodieRowCreateHandle.java:99) at org.apache.hudi.io.storage.row.HoodieRowCreateHandleWithoutMetaFields.<init>(HoodieRowCreateHandleWithoutMetaFields.java:38) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.getRowCreateHandle(BulkInsertDataInternalWriterHelper.java:174) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.write(BulkInsertDataInternalWriterHelper.java:149) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:48) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:35) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:419) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:457) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.http.client.HttpResponseException: Server Error at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69) at org.apache.http.client.fluent.Response.handleResponse(Response.java:90) at org.apache.http.client.fluent.Response.returnContent(Response.java:97) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.executeRequestToTimelineServer(TimelineServerBasedWriteMarkers.java:180) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:145) ... 20 more 2022-06-14 13:31:38,932 WARN scheduler.TaskSetManager: Lost task 99.0 in stage 91.0 (TID 11616) (ip-172-31-82-38.ec2.internal executor 21): org.apache.hudi.exception.HoodieRemoteException: Failed to create marker file ws_sold_date_sk=2452502/54ae3ed1-8331-4028-a5ed-4034e353fbd6-0_99-11616-0_20220614132711019.parquet.marker.CREATE Server Error at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:149) at org.apache.hudi.table.marker.WriteMarkers.create(WriteMarkers.java:64) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.createMarkerFile(HoodieRowCreateHandle.java:192) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.<init>(HoodieRowCreateHandle.java:99) at org.apache.hudi.io.storage.row.HoodieRowCreateHandleWithoutMetaFields.<init>(HoodieRowCreateHandleWithoutMetaFields.java:38) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.getRowCreateHandle(BulkInsertDataInternalWriterHelper.java:174) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.write(BulkInsertDataInternalWriterHelper.java:149) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:48) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:35) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:419) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:457) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.http.client.HttpResponseException: Server Error at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69) at org.apache.http.client.fluent.Response.handleResponse(Response.java:90) at org.apache.http.client.fluent.Response.returnContent(Response.java:97) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.executeRequestToTimelineServer(TimelineServerBasedWriteMarkers.java:180) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:145) ... 20 more 2022-06-14 13:31:38,933 WARN scheduler.TaskSetManager: Lost task 3.0 in stage 91.0 (TID 11520) (ip-172-31-95-89.ec2.internal executor 19): org.apache.hudi.exception.HoodieRemoteException: Failed to create marker file ws_sold_date_sk=2452028/9b6511e2-73a0-432a-b886-88a90c73cb3e-0_3-11520-0_20220614132711019.parquet.marker.CREATE Server Error at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:149) at org.apache.hudi.table.marker.WriteMarkers.create(WriteMarkers.java:64) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.createMarkerFile(HoodieRowCreateHandle.java:192) at org.apache.hudi.io.storage.row.HoodieRowCreateHandle.<init>(HoodieRowCreateHandle.java:99) at org.apache.hudi.io.storage.row.HoodieRowCreateHandleWithoutMetaFields.<init>(HoodieRowCreateHandleWithoutMetaFields.java:38) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.getRowCreateHandle(BulkInsertDataInternalWriterHelper.java:174) at org.apache.hudi.internal.BulkInsertDataInternalWriterHelper.write(BulkInsertDataInternalWriterHelper.java:149) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:48) at org.apache.hudi.spark3.internal.HoodieBulkInsertDataInternalWriter.write(HoodieBulkInsertDataInternalWriter.java:35) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:419) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496) at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:457) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.http.client.HttpResponseException: Server Error at org.apache.http.impl.client.AbstractResponseHandler.handleResponse(AbstractResponseHandler.java:69) at org.apache.http.client.fluent.Response.handleResponse(Response.java:90) at org.apache.http.client.fluent.Response.returnContent(Response.java:97) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.executeRequestToTimelineServer(TimelineServerBasedWriteMarkers.java:180) at org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.create(TimelineServerBasedWriteMarkers.java:145) ... 20 more
asdfa
Attachments
Issue Links
- relates to
-
HUDI-4342 Improve handling of 5xx in timeline server
- Closed