Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
In the File Channel, we map the checkpoint file. Java cannot seem to be able to map a single file of more than 2GB, since MappedByteBuffer is indexed on int. So even if we get a LongBuffer using asLongBuffer (since that is only a view on the real buffer), the checkpointing will fail when the data is being put, with a nasty exception that looks like:
2014-10-16 13:48:54,684 (Log-BackgroundWorker-FileChannel1) [ERROR - org.apache.flume.channel.file.Log$BackgroundWorker.run(Log.java:1188)] General error in checkpoint worker java.lang.IndexOutOfBoundsException at java.nio.Buffer.checkIndex(Buffer.java:512) at java.nio.DirectLongBufferS.put(DirectLongBufferS.java:270) at org.apache.flume.channel.file.EventQueueBackingStoreFile.checkpoint(EventQueueBackingStoreFile.java:269) at org.apache.flume.channel.file.FlumeEventQueue.checkpoint(FlumeEventQueue.java:145) at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:991) at org.apache.flume.channel.file.Log.writeCheckpoint(Log.java:968) at org.apache.flume.channel.file.Log.access$200(Log.java:75) at org.apache.flume.channel.file.Log$BackgroundWorker.run(Log.java:1183) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)