[IGNITE-8780] File I/O operations must be retried if buffer hasn't read/written completely

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5
• Fix Version/s: 2.7
• Component/s: cache
• Labels: important
• Assignee: Alexey Stelmak
• Reporter: Pavel Kovalenko
• Created: 13/Jun/18 14:18
• Updated: 04/Jul/18 18:33
• Resolved: 04/Jul/18 18:33

Description

Currently we don't actually ensure that we write or read some buffer completely:
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager#writeCheckpointEntry
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager#nodeStart

As result we may not write to the disk actual data and after node restart we can get BufferUnderflowException, like this:

java.nio.BufferUnderflowException
    at java.nio.Buffer.nextGetIndex(Buffer.java:506)
    at java.nio.HeapByteBuffer.getLong(HeapByteBuffer.java:412)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readPointer(GridCacheDatabaseSharedManager.java:1915)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointStatus(GridCacheDatabaseSharedManager.java:1892)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:565)
    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:525)
    at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:700)
    at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1738)
    at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:985)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
    at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:671)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:596)
    at org.apache.ignite.Ignition.start(Ignition.java:327)
    at org.apache.ignite.ci.db.TcHelperDb.start(TcHelperDb.java:67)
    at org.apache.ignite.ci.web.CtxListener.contextInitialized(CtxListener.java:37)
    at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:890)
    at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:532)
    at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:853)
    at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:344)
    at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1501)
    at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1463)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:785)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
    at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
    at org.eclipse.jetty.server.Server.start(Server.java:452)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113)
    at org.eclipse.jetty.server.Server.doStart(Server.java:419)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
    at org.apache.ignite.ci.web.Launcher.runServer(Launcher.java:68)
    at org.apache.ignite.ci.TcHelperJettyLauncher.main(TcHelperJettyLauncher.java:10)

and node become into unrecoverable state.