[DRILL-8426] Endless retrying zk set data for a large query

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.21.1
• Fix Version/s: 1.21.2
• Component/s: Execution - Flow
• Assignee: Maksym Rymar
• Reporter: Maksym Rymar
• Created: 27/Apr/23 20:14
• Updated: 01/Jan/24 06:23
• Resolved: 01/Jan/24 06:23

Description

If to issue a large query (bigger than 1MB) Drill can fall down into infinite loop of retries to set data to zookeeper.

In zookeeper logs you will see repeating errors like this:

java.io.IOException: Len error 112569

In drillbit logs you may see errors like this:

Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /drill/magic-drillbits/ea569524-abaa-41e2-9f69-7857f3a04b6c
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:2419)
        at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:291)
        at org.apache.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:287)
        at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:67)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:81)
        at org.apache.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:284)
        at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:270)
        at org.apache.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:33)
        at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.updateService(ServiceDiscoveryImpl.java:208)

Issue Links

• relates to: ZOOKEEPER-4053 ConnectionLossException is vague for failing to read/write large znode