Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-675

maximum frame size of frameDecoder should be increased

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      As Repartitioner::scheduleHashShuffledFetches re-optimizes the number of tasks, sometimes one task may fetch results of thousands of tasks.

      In my case, there is a case that one task fetches from 9240 tasks,
      and got following exception

      2014-03-07 13:29:24,240 ERROR rpc.AsyncRpcClient (AsyncRpcClient.java:exceptionCaught(220)) - skt-rf-01/50.1.103.1:28093,class org.apache.tajo.ipc.QueryMasterProtocol,Adjusted frame length exceeds 2097152: 7729393 - discarded
      org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 2097152: 7729393 - discarded
              at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:417)
              at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:405)
              at org.jboss.netty.handler.codec.frame.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:320)
              at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
              at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
              at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
              at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      2014-03-07 13:29:24,242 ERROR rpc.AsyncRpcClient (AsyncRpcClient.java:exceptionCaught(235)) - RPC Exception:org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 2097152: 7729393 - discarded
      

      The reason seems that frameDecoder setting in getPipeline() of ProtoPipelineFactory.java is 2MB

        public ChannelPipeline getPipeline() throws Exception {
          ChannelPipeline p = Channels.pipeline();
          p.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(1048576*2, 0, 4,
              0, 4));
          p.addLast("protobufDecoder", new ProtobufDecoder(defaultInstance));
          p.addLast("frameEncoder", new LengthFieldPrepender(4));
          p.addLast("protobufEncoder", new ProtobufEncoder());
          p.addLast("handler", handler);
          return p;
        }
      

      I think temporal solution for the issue is just increasing the size.
      However, I'm not sure what the proper size is.

        Activity

        Hide
        charsyam DaeMyung Kang added a comment -

        Keuntae Park Could you review https://issues.apache.org/jira/browse/TAJO-623?

        I think it is connected with this issue.

        Show
        charsyam DaeMyung Kang added a comment - Keuntae Park Could you review https://issues.apache.org/jira/browse/TAJO-623? I think it is connected with this issue.
        Hide
        jhkim Jinho Kim added a comment -

        I think a frame should be variable length and we need refactoring the fetch URIs. I will start working for fetch uri issue
        http://docs.jboss.org/netty/3.2/api/org/jboss/netty/handler/codec/protobuf/ProtobufVarint32FrameDecoder.html

        Show
        jhkim Jinho Kim added a comment - I think a frame should be variable length and we need refactoring the fetch URIs. I will start working for fetch uri issue http://docs.jboss.org/netty/3.2/api/org/jboss/netty/handler/codec/protobuf/ProtobufVarint32FrameDecoder.html
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12641025/TAJO-675.patch
        against master revision f1f36ec.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-rpc.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/363//testReport/
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/363//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641025/TAJO-675.patch against master revision f1f36ec. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-rpc. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/363//testReport/ Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/363//console This message is automatically generated.
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1

        Ship it. The patch looks reasonable.

        Show
        hyunsik Hyunsik Choi added a comment - +1 Ship it. The patch looks reasonable.
        Hide
        jhkim Jinho Kim added a comment -

        Thank you for the review.
        I've just committed it to both master branch and branch-0.8.0.

        Show
        jhkim Jinho Kim added a comment - Thank you for the review. I've just committed it to both master branch and branch-0.8.0.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #191 (See https://builds.apache.org/job/Tajo-master-build/191/)
        TAJO-675: maximum frame size of frameDecoder should be increased. (jinho) (jinossy: rev c367377fa2887458fd246b7481cfc49865dc5ff9)

        • CHANGES.txt
        • tajo-rpc/src/main/java/org/apache/tajo/rpc/ProtoPipelineFactory.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #191 (See https://builds.apache.org/job/Tajo-master-build/191/ ) TAJO-675 : maximum frame size of frameDecoder should be increased. (jinho) (jinossy: rev c367377fa2887458fd246b7481cfc49865dc5ff9) CHANGES.txt tajo-rpc/src/main/java/org/apache/tajo/rpc/ProtoPipelineFactory.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.8.0-build #83 (See https://builds.apache.org/job/Tajo-0.8.0-build/83/)
        TAJO-675: maximum frame size of frameDecoder should be increased. (jinho) (jinossy: rev afa496cb2b870eb7595c6ee4114499dc34ee045b)

        • CHANGES.txt
        • tajo-rpc/src/main/java/org/apache/tajo/rpc/ProtoPipelineFactory.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #83 (See https://builds.apache.org/job/Tajo-0.8.0-build/83/ ) TAJO-675 : maximum frame size of frameDecoder should be increased. (jinho) (jinossy: rev afa496cb2b870eb7595c6ee4114499dc34ee045b) CHANGES.txt tajo-rpc/src/main/java/org/apache/tajo/rpc/ProtoPipelineFactory.java

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            sirpkt Keuntae Park
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development