Hadoop Common
  1. Hadoop Common
  2. HADOOP-2846

Large input data-sets throw "java.net.SocketTimeoutException: timed out waiting for rpc response" exception

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.15.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Pig scripts can run over a data set of 1 day. Using the same script and same number of nodes on a larger data set (of 30 days) fails and throws following exception after 1+ hour of running.

      java.net.SocketTimeoutException: timed out waiting for rpc response
      at org.apache.hadoop.ipc.Client.call(Client.java:484)
      at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:184)
      at $Proxy1.getJobStatus(Unknown Source)
      at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
      at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
      at $Proxy1.getJobStatus(Unknown Source)
      at org.apache.hadoop.mapred.JobClient$NetworkedJob.ensureFreshStatus(JobClient.java:182)
      at org.apache.hadoop.mapred.JobClient$NetworkedJob.isComplete(JobClient.java:237)
      at org.apache.pig.impl.mapreduceExec.MapReduceLauncher.launchPig(MapReduceLauncher.java:189)
      at org.apache.pig.impl.physicalLayer.POMapreduce.open(POMapreduce.java:136)
      at org.apache.pig.impl.physicalLayer.POMapreduce.open(POMapreduce.java:129)
      at org.apache.pig.impl.physicalLayer.POMapreduce.open(POMapreduce.java:129)
      at org.apache.pig.impl.physicalLayer.PhysicalPlan.exec(PhysicalPlan.java:39)
      at org.apache.pig.impl.physicalLayer.IntermedResult.exec(IntermedResult.java:122)
      at org.apache.pig.PigServer.store(PigServer.java:445)
      at org.apache.pig.PigServer.store(PigServer.java:413)
      at org.apache.pig.tools.grunt.GruntParser.processStore(GruntParser.java:135)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:327)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:54)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:54)
      at org.apache.pig.Main.main(Main.java:258)
      timed out waiting for rpc response

      Re-runing always hits the same at %3 progress.

        Activity

        Hide
        Owen O'Malley added a comment -

        0.15.3 is already release, so this issue can't be assigned to it. It isn't clear what the problem is, so I'm unsetting the fix version.

        Show
        Owen O'Malley added a comment - 0.15.3 is already release, so this issue can't be assigned to it. It isn't clear what the problem is, so I'm unsetting the fix version.

          People

          • Assignee:
            Unassigned
            Reporter:
            Amir Youssefi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development