Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3272

While running a pig job,showing delay after Snappy native library get loaded

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 0.10.1
    • None
    • data
    • None

    Description

      2013-04-09 06:29:42,639 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
      2013-04-09 06:29:43,947 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 1024
      2013-04-09 06:29:44,434 INFO org.apache.hadoop.mapred.MapTask: data buffer = 816043776/1020054736
      2013-04-09 06:29:44,435 INFO org.apache.hadoop.mapred.MapTask: record buffer = 2684354/3355443
      2013-04-09 06:29:44,448 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is available
      2013-04-09 06:29:44,448 INFO org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library loaded
      2013-04-09 06:48:29,783 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
      2013-04-09 06:48:29,783 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 423372603; bufvoid = 1020054736
      2013-04-09 06:48:29,783 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 2684354; length = 3355443
      2013-04-09 06:49:32,969 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new compressor

      In the above log you will see, after Snappy native library loaded it took around 20mins to proceed.
      Can anyone tell me the issue and the resolution?
      is it due to some network issue?

      Attachments

        Activity

          People

            Unassigned Unassigned
            sambit Sambit Sahoo
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: