Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-4531

[WebHCat] Collecting task logs to hdfs

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: HCatalog, WebHCat
    • Labels:
      None
    • Release Note:
      Hide
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is:

      logs/$job_id (directory for $job_id)
      logs/$job_id/job.xml.html
      logs/$job_id/$attempt_id (directory for $attempt_id)
      logs/$job_id/$attempt_id/stderr
      logs/$job_id/$attempt_id/stdout
      logs/$job_id/$attempt_id/syslog

      NO PRECOMMIT TESTS
      Show
      In POST pig/hive/jar/steaming request, if statusdir is set and enablelog=true, webhcat will create a $statusdir/logs directory after task finish. The attempts here include completed attempts and failed attempts. The layout for logs directory is: logs/$job_id (directory for $job_id) logs/$job_id/job.xml.html logs/$job_id/$attempt_id (directory for $attempt_id) logs/$job_id/$attempt_id/stderr logs/$job_id/$attempt_id/stdout logs/$job_id/$attempt_id/syslog NO PRECOMMIT TESTS

      Description

      It would be nice we collect task logs after job finish. This is similar to what Amazon EMR does.

        Attachments

        1. samplestatusdirwithlist.tar.gz
          109 kB
          Jianyong Dai
        2. HIVE-4531-9.patch
          65 kB
          Jianyong Dai
        3. HIVE-4531-8.patch
          59 kB
          Jianyong Dai
        4. HIVE-4531-7.patch
          60 kB
          Jianyong Dai
        5. HIVE-4531-6.patch
          60 kB
          Jianyong Dai
        6. HIVE-4531-5.patch
          62 kB
          Jianyong Dai
        7. HIVE-4531-4.patch
          59 kB
          Jianyong Dai
        8. HIVE-4531-3.patch
          55 kB
          Jianyong Dai
        9. HIVE-4531-2.patch
          54 kB
          Jianyong Dai
        10. HIVE-4531-11.patch
          67 kB
          Jianyong Dai
        11. HIVE-4531-10.patch
          67 kB
          Jianyong Dai
        12. HIVE-4531-1.patch
          53 kB
          Jianyong Dai

          Issue Links

            Activity

              People

              • Assignee:
                daijy Jianyong Dai
                Reporter:
                daijy Jianyong Dai
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: