Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      herriot

      Description

      Hadoop would benefit from having a large-scale, automated, test-framework. This jira is meant to be a master-jira to track relevant work.


      The proposal is a junit-based, large-scale test framework which would run against real clusters.

      There are several pieces we need to achieve this goal:

      1. A set of utilities we can use in junit-based tests to work with real, large-scale hadoop clusters. E.g. utilities to bring up to deploy, start & stop clusters, bring down tasktrackers, datanodes, entire racks of both etc.
      2. Enhanced control-ability and inspect-ability of the various components in the system e.g. daemons such as namenode, jobtracker should expose their data-structures for query/manipulation etc. Tests would be much more relevant if we could for e.g. query for specific states of the jobtracker, scheduler etc. Clearly these apis should not be part of the production clusters - hence the proposal is to use aspectj to weave these new apis to debug-deployments.

      Related note: we should break up our tests into at least 3 categories:

      1. src/test/unit -> Real unit tests using mock objects (e.g. HDFS-669 & MAPREDUCE-1050).
      2. src/test/integration -> Current junit tests with Mini* clusters etc.
      3. src/test/system -> HADOOP-6332 and it's children
      1. 6332_v1.patch
        20 kB
        Sharad Agarwal
      2. 6332_v2.patch
        24 kB
        Sharad Agarwal
      3. 6332.patch
        183 kB
        Konstantin Boudnik
      4. 6332.patch
        183 kB
        Konstantin Boudnik
      5. 6332.patch
        86 kB
        Sharad Agarwal
      6. 6332-phase2.fix1.patch
        2 kB
        Konstantin Boudnik
      7. 6332-phase2.fix2.patch
        1 kB
        Konstantin Boudnik
      8. 6332-phase2.patch
        358 kB
        Konstantin Boudnik
      9. HADOOP-6332.0.22.patch
        267 kB
        Konstantin Boudnik
      10. HADOOP-6332.0.22.patch
        267 kB
        Konstantin Boudnik
      11. HADOOP-6332.0.22.patch
        249 kB
        Konstantin Boudnik
      12. HADOOP-6332.0.22.patch
        254 kB
        Konstantin Boudnik
      13. HADOOP-6332.0.22.patch
        253 kB
        Konstantin Boudnik
      14. HADOOP-6332.0.22.patch
        253 kB
        Konstantin Boudnik
      15. HADOOP-6332.0.22.patch
        253 kB
        Konstantin Boudnik
      16. HADOOP-6332.0.22.patch
        256 kB
        Konstantin Boudnik
      17. HADOOP-6332.patch
        13 kB
        Konstantin Boudnik
      18. HADOOP-6332.patch
        13 kB
        Konstantin Boudnik
      19. HADOOP-6332-MR.patch
        13 kB
        Konstantin Boudnik
      20. HADOOP-6332-MR.patch
        11 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Arun C Murthy created issue -
          Arun C Murthy made changes -
          Field Original Value New Value
          Description Hadoop would benefit from having a large-scale, automated, test-framework.

          This jira is meant to be a master-jira to track relevant details.
          Hadoop would benefit from having a large-scale, automated, test-framework. This jira is meant to be a master-jira to track relevant work.

          ----

          The proposal is a junit-based, large-scale test framework which would run against _real_ clusters.

          There are several pieces we need to achieve this goal:

          # A set of utilities we can use in junit-based tests to work with real, large-scale hadoop clusters. E.g. utilities to bring up to deploy, start & stop clusters, bring down tasktrackers, datanodes, entire racks of both etc.
          # Enhanced control-ability and inspect-ability of the various components in the system e.g. daemons such as namenode, jobtracker should expose their data-structures for query/manipulation etc. Tests would be much more relevant if we could for e.g. query for specific states of the jobtracker, scheduler etc. Clearly these apis should _not_ be part of the production clusters - hence the proposal is to use aspectj to weave these new apis to debug-deployments.

          ----

          Related note: we should break up our tests into at least 3 categories:
          # src/test/unit -> Real unit tests using mock objects (e.g. HDFS-669 & MAPREDUCE-1050).
          # src/test/integration -> Current junit tests with Mini* clusters etc.
          # src/test/system -> HADOOP-6332 and it's children
          Arun C Murthy made changes -
          Link This issue incorporates MAPREDUCE-1154 [ MAPREDUCE-1154 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HADOOP-6248 [ HADOOP-6248 ]
          Konstantin Boudnik made changes -
          Link This issue is related to HDFS-708 [ HDFS-708 ]
          Arun C Murthy made changes -
          Assignee Arun C Murthy [ acmurthy ]
          Sharad Agarwal made changes -
          Attachment 6332_v1.patch [ 12426517 ]
          Konstantin Boudnik made changes -
          Link This issue is related to HADOOP-1257 [ HADOOP-1257 ]
          Sharad Agarwal made changes -
          Attachment 6332_v2.patch [ 12427707 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332-MR.patch [ 12428228 ]
          Attachment HADOOP-6332.patch [ 12428229 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.patch [ 12428237 ]
          Attachment HADOOP-6332-MR.patch [ 12428238 ]
          Sharad Agarwal made changes -
          Attachment 6332.patch [ 12434779 ]
          Konstantin Boudnik made changes -
          Assignee Arun C Murthy [ acmurthy ] Sharad Agarwal [ sharadag ]
          Konstantin Boudnik made changes -
          Attachment 6332.patch [ 12438073 ]
          Konstantin Boudnik made changes -
          Attachment 6332.patch [ 12438210 ]
          Konstantin Boudnik made changes -
          Link This issue blocks MAPREDUCE-1616 [ MAPREDUCE-1616 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1616 [ MAPREDUCE-1616 ]
          Konstantin Boudnik made changes -
          Link This issue blocks MAPREDUCE-1616 [ MAPREDUCE-1616 ]
          Sharad Agarwal made changes -
          Link This issue blocks MAPREDUCE-1653 [ MAPREDUCE-1653 ]
          Sharad Agarwal made changes -
          Link This issue blocks MAPREDUCE-1653 [ MAPREDUCE-1653 ]
          Sharad Agarwal made changes -
          Link This issue is required by MAPREDUCE-1653 [ MAPREDUCE-1653 ]
          Sharad Agarwal made changes -
          Link This issue is required by MAPREDUCE-1654 [ MAPREDUCE-1654 ]
          Sharad Agarwal made changes -
          Link This issue is required by MAPREDUCE-1655 [ MAPREDUCE-1655 ]
          Konstantin Boudnik made changes -
          Link This issue blocks MAPREDUCE-1646 [ MAPREDUCE-1646 ]
          Iyappan Srinivasan made changes -
          Link This issue is required by MAPREDUCE-1671 [ MAPREDUCE-1671 ]
          Iyappan Srinivasan made changes -
          Link This issue is required by MAPREDUCE-1672 [ MAPREDUCE-1672 ]
          Iyappan Srinivasan made changes -
          Link This issue is required by MAPREDUCE-1677 [ MAPREDUCE-1677 ]
          Iyappan Srinivasan made changes -
          Link This issue is required by MAPREDUCE-1676 [ MAPREDUCE-1676 ]
          Vinay Kumar Thota made changes -
          Link This issue is required by MAPREDUCE-1693 [ MAPREDUCE-1693 ]
          Iyappan Srinivasan made changes -
          Link This issue is required by MAPREDUCE-1730 [ MAPREDUCE-1730 ]
          Iyappan Srinivasan made changes -
          Link This issue is required by MAPREDUCE-1741 [ MAPREDUCE-1741 ]
          Konstantin Boudnik made changes -
          Attachment 6332-phase2.patch [ 12443539 ]
          Vinay Kumar Thota made changes -
          Link This issue is required by MAPREDUCE-1731 [ MAPREDUCE-1731 ]
          Konstantin Boudnik made changes -
          Attachment 6332-phase2.fix1.patch [ 12443668 ]
          Vinay Kumar Thota made changes -
          Link This issue is required by MAPREDUCE-1753 [ MAPREDUCE-1753 ]
          Vinay Kumar Thota made changes -
          Link This issue is blocked by MAPREDUCE-1713 [ MAPREDUCE-1713 ]
          Vinay Kumar Thota made changes -
          Link This issue is blocked by MAPREDUCE-1713 [ MAPREDUCE-1713 ]
          Vinay Kumar Thota made changes -
          Link This issue requires MAPREDUCE-1713 [ MAPREDUCE-1713 ]
          Konstantin Boudnik made changes -
          Attachment 6332-phase2.fix2.patch [ 12443788 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1760 [ MAPREDUCE-1760 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1710 [ MAPREDUCE-1710 ]
          Konstantin Boudnik made changes -
          Assignee Sharad Agarwal [ sharadag ] Konstantin Boudnik [ cos ]
          Konstantin Boudnik made changes -
          Link This issue relates to HDFS-1134 [ HDFS-1134 ]
          Konstantin Boudnik made changes -
          Link This issue relates to MAPREDUCE-1774 [ MAPREDUCE-1774 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12443924 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12444006 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12444019 ]
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.22.0 [ 12314296 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Fix Version/s 0.21.0 [ 12313563 ]
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12444027 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12444028 ]
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12444166 ]
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12444175 ]
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.patch [ 12444342 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.patch [ 12444342 ]
          Konstantin Boudnik made changes -
          Attachment HADOOP-6332.0.22.patch [ 12444343 ]
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Konstantin Boudnik made changes -
          Affects Version/s 0.21.0 [ 12313563 ]
          Affects Version/s 0.22.0 [ 12314296 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1760 [ MAPREDUCE-1760 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1616 [ MAPREDUCE-1616 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1654 [ MAPREDUCE-1654 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1671 [ MAPREDUCE-1671 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1672 [ MAPREDUCE-1672 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1677 [ MAPREDUCE-1677 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1741 [ MAPREDUCE-1741 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1676 [ MAPREDUCE-1676 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1730 [ MAPREDUCE-1730 ]
          Konstantin Boudnik made changes -
          Link This issue blocks HADOOP-6752 [ HADOOP-6752 ]
          Konstantin Boudnik made changes -
          Link This issue blocks MAPREDUCE-1646 [ MAPREDUCE-1646 ]
          Konstantin Boudnik made changes -
          Link This issue requires MAPREDUCE-1713 [ MAPREDUCE-1713 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1693 [ MAPREDUCE-1693 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1731 [ MAPREDUCE-1731 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1753 [ MAPREDUCE-1753 ]
          Konstantin Boudnik made changes -
          Link This issue is required by MAPREDUCE-1710 [ MAPREDUCE-1710 ]
          Konstantin Boudnik made changes -
          Link This issue blocks HADOOP-6777 [ HADOOP-6777 ]
          Konstantin Boudnik made changes -
          Link This issue blocks HADOOP-6772 [ HADOOP-6772 ]
          Vinay Kumar Thota made changes -
          Link This issue blocks HDFS-1167 [ HDFS-1167 ]
          Konstantin Boudnik made changes -
          Link This issue blocks HDFS-1167 [ HDFS-1167 ]
          Konstantin Boudnik made changes -
          Link This issue relates to HADOOP-6784 [ HADOOP-6784 ]
          Konstantin Boudnik made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Fix Version/s 0.21.0 [ 12313563 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Resolution Fixed [ 1 ]
          Konstantin Boudnik made changes -
          Link This issue is related to HADOOP-6786 [ HADOOP-6786 ]
          Vinay Kumar Thota made changes -
          Link This issue blocks HADOOP-6788 [ HADOOP-6788 ]
          Konstantin Boudnik made changes -
          Link This issue is related to HADOOP-6789 [ HADOOP-6789 ]
          Konstantin Boudnik made changes -
          Link This issue is related to HADOOP-6790 [ HADOOP-6790 ]
          Konstantin Boudnik made changes -
          Link This issue relates to HADOOP-6828 [ HADOOP-6828 ]
          Vinay Kumar Thota made changes -
          Link This issue is depended upon by HADOOP-6819 [ HADOOP-6819 ]
          Vinay Kumar Thota made changes -
          Link This issue blocks HADOOP-6836 [ HADOOP-6836 ]
          Vinay Kumar Thota made changes -
          Link This issue is depended upon by HADOOP-6869 [ HADOOP-6869 ]
          Konstantin Boudnik made changes -
          Link This issue relates to HDFS-1341 [ HDFS-1341 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Konstantin Boudnik made changes -
          Tags herriot
          Gavin made changes -
          Link This issue blocks HADOOP-6836 [ HADOOP-6836 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-6836 [ HADOOP-6836 ]
          Gavin made changes -
          Link This issue blocks HADOOP-6777 [ HADOOP-6777 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-6777 [ HADOOP-6777 ]
          Gavin made changes -
          Link This issue blocks HADOOP-6772 [ HADOOP-6772 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-6772 [ HADOOP-6772 ]
          Gavin made changes -
          Link This issue blocks HADOOP-6788 [ HADOOP-6788 ]
          Gavin made changes -
          Link This issue is depended upon by HADOOP-6788 [ HADOOP-6788 ]

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development