Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2923

Integration job should run full data load + exhaustive.

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      As far as I can tell, our impala-cdh5-integration job loads data from snapshot and does an exhaustive test run. However, most integration problems in the past have show up when doing a full data load, so this job should do:
      1. Full data load (not from snapshot)
      2. Full exhaustive test run

      Job is here:
      http://sandbox.jenkins.cloudera.com/view/Impala/view/Nightly-Builds/job/impala-cdh5-integration/

      As a result of not doing the full data load in that job, we have to manually run the full data load before upgrading our dependencies.

        Activity

        Hide
        mikesbrown Michael Brown added a comment -

        For the above, I mean to say it was filed as upstream issue, but it's really an internal-Cloudera issue. Either way, it's now fixed.

        Show
        mikesbrown Michael Brown added a comment - For the above, I mean to say it was filed as upstream issue, but it's really an internal-Cloudera issue. Either way, it's now fixed.
        Hide
        mikesbrown Michael Brown added a comment -

        This issue was filed back when Apache Impala (incubating) wasn't incubating with Apache. There are no code changes to track here.

        Show
        mikesbrown Michael Brown added a comment - This issue was filed back when Apache Impala (incubating) wasn't incubating with Apache. There are no code changes to track here.
        Hide
        mikesbrown Michael Brown added a comment -

        hbase 1.0.0 -> 1.2.0 is complicating testing the deepcopy, and will also cause the real integration job to fail until we do something with it. There's also just being cowboys about it and toggling the flags without doing comprehensive testing...

        Show
        mikesbrown Michael Brown added a comment - hbase 1.0.0 -> 1.2.0 is complicating testing the deepcopy, and will also cause the real integration job to fail until we do something with it. There's also just being cowboys about it and toggling the flags without doing comprehensive testing...
        Hide
        mikesbrown Michael Brown added a comment -

        Deepcopy ran; modifications for email and dependent jobs done. Test job underway now.

        Show
        mikesbrown Michael Brown added a comment - Deepcopy ran; modifications for email and dependent jobs done. Test job underway now.
        Hide
        alex.behm Alexander Behm added a comment -

        Michael Brown, I believe the answer is "yes". It should be that simple, and it should be safe to copy the job for testing.

        Show
        alex.behm Alexander Behm added a comment - Michael Brown , I believe the answer is "yes". It should be that simple, and it should be safe to copy the job for testing.
        Hide
        mikesbrown Michael Brown added a comment -

        Alexander Behm is the change here as simple as toggling the SKIP_* environment variables to false? If that's the case, I can copy the build and test my copy with the changed environment variables to make sure it performs the full data load. From reading the build configuration, the build triggers other builds. I suppose to be safe I should perform a "deep copy" on the integration job, i.e., also copy its triggering jobs and run those copies as part of my test. That way I don't clog up the real integration job chain.

        Show
        mikesbrown Michael Brown added a comment - Alexander Behm is the change here as simple as toggling the SKIP_* environment variables to false? If that's the case, I can copy the build and test my copy with the changed environment variables to make sure it performs the full data load. From reading the build configuration, the build triggers other builds. I suppose to be safe I should perform a "deep copy" on the integration job, i.e., also copy its triggering jobs and run those copies as part of my test. That way I don't clog up the real integration job chain.
        Hide
        alex.behm Alexander Behm added a comment -
        Show
        alex.behm Alexander Behm added a comment - FYI Harrison Sheinblatt

          People

          • Assignee:
            mikesbrown Michael Brown
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development