Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4207

data load failure: load_nested.py: AttributeError: 'Namespace' object has no attribute 'use_hive'

    Details

      Description

      One of Sahil Takiar's recent changes caused this:

      http://sandbox.jenkins.cloudera.com/job/impala-asf-master-core-data-load/83/

      21:38:47 Loading nested data (logging to load-nested.log)... FAILED
      ...
      21:38:49 Traceback (most recent call last):
      21:38:49   File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/load_nested.py", line 305, in <module>
      21:38:49     cluster = cli_options.create_cluster(args)
      21:38:49   File "/data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/tests/comparison/cli_options.py", line 123, in create_cluster
      21:38:49     elif args.use_hive:
      21:38:49 AttributeError: 'Namespace' object has no attribute 'use_hive'
      21:38:49 Error in /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/testdata/bin/create-load-data.sh at line 45: while [ -n "$*" ]
      21:38:49 Error in /data/jenkins/workspace/impala-asf-master-core-data-load/repos/Impala/buildall.sh at line 407: "${IMPALA_HOME}/testdata/bin/create-load-data.sh" ${CREATE_LOAD_DATA_ARGS} <<< Y
      

        Activity

        Hide
        mikesbrown Michael Brown added a comment -

        The stress test is technically prone to this too, but won't tend to due to luck.

        Show
        mikesbrown Michael Brown added a comment - The stress test is technically prone to this too, but won't tend to due to luck.
        Show
        mikesbrown Michael Brown added a comment - https://gerrit.cloudera.org/#/c/4555/
        Show
        dknupp David Knupp added a comment - It caused this failure as well: http://sandbox.jenkins.cloudera.com/view/Impala/view/Experimental/job/impala-asf-master-core-ubuntu14.04-with-dataload/7/console
        Hide
        mikesbrown Michael Brown added a comment -
        commit a35e4380966346b47a81eeb2500e01bf97fa8f95
        Author: Michael Brown <mikeb@cloudera.com>
        Date:   Wed Sep 28 08:29:19 2016 -0700
        
            IMPALA-4207: test infra: move Hive options from connection to cluster options
        
            Various test tools and frameworks, including the stress test, random
            query generator, and nested types loader, share common modules. This
            change
        
              IMPALA-3980: qgen: re-enable Hive as a target database
        
            made changes to tests.comparison.cli_options, the shared command line
            option module, and to tests.comparison.cluster, the shared module for
            modeling various Impala clusters. Those changes were for the random
            query generator, but didn't take into account the other shared entry
            points. It was possible to call some of those entry points in such a way
            as to produce an exception, because the Hive-related options are now
            required for miniclusters, but the Hive-related options weren't always
            being initialized in those entry points.
        
            The simple fix is to say that, because Hive settings are now needed to
            create Minicluster objects, make the Hive options initialized with
            cluster options, not connection options. While I was making these
            changes, I fixed all flake8 problems in this file.
        
            Testing:
        
            - qgen/minicluster unit tests (regression test)
            - full private data load job, including load_nested.py (bug
              verification)
            - data_generator.py run (regression test), long enough to verify
              connection to the minicluster, using both Hive and Impala
            - discrepancy_searcher.py run (regression test), long enough verify
              connection to the minicluster, using both Hive and Impala
            - concurrent_select.py (in typical mode using a CM host, this is a
              regression check; from the command line against the minicluster, this
              is a bug verification)
        
            Change-Id: I2a2915e6db85ddb3d8e1bce8035eccd0c9324b4b
            Reviewed-on: http://gerrit.cloudera.org:8080/4555
            Reviewed-by: Michael Brown <mikeb@cloudera.com>
            Reviewed-by: Ishaan Joshi <ishaan@cloudera.com>
            Tested-by: Internal Jenkins
        
        Show
        mikesbrown Michael Brown added a comment - commit a35e4380966346b47a81eeb2500e01bf97fa8f95 Author: Michael Brown <mikeb@cloudera.com> Date: Wed Sep 28 08:29:19 2016 -0700 IMPALA-4207: test infra: move Hive options from connection to cluster options Various test tools and frameworks, including the stress test, random query generator, and nested types loader, share common modules. This change IMPALA-3980: qgen: re-enable Hive as a target database made changes to tests.comparison.cli_options, the shared command line option module, and to tests.comparison.cluster, the shared module for modeling various Impala clusters. Those changes were for the random query generator, but didn't take into account the other shared entry points. It was possible to call some of those entry points in such a way as to produce an exception, because the Hive-related options are now required for miniclusters, but the Hive-related options weren't always being initialized in those entry points. The simple fix is to say that, because Hive settings are now needed to create Minicluster objects, make the Hive options initialized with cluster options, not connection options. While I was making these changes, I fixed all flake8 problems in this file. Testing: - qgen/minicluster unit tests (regression test) - full private data load job, including load_nested.py (bug verification) - data_generator.py run (regression test), long enough to verify connection to the minicluster, using both Hive and Impala - discrepancy_searcher.py run (regression test), long enough verify connection to the minicluster, using both Hive and Impala - concurrent_select.py (in typical mode using a CM host, this is a regression check; from the command line against the minicluster, this is a bug verification) Change-Id: I2a2915e6db85ddb3d8e1bce8035eccd0c9324b4b Reviewed-on: http://gerrit.cloudera.org:8080/4555 Reviewed-by: Michael Brown <mikeb@cloudera.com> Reviewed-by: Ishaan Joshi <ishaan@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            mikesbrown Michael Brown
            Reporter:
            mikesbrown Michael Brown
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development