Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6149

single_node_perf_run.py doesn't work for scale factor 1

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Trivial
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.10.0
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      I tried to do a test run of TPC-H SF 1 using the single_node_perf_run.py script, but run into a few problems:

      • It isn't aware that those databases are specially named and don't include the scale factor in the DB name.
      • When it tries to --load data for SF 1, it gets confused for a similar reason - the preload script generates data into /home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch but the data load wants it to be in a slightly different directory:
        Generating TPC-H data into /home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch
        0: jdbc:hive2://localhost:11050/default> LOAD DATA LOCAL INPATH '/home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch1/lineitem'
        . . . . . . . . . . . . . . . . . . . .> OVERWRITE INTO TABLE tpch1.lineitem;
        going to print operations logs
        printed operations logs
        Getting log thread is interrupted, since query is done!
        Error: Error while compiling statement: FAILED: SemanticException Line 1:23 Invalid path ''/home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch1/lineitem'': No files matching path file:/home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch1/lineitem (state=
        42000,code=40000)
        org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Line 1:23 Invalid path ''/home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch1/lineitem'': No files matching path file:/home/tarmstrong/Impala/incubator-impala/test
        data/impala-data/tpch1/lineitem
                at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:241)
                at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:227)
                at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:255)
                at org.apache.hive.beeline.Commands.executeInternal(Commands.java:989)
                at org.apache.hive.beeline.Commands.execute(Commands.java:1177)
                at org.apache.hive.beeline.Commands.sql(Commands.java:1091)
                at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1176)
                at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1010)
                at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:987)
                at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:914)
                at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:518)
                at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:606)
                at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
                at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
        Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Line 1:23 Invalid path ''/home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch1/lineitem'': No files matching path file:/home/tarmstrong/Impala/incubator-
        impala/testdata/impala-data/tpch1/lineitem
                at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
                at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:187)
                at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:271)
                at org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)
                at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:439)
                at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:416)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:606)
                at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
                at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
                at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:421)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
                at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
                at com.sun.proxy.$Proxy19.executeStatementAsync(Unknown Source)
                at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:282)
                at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:501)
                at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
                at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
                at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
                at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
                at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
                at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                at java.lang.Thread.run(Thread.java:745)
        Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:23 Invalid path ''/home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch1/lineitem'': No files matching path file:/home/tarmstrong/Impala/incubator-impala/testdata/impala-data/tpch1/lineitem
                at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.applyConstraints(LoadSemanticAnalyzer.java:139)
                at org.apache.hadoop.hive.ql.parse.LoadSemanticAnalyzer.analyzeInternal(LoadSemanticAnalyzer.java:218)
                at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)
                at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:558)
                at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1356)
                at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1343)
                at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:185)
                ... 27 more
        
        Closing: 0: jdbc:hive2://localhost:11050/default;auth=none
        Error executing file from Hive: load-tpch-core-hive-generated.sql
        tarmstrong@tarmstrong-box2:~/Impala/incubator-impala$ /home/tarmstrong/Impala/incubator-impala/bin/load-data.py --workloads tpch --scale_factor 1 --table_formats text/none,parquet/none^C
        

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: