Index: . =================================================================== --- . (revision 1661246) +++ . (working copy) Property changes on: . ___________________________________________________________________ Modified: svn:mergeinfo Merged /hive/branches/branch-1.1:r1658284,1659437,1659724 Merged /hive/trunk:r1658024-1661246 Merged /hive/branches/spark:r1657402-1660298 Index: RELEASE_NOTES.txt =================================================================== --- RELEASE_NOTES.txt (revision 1661246) +++ RELEASE_NOTES.txt (working copy) @@ -1,3 +1,481 @@ +Release Notes - Hive - Version 1.1.0 + +** Sub-task + * [HIVE-7073] - Implement Binary in ParquetSerDe + * [HIVE-8121] - Create micro-benchmarks for ParquetSerde and evaluate performance + * [HIVE-8122] - Make use of SearchArgument classes for Parquet SERDE + * [HIVE-8130] - Support Date in Avro + * [HIVE-8131] - Support timestamp in Avro + * [HIVE-8362] - Investigate flaky test parallel.q + * [HIVE-8651] - CBO: sort column changed in infer_bucket_sort test + * [HIVE-8707] - Fix ordering differences due to Java 8 HashMap function + * [HIVE-8718] - Refactoring: move mapLocalWork field from MapWork to BaseWork + * [HIVE-8773] - Fix TestWebHCatE2e#getStatus for Java8 + * [HIVE-8862] - Fix ordering diferences on TestParse tests due to Java8 + * [HIVE-8922] - CBO: assorted date and timestamp issues + * [HIVE-8923] - HIVE-8512 needs to be fixed also for CBO + * [HIVE-8936] - Add SORT_QUERY_RESULTS for join tests that do not guarantee order + * [HIVE-8962] - Add SORT_QUERY_RESULTS for join tests that do not guarantee order #2 + * [HIVE-9030] - CBO: Plans with comparison of values with different types + * [HIVE-9033] - Fix ordering differences due to Java8 (part 2) + * [HIVE-9034] - CBO: type change in literal_ints.q + * [HIVE-9035] - CBO: Disable PPD when functions are non-deterministic (ppd_random.q - non-deterministic udf rand() pushed above join) + * [HIVE-9043] - HiveException: Conflict on row inspector for {table} + * [HIVE-9066] - temporarily disable CBO for non-deterministic functions + * [HIVE-9104] - windowing.q failed when mapred.reduce.tasks is set to larger than one + * [HIVE-9109] - Add support for Java 8 specific q-test out files + * [HIVE-9127] - Improve CombineHiveInputFormat.getSplit performance + * [HIVE-9133] - CBO (Calcite Return Path): Refactor Semantic Analyzer to Move CBO code out + * [HIVE-9153] - Perf enhancement on CombineHiveInputFormat and HiveInputFormat + * [HIVE-9161] - Fix ordering differences on UDF functions due to Java8 + * [HIVE-9181] - Fix SkewJoinOptimizer related Java 8 ordering differences + * [HIVE-9193] - Fix ordering differences due to Java 8 (Part 3) + * [HIVE-9194] - Support select distinct * + * [HIVE-9200] - CBO (Calcite Return Path): Inline Join, Properties + * [HIVE-9206] - Fix Desc Formatted related Java 8 ordering differences + * [HIVE-9222] - Fix ordering differences due to Java 8 (Part 4) + * [HIVE-9224] - CBO (Calcite Return Path): Inline Table, Properties + * [HIVE-9239] - Fix ordering differences due to Java 8 (Part 5) + * [HIVE-9241] - Fix TestCliDriver.testCliDriver_subquery_multiinsert + * [HIVE-9257] - Merge from spark to trunk January 2015 + * [HIVE-9259] - Fix ClassCastException when CBO is enabled for HOS [Spark Branch] + * [HIVE-9264] - Merge encryption branch to trunk + * [HIVE-9292] - CBO (Calcite Return Path): Inline GroupBy, Properties + * [HIVE-9315] - CBO (Calcite Return Path): Inline FileSinkOperator, Properties + * [HIVE-9352] - Merge from spark to trunk (follow-up of HIVE-9257) + * [HIVE-9409] - Avoid ser/de loggers as logging framework can be incompatible on driver and workers + * [HIVE-9410] - ClassNotFoundException occurs during hive query case execution with UDF defined [Spark Branch] + * [HIVE-9428] - LocalSparkJobStatus may return failed job as successful [Spark Branch] + * [HIVE-9431] - CBO (Calcite Return Path): Removing AST from ParseContext + * [HIVE-9434] - Shim the method Path.getPathWithoutSchemeAndAuthority + * [HIVE-9444] - CBO (Calcite Return Path): Rewrite GlobalLimitOptimizer + * [HIVE-9449] - Push YARN configuration to Spark while deply Spark on YARN[Spark Branch] + * [HIVE-9450] - [Parquet] Check all data types work for Parquet in Group By operator + * [HIVE-9477] - No error thrown when global limit optimization failed to find enough number of rows [Spark Branch] + * [HIVE-9487] - Make Remote Spark Context secure [Spark Branch] + * [HIVE-9493] - Failed job may not throw exceptions [Spark Branch] + +** Bug + * [HIVE-1344] - error in select disinct + * [HIVE-1654] - select distinct should allow column name regex + * [HIVE-1869] - TestMTQueries failing on jenkins + * [HIVE-3781] - Index related events should be delivered to metastore event listener + * [HIVE-4009] - CLI Tests fail randomly due to MapReduce LocalJobRunner race condition + * [HIVE-5536] - Incorrect Operation Name is passed to hookcontext + * [HIVE-5631] - Index creation on a skew table fails + * [HIVE-5664] - Drop cascade database fails when the db has any tables with indexes + * [HIVE-5865] - AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' + * [HIVE-6165] - Unify HivePreparedStatement from jdbc:hive and jdbc:hive2 + * [HIVE-6198] - ORC file and struct column names are case sensitive + * [HIVE-6308] - COLUMNS_V2 Metastore table not populated for tables created without an explicit column list. + * [HIVE-6421] - abs() should preserve precision/scale of decimal input + * [HIVE-6468] - HS2 & Metastore using SASL out of memory error when curl sends a get request + * [HIVE-6623] - Add "owner" tag to ptest2 created instances + * [HIVE-6679] - HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable + * [HIVE-6683] - Beeline does not accept comments at end of line + * [HIVE-6914] - parquet-hive cannot write nested map (map value is map) + * [HIVE-7024] - Escape control characters for explain result + * [HIVE-7069] - Zookeeper connection leak + * [HIVE-7270] - SerDe Properties are not considered by show create table Command + * [HIVE-7932] - It may cause NP exception when add accessed columns to ReadEntity + * [HIVE-7951] - InputFormats implementing (Job)Configurable should not be cached + * [HIVE-7997] - Potential null pointer reference in ObjectInspectorUtils#compareTypes() + * [HIVE-8099] - IN operator for partition column fails when the partition column type is DATE + * [HIVE-8182] - beeline fails when executing multiple-line queries with trailing spaces + * [HIVE-8257] - Accumulo introduces old hadoop-client dependency + * [HIVE-8266] - create function using statement compilation should include resource URI entity + * [HIVE-8284] - Equality comparison is done between two floating point variables in HiveRelMdUniqueKeys#getUniqueKeys() + * [HIVE-8295] - Add batch retrieve partition objects for metastore direct sql + * [HIVE-8308] - Acid related table properties should be defined in one place and should be case insensitive + * [HIVE-8317] - WebHCat pom should explicitly depend on jersey-core + * [HIVE-8326] - Using DbTxnManager with concurrency off results in run time error + * [HIVE-8330] - HiveResultSet.findColumn() parameters are case sensitive + * [HIVE-8338] - Add ip and command to semantic analyzer hook context + * [HIVE-8345] - q-test for Avro date support + * [HIVE-8359] - Map containing null values are not correctly written in Parquet files + * [HIVE-8374] - schematool fails on Postgres versions < 9.2 + * [HIVE-8381] - Update hive version on trunk to 0.15 + * [HIVE-8387] - add retry logic to ZooKeeperStorage in WebHCat + * [HIVE-8448] - Union All might not work due to the type conversion issue + * [HIVE-8450] - Create table like does not copy over table properties + * [HIVE-8485] - HMS on Oracle incompatibility + * [HIVE-8491] - Fix build name in ptest pre-commit message + * [HIVE-8500] - beeline does not need to set hive.aux.jars.path + * [HIVE-8512] - queries with star and gby produce incorrect results + * [HIVE-8518] - Compile time skew join optimization returns duplicated results + * [HIVE-8523] - Potential null dereference in DDLSemanticAnalyzer#addInputsOutputsAlterTable() + * [HIVE-8532] - return code of "source xxx" clause is missing + * [HIVE-8556] - introduce overflow control and sanity check to BytesBytesMapJoin + * [HIVE-8564] - DROP TABLE IF EXISTS throws exception if the table does not exist. + * [HIVE-8565] - beeline may go into an infinite loop when using EOF + * [HIVE-8576] - Guaranteed NPE in StatsRulesProcFactory + * [HIVE-8594] - Wrong condition in SettableConfigUpdater#setHiveConfWhiteList() + * [HIVE-8600] - Add option to log explain output for query + * [HIVE-8610] - Compile time skew join optimization doesn't work with auto map join + * [HIVE-8611] - grant/revoke syntax should support additional objects for authorization plugins + * [HIVE-8612] - Support metadata result filter hooks + * [HIVE-8613] - percentile_approx raise a comparator error + * [HIVE-8627] - Compute stats on a table from impala caused the table to be corrupted + * [HIVE-8634] - HiveServer2 fair scheduler queue mapping doesn't handle the secondary groups rules correctly + * [HIVE-8636] - CBO: split cbo_correctness test + * [HIVE-8666] - hive.metastore.server.max.threads default is too high + * [HIVE-8680] - Set Max Message for Binary Thrift endpoints + * [HIVE-8693] - Separate out fair scheduler dependency from hadoop 0.23 shim + * [HIVE-8706] - Table statistic collection on counter failed due to table name character case. + * [HIVE-8708] - Add query id to explain log option + * [HIVE-8715] - Hive 14 upgrade scripts can fail for statistics if database was created using auto-create + * [HIVE-8720] - Update orc_merge tests to make it consistent across OS'es + * [HIVE-8728] - Fix ptf.q determinism + * [HIVE-8730] - schemaTool failure when date partition has non-date value + * [HIVE-8736] - add ordering to cbo_correctness to make result consistent + * [HIVE-8739] - handle Derby and Oracle errors with joins and filters in Direct SQL in a invalid-DB-specific path + * [HIVE-8757] - YARN dep in scheduler shim should be optional + * [HIVE-8762] - HiveMetaStore.BooleanPointer should be replaced with an AtomicBoolean + * [HIVE-8784] - Querying partition does not work with JDO enabled against PostgreSQL + * [HIVE-8791] - Hive permission inheritance throws exception S3 + * [HIVE-8796] - TestCliDriver acid tests with decimal needs benchmark to be updated + * [HIVE-8797] - Simultaneous dynamic inserts can result in "partition already exists" error + * [HIVE-8803] - DESC SCHEMA is not working + * [HIVE-8805] - CBO skipped due to SemanticException: Line 0:-1 Both left and right aliases encountered in JOIN 'avg_cs_ext_discount_amt' + * [HIVE-8808] - HiveInputFormat caching cannot work with all input formats + * [HIVE-8811] - Dynamic partition pruning can result in NPE during query compilation + * [HIVE-8812] - TestMinimrCliDriver failure if run in the same command as TestHBaseNegativeCliDriver + * [HIVE-8825] - SQLCompletor catches Throwable and ignores it + * [HIVE-8827] - Remove SSLv2Hello from list of disabled protocols + * [HIVE-8830] - hcatalog process don't exit because of non daemon thread + * [HIVE-8845] - Switch to Tez 0.5.2 + * [HIVE-8847] - Fix bugs in jenkins scripts + * [HIVE-8848] - data loading from text files or text file processing doesn't handle nulls correctly + * [HIVE-8850] - ObjectStore:: rollbackTransaction() needs to be looked into further. + * [HIVE-8863] - Cannot drop table with uppercase name after "compute statistics for columns" + * [HIVE-8866] - Vectorization on partitioned table throws ArrayIndexOutOfBoundsException when partitions are not of same #of columns + * [HIVE-8869] - RowSchema not updated for some ops when columns are pruned + * [HIVE-8870] - errors when selecting a struct field within an array from ORC based tables + * [HIVE-8872] - Hive view of HBase range scan intermittently returns incorrect data. + * [HIVE-8873] - Switch to calcite 0.9.2 + * [HIVE-8874] - Error Accessing HBase from Hive via Oozie on Kerberos 5.0.1 cluster + * [HIVE-8875] - hive.optimize.sort.dynamic.partition should be turned off for ACID + * [HIVE-8876] - incorrect upgrade script for Oracle (13->14) + * [HIVE-8877] - improve context logging during job submission via WebHCat + * [HIVE-8879] - Upgrade derby version to address race candition + * [HIVE-8880] - non-synchronized access to split list in OrcInputFormat + * [HIVE-8881] - Receiving json "{"error":"Could not find job job_1415748506143_0002"}" when web client tries to fetch all jobs from webhcat where HDFS does not have the data. + * [HIVE-8886] - Some Vectorized String CONCAT expressions result in runtime error Vectorization: Unsuported vector output type: StringGroup + * [HIVE-8888] - Mapjoin with LateralViewJoin generates wrong plan in Tez + * [HIVE-8889] - JDBC Driver ResultSet.getXXXXXX(String columnLabel) methods Broken + * [HIVE-8890] - HiveServer2 dynamic service discovery: use persistent ephemeral nodes curator recipe + * [HIVE-8891] - Another possible cause to NucleusObjectNotFoundException from drops/rollback + * [HIVE-8893] - Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode + * [HIVE-8901] - increase retry attempt, interval on metastore database errors + * [HIVE-8909] - Hive doesn't correctly read Parquet nested types + * [HIVE-8914] - HDFSCleanup thread holds reference to FileSystem + * [HIVE-8916] - Handle user@domain username under LDAP authentication + * [HIVE-8917] - HIVE-5679 adds two thread safety problems + * [HIVE-8926] - Projections that only swap input columns are identified incorrectly as identity projections + * [HIVE-8938] - Compiler should save the transform URI as input entity + * [HIVE-8944] - TestCompactor fails with IncompatibleClassChangeError + * [HIVE-8947] - HIVE-8876 also affects Postgres < 9.2 + * [HIVE-8948] - TestStreaming is flaky + * [HIVE-8964] - Some TestMiniTezCliDriver tests taking two hours + * [HIVE-8965] - Enhance PTest to kill all processes between tests and to report when a TEST*.xml file is not generated + * [HIVE-8967] - Fix bucketmapjoin7.q determinism + * [HIVE-8975] - Possible performance regression on bucket_map_join_tez2.q + * [HIVE-8978] - Fix test determinism issue for qfile: smb_mapjoin_1.q etc + * [HIVE-8990] - mapjoin_mapjoin.q is failing on Tez (missed golden file update) + * [HIVE-9001] - Ship with log4j.properties file that has a reliable time based rolling policy + * [HIVE-9003] - Vectorized IF expr broken for the scalar and scalar case + * [HIVE-9006] - hiveserver thrift api version is still 6 + * [HIVE-9011] - Fix parquet_join.q determinism + * [HIVE-9024] - NullPointerException when starting webhcat server if templeton.hive.properties is not set + * [HIVE-9025] - join38.q (without map join) produces incorrect result when testing with multiple reducers + * [HIVE-9032] - Help for orcfiledump script does not reflect new options + * [HIVE-9048] - Hive build failed on hadoop-1 after HIVE-8828. + * [HIVE-9051] - TezJobMonitor in-place updates logs too often to logfile + * [HIVE-9053] - select constant in union all followed by group by gives wrong result + * [HIVE-9055] - Tez: union all followed by group by followed by another union all gives error + * [HIVE-9060] - Fix child operator references after NonBlockingOpDeDupProc + * [HIVE-9067] - OrcFileMergeOperator may create merge file that does not match properties of input files + * [HIVE-9077] - Set completer in CliDriver is not working + * [HIVE-9090] - Rename "Tez File Merge Work" to smaller name + * [HIVE-9096] - GenericUDF may be left unclosed in PartitionPrune#visitCall() + * [HIVE-9108] - Fix for HIVE-8735 is incorrect (stats with long paths) + * [HIVE-9111] - Potential NPE in OrcStruct for list and map types + * [HIVE-9112] - Query may generate different results depending on the number of reducers + * [HIVE-9113] - Explain on query failed with NPE + * [HIVE-9114] - union all query in cbo test has undefined ordering + * [HIVE-9120] - Hive Query log does not work when hive.exec.parallel is true + * [HIVE-9122] - Need to remove additional references to hive-shims-common-secure, hive-shims-0.20 + * [HIVE-9126] - Backport HIVE-8827 (Remove SSLv2Hello from list of disabled protocols) to 0.14 branch + * [HIVE-9129] - Migrate to newer Calcite snapshot, where ByteString is now in org.apache.calcite.avatica.util + * [HIVE-9130] - vector_partition_diff_num_cols result is not updated after CBO upgrade + * [HIVE-9131] - MiniTez optimize_nullscan test is unstable + * [HIVE-9141] - HiveOnTez: mix of union all, distinct, group by generates error + * [HIVE-9149] - Add unit test to test implicit conversion during dynamic partitioning/distribute by + * [HIVE-9150] - Unrelated types are compared in GenTezWork#getFollowingWorkIndex() + * [HIVE-9154] - Cache pathToPartitionInfo in context aware record reader + * [HIVE-9155] - HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql + * [HIVE-9162] - stats19 test is environment-dependant + * [HIVE-9166] - Place an upper bound for SARG CNF conversion + * [HIVE-9168] - Vectorized Coalesce for strings is broken + * [HIVE-9177] - Fix child operator references after NonBlockingOpDeDupProc (II) + * [HIVE-9195] - CBO changes constant to column type + * [HIVE-9197] - fix lvj_mapjoin.q diff in trunk + * [HIVE-9199] - Excessive exclusive lock used in some DDLs with DummyTxnManager + * [HIVE-9203] - CREATE TEMPORARY FUNCTION hangs trying to acquire lock + * [HIVE-9205] - Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist + * [HIVE-9215] - Some mapjoin queries broken with IdentityProjectRemover with PPD + * [HIVE-9221] - Remove deprecation warning for hive.metastore.local + * [HIVE-9234] - HiveServer2 leaks FileSystem objects in FileSystem.CACHE + * [HIVE-9242] - Many places in CBO code eat exceptions + * [HIVE-9243] - Static Map in IOContext is not thread safe + * [HIVE-9249] - java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables + * [HIVE-9255] - Fastpath for limited fetches from unpartitioned tables + * [HIVE-9296] - Need to add schema upgrade changes for queueing events in the database + * [HIVE-9299] - Reuse Configuration in AvroSerdeUtils + * [HIVE-9300] - Make TCompactProtocol configurable + * [HIVE-9301] - Potential null dereference in MoveTask#createTargetPath() + * [HIVE-9309] - schematool fails on Postgres 8.1 + * [HIVE-9310] - CLI JLine does not flush history back to ~/.hivehistory + * [HIVE-9316] - TestSqoop tests in WebHCat testsuite hardcode libdir path to hdfs + * [HIVE-9317] - move Microsoft copyright to NOTICE file + * [HIVE-9321] - Notification message size can be arbitrarily long, DbNotificationListener limits to 1024 + * [HIVE-9330] - DummyTxnManager will throw NPE if WriteEntity writeType has not been set + * [HIVE-9331] - get rid of pre-optimized-hashtable memory optimizations + * [HIVE-9344] - Fix flaky test optimize_nullscan + * [HIVE-9347] - Bug with max() together with rank() and grouping sets + * [HIVE-9351] - Running Hive Jobs with Tez cause templeton to never report percent complete + * [HIVE-9353] - make TABLE keyword optional in INSERT INTO TABLE foo... + * [HIVE-9359] - Export of a large table causes OOM in Metastore and Client + * [HIVE-9361] - Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable + * [HIVE-9366] - wrong date in description annotation in date_add() and date_sub() udf + * [HIVE-9369] - fix arguments length checking in Upper and Lower UDF + * [HIVE-9377] - UDF in_file() in WHERE predicate causes NPE. + * [HIVE-9381] - HCatalog hardcodes maximum append limit to 1000. + * [HIVE-9382] - Query got rerun with Global Limit optimization on and Fetch optimization off + * [HIVE-9386] - FileNotFoundException when using in_file() + * [HIVE-9390] - Enhance retry logic wrt DB access in TxnHandler + * [HIVE-9393] - reduce noisy log level of ColumnarSerDe.java:116 from INFO to DEBUG + * [HIVE-9396] - date_add()/date_sub() should allow tinyint/smallint/bigint arguments in addition to int + * [HIVE-9401] - Backport: Fastpath for limited fetches from unpartitioned tables + * [HIVE-9404] - NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct() + * [HIVE-9414] - Fixup post HIVE-9264 - Merge encryption branch to trunk + * [HIVE-9437] - Beeline does not add any existing HADOOP_CLASSPATH + * [HIVE-9440] - Folders may not be pruned for Hadoop 2 + * [HIVE-9441] - Remove call to deprecated Calcite method + * [HIVE-9443] - ORC PPD - fix fuzzy case evaluation of IS_NULL + * [HIVE-9445] - Revert HIVE-5700 - enforce single date format for partition column storage + * [HIVE-9446] - JDBC DatabaseMetadata.getColumns() does not work for temporary tables + * [HIVE-9448] - Merge spark to trunk 1/23/15 + * [HIVE-9454] - Test failures due to new Calcite version + * [HIVE-9462] - HIVE-8577 - breaks type evolution + * [HIVE-9473] - sql std auth should disallow built-in udfs that allow any java methods to be called + * [HIVE-9475] - HiveMetastoreClient.tableExists does not work + * [HIVE-9476] - Beeline fails to start on trunk + * [HIVE-9502] - Parquet cannot read Map types from files written with Hive <= 0.12 + * [HIVE-9514] - schematool is broken in hive 1.0.0 + * [HIVE-9519] - Bump up spark client connection timeout + * [HIVE-9593] - ORC Reader should ignore unknown metadata streams + * [HIVE-9610] - Continuation of HIVE-9438 - The standalone-jdbc jar missing some classes + * [HIVE-9611] - Allow SPARK_HOME as well as spark.home to define sparks location + * [HIVE-9621] - HiveServer2 http mode - embedded jetty should use SynchronousQueue + * [HIVE-9646] - Beeline doesn't show Spark job progress info [Spark Branch] + * [HIVE-9651] - FileSinkOperator does not pass in conf to serde.initialize + +** Improvement + * [HIVE-2828] - make timestamp accessible in the hbase KeyValue + * [HIVE-3187] - support ISO-2012 timestamp literals + * [HIVE-3280] - Make HiveMetaStoreClient a public API + * [HIVE-4639] - Add has null flag to ORC internal index + * [HIVE-4766] - Support HS2 client login timeout when the thrift thread max# is reached + * [HIVE-4809] - ReduceSinkOperator of PTFOperator can have redundant key columns + * [HIVE-5718] - Support direct fetch for lateral views, sub queries, etc. + * [HIVE-6148] - Support arbitrary structs stored in HBase + * [HIVE-7032] - Remove Triple Negative in Error 10129 + * [HIVE-7313] - Allow in-memory/ssd session-level temp-tables + * [HIVE-7408] - HCatPartition needs getPartCols method + * [HIVE-7550] - Extend cached evaluation to multiple expressions + * [HIVE-7605] - add more javadoc to HivePrivilegeObject + * [HIVE-7685] - Parquet memory manager + * [HIVE-7858] - Parquet compression should be configurable via table property + * [HIVE-7868] - AvroSerDe error handling could be improved + * [HIVE-7896] - orcfiledump should be able to dump data + * [HIVE-7977] - Avoid creating serde for partitions if possible in FetchTask + * [HIVE-8094] - add LIKE keyword support for SHOW FUNCTIONS + * [HIVE-8155] - In select statement after * any random characters are allowed in hive but in RDBMS its not allowed + * [HIVE-8337] - Change default of hive.warehouse.subdir.inherit.perms to true + * [HIVE-8347] - Use base-64 encoding instead of custom encoding for serialized objects + * [HIVE-8357] - Path type entities should use qualified path rather than string + * [HIVE-8395] - CBO: enable by default + * [HIVE-8424] - Support fair scheduler user queue mapping in non-impersonation mode + * [HIVE-8449] - webhcat startup does not create the log directory + * [HIVE-8469] - Add parquet.compression as a Serde Property + * [HIVE-8552] - Remove hard code of offline and make addHost api public + * [HIVE-8642] - Hive stack() UDTF Doesn't Support NULL Insert Values + * [HIVE-8661] - JDBC MinimizeJAR should be configurable in pom.xml + * [HIVE-8710] - Add more tests for transactional inserts + * [HIVE-8813] - Allow tests to be excluded based on pattern/regex + * [HIVE-8823] - Add additional serde properties for parquet + * [HIVE-8829] - Upgrade to Thrift 0.9.2 + * [HIVE-8839] - Support "alter table .. add/replace columns cascade" + * [HIVE-8910] - Refactoring of PassThroughOutputFormat + * [HIVE-8933] - Check release builds for SNAPSHOT dependencies + * [HIVE-8971] - HIVE-8965 exposed some classes which start with Test but are not tests + * [HIVE-8976] - Make nine additional tests deterministic + * [HIVE-8977] - TestParquetDirect should be abstract + * [HIVE-8983] - PTest Backup spark.log in addition to hive.log + * [HIVE-8988] - Support advanced aggregation in Hive to Calcite path + * [HIVE-8996] - Rename getUGIForConf + * [HIVE-9037] - Improve explain plan to show joining keys for shuffle join + * [HIVE-9049] - Metastore should use TCompactProtocol as opposed to TBinaryProtocol + * [HIVE-9062] - Explain plan doesn't print join keys for Tez shuffle join + * [HIVE-9106] - improve the performance of null scan optimizer when several table scans share a physical path + * [HIVE-9119] - ZooKeeperHiveLockManager does not use zookeeper in the proper way + * [HIVE-9140] - Add Calcite's ReduceExpressionRules to Hive + * [HIVE-9158] - Multiple LDAP server URLs in hive.server2.authentication.ldap.url + * [HIVE-9176] - Delegation token interval should be configurable in HadoopThriftAuthBridge + * [HIVE-9189] - Add ProjectRemove rule on CBO path + * [HIVE-9226] - Beeline interweaves the query result and query log sometimes + * [HIVE-9244] - Upgrade 0.23 hadoop-shims to latest stable hadoop-2.6.0 + * [HIVE-9314] - Writable object inspector should use Writable classes from Hadoop for get + * [HIVE-9318] - Add UnionMerge rule on cbo path + * [HIVE-9341] - Apply ColumnPrunning for noop PTFs + * [HIVE-9357] - Create ADD_MONTHS UDF + * [HIVE-9358] - Create LAST_DAY UDF + * [HIVE-9367] - CombineFileInputFormatShim#getDirIndices is expensive + * [HIVE-9372] - Parallel checking non-combinable paths in CombineHiveInputFormat + * [HIVE-9383] - Improve schema verification error message + * [HIVE-9538] - Exclude thirdparty directory from tarballs + * [HIVE-9549] - Include missing directories in source tarball + * [HIVE-9586] - Too verbose log can hurt performance, we should always check log level first + * [HIVE-9608] - Define SPARK_HOME if not defined automagically + +** New Feature + * [HIVE-3405] - UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase + * [HIVE-7122] - Storage format for create like table + * [HIVE-8435] - Add identity project remover optimization + * [HIVE-9174] - Enable queuing of HCatalog notification events in metastore DB + * [HIVE-9175] - Add alters to list of events handled by NotificationListener + * [HIVE-9184] - Modify HCatClient to support new notification methods in HiveMetaStoreClient + +** Task + * [HIVE-6977] - Delete Hiveserver1 + * [HIVE-7111] - Extend join transitivity PPD to non-column expressions + * [HIVE-8351] - Allow testing of both hadoop-1 and hadoop-2 for precommits + * [HIVE-8370] - Fix tabs in DefaultHBaseValueFactory + * [HIVE-8465] - Fix some minor test fails on trunk + * [HIVE-8608] - Move jenkins scripts to source control + * [HIVE-8609] - Move beeline to jline2 + * [HIVE-8826] - Remove jdbm from top level license file + * [HIVE-8828] - Remove hadoop 20 shims + * [HIVE-8894] - Move calcite.version to root pom + * [HIVE-8935] - Add debug logging around token stores + * [HIVE-8974] - Upgrade to Calcite 1.0.0-SNAPSHOT (with lots of renames) + * [HIVE-8979] - Merge shims/common-secure into shims/common + * [HIVE-8989] - Make groupby_multi_single_reducer.q and smb_mapjoin_3.q deterministic + * [HIVE-9183] - Compilation against HBase 1.0.0 fails due to missing dependency on high_scale_lib.Counter + * [HIVE-9304] - [Refactor] remove unused method in SemAly + * [HIVE-9345] - Add encryption branch to pre-commit script + * [HIVE-9402] - Create GREATEST and LEAST udf + * [HIVE-9407] - Fix spacing in pom + * [HIVE-9408] - Add hook interface so queries can be redacted before being placed in job.xml + * [HIVE-9527] - Include dot files in tarball + * [HIVE-9554] - Rename 0.15 upgrade scripts to 1.1 + * [HIVE-9672] - Update RELEASE_NOTES on trunk to reflect the 1.0.0 release + +** Test + * [HIVE-7924] - auto_sortmerge_join_8 sometimes fails with OOM + * [HIVE-8256] - Add SORT_QUERY_RESULTS for test that doesn't guarantee order #2 + * [HIVE-8327] - mvn site -Pfindbugs + * [HIVE-8573] - Fix some non-deterministic vectorization tests + * [HIVE-8601] - Fix auto_sortmerge_join_16 determinism + * [HIVE-8602] - Add SORT_QUERY_RESULTS for skewjoinopt2 + * [HIVE-8753] - TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce failing on trunk + * [HIVE-8801] - Make orc_merge_incompat1.q deterministic across platforms + * [HIVE-8929] - incorrect error message for cbo path + * [HIVE-9091] - Add additional unit tests for HiveSQLException + * [HIVE-9173] - Fix auto_join29.q, auto_join_without_localtask.q determinism + * [HIVE-9290] - Make some test results deterministic + * [HIVE-9360] - TestSparkClient throws Timeoutexception + * [HIVE-9403] - File tests determinism with multiple reducers + * [HIVE-9417] - Fix failing test groupby_grouping_window.q on trunk + * [HIVE-9435] - Fix auto_join21.q for Tez + + + +Release Notes - Hive - Version 1.0.0 + +** Bug + * [HIVE-5631] - Index creation on a skew table fails + * [HIVE-5664] - Drop cascade database fails when the db has any tables with indexes + * [HIVE-6198] - ORC file and struct column names are case sensitive + * [HIVE-6468] - HS2 & Metastore using SASL out of memory error when curl sends a get request + * [HIVE-7270] - SerDe Properties are not considered by show create table Command + * [HIVE-8099] - IN operator for partition column fails when the partition column type is DATE + * [HIVE-8295] - Add batch retrieve partition objects for metastore direct sql + * [HIVE-8374] - schematool fails on Postgres versions < 9.2 + * [HIVE-8485] - HMS on Oracle incompatibility + * [HIVE-8706] - Table statistic collection on counter failed due to table name character case. + * [HIVE-8715] - Hive 14 upgrade scripts can fail for statistics if database was created using auto-create + * [HIVE-8739] - handle Derby and Oracle errors with joins and filters in Direct SQL in a invalid-DB-specific path + * [HIVE-8784] - Querying partition does not work with JDO enabled against PostgreSQL + * [HIVE-8805] - CBO skipped due to SemanticException: Line 0:-1 Both left and right aliases encountered in JOIN 'avg_cs_ext_discount_amt' + * [HIVE-8807] - Obsolete default values in webhcat-default.xml + * [HIVE-8811] - Dynamic partition pruning can result in NPE during query compilation + * [HIVE-8827] - Remove SSLv2Hello from list of disabled protocols + * [HIVE-8830] - hcatalog process don't exit because of non daemon thread + * [HIVE-8845] - Switch to Tez 0.5.2 + * [HIVE-8866] - Vectorization on partitioned table throws ArrayIndexOutOfBoundsException when partitions are not of same #of columns + * [HIVE-8870] - errors when selecting a struct field within an array from ORC based tables + * [HIVE-8873] - Switch to calcite 0.9.2 + * [HIVE-8876] - incorrect upgrade script for Oracle (13->14) + * [HIVE-8880] - non-synchronized access to split list in OrcInputFormat + * [HIVE-8886] - Some Vectorized String CONCAT expressions result in runtime error Vectorization: Unsuported vector output type: StringGroup + * [HIVE-8888] - Mapjoin with LateralViewJoin generates wrong plan in Tez + * [HIVE-8947] - HIVE-8876 also affects Postgres < 9.2 + * [HIVE-8966] - Delta files created by hive hcatalog streaming cannot be compacted + * [HIVE-9003] - Vectorized IF expr broken for the scalar and scalar case + * [HIVE-9025] - join38.q (without map join) produces incorrect result when testing with multiple reducers + * [HIVE-9038] - Join tests fail on Tez + * [HIVE-9051] - TezJobMonitor in-place updates logs too often to logfile + * [HIVE-9053] - select constant in union all followed by group by gives wrong result + * [HIVE-9067] - OrcFileMergeOperator may create merge file that does not match properties of input files + * [HIVE-9090] - Rename "Tez File Merge Work" to smaller name + * [HIVE-9108] - Fix for HIVE-8735 is incorrect (stats with long paths) + * [HIVE-9111] - Potential NPE in OrcStruct for list and map types + * [HIVE-9112] - Query may generate different results depending on the number of reducers + * [HIVE-9114] - union all query in cbo test has undefined ordering + * [HIVE-9126] - Backport HIVE-8827 (Remove SSLv2Hello from list of disabled protocols) to 0.14 branch + * [HIVE-9141] - HiveOnTez: mix of union all, distinct, group by generates error + * [HIVE-9155] - HIVE_LOCKS uses int instead of bigint hive-txn-schema-0.14.0.mssql.sql + * [HIVE-9162] - stats19 test is environment-dependant + * [HIVE-9166] - Place an upper bound for SARG CNF conversion + * [HIVE-9168] - Vectorized Coalesce for strings is broken + * [HIVE-9205] - Change default tez install directory to use /tmp instead of /user and create the directory if it does not exist + * [HIVE-9234] - HiveServer2 leaks FileSystem objects in FileSystem.CACHE + * [HIVE-9249] - java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org.apache.hadoop.hive.common.type.HiveVarchar when joining tables + * [HIVE-9278] - Cached expression feature broken in one case + * [HIVE-9317] - move Microsoft copyright to NOTICE file + * [HIVE-9359] - Export of a large table causes OOM in Metastore and Client + * [HIVE-9361] - Intermittent NPE in SessionHiveMetaStoreClient.alterTempTable + * [HIVE-9390] - Enhance retry logic wrt DB access in TxnHandler + * [HIVE-9401] - Backport: Fastpath for limited fetches from unpartitioned tables + * [HIVE-9404] - NPE in org.apache.hadoop.hive.metastore.txn.TxnHandler.determineDatabaseProduct() + * [HIVE-9436] - RetryingMetaStoreClient does not retry JDOExceptions + * [HIVE-9473] - sql std auth should disallow built-in udfs that allow any java methods to be called + * [HIVE-9514] - schematool is broken in hive 1.0.0 + + +** Improvement + * [HIVE-3280] - Make HiveMetaStoreClient a public API + * [HIVE-8933] - Check release builds for SNAPSHOT dependencies + + +** Task + * [HIVE-6977] - Delete Hiveserver1 + + + + Release Notes - Hive - Version 0.14.0 ** Sub-task @@ -4498,4 +4976,3 @@ * [HIVE-1359] - Unit test should be shim-aware - Index: accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloRowSerializer.java =================================================================== --- accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloRowSerializer.java (revision 1661246) +++ accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloRowSerializer.java (working copy) @@ -30,8 +30,8 @@ import org.apache.hadoop.hive.serde2.ByteStream; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeUtils; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; import org.apache.hadoop.hive.serde2.lazy.LazyUtils; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; @@ -56,12 +56,12 @@ private final int rowIdOffset; private final ByteStream.Output output; - private final SerDeParameters serDeParams; + private final LazySerDeParameters serDeParams; private final List mappings; private final ColumnVisibility visibility; private final AccumuloRowIdFactory rowIdFactory; - public AccumuloRowSerializer(int primaryKeyOffset, SerDeParameters serDeParams, + public AccumuloRowSerializer(int primaryKeyOffset, LazySerDeParameters serDeParams, List mappings, ColumnVisibility visibility, AccumuloRowIdFactory rowIdFactory) { Preconditions.checkArgument(primaryKeyOffset >= 0, "A valid offset to the mapping for the Accumulo RowID is required, received " @@ -254,7 +254,7 @@ /** * Recursively serialize an Object using its {@link ObjectInspector}, respecting the - * separators defined by the {@link SerDeParameters}. + * separators defined by the {@link LazySerDeParameters}. * @param oi ObjectInspector for the current object * @param value The current object * @param output A buffer output is written to Index: accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDe.java =================================================================== --- accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDe.java (revision 1661246) +++ accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDe.java (working copy) @@ -32,7 +32,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeStats; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; @@ -57,7 +57,7 @@ public void initialize(Configuration conf, Properties properties) throws SerDeException { accumuloSerDeParameters = new AccumuloSerDeParameters(conf, properties, getClass().getName()); - final SerDeParameters serDeParams = accumuloSerDeParameters.getSerDeParameters(); + final LazySerDeParameters serDeParams = accumuloSerDeParameters.getSerDeParameters(); final List mappings = accumuloSerDeParameters.getColumnMappings(); final List columnTypes = accumuloSerDeParameters.getHiveColumnTypes(); final AccumuloRowIdFactory factory = accumuloSerDeParameters.getRowIdFactory(); @@ -83,7 +83,7 @@ } protected ArrayList getColumnObjectInspectors(List columnTypes, - SerDeParameters serDeParams, List mappings, AccumuloRowIdFactory factory) + LazySerDeParameters serDeParams, List mappings, AccumuloRowIdFactory factory) throws SerDeException { ArrayList columnObjectInspectors = new ArrayList( columnTypes.size()); Index: accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDeParameters.java =================================================================== --- accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDeParameters.java (revision 1661246) +++ accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/AccumuloSerDeParameters.java (working copy) @@ -30,8 +30,7 @@ import org.apache.hadoop.hive.accumulo.columns.HiveAccumuloRowIdColumnMapping; import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.SerDeException; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; import org.apache.hadoop.util.ReflectionUtils; import org.apache.log4j.Logger; @@ -62,7 +61,7 @@ private Properties tableProperties; private String serdeName; - private SerDeParameters lazySerDeParameters; + private LazySerDeParameters lazySerDeParameters; private AccumuloRowIdFactory rowIdFactory; public AccumuloSerDeParameters(Configuration conf, Properties tableProperties, String serdeName) @@ -71,7 +70,7 @@ this.tableProperties = tableProperties; this.serdeName = serdeName; - lazySerDeParameters = LazySimpleSerDe.initSerdeParams(conf, tableProperties, serdeName); + lazySerDeParameters = new LazySerDeParameters(conf, tableProperties, serdeName); // The default encoding for this table when not otherwise specified String defaultStorage = tableProperties.getProperty(DEFAULT_STORAGE_TYPE); @@ -134,7 +133,7 @@ return new DefaultAccumuloRowIdFactory(); } - public SerDeParameters getSerDeParameters() { + public LazySerDeParameters getSerDeParameters() { return lazySerDeParameters; } Index: accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java =================================================================== --- accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java (revision 1661246) +++ accumulo-handler/src/java/org/apache/hadoop/hive/accumulo/serde/DefaultAccumuloRowIdFactory.java (working copy) @@ -29,7 +29,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; import org.apache.hadoop.hive.serde2.lazy.LazyObjectBase; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.StructField; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; @@ -42,7 +42,7 @@ public class DefaultAccumuloRowIdFactory implements AccumuloRowIdFactory { protected AccumuloSerDeParameters accumuloSerDeParams; - protected LazySimpleSerDe.SerDeParameters serdeParams; + protected LazySerDeParameters serdeParams; protected Properties properties; protected HiveAccumuloRowIdColumnMapping rowIdMapping; protected AccumuloRowSerializer serializer; Index: accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestLazyAccumuloRow.java =================================================================== --- accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestLazyAccumuloRow.java (revision 1661246) +++ accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/TestLazyAccumuloRow.java (working copy) @@ -33,7 +33,7 @@ import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; import org.apache.hadoop.hive.serde2.lazy.LazyInteger; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.LazyString; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector; import org.apache.hadoop.hive.serde2.lazydio.LazyDioInteger; @@ -58,7 +58,7 @@ TypeInfoFactory.intTypeInfo, TypeInfoFactory.intTypeInfo, TypeInfoFactory.intTypeInfo); LazySimpleStructObjectInspector objectInspector = (LazySimpleStructObjectInspector) LazyFactory - .createLazyStructInspector(columns, types, LazySimpleSerDe.DefaultSeparators, new Text( + .createLazyStructInspector(columns, types, LazySerDeParameters.DefaultSeparators, new Text( "\\N"), false, false, (byte) '\\'); DefaultAccumuloRowIdFactory rowIdFactory = new DefaultAccumuloRowIdFactory(); @@ -119,7 +119,7 @@ TypeInfoFactory.intTypeInfo, TypeInfoFactory.intTypeInfo, TypeInfoFactory.intTypeInfo); LazySimpleStructObjectInspector objectInspector = (LazySimpleStructObjectInspector) LazyFactory - .createLazyStructInspector(columns, types, LazySimpleSerDe.DefaultSeparators, new Text( + .createLazyStructInspector(columns, types, LazySerDeParameters.DefaultSeparators, new Text( "\\N"), false, false, (byte) '\\'); DefaultAccumuloRowIdFactory rowIdFactory = new DefaultAccumuloRowIdFactory(); @@ -202,7 +202,7 @@ TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME)); LazySimpleStructObjectInspector objectInspector = (LazySimpleStructObjectInspector) LazyFactory - .createLazyStructInspector(columns, types, LazySimpleSerDe.DefaultSeparators, new Text( + .createLazyStructInspector(columns, types, LazySerDeParameters.DefaultSeparators, new Text( "\\N"), false, false, (byte) '\\'); DefaultAccumuloRowIdFactory rowIdFactory = new DefaultAccumuloRowIdFactory(); Index: accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTableOutputFormat.java =================================================================== --- accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTableOutputFormat.java (revision 1661246) +++ accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/mr/TestHiveAccumuloTableOutputFormat.java (working copy) @@ -43,7 +43,7 @@ import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.lazy.ByteArrayRef; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.LazyStruct; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory; @@ -150,7 +150,7 @@ tableProperties.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(',').join(types)); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); AccumuloRowSerializer serializer = new AccumuloRowSerializer(0, serDeParams, accumuloSerDeParams.getColumnMappings(), AccumuloSerDeParameters.DEFAULT_VISIBILITY_LABEL, @@ -243,7 +243,7 @@ tableProperties.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(',').join(types)); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); AccumuloRowSerializer serializer = new AccumuloRowSerializer(0, serDeParams, accumuloSerDeParams.getColumnMappings(), new ColumnVisibility("foo"), @@ -332,7 +332,7 @@ tableProperties.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(',').join(types)); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); AccumuloRowSerializer serializer = new AccumuloRowSerializer(0, serDeParams, accumuloSerDeParams.getColumnMappings(), AccumuloSerDeParameters.DEFAULT_VISIBILITY_LABEL, @@ -432,7 +432,7 @@ tableProperties.setProperty(AccumuloSerDeParameters.DEFAULT_STORAGE_TYPE, ColumnEncoding.BINARY.getName()); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); AccumuloRowSerializer serializer = new AccumuloRowSerializer(0, serDeParams, accumuloSerDeParams.getColumnMappings(), AccumuloSerDeParameters.DEFAULT_VISIBILITY_LABEL, Index: accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloRowSerializer.java =================================================================== --- accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloRowSerializer.java (revision 1661246) +++ accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloRowSerializer.java (working copy) @@ -35,7 +35,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.lazy.ByteArrayRef; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.LazyStruct; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory; @@ -109,7 +109,7 @@ tableProperties.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(',').join(typeNames)); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); LazySimpleStructObjectInspector oi = (LazySimpleStructObjectInspector) LazyFactory .createLazyStructInspector(columns, types, serDeParams.getSeparators(), @@ -178,7 +178,7 @@ tableProperties.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(',').join(typeNames)); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); LazySimpleStructObjectInspector oi = (LazySimpleStructObjectInspector) LazyFactory .createLazyStructInspector(columns, types, serDeParams.getSeparators(), @@ -251,7 +251,7 @@ tableProperties.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(',').join(typeNames)); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); TypeInfo stringTypeInfo = TypeInfoFactory.getPrimitiveTypeInfo(serdeConstants.STRING_TYPE_NAME); LazyStringObjectInspector stringOI = (LazyStringObjectInspector) LazyFactory Index: accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloSerDe.java =================================================================== --- accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloSerDe.java (revision 1661246) +++ accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestAccumuloSerDe.java (working copy) @@ -44,7 +44,7 @@ import org.apache.hadoop.hive.serde2.lazy.LazyArray; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; import org.apache.hadoop.hive.serde2.lazy.LazyMap; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.LazyString; import org.apache.hadoop.hive.serde2.lazy.LazyStruct; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector; @@ -389,7 +389,7 @@ tableProperties.setProperty(serdeConstants.LIST_COLUMN_TYPES, Joiner.on(',').join(types)); AccumuloSerDeParameters accumuloSerDeParams = new AccumuloSerDeParameters(new Configuration(), tableProperties, AccumuloSerDe.class.getSimpleName()); - SerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDeParams.getSerDeParameters(); byte[] seps = serDeParams.getSeparators(); Index: accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestDefaultAccumuloRowIdFactory.java =================================================================== --- accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestDefaultAccumuloRowIdFactory.java (revision 1661246) +++ accumulo-handler/src/test/org/apache/hadoop/hive/accumulo/serde/TestDefaultAccumuloRowIdFactory.java (working copy) @@ -25,7 +25,7 @@ import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.lazy.LazyObjectBase; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.LazyString; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector; @@ -33,8 +33,6 @@ import org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyPrimitiveObjectInspectorFactory; import org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; import org.junit.Assert; import org.junit.Test; @@ -60,7 +58,7 @@ AccumuloRowIdFactory factory = accumuloSerDe.getParams().getRowIdFactory(); List columnTypes = accumuloSerDe.getParams().getHiveColumnTypes(); ColumnMapper mapper = accumuloSerDe.getParams().getColumnMapper(); - SerDeParameters serDeParams = accumuloSerDe.getParams().getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDe.getParams().getSerDeParameters(); List OIs = accumuloSerDe.getColumnObjectInspectors(columnTypes, serDeParams, mapper.getColumnMappings(), factory); @@ -85,7 +83,7 @@ AccumuloRowIdFactory factory = accumuloSerDe.getParams().getRowIdFactory(); List columnTypes = accumuloSerDe.getParams().getHiveColumnTypes(); ColumnMapper mapper = accumuloSerDe.getParams().getColumnMapper(); - SerDeParameters serDeParams = accumuloSerDe.getParams().getSerDeParameters(); + LazySerDeParameters serDeParams = accumuloSerDe.getParams().getSerDeParameters(); List OIs = accumuloSerDe.getColumnObjectInspectors(columnTypes, serDeParams, mapper.getColumnMappings(), factory); Index: bin/ext/beeline.sh =================================================================== --- bin/ext/beeline.sh (revision 1661246) +++ bin/ext/beeline.sh (working copy) @@ -25,7 +25,12 @@ superCsvJarPath=`ls ${HIVE_LIB}/super-csv-*.jar` jlineJarPath=`ls ${HIVE_LIB}/jline-*.jar` jdbcStandaloneJarPath=`ls ${HIVE_LIB}/hive-jdbc-*-standalone.jar` - export HADOOP_CLASSPATH=${HIVE_CONF_DIR}:${beelineJarPath}:${superCsvJarPath}:${jlineJarPath}:${jdbcStandaloneJarPath} + hadoopClasspath="" + if [[ -n "${HADOOP_CLASSPATH}" ]] + then + hadoopClasspath="${HADOOP_CLASSPATH}:" + fi + export HADOOP_CLASSPATH="${hadoopClasspath}${HIVE_CONF_DIR}:${beelineJarPath}:${superCsvJarPath}:${jlineJarPath}:${jdbcStandaloneJarPath}" export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dlog4j.configuration=beeline-log4j.properties " exec $HADOOP jar ${beelineJarPath} $CLASS $HIVE_OPTS "$@" Index: bin/hive =================================================================== --- bin/hive (revision 1661246) +++ bin/hive (working copy) @@ -72,6 +72,17 @@ . "${HIVE_CONF_DIR}/hive-env.sh" fi +if [[ -z "$SPARK_HOME" ]] +then + bin=`dirname "$0"` + # many hadoop installs are in dir/{spark,hive,hadoop,..} + sparkHome=$(readlink -f $bin/../../spark) + if [[ -d $sparkHome ]] + then + export SPARK_HOME=$sparkHome + fi +fi + CLASSPATH="${HIVE_CONF_DIR}" HIVE_LIB=${HIVE_HOME}/lib Index: common/src/java/org/apache/hadoop/hive/common/JavaUtils.java =================================================================== --- common/src/java/org/apache/hadoop/hive/common/JavaUtils.java (revision 1661246) +++ common/src/java/org/apache/hadoop/hive/common/JavaUtils.java (working copy) @@ -70,9 +70,9 @@ return classLoader; } - public static void closeClassLoadersTo(ClassLoader current, ClassLoader stop) { + public static boolean closeClassLoadersTo(ClassLoader current, ClassLoader stop) { if (!isValidHierarchy(current, stop)) { - return; + return false; } for (; current != null && current != stop; current = current.getParent()) { try { @@ -82,6 +82,7 @@ Arrays.toString(((URLClassLoader) current).getURLs()), e); } } + return true; } // check before closing loaders, not to close app-classloader, etc. by mistake Index: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java =================================================================== --- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (revision 1661246) +++ common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (working copy) @@ -333,7 +333,7 @@ "When hive.exec.mode.local.auto is true, the number of tasks should less than this for local mode."), DROPIGNORESNONEXISTENT("hive.exec.drop.ignorenonexistent", true, - "Do not report an error if DROP TABLE/VIEW/Index specifies a non-existent table/view/index"), + "Do not report an error if DROP TABLE/VIEW/Index/Function specifies a non-existent table/view/index/function"), HIVEIGNOREMAPJOINHINT("hive.ignore.mapjoin.hint", true, "Ignore the mapjoin hint"), @@ -593,8 +593,8 @@ "List of comma separated keys occurring in table properties which will get inherited to newly created partitions. \n" + "* implies all the keys will get inherited."), METASTORE_FILTER_HOOK("hive.metastore.filter.hook", "org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl", - "Metastore hook class for filtering the metadata read results. If hive.security.authorization.manager\n" + - "is set to instance of HiveAuthorizerFactory, then this value is ignored."), + "Metastore hook class for filtering the metadata read results. If hive.security.authorization.manager" + + "is set to instance of HiveAuthorizerFactory, then this value is ignored."), FIRE_EVENTS_FOR_DML("hive.metastore.dml.events", false, "If true, the metastore will be asked" + " to fire events for DML operations"), Index: contrib/src/java/org/apache/hadoop/hive/contrib/genericudf/example/GenericUDFDBOutput.java =================================================================== --- contrib/src/java/org/apache/hadoop/hive/contrib/genericudf/example/GenericUDFDBOutput.java (revision 1661246) +++ contrib/src/java/org/apache/hadoop/hive/contrib/genericudf/example/GenericUDFDBOutput.java (working copy) @@ -29,7 +29,6 @@ import org.apache.hadoop.hive.ql.metadata.HiveException; import org.apache.hadoop.hive.ql.udf.UDFType; import org.apache.hadoop.hive.ql.udf.generic.GenericUDF; -import org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils; import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector; @@ -159,17 +158,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("dboutput("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(","); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("dboutput", children, ","); } } Index: contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java =================================================================== --- contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java (revision 1661246) +++ contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java (working copy) @@ -35,6 +35,7 @@ import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; import org.apache.hadoop.hive.serde2.lazy.LazyStruct; import org.apache.hadoop.hive.serde2.lazy.LazyUtils; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; @@ -62,7 +63,8 @@ serdeConstants.SERIALIZATION_LAST_COLUMN_TAKES_REST, serdeConstants.ESCAPE_CHAR, serdeConstants.SERIALIZATION_ENCODING, - LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS}) + LazySerDeParameters.SERIALIZATION_EXTEND_NESTING_LEVELS, + LazySerDeParameters.SERIALIZATION_EXTEND_ADDITIONAL_NESTING_LEVELS}) public class MultiDelimitSerDe extends AbstractSerDe { private static final Log LOG = LogFactory.getLog(MultiDelimitSerDe.class.getName()); private static final byte[] DEFAULT_SEPARATORS = {(byte) 1, (byte) 2, (byte) 3}; @@ -85,7 +87,7 @@ // The wrapper for byte array private ByteArrayRef byteArrayRef; - private LazySimpleSerDe.SerDeParameters serdeParams = null; + private LazySerDeParameters serdeParams = null; // The output stream of serialized objects private final ByteStream.Output serializeStream = new ByteStream.Output(); // The Writable to return in serialize @@ -94,7 +96,7 @@ @Override public void initialize(Configuration conf, Properties tbl) throws SerDeException { // get the SerDe parameters - serdeParams = LazySimpleSerDe.initSerdeParams(conf, tbl, getClass().getName()); + serdeParams = new LazySerDeParameters(conf, tbl, getClass().getName()); fieldDelimited = tbl.getProperty(serdeConstants.FIELD_DELIM); if (fieldDelimited == null || fieldDelimited.isEmpty()) { @@ -103,12 +105,12 @@ // get the collection separator and map key separator // TODO: use serdeConstants.COLLECTION_DELIM when the typo is fixed - collSep = LazySimpleSerDe.getByte(tbl.getProperty(COLLECTION_DELIM), + collSep = LazyUtils.getByte(tbl.getProperty(COLLECTION_DELIM), DEFAULT_SEPARATORS[1]); - keySep = LazySimpleSerDe.getByte(tbl.getProperty(serdeConstants.MAPKEY_DELIM), + keySep = LazyUtils.getByte(tbl.getProperty(serdeConstants.MAPKEY_DELIM), DEFAULT_SEPARATORS[2]); - serdeParams.getSeparators()[1] = collSep; - serdeParams.getSeparators()[2] = keySep; + serdeParams.setSeparator(1, collSep); + serdeParams.setSeparator(2, keySep); // Create the ObjectInspectors for the fields cachedObjectInspector = LazyFactory.createLazyStructInspector(serdeParams Index: contrib/src/test/results/clientnegative/invalid_row_sequence.q.out =================================================================== --- contrib/src/test/results/clientnegative/invalid_row_sequence.q.out (revision 1661246) +++ contrib/src/test/results/clientnegative/invalid_row_sequence.q.out (working copy) @@ -2,12 +2,10 @@ drop temporary function row_sequence PREHOOK: type: DROPFUNCTION -PREHOOK: Output: row_sequence POSTHOOK: query: -- Verify that a stateful UDF cannot be used outside of the SELECT list drop temporary function row_sequence POSTHOOK: type: DROPFUNCTION -POSTHOOK: Output: row_sequence PREHOOK: query: create temporary function row_sequence as 'org.apache.hadoop.hive.contrib.udf.UDFRowSequence' PREHOOK: type: CREATEFUNCTION Index: data/conf/hive-log4j.properties =================================================================== --- data/conf/hive-log4j.properties (revision 1661246) +++ data/conf/hive-log4j.properties (working copy) @@ -93,3 +93,5 @@ log4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,DRFA log4j.logger.org.apache.hadoop.hive.ql.log.PerfLogger=WARN,DRFA log4j.logger.org.apache.hadoop.hive.ql.exec.Operator=INFO,DRFA +log4j.logger.org.apache.hadoop.hive.serde2.lazy=INFO,DRFA +log4j.logger.org.apache.hadoop.hive.metastore.ObjectStore=INFO,DRFA Index: hbase-handler/pom.xml =================================================================== --- hbase-handler/pom.xml (revision 1661246) +++ hbase-handler/pom.xml (working copy) Property changes on: hbase-handler/pom.xml ___________________________________________________________________ Modified: svn:mergeinfo Merged /hive/trunk/hbase-handler/pom.xml:r1658024-1661246 Merged /hive/branches/spark/hbase-handler/pom.xml:r1657402-1660298 Index: hbase-handler/src/java/org/apache/hadoop/hive/hbase/DefaultHBaseKeyFactory.java =================================================================== --- hbase-handler/src/java/org/apache/hadoop/hive/hbase/DefaultHBaseKeyFactory.java (revision 1661246) +++ hbase-handler/src/java/org/apache/hadoop/hive/hbase/DefaultHBaseKeyFactory.java (working copy) @@ -22,10 +22,11 @@ import java.util.Properties; import com.google.common.annotations.VisibleForTesting; + import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; import org.apache.hadoop.hive.serde2.lazy.LazyObjectBase; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.StructField; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; @@ -33,7 +34,7 @@ public class DefaultHBaseKeyFactory extends AbstractHBaseKeyFactory implements HBaseKeyFactory { - protected SerDeParameters serdeParams; + protected LazySerDeParameters serdeParams; protected HBaseRowSerializer serializer; @Override @@ -59,7 +60,7 @@ } @VisibleForTesting - static DefaultHBaseKeyFactory forTest(SerDeParameters params, ColumnMappings mappings) { + static DefaultHBaseKeyFactory forTest(LazySerDeParameters params, ColumnMappings mappings) { DefaultHBaseKeyFactory factory = new DefaultHBaseKeyFactory(); factory.serdeParams = params; factory.keyMapping = mappings.getKeyMapping(); Index: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java =================================================================== --- hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java (revision 1661246) +++ hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseLazyObjectFactory.java (working copy) @@ -23,7 +23,7 @@ import org.apache.hadoop.hive.hbase.struct.HBaseValueFactory; import org.apache.hadoop.hive.serde2.SerDeException; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyObjectInspectorFactory; import org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyObjectInspectorParameters; import org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyObjectInspectorParametersImpl; @@ -37,7 +37,7 @@ public class HBaseLazyObjectFactory { public static ObjectInspector createLazyHBaseStructInspector( - SerDeParameters serdeParams, int index, HBaseKeyFactory keyFactory, List valueFactories) throws SerDeException { + LazySerDeParameters serdeParams, int index, HBaseKeyFactory keyFactory, List valueFactories) throws SerDeException { List columnTypes = serdeParams.getColumnTypes(); ArrayList columnObjectInspectors = new ArrayList( columnTypes.size()); Index: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java =================================================================== --- hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java (revision 1661246) +++ hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseRowSerializer.java (working copy) @@ -27,8 +27,8 @@ import org.apache.hadoop.hive.serde2.ByteStream; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeUtils; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; import org.apache.hadoop.hive.serde2.lazy.LazyUtils; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; @@ -44,7 +44,7 @@ private final HBaseKeyFactory keyFactory; private final HBaseSerDeParameters hbaseParam; - private final LazySimpleSerDe.SerDeParameters serdeParam; + private final LazySerDeParameters serdeParam; private final int keyIndex; private final int timestampIndex; @@ -54,9 +54,7 @@ private final byte[] separators; // the separators array private final boolean escaped; // whether we need to escape the data when writing out private final byte escapeChar; // which char to use as the escape char, e.g. '\\' - private final boolean[] needsEscape; // which chars need to be escaped. This array should have size - // of 128. Negative byte values (or byte values >= 128) are - // never escaped. + private final boolean[] needsEscape; // which chars need to be escaped. private final long putTimestamp; private final ByteStream.Output output = new ByteStream.Output(); Index: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java =================================================================== --- hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java (revision 1661246) +++ hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java (working copy) @@ -34,7 +34,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeSpec; import org.apache.hadoop.hive.serde2.SerDeStats; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.io.Writable; @@ -51,7 +51,8 @@ serdeConstants.SERIALIZATION_LAST_COLUMN_TAKES_REST, serdeConstants.ESCAPE_CHAR, serdeConstants.SERIALIZATION_ENCODING, - LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS, + LazySerDeParameters.SERIALIZATION_EXTEND_NESTING_LEVELS, + LazySerDeParameters.SERIALIZATION_EXTEND_ADDITIONAL_NESTING_LEVELS, HBaseSerDe.HBASE_COLUMNS_MAPPING, HBaseSerDe.HBASE_TABLE_NAME, HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE, @@ -240,7 +241,7 @@ return new ColumnMappings(columnsMapping, rowKeyIndex, timestampIndex); } - public LazySimpleSerDe.SerDeParameters getSerdeParams() { + public LazySerDeParameters getSerdeParams() { return serdeParams.getSerdeParams(); } Index: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java =================================================================== --- hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java (revision 1661246) +++ hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDeParameters.java (working copy) @@ -33,8 +33,7 @@ import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; import org.apache.hadoop.util.ReflectionUtils; @@ -50,7 +49,7 @@ public static final String AVRO_SERIALIZATION_TYPE = "avro"; public static final String STRUCT_SERIALIZATION_TYPE = "struct"; - private final SerDeParameters serdeParams; + private final LazySerDeParameters serdeParams; private final Configuration job; @@ -92,7 +91,7 @@ columnMappings.toTypesString(tbl, job, autogenerate)); } - this.serdeParams = LazySimpleSerDe.initSerdeParams(job, tbl, serdeName); + this.serdeParams = new LazySerDeParameters(job, tbl, serdeName); this.putTimestamp = Long.valueOf(tbl.getProperty(HBaseSerDe.HBASE_PUT_TIMESTAMP, "-1")); columnMappings.setHiveColumnDescription(serdeName, serdeParams.getColumnNames(), @@ -114,7 +113,7 @@ return serdeParams.getColumnTypes(); } - public SerDeParameters getSerdeParams() { + public LazySerDeParameters getSerdeParams() { return serdeParams; } Index: hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java =================================================================== --- hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java (revision 1661246) +++ hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java (working copy) @@ -88,10 +88,16 @@ private static final String HBASE_SNAPSHOT_TABLE_DIR_KEY = "hbase.TableSnapshotInputFormat.table.dir"; /** HBase-internal config by which input format received restore dir after HBASE-11335. */ private static final String HBASE_SNAPSHOT_RESTORE_DIR_KEY = "hbase.TableSnapshotInputFormat.restore.dir"; - /** HBase config by which a SlabCache is sized. */ - private static final String HBASE_OFFHEAP_PCT_KEY = "hbase.offheapcache.percentage"; - /** HBase config by which a BucketCache is sized. */ - private static final String HBASE_BUCKETCACHE_SIZE_KEY = "hbase.bucketcache.size"; + private static final String[] HBASE_CACHE_KEYS = new String[] { + /** HBase config by which a SlabCache is sized. From HBase [0.98.3, 1.0.0) */ + "hbase.offheapcache.percentage", + /** HBase config by which a BucketCache is sized. */ + "hbase.bucketcache.size", + /** HBase config by which the bucket cache implementation is chosen. From HBase 0.98.10+ */ + "hbase.bucketcache.ioengine", + /** HBase config by which a BlockCache is sized. */ + "hfile.block.cache.size" + }; final static public String DEFAULT_PREFIX = "default."; @@ -395,8 +401,14 @@ TableMapReduceUtil.resetCacheConfig(hbaseConf); // copy over configs touched by above method - jobProperties.put(HBASE_OFFHEAP_PCT_KEY, hbaseConf.get(HBASE_OFFHEAP_PCT_KEY)); - jobProperties.put(HBASE_BUCKETCACHE_SIZE_KEY, hbaseConf.get(HBASE_BUCKETCACHE_SIZE_KEY)); + for (String cacheKey : HBASE_CACHE_KEYS) { + final String value = hbaseConf.get(cacheKey); + if (value != null) { + jobProperties.put(cacheKey, value); + } else { + jobProperties.remove(cacheKey); + } + } } catch (IOException e) { throw new IllegalArgumentException(e); } Index: hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/DefaultHBaseValueFactory.java =================================================================== --- hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/DefaultHBaseValueFactory.java (revision 1661246) +++ hbase-handler/src/java/org/apache/hadoop/hive/hbase/struct/DefaultHBaseValueFactory.java (working copy) @@ -27,7 +27,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; import org.apache.hadoop.hive.serde2.lazy.LazyObjectBase; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; import org.apache.hadoop.hive.serde2.objectinspector.StructField; @@ -38,7 +38,7 @@ * */ public class DefaultHBaseValueFactory implements HBaseValueFactory { - protected LazySimpleSerDe.SerDeParameters serdeParams; + protected LazySerDeParameters serdeParams; protected ColumnMappings columnMappings; protected HBaseSerDeParameters hbaseParams; protected Properties properties; Index: hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java =================================================================== --- hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java (revision 1661246) +++ hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java (working copy) @@ -61,8 +61,8 @@ import org.apache.hadoop.hive.serde2.io.DoubleWritable; import org.apache.hadoop.hive.serde2.io.ShortWritable; import org.apache.hadoop.hive.serde2.lazy.LazyPrimitive; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; import org.apache.hadoop.hive.serde2.lazy.LazyStruct; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.StructField; import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; import org.apache.hadoop.io.BooleanWritable; @@ -1413,7 +1413,7 @@ "org.apache.hadoop.hive.hbase.avro.Employee"); tbl.setProperty(HBaseSerDe.HBASE_COLUMNS_MAPPING, "cola:prefixB_.*"); tbl.setProperty(HBaseSerDe.HBASE_AUTOGENERATE_STRUCT, "true"); - tbl.setProperty(LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS, "true"); + tbl.setProperty(LazySerDeParameters.SERIALIZATION_EXTEND_NESTING_LEVELS, "true"); return tbl; } Index: hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java =================================================================== --- hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java (revision 1661246) +++ hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/DelimitedInputWriter.java (working copy) @@ -31,6 +31,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.io.BytesWritable; import java.io.IOException; @@ -89,7 +90,7 @@ throws ClassNotFoundException, ConnectionError, SerializationError, InvalidColumn, StreamingException { this(colNamesForFields, delimiter, endPoint, conf, - (char) LazySimpleSerDe.DefaultSeparators[0]); + (char) LazySerDeParameters.DefaultSeparators[0]); } /** Index: itests/hive-jmh/pom.xml =================================================================== --- itests/hive-jmh/pom.xml (revision 1661246) +++ itests/hive-jmh/pom.xml (working copy) @@ -62,6 +62,16 @@ + hadoop-1 + + + org.apache.hadoop + hadoop-core + ${hadoop-20S.version} + + + + hadoop-2 Index: itests/src/test/resources/testconfiguration.properties =================================================================== --- itests/src/test/resources/testconfiguration.properties (revision 1661246) +++ itests/src/test/resources/testconfiguration.properties (working copy) @@ -140,6 +140,7 @@ orc_vectorization_ppd.q,\ parallel.q,\ ptf.q,\ + ptf_matchpath.q,\ ptf_streaming.q,\ sample1.q,\ selectDistinctStar.q,\ @@ -167,6 +168,8 @@ union7.q,\ union8.q,\ union9.q,\ + unionDistinct_1.q,\ + unionDistinct_2.q,\ update_after_multiple_inserts.q,\ update_all_non_partitioned.q,\ update_all_partitioned.q,\ @@ -572,6 +575,7 @@ bucketsortoptimize_insert_6.q, \ bucketsortoptimize_insert_7.q, \ bucketsortoptimize_insert_8.q, \ + cbo_gby_empty.q, \ column_access_stats.q, \ count.q, \ create_merge_compressed.q, \ @@ -907,6 +911,7 @@ transform_ppr1.q, \ transform_ppr2.q, \ udf_example_add.q, \ + udf_in_file.q, \ union.q, \ union10.q, \ union11.q, \ @@ -953,6 +958,7 @@ union_remove_8.q, \ union_remove_9.q, \ uniquejoin.q, \ + union_view.q, \ varchar_join1.q, \ vector_between_in.q, \ vector_cast_constant.q, \ Index: itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java =================================================================== --- itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java (revision 1661246) +++ itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java (working copy) @@ -1051,7 +1051,7 @@ } command = ""; } - if (SessionState.get() != null) { + if (rc == 0 && SessionState.get() != null) { SessionState.get().setLastCommand(null); // reset } return rc; Index: itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEvaluateNPE.java =================================================================== --- itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEvaluateNPE.java (revision 1661246) +++ itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEvaluateNPE.java (working copy) @@ -76,6 +76,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 1); - return "evaluate_npe(" + children[0] + ")"; + return getStandardDisplayString("evaluate_npe", children); } } Index: itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaBoolean.java =================================================================== --- itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaBoolean.java (revision 1661246) +++ itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaBoolean.java (working copy) @@ -54,6 +54,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 1); - return "TestGetJavaBoolean(" + children[0] + ")"; + return getStandardDisplayString("TestGetJavaBoolean", children); } } Index: itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaString.java =================================================================== --- itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaString.java (revision 1661246) +++ itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestGetJavaString.java (working copy) @@ -50,6 +50,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 1); - return "GenericUDFTestGetJavaString(" + children[0] + ")"; + return getStandardDisplayString("GenericUDFTestGetJavaString", children); } } Index: itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestTranslate.java =================================================================== --- itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestTranslate.java (revision 1661246) +++ itests/util/src/main/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTestTranslate.java (working copy) @@ -119,7 +119,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 3); - return "translate(" + children[0] + "," + children[1] + "," + children[2] - + ")"; + return getStandardDisplayString("translate", children, ","); } } Index: jdbc/pom.xml =================================================================== --- jdbc/pom.xml (revision 1661246) +++ jdbc/pom.xml (working copy) @@ -162,14 +162,13 @@ org.apache.hive.shims:hive-shims-common - org/apache/hadoop/hive/shims/* - org/apache/hadoop/hive/thrift/* + ** org.apache.hive.shims:hive-shims-0.23 - org/apache/hadoop/hive/thrift/* + ** Index: llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataProducer.java =================================================================== --- llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataProducer.java (revision 1661246) +++ llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataProducer.java (working copy) @@ -45,6 +45,7 @@ import org.apache.hadoop.hive.ql.io.orc.OrcFile; import org.apache.hadoop.hive.ql.io.orc.OrcInputFormat; import org.apache.hadoop.hive.ql.io.orc.OrcProto.Type; +import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl; import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.SargApplier; import org.apache.hadoop.hive.ql.io.orc.Reader; import org.apache.hadoop.hive.ql.io.orc.StripeInformation; @@ -150,11 +151,23 @@ int stride = metadata.getRowIndexStride(); ArrayList stripeMetadatas = null; boolean[] globalIncludes = null; + boolean[] sargColumns = null; try { globalIncludes = OrcInputFormat.genIncludedColumns(metadata.getTypes(), columnIds, true); if (sarg != null && stride != 0) { + // TODO: move this to a common method + int[] filterColumns = RecordReaderImpl.mapSargColumns(sarg.getLeaves(), columnNames, 0); + // included will not be null, row options will fill the array with trues if null + sargColumns = new boolean[globalIncludes.length]; + for (int i : filterColumns) { + // filter columns may have -1 as index which could be partition column in SARG. + if (i > 0) { + sargColumns[i] = true; + } + } + // If SARG is present, get relevant stripe metadata from cache or readers. - stripeMetadatas = readStripesMetadata(metadata, globalIncludes); + stripeMetadatas = readStripesMetadata(metadata, globalIncludes, sargColumns); } // Now, apply SARG if any; w/o sarg, this will just initialize readState. @@ -245,7 +258,7 @@ stripeMetadata = metadataCache.getStripeMetadata(stripeKey); if (stripeMetadata == null) { ensureMetadataReader(); - stripeMetadata = new OrcStripeMetadata(metadataReader, si, stripeIncludes); + stripeMetadata = new OrcStripeMetadata(metadataReader, si, stripeIncludes, sargColumns); if (DebugUtils.isTraceOrcEnabled()) { LlapIoImpl.LOG.info("Caching stripe " + stripeKey.stripeIx + " metadata with includes: " + DebugUtils.toString(stripeIncludes)); @@ -260,7 +273,7 @@ + " metadata for includes: " + DebugUtils.toString(stripeIncludes)); } ensureMetadataReader(); - updateLoadedIndexes(stripeMetadata, si, stripeIncludes); + updateLoadedIndexes(stripeMetadata, si, stripeIncludes, sargColumns); } // 6.3. Finally, hand off to the stripe reader to produce the data. // This is a sync call that will feed data to the consumer. @@ -302,12 +315,12 @@ * the missing one. This is a temporary cludge until real metadata cache becomes available. */ private void updateLoadedIndexes(OrcStripeMetadata stripeMetadata, - StripeInformation stripe, boolean[] stripeIncludes) throws IOException { + StripeInformation stripe, boolean[] stripeIncludes, boolean[] sargColumns) throws IOException { // We only synchronize on write for now - design of metadata cache is very temporary; // we pre-allocate the array and never remove entries; so readers should be safe. synchronized (stripeMetadata) { if (stripeMetadata.hasAllIndexes(stripeIncludes)) return; - stripeMetadata.loadMissingIndexes(metadataReader, stripe, stripeIncludes); + stripeMetadata.loadMissingIndexes(metadataReader, stripe, stripeIncludes, sargColumns); } } @@ -360,7 +373,7 @@ * Reads the metadata for all stripes in the file. */ private ArrayList readStripesMetadata( - OrcFileMetadata metadata, boolean[] globalInc) throws IOException { + OrcFileMetadata metadata, boolean[] globalInc, boolean[] sargColumns) throws IOException { ArrayList result = new ArrayList(readState.length); OrcBatchKey stripeKey = new OrcBatchKey(internedFilePath, 0, 0); for (int stripeIxMod = 0; stripeIxMod < readState.length; ++stripeIxMod) { @@ -374,7 +387,7 @@ + si.getOffset() + ", " + si.getLength()); } if (value == null) { - value = new OrcStripeMetadata(metadataReader, si, globalInc); + value = new OrcStripeMetadata(metadataReader, si, globalInc, sargColumns); metadataCache.putStripeMetadata(stripeKey, value); if (DebugUtils.isTraceOrcEnabled()) { LlapIoImpl.LOG.info("Caching stripe " + stripeKey.stripeIx @@ -387,7 +400,7 @@ LlapIoImpl.LOG.info("Updating indexes in stripe " + stripeKey.stripeIx + " metadata for includes: " + DebugUtils.toString(globalInc)); } - updateLoadedIndexes(value, si, globalInc); + updateLoadedIndexes(value, si, globalInc, sargColumns); } } result.add(value); @@ -419,7 +432,7 @@ ensureOrcReader(); String[] colNamesForSarg = OrcInputFormat.getSargColumnNames( columnNames, types, globalIncludes, OrcInputFormat.isOriginal(orcReader)); - sargApp = new SargApplier(sarg, colNamesForSarg, rowIndexStride); + sargApp = new SargApplier(sarg, colNamesForSarg, rowIndexStride, types); } // readState should have been initialized by this time with an empty array. for (int stripeIxMod = 0; stripeIxMod < readState.length; ++stripeIxMod) { Index: llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java =================================================================== --- llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java (revision 1661246) +++ llap-server/src/java/org/apache/hadoop/hive/llap/io/metadata/OrcStripeMetadata.java (working copy) @@ -25,40 +25,42 @@ import org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex; import org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream; import org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter; +import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl; import org.apache.hadoop.hive.ql.io.orc.StripeInformation; public class OrcStripeMetadata { List encodings; List streams; - RowIndex[] rowIndexes; + RecordReaderImpl.Index rowIndex; - public OrcStripeMetadata( - MetadataReader mr, StripeInformation stripe, boolean[] includes) throws IOException { + public OrcStripeMetadata(MetadataReader mr, StripeInformation stripe, + boolean[] includes, boolean[] sargColumns) throws IOException { StripeFooter footer = mr.readStripeFooter(stripe); streams = footer.getStreamsList(); encodings = footer.getColumnsList(); - rowIndexes = mr.readRowIndex(stripe, footer, includes, null); + rowIndex = mr.readRowIndex(stripe, footer, includes, null, sargColumns, null); } public boolean hasAllIndexes(boolean[] includes) { for (int i = 0; i < includes.length; ++i) { - if (includes[i] && rowIndexes[i] == null) return false; + if (includes[i] && rowIndex.getRowGroupIndex()[i] == null) return false; } return true; } - public void loadMissingIndexes( - MetadataReader mr, StripeInformation stripe, boolean[] includes) throws IOException { + public void loadMissingIndexes(MetadataReader mr, StripeInformation stripe, boolean[] includes, + boolean[] sargColumns) throws IOException { // TODO: should we save footer to avoid a read here? - rowIndexes = mr.readRowIndex(stripe, null, includes, rowIndexes); + rowIndex = mr.readRowIndex(stripe, null, includes, rowIndex.getRowGroupIndex(), + sargColumns, rowIndex.getBloomFilterIndex()); } public RowIndex[] getRowIndexes() { - return rowIndexes; + return rowIndex.getRowGroupIndex(); } public void setRowIndexes(RowIndex[] rowIndexes) { - this.rowIndexes = rowIndexes; + this.rowIndex.setRowGroupIndex(rowIndexes); } public List getEncodings() { Index: metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java =================================================================== --- metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java (revision 1661246) +++ metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java (working copy) @@ -346,9 +346,11 @@ final Formatter fmt = auditFormatter.get(); ((StringBuilder) fmt.out()).setLength(0); - String address; + String address = null; if (useSasl) { - address = saslServer.getRemoteAddress().toString(); + if (saslServer != null && saslServer.getRemoteAddress() != null) { + address = String.valueOf(saslServer.getRemoteAddress()); + } } else { address = getIpAddress(); } Index: metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java =================================================================== --- metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java (revision 1661246) +++ metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java (working copy) @@ -18,9 +18,9 @@ package org.apache.hadoop.hive.metastore; +import java.util.Collections; import java.util.List; import java.util.Map; -import java.util.SortedSet; import junit.framework.Assert; @@ -28,7 +28,6 @@ import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.metastore.api.AggrStats; import org.apache.hadoop.hive.metastore.api.ColumnStatistics; -import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj; import org.apache.hadoop.hive.metastore.api.CurrentNotificationEventId; import org.apache.hadoop.hive.metastore.api.Database; import org.apache.hadoop.hive.metastore.api.Function; @@ -44,12 +43,10 @@ import org.apache.hadoop.hive.metastore.api.NotificationEventResponse; import org.apache.hadoop.hive.metastore.api.Partition; import org.apache.hadoop.hive.metastore.api.PartitionEventType; -import org.apache.hadoop.hive.metastore.api.PartitionsStatsRequest; import org.apache.hadoop.hive.metastore.api.PrincipalPrivilegeSet; import org.apache.hadoop.hive.metastore.api.PrincipalType; import org.apache.hadoop.hive.metastore.api.PrivilegeBag; import org.apache.hadoop.hive.metastore.api.Role; -import org.apache.hadoop.hive.metastore.api.SetPartitionsStatsRequest; import org.apache.hadoop.hive.metastore.api.Table; import org.apache.hadoop.hive.metastore.api.Type; import org.apache.hadoop.hive.metastore.api.UnknownDBException; @@ -145,13 +142,13 @@ @Override public List getDatabases(String pattern) throws MetaException { - return null; + return Collections.emptyList(); } @Override public List getAllDatabases() throws MetaException { - return null; + return Collections.emptyList(); } @Override @@ -214,7 +211,7 @@ public List getPartitions(String dbName, String tableName, int max) throws MetaException { - return null; + return Collections.emptyList(); } @Override @@ -227,41 +224,41 @@ @Override public List getTables(String dbName, String pattern) throws MetaException { - return null; + return Collections.emptyList(); } @Override public List getTableObjectsByName(String dbname, List tableNames) throws MetaException, UnknownDBException { - return null; + return Collections.emptyList(); } @Override public List getAllTables(String dbName) throws MetaException { - return null; + return Collections.emptyList(); } @Override public List listTableNamesByFilter(String dbName, String filter, short max_tables) throws MetaException, UnknownDBException { - return null; + return Collections.emptyList(); } @Override public List listPartitionNames(String db_name, String tbl_name, short max_parts) throws MetaException { - return null; + return Collections.emptyList(); } @Override public List listPartitionNamesByFilter(String db_name, String tbl_name, String filter, short max_parts) throws MetaException { - return null; + return Collections.emptyList(); } @Override @@ -310,7 +307,7 @@ public List listIndexNames(String dbName, String origTableName, short max) throws MetaException { - return null; + return Collections.emptyList(); } @Override @@ -324,14 +321,14 @@ public List getPartitionsByFilter(String dbName, String tblName, String filter, short maxParts) throws MetaException, NoSuchObjectException { - return null; + return Collections.emptyList(); } @Override public List getPartitionsByNames(String dbName, String tblName, List partNames) throws MetaException, NoSuchObjectException { - return null; + return Collections.emptyList(); } @Override @@ -425,35 +422,35 @@ public List listPrincipalGlobalGrants(String principalName, PrincipalType principalType) { - return null; + return Collections.emptyList(); } @Override public List listPrincipalDBGrants(String principalName, PrincipalType principalType, String dbName) { - return null; + return Collections.emptyList(); } @Override public List listAllTableGrants(String principalName, PrincipalType principalType, String dbName, String tableName) { - return null; + return Collections.emptyList(); } @Override public List listPrincipalPartitionGrants(String principalName, PrincipalType principalType, String dbName, String tableName, String partName) { - return null; + return Collections.emptyList(); } @Override public List listPrincipalTableColumnGrants(String principalName, PrincipalType principalType, String dbName, String tableName, String columnName) { - return null; + return Collections.emptyList(); } @Override @@ -461,7 +458,7 @@ PrincipalType principalType, String dbName, String tableName, String partName, String columnName) { - return null; + return Collections.emptyList(); } @Override @@ -487,18 +484,18 @@ @Override public List listRoleNames() { - return null; + return Collections.emptyList(); } @Override public List listRoles(String principalName, PrincipalType principalType) { - return null; + return Collections.emptyList(); } @Override public List listRoleMembers(String roleName) { - return null; + return Collections.emptyList(); } @Override @@ -514,14 +511,14 @@ String userName, List groupNames) throws MetaException, NoSuchObjectException, InvalidObjectException { - return null; + return Collections.emptyList(); } @Override public List listPartitionNamesPs(String db_name, String tbl_name, List part_vals, short max_parts) throws MetaException, NoSuchObjectException { - return null; + return Collections.emptyList(); } @Override @@ -529,7 +526,7 @@ List part_vals, short max_parts, String userName, List groupNames) throws MetaException, InvalidObjectException, NoSuchObjectException { - return null; + return Collections.emptyList(); } @Override @@ -555,7 +552,7 @@ @Override public List getAllTokenIdentifiers() { - return null; + return Collections.emptyList(); } @Override @@ -574,67 +571,67 @@ @Override public String[] getMasterKeys() { - return null; + return new String[0]; } @Override public List listPrincipalDBGrantsAll( String principalName, PrincipalType principalType) { - return null; + return Collections.emptyList(); } @Override public List listPrincipalTableGrantsAll( String principalName, PrincipalType principalType) { - return null; + return Collections.emptyList(); } @Override public List listPrincipalPartitionGrantsAll( String principalName, PrincipalType principalType) { - return null; + return Collections.emptyList(); } @Override public List listPrincipalTableColumnGrantsAll( String principalName, PrincipalType principalType) { - return null; + return Collections.emptyList(); } @Override public List listPrincipalPartitionColumnGrantsAll( String principalName, PrincipalType principalType) { - return null; + return Collections.emptyList(); } @Override public List listGlobalGrantsAll() { - return null; + return Collections.emptyList(); } @Override public List listDBGrantsAll(String dbName) { - return null; + return Collections.emptyList(); } @Override public List listPartitionColumnGrantsAll(String dbName, String tableName, String partitionName, String columnName) { - return null; + return Collections.emptyList(); } @Override public List listTableGrantsAll(String dbName, String tableName) { - return null; + return Collections.emptyList(); } @Override public List listPartitionGrantsAll(String dbName, String tableName, String partitionName) { - return null; + return Collections.emptyList(); } @Override public List listTableColumnGrantsAll(String dbName, String tableName, String columnName) { - return null; + return Collections.emptyList(); } @Override @@ -689,7 +686,7 @@ public List getPartitionColumnStatistics(String dbName, String tblName, List colNames, List partNames) throws MetaException, NoSuchObjectException { - return null; + return Collections.emptyList(); } @Override @@ -738,7 +735,7 @@ @Override public List getFunctions(String dbName, String pattern) throws MetaException { - return null; + return Collections.emptyList(); } @Override Index: packaging/pom.xml =================================================================== --- packaging/pom.xml (revision 1661246) +++ packaging/pom.xml (working copy) @@ -86,28 +86,6 @@ - - org.codehaus.mojo - build-helper-maven-plugin - - - attach-jdbc-driver - package - - attach-artifact - - - - - ${project.build.directory}/${hive.jdbc.driver.jar} - jar - ${hive.jdbc.driver.classifier} - - - - - - Index: packaging/src/main/assembly/src.xml =================================================================== --- packaging/src/main/assembly/src.xml (revision 1661246) +++ packaging/src/main/assembly/src.xml (working copy) @@ -40,6 +40,7 @@ **/.project **/.settings/** itests/thirdparty/** + testutils/ptest2/velocity.log* @@ -74,6 +75,8 @@ jdbc/**/* contrib/**/* metastore/**/* + lib/**/* + findbugs/**/* odbc/**/* packaging/pom.xml packaging/src/**/* @@ -83,7 +86,6 @@ service/**/* shims/**/* spark-client/**/* - testlibs/**/* testutils/**/* / Index: pom.xml =================================================================== --- pom.xml (revision 1661246) +++ pom.xml (working copy) @@ -102,7 +102,7 @@ 3.4 1.7.5 0.8.0.RELEASE - 1.0.0-incubating-SNAPSHOT + 1.0.0-incubating 3.2.6 3.2.10 3.2.9 @@ -173,6 +173,7 @@ + datanucleus datanucleus maven repository @@ -488,10 +489,20 @@ org.apache.curator + curator-client + ${curator.version} + + + org.apache.curator curator-framework ${curator.version} + org.apache.curator + curator-recipes + ${curator.version} + + org.codehaus.groovy groovy-all ${groovy.version} Index: ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java =================================================================== --- ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java (revision 1661246) +++ ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java (working copy) @@ -9,7 +9,7 @@ com.google.protobuf.ExtensionRegistry registry) { } /** - * Protobuf enum {@code org.apache.hadoop.hive.ql.io.orc.CompressionKind} + * Protobuf enum {@code orc.proto.CompressionKind} */ public enum CompressionKind implements com.google.protobuf.ProtocolMessageEnum { @@ -105,7 +105,7 @@ this.value = value; } - // @@protoc_insertion_point(enum_scope:org.apache.hadoop.hive.ql.io.orc.CompressionKind) + // @@protoc_insertion_point(enum_scope:orc.proto.CompressionKind) } public interface IntegerStatisticsOrBuilder @@ -142,7 +142,7 @@ long getSum(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.IntegerStatistics} + * Protobuf type {@code orc.proto.IntegerStatistics} */ public static final class IntegerStatistics extends com.google.protobuf.GeneratedMessage @@ -221,12 +221,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_IntegerStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_IntegerStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.Builder.class); } @@ -421,19 +421,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.IntegerStatistics} + * Protobuf type {@code orc.proto.IntegerStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_IntegerStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_IntegerStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.Builder.class); } @@ -473,7 +473,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_IntegerStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics getDefaultInstanceForType() { @@ -655,7 +655,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.IntegerStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.IntegerStatistics) } static { @@ -663,7 +663,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.IntegerStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.IntegerStatistics) } public interface DoubleStatisticsOrBuilder @@ -700,7 +700,7 @@ double getSum(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.DoubleStatistics} + * Protobuf type {@code orc.proto.DoubleStatistics} */ public static final class DoubleStatistics extends com.google.protobuf.GeneratedMessage @@ -779,12 +779,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DoubleStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DoubleStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.Builder.class); } @@ -979,19 +979,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.DoubleStatistics} + * Protobuf type {@code orc.proto.DoubleStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DoubleStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DoubleStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.Builder.class); } @@ -1031,7 +1031,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DoubleStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics getDefaultInstanceForType() { @@ -1213,7 +1213,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.DoubleStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.DoubleStatistics) } static { @@ -1221,7 +1221,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.DoubleStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.DoubleStatistics) } public interface StringStatisticsOrBuilder @@ -1276,7 +1276,7 @@ long getSum(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StringStatistics} + * Protobuf type {@code orc.proto.StringStatistics} */ public static final class StringStatistics extends com.google.protobuf.GeneratedMessage @@ -1355,12 +1355,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StringStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StringStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.Builder.class); } @@ -1617,19 +1617,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StringStatistics} + * Protobuf type {@code orc.proto.StringStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StringStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StringStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.Builder.class); } @@ -1669,7 +1669,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StringStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics getDefaultInstanceForType() { @@ -1953,7 +1953,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.StringStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.StringStatistics) } static { @@ -1961,7 +1961,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.StringStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.StringStatistics) } public interface BucketStatisticsOrBuilder @@ -1982,7 +1982,7 @@ long getCount(int index); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.BucketStatistics} + * Protobuf type {@code orc.proto.BucketStatistics} */ public static final class BucketStatistics extends com.google.protobuf.GeneratedMessage @@ -2070,12 +2070,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BucketStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BucketStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.Builder.class); } @@ -2243,19 +2243,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.BucketStatistics} + * Protobuf type {@code orc.proto.BucketStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BucketStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BucketStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.Builder.class); } @@ -2291,7 +2291,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BucketStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics getDefaultInstanceForType() { @@ -2432,7 +2432,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.BucketStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.BucketStatistics) } static { @@ -2440,7 +2440,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.BucketStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.BucketStatistics) } public interface DecimalStatisticsOrBuilder @@ -2492,7 +2492,7 @@ getSumBytes(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.DecimalStatistics} + * Protobuf type {@code orc.proto.DecimalStatistics} */ public static final class DecimalStatistics extends com.google.protobuf.GeneratedMessage @@ -2571,12 +2571,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DecimalStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DecimalStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.Builder.class); } @@ -2852,19 +2852,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.DecimalStatistics} + * Protobuf type {@code orc.proto.DecimalStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DecimalStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DecimalStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.Builder.class); } @@ -2904,7 +2904,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DecimalStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics getDefaultInstanceForType() { @@ -3215,7 +3215,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.DecimalStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.DecimalStatistics) } static { @@ -3223,7 +3223,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.DecimalStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.DecimalStatistics) } public interface DateStatisticsOrBuilder @@ -3258,7 +3258,7 @@ int getMaximum(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.DateStatistics} + * Protobuf type {@code orc.proto.DateStatistics} */ public static final class DateStatistics extends com.google.protobuf.GeneratedMessage @@ -3332,12 +3332,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DateStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DateStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.Builder.class); } @@ -3516,19 +3516,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.DateStatistics} + * Protobuf type {@code orc.proto.DateStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DateStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DateStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.Builder.class); } @@ -3566,7 +3566,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_DateStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics getDefaultInstanceForType() { @@ -3724,7 +3724,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.DateStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.DateStatistics) } static { @@ -3732,7 +3732,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.DateStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.DateStatistics) } public interface TimestampStatisticsOrBuilder @@ -3767,7 +3767,7 @@ long getMaximum(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.TimestampStatistics} + * Protobuf type {@code orc.proto.TimestampStatistics} */ public static final class TimestampStatistics extends com.google.protobuf.GeneratedMessage @@ -3841,12 +3841,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_TimestampStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_TimestampStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.Builder.class); } @@ -4025,19 +4025,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.TimestampStatistics} + * Protobuf type {@code orc.proto.TimestampStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_TimestampStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_TimestampStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.Builder.class); } @@ -4075,7 +4075,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_TimestampStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics getDefaultInstanceForType() { @@ -4233,7 +4233,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.TimestampStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.TimestampStatistics) } static { @@ -4241,7 +4241,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.TimestampStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.TimestampStatistics) } public interface BinaryStatisticsOrBuilder @@ -4266,7 +4266,7 @@ long getSum(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.BinaryStatistics} + * Protobuf type {@code orc.proto.BinaryStatistics} */ public static final class BinaryStatistics extends com.google.protobuf.GeneratedMessage @@ -4335,12 +4335,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BinaryStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BinaryStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.Builder.class); } @@ -4495,19 +4495,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.BinaryStatistics} + * Protobuf type {@code orc.proto.BinaryStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BinaryStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BinaryStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.Builder.class); } @@ -4543,7 +4543,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BinaryStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics getDefaultInstanceForType() { @@ -4661,7 +4661,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.BinaryStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.BinaryStatistics) } static { @@ -4669,7 +4669,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.BinaryStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.BinaryStatistics) } public interface ColumnStatisticsOrBuilder @@ -4685,115 +4685,115 @@ */ long getNumberOfValues(); - // optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + // optional .orc.proto.IntegerStatistics intStatistics = 2; /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ boolean hasIntStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics getIntStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatisticsOrBuilder getIntStatisticsOrBuilder(); - // optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + // optional .orc.proto.DoubleStatistics doubleStatistics = 3; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ boolean hasDoubleStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics getDoubleStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatisticsOrBuilder getDoubleStatisticsOrBuilder(); - // optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + // optional .orc.proto.StringStatistics stringStatistics = 4; /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ boolean hasStringStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics getStringStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatisticsOrBuilder getStringStatisticsOrBuilder(); - // optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + // optional .orc.proto.BucketStatistics bucketStatistics = 5; /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ boolean hasBucketStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics getBucketStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatisticsOrBuilder getBucketStatisticsOrBuilder(); - // optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + // optional .orc.proto.DecimalStatistics decimalStatistics = 6; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ boolean hasDecimalStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics getDecimalStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatisticsOrBuilder getDecimalStatisticsOrBuilder(); - // optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + // optional .orc.proto.DateStatistics dateStatistics = 7; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ boolean hasDateStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics getDateStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatisticsOrBuilder getDateStatisticsOrBuilder(); - // optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + // optional .orc.proto.BinaryStatistics binaryStatistics = 8; /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ boolean hasBinaryStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics getBinaryStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatisticsOrBuilder getBinaryStatisticsOrBuilder(); - // optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + // optional .orc.proto.TimestampStatistics timestampStatistics = 9; /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ boolean hasTimestampStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics getTimestampStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatisticsOrBuilder getTimestampStatisticsOrBuilder(); @@ -4808,7 +4808,7 @@ boolean getHasNull(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.ColumnStatistics} + * Protobuf type {@code orc.proto.ColumnStatistics} */ public static final class ColumnStatistics extends com.google.protobuf.GeneratedMessage @@ -4986,12 +4986,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder.class); } @@ -5028,177 +5028,177 @@ return numberOfValues_; } - // optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + // optional .orc.proto.IntegerStatistics intStatistics = 2; public static final int INTSTATISTICS_FIELD_NUMBER = 2; private org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics intStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public boolean hasIntStatistics() { return ((bitField0_ & 0x00000002) == 0x00000002); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics getIntStatistics() { return intStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatisticsOrBuilder getIntStatisticsOrBuilder() { return intStatistics_; } - // optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + // optional .orc.proto.DoubleStatistics doubleStatistics = 3; public static final int DOUBLESTATISTICS_FIELD_NUMBER = 3; private org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics doubleStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public boolean hasDoubleStatistics() { return ((bitField0_ & 0x00000004) == 0x00000004); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics getDoubleStatistics() { return doubleStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatisticsOrBuilder getDoubleStatisticsOrBuilder() { return doubleStatistics_; } - // optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + // optional .orc.proto.StringStatistics stringStatistics = 4; public static final int STRINGSTATISTICS_FIELD_NUMBER = 4; private org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics stringStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public boolean hasStringStatistics() { return ((bitField0_ & 0x00000008) == 0x00000008); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics getStringStatistics() { return stringStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatisticsOrBuilder getStringStatisticsOrBuilder() { return stringStatistics_; } - // optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + // optional .orc.proto.BucketStatistics bucketStatistics = 5; public static final int BUCKETSTATISTICS_FIELD_NUMBER = 5; private org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics bucketStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public boolean hasBucketStatistics() { return ((bitField0_ & 0x00000010) == 0x00000010); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics getBucketStatistics() { return bucketStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatisticsOrBuilder getBucketStatisticsOrBuilder() { return bucketStatistics_; } - // optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + // optional .orc.proto.DecimalStatistics decimalStatistics = 6; public static final int DECIMALSTATISTICS_FIELD_NUMBER = 6; private org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics decimalStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public boolean hasDecimalStatistics() { return ((bitField0_ & 0x00000020) == 0x00000020); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics getDecimalStatistics() { return decimalStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatisticsOrBuilder getDecimalStatisticsOrBuilder() { return decimalStatistics_; } - // optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + // optional .orc.proto.DateStatistics dateStatistics = 7; public static final int DATESTATISTICS_FIELD_NUMBER = 7; private org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics dateStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public boolean hasDateStatistics() { return ((bitField0_ & 0x00000040) == 0x00000040); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics getDateStatistics() { return dateStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatisticsOrBuilder getDateStatisticsOrBuilder() { return dateStatistics_; } - // optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + // optional .orc.proto.BinaryStatistics binaryStatistics = 8; public static final int BINARYSTATISTICS_FIELD_NUMBER = 8; private org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics binaryStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public boolean hasBinaryStatistics() { return ((bitField0_ & 0x00000080) == 0x00000080); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics getBinaryStatistics() { return binaryStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatisticsOrBuilder getBinaryStatisticsOrBuilder() { return binaryStatistics_; } - // optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + // optional .orc.proto.TimestampStatistics timestampStatistics = 9; public static final int TIMESTAMPSTATISTICS_FIELD_NUMBER = 9; private org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics timestampStatistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public boolean hasTimestampStatistics() { return ((bitField0_ & 0x00000100) == 0x00000100); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics getTimestampStatistics() { return timestampStatistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatisticsOrBuilder getTimestampStatisticsOrBuilder() { return timestampStatistics_; @@ -5402,19 +5402,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.ColumnStatistics} + * Protobuf type {@code orc.proto.ColumnStatistics} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnStatistics_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnStatistics_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder.class); } @@ -5508,7 +5508,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnStatistics_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getDefaultInstanceForType() { @@ -5705,18 +5705,18 @@ return this; } - // optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + // optional .orc.proto.IntegerStatistics intStatistics = 2; private org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics intStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatisticsOrBuilder> intStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public boolean hasIntStatistics() { return ((bitField0_ & 0x00000002) == 0x00000002); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics getIntStatistics() { if (intStatisticsBuilder_ == null) { @@ -5726,7 +5726,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public Builder setIntStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics value) { if (intStatisticsBuilder_ == null) { @@ -5742,7 +5742,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public Builder setIntStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.Builder builderForValue) { @@ -5756,7 +5756,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public Builder mergeIntStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics value) { if (intStatisticsBuilder_ == null) { @@ -5775,7 +5775,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public Builder clearIntStatistics() { if (intStatisticsBuilder_ == null) { @@ -5788,7 +5788,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.Builder getIntStatisticsBuilder() { bitField0_ |= 0x00000002; @@ -5796,7 +5796,7 @@ return getIntStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatisticsOrBuilder getIntStatisticsOrBuilder() { if (intStatisticsBuilder_ != null) { @@ -5806,7 +5806,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.IntegerStatistics intStatistics = 2; + * optional .orc.proto.IntegerStatistics intStatistics = 2; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatisticsOrBuilder> @@ -5822,18 +5822,18 @@ return intStatisticsBuilder_; } - // optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + // optional .orc.proto.DoubleStatistics doubleStatistics = 3; private org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics doubleStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatisticsOrBuilder> doubleStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public boolean hasDoubleStatistics() { return ((bitField0_ & 0x00000004) == 0x00000004); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics getDoubleStatistics() { if (doubleStatisticsBuilder_ == null) { @@ -5843,7 +5843,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public Builder setDoubleStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics value) { if (doubleStatisticsBuilder_ == null) { @@ -5859,7 +5859,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public Builder setDoubleStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.Builder builderForValue) { @@ -5873,7 +5873,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public Builder mergeDoubleStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics value) { if (doubleStatisticsBuilder_ == null) { @@ -5892,7 +5892,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public Builder clearDoubleStatistics() { if (doubleStatisticsBuilder_ == null) { @@ -5905,7 +5905,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.Builder getDoubleStatisticsBuilder() { bitField0_ |= 0x00000004; @@ -5913,7 +5913,7 @@ return getDoubleStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatisticsOrBuilder getDoubleStatisticsOrBuilder() { if (doubleStatisticsBuilder_ != null) { @@ -5923,7 +5923,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DoubleStatistics doubleStatistics = 3; + * optional .orc.proto.DoubleStatistics doubleStatistics = 3; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.DoubleStatisticsOrBuilder> @@ -5939,18 +5939,18 @@ return doubleStatisticsBuilder_; } - // optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + // optional .orc.proto.StringStatistics stringStatistics = 4; private org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics stringStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatisticsOrBuilder> stringStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public boolean hasStringStatistics() { return ((bitField0_ & 0x00000008) == 0x00000008); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics getStringStatistics() { if (stringStatisticsBuilder_ == null) { @@ -5960,7 +5960,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public Builder setStringStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics value) { if (stringStatisticsBuilder_ == null) { @@ -5976,7 +5976,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public Builder setStringStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.Builder builderForValue) { @@ -5990,7 +5990,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public Builder mergeStringStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics value) { if (stringStatisticsBuilder_ == null) { @@ -6009,7 +6009,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public Builder clearStringStatistics() { if (stringStatisticsBuilder_ == null) { @@ -6022,7 +6022,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.Builder getStringStatisticsBuilder() { bitField0_ |= 0x00000008; @@ -6030,7 +6030,7 @@ return getStringStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatisticsOrBuilder getStringStatisticsOrBuilder() { if (stringStatisticsBuilder_ != null) { @@ -6040,7 +6040,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.StringStatistics stringStatistics = 4; + * optional .orc.proto.StringStatistics stringStatistics = 4; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.StringStatisticsOrBuilder> @@ -6056,18 +6056,18 @@ return stringStatisticsBuilder_; } - // optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + // optional .orc.proto.BucketStatistics bucketStatistics = 5; private org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics bucketStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatisticsOrBuilder> bucketStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public boolean hasBucketStatistics() { return ((bitField0_ & 0x00000010) == 0x00000010); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics getBucketStatistics() { if (bucketStatisticsBuilder_ == null) { @@ -6077,7 +6077,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public Builder setBucketStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics value) { if (bucketStatisticsBuilder_ == null) { @@ -6093,7 +6093,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public Builder setBucketStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.Builder builderForValue) { @@ -6107,7 +6107,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public Builder mergeBucketStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics value) { if (bucketStatisticsBuilder_ == null) { @@ -6126,7 +6126,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public Builder clearBucketStatistics() { if (bucketStatisticsBuilder_ == null) { @@ -6139,7 +6139,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.Builder getBucketStatisticsBuilder() { bitField0_ |= 0x00000010; @@ -6147,7 +6147,7 @@ return getBucketStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatisticsOrBuilder getBucketStatisticsOrBuilder() { if (bucketStatisticsBuilder_ != null) { @@ -6157,7 +6157,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BucketStatistics bucketStatistics = 5; + * optional .orc.proto.BucketStatistics bucketStatistics = 5; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.BucketStatisticsOrBuilder> @@ -6173,18 +6173,18 @@ return bucketStatisticsBuilder_; } - // optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + // optional .orc.proto.DecimalStatistics decimalStatistics = 6; private org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics decimalStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatisticsOrBuilder> decimalStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public boolean hasDecimalStatistics() { return ((bitField0_ & 0x00000020) == 0x00000020); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics getDecimalStatistics() { if (decimalStatisticsBuilder_ == null) { @@ -6194,7 +6194,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public Builder setDecimalStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics value) { if (decimalStatisticsBuilder_ == null) { @@ -6210,7 +6210,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public Builder setDecimalStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.Builder builderForValue) { @@ -6224,7 +6224,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public Builder mergeDecimalStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics value) { if (decimalStatisticsBuilder_ == null) { @@ -6243,7 +6243,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public Builder clearDecimalStatistics() { if (decimalStatisticsBuilder_ == null) { @@ -6256,7 +6256,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.Builder getDecimalStatisticsBuilder() { bitField0_ |= 0x00000020; @@ -6264,7 +6264,7 @@ return getDecimalStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatisticsOrBuilder getDecimalStatisticsOrBuilder() { if (decimalStatisticsBuilder_ != null) { @@ -6274,7 +6274,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DecimalStatistics decimalStatistics = 6; + * optional .orc.proto.DecimalStatistics decimalStatistics = 6; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.DecimalStatisticsOrBuilder> @@ -6290,18 +6290,18 @@ return decimalStatisticsBuilder_; } - // optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + // optional .orc.proto.DateStatistics dateStatistics = 7; private org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics dateStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatisticsOrBuilder> dateStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public boolean hasDateStatistics() { return ((bitField0_ & 0x00000040) == 0x00000040); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics getDateStatistics() { if (dateStatisticsBuilder_ == null) { @@ -6311,7 +6311,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public Builder setDateStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics value) { if (dateStatisticsBuilder_ == null) { @@ -6327,7 +6327,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public Builder setDateStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.Builder builderForValue) { @@ -6341,7 +6341,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public Builder mergeDateStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics value) { if (dateStatisticsBuilder_ == null) { @@ -6360,7 +6360,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public Builder clearDateStatistics() { if (dateStatisticsBuilder_ == null) { @@ -6373,7 +6373,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.Builder getDateStatisticsBuilder() { bitField0_ |= 0x00000040; @@ -6381,7 +6381,7 @@ return getDateStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatisticsOrBuilder getDateStatisticsOrBuilder() { if (dateStatisticsBuilder_ != null) { @@ -6391,7 +6391,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.DateStatistics dateStatistics = 7; + * optional .orc.proto.DateStatistics dateStatistics = 7; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.DateStatisticsOrBuilder> @@ -6407,18 +6407,18 @@ return dateStatisticsBuilder_; } - // optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + // optional .orc.proto.BinaryStatistics binaryStatistics = 8; private org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics binaryStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatisticsOrBuilder> binaryStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public boolean hasBinaryStatistics() { return ((bitField0_ & 0x00000080) == 0x00000080); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics getBinaryStatistics() { if (binaryStatisticsBuilder_ == null) { @@ -6428,7 +6428,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public Builder setBinaryStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics value) { if (binaryStatisticsBuilder_ == null) { @@ -6444,7 +6444,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public Builder setBinaryStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.Builder builderForValue) { @@ -6458,7 +6458,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public Builder mergeBinaryStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics value) { if (binaryStatisticsBuilder_ == null) { @@ -6477,7 +6477,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public Builder clearBinaryStatistics() { if (binaryStatisticsBuilder_ == null) { @@ -6490,7 +6490,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.Builder getBinaryStatisticsBuilder() { bitField0_ |= 0x00000080; @@ -6498,7 +6498,7 @@ return getBinaryStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatisticsOrBuilder getBinaryStatisticsOrBuilder() { if (binaryStatisticsBuilder_ != null) { @@ -6508,7 +6508,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.BinaryStatistics binaryStatistics = 8; + * optional .orc.proto.BinaryStatistics binaryStatistics = 8; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.BinaryStatisticsOrBuilder> @@ -6524,18 +6524,18 @@ return binaryStatisticsBuilder_; } - // optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + // optional .orc.proto.TimestampStatistics timestampStatistics = 9; private org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics timestampStatistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatisticsOrBuilder> timestampStatisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public boolean hasTimestampStatistics() { return ((bitField0_ & 0x00000100) == 0x00000100); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics getTimestampStatistics() { if (timestampStatisticsBuilder_ == null) { @@ -6545,7 +6545,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public Builder setTimestampStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics value) { if (timestampStatisticsBuilder_ == null) { @@ -6561,7 +6561,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public Builder setTimestampStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.Builder builderForValue) { @@ -6575,7 +6575,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public Builder mergeTimestampStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics value) { if (timestampStatisticsBuilder_ == null) { @@ -6594,7 +6594,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public Builder clearTimestampStatistics() { if (timestampStatisticsBuilder_ == null) { @@ -6607,7 +6607,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.Builder getTimestampStatisticsBuilder() { bitField0_ |= 0x00000100; @@ -6615,7 +6615,7 @@ return getTimestampStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatisticsOrBuilder getTimestampStatisticsOrBuilder() { if (timestampStatisticsBuilder_ != null) { @@ -6625,7 +6625,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.TimestampStatistics timestampStatistics = 9; + * optional .orc.proto.TimestampStatistics timestampStatistics = 9; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.TimestampStatisticsOrBuilder> @@ -6674,7 +6674,7 @@ return this; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.ColumnStatistics) + // @@protoc_insertion_point(builder_scope:orc.proto.ColumnStatistics) } static { @@ -6682,7 +6682,7 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.ColumnStatistics) + // @@protoc_insertion_point(class_scope:orc.proto.ColumnStatistics) } public interface RowIndexEntryOrBuilder @@ -6702,22 +6702,22 @@ */ long getPositions(int index); - // optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + // optional .orc.proto.ColumnStatistics statistics = 2; /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ boolean hasStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getStatistics(); /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getStatisticsOrBuilder(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.RowIndexEntry} + * Protobuf type {@code orc.proto.RowIndexEntry} */ public static final class RowIndexEntry extends com.google.protobuf.GeneratedMessage @@ -6818,12 +6818,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndexEntry_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndexEntry_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder.class); } @@ -6868,23 +6868,23 @@ } private int positionsMemoizedSerializedSize = -1; - // optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + // optional .orc.proto.ColumnStatistics statistics = 2; public static final int STATISTICS_FIELD_NUMBER = 2; private org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics statistics_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public boolean hasStatistics() { return ((bitField0_ & 0x00000001) == 0x00000001); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getStatistics() { return statistics_; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getStatisticsOrBuilder() { return statistics_; @@ -7022,19 +7022,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.RowIndexEntry} + * Protobuf type {@code orc.proto.RowIndexEntry} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntryOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndexEntry_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndexEntry_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder.class); } @@ -7077,7 +7077,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndexEntry_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry getDefaultInstanceForType() { @@ -7231,18 +7231,18 @@ return this; } - // optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + // optional .orc.proto.ColumnStatistics statistics = 2; private org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics statistics_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.getDefaultInstance(); private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder> statisticsBuilder_; /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public boolean hasStatistics() { return ((bitField0_ & 0x00000002) == 0x00000002); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getStatistics() { if (statisticsBuilder_ == null) { @@ -7252,7 +7252,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public Builder setStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) { if (statisticsBuilder_ == null) { @@ -7268,7 +7268,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public Builder setStatistics( org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder builderForValue) { @@ -7282,7 +7282,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public Builder mergeStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) { if (statisticsBuilder_ == null) { @@ -7301,7 +7301,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public Builder clearStatistics() { if (statisticsBuilder_ == null) { @@ -7314,7 +7314,7 @@ return this; } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder getStatisticsBuilder() { bitField0_ |= 0x00000002; @@ -7322,7 +7322,7 @@ return getStatisticsFieldBuilder().getBuilder(); } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getStatisticsOrBuilder() { if (statisticsBuilder_ != null) { @@ -7332,7 +7332,7 @@ } } /** - * optional .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 2; + * optional .orc.proto.ColumnStatistics statistics = 2; */ private com.google.protobuf.SingleFieldBuilder< org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder> @@ -7348,7 +7348,7 @@ return statisticsBuilder_; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.RowIndexEntry) + // @@protoc_insertion_point(builder_scope:orc.proto.RowIndexEntry) } static { @@ -7356,39 +7356,39 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.RowIndexEntry) + // @@protoc_insertion_point(class_scope:orc.proto.RowIndexEntry) } public interface RowIndexOrBuilder extends com.google.protobuf.MessageOrBuilder { - // repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + // repeated .orc.proto.RowIndexEntry entry = 1; /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ java.util.List getEntryList(); /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry getEntry(int index); /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ int getEntryCount(); /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ java.util.List getEntryOrBuilderList(); /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntryOrBuilder getEntryOrBuilder( int index); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.RowIndex} + * Protobuf type {@code orc.proto.RowIndex} */ public static final class RowIndex extends com.google.protobuf.GeneratedMessage @@ -7463,12 +7463,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndex_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndex_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.Builder.class); } @@ -7488,36 +7488,36 @@ return PARSER; } - // repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + // repeated .orc.proto.RowIndexEntry entry = 1; public static final int ENTRY_FIELD_NUMBER = 1; private java.util.List entry_; /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public java.util.List getEntryList() { return entry_; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public java.util.List getEntryOrBuilderList() { return entry_; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public int getEntryCount() { return entry_.size(); } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry getEntry(int index) { return entry_.get(index); } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntryOrBuilder getEntryOrBuilder( int index) { @@ -7634,19 +7634,19 @@ return builder; } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.RowIndex} + * Protobuf type {@code orc.proto.RowIndex} */ public static final class Builder extends com.google.protobuf.GeneratedMessage.Builder implements org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexOrBuilder { public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndex_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndex_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex.Builder.class); } @@ -7687,7 +7687,7 @@ public com.google.protobuf.Descriptors.Descriptor getDescriptorForType() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_RowIndex_descriptor; } public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex getDefaultInstanceForType() { @@ -7782,7 +7782,7 @@ } private int bitField0_; - // repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + // repeated .orc.proto.RowIndexEntry entry = 1; private java.util.List entry_ = java.util.Collections.emptyList(); private void ensureEntryIsMutable() { @@ -7796,7 +7796,7 @@ org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntryOrBuilder> entryBuilder_; /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public java.util.List getEntryList() { if (entryBuilder_ == null) { @@ -7806,7 +7806,7 @@ } } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public int getEntryCount() { if (entryBuilder_ == null) { @@ -7816,7 +7816,7 @@ } } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry getEntry(int index) { if (entryBuilder_ == null) { @@ -7826,7 +7826,7 @@ } } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder setEntry( int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry value) { @@ -7843,7 +7843,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder setEntry( int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder builderForValue) { @@ -7857,7 +7857,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder addEntry(org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry value) { if (entryBuilder_ == null) { @@ -7873,7 +7873,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder addEntry( int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry value) { @@ -7890,7 +7890,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder addEntry( org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder builderForValue) { @@ -7904,7 +7904,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder addEntry( int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder builderForValue) { @@ -7918,7 +7918,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder addAllEntry( java.lang.Iterable values) { @@ -7932,7 +7932,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder clearEntry() { if (entryBuilder_ == null) { @@ -7945,7 +7945,7 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public Builder removeEntry(int index) { if (entryBuilder_ == null) { @@ -7958,14 +7958,14 @@ return this; } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder getEntryBuilder( int index) { return getEntryFieldBuilder().getBuilder(index); } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntryOrBuilder getEntryOrBuilder( int index) { @@ -7975,7 +7975,7 @@ } } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public java.util.List getEntryOrBuilderList() { @@ -7986,14 +7986,14 @@ } } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder addEntryBuilder() { return getEntryFieldBuilder().addBuilder( org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.getDefaultInstance()); } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.Builder addEntryBuilder( int index) { @@ -8001,7 +8001,7 @@ index, org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry.getDefaultInstance()); } /** - * repeated .org.apache.hadoop.hive.ql.io.orc.RowIndexEntry entry = 1; + * repeated .orc.proto.RowIndexEntry entry = 1; */ public java.util.List getEntryBuilderList() { @@ -8022,7 +8022,7 @@ return entryBuilder_; } - // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.RowIndex) + // @@protoc_insertion_point(builder_scope:orc.proto.RowIndex) } static { @@ -8030,19 +8030,1243 @@ defaultInstance.initFields(); } - // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.RowIndex) + // @@protoc_insertion_point(class_scope:orc.proto.RowIndex) } + public interface BloomFilterOrBuilder + extends com.google.protobuf.MessageOrBuilder { + + // optional uint32 numHashFunctions = 1; + /** + * optional uint32 numHashFunctions = 1; + */ + boolean hasNumHashFunctions(); + /** + * optional uint32 numHashFunctions = 1; + */ + int getNumHashFunctions(); + + // repeated fixed64 bitset = 2; + /** + * repeated fixed64 bitset = 2; + */ + java.util.List getBitsetList(); + /** + * repeated fixed64 bitset = 2; + */ + int getBitsetCount(); + /** + * repeated fixed64 bitset = 2; + */ + long getBitset(int index); + } + /** + * Protobuf type {@code orc.proto.BloomFilter} + */ + public static final class BloomFilter extends + com.google.protobuf.GeneratedMessage + implements BloomFilterOrBuilder { + // Use BloomFilter.newBuilder() to construct. + private BloomFilter(com.google.protobuf.GeneratedMessage.Builder builder) { + super(builder); + this.unknownFields = builder.getUnknownFields(); + } + private BloomFilter(boolean noInit) { this.unknownFields = com.google.protobuf.UnknownFieldSet.getDefaultInstance(); } + + private static final BloomFilter defaultInstance; + public static BloomFilter getDefaultInstance() { + return defaultInstance; + } + + public BloomFilter getDefaultInstanceForType() { + return defaultInstance; + } + + private final com.google.protobuf.UnknownFieldSet unknownFields; + @java.lang.Override + public final com.google.protobuf.UnknownFieldSet + getUnknownFields() { + return this.unknownFields; + } + private BloomFilter( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + initFields(); + int mutable_bitField0_ = 0; + com.google.protobuf.UnknownFieldSet.Builder unknownFields = + com.google.protobuf.UnknownFieldSet.newBuilder(); + try { + boolean done = false; + while (!done) { + int tag = input.readTag(); + switch (tag) { + case 0: + done = true; + break; + default: { + if (!parseUnknownField(input, unknownFields, + extensionRegistry, tag)) { + done = true; + } + break; + } + case 8: { + bitField0_ |= 0x00000001; + numHashFunctions_ = input.readUInt32(); + break; + } + case 17: { + if (!((mutable_bitField0_ & 0x00000002) == 0x00000002)) { + bitset_ = new java.util.ArrayList(); + mutable_bitField0_ |= 0x00000002; + } + bitset_.add(input.readFixed64()); + break; + } + case 18: { + int length = input.readRawVarint32(); + int limit = input.pushLimit(length); + if (!((mutable_bitField0_ & 0x00000002) == 0x00000002) && input.getBytesUntilLimit() > 0) { + bitset_ = new java.util.ArrayList(); + mutable_bitField0_ |= 0x00000002; + } + while (input.getBytesUntilLimit() > 0) { + bitset_.add(input.readFixed64()); + } + input.popLimit(limit); + break; + } + } + } + } catch (com.google.protobuf.InvalidProtocolBufferException e) { + throw e.setUnfinishedMessage(this); + } catch (java.io.IOException e) { + throw new com.google.protobuf.InvalidProtocolBufferException( + e.getMessage()).setUnfinishedMessage(this); + } finally { + if (((mutable_bitField0_ & 0x00000002) == 0x00000002)) { + bitset_ = java.util.Collections.unmodifiableList(bitset_); + } + this.unknownFields = unknownFields.build(); + makeExtensionsImmutable(); + } + } + public static final com.google.protobuf.Descriptors.Descriptor + getDescriptor() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilter_descriptor; + } + + protected com.google.protobuf.GeneratedMessage.FieldAccessorTable + internalGetFieldAccessorTable() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilter_fieldAccessorTable + .ensureFieldAccessorsInitialized( + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder.class); + } + + public static com.google.protobuf.Parser PARSER = + new com.google.protobuf.AbstractParser() { + public BloomFilter parsePartialFrom( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + return new BloomFilter(input, extensionRegistry); + } + }; + + @java.lang.Override + public com.google.protobuf.Parser getParserForType() { + return PARSER; + } + + private int bitField0_; + // optional uint32 numHashFunctions = 1; + public static final int NUMHASHFUNCTIONS_FIELD_NUMBER = 1; + private int numHashFunctions_; + /** + * optional uint32 numHashFunctions = 1; + */ + public boolean hasNumHashFunctions() { + return ((bitField0_ & 0x00000001) == 0x00000001); + } + /** + * optional uint32 numHashFunctions = 1; + */ + public int getNumHashFunctions() { + return numHashFunctions_; + } + + // repeated fixed64 bitset = 2; + public static final int BITSET_FIELD_NUMBER = 2; + private java.util.List bitset_; + /** + * repeated fixed64 bitset = 2; + */ + public java.util.List + getBitsetList() { + return bitset_; + } + /** + * repeated fixed64 bitset = 2; + */ + public int getBitsetCount() { + return bitset_.size(); + } + /** + * repeated fixed64 bitset = 2; + */ + public long getBitset(int index) { + return bitset_.get(index); + } + + private void initFields() { + numHashFunctions_ = 0; + bitset_ = java.util.Collections.emptyList(); + } + private byte memoizedIsInitialized = -1; + public final boolean isInitialized() { + byte isInitialized = memoizedIsInitialized; + if (isInitialized != -1) return isInitialized == 1; + + memoizedIsInitialized = 1; + return true; + } + + public void writeTo(com.google.protobuf.CodedOutputStream output) + throws java.io.IOException { + getSerializedSize(); + if (((bitField0_ & 0x00000001) == 0x00000001)) { + output.writeUInt32(1, numHashFunctions_); + } + for (int i = 0; i < bitset_.size(); i++) { + output.writeFixed64(2, bitset_.get(i)); + } + getUnknownFields().writeTo(output); + } + + private int memoizedSerializedSize = -1; + public int getSerializedSize() { + int size = memoizedSerializedSize; + if (size != -1) return size; + + size = 0; + if (((bitField0_ & 0x00000001) == 0x00000001)) { + size += com.google.protobuf.CodedOutputStream + .computeUInt32Size(1, numHashFunctions_); + } + { + int dataSize = 0; + dataSize = 8 * getBitsetList().size(); + size += dataSize; + size += 1 * getBitsetList().size(); + } + size += getUnknownFields().getSerializedSize(); + memoizedSerializedSize = size; + return size; + } + + private static final long serialVersionUID = 0L; + @java.lang.Override + protected java.lang.Object writeReplace() + throws java.io.ObjectStreamException { + return super.writeReplace(); + } + + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom( + com.google.protobuf.ByteString data) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom( + com.google.protobuf.ByteString data, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom(byte[] data) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom( + byte[] data, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom(java.io.InputStream input) + throws java.io.IOException { + return PARSER.parseFrom(input); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom( + java.io.InputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + return PARSER.parseFrom(input, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseDelimitedFrom(java.io.InputStream input) + throws java.io.IOException { + return PARSER.parseDelimitedFrom(input); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseDelimitedFrom( + java.io.InputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + return PARSER.parseDelimitedFrom(input, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom( + com.google.protobuf.CodedInputStream input) + throws java.io.IOException { + return PARSER.parseFrom(input); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parseFrom( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + return PARSER.parseFrom(input, extensionRegistry); + } + + public static Builder newBuilder() { return Builder.create(); } + public Builder newBuilderForType() { return newBuilder(); } + public static Builder newBuilder(org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter prototype) { + return newBuilder().mergeFrom(prototype); + } + public Builder toBuilder() { return newBuilder(this); } + + @java.lang.Override + protected Builder newBuilderForType( + com.google.protobuf.GeneratedMessage.BuilderParent parent) { + Builder builder = new Builder(parent); + return builder; + } + /** + * Protobuf type {@code orc.proto.BloomFilter} + */ + public static final class Builder extends + com.google.protobuf.GeneratedMessage.Builder + implements org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterOrBuilder { + public static final com.google.protobuf.Descriptors.Descriptor + getDescriptor() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilter_descriptor; + } + + protected com.google.protobuf.GeneratedMessage.FieldAccessorTable + internalGetFieldAccessorTable() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilter_fieldAccessorTable + .ensureFieldAccessorsInitialized( + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder.class); + } + + // Construct using org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.newBuilder() + private Builder() { + maybeForceBuilderInitialization(); + } + + private Builder( + com.google.protobuf.GeneratedMessage.BuilderParent parent) { + super(parent); + maybeForceBuilderInitialization(); + } + private void maybeForceBuilderInitialization() { + if (com.google.protobuf.GeneratedMessage.alwaysUseFieldBuilders) { + } + } + private static Builder create() { + return new Builder(); + } + + public Builder clear() { + super.clear(); + numHashFunctions_ = 0; + bitField0_ = (bitField0_ & ~0x00000001); + bitset_ = java.util.Collections.emptyList(); + bitField0_ = (bitField0_ & ~0x00000002); + return this; + } + + public Builder clone() { + return create().mergeFrom(buildPartial()); + } + + public com.google.protobuf.Descriptors.Descriptor + getDescriptorForType() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilter_descriptor; + } + + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter getDefaultInstanceForType() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.getDefaultInstance(); + } + + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter build() { + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter result = buildPartial(); + if (!result.isInitialized()) { + throw newUninitializedMessageException(result); + } + return result; + } + + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter buildPartial() { + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter result = new org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter(this); + int from_bitField0_ = bitField0_; + int to_bitField0_ = 0; + if (((from_bitField0_ & 0x00000001) == 0x00000001)) { + to_bitField0_ |= 0x00000001; + } + result.numHashFunctions_ = numHashFunctions_; + if (((bitField0_ & 0x00000002) == 0x00000002)) { + bitset_ = java.util.Collections.unmodifiableList(bitset_); + bitField0_ = (bitField0_ & ~0x00000002); + } + result.bitset_ = bitset_; + result.bitField0_ = to_bitField0_; + onBuilt(); + return result; + } + + public Builder mergeFrom(com.google.protobuf.Message other) { + if (other instanceof org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter) { + return mergeFrom((org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter)other); + } else { + super.mergeFrom(other); + return this; + } + } + + public Builder mergeFrom(org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter other) { + if (other == org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.getDefaultInstance()) return this; + if (other.hasNumHashFunctions()) { + setNumHashFunctions(other.getNumHashFunctions()); + } + if (!other.bitset_.isEmpty()) { + if (bitset_.isEmpty()) { + bitset_ = other.bitset_; + bitField0_ = (bitField0_ & ~0x00000002); + } else { + ensureBitsetIsMutable(); + bitset_.addAll(other.bitset_); + } + onChanged(); + } + this.mergeUnknownFields(other.getUnknownFields()); + return this; + } + + public final boolean isInitialized() { + return true; + } + + public Builder mergeFrom( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter parsedMessage = null; + try { + parsedMessage = PARSER.parsePartialFrom(input, extensionRegistry); + } catch (com.google.protobuf.InvalidProtocolBufferException e) { + parsedMessage = (org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter) e.getUnfinishedMessage(); + throw e; + } finally { + if (parsedMessage != null) { + mergeFrom(parsedMessage); + } + } + return this; + } + private int bitField0_; + + // optional uint32 numHashFunctions = 1; + private int numHashFunctions_ ; + /** + * optional uint32 numHashFunctions = 1; + */ + public boolean hasNumHashFunctions() { + return ((bitField0_ & 0x00000001) == 0x00000001); + } + /** + * optional uint32 numHashFunctions = 1; + */ + public int getNumHashFunctions() { + return numHashFunctions_; + } + /** + * optional uint32 numHashFunctions = 1; + */ + public Builder setNumHashFunctions(int value) { + bitField0_ |= 0x00000001; + numHashFunctions_ = value; + onChanged(); + return this; + } + /** + * optional uint32 numHashFunctions = 1; + */ + public Builder clearNumHashFunctions() { + bitField0_ = (bitField0_ & ~0x00000001); + numHashFunctions_ = 0; + onChanged(); + return this; + } + + // repeated fixed64 bitset = 2; + private java.util.List bitset_ = java.util.Collections.emptyList(); + private void ensureBitsetIsMutable() { + if (!((bitField0_ & 0x00000002) == 0x00000002)) { + bitset_ = new java.util.ArrayList(bitset_); + bitField0_ |= 0x00000002; + } + } + /** + * repeated fixed64 bitset = 2; + */ + public java.util.List + getBitsetList() { + return java.util.Collections.unmodifiableList(bitset_); + } + /** + * repeated fixed64 bitset = 2; + */ + public int getBitsetCount() { + return bitset_.size(); + } + /** + * repeated fixed64 bitset = 2; + */ + public long getBitset(int index) { + return bitset_.get(index); + } + /** + * repeated fixed64 bitset = 2; + */ + public Builder setBitset( + int index, long value) { + ensureBitsetIsMutable(); + bitset_.set(index, value); + onChanged(); + return this; + } + /** + * repeated fixed64 bitset = 2; + */ + public Builder addBitset(long value) { + ensureBitsetIsMutable(); + bitset_.add(value); + onChanged(); + return this; + } + /** + * repeated fixed64 bitset = 2; + */ + public Builder addAllBitset( + java.lang.Iterable values) { + ensureBitsetIsMutable(); + super.addAll(values, bitset_); + onChanged(); + return this; + } + /** + * repeated fixed64 bitset = 2; + */ + public Builder clearBitset() { + bitset_ = java.util.Collections.emptyList(); + bitField0_ = (bitField0_ & ~0x00000002); + onChanged(); + return this; + } + + // @@protoc_insertion_point(builder_scope:orc.proto.BloomFilter) + } + + static { + defaultInstance = new BloomFilter(true); + defaultInstance.initFields(); + } + + // @@protoc_insertion_point(class_scope:orc.proto.BloomFilter) + } + + public interface BloomFilterIndexOrBuilder + extends com.google.protobuf.MessageOrBuilder { + + // repeated .orc.proto.BloomFilter bloomFilter = 1; + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + java.util.List + getBloomFilterList(); + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter getBloomFilter(int index); + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + int getBloomFilterCount(); + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + java.util.List + getBloomFilterOrBuilderList(); + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterOrBuilder getBloomFilterOrBuilder( + int index); + } + /** + * Protobuf type {@code orc.proto.BloomFilterIndex} + */ + public static final class BloomFilterIndex extends + com.google.protobuf.GeneratedMessage + implements BloomFilterIndexOrBuilder { + // Use BloomFilterIndex.newBuilder() to construct. + private BloomFilterIndex(com.google.protobuf.GeneratedMessage.Builder builder) { + super(builder); + this.unknownFields = builder.getUnknownFields(); + } + private BloomFilterIndex(boolean noInit) { this.unknownFields = com.google.protobuf.UnknownFieldSet.getDefaultInstance(); } + + private static final BloomFilterIndex defaultInstance; + public static BloomFilterIndex getDefaultInstance() { + return defaultInstance; + } + + public BloomFilterIndex getDefaultInstanceForType() { + return defaultInstance; + } + + private final com.google.protobuf.UnknownFieldSet unknownFields; + @java.lang.Override + public final com.google.protobuf.UnknownFieldSet + getUnknownFields() { + return this.unknownFields; + } + private BloomFilterIndex( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + initFields(); + int mutable_bitField0_ = 0; + com.google.protobuf.UnknownFieldSet.Builder unknownFields = + com.google.protobuf.UnknownFieldSet.newBuilder(); + try { + boolean done = false; + while (!done) { + int tag = input.readTag(); + switch (tag) { + case 0: + done = true; + break; + default: { + if (!parseUnknownField(input, unknownFields, + extensionRegistry, tag)) { + done = true; + } + break; + } + case 10: { + if (!((mutable_bitField0_ & 0x00000001) == 0x00000001)) { + bloomFilter_ = new java.util.ArrayList(); + mutable_bitField0_ |= 0x00000001; + } + bloomFilter_.add(input.readMessage(org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.PARSER, extensionRegistry)); + break; + } + } + } + } catch (com.google.protobuf.InvalidProtocolBufferException e) { + throw e.setUnfinishedMessage(this); + } catch (java.io.IOException e) { + throw new com.google.protobuf.InvalidProtocolBufferException( + e.getMessage()).setUnfinishedMessage(this); + } finally { + if (((mutable_bitField0_ & 0x00000001) == 0x00000001)) { + bloomFilter_ = java.util.Collections.unmodifiableList(bloomFilter_); + } + this.unknownFields = unknownFields.build(); + makeExtensionsImmutable(); + } + } + public static final com.google.protobuf.Descriptors.Descriptor + getDescriptor() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilterIndex_descriptor; + } + + protected com.google.protobuf.GeneratedMessage.FieldAccessorTable + internalGetFieldAccessorTable() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilterIndex_fieldAccessorTable + .ensureFieldAccessorsInitialized( + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.Builder.class); + } + + public static com.google.protobuf.Parser PARSER = + new com.google.protobuf.AbstractParser() { + public BloomFilterIndex parsePartialFrom( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + return new BloomFilterIndex(input, extensionRegistry); + } + }; + + @java.lang.Override + public com.google.protobuf.Parser getParserForType() { + return PARSER; + } + + // repeated .orc.proto.BloomFilter bloomFilter = 1; + public static final int BLOOMFILTER_FIELD_NUMBER = 1; + private java.util.List bloomFilter_; + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public java.util.List getBloomFilterList() { + return bloomFilter_; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public java.util.List + getBloomFilterOrBuilderList() { + return bloomFilter_; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public int getBloomFilterCount() { + return bloomFilter_.size(); + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter getBloomFilter(int index) { + return bloomFilter_.get(index); + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterOrBuilder getBloomFilterOrBuilder( + int index) { + return bloomFilter_.get(index); + } + + private void initFields() { + bloomFilter_ = java.util.Collections.emptyList(); + } + private byte memoizedIsInitialized = -1; + public final boolean isInitialized() { + byte isInitialized = memoizedIsInitialized; + if (isInitialized != -1) return isInitialized == 1; + + memoizedIsInitialized = 1; + return true; + } + + public void writeTo(com.google.protobuf.CodedOutputStream output) + throws java.io.IOException { + getSerializedSize(); + for (int i = 0; i < bloomFilter_.size(); i++) { + output.writeMessage(1, bloomFilter_.get(i)); + } + getUnknownFields().writeTo(output); + } + + private int memoizedSerializedSize = -1; + public int getSerializedSize() { + int size = memoizedSerializedSize; + if (size != -1) return size; + + size = 0; + for (int i = 0; i < bloomFilter_.size(); i++) { + size += com.google.protobuf.CodedOutputStream + .computeMessageSize(1, bloomFilter_.get(i)); + } + size += getUnknownFields().getSerializedSize(); + memoizedSerializedSize = size; + return size; + } + + private static final long serialVersionUID = 0L; + @java.lang.Override + protected java.lang.Object writeReplace() + throws java.io.ObjectStreamException { + return super.writeReplace(); + } + + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom( + com.google.protobuf.ByteString data) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom( + com.google.protobuf.ByteString data, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom(byte[] data) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom( + byte[] data, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws com.google.protobuf.InvalidProtocolBufferException { + return PARSER.parseFrom(data, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom(java.io.InputStream input) + throws java.io.IOException { + return PARSER.parseFrom(input); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom( + java.io.InputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + return PARSER.parseFrom(input, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseDelimitedFrom(java.io.InputStream input) + throws java.io.IOException { + return PARSER.parseDelimitedFrom(input); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseDelimitedFrom( + java.io.InputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + return PARSER.parseDelimitedFrom(input, extensionRegistry); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom( + com.google.protobuf.CodedInputStream input) + throws java.io.IOException { + return PARSER.parseFrom(input); + } + public static org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parseFrom( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + return PARSER.parseFrom(input, extensionRegistry); + } + + public static Builder newBuilder() { return Builder.create(); } + public Builder newBuilderForType() { return newBuilder(); } + public static Builder newBuilder(org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex prototype) { + return newBuilder().mergeFrom(prototype); + } + public Builder toBuilder() { return newBuilder(this); } + + @java.lang.Override + protected Builder newBuilderForType( + com.google.protobuf.GeneratedMessage.BuilderParent parent) { + Builder builder = new Builder(parent); + return builder; + } + /** + * Protobuf type {@code orc.proto.BloomFilterIndex} + */ + public static final class Builder extends + com.google.protobuf.GeneratedMessage.Builder + implements org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndexOrBuilder { + public static final com.google.protobuf.Descriptors.Descriptor + getDescriptor() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilterIndex_descriptor; + } + + protected com.google.protobuf.GeneratedMessage.FieldAccessorTable + internalGetFieldAccessorTable() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilterIndex_fieldAccessorTable + .ensureFieldAccessorsInitialized( + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.Builder.class); + } + + // Construct using org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.newBuilder() + private Builder() { + maybeForceBuilderInitialization(); + } + + private Builder( + com.google.protobuf.GeneratedMessage.BuilderParent parent) { + super(parent); + maybeForceBuilderInitialization(); + } + private void maybeForceBuilderInitialization() { + if (com.google.protobuf.GeneratedMessage.alwaysUseFieldBuilders) { + getBloomFilterFieldBuilder(); + } + } + private static Builder create() { + return new Builder(); + } + + public Builder clear() { + super.clear(); + if (bloomFilterBuilder_ == null) { + bloomFilter_ = java.util.Collections.emptyList(); + bitField0_ = (bitField0_ & ~0x00000001); + } else { + bloomFilterBuilder_.clear(); + } + return this; + } + + public Builder clone() { + return create().mergeFrom(buildPartial()); + } + + public com.google.protobuf.Descriptors.Descriptor + getDescriptorForType() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_BloomFilterIndex_descriptor; + } + + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex getDefaultInstanceForType() { + return org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.getDefaultInstance(); + } + + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex build() { + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex result = buildPartial(); + if (!result.isInitialized()) { + throw newUninitializedMessageException(result); + } + return result; + } + + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex buildPartial() { + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex result = new org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex(this); + int from_bitField0_ = bitField0_; + if (bloomFilterBuilder_ == null) { + if (((bitField0_ & 0x00000001) == 0x00000001)) { + bloomFilter_ = java.util.Collections.unmodifiableList(bloomFilter_); + bitField0_ = (bitField0_ & ~0x00000001); + } + result.bloomFilter_ = bloomFilter_; + } else { + result.bloomFilter_ = bloomFilterBuilder_.build(); + } + onBuilt(); + return result; + } + + public Builder mergeFrom(com.google.protobuf.Message other) { + if (other instanceof org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex) { + return mergeFrom((org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex)other); + } else { + super.mergeFrom(other); + return this; + } + } + + public Builder mergeFrom(org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex other) { + if (other == org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex.getDefaultInstance()) return this; + if (bloomFilterBuilder_ == null) { + if (!other.bloomFilter_.isEmpty()) { + if (bloomFilter_.isEmpty()) { + bloomFilter_ = other.bloomFilter_; + bitField0_ = (bitField0_ & ~0x00000001); + } else { + ensureBloomFilterIsMutable(); + bloomFilter_.addAll(other.bloomFilter_); + } + onChanged(); + } + } else { + if (!other.bloomFilter_.isEmpty()) { + if (bloomFilterBuilder_.isEmpty()) { + bloomFilterBuilder_.dispose(); + bloomFilterBuilder_ = null; + bloomFilter_ = other.bloomFilter_; + bitField0_ = (bitField0_ & ~0x00000001); + bloomFilterBuilder_ = + com.google.protobuf.GeneratedMessage.alwaysUseFieldBuilders ? + getBloomFilterFieldBuilder() : null; + } else { + bloomFilterBuilder_.addAllMessages(other.bloomFilter_); + } + } + } + this.mergeUnknownFields(other.getUnknownFields()); + return this; + } + + public final boolean isInitialized() { + return true; + } + + public Builder mergeFrom( + com.google.protobuf.CodedInputStream input, + com.google.protobuf.ExtensionRegistryLite extensionRegistry) + throws java.io.IOException { + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex parsedMessage = null; + try { + parsedMessage = PARSER.parsePartialFrom(input, extensionRegistry); + } catch (com.google.protobuf.InvalidProtocolBufferException e) { + parsedMessage = (org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex) e.getUnfinishedMessage(); + throw e; + } finally { + if (parsedMessage != null) { + mergeFrom(parsedMessage); + } + } + return this; + } + private int bitField0_; + + // repeated .orc.proto.BloomFilter bloomFilter = 1; + private java.util.List bloomFilter_ = + java.util.Collections.emptyList(); + private void ensureBloomFilterIsMutable() { + if (!((bitField0_ & 0x00000001) == 0x00000001)) { + bloomFilter_ = new java.util.ArrayList(bloomFilter_); + bitField0_ |= 0x00000001; + } + } + + private com.google.protobuf.RepeatedFieldBuilder< + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterOrBuilder> bloomFilterBuilder_; + + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public java.util.List getBloomFilterList() { + if (bloomFilterBuilder_ == null) { + return java.util.Collections.unmodifiableList(bloomFilter_); + } else { + return bloomFilterBuilder_.getMessageList(); + } + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public int getBloomFilterCount() { + if (bloomFilterBuilder_ == null) { + return bloomFilter_.size(); + } else { + return bloomFilterBuilder_.getCount(); + } + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter getBloomFilter(int index) { + if (bloomFilterBuilder_ == null) { + return bloomFilter_.get(index); + } else { + return bloomFilterBuilder_.getMessage(index); + } + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder setBloomFilter( + int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter value) { + if (bloomFilterBuilder_ == null) { + if (value == null) { + throw new NullPointerException(); + } + ensureBloomFilterIsMutable(); + bloomFilter_.set(index, value); + onChanged(); + } else { + bloomFilterBuilder_.setMessage(index, value); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder setBloomFilter( + int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder builderForValue) { + if (bloomFilterBuilder_ == null) { + ensureBloomFilterIsMutable(); + bloomFilter_.set(index, builderForValue.build()); + onChanged(); + } else { + bloomFilterBuilder_.setMessage(index, builderForValue.build()); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder addBloomFilter(org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter value) { + if (bloomFilterBuilder_ == null) { + if (value == null) { + throw new NullPointerException(); + } + ensureBloomFilterIsMutable(); + bloomFilter_.add(value); + onChanged(); + } else { + bloomFilterBuilder_.addMessage(value); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder addBloomFilter( + int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter value) { + if (bloomFilterBuilder_ == null) { + if (value == null) { + throw new NullPointerException(); + } + ensureBloomFilterIsMutable(); + bloomFilter_.add(index, value); + onChanged(); + } else { + bloomFilterBuilder_.addMessage(index, value); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder addBloomFilter( + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder builderForValue) { + if (bloomFilterBuilder_ == null) { + ensureBloomFilterIsMutable(); + bloomFilter_.add(builderForValue.build()); + onChanged(); + } else { + bloomFilterBuilder_.addMessage(builderForValue.build()); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder addBloomFilter( + int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder builderForValue) { + if (bloomFilterBuilder_ == null) { + ensureBloomFilterIsMutable(); + bloomFilter_.add(index, builderForValue.build()); + onChanged(); + } else { + bloomFilterBuilder_.addMessage(index, builderForValue.build()); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder addAllBloomFilter( + java.lang.Iterable values) { + if (bloomFilterBuilder_ == null) { + ensureBloomFilterIsMutable(); + super.addAll(values, bloomFilter_); + onChanged(); + } else { + bloomFilterBuilder_.addAllMessages(values); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder clearBloomFilter() { + if (bloomFilterBuilder_ == null) { + bloomFilter_ = java.util.Collections.emptyList(); + bitField0_ = (bitField0_ & ~0x00000001); + onChanged(); + } else { + bloomFilterBuilder_.clear(); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public Builder removeBloomFilter(int index) { + if (bloomFilterBuilder_ == null) { + ensureBloomFilterIsMutable(); + bloomFilter_.remove(index); + onChanged(); + } else { + bloomFilterBuilder_.remove(index); + } + return this; + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder getBloomFilterBuilder( + int index) { + return getBloomFilterFieldBuilder().getBuilder(index); + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterOrBuilder getBloomFilterOrBuilder( + int index) { + if (bloomFilterBuilder_ == null) { + return bloomFilter_.get(index); } else { + return bloomFilterBuilder_.getMessageOrBuilder(index); + } + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public java.util.List + getBloomFilterOrBuilderList() { + if (bloomFilterBuilder_ != null) { + return bloomFilterBuilder_.getMessageOrBuilderList(); + } else { + return java.util.Collections.unmodifiableList(bloomFilter_); + } + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder addBloomFilterBuilder() { + return getBloomFilterFieldBuilder().addBuilder( + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.getDefaultInstance()); + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder addBloomFilterBuilder( + int index) { + return getBloomFilterFieldBuilder().addBuilder( + index, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.getDefaultInstance()); + } + /** + * repeated .orc.proto.BloomFilter bloomFilter = 1; + */ + public java.util.List + getBloomFilterBuilderList() { + return getBloomFilterFieldBuilder().getBuilderList(); + } + private com.google.protobuf.RepeatedFieldBuilder< + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterOrBuilder> + getBloomFilterFieldBuilder() { + if (bloomFilterBuilder_ == null) { + bloomFilterBuilder_ = new com.google.protobuf.RepeatedFieldBuilder< + org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilter.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterOrBuilder>( + bloomFilter_, + ((bitField0_ & 0x00000001) == 0x00000001), + getParentForChildren(), + isClean()); + bloomFilter_ = null; + } + return bloomFilterBuilder_; + } + + // @@protoc_insertion_point(builder_scope:orc.proto.BloomFilterIndex) + } + + static { + defaultInstance = new BloomFilterIndex(true); + defaultInstance.initFields(); + } + + // @@protoc_insertion_point(class_scope:orc.proto.BloomFilterIndex) + } + public interface StreamOrBuilder extends com.google.protobuf.MessageOrBuilder { - // required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1; + // optional .orc.proto.Stream.Kind kind = 1; /** - * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1; + * optional .orc.proto.Stream.Kind kind = 1; */ boolean hasKind(); /** - * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1; + * optional .orc.proto.Stream.Kind kind = 1; */ org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Kind getKind(); @@ -8067,7 +9291,7 @@ long getLength(); } /** - * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Stream} + * Protobuf type {@code orc.proto.Stream} */ public static final class Stream extends com.google.protobuf.GeneratedMessage @@ -8152,12 +9376,12 @@ } public static final com.google.protobuf.Descriptors.Descriptor getDescriptor() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_descriptor; + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Stream_descriptor; } protected com.google.protobuf.GeneratedMessage.FieldAccessorTable internalGetFieldAccessorTable() { - return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_fieldAccessorTable + return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Stream_fieldAccessorTable .ensureFieldAccessorsInitialized( org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder.class); } @@ -8178,7 +9402,7 @@ } /** - * Protobuf enum {@code org.apache.hadoop.hive.ql.io.orc.Stream.Kind} + * Protobuf enum {@code orc.proto.Stream.Kind} * *
      * if you add new index stream kinds, you need to make sure to update
@@ -8215,6 +9439,10 @@
        * ROW_INDEX = 6;
        */
       ROW_INDEX(6, 6),
+      /**
+       * BLOOM_FILTER = 7;
+       */
+      BLOOM_FILTER(7, 7),
       ;
 
       /**
@@ -8245,6 +9473,10 @@
        * ROW_INDEX = 6;
        */
       public static final int ROW_INDEX_VALUE = 6;
+      /**
+       * BLOOM_FILTER = 7;
+       */
+      public static final int BLOOM_FILTER_VALUE = 7;
 
 
       public final int getNumber() { return value; }
@@ -8258,6 +9490,7 @@
           case 4: return DICTIONARY_COUNT;
           case 5: return SECONDARY;
           case 6: return ROW_INDEX;
+          case 7: return BLOOM_FILTER;
           default: return null;
         }
       }
@@ -8306,21 +9539,21 @@
         this.value = value;
       }
 
-      // @@protoc_insertion_point(enum_scope:org.apache.hadoop.hive.ql.io.orc.Stream.Kind)
+      // @@protoc_insertion_point(enum_scope:orc.proto.Stream.Kind)
     }
 
     private int bitField0_;
-    // required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+    // optional .orc.proto.Stream.Kind kind = 1;
     public static final int KIND_FIELD_NUMBER = 1;
     private org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Kind kind_;
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+     * optional .orc.proto.Stream.Kind kind = 1;
      */
     public boolean hasKind() {
       return ((bitField0_ & 0x00000001) == 0x00000001);
     }
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+     * optional .orc.proto.Stream.Kind kind = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Kind getKind() {
       return kind_;
@@ -8368,10 +9601,6 @@
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized != -1) return isInitialized == 1;
 
-      if (!hasKind()) {
-        memoizedIsInitialized = 0;
-        return false;
-      }
       memoizedIsInitialized = 1;
       return true;
     }
@@ -8488,19 +9717,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Stream}
+     * Protobuf type {@code orc.proto.Stream}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.StreamOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Stream_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Stream_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder.class);
       }
@@ -8540,7 +9769,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Stream_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream getDefaultInstanceForType() {
@@ -8601,10 +9830,6 @@
       }
 
       public final boolean isInitialized() {
-        if (!hasKind()) {
-          
-          return false;
-        }
         return true;
       }
 
@@ -8627,22 +9852,22 @@
       }
       private int bitField0_;
 
-      // required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+      // optional .orc.proto.Stream.Kind kind = 1;
       private org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Kind kind_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Kind.PRESENT;
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+       * optional .orc.proto.Stream.Kind kind = 1;
        */
       public boolean hasKind() {
         return ((bitField0_ & 0x00000001) == 0x00000001);
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+       * optional .orc.proto.Stream.Kind kind = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Kind getKind() {
         return kind_;
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+       * optional .orc.proto.Stream.Kind kind = 1;
        */
       public Builder setKind(org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Kind value) {
         if (value == null) {
@@ -8654,7 +9879,7 @@
         return this;
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Stream.Kind kind = 1;
+       * optional .orc.proto.Stream.Kind kind = 1;
        */
       public Builder clearKind() {
         bitField0_ = (bitField0_ & ~0x00000001);
@@ -8729,7 +9954,7 @@
         return this;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.Stream)
+      // @@protoc_insertion_point(builder_scope:orc.proto.Stream)
     }
 
     static {
@@ -8737,19 +9962,19 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.Stream)
+    // @@protoc_insertion_point(class_scope:orc.proto.Stream)
   }
 
   public interface ColumnEncodingOrBuilder
       extends com.google.protobuf.MessageOrBuilder {
 
-    // required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+    // optional .orc.proto.ColumnEncoding.Kind kind = 1;
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+     * optional .orc.proto.ColumnEncoding.Kind kind = 1;
      */
     boolean hasKind();
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+     * optional .orc.proto.ColumnEncoding.Kind kind = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Kind getKind();
 
@@ -8764,7 +9989,7 @@
     int getDictionarySize();
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.ColumnEncoding}
+   * Protobuf type {@code orc.proto.ColumnEncoding}
    */
   public static final class ColumnEncoding extends
       com.google.protobuf.GeneratedMessage
@@ -8844,12 +10069,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnEncoding_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnEncoding_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder.class);
     }
@@ -8870,7 +10095,7 @@
     }
 
     /**
-     * Protobuf enum {@code org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind}
+     * Protobuf enum {@code orc.proto.ColumnEncoding.Kind}
      */
     public enum Kind
         implements com.google.protobuf.ProtocolMessageEnum {
@@ -8966,21 +10191,21 @@
         this.value = value;
       }
 
-      // @@protoc_insertion_point(enum_scope:org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind)
+      // @@protoc_insertion_point(enum_scope:orc.proto.ColumnEncoding.Kind)
     }
 
     private int bitField0_;
-    // required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+    // optional .orc.proto.ColumnEncoding.Kind kind = 1;
     public static final int KIND_FIELD_NUMBER = 1;
     private org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Kind kind_;
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+     * optional .orc.proto.ColumnEncoding.Kind kind = 1;
      */
     public boolean hasKind() {
       return ((bitField0_ & 0x00000001) == 0x00000001);
     }
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+     * optional .orc.proto.ColumnEncoding.Kind kind = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Kind getKind() {
       return kind_;
@@ -9011,10 +10236,6 @@
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized != -1) return isInitialized == 1;
 
-      if (!hasKind()) {
-        memoizedIsInitialized = 0;
-        return false;
-      }
       memoizedIsInitialized = 1;
       return true;
     }
@@ -9124,19 +10345,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.ColumnEncoding}
+     * Protobuf type {@code orc.proto.ColumnEncoding}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncodingOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnEncoding_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnEncoding_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder.class);
       }
@@ -9174,7 +10395,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_ColumnEncoding_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding getDefaultInstanceForType() {
@@ -9228,10 +10449,6 @@
       }
 
       public final boolean isInitialized() {
-        if (!hasKind()) {
-          
-          return false;
-        }
         return true;
       }
 
@@ -9254,22 +10471,22 @@
       }
       private int bitField0_;
 
-      // required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+      // optional .orc.proto.ColumnEncoding.Kind kind = 1;
       private org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Kind kind_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Kind.DIRECT;
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+       * optional .orc.proto.ColumnEncoding.Kind kind = 1;
        */
       public boolean hasKind() {
         return ((bitField0_ & 0x00000001) == 0x00000001);
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+       * optional .orc.proto.ColumnEncoding.Kind kind = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Kind getKind() {
         return kind_;
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+       * optional .orc.proto.ColumnEncoding.Kind kind = 1;
        */
       public Builder setKind(org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Kind value) {
         if (value == null) {
@@ -9281,7 +10498,7 @@
         return this;
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding.Kind kind = 1;
+       * optional .orc.proto.ColumnEncoding.Kind kind = 1;
        */
       public Builder clearKind() {
         bitField0_ = (bitField0_ & ~0x00000001);
@@ -9323,7 +10540,7 @@
         return this;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.ColumnEncoding)
+      // @@protoc_insertion_point(builder_scope:orc.proto.ColumnEncoding)
     }
 
     static {
@@ -9331,64 +10548,64 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.ColumnEncoding)
+    // @@protoc_insertion_point(class_scope:orc.proto.ColumnEncoding)
   }
 
   public interface StripeFooterOrBuilder
       extends com.google.protobuf.MessageOrBuilder {
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+    // repeated .orc.proto.Stream streams = 1;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     java.util.List 
         getStreamsList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream getStreams(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     int getStreamsCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     java.util.List 
         getStreamsOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.StreamOrBuilder getStreamsOrBuilder(
         int index);
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+    // repeated .orc.proto.ColumnEncoding columns = 2;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     java.util.List 
         getColumnsList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding getColumns(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     int getColumnsCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     java.util.List 
         getColumnsOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncodingOrBuilder getColumnsOrBuilder(
         int index);
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StripeFooter}
+   * Protobuf type {@code orc.proto.StripeFooter}
    */
   public static final class StripeFooter extends
       com.google.protobuf.GeneratedMessage
@@ -9474,12 +10691,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeFooter_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeFooter_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.Builder.class);
     }
@@ -9499,72 +10716,72 @@
       return PARSER;
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+    // repeated .orc.proto.Stream streams = 1;
     public static final int STREAMS_FIELD_NUMBER = 1;
     private java.util.List streams_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     public java.util.List getStreamsList() {
       return streams_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     public java.util.List 
         getStreamsOrBuilderList() {
       return streams_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     public int getStreamsCount() {
       return streams_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream getStreams(int index) {
       return streams_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+     * repeated .orc.proto.Stream streams = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.StreamOrBuilder getStreamsOrBuilder(
         int index) {
       return streams_.get(index);
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+    // repeated .orc.proto.ColumnEncoding columns = 2;
     public static final int COLUMNS_FIELD_NUMBER = 2;
     private java.util.List columns_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     public java.util.List getColumnsList() {
       return columns_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     public java.util.List 
         getColumnsOrBuilderList() {
       return columns_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     public int getColumnsCount() {
       return columns_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding getColumns(int index) {
       return columns_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+     * repeated .orc.proto.ColumnEncoding columns = 2;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncodingOrBuilder getColumnsOrBuilder(
         int index) {
@@ -9580,18 +10797,6 @@
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized != -1) return isInitialized == 1;
 
-      for (int i = 0; i < getStreamsCount(); i++) {
-        if (!getStreams(i).isInitialized()) {
-          memoizedIsInitialized = 0;
-          return false;
-        }
-      }
-      for (int i = 0; i < getColumnsCount(); i++) {
-        if (!getColumns(i).isInitialized()) {
-          memoizedIsInitialized = 0;
-          return false;
-        }
-      }
       memoizedIsInitialized = 1;
       return true;
     }
@@ -9701,19 +10906,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StripeFooter}
+     * Protobuf type {@code orc.proto.StripeFooter}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooterOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeFooter_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeFooter_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter.Builder.class);
       }
@@ -9761,7 +10966,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeFooter_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter getDefaultInstanceForType() {
@@ -9869,18 +11074,6 @@
       }
 
       public final boolean isInitialized() {
-        for (int i = 0; i < getStreamsCount(); i++) {
-          if (!getStreams(i).isInitialized()) {
-            
-            return false;
-          }
-        }
-        for (int i = 0; i < getColumnsCount(); i++) {
-          if (!getColumns(i).isInitialized()) {
-            
-            return false;
-          }
-        }
         return true;
       }
 
@@ -9903,7 +11096,7 @@
       }
       private int bitField0_;
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+      // repeated .orc.proto.Stream streams = 1;
       private java.util.List streams_ =
         java.util.Collections.emptyList();
       private void ensureStreamsIsMutable() {
@@ -9917,7 +11110,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.StreamOrBuilder> streamsBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public java.util.List getStreamsList() {
         if (streamsBuilder_ == null) {
@@ -9927,7 +11120,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public int getStreamsCount() {
         if (streamsBuilder_ == null) {
@@ -9937,7 +11130,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream getStreams(int index) {
         if (streamsBuilder_ == null) {
@@ -9947,7 +11140,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder setStreams(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream value) {
@@ -9964,7 +11157,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder setStreams(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder builderForValue) {
@@ -9978,7 +11171,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder addStreams(org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream value) {
         if (streamsBuilder_ == null) {
@@ -9994,7 +11187,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder addStreams(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream value) {
@@ -10011,7 +11204,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder addStreams(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder builderForValue) {
@@ -10025,7 +11218,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder addStreams(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder builderForValue) {
@@ -10039,7 +11232,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder addAllStreams(
           java.lang.Iterable values) {
@@ -10053,7 +11246,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder clearStreams() {
         if (streamsBuilder_ == null) {
@@ -10066,7 +11259,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public Builder removeStreams(int index) {
         if (streamsBuilder_ == null) {
@@ -10079,14 +11272,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder getStreamsBuilder(
           int index) {
         return getStreamsFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StreamOrBuilder getStreamsOrBuilder(
           int index) {
@@ -10096,7 +11289,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public java.util.List 
            getStreamsOrBuilderList() {
@@ -10107,14 +11300,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder addStreamsBuilder() {
         return getStreamsFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.Builder addStreamsBuilder(
           int index) {
@@ -10122,7 +11315,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Stream.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Stream streams = 1;
+       * repeated .orc.proto.Stream streams = 1;
        */
       public java.util.List 
            getStreamsBuilderList() {
@@ -10143,7 +11336,7 @@
         return streamsBuilder_;
       }
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+      // repeated .orc.proto.ColumnEncoding columns = 2;
       private java.util.List columns_ =
         java.util.Collections.emptyList();
       private void ensureColumnsIsMutable() {
@@ -10157,7 +11350,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncodingOrBuilder> columnsBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public java.util.List getColumnsList() {
         if (columnsBuilder_ == null) {
@@ -10167,7 +11360,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public int getColumnsCount() {
         if (columnsBuilder_ == null) {
@@ -10177,7 +11370,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding getColumns(int index) {
         if (columnsBuilder_ == null) {
@@ -10187,7 +11380,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder setColumns(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding value) {
@@ -10204,7 +11397,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder setColumns(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder builderForValue) {
@@ -10218,7 +11411,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder addColumns(org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding value) {
         if (columnsBuilder_ == null) {
@@ -10234,7 +11427,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder addColumns(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding value) {
@@ -10251,7 +11444,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder addColumns(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder builderForValue) {
@@ -10265,7 +11458,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder addColumns(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder builderForValue) {
@@ -10279,7 +11472,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder addAllColumns(
           java.lang.Iterable values) {
@@ -10293,7 +11486,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder clearColumns() {
         if (columnsBuilder_ == null) {
@@ -10306,7 +11499,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public Builder removeColumns(int index) {
         if (columnsBuilder_ == null) {
@@ -10319,14 +11512,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder getColumnsBuilder(
           int index) {
         return getColumnsFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncodingOrBuilder getColumnsOrBuilder(
           int index) {
@@ -10336,7 +11529,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public java.util.List 
            getColumnsOrBuilderList() {
@@ -10347,14 +11540,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder addColumnsBuilder() {
         return getColumnsFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.Builder addColumnsBuilder(
           int index) {
@@ -10362,7 +11555,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnEncoding.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnEncoding columns = 2;
+       * repeated .orc.proto.ColumnEncoding columns = 2;
        */
       public java.util.List 
            getColumnsBuilderList() {
@@ -10383,7 +11576,7 @@
         return columnsBuilder_;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.StripeFooter)
+      // @@protoc_insertion_point(builder_scope:orc.proto.StripeFooter)
     }
 
     static {
@@ -10391,19 +11584,19 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.StripeFooter)
+    // @@protoc_insertion_point(class_scope:orc.proto.StripeFooter)
   }
 
   public interface TypeOrBuilder
       extends com.google.protobuf.MessageOrBuilder {
 
-    // required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+    // optional .orc.proto.Type.Kind kind = 1;
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+     * optional .orc.proto.Type.Kind kind = 1;
      */
     boolean hasKind();
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+     * optional .orc.proto.Type.Kind kind = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Kind getKind();
 
@@ -10472,7 +11665,7 @@
     int getScale();
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Type}
+   * Protobuf type {@code orc.proto.Type}
    */
   public static final class Type extends
       com.google.protobuf.GeneratedMessage
@@ -10597,12 +11790,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Type_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Type_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Type_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Type_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder.class);
     }
@@ -10623,7 +11816,7 @@
     }
 
     /**
-     * Protobuf enum {@code org.apache.hadoop.hive.ql.io.orc.Type.Kind}
+     * Protobuf enum {@code orc.proto.Type.Kind}
      */
     public enum Kind
         implements com.google.protobuf.ProtocolMessageEnum {
@@ -10845,21 +12038,21 @@
         this.value = value;
       }
 
-      // @@protoc_insertion_point(enum_scope:org.apache.hadoop.hive.ql.io.orc.Type.Kind)
+      // @@protoc_insertion_point(enum_scope:orc.proto.Type.Kind)
     }
 
     private int bitField0_;
-    // required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+    // optional .orc.proto.Type.Kind kind = 1;
     public static final int KIND_FIELD_NUMBER = 1;
     private org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Kind kind_;
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+     * optional .orc.proto.Type.Kind kind = 1;
      */
     public boolean hasKind() {
       return ((bitField0_ & 0x00000001) == 0x00000001);
     }
     /**
-     * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+     * optional .orc.proto.Type.Kind kind = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Kind getKind() {
       return kind_;
@@ -10980,10 +12173,6 @@
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized != -1) return isInitialized == 1;
 
-      if (!hasKind()) {
-        memoizedIsInitialized = 0;
-        return false;
-      }
       memoizedIsInitialized = 1;
       return true;
     }
@@ -11140,19 +12329,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Type}
+     * Protobuf type {@code orc.proto.Type}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.TypeOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Type_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Type_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Type_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Type_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder.class);
       }
@@ -11198,7 +12387,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Type_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Type_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type getDefaultInstanceForType() {
@@ -11297,10 +12486,6 @@
       }
 
       public final boolean isInitialized() {
-        if (!hasKind()) {
-          
-          return false;
-        }
         return true;
       }
 
@@ -11323,22 +12508,22 @@
       }
       private int bitField0_;
 
-      // required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+      // optional .orc.proto.Type.Kind kind = 1;
       private org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Kind kind_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Kind.BOOLEAN;
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+       * optional .orc.proto.Type.Kind kind = 1;
        */
       public boolean hasKind() {
         return ((bitField0_ & 0x00000001) == 0x00000001);
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+       * optional .orc.proto.Type.Kind kind = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Kind getKind() {
         return kind_;
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+       * optional .orc.proto.Type.Kind kind = 1;
        */
       public Builder setKind(org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Kind value) {
         if (value == null) {
@@ -11350,7 +12535,7 @@
         return this;
       }
       /**
-       * required .org.apache.hadoop.hive.ql.io.orc.Type.Kind kind = 1;
+       * optional .orc.proto.Type.Kind kind = 1;
        */
       public Builder clearKind() {
         bitField0_ = (bitField0_ & ~0x00000001);
@@ -11617,7 +12802,7 @@
         return this;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.Type)
+      // @@protoc_insertion_point(builder_scope:orc.proto.Type)
     }
 
     static {
@@ -11625,7 +12810,7 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.Type)
+    // @@protoc_insertion_point(class_scope:orc.proto.Type)
   }
 
   public interface StripeInformationOrBuilder
@@ -11682,7 +12867,7 @@
     long getNumberOfRows();
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StripeInformation}
+   * Protobuf type {@code orc.proto.StripeInformation}
    */
   public static final class StripeInformation extends
       com.google.protobuf.GeneratedMessage
@@ -11771,12 +12956,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeInformation_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeInformation_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder.class);
     }
@@ -12019,19 +13204,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StripeInformation}
+     * Protobuf type {@code orc.proto.StripeInformation}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformationOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeInformation_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeInformation_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder.class);
       }
@@ -12075,7 +13260,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeInformation_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation getDefaultInstanceForType() {
@@ -12337,7 +13522,7 @@
         return this;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.StripeInformation)
+      // @@protoc_insertion_point(builder_scope:orc.proto.StripeInformation)
     }
 
     static {
@@ -12345,39 +13530,39 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.StripeInformation)
+    // @@protoc_insertion_point(class_scope:orc.proto.StripeInformation)
   }
 
   public interface UserMetadataItemOrBuilder
       extends com.google.protobuf.MessageOrBuilder {
 
-    // required string name = 1;
+    // optional string name = 1;
     /**
-     * required string name = 1;
+     * optional string name = 1;
      */
     boolean hasName();
     /**
-     * required string name = 1;
+     * optional string name = 1;
      */
     java.lang.String getName();
     /**
-     * required string name = 1;
+     * optional string name = 1;
      */
     com.google.protobuf.ByteString
         getNameBytes();
 
-    // required bytes value = 2;
+    // optional bytes value = 2;
     /**
-     * required bytes value = 2;
+     * optional bytes value = 2;
      */
     boolean hasValue();
     /**
-     * required bytes value = 2;
+     * optional bytes value = 2;
      */
     com.google.protobuf.ByteString getValue();
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.UserMetadataItem}
+   * Protobuf type {@code orc.proto.UserMetadataItem}
    */
   public static final class UserMetadataItem extends
       com.google.protobuf.GeneratedMessage
@@ -12451,12 +13636,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_UserMetadataItem_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_UserMetadataItem_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder.class);
     }
@@ -12477,17 +13662,17 @@
     }
 
     private int bitField0_;
-    // required string name = 1;
+    // optional string name = 1;
     public static final int NAME_FIELD_NUMBER = 1;
     private java.lang.Object name_;
     /**
-     * required string name = 1;
+     * optional string name = 1;
      */
     public boolean hasName() {
       return ((bitField0_ & 0x00000001) == 0x00000001);
     }
     /**
-     * required string name = 1;
+     * optional string name = 1;
      */
     public java.lang.String getName() {
       java.lang.Object ref = name_;
@@ -12504,7 +13689,7 @@
       }
     }
     /**
-     * required string name = 1;
+     * optional string name = 1;
      */
     public com.google.protobuf.ByteString
         getNameBytes() {
@@ -12520,17 +13705,17 @@
       }
     }
 
-    // required bytes value = 2;
+    // optional bytes value = 2;
     public static final int VALUE_FIELD_NUMBER = 2;
     private com.google.protobuf.ByteString value_;
     /**
-     * required bytes value = 2;
+     * optional bytes value = 2;
      */
     public boolean hasValue() {
       return ((bitField0_ & 0x00000002) == 0x00000002);
     }
     /**
-     * required bytes value = 2;
+     * optional bytes value = 2;
      */
     public com.google.protobuf.ByteString getValue() {
       return value_;
@@ -12545,14 +13730,6 @@
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized != -1) return isInitialized == 1;
 
-      if (!hasName()) {
-        memoizedIsInitialized = 0;
-        return false;
-      }
-      if (!hasValue()) {
-        memoizedIsInitialized = 0;
-        return false;
-      }
       memoizedIsInitialized = 1;
       return true;
     }
@@ -12662,19 +13839,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.UserMetadataItem}
+     * Protobuf type {@code orc.proto.UserMetadataItem}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItemOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_UserMetadataItem_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_UserMetadataItem_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder.class);
       }
@@ -12712,7 +13889,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_UserMetadataItem_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem getDefaultInstanceForType() {
@@ -12768,14 +13945,6 @@
       }
 
       public final boolean isInitialized() {
-        if (!hasName()) {
-          
-          return false;
-        }
-        if (!hasValue()) {
-          
-          return false;
-        }
         return true;
       }
 
@@ -12798,16 +13967,16 @@
       }
       private int bitField0_;
 
-      // required string name = 1;
+      // optional string name = 1;
       private java.lang.Object name_ = "";
       /**
-       * required string name = 1;
+       * optional string name = 1;
        */
       public boolean hasName() {
         return ((bitField0_ & 0x00000001) == 0x00000001);
       }
       /**
-       * required string name = 1;
+       * optional string name = 1;
        */
       public java.lang.String getName() {
         java.lang.Object ref = name_;
@@ -12821,7 +13990,7 @@
         }
       }
       /**
-       * required string name = 1;
+       * optional string name = 1;
        */
       public com.google.protobuf.ByteString
           getNameBytes() {
@@ -12837,7 +14006,7 @@
         }
       }
       /**
-       * required string name = 1;
+       * optional string name = 1;
        */
       public Builder setName(
           java.lang.String value) {
@@ -12850,7 +14019,7 @@
         return this;
       }
       /**
-       * required string name = 1;
+       * optional string name = 1;
        */
       public Builder clearName() {
         bitField0_ = (bitField0_ & ~0x00000001);
@@ -12859,7 +14028,7 @@
         return this;
       }
       /**
-       * required string name = 1;
+       * optional string name = 1;
        */
       public Builder setNameBytes(
           com.google.protobuf.ByteString value) {
@@ -12872,22 +14041,22 @@
         return this;
       }
 
-      // required bytes value = 2;
+      // optional bytes value = 2;
       private com.google.protobuf.ByteString value_ = com.google.protobuf.ByteString.EMPTY;
       /**
-       * required bytes value = 2;
+       * optional bytes value = 2;
        */
       public boolean hasValue() {
         return ((bitField0_ & 0x00000002) == 0x00000002);
       }
       /**
-       * required bytes value = 2;
+       * optional bytes value = 2;
        */
       public com.google.protobuf.ByteString getValue() {
         return value_;
       }
       /**
-       * required bytes value = 2;
+       * optional bytes value = 2;
        */
       public Builder setValue(com.google.protobuf.ByteString value) {
         if (value == null) {
@@ -12899,7 +14068,7 @@
         return this;
       }
       /**
-       * required bytes value = 2;
+       * optional bytes value = 2;
        */
       public Builder clearValue() {
         bitField0_ = (bitField0_ & ~0x00000002);
@@ -12908,7 +14077,7 @@
         return this;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.UserMetadataItem)
+      // @@protoc_insertion_point(builder_scope:orc.proto.UserMetadataItem)
     }
 
     static {
@@ -12916,39 +14085,39 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.UserMetadataItem)
+    // @@protoc_insertion_point(class_scope:orc.proto.UserMetadataItem)
   }
 
   public interface StripeStatisticsOrBuilder
       extends com.google.protobuf.MessageOrBuilder {
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+    // repeated .orc.proto.ColumnStatistics colStats = 1;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     java.util.List 
         getColStatsList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getColStats(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     int getColStatsCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     java.util.List 
         getColStatsOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getColStatsOrBuilder(
         int index);
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StripeStatistics}
+   * Protobuf type {@code orc.proto.StripeStatistics}
    */
   public static final class StripeStatistics extends
       com.google.protobuf.GeneratedMessage
@@ -13023,12 +14192,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeStatistics_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeStatistics_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder.class);
     }
@@ -13048,36 +14217,36 @@
       return PARSER;
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+    // repeated .orc.proto.ColumnStatistics colStats = 1;
     public static final int COLSTATS_FIELD_NUMBER = 1;
     private java.util.List colStats_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     public java.util.List getColStatsList() {
       return colStats_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     public java.util.List 
         getColStatsOrBuilderList() {
       return colStats_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     public int getColStatsCount() {
       return colStats_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getColStats(int index) {
       return colStats_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+     * repeated .orc.proto.ColumnStatistics colStats = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getColStatsOrBuilder(
         int index) {
@@ -13194,19 +14363,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.StripeStatistics}
+     * Protobuf type {@code orc.proto.StripeStatistics}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatisticsOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeStatistics_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeStatistics_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder.class);
       }
@@ -13247,7 +14416,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_StripeStatistics_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics getDefaultInstanceForType() {
@@ -13342,7 +14511,7 @@
       }
       private int bitField0_;
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+      // repeated .orc.proto.ColumnStatistics colStats = 1;
       private java.util.List colStats_ =
         java.util.Collections.emptyList();
       private void ensureColStatsIsMutable() {
@@ -13356,7 +14525,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder> colStatsBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public java.util.List getColStatsList() {
         if (colStatsBuilder_ == null) {
@@ -13366,7 +14535,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public int getColStatsCount() {
         if (colStatsBuilder_ == null) {
@@ -13376,7 +14545,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getColStats(int index) {
         if (colStatsBuilder_ == null) {
@@ -13386,7 +14555,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder setColStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) {
@@ -13403,7 +14572,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder setColStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder builderForValue) {
@@ -13417,7 +14586,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder addColStats(org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) {
         if (colStatsBuilder_ == null) {
@@ -13433,7 +14602,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder addColStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) {
@@ -13450,7 +14619,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder addColStats(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder builderForValue) {
@@ -13464,7 +14633,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder addColStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder builderForValue) {
@@ -13478,7 +14647,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder addAllColStats(
           java.lang.Iterable values) {
@@ -13492,7 +14661,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder clearColStats() {
         if (colStatsBuilder_ == null) {
@@ -13505,7 +14674,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public Builder removeColStats(int index) {
         if (colStatsBuilder_ == null) {
@@ -13518,14 +14687,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder getColStatsBuilder(
           int index) {
         return getColStatsFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getColStatsOrBuilder(
           int index) {
@@ -13535,7 +14704,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public java.util.List 
            getColStatsOrBuilderList() {
@@ -13546,14 +14715,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder addColStatsBuilder() {
         return getColStatsFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder addColStatsBuilder(
           int index) {
@@ -13561,7 +14730,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics colStats = 1;
+       * repeated .orc.proto.ColumnStatistics colStats = 1;
        */
       public java.util.List 
            getColStatsBuilderList() {
@@ -13582,7 +14751,7 @@
         return colStatsBuilder_;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.StripeStatistics)
+      // @@protoc_insertion_point(builder_scope:orc.proto.StripeStatistics)
     }
 
     static {
@@ -13590,39 +14759,39 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.StripeStatistics)
+    // @@protoc_insertion_point(class_scope:orc.proto.StripeStatistics)
   }
 
   public interface MetadataOrBuilder
       extends com.google.protobuf.MessageOrBuilder {
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+    // repeated .orc.proto.StripeStatistics stripeStats = 1;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     java.util.List 
         getStripeStatsList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics getStripeStats(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     int getStripeStatsCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     java.util.List 
         getStripeStatsOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatisticsOrBuilder getStripeStatsOrBuilder(
         int index);
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Metadata}
+   * Protobuf type {@code orc.proto.Metadata}
    */
   public static final class Metadata extends
       com.google.protobuf.GeneratedMessage
@@ -13697,12 +14866,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Metadata_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Metadata_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.Metadata.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Metadata.Builder.class);
     }
@@ -13722,36 +14891,36 @@
       return PARSER;
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+    // repeated .orc.proto.StripeStatistics stripeStats = 1;
     public static final int STRIPESTATS_FIELD_NUMBER = 1;
     private java.util.List stripeStats_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     public java.util.List getStripeStatsList() {
       return stripeStats_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     public java.util.List 
         getStripeStatsOrBuilderList() {
       return stripeStats_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     public int getStripeStatsCount() {
       return stripeStats_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics getStripeStats(int index) {
       return stripeStats_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+     * repeated .orc.proto.StripeStatistics stripeStats = 1;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatisticsOrBuilder getStripeStatsOrBuilder(
         int index) {
@@ -13868,19 +15037,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Metadata}
+     * Protobuf type {@code orc.proto.Metadata}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.MetadataOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Metadata_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Metadata_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.Metadata.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Metadata.Builder.class);
       }
@@ -13921,7 +15090,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Metadata_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Metadata getDefaultInstanceForType() {
@@ -14016,7 +15185,7 @@
       }
       private int bitField0_;
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+      // repeated .orc.proto.StripeStatistics stripeStats = 1;
       private java.util.List stripeStats_ =
         java.util.Collections.emptyList();
       private void ensureStripeStatsIsMutable() {
@@ -14030,7 +15199,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatisticsOrBuilder> stripeStatsBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public java.util.List getStripeStatsList() {
         if (stripeStatsBuilder_ == null) {
@@ -14040,7 +15209,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public int getStripeStatsCount() {
         if (stripeStatsBuilder_ == null) {
@@ -14050,7 +15219,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics getStripeStats(int index) {
         if (stripeStatsBuilder_ == null) {
@@ -14060,7 +15229,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder setStripeStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics value) {
@@ -14077,7 +15246,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder setStripeStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder builderForValue) {
@@ -14091,7 +15260,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder addStripeStats(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics value) {
         if (stripeStatsBuilder_ == null) {
@@ -14107,7 +15276,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder addStripeStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics value) {
@@ -14124,7 +15293,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder addStripeStats(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder builderForValue) {
@@ -14138,7 +15307,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder addStripeStats(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder builderForValue) {
@@ -14152,7 +15321,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder addAllStripeStats(
           java.lang.Iterable values) {
@@ -14166,7 +15335,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder clearStripeStats() {
         if (stripeStatsBuilder_ == null) {
@@ -14179,7 +15348,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public Builder removeStripeStats(int index) {
         if (stripeStatsBuilder_ == null) {
@@ -14192,14 +15361,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder getStripeStatsBuilder(
           int index) {
         return getStripeStatsFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatisticsOrBuilder getStripeStatsOrBuilder(
           int index) {
@@ -14209,7 +15378,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public java.util.List 
            getStripeStatsOrBuilderList() {
@@ -14220,14 +15389,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder addStripeStatsBuilder() {
         return getStripeStatsFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.Builder addStripeStatsBuilder(
           int index) {
@@ -14235,7 +15404,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeStatistics.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeStatistics stripeStats = 1;
+       * repeated .orc.proto.StripeStatistics stripeStats = 1;
        */
       public java.util.List 
            getStripeStatsBuilderList() {
@@ -14256,7 +15425,7 @@
         return stripeStatsBuilder_;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.Metadata)
+      // @@protoc_insertion_point(builder_scope:orc.proto.Metadata)
     }
 
     static {
@@ -14264,7 +15433,7 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.Metadata)
+    // @@protoc_insertion_point(class_scope:orc.proto.Metadata)
   }
 
   public interface FooterOrBuilder
@@ -14290,77 +15459,77 @@
      */
     long getContentLength();
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+    // repeated .orc.proto.StripeInformation stripes = 3;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     java.util.List 
         getStripesList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation getStripes(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     int getStripesCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     java.util.List 
         getStripesOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformationOrBuilder getStripesOrBuilder(
         int index);
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+    // repeated .orc.proto.Type types = 4;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     java.util.List 
         getTypesList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.Type getTypes(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     int getTypesCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     java.util.List 
         getTypesOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.TypeOrBuilder getTypesOrBuilder(
         int index);
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+    // repeated .orc.proto.UserMetadataItem metadata = 5;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     java.util.List 
         getMetadataList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem getMetadata(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     int getMetadataCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     java.util.List 
         getMetadataOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItemOrBuilder getMetadataOrBuilder(
         int index);
@@ -14375,27 +15544,27 @@
      */
     long getNumberOfRows();
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+    // repeated .orc.proto.ColumnStatistics statistics = 7;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     java.util.List 
         getStatisticsList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getStatistics(int index);
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     int getStatisticsCount();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     java.util.List 
         getStatisticsOrBuilderList();
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getStatisticsOrBuilder(
         int index);
@@ -14411,7 +15580,7 @@
     int getRowIndexStride();
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Footer}
+   * Protobuf type {@code orc.proto.Footer}
    */
   public static final class Footer extends
       com.google.protobuf.GeneratedMessage
@@ -14539,12 +15708,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Footer_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Footer_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.Builder.class);
     }
@@ -14597,108 +15766,108 @@
       return contentLength_;
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+    // repeated .orc.proto.StripeInformation stripes = 3;
     public static final int STRIPES_FIELD_NUMBER = 3;
     private java.util.List stripes_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     public java.util.List getStripesList() {
       return stripes_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     public java.util.List 
         getStripesOrBuilderList() {
       return stripes_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     public int getStripesCount() {
       return stripes_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation getStripes(int index) {
       return stripes_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+     * repeated .orc.proto.StripeInformation stripes = 3;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformationOrBuilder getStripesOrBuilder(
         int index) {
       return stripes_.get(index);
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+    // repeated .orc.proto.Type types = 4;
     public static final int TYPES_FIELD_NUMBER = 4;
     private java.util.List types_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     public java.util.List getTypesList() {
       return types_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     public java.util.List 
         getTypesOrBuilderList() {
       return types_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     public int getTypesCount() {
       return types_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type getTypes(int index) {
       return types_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+     * repeated .orc.proto.Type types = 4;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.TypeOrBuilder getTypesOrBuilder(
         int index) {
       return types_.get(index);
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+    // repeated .orc.proto.UserMetadataItem metadata = 5;
     public static final int METADATA_FIELD_NUMBER = 5;
     private java.util.List metadata_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     public java.util.List getMetadataList() {
       return metadata_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     public java.util.List 
         getMetadataOrBuilderList() {
       return metadata_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     public int getMetadataCount() {
       return metadata_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem getMetadata(int index) {
       return metadata_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+     * repeated .orc.proto.UserMetadataItem metadata = 5;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItemOrBuilder getMetadataOrBuilder(
         int index) {
@@ -14721,36 +15890,36 @@
       return numberOfRows_;
     }
 
-    // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+    // repeated .orc.proto.ColumnStatistics statistics = 7;
     public static final int STATISTICS_FIELD_NUMBER = 7;
     private java.util.List statistics_;
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     public java.util.List getStatisticsList() {
       return statistics_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     public java.util.List 
         getStatisticsOrBuilderList() {
       return statistics_;
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     public int getStatisticsCount() {
       return statistics_.size();
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getStatistics(int index) {
       return statistics_.get(index);
     }
     /**
-     * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+     * repeated .orc.proto.ColumnStatistics statistics = 7;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getStatisticsOrBuilder(
         int index) {
@@ -14788,18 +15957,6 @@
       byte isInitialized = memoizedIsInitialized;
       if (isInitialized != -1) return isInitialized == 1;
 
-      for (int i = 0; i < getTypesCount(); i++) {
-        if (!getTypes(i).isInitialized()) {
-          memoizedIsInitialized = 0;
-          return false;
-        }
-      }
-      for (int i = 0; i < getMetadataCount(); i++) {
-        if (!getMetadata(i).isInitialized()) {
-          memoizedIsInitialized = 0;
-          return false;
-        }
-      }
       memoizedIsInitialized = 1;
       return true;
     }
@@ -14951,19 +16108,19 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.Footer}
+     * Protobuf type {@code orc.proto.Footer}
      */
     public static final class Builder extends
         com.google.protobuf.GeneratedMessage.Builder
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.FooterOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Footer_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Footer_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer.Builder.class);
       }
@@ -15033,7 +16190,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_Footer_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer getDefaultInstanceForType() {
@@ -15241,18 +16398,6 @@
       }
 
       public final boolean isInitialized() {
-        for (int i = 0; i < getTypesCount(); i++) {
-          if (!getTypes(i).isInitialized()) {
-            
-            return false;
-          }
-        }
-        for (int i = 0; i < getMetadataCount(); i++) {
-          if (!getMetadata(i).isInitialized()) {
-            
-            return false;
-          }
-        }
         return true;
       }
 
@@ -15341,7 +16486,7 @@
         return this;
       }
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+      // repeated .orc.proto.StripeInformation stripes = 3;
       private java.util.List stripes_ =
         java.util.Collections.emptyList();
       private void ensureStripesIsMutable() {
@@ -15355,7 +16500,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformationOrBuilder> stripesBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public java.util.List getStripesList() {
         if (stripesBuilder_ == null) {
@@ -15365,7 +16510,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public int getStripesCount() {
         if (stripesBuilder_ == null) {
@@ -15375,7 +16520,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation getStripes(int index) {
         if (stripesBuilder_ == null) {
@@ -15385,7 +16530,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder setStripes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation value) {
@@ -15402,7 +16547,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder setStripes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder builderForValue) {
@@ -15416,7 +16561,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder addStripes(org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation value) {
         if (stripesBuilder_ == null) {
@@ -15432,7 +16577,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder addStripes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation value) {
@@ -15449,7 +16594,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder addStripes(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder builderForValue) {
@@ -15463,7 +16608,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder addStripes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder builderForValue) {
@@ -15477,7 +16622,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder addAllStripes(
           java.lang.Iterable values) {
@@ -15491,7 +16636,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder clearStripes() {
         if (stripesBuilder_ == null) {
@@ -15504,7 +16649,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public Builder removeStripes(int index) {
         if (stripesBuilder_ == null) {
@@ -15517,14 +16662,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder getStripesBuilder(
           int index) {
         return getStripesFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformationOrBuilder getStripesOrBuilder(
           int index) {
@@ -15534,7 +16679,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public java.util.List 
            getStripesOrBuilderList() {
@@ -15545,14 +16690,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder addStripesBuilder() {
         return getStripesFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.Builder addStripesBuilder(
           int index) {
@@ -15560,7 +16705,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeInformation.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.StripeInformation stripes = 3;
+       * repeated .orc.proto.StripeInformation stripes = 3;
        */
       public java.util.List 
            getStripesBuilderList() {
@@ -15581,7 +16726,7 @@
         return stripesBuilder_;
       }
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+      // repeated .orc.proto.Type types = 4;
       private java.util.List types_ =
         java.util.Collections.emptyList();
       private void ensureTypesIsMutable() {
@@ -15595,7 +16740,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.Type, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.TypeOrBuilder> typesBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public java.util.List getTypesList() {
         if (typesBuilder_ == null) {
@@ -15605,7 +16750,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public int getTypesCount() {
         if (typesBuilder_ == null) {
@@ -15615,7 +16760,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type getTypes(int index) {
         if (typesBuilder_ == null) {
@@ -15625,7 +16770,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder setTypes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type value) {
@@ -15642,7 +16787,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder setTypes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder builderForValue) {
@@ -15656,7 +16801,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder addTypes(org.apache.hadoop.hive.ql.io.orc.OrcProto.Type value) {
         if (typesBuilder_ == null) {
@@ -15672,7 +16817,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder addTypes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type value) {
@@ -15689,7 +16834,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder addTypes(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder builderForValue) {
@@ -15703,7 +16848,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder addTypes(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder builderForValue) {
@@ -15717,7 +16862,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder addAllTypes(
           java.lang.Iterable values) {
@@ -15731,7 +16876,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder clearTypes() {
         if (typesBuilder_ == null) {
@@ -15744,7 +16889,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public Builder removeTypes(int index) {
         if (typesBuilder_ == null) {
@@ -15757,14 +16902,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder getTypesBuilder(
           int index) {
         return getTypesFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.TypeOrBuilder getTypesOrBuilder(
           int index) {
@@ -15774,7 +16919,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public java.util.List 
            getTypesOrBuilderList() {
@@ -15785,14 +16930,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder addTypesBuilder() {
         return getTypesFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.Builder addTypesBuilder(
           int index) {
@@ -15800,7 +16945,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.Type.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.Type types = 4;
+       * repeated .orc.proto.Type types = 4;
        */
       public java.util.List 
            getTypesBuilderList() {
@@ -15821,7 +16966,7 @@
         return typesBuilder_;
       }
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+      // repeated .orc.proto.UserMetadataItem metadata = 5;
       private java.util.List metadata_ =
         java.util.Collections.emptyList();
       private void ensureMetadataIsMutable() {
@@ -15835,7 +16980,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItemOrBuilder> metadataBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public java.util.List getMetadataList() {
         if (metadataBuilder_ == null) {
@@ -15845,7 +16990,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public int getMetadataCount() {
         if (metadataBuilder_ == null) {
@@ -15855,7 +17000,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem getMetadata(int index) {
         if (metadataBuilder_ == null) {
@@ -15865,7 +17010,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder setMetadata(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem value) {
@@ -15882,7 +17027,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder setMetadata(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder builderForValue) {
@@ -15896,7 +17041,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder addMetadata(org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem value) {
         if (metadataBuilder_ == null) {
@@ -15912,7 +17057,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder addMetadata(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem value) {
@@ -15929,7 +17074,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder addMetadata(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder builderForValue) {
@@ -15943,7 +17088,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder addMetadata(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder builderForValue) {
@@ -15957,7 +17102,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder addAllMetadata(
           java.lang.Iterable values) {
@@ -15971,7 +17116,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder clearMetadata() {
         if (metadataBuilder_ == null) {
@@ -15984,7 +17129,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public Builder removeMetadata(int index) {
         if (metadataBuilder_ == null) {
@@ -15997,14 +17142,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder getMetadataBuilder(
           int index) {
         return getMetadataFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItemOrBuilder getMetadataOrBuilder(
           int index) {
@@ -16014,7 +17159,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public java.util.List 
            getMetadataOrBuilderList() {
@@ -16025,14 +17170,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder addMetadataBuilder() {
         return getMetadataFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.Builder addMetadataBuilder(
           int index) {
@@ -16040,7 +17185,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.UserMetadataItem.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.UserMetadataItem metadata = 5;
+       * repeated .orc.proto.UserMetadataItem metadata = 5;
        */
       public java.util.List 
            getMetadataBuilderList() {
@@ -16094,7 +17239,7 @@
         return this;
       }
 
-      // repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+      // repeated .orc.proto.ColumnStatistics statistics = 7;
       private java.util.List statistics_ =
         java.util.Collections.emptyList();
       private void ensureStatisticsIsMutable() {
@@ -16108,7 +17253,7 @@
           org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder> statisticsBuilder_;
 
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public java.util.List getStatisticsList() {
         if (statisticsBuilder_ == null) {
@@ -16118,7 +17263,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public int getStatisticsCount() {
         if (statisticsBuilder_ == null) {
@@ -16128,7 +17273,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics getStatistics(int index) {
         if (statisticsBuilder_ == null) {
@@ -16138,7 +17283,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder setStatistics(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) {
@@ -16155,7 +17300,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder setStatistics(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder builderForValue) {
@@ -16169,7 +17314,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder addStatistics(org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) {
         if (statisticsBuilder_ == null) {
@@ -16185,7 +17330,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder addStatistics(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics value) {
@@ -16202,7 +17347,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder addStatistics(
           org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder builderForValue) {
@@ -16216,7 +17361,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder addStatistics(
           int index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder builderForValue) {
@@ -16230,7 +17375,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder addAllStatistics(
           java.lang.Iterable values) {
@@ -16244,7 +17389,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder clearStatistics() {
         if (statisticsBuilder_ == null) {
@@ -16257,7 +17402,7 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public Builder removeStatistics(int index) {
         if (statisticsBuilder_ == null) {
@@ -16270,14 +17415,14 @@
         return this;
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder getStatisticsBuilder(
           int index) {
         return getStatisticsFieldBuilder().getBuilder(index);
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatisticsOrBuilder getStatisticsOrBuilder(
           int index) {
@@ -16287,7 +17432,7 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public java.util.List 
            getStatisticsOrBuilderList() {
@@ -16298,14 +17443,14 @@
         }
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder addStatisticsBuilder() {
         return getStatisticsFieldBuilder().addBuilder(
             org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.Builder addStatisticsBuilder(
           int index) {
@@ -16313,7 +17458,7 @@
             index, org.apache.hadoop.hive.ql.io.orc.OrcProto.ColumnStatistics.getDefaultInstance());
       }
       /**
-       * repeated .org.apache.hadoop.hive.ql.io.orc.ColumnStatistics statistics = 7;
+       * repeated .orc.proto.ColumnStatistics statistics = 7;
        */
       public java.util.List 
            getStatisticsBuilderList() {
@@ -16367,7 +17512,7 @@
         return this;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.Footer)
+      // @@protoc_insertion_point(builder_scope:orc.proto.Footer)
     }
 
     static {
@@ -16375,7 +17520,7 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.Footer)
+    // @@protoc_insertion_point(class_scope:orc.proto.Footer)
   }
 
   public interface PostScriptOrBuilder
@@ -16391,13 +17536,13 @@
      */
     long getFooterLength();
 
-    // optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+    // optional .orc.proto.CompressionKind compression = 2;
     /**
-     * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+     * optional .orc.proto.CompressionKind compression = 2;
      */
     boolean hasCompression();
     /**
-     * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+     * optional .orc.proto.CompressionKind compression = 2;
      */
     org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind getCompression();
 
@@ -16503,7 +17648,7 @@
         getMagicBytes();
   }
   /**
-   * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.PostScript}
+   * Protobuf type {@code orc.proto.PostScript}
    *
    * 
    * Serialized length must be less that 255 bytes
@@ -16631,12 +17776,12 @@
     }
     public static final com.google.protobuf.Descriptors.Descriptor
         getDescriptor() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_descriptor;
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_PostScript_descriptor;
     }
 
     protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
         internalGetFieldAccessorTable() {
-      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_fieldAccessorTable
+      return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_PostScript_fieldAccessorTable
           .ensureFieldAccessorsInitialized(
               org.apache.hadoop.hive.ql.io.orc.OrcProto.PostScript.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.PostScript.Builder.class);
     }
@@ -16673,17 +17818,17 @@
       return footerLength_;
     }
 
-    // optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+    // optional .orc.proto.CompressionKind compression = 2;
     public static final int COMPRESSION_FIELD_NUMBER = 2;
     private org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind compression_;
     /**
-     * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+     * optional .orc.proto.CompressionKind compression = 2;
      */
     public boolean hasCompression() {
       return ((bitField0_ & 0x00000002) == 0x00000002);
     }
     /**
-     * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+     * optional .orc.proto.CompressionKind compression = 2;
      */
     public org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind getCompression() {
       return compression_;
@@ -17018,7 +18163,7 @@
       return builder;
     }
     /**
-     * Protobuf type {@code org.apache.hadoop.hive.ql.io.orc.PostScript}
+     * Protobuf type {@code orc.proto.PostScript}
      *
      * 
      * Serialized length must be less that 255 bytes
@@ -17029,12 +18174,12 @@
        implements org.apache.hadoop.hive.ql.io.orc.OrcProto.PostScriptOrBuilder {
       public static final com.google.protobuf.Descriptors.Descriptor
           getDescriptor() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_PostScript_descriptor;
       }
 
       protected com.google.protobuf.GeneratedMessage.FieldAccessorTable
           internalGetFieldAccessorTable() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_fieldAccessorTable
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_PostScript_fieldAccessorTable
             .ensureFieldAccessorsInitialized(
                 org.apache.hadoop.hive.ql.io.orc.OrcProto.PostScript.class, org.apache.hadoop.hive.ql.io.orc.OrcProto.PostScript.Builder.class);
       }
@@ -17082,7 +18227,7 @@
 
       public com.google.protobuf.Descriptors.Descriptor
           getDescriptorForType() {
-        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_descriptor;
+        return org.apache.hadoop.hive.ql.io.orc.OrcProto.internal_static_orc_proto_PostScript_descriptor;
       }
 
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.PostScript getDefaultInstanceForType() {
@@ -17236,22 +18381,22 @@
         return this;
       }
 
-      // optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+      // optional .orc.proto.CompressionKind compression = 2;
       private org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind compression_ = org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind.NONE;
       /**
-       * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+       * optional .orc.proto.CompressionKind compression = 2;
        */
       public boolean hasCompression() {
         return ((bitField0_ & 0x00000002) == 0x00000002);
       }
       /**
-       * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+       * optional .orc.proto.CompressionKind compression = 2;
        */
       public org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind getCompression() {
         return compression_;
       }
       /**
-       * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+       * optional .orc.proto.CompressionKind compression = 2;
        */
       public Builder setCompression(org.apache.hadoop.hive.ql.io.orc.OrcProto.CompressionKind value) {
         if (value == null) {
@@ -17263,7 +18408,7 @@
         return this;
       }
       /**
-       * optional .org.apache.hadoop.hive.ql.io.orc.CompressionKind compression = 2;
+       * optional .orc.proto.CompressionKind compression = 2;
        */
       public Builder clearCompression() {
         bitField0_ = (bitField0_ & ~0x00000002);
@@ -17601,7 +18746,7 @@
         return this;
       }
 
-      // @@protoc_insertion_point(builder_scope:org.apache.hadoop.hive.ql.io.orc.PostScript)
+      // @@protoc_insertion_point(builder_scope:orc.proto.PostScript)
     }
 
     static {
@@ -17609,114 +18754,124 @@
       defaultInstance.initFields();
     }
 
-    // @@protoc_insertion_point(class_scope:org.apache.hadoop.hive.ql.io.orc.PostScript)
+    // @@protoc_insertion_point(class_scope:orc.proto.PostScript)
   }
 
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_descriptor;
+    internal_static_orc_proto_IntegerStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_fieldAccessorTable;
+      internal_static_orc_proto_IntegerStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_descriptor;
+    internal_static_orc_proto_DoubleStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_fieldAccessorTable;
+      internal_static_orc_proto_DoubleStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_descriptor;
+    internal_static_orc_proto_StringStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_fieldAccessorTable;
+      internal_static_orc_proto_StringStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_descriptor;
+    internal_static_orc_proto_BucketStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_fieldAccessorTable;
+      internal_static_orc_proto_BucketStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_descriptor;
+    internal_static_orc_proto_DecimalStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_fieldAccessorTable;
+      internal_static_orc_proto_DecimalStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_descriptor;
+    internal_static_orc_proto_DateStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_fieldAccessorTable;
+      internal_static_orc_proto_DateStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_descriptor;
+    internal_static_orc_proto_TimestampStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_fieldAccessorTable;
+      internal_static_orc_proto_TimestampStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_descriptor;
+    internal_static_orc_proto_BinaryStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_fieldAccessorTable;
+      internal_static_orc_proto_BinaryStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_descriptor;
+    internal_static_orc_proto_ColumnStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_fieldAccessorTable;
+      internal_static_orc_proto_ColumnStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_descriptor;
+    internal_static_orc_proto_RowIndexEntry_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_fieldAccessorTable;
+      internal_static_orc_proto_RowIndexEntry_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_descriptor;
+    internal_static_orc_proto_RowIndex_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_fieldAccessorTable;
+      internal_static_orc_proto_RowIndex_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_descriptor;
+    internal_static_orc_proto_BloomFilter_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_fieldAccessorTable;
+      internal_static_orc_proto_BloomFilter_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_descriptor;
+    internal_static_orc_proto_BloomFilterIndex_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_fieldAccessorTable;
+      internal_static_orc_proto_BloomFilterIndex_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_descriptor;
+    internal_static_orc_proto_Stream_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_fieldAccessorTable;
+      internal_static_orc_proto_Stream_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_Type_descriptor;
+    internal_static_orc_proto_ColumnEncoding_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_Type_fieldAccessorTable;
+      internal_static_orc_proto_ColumnEncoding_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_descriptor;
+    internal_static_orc_proto_StripeFooter_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_fieldAccessorTable;
+      internal_static_orc_proto_StripeFooter_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_descriptor;
+    internal_static_orc_proto_Type_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_fieldAccessorTable;
+      internal_static_orc_proto_Type_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_descriptor;
+    internal_static_orc_proto_StripeInformation_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_fieldAccessorTable;
+      internal_static_orc_proto_StripeInformation_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_descriptor;
+    internal_static_orc_proto_UserMetadataItem_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_fieldAccessorTable;
+      internal_static_orc_proto_UserMetadataItem_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_descriptor;
+    internal_static_orc_proto_StripeStatistics_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_fieldAccessorTable;
+      internal_static_orc_proto_StripeStatistics_fieldAccessorTable;
   private static com.google.protobuf.Descriptors.Descriptor
-    internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_descriptor;
+    internal_static_orc_proto_Metadata_descriptor;
   private static
     com.google.protobuf.GeneratedMessage.FieldAccessorTable
-      internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_fieldAccessorTable;
+      internal_static_orc_proto_Metadata_fieldAccessorTable;
+  private static com.google.protobuf.Descriptors.Descriptor
+    internal_static_orc_proto_Footer_descriptor;
+  private static
+    com.google.protobuf.GeneratedMessage.FieldAccessorTable
+      internal_static_orc_proto_Footer_fieldAccessorTable;
+  private static com.google.protobuf.Descriptors.Descriptor
+    internal_static_orc_proto_PostScript_descriptor;
+  private static
+    com.google.protobuf.GeneratedMessage.FieldAccessorTable
+      internal_static_orc_proto_PostScript_fieldAccessorTable;
 
   public static com.google.protobuf.Descriptors.FileDescriptor
       getDescriptor() {
@@ -17726,220 +18881,223 @@
       descriptor;
   static {
     java.lang.String[] descriptorData = {
-      "\n\017orc_proto.proto\022 org.apache.hadoop.hiv" +
-      "e.ql.io.orc\"B\n\021IntegerStatistics\022\017\n\007mini" +
-      "mum\030\001 \001(\022\022\017\n\007maximum\030\002 \001(\022\022\013\n\003sum\030\003 \001(\022\"" +
-      "A\n\020DoubleStatistics\022\017\n\007minimum\030\001 \001(\001\022\017\n\007" +
-      "maximum\030\002 \001(\001\022\013\n\003sum\030\003 \001(\001\"A\n\020StringStat" +
-      "istics\022\017\n\007minimum\030\001 \001(\t\022\017\n\007maximum\030\002 \001(\t" +
-      "\022\013\n\003sum\030\003 \001(\022\"%\n\020BucketStatistics\022\021\n\005cou" +
-      "nt\030\001 \003(\004B\002\020\001\"B\n\021DecimalStatistics\022\017\n\007min" +
-      "imum\030\001 \001(\t\022\017\n\007maximum\030\002 \001(\t\022\013\n\003sum\030\003 \001(\t" +
-      "\"2\n\016DateStatistics\022\017\n\007minimum\030\001 \001(\021\022\017\n\007m",
-      "aximum\030\002 \001(\021\"7\n\023TimestampStatistics\022\017\n\007m" +
-      "inimum\030\001 \001(\022\022\017\n\007maximum\030\002 \001(\022\"\037\n\020BinaryS" +
-      "tatistics\022\013\n\003sum\030\001 \001(\022\"\255\005\n\020ColumnStatist" +
-      "ics\022\026\n\016numberOfValues\030\001 \001(\004\022J\n\rintStatis" +
-      "tics\030\002 \001(\01323.org.apache.hadoop.hive.ql.i" +
-      "o.orc.IntegerStatistics\022L\n\020doubleStatist" +
-      "ics\030\003 \001(\01322.org.apache.hadoop.hive.ql.io" +
-      ".orc.DoubleStatistics\022L\n\020stringStatistic" +
-      "s\030\004 \001(\01322.org.apache.hadoop.hive.ql.io.o" +
-      "rc.StringStatistics\022L\n\020bucketStatistics\030",
-      "\005 \001(\01322.org.apache.hadoop.hive.ql.io.orc" +
-      ".BucketStatistics\022N\n\021decimalStatistics\030\006" +
-      " \001(\01323.org.apache.hadoop.hive.ql.io.orc." +
-      "DecimalStatistics\022H\n\016dateStatistics\030\007 \001(" +
-      "\01320.org.apache.hadoop.hive.ql.io.orc.Dat" +
-      "eStatistics\022L\n\020binaryStatistics\030\010 \001(\01322." +
-      "org.apache.hadoop.hive.ql.io.orc.BinaryS" +
-      "tatistics\022R\n\023timestampStatistics\030\t \001(\01325" +
-      ".org.apache.hadoop.hive.ql.io.orc.Timest" +
-      "ampStatistics\022\017\n\007hasNull\030\n \001(\010\"n\n\rRowInd",
-      "exEntry\022\025\n\tpositions\030\001 \003(\004B\002\020\001\022F\n\nstatis" +
-      "tics\030\002 \001(\01322.org.apache.hadoop.hive.ql.i" +
-      "o.orc.ColumnStatistics\"J\n\010RowIndex\022>\n\005en" +
-      "try\030\001 \003(\0132/.org.apache.hadoop.hive.ql.io" +
-      ".orc.RowIndexEntry\"\331\001\n\006Stream\022;\n\004kind\030\001 " +
-      "\002(\0162-.org.apache.hadoop.hive.ql.io.orc.S" +
-      "tream.Kind\022\016\n\006column\030\002 \001(\r\022\016\n\006length\030\003 \001" +
-      "(\004\"r\n\004Kind\022\013\n\007PRESENT\020\000\022\010\n\004DATA\020\001\022\n\n\006LEN" +
-      "GTH\020\002\022\023\n\017DICTIONARY_DATA\020\003\022\024\n\020DICTIONARY" +
-      "_COUNT\020\004\022\r\n\tSECONDARY\020\005\022\r\n\tROW_INDEX\020\006\"\263",
-      "\001\n\016ColumnEncoding\022C\n\004kind\030\001 \002(\01625.org.ap" +
-      "ache.hadoop.hive.ql.io.orc.ColumnEncodin" +
-      "g.Kind\022\026\n\016dictionarySize\030\002 \001(\r\"D\n\004Kind\022\n" +
-      "\n\006DIRECT\020\000\022\016\n\nDICTIONARY\020\001\022\r\n\tDIRECT_V2\020" +
-      "\002\022\021\n\rDICTIONARY_V2\020\003\"\214\001\n\014StripeFooter\0229\n" +
-      "\007streams\030\001 \003(\0132(.org.apache.hadoop.hive." +
-      "ql.io.orc.Stream\022A\n\007columns\030\002 \003(\01320.org." +
-      "apache.hadoop.hive.ql.io.orc.ColumnEncod" +
-      "ing\"\370\002\n\004Type\0229\n\004kind\030\001 \002(\0162+.org.apache." +
-      "hadoop.hive.ql.io.orc.Type.Kind\022\024\n\010subty",
-      "pes\030\002 \003(\rB\002\020\001\022\022\n\nfieldNames\030\003 \003(\t\022\025\n\rmax" +
-      "imumLength\030\004 \001(\r\022\021\n\tprecision\030\005 \001(\r\022\r\n\005s" +
-      "cale\030\006 \001(\r\"\321\001\n\004Kind\022\013\n\007BOOLEAN\020\000\022\010\n\004BYTE" +
-      "\020\001\022\t\n\005SHORT\020\002\022\007\n\003INT\020\003\022\010\n\004LONG\020\004\022\t\n\005FLOA" +
-      "T\020\005\022\n\n\006DOUBLE\020\006\022\n\n\006STRING\020\007\022\n\n\006BINARY\020\010\022" +
-      "\r\n\tTIMESTAMP\020\t\022\010\n\004LIST\020\n\022\007\n\003MAP\020\013\022\n\n\006STR" +
-      "UCT\020\014\022\t\n\005UNION\020\r\022\013\n\007DECIMAL\020\016\022\010\n\004DATE\020\017\022" +
-      "\013\n\007VARCHAR\020\020\022\010\n\004CHAR\020\021\"x\n\021StripeInformat" +
-      "ion\022\016\n\006offset\030\001 \001(\004\022\023\n\013indexLength\030\002 \001(\004" +
-      "\022\022\n\ndataLength\030\003 \001(\004\022\024\n\014footerLength\030\004 \001",
-      "(\004\022\024\n\014numberOfRows\030\005 \001(\004\"/\n\020UserMetadata" +
-      "Item\022\014\n\004name\030\001 \002(\t\022\r\n\005value\030\002 \002(\014\"X\n\020Str" +
-      "ipeStatistics\022D\n\010colStats\030\001 \003(\01322.org.ap" +
-      "ache.hadoop.hive.ql.io.orc.ColumnStatist" +
-      "ics\"S\n\010Metadata\022G\n\013stripeStats\030\001 \003(\01322.o" +
-      "rg.apache.hadoop.hive.ql.io.orc.StripeSt" +
-      "atistics\"\356\002\n\006Footer\022\024\n\014headerLength\030\001 \001(" +
-      "\004\022\025\n\rcontentLength\030\002 \001(\004\022D\n\007stripes\030\003 \003(" +
-      "\01323.org.apache.hadoop.hive.ql.io.orc.Str" +
-      "ipeInformation\0225\n\005types\030\004 \003(\0132&.org.apac",
-      "he.hadoop.hive.ql.io.orc.Type\022D\n\010metadat" +
-      "a\030\005 \003(\01322.org.apache.hadoop.hive.ql.io.o" +
-      "rc.UserMetadataItem\022\024\n\014numberOfRows\030\006 \001(" +
-      "\004\022F\n\nstatistics\030\007 \003(\01322.org.apache.hadoo" +
-      "p.hive.ql.io.orc.ColumnStatistics\022\026\n\016row" +
-      "IndexStride\030\010 \001(\r\"\334\001\n\nPostScript\022\024\n\014foot" +
-      "erLength\030\001 \001(\004\022F\n\013compression\030\002 \001(\01621.or" +
-      "g.apache.hadoop.hive.ql.io.orc.Compressi" +
-      "onKind\022\034\n\024compressionBlockSize\030\003 \001(\004\022\023\n\007" +
-      "version\030\004 \003(\rB\002\020\001\022\026\n\016metadataLength\030\005 \001(",
-      "\004\022\025\n\rwriterVersion\030\006 \001(\r\022\016\n\005magic\030\300> \001(\t" +
-      "*:\n\017CompressionKind\022\010\n\004NONE\020\000\022\010\n\004ZLIB\020\001\022" +
-      "\n\n\006SNAPPY\020\002\022\007\n\003LZO\020\003"
+      "\n\017orc_proto.proto\022\torc.proto\"B\n\021IntegerS" +
+      "tatistics\022\017\n\007minimum\030\001 \001(\022\022\017\n\007maximum\030\002 " +
+      "\001(\022\022\013\n\003sum\030\003 \001(\022\"A\n\020DoubleStatistics\022\017\n\007" +
+      "minimum\030\001 \001(\001\022\017\n\007maximum\030\002 \001(\001\022\013\n\003sum\030\003 " +
+      "\001(\001\"A\n\020StringStatistics\022\017\n\007minimum\030\001 \001(\t" +
+      "\022\017\n\007maximum\030\002 \001(\t\022\013\n\003sum\030\003 \001(\022\"%\n\020Bucket" +
+      "Statistics\022\021\n\005count\030\001 \003(\004B\002\020\001\"B\n\021Decimal" +
+      "Statistics\022\017\n\007minimum\030\001 \001(\t\022\017\n\007maximum\030\002" +
+      " \001(\t\022\013\n\003sum\030\003 \001(\t\"2\n\016DateStatistics\022\017\n\007m" +
+      "inimum\030\001 \001(\021\022\017\n\007maximum\030\002 \001(\021\"7\n\023Timesta",
+      "mpStatistics\022\017\n\007minimum\030\001 \001(\022\022\017\n\007maximum" +
+      "\030\002 \001(\022\"\037\n\020BinaryStatistics\022\013\n\003sum\030\001 \001(\022\"" +
+      "\365\003\n\020ColumnStatistics\022\026\n\016numberOfValues\030\001" +
+      " \001(\004\0223\n\rintStatistics\030\002 \001(\0132\034.orc.proto." +
+      "IntegerStatistics\0225\n\020doubleStatistics\030\003 " +
+      "\001(\0132\033.orc.proto.DoubleStatistics\0225\n\020stri" +
+      "ngStatistics\030\004 \001(\0132\033.orc.proto.StringSta" +
+      "tistics\0225\n\020bucketStatistics\030\005 \001(\0132\033.orc." +
+      "proto.BucketStatistics\0227\n\021decimalStatist" +
+      "ics\030\006 \001(\0132\034.orc.proto.DecimalStatistics\022",
+      "1\n\016dateStatistics\030\007 \001(\0132\031.orc.proto.Date" +
+      "Statistics\0225\n\020binaryStatistics\030\010 \001(\0132\033.o" +
+      "rc.proto.BinaryStatistics\022;\n\023timestampSt" +
+      "atistics\030\t \001(\0132\036.orc.proto.TimestampStat" +
+      "istics\022\017\n\007hasNull\030\n \001(\010\"W\n\rRowIndexEntry" +
+      "\022\025\n\tpositions\030\001 \003(\004B\002\020\001\022/\n\nstatistics\030\002 " +
+      "\001(\0132\033.orc.proto.ColumnStatistics\"3\n\010RowI" +
+      "ndex\022\'\n\005entry\030\001 \003(\0132\030.orc.proto.RowIndex" +
+      "Entry\"7\n\013BloomFilter\022\030\n\020numHashFunctions" +
+      "\030\001 \001(\r\022\016\n\006bitset\030\002 \003(\006\"?\n\020BloomFilterInd",
+      "ex\022+\n\013bloomFilter\030\001 \003(\0132\026.orc.proto.Bloo" +
+      "mFilter\"\325\001\n\006Stream\022$\n\004kind\030\001 \001(\0162\026.orc.p" +
+      "roto.Stream.Kind\022\016\n\006column\030\002 \001(\r\022\016\n\006leng" +
+      "th\030\003 \001(\004\"\204\001\n\004Kind\022\013\n\007PRESENT\020\000\022\010\n\004DATA\020\001" +
+      "\022\n\n\006LENGTH\020\002\022\023\n\017DICTIONARY_DATA\020\003\022\024\n\020DIC" +
+      "TIONARY_COUNT\020\004\022\r\n\tSECONDARY\020\005\022\r\n\tROW_IN" +
+      "DEX\020\006\022\020\n\014BLOOM_FILTER\020\007\"\234\001\n\016ColumnEncodi" +
+      "ng\022,\n\004kind\030\001 \001(\0162\036.orc.proto.ColumnEncod" +
+      "ing.Kind\022\026\n\016dictionarySize\030\002 \001(\r\"D\n\004Kind" +
+      "\022\n\n\006DIRECT\020\000\022\016\n\nDICTIONARY\020\001\022\r\n\tDIRECT_V",
+      "2\020\002\022\021\n\rDICTIONARY_V2\020\003\"^\n\014StripeFooter\022\"" +
+      "\n\007streams\030\001 \003(\0132\021.orc.proto.Stream\022*\n\007co" +
+      "lumns\030\002 \003(\0132\031.orc.proto.ColumnEncoding\"\341" +
+      "\002\n\004Type\022\"\n\004kind\030\001 \001(\0162\024.orc.proto.Type.K" +
+      "ind\022\024\n\010subtypes\030\002 \003(\rB\002\020\001\022\022\n\nfieldNames\030" +
+      "\003 \003(\t\022\025\n\rmaximumLength\030\004 \001(\r\022\021\n\tprecisio" +
+      "n\030\005 \001(\r\022\r\n\005scale\030\006 \001(\r\"\321\001\n\004Kind\022\013\n\007BOOLE" +
+      "AN\020\000\022\010\n\004BYTE\020\001\022\t\n\005SHORT\020\002\022\007\n\003INT\020\003\022\010\n\004LO" +
+      "NG\020\004\022\t\n\005FLOAT\020\005\022\n\n\006DOUBLE\020\006\022\n\n\006STRING\020\007\022" +
+      "\n\n\006BINARY\020\010\022\r\n\tTIMESTAMP\020\t\022\010\n\004LIST\020\n\022\007\n\003",
+      "MAP\020\013\022\n\n\006STRUCT\020\014\022\t\n\005UNION\020\r\022\013\n\007DECIMAL\020" +
+      "\016\022\010\n\004DATE\020\017\022\013\n\007VARCHAR\020\020\022\010\n\004CHAR\020\021\"x\n\021St" +
+      "ripeInformation\022\016\n\006offset\030\001 \001(\004\022\023\n\013index" +
+      "Length\030\002 \001(\004\022\022\n\ndataLength\030\003 \001(\004\022\024\n\014foot" +
+      "erLength\030\004 \001(\004\022\024\n\014numberOfRows\030\005 \001(\004\"/\n\020" +
+      "UserMetadataItem\022\014\n\004name\030\001 \001(\t\022\r\n\005value\030" +
+      "\002 \001(\014\"A\n\020StripeStatistics\022-\n\010colStats\030\001 " +
+      "\003(\0132\033.orc.proto.ColumnStatistics\"<\n\010Meta" +
+      "data\0220\n\013stripeStats\030\001 \003(\0132\033.orc.proto.St" +
+      "ripeStatistics\"\222\002\n\006Footer\022\024\n\014headerLengt",
+      "h\030\001 \001(\004\022\025\n\rcontentLength\030\002 \001(\004\022-\n\007stripe" +
+      "s\030\003 \003(\0132\034.orc.proto.StripeInformation\022\036\n" +
+      "\005types\030\004 \003(\0132\017.orc.proto.Type\022-\n\010metadat" +
+      "a\030\005 \003(\0132\033.orc.proto.UserMetadataItem\022\024\n\014" +
+      "numberOfRows\030\006 \001(\004\022/\n\nstatistics\030\007 \003(\0132\033" +
+      ".orc.proto.ColumnStatistics\022\026\n\016rowIndexS" +
+      "tride\030\010 \001(\r\"\305\001\n\nPostScript\022\024\n\014footerLeng" +
+      "th\030\001 \001(\004\022/\n\013compression\030\002 \001(\0162\032.orc.prot" +
+      "o.CompressionKind\022\034\n\024compressionBlockSiz" +
+      "e\030\003 \001(\004\022\023\n\007version\030\004 \003(\rB\002\020\001\022\026\n\016metadata",
+      "Length\030\005 \001(\004\022\025\n\rwriterVersion\030\006 \001(\r\022\016\n\005m" +
+      "agic\030\300> \001(\t*:\n\017CompressionKind\022\010\n\004NONE\020\000" +
+      "\022\010\n\004ZLIB\020\001\022\n\n\006SNAPPY\020\002\022\007\n\003LZO\020\003B\"\n org.a" +
+      "pache.hadoop.hive.ql.io.orc"
     };
     com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner assigner =
       new com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner() {
         public com.google.protobuf.ExtensionRegistry assignDescriptors(
             com.google.protobuf.Descriptors.FileDescriptor root) {
           descriptor = root;
-          internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_descriptor =
+          internal_static_orc_proto_IntegerStatistics_descriptor =
             getDescriptor().getMessageTypes().get(0);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_IntegerStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_IntegerStatistics_descriptor,
+              internal_static_orc_proto_IntegerStatistics_descriptor,
               new java.lang.String[] { "Minimum", "Maximum", "Sum", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_descriptor =
+          internal_static_orc_proto_DoubleStatistics_descriptor =
             getDescriptor().getMessageTypes().get(1);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_DoubleStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_DoubleStatistics_descriptor,
+              internal_static_orc_proto_DoubleStatistics_descriptor,
               new java.lang.String[] { "Minimum", "Maximum", "Sum", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_descriptor =
+          internal_static_orc_proto_StringStatistics_descriptor =
             getDescriptor().getMessageTypes().get(2);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_StringStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_StringStatistics_descriptor,
+              internal_static_orc_proto_StringStatistics_descriptor,
               new java.lang.String[] { "Minimum", "Maximum", "Sum", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_descriptor =
+          internal_static_orc_proto_BucketStatistics_descriptor =
             getDescriptor().getMessageTypes().get(3);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_BucketStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_BucketStatistics_descriptor,
+              internal_static_orc_proto_BucketStatistics_descriptor,
               new java.lang.String[] { "Count", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_descriptor =
+          internal_static_orc_proto_DecimalStatistics_descriptor =
             getDescriptor().getMessageTypes().get(4);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_DecimalStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_DecimalStatistics_descriptor,
+              internal_static_orc_proto_DecimalStatistics_descriptor,
               new java.lang.String[] { "Minimum", "Maximum", "Sum", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_descriptor =
+          internal_static_orc_proto_DateStatistics_descriptor =
             getDescriptor().getMessageTypes().get(5);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_DateStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_DateStatistics_descriptor,
+              internal_static_orc_proto_DateStatistics_descriptor,
               new java.lang.String[] { "Minimum", "Maximum", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_descriptor =
+          internal_static_orc_proto_TimestampStatistics_descriptor =
             getDescriptor().getMessageTypes().get(6);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_TimestampStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_TimestampStatistics_descriptor,
+              internal_static_orc_proto_TimestampStatistics_descriptor,
               new java.lang.String[] { "Minimum", "Maximum", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_descriptor =
+          internal_static_orc_proto_BinaryStatistics_descriptor =
             getDescriptor().getMessageTypes().get(7);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_BinaryStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_BinaryStatistics_descriptor,
+              internal_static_orc_proto_BinaryStatistics_descriptor,
               new java.lang.String[] { "Sum", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_descriptor =
+          internal_static_orc_proto_ColumnStatistics_descriptor =
             getDescriptor().getMessageTypes().get(8);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_ColumnStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnStatistics_descriptor,
+              internal_static_orc_proto_ColumnStatistics_descriptor,
               new java.lang.String[] { "NumberOfValues", "IntStatistics", "DoubleStatistics", "StringStatistics", "BucketStatistics", "DecimalStatistics", "DateStatistics", "BinaryStatistics", "TimestampStatistics", "HasNull", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_descriptor =
+          internal_static_orc_proto_RowIndexEntry_descriptor =
             getDescriptor().getMessageTypes().get(9);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_fieldAccessorTable = new
+          internal_static_orc_proto_RowIndexEntry_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndexEntry_descriptor,
+              internal_static_orc_proto_RowIndexEntry_descriptor,
               new java.lang.String[] { "Positions", "Statistics", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_descriptor =
+          internal_static_orc_proto_RowIndex_descriptor =
             getDescriptor().getMessageTypes().get(10);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_fieldAccessorTable = new
+          internal_static_orc_proto_RowIndex_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_RowIndex_descriptor,
+              internal_static_orc_proto_RowIndex_descriptor,
               new java.lang.String[] { "Entry", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_descriptor =
+          internal_static_orc_proto_BloomFilter_descriptor =
             getDescriptor().getMessageTypes().get(11);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_fieldAccessorTable = new
+          internal_static_orc_proto_BloomFilter_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_Stream_descriptor,
-              new java.lang.String[] { "Kind", "Column", "Length", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_descriptor =
+              internal_static_orc_proto_BloomFilter_descriptor,
+              new java.lang.String[] { "NumHashFunctions", "Bitset", });
+          internal_static_orc_proto_BloomFilterIndex_descriptor =
             getDescriptor().getMessageTypes().get(12);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_fieldAccessorTable = new
+          internal_static_orc_proto_BloomFilterIndex_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_ColumnEncoding_descriptor,
-              new java.lang.String[] { "Kind", "DictionarySize", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_descriptor =
+              internal_static_orc_proto_BloomFilterIndex_descriptor,
+              new java.lang.String[] { "BloomFilter", });
+          internal_static_orc_proto_Stream_descriptor =
             getDescriptor().getMessageTypes().get(13);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_fieldAccessorTable = new
+          internal_static_orc_proto_Stream_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_StripeFooter_descriptor,
-              new java.lang.String[] { "Streams", "Columns", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Type_descriptor =
+              internal_static_orc_proto_Stream_descriptor,
+              new java.lang.String[] { "Kind", "Column", "Length", });
+          internal_static_orc_proto_ColumnEncoding_descriptor =
             getDescriptor().getMessageTypes().get(14);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Type_fieldAccessorTable = new
+          internal_static_orc_proto_ColumnEncoding_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_Type_descriptor,
-              new java.lang.String[] { "Kind", "Subtypes", "FieldNames", "MaximumLength", "Precision", "Scale", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_descriptor =
+              internal_static_orc_proto_ColumnEncoding_descriptor,
+              new java.lang.String[] { "Kind", "DictionarySize", });
+          internal_static_orc_proto_StripeFooter_descriptor =
             getDescriptor().getMessageTypes().get(15);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_fieldAccessorTable = new
+          internal_static_orc_proto_StripeFooter_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_StripeInformation_descriptor,
-              new java.lang.String[] { "Offset", "IndexLength", "DataLength", "FooterLength", "NumberOfRows", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_descriptor =
+              internal_static_orc_proto_StripeFooter_descriptor,
+              new java.lang.String[] { "Streams", "Columns", });
+          internal_static_orc_proto_Type_descriptor =
             getDescriptor().getMessageTypes().get(16);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_fieldAccessorTable = new
+          internal_static_orc_proto_Type_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_UserMetadataItem_descriptor,
-              new java.lang.String[] { "Name", "Value", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_descriptor =
+              internal_static_orc_proto_Type_descriptor,
+              new java.lang.String[] { "Kind", "Subtypes", "FieldNames", "MaximumLength", "Precision", "Scale", });
+          internal_static_orc_proto_StripeInformation_descriptor =
             getDescriptor().getMessageTypes().get(17);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_fieldAccessorTable = new
+          internal_static_orc_proto_StripeInformation_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_StripeStatistics_descriptor,
-              new java.lang.String[] { "ColStats", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_descriptor =
+              internal_static_orc_proto_StripeInformation_descriptor,
+              new java.lang.String[] { "Offset", "IndexLength", "DataLength", "FooterLength", "NumberOfRows", });
+          internal_static_orc_proto_UserMetadataItem_descriptor =
             getDescriptor().getMessageTypes().get(18);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_fieldAccessorTable = new
+          internal_static_orc_proto_UserMetadataItem_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_Metadata_descriptor,
-              new java.lang.String[] { "StripeStats", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_descriptor =
+              internal_static_orc_proto_UserMetadataItem_descriptor,
+              new java.lang.String[] { "Name", "Value", });
+          internal_static_orc_proto_StripeStatistics_descriptor =
             getDescriptor().getMessageTypes().get(19);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_fieldAccessorTable = new
+          internal_static_orc_proto_StripeStatistics_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_Footer_descriptor,
-              new java.lang.String[] { "HeaderLength", "ContentLength", "Stripes", "Types", "Metadata", "NumberOfRows", "Statistics", "RowIndexStride", });
-          internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_descriptor =
+              internal_static_orc_proto_StripeStatistics_descriptor,
+              new java.lang.String[] { "ColStats", });
+          internal_static_orc_proto_Metadata_descriptor =
             getDescriptor().getMessageTypes().get(20);
-          internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_fieldAccessorTable = new
+          internal_static_orc_proto_Metadata_fieldAccessorTable = new
             com.google.protobuf.GeneratedMessage.FieldAccessorTable(
-              internal_static_org_apache_hadoop_hive_ql_io_orc_PostScript_descriptor,
+              internal_static_orc_proto_Metadata_descriptor,
+              new java.lang.String[] { "StripeStats", });
+          internal_static_orc_proto_Footer_descriptor =
+            getDescriptor().getMessageTypes().get(21);
+          internal_static_orc_proto_Footer_fieldAccessorTable = new
+            com.google.protobuf.GeneratedMessage.FieldAccessorTable(
+              internal_static_orc_proto_Footer_descriptor,
+              new java.lang.String[] { "HeaderLength", "ContentLength", "Stripes", "Types", "Metadata", "NumberOfRows", "Statistics", "RowIndexStride", });
+          internal_static_orc_proto_PostScript_descriptor =
+            getDescriptor().getMessageTypes().get(22);
+          internal_static_orc_proto_PostScript_fieldAccessorTable = new
+            com.google.protobuf.GeneratedMessage.FieldAccessorTable(
+              internal_static_orc_proto_PostScript_descriptor,
               new java.lang.String[] { "FooterLength", "Compression", "CompressionBlockSize", "Version", "MetadataLength", "WriterVersion", "Magic", });
           return null;
         }
Index: ql/src/java/org/apache/hadoop/hive/ql/Driver.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/Driver.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/Driver.java	(working copy)
@@ -47,7 +47,6 @@
 import org.apache.hadoop.hive.ql.exec.ConditionalTask;
 import org.apache.hadoop.hive.ql.exec.ExplainTask;
 import org.apache.hadoop.hive.ql.exec.FetchTask;
-import org.apache.hadoop.hive.ql.exec.MoveTask;
 import org.apache.hadoop.hive.ql.exec.Operator;
 import org.apache.hadoop.hive.ql.exec.TableScanOperator;
 import org.apache.hadoop.hive.ql.exec.Task;
@@ -64,8 +63,8 @@
 import org.apache.hadoop.hive.ql.hooks.PostExecute;
 import org.apache.hadoop.hive.ql.hooks.PreExecute;
 import org.apache.hadoop.hive.ql.hooks.ReadEntity;
+import org.apache.hadoop.hive.ql.hooks.Redactor;
 import org.apache.hadoop.hive.ql.hooks.WriteEntity;
-import org.apache.hadoop.hive.ql.hooks.Redactor;
 import org.apache.hadoop.hive.ql.lockmgr.HiveLock;
 import org.apache.hadoop.hive.ql.lockmgr.HiveLockMode;
 import org.apache.hadoop.hive.ql.lockmgr.HiveLockObj;
@@ -153,6 +152,9 @@
 
   private String userName;
 
+  // HS2 operation handle guid string
+  private String operationId;
+
   private boolean checkConcurrency() {
     boolean supportConcurrency = conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
     if (!supportConcurrency) {
@@ -1328,7 +1330,8 @@
       resStream = null;
 
       SessionState ss = SessionState.get();
-      HookContext hookContext = new HookContext(plan, conf, ctx.getPathToCS(), ss.getUserName(), ss.getUserIpAddress());
+      HookContext hookContext = new HookContext(plan, conf, ctx.getPathToCS(), ss.getUserName(),
+          ss.getUserIpAddress(), operationId);
       hookContext.setHookType(HookContext.HookType.PRE_EXEC_HOOK);
 
       for (Hook peh : getHooks(HiveConf.ConfVars.PREEXECHOOKS)) {
@@ -1625,8 +1628,7 @@
 
     cxt.launching(tskRun);
     // Launch Task
-    if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.EXECPARALLEL)
-        && (tsk.isMapRedTask() || (tsk instanceof MoveTask))) {
+    if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.EXECPARALLEL) && tsk.isMapRedTask()) {
       // Launch it in the parallel mode, as a separate thread only for MR tasks
       if (LOG.isInfoEnabled()){
         LOG.info("Starting task [" + tsk + "] in parallel");
@@ -1784,4 +1786,12 @@
     return errorMessage;
   }
 
+  /**
+   * Set the HS2 operation handle's guid string
+   * @param opId base64 encoded guid string
+   */
+  public void setOperationId(String opId) {
+    this.operationId = opId;
+  }
+
 }
Index: ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java	(working copy)
@@ -428,6 +428,7 @@
   ALTER_TABLE_PARTITION_CASCADE_NOT_SUPPORTED(10300,
       "Alter table partition type {0} does not support cascade", true),
 
+  DROP_NATIVE_FUNCTION(10301, "Cannot drop native function"),
 
   //========================== 20000 range starts here ========================//
   SCRIPT_INIT_ERROR(20000, "Unable to initialize custom script."),
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/CommonFunctionInfo.java	(working copy)
@@ -1,25 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.hadoop.hive.ql.exec;
-
-/**
- * Interface for common functionality between FunctionInfo/WindowFunctionInfo
- */
-public interface CommonFunctionInfo {
-  Class getFunctionClass();
-}
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java	(working copy)
@@ -33,7 +33,6 @@
 import java.util.HashMap;
 import java.util.HashSet;
 import java.util.LinkedHashMap;
-import java.util.LinkedList;
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
@@ -478,11 +477,11 @@
           json.put(ent.getKey().toString(), ent.getValue().toString());
         }
       }
-      else if (ent.getValue() instanceof Serializable) {
+      else if (ent.getValue() != null) {
         if (out != null) {
           out.println();
         }
-        JSONObject jsonOut = outputPlan((Serializable) ent.getValue(), out,
+        JSONObject jsonOut = outputPlan(ent.getValue(), out,
             extended, jsonOutput, jsonOutput ? 0 : indent + 2);
         if (jsonOutput) {
           json.put(ent.getKey().toString(), jsonOut);
@@ -518,11 +517,11 @@
         }
         nl = true;
       }
-      else if (o instanceof Serializable) {
+      else {
         if (first_el && (out != null) && hasHeader) {
           out.println();
         }
-        JSONObject jsonOut = outputPlan((Serializable) o, out, extended,
+        JSONObject jsonOut = outputPlan(o, out, extended,
             jsonOutput, jsonOutput ? 0 : (hasHeader ? indent + 2 : indent));
         if (jsonOutput) {
           outputArray.put(jsonOut);
@@ -553,12 +552,12 @@
     return false;
   }
 
-  private JSONObject outputPlan(Serializable work,
+  private JSONObject outputPlan(Object work,
       PrintStream out, boolean extended, boolean jsonOutput, int indent) throws Exception {
     return outputPlan(work, out, extended, jsonOutput, indent, "");
   }
 
-  private JSONObject outputPlan(Serializable work, PrintStream out,
+  private JSONObject outputPlan(Object work, PrintStream out,
       boolean extended, boolean jsonOutput, int indent, String appendToHeader) throws Exception {
     // Check if work has an explain annotation
     Annotation note = AnnotationUtils.getAnnotation(work.getClass(), Explain.class);
@@ -678,7 +677,7 @@
           }
 
           // Try this as a map
-          try {
+          if (val instanceof Map) {
             // Go through the map and print out the stuff
             Map mp = (Map) val;
 
@@ -692,22 +691,10 @@
             }
             continue;
           }
-          catch (ClassCastException ce) {
-            // Ignore - all this means is that this is not a map
-          }
 
           // Try this as a list
-          try {
-            List l;
-
-            try {
-              l = (List) val;
-            } catch (ClassCastException e) {
-              Set s = (Set) val;
-              l = new LinkedList();
-              l.addAll(s);
-            }
-
+          if (val instanceof List || val instanceof Set) {
+            List l = val instanceof List ? (List)val : new ArrayList((Set)val);
             if (out != null && !skipHeader && l != null && !l.isEmpty()) {
               out.print(header);
             }
@@ -720,18 +707,13 @@
 
             continue;
           }
-          catch (ClassCastException ce) {
-            // Ignore
-          }
 
           // Finally check if it is serializable
           try {
-            Serializable s = (Serializable) val;
-
             if (!skipHeader && out != null) {
               out.println(header);
             }
-            JSONObject jsonOut = outputPlan(s, out, extended, jsonOutput, ind);
+            JSONObject jsonOut = outputPlan(val, out, extended, jsonOutput, ind);
             if (jsonOutput) {
               if (!skipHeader) {
                 json.put(header, jsonOut);
@@ -779,7 +761,7 @@
     return true;
   }
 
-  private JSONObject outputPlan(Task task,
+  private JSONObject outputPlan(Task task,
       PrintStream out, JSONObject parentJSON, boolean extended,
       boolean jsonOutput, int indent) throws Exception {
 
@@ -805,7 +787,7 @@
     return null;
   }
 
-  private JSONObject outputDependencies(Task task,
+  private JSONObject outputDependencies(Task task,
       PrintStream out, JSONObject parentJson, boolean jsonOutput, boolean taskType, int indent)
       throws Exception {
 
@@ -830,7 +812,7 @@
     else {
       StringBuffer s = new StringBuffer();
       first = true;
-      for (Task parent : task.getParentTasks()) {
+      for (Task parent : task.getParentTasks()) {
         if (!first) {
           s.append(", ");
         }
@@ -847,7 +829,7 @@
       }
     }
 
-    Task currBackupTask = task.getBackupTask();
+    Task currBackupTask = task.getBackupTask();
     if (currBackupTask != null) {
       if (out != null) {
         out.print(" has a backup stage: ");
@@ -862,7 +844,7 @@
         && ((ConditionalTask) task).getListTasks() != null) {
       StringBuffer s = new StringBuffer();
       first = true;
-      for (Task con : ((ConditionalTask) task).getListTasks()) {
+      for (Task con : ((ConditionalTask) task).getListTasks()) {
         if (!first) {
           s.append(", ");
         }
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java	(working copy)
@@ -341,7 +341,7 @@
       statsCollectRawDataSize = conf.isStatsCollectRawDataSize();
       statsFromRecordWriter = new boolean[numFiles];
       serializer = (Serializer) conf.getTableInfo().getDeserializerClass().newInstance();
-      serializer.initialize(null, conf.getTableInfo().getProperties());
+      serializer.initialize(hconf, conf.getTableInfo().getProperties());
       outputClass = serializer.getSerializedClass();
 
       if (isLogInfoEnabled) {
@@ -418,12 +418,14 @@
 
       numRows = 0;
 
-      String context = jc.get(Operator.CONTEXT_NAME_KEY, "");
-      if (context != null && !context.isEmpty()) {
-        context = "_" + context.replace(" ","_");
+      String suffix = Integer.toString(conf.getDestTableId());
+      String fullName = conf.getTableInfo().getTableName();
+      if (fullName != null) {
+        suffix = suffix + "_" + fullName.toLowerCase();
       }
-      statsMap.put(Counter.RECORDS_OUT + context, row_count);
 
+      statsMap.put(Counter.RECORDS_OUT + "_" + suffix, row_count);
+
       initializeChildren(hconf);
     } catch (HiveException e) {
       throw e;
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionInfo.java	(working copy)
@@ -18,6 +18,7 @@
 
 package org.apache.hadoop.hive.ql.exec;
 
+import org.apache.hadoop.hive.ql.session.SessionState;
 import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge;
 import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFResolver;
 import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
@@ -27,18 +28,28 @@
 import org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction;
 import org.apache.hive.common.util.AnnotationUtils;
 
+import java.util.concurrent.atomic.AtomicBoolean;
+
 /**
  * FunctionInfo.
  *
  */
-public class FunctionInfo implements CommonFunctionInfo {
+public class FunctionInfo {
 
-  private final boolean isNative;
+  public static enum FunctionType {
+    BUILTIN, PERSISTENT, TEMPORARY;
+  }
 
+  private final FunctionType functionType;
+
   private final boolean isInternalTableFunction;
 
   private final String displayName;
 
+  private final FunctionResource[] resources;
+
+  private String className;
+
   private GenericUDF genericUDF;
 
   private GenericUDTF genericUDTF;
@@ -49,38 +60,55 @@
 
   private boolean blockedFunction;
 
+  // for persistent function
+  // if the function is dropped, all functions registered to sessions are needed to be reloaded
+  private AtomicBoolean discarded;
+
+  public FunctionInfo(String displayName, String className, FunctionResource... resources) {
+    this.functionType = FunctionType.PERSISTENT;
+    this.displayName = displayName;
+    this.className = className;
+    this.isInternalTableFunction = false;
+    this.resources = resources;
+    this.discarded = new AtomicBoolean(false);  // shared to all session functions
+  }
+
   public FunctionInfo(boolean isNative, String displayName,
-      GenericUDF genericUDF) {
-    this.isNative = isNative;
+      GenericUDF genericUDF, FunctionResource... resources) {
+    this.functionType = isNative ? FunctionType.BUILTIN : FunctionType.TEMPORARY;
     this.displayName = displayName;
     this.genericUDF = genericUDF;
     this.isInternalTableFunction = false;
+    this.resources = resources;
   }
 
   public FunctionInfo(boolean isNative, String displayName,
-      GenericUDAFResolver genericUDAFResolver) {
-    this.isNative = isNative;
+      GenericUDAFResolver genericUDAFResolver, FunctionResource... resources) {
+    this.functionType = isNative ? FunctionType.BUILTIN : FunctionType.TEMPORARY;
     this.displayName = displayName;
     this.genericUDAFResolver = genericUDAFResolver;
     this.isInternalTableFunction = false;
+    this.resources = resources;
   }
 
   public FunctionInfo(boolean isNative, String displayName,
-      GenericUDTF genericUDTF) {
-    this.isNative = isNative;
+      GenericUDTF genericUDTF, FunctionResource... resources) {
+    this.functionType = isNative ? FunctionType.BUILTIN : FunctionType.TEMPORARY;
     this.displayName = displayName;
     this.genericUDTF = genericUDTF;
     this.isInternalTableFunction = false;
+    this.resources = resources;
   }
 
-  public FunctionInfo(String displayName, Class tFnCls)
-  {
+  public FunctionInfo(boolean isNative, String displayName, Class tFnCls,
+      FunctionResource... resources) {
+    this.functionType = isNative ? FunctionType.BUILTIN : FunctionType.TEMPORARY;
     this.displayName = displayName;
     this.tableFunctionResolver = tFnCls;
     PartitionTableFunctionDescription def = AnnotationUtils.getAnnotation(
         tableFunctionResolver, PartitionTableFunctionDescription.class);
-    this.isNative = (def == null) ? false : def.isInternal();
-    this.isInternalTableFunction = isNative;
+    this.isInternalTableFunction = def != null && def.isInternal();
+    this.resources = resources;
   }
 
   /**
@@ -154,7 +182,7 @@
    * Native functions cannot be unregistered.
    */
   public boolean isNative() {
-    return isNative;
+    return functionType == FunctionType.BUILTIN || functionType == FunctionType.PERSISTENT;
   }
 
   /**
@@ -201,4 +229,56 @@
     this.blockedFunction = blockedFunction;
   }
 
+  public boolean isBuiltIn() {
+    return functionType == FunctionType.BUILTIN;
+  }
+
+  public boolean isPersistent() {
+    return functionType == FunctionType.PERSISTENT;
+  }
+
+  public String getClassName() {
+    return className;
+  }
+
+  public FunctionResource[] getResources() {
+    return resources;
+  }
+
+  public void discarded() {
+    if (discarded != null) {
+      discarded.set(true);
+    }
+  }
+
+  // for persistent function
+  public boolean isDiscarded() {
+    return discarded != null && discarded.get();
+  }
+
+  // for persistent function
+  public void shareStateWith(FunctionInfo function) {
+    if (function != null) {
+      function.discarded = discarded;
+    }
+  }
+
+  public static class FunctionResource {
+    private final SessionState.ResourceType resourceType;
+    private final String resourceURI;
+    public FunctionResource(SessionState.ResourceType resourceType, String resourceURI) {
+      this.resourceType = resourceType;
+      this.resourceURI = resourceURI;
+    }
+    public SessionState.ResourceType getResourceType() {
+      return resourceType;
+    }
+    public String getResourceURI() {
+      return resourceURI;
+    }
+    @Override
+    public String toString() {
+      return resourceType + ":" + resourceURI;
+    }
+  }
 }
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java	(working copy)
@@ -18,35 +18,23 @@
 
 package org.apache.hadoop.hive.ql.exec;
 
-import java.io.InputStream;
 import java.lang.reflect.Method;
-import java.net.URL;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.Collections;
 import java.util.EnumMap;
 import java.util.HashSet;
 import java.util.LinkedHashSet;
 import java.util.Iterator;
-import java.util.LinkedHashMap;
 import java.util.List;
-import java.util.Map;
-import java.util.Map.Entry;
 import java.util.Set;
 import java.util.TreeSet;
 import java.util.regex.Pattern;
 import java.util.regex.PatternSyntaxException;
 
-import javax.xml.parsers.DocumentBuilder;
-import javax.xml.parsers.DocumentBuilderFactory;
-
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
-import org.apache.hadoop.hive.conf.HiveConf;
-import org.apache.hadoop.hive.metastore.api.Function;
-import org.apache.hadoop.hive.metastore.api.MetaException;
-import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
-import org.apache.hadoop.hive.ql.exec.FunctionUtils.UDFClassType;
-import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.exec.FunctionInfo.FunctionResource;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
 import org.apache.hadoop.hive.ql.parse.SemanticException;
 import org.apache.hadoop.hive.ql.plan.ExprNodeDesc;
@@ -142,30 +130,16 @@
 import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
 import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory;
 import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils;
-import org.apache.hadoop.io.IOUtils;
 import org.apache.hadoop.util.ReflectionUtils;
 import org.apache.hive.common.util.AnnotationUtils;
-import org.w3c.dom.Document;
-import org.w3c.dom.Element;
-import org.w3c.dom.NodeList;
 
-import com.google.common.base.Splitter;
-import com.google.common.collect.Lists;
-
-
 /**
  * FunctionRegistry.
  */
 public final class FunctionRegistry {
 
-  private static Log LOG = LogFactory.getLog(FunctionRegistry.class);
+  private static final Log LOG = LogFactory.getLog(FunctionRegistry.class);
 
-  /**
-   * The mapping from expression function names to expression classes.
-   */
-  static Map mFunctions = Collections.synchronizedMap(new LinkedHashMap());
-
-  static Set> nativeUdfs = Collections.synchronizedSet(new HashSet>());
   /*
    * PTF variables
    * */
@@ -182,570 +156,330 @@
   private static final String NOOP_MAP_TABLE_FUNCTION = "noopwithmap";
   private static final String NOOP_STREAMING_TABLE_FUNCTION = "noopstreaming";
   private static final String NOOP_STREAMING_MAP_TABLE_FUNCTION = "noopwithmapstreaming";
+  private static final String MATCH_PATH_TABLE_FUNCTION = "matchpath";
 
-  static Map windowFunctions = Collections.synchronizedMap(new LinkedHashMap());
+  public static final Set HIVE_OPERATORS = new HashSet();
 
   static {
-    registerGenericUDF("concat", GenericUDFConcat.class);
-    registerUDF("substr", UDFSubstr.class, false);
-    registerUDF("substring", UDFSubstr.class, false);
-    registerUDF("space", UDFSpace.class, false);
-    registerUDF("repeat", UDFRepeat.class, false);
-    registerUDF("ascii", UDFAscii.class, false);
-    registerGenericUDF("lpad", GenericUDFLpad.class);
-    registerGenericUDF("rpad", GenericUDFRpad.class);
+    HIVE_OPERATORS.addAll(Arrays.asList(
+        "+", "-", "*", "/", "%", "div", "&", "|", "^", "~",
+        "and", "or", "not", "!",
+        "=", "==", "<=>", "!=", "<>", "<", "<=", ">", ">=",
+        "index"));
+  }
 
-    registerGenericUDF("size", GenericUDFSize.class);
+  // registry for system functions
+  private static final Registry system = new Registry(true);
 
-    registerGenericUDF("round", GenericUDFRound.class);
-    registerGenericUDF("floor", GenericUDFFloor.class);
-    registerUDF("sqrt", UDFSqrt.class, false);
-    registerGenericUDF("ceil", GenericUDFCeil.class);
-    registerGenericUDF("ceiling", GenericUDFCeil.class);
-    registerUDF("rand", UDFRand.class, false);
-    registerGenericUDF("abs", GenericUDFAbs.class);
-    registerGenericUDF("pmod", GenericUDFPosMod.class);
+  static {
+    system.registerGenericUDF("concat", GenericUDFConcat.class);
+    system.registerUDF("substr", UDFSubstr.class, false);
+    system.registerUDF("substring", UDFSubstr.class, false);
+    system.registerUDF("space", UDFSpace.class, false);
+    system.registerUDF("repeat", UDFRepeat.class, false);
+    system.registerUDF("ascii", UDFAscii.class, false);
+    system.registerGenericUDF("lpad", GenericUDFLpad.class);
+    system.registerGenericUDF("rpad", GenericUDFRpad.class);
+    system.registerGenericUDF("levenshtein", GenericUDFLevenstein.class);
 
-    registerUDF("ln", UDFLn.class, false);
-    registerUDF("log2", UDFLog2.class, false);
-    registerUDF("sin", UDFSin.class, false);
-    registerUDF("asin", UDFAsin.class, false);
-    registerUDF("cos", UDFCos.class, false);
-    registerUDF("acos", UDFAcos.class, false);
-    registerUDF("log10", UDFLog10.class, false);
-    registerUDF("log", UDFLog.class, false);
-    registerUDF("exp", UDFExp.class, false);
-    registerGenericUDF("power", GenericUDFPower.class);
-    registerGenericUDF("pow", GenericUDFPower.class);
-    registerUDF("sign", UDFSign.class, false);
-    registerUDF("pi", UDFPI.class, false);
-    registerUDF("degrees", UDFDegrees.class, false);
-    registerUDF("radians", UDFRadians.class, false);
-    registerUDF("atan", UDFAtan.class, false);
-    registerUDF("tan", UDFTan.class, false);
-    registerUDF("e", UDFE.class, false);
+    system.registerGenericUDF("size", GenericUDFSize.class);
 
-    registerUDF("conv", UDFConv.class, false);
-    registerUDF("bin", UDFBin.class, false);
-    registerUDF("hex", UDFHex.class, false);
-    registerUDF("unhex", UDFUnhex.class, false);
-    registerUDF("base64", UDFBase64.class, false);
-    registerUDF("unbase64", UDFUnbase64.class, false);
+    system.registerGenericUDF("round", GenericUDFRound.class);
+    system.registerGenericUDF("floor", GenericUDFFloor.class);
+    system.registerUDF("sqrt", UDFSqrt.class, false);
+    system.registerGenericUDF("ceil", GenericUDFCeil.class);
+    system.registerGenericUDF("ceiling", GenericUDFCeil.class);
+    system.registerUDF("rand", UDFRand.class, false);
+    system.registerGenericUDF("abs", GenericUDFAbs.class);
+    system.registerGenericUDF("pmod", GenericUDFPosMod.class);
 
-    registerGenericUDF("encode", GenericUDFEncode.class);
-    registerGenericUDF("decode", GenericUDFDecode.class);
+    system.registerUDF("ln", UDFLn.class, false);
+    system.registerUDF("log2", UDFLog2.class, false);
+    system.registerUDF("sin", UDFSin.class, false);
+    system.registerUDF("asin", UDFAsin.class, false);
+    system.registerUDF("cos", UDFCos.class, false);
+    system.registerUDF("acos", UDFAcos.class, false);
+    system.registerUDF("log10", UDFLog10.class, false);
+    system.registerUDF("log", UDFLog.class, false);
+    system.registerUDF("exp", UDFExp.class, false);
+    system.registerGenericUDF("power", GenericUDFPower.class);
+    system.registerGenericUDF("pow", GenericUDFPower.class);
+    system.registerUDF("sign", UDFSign.class, false);
+    system.registerUDF("pi", UDFPI.class, false);
+    system.registerUDF("degrees", UDFDegrees.class, false);
+    system.registerUDF("radians", UDFRadians.class, false);
+    system.registerUDF("atan", UDFAtan.class, false);
+    system.registerUDF("tan", UDFTan.class, false);
+    system.registerUDF("e", UDFE.class, false);
 
-    registerGenericUDF("upper", GenericUDFUpper.class);
-    registerGenericUDF("lower", GenericUDFLower.class);
-    registerGenericUDF("ucase", GenericUDFUpper.class);
-    registerGenericUDF("lcase", GenericUDFLower.class);
-    registerGenericUDF("trim", GenericUDFTrim.class);
-    registerGenericUDF("ltrim", GenericUDFLTrim.class);
-    registerGenericUDF("rtrim", GenericUDFRTrim.class);
-    registerUDF("length", UDFLength.class, false);
-    registerUDF("reverse", UDFReverse.class, false);
-    registerGenericUDF("field", GenericUDFField.class);
-    registerUDF("find_in_set", UDFFindInSet.class, false);
-    registerGenericUDF("initcap", GenericUDFInitCap.class);
+    system.registerUDF("conv", UDFConv.class, false);
+    system.registerUDF("bin", UDFBin.class, false);
+    system.registerUDF("hex", UDFHex.class, false);
+    system.registerUDF("unhex", UDFUnhex.class, false);
+    system.registerUDF("base64", UDFBase64.class, false);
+    system.registerUDF("unbase64", UDFUnbase64.class, false);
 
-    registerUDF("like", UDFLike.class, true);
-    registerUDF("rlike", UDFRegExp.class, true);
-    registerUDF("regexp", UDFRegExp.class, true);
-    registerUDF("regexp_replace", UDFRegExpReplace.class, false);
-    registerUDF("regexp_extract", UDFRegExpExtract.class, false);
-    registerUDF("parse_url", UDFParseUrl.class, false);
-    registerGenericUDF("nvl", GenericUDFNvl.class);
-    registerGenericUDF("split", GenericUDFSplit.class);
-    registerGenericUDF("str_to_map", GenericUDFStringToMap.class);
-    registerGenericUDF("translate", GenericUDFTranslate.class);
+    system.registerGenericUDF("encode", GenericUDFEncode.class);
+    system.registerGenericUDF("decode", GenericUDFDecode.class);
 
-    registerGenericUDF(UNARY_PLUS_FUNC_NAME, GenericUDFOPPositive.class);
-    registerGenericUDF(UNARY_MINUS_FUNC_NAME, GenericUDFOPNegative.class);
+    system.registerGenericUDF("upper", GenericUDFUpper.class);
+    system.registerGenericUDF("lower", GenericUDFLower.class);
+    system.registerGenericUDF("ucase", GenericUDFUpper.class);
+    system.registerGenericUDF("lcase", GenericUDFLower.class);
+    system.registerGenericUDF("trim", GenericUDFTrim.class);
+    system.registerGenericUDF("ltrim", GenericUDFLTrim.class);
+    system.registerGenericUDF("rtrim", GenericUDFRTrim.class);
+    system.registerUDF("length", UDFLength.class, false);
+    system.registerUDF("reverse", UDFReverse.class, false);
+    system.registerGenericUDF("field", GenericUDFField.class);
+    system.registerUDF("find_in_set", UDFFindInSet.class, false);
+    system.registerGenericUDF("initcap", GenericUDFInitCap.class);
 
-    registerUDF("day", UDFDayOfMonth.class, false);
-    registerUDF("dayofmonth", UDFDayOfMonth.class, false);
-    registerUDF("month", UDFMonth.class, false);
-    registerUDF("year", UDFYear.class, false);
-    registerUDF("hour", UDFHour.class, false);
-    registerUDF("minute", UDFMinute.class, false);
-    registerUDF("second", UDFSecond.class, false);
-    registerUDF("from_unixtime", UDFFromUnixTime.class, false);
-    registerGenericUDF("to_date", GenericUDFDate.class);
-    registerUDF("weekofyear", UDFWeekOfYear.class, false);
-    registerGenericUDF("last_day", GenericUDFLastDay.class);
+    system.registerUDF("like", UDFLike.class, true);
+    system.registerUDF("rlike", UDFRegExp.class, true);
+    system.registerUDF("regexp", UDFRegExp.class, true);
+    system.registerUDF("regexp_replace", UDFRegExpReplace.class, false);
+    system.registerUDF("regexp_extract", UDFRegExpExtract.class, false);
+    system.registerUDF("parse_url", UDFParseUrl.class, false);
+    system.registerGenericUDF("nvl", GenericUDFNvl.class);
+    system.registerGenericUDF("split", GenericUDFSplit.class);
+    system.registerGenericUDF("str_to_map", GenericUDFStringToMap.class);
+    system.registerGenericUDF("translate", GenericUDFTranslate.class);
 
-    registerGenericUDF("date_add", GenericUDFDateAdd.class);
-    registerGenericUDF("date_sub", GenericUDFDateSub.class);
-    registerGenericUDF("datediff", GenericUDFDateDiff.class);
-    registerGenericUDF("add_months", GenericUDFAddMonths.class);
+    system.registerGenericUDF(UNARY_PLUS_FUNC_NAME, GenericUDFOPPositive.class);
+    system.registerGenericUDF(UNARY_MINUS_FUNC_NAME, GenericUDFOPNegative.class);
 
-    registerUDF("get_json_object", UDFJson.class, false);
+    system.registerUDF("day", UDFDayOfMonth.class, false);
+    system.registerUDF("dayofmonth", UDFDayOfMonth.class, false);
+    system.registerUDF("month", UDFMonth.class, false);
+    system.registerUDF("year", UDFYear.class, false);
+    system.registerUDF("hour", UDFHour.class, false);
+    system.registerUDF("minute", UDFMinute.class, false);
+    system.registerUDF("second", UDFSecond.class, false);
+    system.registerUDF("from_unixtime", UDFFromUnixTime.class, false);
+    system.registerGenericUDF("to_date", GenericUDFDate.class);
+    system.registerUDF("weekofyear", UDFWeekOfYear.class, false);
+    system.registerGenericUDF("last_day", GenericUDFLastDay.class);
+    system.registerGenericUDF("next_day", GenericUDFNextDay.class);
 
-    registerUDF("xpath_string", UDFXPathString.class, false);
-    registerUDF("xpath_boolean", UDFXPathBoolean.class, false);
-    registerUDF("xpath_number", UDFXPathDouble.class, false);
-    registerUDF("xpath_double", UDFXPathDouble.class, false);
-    registerUDF("xpath_float", UDFXPathFloat.class, false);
-    registerUDF("xpath_long", UDFXPathLong.class, false);
-    registerUDF("xpath_int", UDFXPathInteger.class, false);
-    registerUDF("xpath_short", UDFXPathShort.class, false);
-    registerGenericUDF("xpath", GenericUDFXPath.class);
+    system.registerGenericUDF("date_add", GenericUDFDateAdd.class);
+    system.registerGenericUDF("date_sub", GenericUDFDateSub.class);
+    system.registerGenericUDF("datediff", GenericUDFDateDiff.class);
+    system.registerGenericUDF("add_months", GenericUDFAddMonths.class);
 
-    registerGenericUDF("+", GenericUDFOPPlus.class);
-    registerGenericUDF("-", GenericUDFOPMinus.class);
-    registerGenericUDF("*", GenericUDFOPMultiply.class);
-    registerGenericUDF("/", GenericUDFOPDivide.class);
-    registerGenericUDF("%", GenericUDFOPMod.class);
-    registerUDF("div", UDFOPLongDivide.class, true);
+    system.registerUDF("get_json_object", UDFJson.class, false);
 
-    registerUDF("&", UDFOPBitAnd.class, true);
-    registerUDF("|", UDFOPBitOr.class, true);
-    registerUDF("^", UDFOPBitXor.class, true);
-    registerUDF("~", UDFOPBitNot.class, true);
+    system.registerUDF("xpath_string", UDFXPathString.class, false);
+    system.registerUDF("xpath_boolean", UDFXPathBoolean.class, false);
+    system.registerUDF("xpath_number", UDFXPathDouble.class, false);
+    system.registerUDF("xpath_double", UDFXPathDouble.class, false);
+    system.registerUDF("xpath_float", UDFXPathFloat.class, false);
+    system.registerUDF("xpath_long", UDFXPathLong.class, false);
+    system.registerUDF("xpath_int", UDFXPathInteger.class, false);
+    system.registerUDF("xpath_short", UDFXPathShort.class, false);
+    system.registerGenericUDF("xpath", GenericUDFXPath.class);
 
-    registerGenericUDF("current_database", UDFCurrentDB.class);
-    registerGenericUDF("current_date", GenericUDFCurrentDate.class);
-    registerGenericUDF("current_timestamp", GenericUDFCurrentTimestamp.class);
-    registerGenericUDF("current_user", GenericUDFCurrentUser.class);
+    system.registerGenericUDF("+", GenericUDFOPPlus.class);
+    system.registerGenericUDF("-", GenericUDFOPMinus.class);
+    system.registerGenericUDF("*", GenericUDFOPMultiply.class);
+    system.registerGenericUDF("/", GenericUDFOPDivide.class);
+    system.registerGenericUDF("%", GenericUDFOPMod.class);
+    system.registerUDF("div", UDFOPLongDivide.class, true);
 
-    registerGenericUDF("isnull", GenericUDFOPNull.class);
-    registerGenericUDF("isnotnull", GenericUDFOPNotNull.class);
+    system.registerUDF("&", UDFOPBitAnd.class, true);
+    system.registerUDF("|", UDFOPBitOr.class, true);
+    system.registerUDF("^", UDFOPBitXor.class, true);
+    system.registerUDF("~", UDFOPBitNot.class, true);
 
-    registerGenericUDF("if", GenericUDFIf.class);
-    registerGenericUDF("in", GenericUDFIn.class);
-    registerGenericUDF("and", GenericUDFOPAnd.class);
-    registerGenericUDF("or", GenericUDFOPOr.class);
-    registerGenericUDF("=", GenericUDFOPEqual.class);
-    registerGenericUDF("==", GenericUDFOPEqual.class);
-    registerGenericUDF("<=>", GenericUDFOPEqualNS.class);
-    registerGenericUDF("!=", GenericUDFOPNotEqual.class);
-    registerGenericUDF("<>", GenericUDFOPNotEqual.class);
-    registerGenericUDF("<", GenericUDFOPLessThan.class);
-    registerGenericUDF("<=", GenericUDFOPEqualOrLessThan.class);
-    registerGenericUDF(">", GenericUDFOPGreaterThan.class);
-    registerGenericUDF(">=", GenericUDFOPEqualOrGreaterThan.class);
-    registerGenericUDF("not", GenericUDFOPNot.class);
-    registerGenericUDF("!", GenericUDFOPNot.class);
-    registerGenericUDF("between", GenericUDFBetween.class);
+    system.registerGenericUDF("current_database", UDFCurrentDB.class);
+    system.registerGenericUDF("current_date", GenericUDFCurrentDate.class);
+    system.registerGenericUDF("current_timestamp", GenericUDFCurrentTimestamp.class);
+    system.registerGenericUDF("current_user", GenericUDFCurrentUser.class);
 
-    registerGenericUDF("ewah_bitmap_and", GenericUDFEWAHBitmapAnd.class);
-    registerGenericUDF("ewah_bitmap_or", GenericUDFEWAHBitmapOr.class);
-    registerGenericUDF("ewah_bitmap_empty", GenericUDFEWAHBitmapEmpty.class);
+    system.registerGenericUDF("isnull", GenericUDFOPNull.class);
+    system.registerGenericUDF("isnotnull", GenericUDFOPNotNull.class);
 
+    system.registerGenericUDF("if", GenericUDFIf.class);
+    system.registerGenericUDF("in", GenericUDFIn.class);
+    system.registerGenericUDF("and", GenericUDFOPAnd.class);
+    system.registerGenericUDF("or", GenericUDFOPOr.class);
+    system.registerGenericUDF("=", GenericUDFOPEqual.class);
+    system.registerGenericUDF("==", GenericUDFOPEqual.class);
+    system.registerGenericUDF("<=>", GenericUDFOPEqualNS.class);
+    system.registerGenericUDF("!=", GenericUDFOPNotEqual.class);
+    system.registerGenericUDF("<>", GenericUDFOPNotEqual.class);
+    system.registerGenericUDF("<", GenericUDFOPLessThan.class);
+    system.registerGenericUDF("<=", GenericUDFOPEqualOrLessThan.class);
+    system.registerGenericUDF(">", GenericUDFOPGreaterThan.class);
+    system.registerGenericUDF(">=", GenericUDFOPEqualOrGreaterThan.class);
+    system.registerGenericUDF("not", GenericUDFOPNot.class);
+    system.registerGenericUDF("!", GenericUDFOPNot.class);
+    system.registerGenericUDF("between", GenericUDFBetween.class);
 
+    system.registerGenericUDF("ewah_bitmap_and", GenericUDFEWAHBitmapAnd.class);
+    system.registerGenericUDF("ewah_bitmap_or", GenericUDFEWAHBitmapOr.class);
+    system.registerGenericUDF("ewah_bitmap_empty", GenericUDFEWAHBitmapEmpty.class);
+
     // Aliases for Java Class Names
     // These are used in getImplicitConvertUDFMethod
-    registerUDF(serdeConstants.BOOLEAN_TYPE_NAME, UDFToBoolean.class, false,
-        UDFToBoolean.class.getSimpleName());
-    registerUDF(serdeConstants.TINYINT_TYPE_NAME, UDFToByte.class, false,
-        UDFToByte.class.getSimpleName());
-    registerUDF(serdeConstants.SMALLINT_TYPE_NAME, UDFToShort.class, false,
-        UDFToShort.class.getSimpleName());
-    registerUDF(serdeConstants.INT_TYPE_NAME, UDFToInteger.class, false,
-        UDFToInteger.class.getSimpleName());
-    registerUDF(serdeConstants.BIGINT_TYPE_NAME, UDFToLong.class, false,
-        UDFToLong.class.getSimpleName());
-    registerUDF(serdeConstants.FLOAT_TYPE_NAME, UDFToFloat.class, false,
-        UDFToFloat.class.getSimpleName());
-    registerUDF(serdeConstants.DOUBLE_TYPE_NAME, UDFToDouble.class, false,
-        UDFToDouble.class.getSimpleName());
-    registerUDF(serdeConstants.STRING_TYPE_NAME, UDFToString.class, false,
-        UDFToString.class.getSimpleName());
+    system.registerUDF(serdeConstants.BOOLEAN_TYPE_NAME, UDFToBoolean.class, false, UDFToBoolean.class.getSimpleName());
+    system.registerUDF(serdeConstants.TINYINT_TYPE_NAME, UDFToByte.class, false, UDFToByte.class.getSimpleName());
+    system.registerUDF(serdeConstants.SMALLINT_TYPE_NAME, UDFToShort.class, false, UDFToShort.class.getSimpleName());
+    system.registerUDF(serdeConstants.INT_TYPE_NAME, UDFToInteger.class, false, UDFToInteger.class.getSimpleName());
+    system.registerUDF(serdeConstants.BIGINT_TYPE_NAME, UDFToLong.class, false, UDFToLong.class.getSimpleName());
+    system.registerUDF(serdeConstants.FLOAT_TYPE_NAME, UDFToFloat.class, false, UDFToFloat.class.getSimpleName());
+    system.registerUDF(serdeConstants.DOUBLE_TYPE_NAME, UDFToDouble.class, false, UDFToDouble.class.getSimpleName());
+    system.registerUDF(serdeConstants.STRING_TYPE_NAME, UDFToString.class, false, UDFToString.class.getSimpleName());
 
-    registerGenericUDF(serdeConstants.DATE_TYPE_NAME,
-        GenericUDFToDate.class);
-    registerGenericUDF(serdeConstants.TIMESTAMP_TYPE_NAME,
-        GenericUDFTimestamp.class);
-    registerGenericUDF(serdeConstants.BINARY_TYPE_NAME,
-        GenericUDFToBinary.class);
-    registerGenericUDF(serdeConstants.DECIMAL_TYPE_NAME,
-        GenericUDFToDecimal.class);
-    registerGenericUDF(serdeConstants.VARCHAR_TYPE_NAME,
-        GenericUDFToVarchar.class);
-    registerGenericUDF(serdeConstants.CHAR_TYPE_NAME,
-        GenericUDFToChar.class);
+    system.registerGenericUDF(serdeConstants.DATE_TYPE_NAME, GenericUDFToDate.class);
+    system.registerGenericUDF(serdeConstants.TIMESTAMP_TYPE_NAME, GenericUDFTimestamp.class);
+    system.registerGenericUDF(serdeConstants.BINARY_TYPE_NAME, GenericUDFToBinary.class);
+    system.registerGenericUDF(serdeConstants.DECIMAL_TYPE_NAME, GenericUDFToDecimal.class);
+    system.registerGenericUDF(serdeConstants.VARCHAR_TYPE_NAME, GenericUDFToVarchar.class);
+    system.registerGenericUDF(serdeConstants.CHAR_TYPE_NAME, GenericUDFToChar.class);
 
     // Aggregate functions
-    registerGenericUDAF("max", new GenericUDAFMax());
-    registerGenericUDAF("min", new GenericUDAFMin());
+    system.registerGenericUDAF("max", new GenericUDAFMax());
+    system.registerGenericUDAF("min", new GenericUDAFMin());
 
-    registerGenericUDAF("sum", new GenericUDAFSum());
-    registerGenericUDAF("count", new GenericUDAFCount());
-    registerGenericUDAF("avg", new GenericUDAFAverage());
-    registerGenericUDAF("std", new GenericUDAFStd());
-    registerGenericUDAF("stddev", new GenericUDAFStd());
-    registerGenericUDAF("stddev_pop", new GenericUDAFStd());
-    registerGenericUDAF("stddev_samp", new GenericUDAFStdSample());
-    registerGenericUDAF("variance", new GenericUDAFVariance());
-    registerGenericUDAF("var_pop", new GenericUDAFVariance());
-    registerGenericUDAF("var_samp", new GenericUDAFVarianceSample());
-    registerGenericUDAF("covar_pop", new GenericUDAFCovariance());
-    registerGenericUDAF("covar_samp", new GenericUDAFCovarianceSample());
-    registerGenericUDAF("corr", new GenericUDAFCorrelation());
-    registerGenericUDAF("histogram_numeric", new GenericUDAFHistogramNumeric());
-    registerGenericUDAF("percentile_approx", new GenericUDAFPercentileApprox());
-    registerGenericUDAF("collect_set", new GenericUDAFCollectSet());
-    registerGenericUDAF("collect_list", new GenericUDAFCollectList());
+    system.registerGenericUDAF("sum", new GenericUDAFSum());
+    system.registerGenericUDAF("count", new GenericUDAFCount());
+    system.registerGenericUDAF("avg", new GenericUDAFAverage());
+    system.registerGenericUDAF("std", new GenericUDAFStd());
+    system.registerGenericUDAF("stddev", new GenericUDAFStd());
+    system.registerGenericUDAF("stddev_pop", new GenericUDAFStd());
+    system.registerGenericUDAF("stddev_samp", new GenericUDAFStdSample());
+    system.registerGenericUDAF("variance", new GenericUDAFVariance());
+    system.registerGenericUDAF("var_pop", new GenericUDAFVariance());
+    system.registerGenericUDAF("var_samp", new GenericUDAFVarianceSample());
+    system.registerGenericUDAF("covar_pop", new GenericUDAFCovariance());
+    system.registerGenericUDAF("covar_samp", new GenericUDAFCovarianceSample());
+    system.registerGenericUDAF("corr", new GenericUDAFCorrelation());
+    system.registerGenericUDAF("histogram_numeric", new GenericUDAFHistogramNumeric());
+    system.registerGenericUDAF("percentile_approx", new GenericUDAFPercentileApprox());
+    system.registerGenericUDAF("collect_set", new GenericUDAFCollectSet());
+    system.registerGenericUDAF("collect_list", new GenericUDAFCollectList());
 
-    registerGenericUDAF("ngrams", new GenericUDAFnGrams());
-    registerGenericUDAF("context_ngrams", new GenericUDAFContextNGrams());
+    system.registerGenericUDAF("ngrams", new GenericUDAFnGrams());
+    system.registerGenericUDAF("context_ngrams", new GenericUDAFContextNGrams());
 
-    registerGenericUDAF("ewah_bitmap", new GenericUDAFEWAHBitmap());
+    system.registerGenericUDAF("ewah_bitmap", new GenericUDAFEWAHBitmap());
 
-    registerGenericUDAF("compute_stats" , new GenericUDAFComputeStats());
+    system.registerGenericUDAF("compute_stats", new GenericUDAFComputeStats());
 
-    registerUDAF("percentile", UDAFPercentile.class);
+    system.registerUDAF("percentile", UDAFPercentile.class);
 
 
     // Generic UDFs
-    registerGenericUDF("reflect", GenericUDFReflect.class);
-    registerGenericUDF("reflect2", GenericUDFReflect2.class);
-    registerGenericUDF("java_method", GenericUDFReflect.class);
+    system.registerGenericUDF("reflect", GenericUDFReflect.class);
+    system.registerGenericUDF("reflect2", GenericUDFReflect2.class);
+    system.registerGenericUDF("java_method", GenericUDFReflect.class);
 
-    registerGenericUDF("array", GenericUDFArray.class);
-    registerGenericUDF("assert_true", GenericUDFAssertTrue.class);
-    registerGenericUDF("map", GenericUDFMap.class);
-    registerGenericUDF("struct", GenericUDFStruct.class);
-    registerGenericUDF("named_struct", GenericUDFNamedStruct.class);
-    registerGenericUDF("create_union", GenericUDFUnion.class);
+    system.registerGenericUDF("array", GenericUDFArray.class);
+    system.registerGenericUDF("assert_true", GenericUDFAssertTrue.class);
+    system.registerGenericUDF("map", GenericUDFMap.class);
+    system.registerGenericUDF("struct", GenericUDFStruct.class);
+    system.registerGenericUDF("named_struct", GenericUDFNamedStruct.class);
+    system.registerGenericUDF("create_union", GenericUDFUnion.class);
 
-    registerGenericUDF("case", GenericUDFCase.class);
-    registerGenericUDF("when", GenericUDFWhen.class);
-    registerGenericUDF("hash", GenericUDFHash.class);
-    registerGenericUDF("coalesce", GenericUDFCoalesce.class);
-    registerGenericUDF("index", GenericUDFIndex.class);
-    registerGenericUDF("in_file", GenericUDFInFile.class);
-    registerGenericUDF("instr", GenericUDFInstr.class);
-    registerGenericUDF("locate", GenericUDFLocate.class);
-    registerGenericUDF("elt", GenericUDFElt.class);
-    registerGenericUDF("concat_ws", GenericUDFConcatWS.class);
-    registerGenericUDF("sort_array", GenericUDFSortArray.class);
-    registerGenericUDF("array_contains", GenericUDFArrayContains.class);
-    registerGenericUDF("sentences", GenericUDFSentences.class);
-    registerGenericUDF("map_keys", GenericUDFMapKeys.class);
-    registerGenericUDF("map_values", GenericUDFMapValues.class);
-    registerGenericUDF("format_number", GenericUDFFormatNumber.class);
-    registerGenericUDF("printf", GenericUDFPrintf.class);
-    registerGenericUDF("greatest", GenericUDFGreatest.class);
-    registerGenericUDF("least", GenericUDFLeast.class);
+    system.registerGenericUDF("case", GenericUDFCase.class);
+    system.registerGenericUDF("when", GenericUDFWhen.class);
+    system.registerGenericUDF("hash", GenericUDFHash.class);
+    system.registerGenericUDF("coalesce", GenericUDFCoalesce.class);
+    system.registerGenericUDF("index", GenericUDFIndex.class);
+    system.registerGenericUDF("in_file", GenericUDFInFile.class);
+    system.registerGenericUDF("instr", GenericUDFInstr.class);
+    system.registerGenericUDF("locate", GenericUDFLocate.class);
+    system.registerGenericUDF("elt", GenericUDFElt.class);
+    system.registerGenericUDF("concat_ws", GenericUDFConcatWS.class);
+    system.registerGenericUDF("sort_array", GenericUDFSortArray.class);
+    system.registerGenericUDF("array_contains", GenericUDFArrayContains.class);
+    system.registerGenericUDF("sentences", GenericUDFSentences.class);
+    system.registerGenericUDF("map_keys", GenericUDFMapKeys.class);
+    system.registerGenericUDF("map_values", GenericUDFMapValues.class);
+    system.registerGenericUDF("format_number", GenericUDFFormatNumber.class);
+    system.registerGenericUDF("printf", GenericUDFPrintf.class);
+    system.registerGenericUDF("greatest", GenericUDFGreatest.class);
+    system.registerGenericUDF("least", GenericUDFLeast.class);
 
-    registerGenericUDF("from_utc_timestamp", GenericUDFFromUtcTimestamp.class);
-    registerGenericUDF("to_utc_timestamp", GenericUDFToUtcTimestamp.class);
+    system.registerGenericUDF("from_utc_timestamp", GenericUDFFromUtcTimestamp.class);
+    system.registerGenericUDF("to_utc_timestamp", GenericUDFToUtcTimestamp.class);
 
-    registerGenericUDF("unix_timestamp", GenericUDFUnixTimeStamp.class);
-    registerGenericUDF("to_unix_timestamp", GenericUDFToUnixTimeStamp.class);
+    system.registerGenericUDF("unix_timestamp", GenericUDFUnixTimeStamp.class);
+    system.registerGenericUDF("to_unix_timestamp", GenericUDFToUnixTimeStamp.class);
 
     // Generic UDTF's
-    registerGenericUDTF("explode", GenericUDTFExplode.class);
-    registerGenericUDTF("inline", GenericUDTFInline.class);
-    registerGenericUDTF("json_tuple", GenericUDTFJSONTuple.class);
-    registerGenericUDTF("parse_url_tuple", GenericUDTFParseUrlTuple.class);
-    registerGenericUDTF("posexplode", GenericUDTFPosExplode.class);
-    registerGenericUDTF("stack", GenericUDTFStack.class);
+    system.registerGenericUDTF("explode", GenericUDTFExplode.class);
+    system.registerGenericUDTF("inline", GenericUDTFInline.class);
+    system.registerGenericUDTF("json_tuple", GenericUDTFJSONTuple.class);
+    system.registerGenericUDTF("parse_url_tuple", GenericUDTFParseUrlTuple.class);
+    system.registerGenericUDTF("posexplode", GenericUDTFPosExplode.class);
+    system.registerGenericUDTF("stack", GenericUDTFStack.class);
 
     //PTF declarations
-    registerGenericUDF(LEAD_FUNC_NAME, GenericUDFLead.class);
-    registerGenericUDF(LAG_FUNC_NAME, GenericUDFLag.class);
+    system.registerGenericUDF(LEAD_FUNC_NAME, GenericUDFLead.class);
+    system.registerGenericUDF(LAG_FUNC_NAME, GenericUDFLag.class);
 
-    registerWindowFunction("row_number", new GenericUDAFRowNumber());
-    registerWindowFunction("rank", new GenericUDAFRank());
-    registerWindowFunction("dense_rank", new GenericUDAFDenseRank());
-    registerWindowFunction("percent_rank", new GenericUDAFPercentRank());
-    registerWindowFunction("cume_dist", new GenericUDAFCumeDist());
-    registerWindowFunction("ntile", new GenericUDAFNTile());
-    registerWindowFunction("first_value", new GenericUDAFFirstValue());
-    registerWindowFunction("last_value", new GenericUDAFLastValue());
-    registerWindowFunction(LEAD_FUNC_NAME, new GenericUDAFLead(), false);
-    registerWindowFunction(LAG_FUNC_NAME, new GenericUDAFLag(), false);
+    system.registerGenericUDAF("row_number", new GenericUDAFRowNumber());
+    system.registerGenericUDAF("rank", new GenericUDAFRank());
+    system.registerGenericUDAF("dense_rank", new GenericUDAFDenseRank());
+    system.registerGenericUDAF("percent_rank", new GenericUDAFPercentRank());
+    system.registerGenericUDAF("cume_dist", new GenericUDAFCumeDist());
+    system.registerGenericUDAF("ntile", new GenericUDAFNTile());
+    system.registerGenericUDAF("first_value", new GenericUDAFFirstValue());
+    system.registerGenericUDAF("last_value", new GenericUDAFLastValue());
+    system.registerWindowFunction(LEAD_FUNC_NAME, new GenericUDAFLead());
+    system.registerWindowFunction(LAG_FUNC_NAME, new GenericUDAFLag());
 
-    registerTableFunction(NOOP_TABLE_FUNCTION, NoopResolver.class);
-    registerTableFunction(NOOP_MAP_TABLE_FUNCTION, NoopWithMapResolver.class);
-    registerTableFunction(NOOP_STREAMING_TABLE_FUNCTION, NoopStreamingResolver.class);
-    registerTableFunction(NOOP_STREAMING_MAP_TABLE_FUNCTION, NoopWithMapStreamingResolver.class);
-    registerTableFunction(WINDOWING_TABLE_FUNCTION,  WindowingTableFunctionResolver.class);
-    registerTableFunction("matchpath", MatchPathResolver.class);
+    system.registerTableFunction(NOOP_TABLE_FUNCTION, NoopResolver.class);
+    system.registerTableFunction(NOOP_MAP_TABLE_FUNCTION, NoopWithMapResolver.class);
+    system.registerTableFunction(NOOP_STREAMING_TABLE_FUNCTION, NoopStreamingResolver.class);
+    system.registerTableFunction(NOOP_STREAMING_MAP_TABLE_FUNCTION, NoopWithMapStreamingResolver.class);
+    system.registerTableFunction(WINDOWING_TABLE_FUNCTION, WindowingTableFunctionResolver.class);
+    system.registerTableFunction(MATCH_PATH_TABLE_FUNCTION, MatchPathResolver.class);
   }
 
-  public static void registerTemporaryUDF(String functionName,
-      Class UDFClass, boolean isOperator) {
-    registerUDF(false, functionName, UDFClass, isOperator);
-  }
-
-  static void registerUDF(String functionName, Class UDFClass,
-      boolean isOperator) {
-    registerUDF(true, functionName, UDFClass, isOperator);
-  }
-
-  public static void registerUDF(boolean isNative, String functionName,
-      Class UDFClass, boolean isOperator) {
-    registerUDF(isNative, functionName, UDFClass, isOperator, functionName
-        .toLowerCase());
-  }
-
-  public static void registerUDF(String functionName,
-      Class UDFClass, boolean isOperator, String displayName) {
-    registerUDF(true, functionName, UDFClass, isOperator, displayName);
-  }
-
-  public static void registerUDF(boolean isNative, String functionName,
-      Class UDFClass, boolean isOperator, String displayName) {
-    if (UDF.class.isAssignableFrom(UDFClass)) {
-      FunctionInfo fI = new FunctionInfo(isNative, displayName,
-          new GenericUDFBridge(displayName, isOperator, UDFClass.getName()));
-      mFunctions.put(functionName.toLowerCase(), fI);
-      registerNativeStatus(fI);
-    } else {
-      throw new RuntimeException("Registering UDF Class " + UDFClass
-          + " which does not extend " + UDF.class);
-    }
-  }
-
-  public static void registerTemporaryGenericUDF(String functionName,
-      Class genericUDFClass) {
-    registerGenericUDF(false, functionName, genericUDFClass);
-  }
-
-  static void registerGenericUDF(String functionName,
-      Class genericUDFClass) {
-    registerGenericUDF(true, functionName, genericUDFClass);
-  }
-
-  public static void registerGenericUDF(boolean isNative, String functionName,
-      Class genericUDFClass) {
-    if (GenericUDF.class.isAssignableFrom(genericUDFClass)) {
-      FunctionInfo fI = new FunctionInfo(isNative, functionName,
-          ReflectionUtils.newInstance(genericUDFClass, null));
-      mFunctions.put(functionName.toLowerCase(), fI);
-      registerNativeStatus(fI);
-    } else {
-      throw new RuntimeException("Registering GenericUDF Class "
-          + genericUDFClass + " which does not extend " + GenericUDF.class);
-    }
-  }
-
-  public static void registerTemporaryGenericUDTF(String functionName,
-      Class genericUDTFClass) {
-    registerGenericUDTF(false, functionName, genericUDTFClass);
-  }
-
-  static void registerGenericUDTF(String functionName,
-      Class genericUDTFClass) {
-    registerGenericUDTF(true, functionName, genericUDTFClass);
-  }
-
-  public static void registerGenericUDTF(boolean isNative, String functionName,
-      Class genericUDTFClass) {
-    if (GenericUDTF.class.isAssignableFrom(genericUDTFClass)) {
-      FunctionInfo fI = new FunctionInfo(isNative, functionName,
-          ReflectionUtils.newInstance(genericUDTFClass, null));
-      mFunctions.put(functionName.toLowerCase(), fI);
-      registerNativeStatus(fI);
-    } else {
-      throw new RuntimeException("Registering GenericUDTF Class "
-          + genericUDTFClass + " which does not extend " + GenericUDTF.class);
-    }
-  }
-
-  private static FunctionInfo getFunctionInfoFromMetastore(String functionName) {
-    FunctionInfo ret = null;
-
-    try {
-      String dbName;
-      String fName;
-      if (FunctionUtils.isQualifiedFunctionName(functionName)) {
-        String[] parts = FunctionUtils.splitQualifiedFunctionName(functionName);
-        dbName = parts[0];
-        fName = parts[1];
-      } else {
-        // otherwise, qualify using current db
-        dbName = SessionState.get().getCurrentDatabase().toLowerCase();
-        fName = functionName;
-      }
-
-      // Try looking up function in the metastore
-      HiveConf conf = SessionState.get().getConf();
-      Function func = Hive.get(conf).getFunction(dbName, fName);
-      if (func != null) {
-        // Found UDF in metastore - now add it to the function registry
-        // At this point we should add any relevant jars that would be needed for the UDf.
-        try {
-          FunctionTask.addFunctionResources(func.getResourceUris());
-        } catch (Exception e) {
-          LOG.error("Unable to load resources for " + dbName + "." + fName + ":" + e.getMessage(), e);
-          return null;
-        }
-
-        Class udfClass = Class.forName(func.getClassName(), true, Utilities.getSessionSpecifiedClassLoader());
-        if (registerTemporaryFunction(functionName, udfClass)) {
-          ret = mFunctions.get(functionName);
-        } else {
-          LOG.error(func.getClassName() + " is not a valid UDF class and was not registered.");
-        }
-      }
-    } catch (HiveException e) {
-      if (!((e.getCause() != null) && (e.getCause() instanceof MetaException)) &&
-         (e.getCause().getCause() != null) && (e.getCause().getCause() instanceof NoSuchObjectException))  {
-         LOG.info("Unable to lookup UDF in metastore: " + e);
-      }
-    } catch (ClassNotFoundException e) {
-      // Lookup of UDf class failed
-      LOG.error("Unable to load UDF class: " + e);
-    }
-
-    return ret;
-  }
-
-  private static  T getQualifiedFunctionInfo(
-      Map mFunctions, String functionName) {
-    T functionInfo =  mFunctions.get(functionName);
-    if (functionInfo == null) {
-      // Try looking up in metastore.
-      FunctionInfo fi = getFunctionInfoFromMetastore(functionName);
-      if (fi != null) {
-        // metastore lookup resulted in function getting added to mFunctions, try again
-        functionInfo = mFunctions.get(functionName);
-      }
-    }
-
-    // HIVE-6672: In HiveServer2 the JARs for this UDF may have been loaded by a different thread,
-    // and the current thread may not be able to resolve the UDF. Test for this condition
-    // and if necessary load the JARs in this thread.
-    if (functionInfo != null) {
-      loadFunctionResourcesIfNecessary(functionName, functionInfo);
-    }
-
-    return functionInfo;
-  }
-
-  private static void checkFunctionClass(CommonFunctionInfo cfi) throws ClassNotFoundException {
-    // This call will fail for non-generic UDFs using GenericUDFBridge
-    Class udfClass = cfi.getFunctionClass();
-    // Even if we have a reference to the class (which will be the case for GenericUDFs),
-    // the classloader may not be able to resolve the class, which would mean reflection-based
-    // methods would fail such as for plan deserialization. Make sure this works too.
-    Class.forName(udfClass.getName(), true, Utilities.getSessionSpecifiedClassLoader());
-  }
-
-  private static void loadFunctionResourcesIfNecessary(String functionName, CommonFunctionInfo cfi) {
-    try {
-      // Check if the necessary JARs have been loaded for this function.
-      checkFunctionClass(cfi);
-    } catch (Exception e) {
-      // Unable to resolve the UDF with the classloader.
-      // Look up the function in the metastore and load any resources.
-      LOG.debug("Attempting to reload resources for " + functionName);
-      try {
-        String[] parts = FunctionUtils.getQualifiedFunctionNameParts(functionName);
-        HiveConf conf = SessionState.get().getConf();
-        Function func = Hive.get(conf).getFunction(parts[0], parts[1]);
-        if (func != null) {
-          FunctionTask.addFunctionResources(func.getResourceUris());
-          // Check again now that we've loaded the resources in this thread.
-          checkFunctionClass(cfi);
-        } else {
-          // Couldn't find the function .. just rethrow the original error
-          LOG.error("Unable to reload resources for " + functionName);
-          throw e;
-        }
-      } catch (Exception err) {
-        throw new RuntimeException(err);
-      }
-    }
-  }
-
-  public static String getNormalizedFunctionName(String fn) {
+  public static String getNormalizedFunctionName(String fn) throws SemanticException {
     // Does the same thing as getFunctionInfo, except for getting the function info.
     fn = fn.toLowerCase();
-    return (FunctionUtils.isQualifiedFunctionName(fn) || mFunctions.get(fn) != null) ? fn
+    return (FunctionUtils.isQualifiedFunctionName(fn) || getFunctionInfo(fn) != null) ? fn
         : FunctionUtils.qualifyFunctionName(
-            fn, SessionState.get().getCurrentDatabase().toLowerCase());
+        fn, SessionState.get().getCurrentDatabase().toLowerCase());
   }
 
-  private static  T getFunctionInfo(
-      Map mFunctions, String functionName) {
-    functionName = functionName.toLowerCase();
-    T functionInfo = null;
-    if (FunctionUtils.isQualifiedFunctionName(functionName)) {
-      functionInfo = getQualifiedFunctionInfo(mFunctions, functionName);
-    } else {
-      // First try without qualifiers - would resolve builtin/temp functions.
-      // Otherwise try qualifying with current db name.
-      functionInfo =  mFunctions.get(functionName);
-      if (functionInfo == null && !FunctionUtils.isQualifiedFunctionName(functionName)) {
-        String qualifiedName = FunctionUtils.qualifyFunctionName(functionName,
-            SessionState.get().getCurrentDatabase().toLowerCase());
-        functionInfo = getQualifiedFunctionInfo(mFunctions, qualifiedName);
-      }
-    }
-    return functionInfo;
+  public static FunctionInfo getFunctionInfo(String functionName) throws SemanticException {
+    FunctionInfo info = getTemporaryFunctionInfo(functionName);
+    return info != null ? info : system.getFunctionInfo(functionName);
   }
 
-  public static FunctionInfo getFunctionInfo(String functionName) throws SemanticException {
-    FunctionInfo functionInfo = getFunctionInfo(mFunctions, functionName);
-    if (functionInfo != null && functionInfo.isBlockedFunction()) {
-      throw new SemanticException ("UDF " + functionName + " is not allowed");
-    }
-    return functionInfo;
+  public static FunctionInfo getTemporaryFunctionInfo(String functionName) throws SemanticException {
+    Registry registry = SessionState.getRegistry();
+    return registry == null ? null : registry.getFunctionInfo(functionName);
   }
 
-  /**
-   * Returns a set of registered function names. This is used for the CLI
-   * command "SHOW FUNCTIONS;"
-   *
-   * @return set of strings contains function names
-   */
-  public static Set getFunctionNames() {
-    return getFunctionNames(true);
+  public static WindowFunctionInfo getWindowFunctionInfo(String functionName)
+      throws SemanticException {
+    Registry registry = SessionState.getRegistry();
+    WindowFunctionInfo info =
+        registry == null ? null : registry.getWindowFunctionInfo(functionName);
+    return info != null ? info : system.getWindowFunctionInfo(functionName);
   }
 
-  private static Set getFunctionNames(boolean searchMetastore) {
-    Set functionNames = mFunctions.keySet();
-    if (searchMetastore) {
-      functionNames = new HashSet(functionNames);
-      try {
-        Hive db = getHive();
-        List dbNames = db.getAllDatabases();
-
-        for (String dbName : dbNames) {
-          List funcNames = db.getFunctions(dbName, "*");
-          for (String funcName : funcNames) {
-            functionNames.add(FunctionUtils.qualifyFunctionName(funcName, dbName));
-          }
-        }
-      } catch (Exception e) {
-        LOG.error(e);
-        // Continue on, we can still return the functions we've gotten to this point.
-      }
+  public static Set getFunctionNames() {
+    Set names = new TreeSet();
+    names.addAll(system.getCurrentFunctionNames());
+    if (SessionState.getRegistry() != null) {
+      names.addAll(SessionState.getRegistry().getCurrentFunctionNames());
     }
-    return functionNames;
+    return names;
   }
 
-  public static Hive getHive() throws HiveException {
-    return Hive.get(SessionState.get().getConf());
-  }
-
-  /**
-   * Returns a set of registered function names. This is used for the CLI
-   * command "SHOW FUNCTIONS 'regular expression';" Returns an empty set when
-   * the regular expression is not valid.
-   *
-   * @param funcPatternStr
-   *          regular expression of the interested function names
-   * @return set of strings contains function names
-   */
   public static Set getFunctionNames(String funcPatternStr) {
     Set funcNames = new TreeSet();
-    Pattern funcPattern = null;
-    try {
-      funcPattern = Pattern.compile(funcPatternStr);
-    } catch (PatternSyntaxException e) {
-      return funcNames;
+    funcNames.addAll(system.getFunctionNames(funcPatternStr));
+    if (SessionState.getRegistry() != null) {
+      funcNames.addAll(SessionState.getRegistry().getFunctionNames(funcPatternStr));
     }
-    for (String funcName : mFunctions.keySet()) {
-      if (funcPattern.matcher(funcName).matches()) {
-        funcNames.add(funcName);
-      }
-    }
     return funcNames;
   }
 
@@ -759,7 +493,7 @@
    */
   public static Set getFunctionNamesByLikePattern(String funcPatternStr) {
     Set funcNames = new TreeSet();
-    Set allFuncs = getFunctionNames(true);
+    Set allFuncs = getFunctionNames();
     String[] subpatterns = funcPatternStr.trim().split("\\|");
     for (String subpattern : subpatterns) {
       subpattern = "(?i)" + subpattern.replaceAll("\\*", ".*");
@@ -780,35 +514,19 @@
   /**
    * Returns the set of synonyms of the supplied function.
    *
-   * @param funcName
-   *          the name of the function
+   * @param funcName the name of the function
    * @return Set of synonyms for funcName
    */
-  public static Set getFunctionSynonyms(String funcName) {
-    // Must be deterministic order map for consistent q-test output across Java versions - see HIVE-9161
-    Set synonyms = new LinkedHashSet();
-
-    FunctionInfo funcInfo;
-    try {
-      funcInfo = getFunctionInfo(funcName);
-    } catch (SemanticException e) {
-      LOG.warn("Failed to load " + funcName);
-      funcInfo = null;
-    }
+  public static Set getFunctionSynonyms(String funcName) throws SemanticException {
+    FunctionInfo funcInfo = getFunctionInfo(funcName);
     if (null == funcInfo) {
-      return synonyms;
+      return Collections.emptySet();
     }
-
-    Class funcClass = funcInfo.getFunctionClass();
-    for (String name : mFunctions.keySet()) {
-      if (name.equals(funcName)) {
-        continue;
-      }
-      if (mFunctions.get(name).getFunctionClass().equals(funcClass)) {
-        synonyms.add(name);
-      }
+    Set synonyms = new LinkedHashSet();
+    system.getFunctionSynonyms(funcName, funcInfo, synonyms);
+    if (SessionState.getRegistry() != null) {
+      SessionState.getRegistry().getFunctionSynonyms(funcName, funcInfo, synonyms);
     }
-
     return synonyms;
   }
 
@@ -845,20 +563,20 @@
    */
   public static boolean isNumericType(PrimitiveTypeInfo typeInfo) {
     switch (typeInfo.getPrimitiveCategory()) {
-    case BYTE:
-    case SHORT:
-    case INT:
-    case LONG:
-    case DECIMAL:
-    case FLOAT:
-    case DOUBLE:
-    case STRING: // String or string equivalent is considered numeric when used in arithmetic operator.
-    case VARCHAR:
-    case CHAR:
-    case VOID: // NULL is considered numeric type for arithmetic operators.
-      return true;
-    default:
-      return false;
+      case BYTE:
+      case SHORT:
+      case INT:
+      case LONG:
+      case DECIMAL:
+      case FLOAT:
+      case DOUBLE:
+      case STRING: // String or string equivalent is considered numeric when used in arithmetic operator.
+      case VARCHAR:
+      case CHAR:
+      case VOID: // NULL is considered numeric type for arithmetic operators.
+        return true;
+      default:
+        return false;
     }
   }
 
@@ -871,14 +589,14 @@
    */
   public static boolean isExactNumericType(PrimitiveTypeInfo typeInfo) {
     switch (typeInfo.getPrimitiveCategory()) {
-    case BYTE:
-    case SHORT:
-    case INT:
-    case LONG:
-    case DECIMAL:
-      return true;
-    default:
-      return false;
+      case BYTE:
+      case SHORT:
+      case INT:
+      case LONG:
+      case DECIMAL:
+        return true;
+      default:
+        return false;
     }
   }
 
@@ -922,11 +640,11 @@
             TypeInfoUtils.getCharacterLengthForType(b));
         return TypeInfoFactory.getVarcharTypeInfo(maxLength);
       case DECIMAL:
-      return HiveDecimalUtils.getDecimalTypeForPrimitiveCategories(a, b);
+        return HiveDecimalUtils.getDecimalTypeForPrimitiveCategories(a, b);
       default:
         // Type doesn't require any qualifiers.
         return TypeInfoFactory.getPrimitiveTypeInfo(
-          PrimitiveObjectInspectorUtils.getTypeEntryFromPrimitiveCategory(typeCategory).typeName);
+            PrimitiveObjectInspectorUtils.getTypeEntryFromPrimitiveCategory(typeCategory).typeName);
     }
   }
 
@@ -1153,7 +871,6 @@
       return null;
     }
 
-    GenericUDAFEvaluator udafEvaluator = null;
     ObjectInspector args[] = new ObjectInspector[argumentOIs.size()];
     // Can't use toArray here because Java is dumb when it comes to
     // generics + arrays.
@@ -1164,6 +881,8 @@
     GenericUDAFParameterInfo paramInfo =
         new SimpleGenericUDAFParameterInfo(
             args, isDistinct, isAllColumns);
+
+    GenericUDAFEvaluator udafEvaluator;
     if (udafResolver instanceof GenericUDAFResolver2) {
       udafEvaluator =
           ((GenericUDAFResolver2) udafResolver).getEvaluator(paramInfo);
@@ -1173,24 +892,14 @@
     return udafEvaluator;
   }
 
-  @SuppressWarnings("deprecation")
   public static GenericUDAFEvaluator getGenericWindowingEvaluator(String name,
       List argumentOIs, boolean isDistinct,
       boolean isAllColumns) throws SemanticException {
-
-    WindowFunctionInfo finfo = windowFunctions.get(name.toLowerCase());
-    if (finfo == null) { return null;}
-    if ( !name.toLowerCase().equals(LEAD_FUNC_NAME) &&
-    !name.toLowerCase().equals(LAG_FUNC_NAME) ) {
-    return getGenericUDAFEvaluator(name, argumentOIs, isDistinct, isAllColumns);
-    }
-
-    // this must be lead/lag UDAF
-    ObjectInspector args[] = new ObjectInspector[argumentOIs.size()];
-    GenericUDAFResolver udafResolver = finfo.getfInfo().getGenericUDAFResolver();
-    GenericUDAFParameterInfo paramInfo = new SimpleGenericUDAFParameterInfo(
-    argumentOIs.toArray(args), isDistinct, isAllColumns);
-    return ((GenericUDAFResolver2) udafResolver).getEvaluator(paramInfo);
+    Registry registry = SessionState.getRegistry();
+    GenericUDAFEvaluator evaluator = registry == null ? null :
+        registry.getGenericWindowingEvaluator(name, argumentOIs, isDistinct, isAllColumns);
+    return evaluator != null ? evaluator :
+        system.getGenericWindowingEvaluator(name, argumentOIs, isDistinct, isAllColumns);
   }
 
   /**
@@ -1214,62 +923,8 @@
     return getMethodInternal(udfClass, mlist, exact, argumentClasses);
   }
 
-  public static void registerTemporaryGenericUDAF(String functionName,
-      GenericUDAFResolver genericUDAFResolver) {
-    registerGenericUDAF(false, functionName, genericUDAFResolver);
-  }
-
-  static void registerGenericUDAF(String functionName,
-      GenericUDAFResolver genericUDAFResolver) {
-    registerGenericUDAF(true, functionName, genericUDAFResolver);
-  }
-
-  public static void registerGenericUDAF(boolean isNative, String functionName,
-      GenericUDAFResolver genericUDAFResolver) {
-    FunctionInfo fi = new FunctionInfo(isNative, functionName.toLowerCase(), genericUDAFResolver);
-    mFunctions.put(functionName.toLowerCase(), fi);
-
-    // All aggregate functions should also be usable as window functions
-    addFunctionInfoToWindowFunctions(functionName, fi);
-
-    registerNativeStatus(fi);
-  }
-
-  public static void registerTemporaryUDAF(String functionName,
-      Class udafClass) {
-    registerUDAF(false, functionName, udafClass);
-  }
-
-  static void registerUDAF(String functionName, Class udafClass) {
-    registerUDAF(true, functionName, udafClass);
-  }
-
-  public static void registerUDAF(boolean isNative, String functionName,
-      Class udafClass) {
-    FunctionInfo fi = new FunctionInfo(isNative,
-        functionName.toLowerCase(), new GenericUDAFBridge(
-        ReflectionUtils.newInstance(udafClass, null)));
-    mFunctions.put(functionName.toLowerCase(), fi);
-
-    // All aggregate functions should also be usable as window functions
-    addFunctionInfoToWindowFunctions(functionName, fi);
-
-    registerNativeStatus(fi);
-  }
-
-  public static void unregisterTemporaryUDF(String functionName) throws HiveException {
-    FunctionInfo fi = mFunctions.get(functionName.toLowerCase());
-    if (fi != null) {
-      if (!fi.isNative()) {
-        mFunctions.remove(functionName.toLowerCase());
-      } else {
-        throw new HiveException("Function " + functionName
-            + " is hive native, it can't be dropped");
-      }
-    }
-  }
-
-  public static GenericUDAFResolver getGenericUDAFResolver(String functionName) throws SemanticException {
+  public static GenericUDAFResolver getGenericUDAFResolver(String functionName)
+      throws SemanticException {
     if (LOG.isDebugEnabled()) {
       LOG.debug("Looking up GenericUDAF: " + functionName);
     }
@@ -1466,8 +1121,8 @@
       }
       if (LOG.isDebugEnabled()) {
         LOG.debug("Method " + (match ? "did" : "didn't") + " match: passed = "
-                  + argumentsPassed + " accepted = " + argumentsAccepted +
-                  " method = " + m);
+            + argumentsPassed + " accepted = " + argumentsAccepted +
+            " method = " + m);
       }
       if (match) {
         // Always choose the function with least implicit conversions.
@@ -1569,7 +1224,11 @@
    * @throws SemanticException
    */
   public static GenericUDF getGenericUDFForIndex() {
-    return FunctionRegistry.getFunctionInfo(mFunctions, "index").getGenericUDF();
+    try {
+      return FunctionRegistry.getFunctionInfo("index").getGenericUDF();
+    } catch (SemanticException e) {
+      throw new RuntimeException("hive operator -- never be thrown", e);
+    }
   }
 
   /**
@@ -1577,7 +1236,11 @@
    * @throws SemanticException
    */
   public static GenericUDF getGenericUDFForAnd() {
-    return FunctionRegistry.getFunctionInfo(mFunctions, "and").getGenericUDF();
+    try {
+      return FunctionRegistry.getFunctionInfo("and").getGenericUDF();
+    } catch (SemanticException e) {
+      throw new RuntimeException("hive operator -- never be thrown", e);
+    }
   }
 
   /**
@@ -1588,7 +1251,7 @@
       return null;
     }
 
-    GenericUDF clonedUDF = null;
+    GenericUDF clonedUDF;
     if (genericUDF instanceof GenericUDFBridge) {
       GenericUDFBridge bridge = (GenericUDFBridge) genericUDF;
       clonedUDF = new GenericUDFBridge(bridge.getUdfName(), bridge.isOperator(),
@@ -1598,8 +1261,7 @@
       clonedUDF = new GenericUDFMacro(bridge.getMacroName(), bridge.getBody(),
           bridge.getColNames(), bridge.getColTypes());
     } else {
-      clonedUDF = ReflectionUtils
-          .newInstance(genericUDF.getClass(), null);
+      clonedUDF = ReflectionUtils.newInstance(genericUDF.getClass(), null);
     }
 
     if (clonedUDF != null) {
@@ -1637,8 +1299,7 @@
     if (null == genericUDTF) {
       return null;
     }
-    return ReflectionUtils.newInstance(genericUDTF.getClass(),
-        null);
+    return ReflectionUtils.newInstance(genericUDTF.getClass(), null);
   }
 
   /**
@@ -1743,8 +1404,7 @@
    * Returns whether the exprNodeDesc is a node of "positive".
    */
   public static boolean isOpPositive(ExprNodeDesc desc) {
-    Class udfClass = getGenericUDFClassFromExprDesc(desc);
-    return GenericUDFOPPositive.class == udfClass;
+    return GenericUDFOPPositive.class == getGenericUDFClassFromExprDesc(desc);
   }
 
   /**
@@ -1788,103 +1448,41 @@
    * @return true if udfClass's type was recognized (so registration
    * succeeded); false otherwise
    */
-  public static boolean registerTemporaryFunction(
-    String functionName, Class udfClass) {
+  public static FunctionInfo registerTemporaryUDF(
+      String functionName, Class udfClass, FunctionResource... resources) {
+    return SessionState.getRegistryForWrite().registerFunction(
+        functionName, udfClass, resources);
+  }
 
-    UDFClassType udfClassType = FunctionUtils.getUDFClassType(udfClass);
-    switch (udfClassType) {
-    case UDF:
-      FunctionRegistry.registerTemporaryUDF(
-        functionName, (Class) udfClass, false);
-      break;
-    case GENERIC_UDF:
-      FunctionRegistry.registerTemporaryGenericUDF(
-        functionName, (Class) udfClass);
-      break;
-    case GENERIC_UDTF:
-      FunctionRegistry.registerTemporaryGenericUDTF(
-        functionName, (Class) udfClass);
-      break;
-    case UDAF:
-      FunctionRegistry.registerTemporaryUDAF(
-        functionName, (Class) udfClass);
-      break;
-    case GENERIC_UDAF_RESOLVER:
-      FunctionRegistry.registerTemporaryGenericUDAF(
-        functionName, (GenericUDAFResolver)
-        ReflectionUtils.newInstance(udfClass, null));
-      break;
-    case TABLE_FUNCTION_RESOLVER:
-      FunctionRegistry.registerTableFunction(
-        functionName, (Class)udfClass);
-      break;
-    default:
-      return false;
+  public static void unregisterTemporaryUDF(String functionName) throws HiveException {
+    if (SessionState.getRegistry() != null) {
+      SessionState.getRegistry().unregisterFunction(functionName);
     }
-    return true;
-
   }
 
   /**
-   * Registers thae appropriate kind of temporary function based on a class's
+   * Registers the appropriate kind of temporary function based on a class's
    * type.
    *
    * @param macroName name under which to register the macro
    *
    * @param body the expression which the macro evaluates to
-   *
-   * @param colNames the names of the arguments to the macro
-   *
-   * @param colTypes the types of the arguments to the macro
+   *@param colNames the names of the arguments to the macro
+   *@param colTypes the types of the arguments to the macro
    */
   public static void registerTemporaryMacro(
-    String macroName, ExprNodeDesc body,
-    List colNames, List colTypes) {
+      String macroName, ExprNodeDesc body, List colNames, List colTypes) {
+    SessionState.getRegistryForWrite().registerMacro(macroName, body, colNames, colTypes);
+  }
 
-    FunctionInfo fI = new FunctionInfo(false, macroName,
-        new GenericUDFMacro(macroName, body, colNames, colTypes));
-    mFunctions.put(macroName.toLowerCase(), fI);
-    registerNativeStatus(fI);
+  public static FunctionInfo registerPermanentFunction(String functionName,
+      String className, boolean registerToSession, FunctionResource[] resources) {
+    return system.registerPermanentFunction(functionName, className, registerToSession, resources);
   }
 
-  /**
-   * Registers Hive functions from a plugin jar, using metadata from
-   * the jar's META-INF/class-info.xml.
-   *
-   * @param jarLocation URL for reading jar file
-   *
-   * @param classLoader classloader to use for loading function classes
-   */
-  public static void registerFunctionsFromPluginJar(
-    URL jarLocation,
-    ClassLoader classLoader) throws Exception {
-
-    URL url = new URL("jar:" + jarLocation + "!/META-INF/class-info.xml");
-    InputStream inputStream = null;
-    try {
-      inputStream = url.openStream();
-      DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
-      DocumentBuilder docBuilder = dbf.newDocumentBuilder();
-      Document doc = docBuilder.parse(inputStream);
-      Element root = doc.getDocumentElement();
-      if (!root.getTagName().equals("ClassList")) {
-        return;
-      }
-      NodeList children = root.getElementsByTagName("Class");
-      for (int i = 0; i < children.getLength(); ++i) {
-        Element child = (Element) children.item(i);
-        String javaName = child.getAttribute("javaname");
-        String sqlName = child.getAttribute("sqlname");
-        Class udfClass = Class.forName(javaName, true, classLoader);
-        boolean registered = registerTemporaryFunction(sqlName, udfClass);
-        if (!registered) {
-          throw new RuntimeException(
-            "Class " + udfClass + " is not a Hive function implementation");
-        }
-      }
-    } finally {
-      IOUtils.closeStream(inputStream);
-    }
+  public static void unregisterPermanentFunction(String functionName) throws HiveException {
+    system.unregisterFunction(functionName);
+    unregisterTemporaryUDF(functionName);
   }
 
   private FunctionRegistry() {
@@ -1894,43 +1492,7 @@
 
   //---------PTF functions------------
 
-  public static void registerWindowFunction(String name, GenericUDAFResolver wFn)
-  {
-    registerWindowFunction(name, wFn, true);
-  }
-
   /**
-   * Typically a WindowFunction is the same as a UDAF. The only exceptions are Lead & Lag UDAFs. These
-   * are not registered as regular UDAFs because
-   * - we plan to support Lead & Lag as UDFs (usable only within argument expressions
-   *   of UDAFs when windowing is involved). Since mFunctions holds both UDFs and UDAFs we cannot
-   *   add both FunctionInfos to mFunctions.
-   * We choose to only register UDFs in mFunctions. The implication of this is that Lead/Lag UDAFs
-   * are only usable when windowing is involved.
-   *
-   * @param name
-   * @param wFn
-   * @param registerAsUDAF
-   */
-  public static void registerWindowFunction(String name, GenericUDAFResolver wFn, boolean registerAsUDAF)
-  {
-    FunctionInfo fInfo = null;
-    if (registerAsUDAF) {
-      // Just register the function normally, will also get added to window functions.
-      registerGenericUDAF(true, name, wFn);
-    }
-    else {
-      name = name.toLowerCase();
-      fInfo = new FunctionInfo(true, name, wFn);
-      addFunctionInfoToWindowFunctions(name, fInfo);
-    }
-  }
-
-  public static WindowFunctionInfo getWindowFunctionInfo(String functionName) {
-    return getFunctionInfo(windowFunctions, functionName);
-  }
-
-  /**
    * Both UDF and UDAF functions can imply order for analytical functions
    *
    * @param functionName
@@ -1942,13 +1504,11 @@
   public static boolean impliesOrder(String functionName) throws SemanticException {
 
     FunctionInfo info = getFunctionInfo(functionName);
-    if (info != null) {
-      if (info.isGenericUDF()) {
-        UDFType type =
-            AnnotationUtils.getAnnotation(info.getGenericUDF().getClass(), UDFType.class);
-        if (type != null) {
-          return type.impliesOrder();
-        }
+    if (info != null && info.isGenericUDF()) {
+      UDFType type =
+          AnnotationUtils.getAnnotation(info.getGenericUDF().getClass(), UDFType.class);
+      if (type != null) {
+        return type.impliesOrder();
       }
     }
     WindowFunctionInfo windowInfo = getWindowFunctionInfo(functionName);
@@ -1958,30 +1518,23 @@
     return false;
   }
 
-  static private void addFunctionInfoToWindowFunctions(String functionName,
-      FunctionInfo functionInfo) {
-    // Assumes that the caller has already verified that functionInfo is for an aggregate function
-    WindowFunctionInfo wInfo = new WindowFunctionInfo(functionInfo);
-    windowFunctions.put(functionName.toLowerCase(), wInfo);
-  }
-
-  public static boolean isTableFunction(String name) throws SemanticException
-  {
-    FunctionInfo tFInfo = getFunctionInfo(name);
+  public static boolean isTableFunction(String functionName)
+      throws SemanticException {
+    FunctionInfo tFInfo = getFunctionInfo(functionName);
     return tFInfo != null && !tFInfo.isInternalTableFunction() && tFInfo.isTableFunction();
   }
 
-  public static TableFunctionResolver getTableFunctionResolver(String name) throws SemanticException
-  {
-    FunctionInfo tfInfo = getFunctionInfo(name);
-    if(tfInfo.isTableFunction()) {
+  public static TableFunctionResolver getTableFunctionResolver(String functionName)
+      throws SemanticException {
+    FunctionInfo tfInfo = getFunctionInfo(functionName);
+    if (tfInfo != null && tfInfo.isTableFunction()) {
       return (TableFunctionResolver) ReflectionUtils.newInstance(tfInfo.getFunctionClass(), null);
     }
     return null;
   }
 
-  public static TableFunctionResolver getWindowingTableFunction() throws SemanticException
-  {
+  public static TableFunctionResolver getWindowingTableFunction()
+      throws SemanticException {
     return getTableFunctionResolver(WINDOWING_TABLE_FUNCTION);
   }
 
@@ -1993,13 +1546,6 @@
         fnName.equals(NOOP_STREAMING_TABLE_FUNCTION);
   }
 
-  public static void registerTableFunction(String name, Class tFnCls)
-  {
-    FunctionInfo tInfo = new FunctionInfo(name, tFnCls);
-    mFunctions.put(name.toLowerCase(), tInfo);
-    registerNativeStatus(tInfo);
-  }
-
   /**
    * Use this to check if function is ranking function
    *
@@ -2025,78 +1571,23 @@
 
   /**
    * @param fnExpr Function expression.
-   * @return True iff the fnExpr represents a hive built-in function.
+   * @return True iff the fnExpr represents a hive built-in function (native, non-permanent)
    */
-  public static boolean isNativeFuncExpr(ExprNodeGenericFuncDesc fnExpr) {
-    Class udfClass = null;
-
-    GenericUDF udf = fnExpr.getGenericUDF();
-    if (udf instanceof GenericUDFBridge) {
-      udfClass = ((GenericUDFBridge) udf).getUdfClass();
-    } else {
-      udfClass = udf.getClass();
+  public static boolean isBuiltInFuncExpr(ExprNodeGenericFuncDesc fnExpr) {
+    Class udfClass = FunctionRegistry.getGenericUDFClassFromExprDesc(fnExpr);
+    if (udfClass != null) {
+      return system.isBuiltInFunc(udfClass);
     }
-
-    return nativeUdfs.contains(udfClass);
+    return false;
   }
 
-  private static void registerNativeStatus(FunctionInfo fi) {
-    if (!fi.isNative()) {
-      return;
-    }
-    nativeUdfs.add(fi.getFunctionClass());
-  }
-
   /**
    * Setup blocked flag for all builtin UDFs as per udf whitelist and blacklist
-   * @param whiteList
-   * @param blackList
+   * @param whiteListStr
+   * @param blackListStr
    */
   public static void setupPermissionsForBuiltinUDFs(String whiteListStr,
       String blackListStr) {
-    List whiteList = Lists.newArrayList(
-        Splitter.on(",").trimResults().omitEmptyStrings().split(whiteListStr));
-    List blackList = Lists.newArrayList(
-        Splitter.on(",").trimResults().omitEmptyStrings().split(blackListStr));
-
-    for ( Entry funcEntry : mFunctions.entrySet()) {
-      funcEntry.getValue().setBlockedFunction(
-          isUdfBlocked(funcEntry.getKey(), whiteList, blackList));
-    }
+    system.setupPermissionsForUDFs(whiteListStr, blackListStr);
   }
-
-  /**
-   * Check if the function belongs to whitelist or blacklist
-   * @param functionName
-   * @param whiteList
-   * @param blackList
-   * @return true if the given udf is to be blocked
-   */
-  private static boolean isUdfBlocked(String functionName,
-      List whiteList, List blackList) {
-    boolean inWhiteList = false;
-    boolean inBlackList = false;
-
-    if (whiteList.isEmpty()) {
-      // if whitelist is empty, all udfs are allowed
-      inWhiteList = true;
-    } else {
-      for (String allowedFunction : whiteList) {
-        if (functionName.equalsIgnoreCase(allowedFunction)) {
-          inWhiteList = true;
-          break;
-        }
-      }
-    }
-
-    for (String blockedFunction : blackList) {
-      if (functionName.equalsIgnoreCase(blockedFunction)) {
-        inBlackList = true;
-        break;
-      }
-    }
-
-    // blacklist setting takes presendence on whitelist
-    return !inWhiteList || inBlackList;
-  }
-}
+}
\ No newline at end of file
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java	(working copy)
@@ -23,18 +23,19 @@
 import java.io.IOException;
 import java.util.List;
 
+import com.google.common.collect.HashMultimap;
+import com.google.common.collect.Multimap;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.hive.common.JavaUtils;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.metastore.api.Function;
 import org.apache.hadoop.hive.metastore.api.PrincipalType;
 import org.apache.hadoop.hive.metastore.api.ResourceType;
 import org.apache.hadoop.hive.metastore.api.ResourceUri;
+import org.apache.hadoop.hive.ql.exec.FunctionInfo.FunctionResource;
 import org.apache.hadoop.hive.ql.DriverContext;
 import org.apache.hadoop.hive.ql.QueryPlan;
-import org.apache.hadoop.hive.ql.exec.FunctionUtils.UDFClassType;
 import org.apache.hadoop.hive.ql.metadata.Hive;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
 import org.apache.hadoop.hive.ql.plan.CreateFunctionDesc;
@@ -98,6 +99,16 @@
       }
     }
 
+    if (work.getReloadFunctionDesc() != null) {
+      try {
+        Hive.reloadFunctions();
+      } catch (Exception e) {
+        setException(e);
+        LOG.error(stringifyException(e));
+        return 1;
+      }
+    }
+
     CreateMacroDesc createMacroDesc = work.getCreateMacroDesc();
     if (createMacroDesc != null) {
       return createMacro(createMacroDesc);
@@ -110,6 +121,7 @@
     return 0;
   }
 
+  // todo authorization
   private int createPermanentFunction(Hive db, CreateFunctionDesc createFunctionDesc)
       throws HiveException, IOException {
     String[] qualifiedNameParts = FunctionUtils.getQualifiedFunctionNameParts(
@@ -118,64 +130,45 @@
     String funcName = qualifiedNameParts[1];
     String registeredName = FunctionUtils.qualifyFunctionName(funcName, dbName);
     String className = createFunctionDesc.getClassName();
-    boolean addedToRegistry = false;
 
-    try {
-      // For permanent functions, check for any resources from local filesystem.
-      checkLocalFunctionResources(db, createFunctionDesc.getResources());
+    List resources = createFunctionDesc.getResources();
 
-      // Add any required resources
-      addFunctionResources(createFunctionDesc.getResources());
+    // For permanent functions, check for any resources from local filesystem.
+    checkLocalFunctionResources(db, createFunctionDesc.getResources());
 
-      // UDF class should exist
-      Class udfClass = getUdfClass(createFunctionDesc);
-      if (FunctionUtils.getUDFClassType(udfClass) == UDFClassType.UNKNOWN) {
-        console.printError("FAILED: Class " + createFunctionDesc.getClassName()
-            + " does not implement UDF, GenericUDF, or UDAF");
-        return 1;
-      }
-
-      // TODO: There should be a registerPermanentFunction()
-      addedToRegistry = FunctionRegistry.registerTemporaryFunction(registeredName, udfClass);
-      if (!addedToRegistry) {
-        console.printError("Failed to register " + registeredName
-            + " using class " + createFunctionDesc.getClassName());
-        return 1;
-      }
-
-      // Add to metastore
-      Function func = new Function(
-          funcName,
-          dbName,
-          className,
-          SessionState.get().getUserName(),
-          PrincipalType.USER,
-          (int) (System.currentTimeMillis() / 1000),
-          org.apache.hadoop.hive.metastore.api.FunctionType.JAVA,
-          createFunctionDesc.getResources()
-          );
-      db.createFunction(func);
-      return 0;
-    } catch (ClassNotFoundException e) {
-      console.printError("FAILED: Class " + createFunctionDesc.getClassName() + " not found");
-      LOG.info("create function: " + StringUtils.stringifyException(e));
-      if (addedToRegistry) {
-        FunctionRegistry.unregisterTemporaryUDF(registeredName);
-      }
+    FunctionInfo registered = FunctionRegistry.registerPermanentFunction(
+        registeredName, className, true, toFunctionResource(resources));
+    if (registered == null) {
+      console.printError("Failed to register " + registeredName
+          + " using class " + createFunctionDesc.getClassName());
       return 1;
     }
+
+    // Add to metastore
+    Function func = new Function(
+        funcName,
+        dbName,
+        className,
+        SessionState.get().getUserName(),
+        PrincipalType.USER,
+        (int) (System.currentTimeMillis() / 1000),
+        org.apache.hadoop.hive.metastore.api.FunctionType.JAVA,
+        resources
+    );
+    db.createFunction(func);
+    return 0;
   }
 
   private int createTemporaryFunction(CreateFunctionDesc createFunctionDesc) {
     try {
       // Add any required resources
-      addFunctionResources(createFunctionDesc.getResources());
+      FunctionResource[] resources = toFunctionResource(createFunctionDesc.getResources());
+      addFunctionResources(resources);
 
       Class udfClass = getUdfClass(createFunctionDesc);
-      boolean registered = FunctionRegistry.registerTemporaryFunction(
-        createFunctionDesc.getFunctionName(),
-        udfClass);
-      if (registered) {
+      FunctionInfo registered = FunctionRegistry.registerTemporaryUDF(
+          createFunctionDesc.getFunctionName(), udfClass, resources);
+      if (registered != null) {
         return 0;
       }
       console.printError("FAILED: Class " + createFunctionDesc.getClassName()
@@ -198,14 +191,14 @@
       createMacroDesc.getMacroName(),
       createMacroDesc.getBody(),
       createMacroDesc.getColNames(),
-      createMacroDesc.getColTypes());
+      createMacroDesc.getColTypes()
+    );
     return 0;
   }
 
   private int dropMacro(DropMacroDesc dropMacroDesc) {
     try {
-      FunctionRegistry.unregisterTemporaryUDF(dropMacroDesc
-          .getMacroName());
+      FunctionRegistry.unregisterTemporaryUDF(dropMacroDesc.getMacroName());
       return 0;
     } catch (HiveException e) {
       LOG.info("drop macro: " + StringUtils.stringifyException(e));
@@ -213,6 +206,7 @@
     }
   }
 
+  // todo authorization
   private int dropPermanentFunction(Hive db, DropFunctionDesc dropFunctionDesc) {
     try {
       String[] qualifiedNameParts = FunctionUtils.getQualifiedFunctionNameParts(
@@ -221,8 +215,7 @@
       String funcName = qualifiedNameParts[1];
 
       String registeredName = FunctionUtils.qualifyFunctionName(funcName, dbName);
-      // TODO: there should be a unregisterPermanentUDF()
-      FunctionRegistry.unregisterTemporaryUDF(registeredName);
+      FunctionRegistry.unregisterPermanentFunction(registeredName);
       db.dropFunction(dbName, funcName);
 
       return 0;
@@ -235,8 +228,7 @@
 
   private int dropTemporaryFunction(DropFunctionDesc dropFunctionDesc) {
     try {
-      FunctionRegistry.unregisterTemporaryUDF(dropFunctionDesc
-          .getFunctionName());
+      FunctionRegistry.unregisterTemporaryUDF(dropFunctionDesc.getFunctionName());
       return 0;
     } catch (HiveException e) {
       LOG.info("drop function: " + StringUtils.stringifyException(e));
@@ -275,6 +267,19 @@
     }
   }
 
+  public static FunctionResource[] toFunctionResource(List resources)
+      throws HiveException {
+    if (resources == null) {
+      return null;
+    }
+    FunctionResource[] converted = new FunctionResource[resources.size()];
+    for (int i = 0; i < converted.length; i++) {
+      ResourceUri resource = resources.get(i);
+      SessionState.ResourceType type = getResourceType(resource.getResourceType());
+      converted[i] = new FunctionResource(type, resource.getUri());
+    }
+    return converted;
+  }
 
   private static SessionState.ResourceType getResourceType(ResourceType rt) throws HiveException {
     switch (rt) {
@@ -289,15 +294,15 @@
     }
   }
 
-  public static void addFunctionResources(List resources) throws HiveException {
+  public static void addFunctionResources(FunctionResource[] resources) throws HiveException {
     if (resources != null) {
-      for (ResourceUri res : resources) {
-        String addedResource =
-            SessionState.get().add_resource(getResourceType(res.getResourceType()), res.getUri());
-        if (addedResource == null) {
-          throw new HiveException("Unable to load " + res.getResourceType() + " " + res.getUri());
-        }
+      Multimap mappings = HashMultimap.create();
+      for (FunctionResource res : resources) {
+        mappings.put(res.getResourceType(), res.getResourceURI());
       }
+      for (SessionState.ResourceType type : mappings.keys()) {
+        SessionState.get().add_resources(type, mappings.get(type));
+      }
     }
   }
 
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java	(working copy)
@@ -19,6 +19,7 @@
 package org.apache.hadoop.hive.ql.exec;
 
 import java.io.Serializable;
+import java.util.Collection;
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ConcurrentMap;
@@ -223,7 +224,12 @@
     int ret = 0;
 
     try {
-      List partitions = getPartitionsList();
+      Collection partitions = null;
+      if (work.getPrunedPartitionList() == null) {
+        partitions = getPartitionsList();
+      } else {
+        partitions = work.getPrunedPartitionList().getPartitions();
+      }
 
       // non-partitioned table
       if (partitions == null) {
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java	(working copy)
@@ -2100,8 +2100,8 @@
     }
     ClassLoader sessionCL = state.getConf().getClassLoader();
     if (sessionCL != null) {
-      if (LOG.isDebugEnabled()) {
-        LOG.debug("Use session specified class loader");
+      if (LOG.isTraceEnabled()) {
+        LOG.trace("Use session specified class loader");  //it's normal case
       }
       return sessionCL;
     }
@@ -2111,6 +2111,17 @@
     return JavaUtils.getClassLoader();
   }
 
+  public static void restoreSessionSpecifiedClassLoader(ClassLoader prev) {
+    SessionState state = SessionState.get();
+    if (state != null && state.getConf() != null) {
+      ClassLoader current = state.getConf().getClassLoader();
+      if (current != prev && JavaUtils.closeClassLoadersTo(current, prev)) {
+        Thread.currentThread().setContextClassLoader(prev);
+        state.getConf().setClassLoader(prev);
+      }
+    }
+  }
+
   /**
    * Create a URL from a string representing a path to a local file.
    * The path string can be just a path, or can start with file:/, file:///
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/WindowFunctionInfo.java	(working copy)
@@ -22,23 +22,20 @@
 import org.apache.hive.common.util.AnnotationUtils;
 
 @SuppressWarnings("deprecation")
-public class WindowFunctionInfo implements CommonFunctionInfo {
-  boolean supportsWindow = true;
-  boolean pivotResult = false;
-  boolean impliesOrder = false;
-  FunctionInfo fInfo;
+public class WindowFunctionInfo extends FunctionInfo {
 
-  WindowFunctionInfo(FunctionInfo fInfo) {
-    assert fInfo.isGenericUDAF();
-    this.fInfo = fInfo;
-    Class wfnCls = fInfo.getGenericUDAFResolver().getClass();
+  private final boolean supportsWindow;
+  private final boolean pivotResult;
+  private final boolean impliesOrder;
+
+  public WindowFunctionInfo(boolean isNative, String functionName,
+      GenericUDAFResolver resolver, FunctionResource[] resources) {
+    super(isNative, functionName, resolver, resources);
     WindowFunctionDescription def =
-          AnnotationUtils.getAnnotation(wfnCls, WindowFunctionDescription.class);
-    if ( def != null) {
-      supportsWindow = def.supportsWindow();
-      pivotResult = def.pivotResult();
-      impliesOrder = def.impliesOrder();
-    }
+        AnnotationUtils.getAnnotation(resolver.getClass(), WindowFunctionDescription.class);
+    supportsWindow = def == null ? true : def.supportsWindow();
+    pivotResult = def == null ? false : def.pivotResult();
+    impliesOrder = def == null ? false : def.impliesOrder();
   }
 
   public boolean isSupportsWindow() {
@@ -52,12 +49,4 @@
   public boolean isImpliesOrder() {
     return impliesOrder;
   }
-  public FunctionInfo getfInfo() {
-    return fInfo;
-  }
-
-  @Override
-  public Class getFunctionClass() {
-    return getfInfo().getFunctionClass();
-  }
 }
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java	(working copy)
@@ -395,6 +395,7 @@
       if (pwd != null) {
         HiveConf.setVar(job, HiveConf.ConfVars.METASTOREPWD, "HIVE");
       }
+      LOG.error(job.get("mapreduce.framework.name"));
       JobClient jc = new JobClient(job);
       // make this client wait if job tracker is not behaving well.
       Throttle.checkJobTracker(job, LOG);
@@ -665,7 +666,7 @@
     }
 
     if(UserGroupInformation.isSecurityEnabled()){
-      String hadoopAuthToken = UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION;
+      String hadoopAuthToken = System.getenv(UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION);
       if(hadoopAuthToken != null){
         conf.set("mapreduce.job.credentials.binary", hadoopAuthToken);
       }
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java	(working copy)
@@ -22,7 +22,6 @@
 import java.util.Iterator;
 import java.util.NoSuchElementException;
 
-import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hive.ql.io.HiveKey;
 import org.apache.hadoop.io.BytesWritable;
 import org.apache.hadoop.mapred.OutputCollector;
@@ -49,9 +48,9 @@
   private final HiveKVResultCache lastRecordOutput;
   private boolean iteratorAlreadyCreated = false;
 
-  public HiveBaseFunctionResultList(Configuration conf, Iterator inputIterator) {
+  public HiveBaseFunctionResultList(Iterator inputIterator) {
     this.inputIterator = inputIterator;
-    this.lastRecordOutput = new HiveKVResultCache(conf);
+    this.lastRecordOutput = new HiveKVResultCache();
   }
 
   @Override
@@ -87,8 +86,6 @@
         return true;
       }
 
-      lastRecordOutput.clear();
-
       // Process the records in the input iterator until
       //  - new output records are available for serving downstream operator,
       //  - input records are exhausted or
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveKVResultCache.java	(working copy)
@@ -17,141 +17,251 @@
  */
 package org.apache.hadoop.hive.ql.exec.spark;
 
-import java.util.ArrayList;
-import java.util.List;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
 
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.hive.conf.HiveConf;
-import org.apache.hadoop.hive.ql.exec.persistence.RowContainer;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.hive.common.ObjectPair;
 import org.apache.hadoop.hive.ql.io.HiveKey;
-import org.apache.hadoop.hive.ql.metadata.HiveException;
-import org.apache.hadoop.hive.ql.plan.PlanUtils;
-import org.apache.hadoop.hive.ql.plan.TableDesc;
-import org.apache.hadoop.hive.serde.serdeConstants;
-import org.apache.hadoop.hive.serde2.SerDe;
-import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
-import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils;
-import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.ObjectInspectorCopyOption;
 import org.apache.hadoop.io.BytesWritable;
-import org.apache.hadoop.mapred.Reporter;
 
 import scala.Tuple2;
 
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 
 /**
- * Wrapper around {@link org.apache.hadoop.hive.ql.exec.persistence.RowContainer}
- *
- * This class is thread safe.
+ * A cache with fixed buffer. If the buffer is full, new entries will
+ * be written to disk. This class is thread safe since multiple threads
+ * could access it (doesn't have to be concurrently), for example,
+ * the StreamThread in ScriptOperator.
  */
-@SuppressWarnings({"deprecation", "unchecked", "rawtypes"})
-public class HiveKVResultCache {
+@SuppressWarnings("unchecked")
+class HiveKVResultCache {
+  private static final Log LOG = LogFactory.getLog(HiveKVResultCache.class);
 
-  public static final int IN_MEMORY_CACHE_SIZE = 512;
-  private static final String COL_NAMES = "key,value";
-  private static final String COL_TYPES =
-      serdeConstants.BINARY_TYPE_NAME + ":" + serdeConstants.BINARY_TYPE_NAME;
+  @VisibleForTesting
+  static final int IN_MEMORY_NUM_ROWS = 1024;
 
-  // Used to cache rows added while container is iterated.
-  private RowContainer backupContainer;
+  private ObjectPair[] writeBuffer;
+  private ObjectPair[] readBuffer;
 
-  private RowContainer container;
-  private Configuration conf;
-  private int cursor = 0;
+  private File parentFile;
+  private File tmpFile;
 
-  public HiveKVResultCache(Configuration conf) {
-    container = initRowContainer(conf);
-    this.conf = conf;
+  private int readCursor = 0;
+  private int writeCursor = 0;
+
+  // Indicate if the read buffer has data, for example,
+  // when in reading, data on disk could be pull in
+  private boolean readBufferUsed = false;
+  private int rowsInReadBuffer = 0;
+
+  private Input input;
+  private Output output;
+
+  public HiveKVResultCache() {
+    writeBuffer = new ObjectPair[IN_MEMORY_NUM_ROWS];
+    readBuffer = new ObjectPair[IN_MEMORY_NUM_ROWS];
+    for (int i = 0; i < IN_MEMORY_NUM_ROWS; i++) {
+      writeBuffer[i] = new ObjectPair();
+      readBuffer[i] = new ObjectPair();
+    }
   }
 
-  private static RowContainer initRowContainer(Configuration conf) {
-    RowContainer container;
-    try {
-      container = new RowContainer(IN_MEMORY_CACHE_SIZE, conf, Reporter.NULL);
+  private void switchBufferAndResetCursor() {
+    ObjectPair[] tmp = readBuffer;
+    rowsInReadBuffer = writeCursor;
+    readBuffer = writeBuffer;
+    readBufferUsed = true;
+    readCursor = 0;
+    writeBuffer = tmp;
+    writeCursor = 0;
+  }
 
-      String fileFormat = HiveConf.getVar(conf, HiveConf.ConfVars.HIVEQUERYRESULTFILEFORMAT);
-      TableDesc tableDesc =
-          PlanUtils.getDefaultQueryOutputTableDesc(COL_NAMES, COL_TYPES, fileFormat);
+  private void setupOutput() throws IOException {
+    if (parentFile == null) {
+      while (true) {
+        parentFile = File.createTempFile("hive-resultcache", "");
+        if (parentFile.delete() && parentFile.mkdir()) {
+          parentFile.deleteOnExit();
+          break;
+        }
+        if (LOG.isDebugEnabled()) {
+          LOG.debug("Retry creating tmp result-cache directory...");
+        }
+      }
+    }
 
-      SerDe serDe = (SerDe) tableDesc.getDeserializer();
-      ObjectInspector oi = ObjectInspectorUtils.getStandardObjectInspector(
-          serDe.getObjectInspector(), ObjectInspectorCopyOption.WRITABLE);
+    if (tmpFile == null || input != null) {
+      tmpFile = File.createTempFile("ResultCache", ".tmp", parentFile);
+      LOG.info("ResultCache created temp file " + tmpFile.getAbsolutePath());
+      tmpFile.deleteOnExit();
+    }
 
-      container.setSerDe(serDe, oi);
-      container.setTableDesc(tableDesc);
-    } catch (Exception ex) {
-      throw new RuntimeException("Failed to create RowContainer", ex);
+    FileOutputStream fos = null;
+    try {
+      fos = new FileOutputStream(tmpFile);
+      output = new Output(fos);
+    } finally {
+      if (output == null && fos != null) {
+        fos.close();
+      }
     }
-    return container;
   }
 
-  public void add(HiveKey key, BytesWritable value) {
-    byte[] hiveKeyBytes = KryoSerializer.serialize(key);
-    BytesWritable wrappedHiveKey = new BytesWritable(hiveKeyBytes);
-    List row = new ArrayList(2);
-    row.add(wrappedHiveKey);
-    row.add(value);
+  private BytesWritable readValue(Input input) {
+    return new BytesWritable(input.readBytes(input.readInt()));
+  }
 
-    synchronized (this) {
-      try {
-        if (cursor == 0) {
-          container.addRow(row);
-        } else {
-          if (backupContainer == null) {
-            backupContainer = initRowContainer(conf);
+  private void writeValue(Output output, BytesWritable bytesWritable) {
+    int size = bytesWritable.getLength();
+    output.writeInt(size);
+    output.writeBytes(bytesWritable.getBytes(), 0, size);
+  }
+
+  private HiveKey readHiveKey(Input input) {
+    HiveKey hiveKey = new HiveKey(
+      input.readBytes(input.readInt()), input.readInt());
+    hiveKey.setDistKeyLength(input.readInt());
+    return hiveKey;
+  }
+
+  private void writeHiveKey(Output output, HiveKey hiveKey) {
+    int size = hiveKey.getLength();
+    output.writeInt(size);
+    output.writeBytes(hiveKey.getBytes(), 0, size);
+    output.writeInt(hiveKey.hashCode());
+    output.writeInt(hiveKey.getDistKeyLength());
+  }
+
+  public synchronized void add(HiveKey key, BytesWritable value) {
+    if (writeCursor >= IN_MEMORY_NUM_ROWS) { // Write buffer is full
+      if (!readBufferUsed) { // Read buffer isn't used, switch buffer
+        switchBufferAndResetCursor();
+      } else {
+        // Need to spill from write buffer to disk
+        try {
+          if (output == null) {
+            setupOutput();
           }
-          backupContainer.addRow(row);
+          for (int i = 0; i < IN_MEMORY_NUM_ROWS; i++) {
+            ObjectPair pair = writeBuffer[i];
+            writeHiveKey(output, pair.getFirst());
+            writeValue(output, pair.getSecond());
+            pair.setFirst(null);
+            pair.setSecond(null);
+          }
+          writeCursor = 0;
+        } catch (Exception e) {
+          clear(); // Clean up the cache
+          throw new RuntimeException("Failed to spill rows to disk", e);
         }
-      } catch (HiveException ex) {
-        throw new RuntimeException("Failed to add KV pair to RowContainer", ex);
       }
     }
+    ObjectPair pair = writeBuffer[writeCursor++];
+    pair.setFirst(key);
+    pair.setSecond(value);
   }
 
   public synchronized void clear() {
-    if (cursor == 0) {
-      return;
+    writeCursor = readCursor = rowsInReadBuffer = 0;
+    readBufferUsed = false;
+
+    if (parentFile != null) {
+      if (input != null) {
+        try {
+          input.close();
+        } catch (Throwable ignored) {
+        }
+        input = null;
+      }
+      if (output != null) {
+        try {
+          output.close();
+        } catch (Throwable ignored) {
+        }
+        output = null;
+      }
+      try {
+        FileUtil.fullyDelete(parentFile);
+      } catch (Throwable ignored) {
+      }
+      parentFile = null;
+      tmpFile = null;
     }
-    try {
-      container.clearRows();
-    } catch (HiveException ex) {
-      throw new RuntimeException("Failed to clear rows in RowContainer", ex);
-    }
-    cursor = 0;
   }
 
   public synchronized boolean hasNext() {
-    if (container.rowCount() > 0 && cursor < container.rowCount()) {
-      return true;
-    }
-    if (backupContainer == null
-        || backupContainer.rowCount() == 0) {
-      return false;
-    }
-    clear();
-    // Switch containers
-    RowContainer tmp = container;
-    container = backupContainer;
-    backupContainer = tmp;
-    return true;
+    return readBufferUsed || writeCursor > 0;
   }
 
-  public Tuple2 next() {
-    try {
-      List row;
-      synchronized (this) {
-        Preconditions.checkState(hasNext());
-        if (cursor == 0) {
-          row = container.first();
+  public synchronized Tuple2 next() {
+    Preconditions.checkState(hasNext());
+    if (!readBufferUsed) {
+      try {
+        if (input == null && output != null) {
+          // Close output stream if open
+          output.close();
+          output = null;
+
+          FileInputStream fis = null;
+          try {
+            fis = new FileInputStream(tmpFile);
+            input = new Input(fis);
+          } finally {
+            if (input == null && fis != null) {
+              fis.close();
+            }
+          }
+        }
+        if (input != null) {
+          // Load next batch from disk
+          for (int i = 0; i < IN_MEMORY_NUM_ROWS; i++) {
+            ObjectPair pair = readBuffer[i];
+            pair.setFirst(readHiveKey(input));
+            pair.setSecond(readValue(input));
+          }
+          if (input.eof()) {
+            input.close();
+            input = null;
+          }
+          rowsInReadBuffer = IN_MEMORY_NUM_ROWS;
+          readBufferUsed = true;
+          readCursor = 0;
+        } else if (writeCursor == 1) {
+          ObjectPair pair = writeBuffer[0];
+          Tuple2 row = new Tuple2(
+            pair.getFirst(), pair.getSecond());
+          pair.setFirst(null);
+          pair.setSecond(null);
+          writeCursor = 0;
+          return row;
         } else {
-          row = container.next();
+          // No record on disk, more data in write buffer
+          switchBufferAndResetCursor();
         }
-        cursor++;
+      } catch (Exception e) {
+        clear(); // Clean up the cache
+        throw new RuntimeException("Failed to load rows from disk", e);
       }
-      HiveKey key = KryoSerializer.deserialize(row.get(0).getBytes(), HiveKey.class);
-      return new Tuple2(key, row.get(1));
-    } catch (HiveException ex) {
-      throw new RuntimeException("Failed to get row from RowContainer", ex);
     }
+    ObjectPair pair = readBuffer[readCursor];
+    Tuple2 row = new Tuple2(
+      pair.getFirst(), pair.getSecond());
+    pair.setFirst(null);
+    pair.setSecond(null);
+    if (++readCursor >= rowsInReadBuffer) {
+      readBufferUsed = false;
+      rowsInReadBuffer = 0;
+      readCursor = 0;
+    }
+    return row;
   }
 }
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunction.java	(working copy)
@@ -18,6 +18,8 @@
 
 package org.apache.hadoop.hive.ql.exec.spark;
 
+import java.util.Iterator;
+
 import org.apache.hadoop.hive.ql.exec.Utilities;
 import org.apache.hadoop.hive.ql.io.HiveKey;
 import org.apache.hadoop.hive.ql.io.merge.MergeFileMapper;
@@ -25,8 +27,6 @@
 
 import scala.Tuple2;
 
-import java.util.Iterator;
-
 public class HiveMapFunction extends HivePairFlatMapFunction<
   Iterator>, HiveKey, BytesWritable> {
 
@@ -51,7 +51,7 @@
       mapRecordHandler = new SparkMapRecordHandler();
     }
 
-    HiveMapFunctionResultList result = new HiveMapFunctionResultList(jobConf, it, mapRecordHandler);
+    HiveMapFunctionResultList result = new HiveMapFunctionResultList(it, mapRecordHandler);
     mapRecordHandler.init(jobConf, result, sparkReporter);
 
     return result;
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveMapFunctionResultList.java	(working copy)
@@ -17,27 +17,28 @@
  */
 package org.apache.hadoop.hive.ql.exec.spark;
 
-import org.apache.hadoop.conf.Configuration;
+import java.io.IOException;
+import java.util.Iterator;
+
 import org.apache.hadoop.io.BytesWritable;
+
 import scala.Tuple2;
 
-import java.io.IOException;
-import java.util.Iterator;
-
 public class HiveMapFunctionResultList extends
     HiveBaseFunctionResultList> {
+  private static final long serialVersionUID = 1L;
   private final SparkRecordHandler recordHandler;
 
   /**
    * Instantiate result set Iterable for Map function output.
    *
-   * @param conf Hive configuration.
    * @param inputIterator Input record iterator.
    * @param handler Initialized {@link SparkMapRecordHandler} instance.
    */
-  public HiveMapFunctionResultList(Configuration conf,
-      Iterator> inputIterator, SparkRecordHandler handler) {
-    super(conf, inputIterator);
+  public HiveMapFunctionResultList(
+      Iterator> inputIterator,
+      SparkRecordHandler handler) {
+    super(inputIterator);
     recordHandler = handler;
   }
 
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java	(working copy)
@@ -18,13 +18,13 @@
 
 package org.apache.hadoop.hive.ql.exec.spark;
 
+import java.util.Iterator;
+
 import org.apache.hadoop.hive.ql.io.HiveKey;
 import org.apache.hadoop.io.BytesWritable;
 
 import scala.Tuple2;
 
-import java.util.Iterator;
-
 public class HiveReduceFunction extends HivePairFlatMapFunction<
   Iterator>>, HiveKey, BytesWritable> {
 
@@ -42,7 +42,7 @@
 
     SparkReduceRecordHandler reducerRecordhandler = new SparkReduceRecordHandler();
     HiveReduceFunctionResultList result =
-        new HiveReduceFunctionResultList(jobConf, it, reducerRecordhandler);
+        new HiveReduceFunctionResultList(it, reducerRecordhandler);
     reducerRecordhandler.init(jobConf, result, sparkReporter);
 
     return result;
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunctionResultList.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunctionResultList.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunctionResultList.java	(working copy)
@@ -17,29 +17,29 @@
  */
 package org.apache.hadoop.hive.ql.exec.spark;
 
-import org.apache.hadoop.conf.Configuration;
+import java.io.IOException;
+import java.util.Iterator;
+
 import org.apache.hadoop.hive.ql.io.HiveKey;
 import org.apache.hadoop.io.BytesWritable;
+
 import scala.Tuple2;
 
-import java.io.IOException;
-import java.util.Iterator;
-
 public class HiveReduceFunctionResultList extends
     HiveBaseFunctionResultList>> {
+  private static final long serialVersionUID = 1L;
   private final SparkReduceRecordHandler reduceRecordHandler;
 
   /**
    * Instantiate result set Iterable for Reduce function output.
    *
-   * @param conf Hive configuration.
    * @param inputIterator Input record iterator.
    * @param reducer Initialized {@link org.apache.hadoop.hive.ql.exec.mr.ExecReducer} instance.
    */
-  public HiveReduceFunctionResultList(Configuration conf,
+  public HiveReduceFunctionResultList(
       Iterator>> inputIterator,
-    SparkReduceRecordHandler reducer) {
-    super(conf, inputIterator);
+      SparkReduceRecordHandler reducer) {
+    super(inputIterator);
     this.reduceRecordHandler = reducer;
   }
 
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java	(working copy)
@@ -19,11 +19,11 @@
 package org.apache.hadoop.hive.ql.exec.spark;
 
 import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
 import java.io.DataInputStream;
 import java.io.DataOutputStream;
 import java.io.IOException;
 
+import org.apache.commons.io.output.ByteArrayOutputStream;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.apache.hadoop.hive.ql.exec.Utilities;
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java	(working copy)
@@ -22,8 +22,8 @@
 
 import java.io.IOException;
 import java.io.Serializable;
-import java.net.MalformedURLException;
-import java.net.URL;
+import java.net.URI;
+import java.net.URISyntaxException;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
@@ -77,8 +77,8 @@
   private transient SparkConf sparkConf;
   private transient HiveConf hiveConf;
 
-  private transient List localJars = new ArrayList();
-  private transient List localFiles = new ArrayList();
+  private transient List localJars = new ArrayList();
+  private transient List localFiles = new ArrayList();
 
   private final transient long sparkClientTimtout;
 
@@ -128,7 +128,7 @@
     return new RemoteSparkJobRef(hiveConf, jobHandle, sparkJobStatus);
   }
 
-  private void refreshLocalResources(SparkWork sparkWork, HiveConf conf) {
+  private void refreshLocalResources(SparkWork sparkWork, HiveConf conf) throws IOException {
     // add hive-exec jar
     addJars((new JobConf(this.getClass())).getJar());
 
@@ -160,30 +160,32 @@
     addResources(addedArchives);
   }
 
-  private void addResources(String addedFiles) {
+  private void addResources(String addedFiles) throws IOException {
     for (String addedFile : CSV_SPLITTER.split(Strings.nullToEmpty(addedFiles))) {
       try {
-        URL fileUrl = SparkUtilities.getURL(addedFile);
-        if (fileUrl != null && !localFiles.contains(fileUrl)) {
-          localFiles.add(fileUrl);
-          remoteClient.addFile(fileUrl);
+        URI fileUri = SparkUtilities.getURI(addedFile);
+        if (fileUri != null && !localFiles.contains(fileUri)) {
+          fileUri = SparkUtilities.uploadToHDFS(fileUri, hiveConf);
+          localFiles.add(fileUri);
+          remoteClient.addFile(fileUri);
         }
-      } catch (MalformedURLException e) {
-        LOG.warn("Failed to add file:" + addedFile);
+      } catch (URISyntaxException e) {
+        LOG.warn("Failed to add file:" + addedFile, e);
       }
     }
   }
 
-  private void addJars(String addedJars) {
+  private void addJars(String addedJars) throws IOException {
     for (String addedJar : CSV_SPLITTER.split(Strings.nullToEmpty(addedJars))) {
       try {
-        URL jarUrl = SparkUtilities.getURL(addedJar);
-        if (jarUrl != null && !localJars.contains(jarUrl)) {
-          localJars.add(jarUrl);
-          remoteClient.addJar(jarUrl);
+        URI jarUri = SparkUtilities.getURI(addedJar);
+        if (jarUri != null && !localJars.contains(jarUri)) {
+          jarUri = SparkUtilities.uploadToHDFS(jarUri, hiveConf);
+          localJars.add(jarUri);
+          remoteClient.addJar(jarUri);
         }
-      } catch (MalformedURLException e) {
-        LOG.warn("Failed to add jar:" + addedJar);
+      } catch (URISyntaxException e) {
+        LOG.warn("Failed to add jar:" + addedJar, e);
       }
     }
   }
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java	(working copy)
@@ -18,11 +18,15 @@
 package org.apache.hadoop.hive.ql.exec.spark;
 
 import java.io.File;
+import java.io.IOException;
 import java.net.MalformedURLException;
 import java.net.URI;
 import java.net.URISyntaxException;
 import java.net.URL;
 
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.ql.exec.spark.session.SparkSession;
 import org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManager;
@@ -50,25 +54,50 @@
     return copy;
   }
 
-  public static URL getURL(String path) throws MalformedURLException {
+  public static URI getURI(String path) throws URISyntaxException {
     if (path == null) {
       return null;
     }
 
-    URL url = null;
-    try {
       URI uri = new URI(path);
-      if (uri.getScheme() != null) {
-        url = uri.toURL();
-      } else {
+      if (uri.getScheme() == null) {
         // if no file schema in path, we assume it's file on local fs.
-        url = new File(path).toURI().toURL();
+        uri = new File(path).toURI();
       }
-    } catch (URISyntaxException e) {
-      // do nothing here, just return null if input path is not a valid URI.
+
+    return uri;
+  }
+
+  /**
+   * Copies local file to HDFS in yarn-cluster mode.
+   *
+   * @param source
+   * @param conf
+   * @return
+   * @throws IOException
+   */
+  public static URI uploadToHDFS(URI source, HiveConf conf) throws IOException {
+    URI result = source;
+    if (conf.get("spark.master").equals("yarn-cluster")) {
+      if (!source.getScheme().equals("hdfs")) {
+        Path tmpDir = SessionState.getHDFSSessionPath(conf);
+        FileSystem fileSystem = FileSystem.get(conf);
+        fileSystem.copyFromLocalFile(new Path(source.getPath()), tmpDir);
+        String filePath = tmpDir + File.separator + getFileName(source);
+        Path fullPath = fileSystem.getFileStatus(new Path(filePath)).getPath();
+        result = fullPath.toUri();
+      }
     }
+    return result;
+  }
 
-    return url;
+  private static String getFileName(URI uri) {
+    if (uri == null) {
+      return null;
+    }
+
+    String[] splits = uri.getPath().split(File.separator);
+    return  splits[splits.length-1];
   }
 
   public static SparkSession getSparkSession(HiveConf conf,
Index: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java	(working copy)
@@ -21,6 +21,7 @@
 import static org.apache.tez.dag.api.client.DAGStatus.State.RUNNING;
 import static org.fusesource.jansi.Ansi.ansi;
 import static org.fusesource.jansi.internal.CLibrary.STDOUT_FILENO;
+import static org.fusesource.jansi.internal.CLibrary.STDERR_FILENO;
 import static org.fusesource.jansi.internal.CLibrary.isatty;
 
 import org.apache.hadoop.hive.conf.HiveConf;
@@ -167,6 +168,9 @@
       if (isatty(STDOUT_FILENO) == 0) {
         return false;
       }
+      if (isatty(STDERR_FILENO) == 0) {
+        return false;
+      }
     } catch (NoClassDefFoundError ignore) {
       // These errors happen if the JNI lib is not available for your platform.
       return false;
Index: ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java	(working copy)
@@ -67,7 +67,7 @@
 
   private static final String DELIMITER = " ";
 
-  private static final String ROW_COUNT_PATTERN = "TABLE_ID_(\\d+)_ROWCOUNT";
+  private static final String ROW_COUNT_PATTERN = "RECORDS_OUT_(\\d+)(_)*(\\S+)*";
 
   private static final Pattern rowCountPattern = Pattern.compile(ROW_COUNT_PATTERN);
 
@@ -343,6 +343,10 @@
 
     if (m.find()) {
       String tuple = m.group(1);
+      String tableName = m.group(3);
+      if (tableName != null)
+        return tableName;
+
       return idToTableMap.get(tuple);
     }
     return null;
Index: ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java	(working copy)
@@ -17,11 +17,9 @@
  */
 package org.apache.hadoop.hive.ql.hooks;
 
-import com.google.common.util.concurrent.ThreadFactoryBuilder;
-
 import java.util.List;
+import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
-import java.util.concurrent.ExecutorService;
 import java.util.concurrent.TimeUnit;
 
 import org.apache.commons.logging.Log;
@@ -37,10 +35,9 @@
 import org.apache.hadoop.yarn.api.records.timeline.TimelineEvent;
 import org.apache.hadoop.yarn.client.api.TimelineClient;
 import org.apache.hadoop.yarn.conf.YarnConfiguration;
-
 import org.json.JSONObject;
 
-import static org.apache.hadoop.hive.ql.hooks.HookContext.HookType.*;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
 
 /**
  * ATSHook sends query + plan info to Yarn App Timeline Server. To enable (hadoop 2.4 and up) set
@@ -55,7 +52,7 @@
   private enum EntityTypes { HIVE_QUERY_ID };
   private enum EventTypes { QUERY_SUBMITTED, QUERY_COMPLETED };
   private enum OtherInfoTypes { QUERY, STATUS, TEZ, MAPRED };
-  private enum PrimaryFilterTypes { user };
+  private enum PrimaryFilterTypes { user, operationid };
   private static final int WAIT_TIME = 3;
 
   public ATSHook() {
@@ -101,6 +98,7 @@
               return;
             }
             String queryId = plan.getQueryId();
+            String opId = hookContext.getOperationId();
             long queryStartTime = plan.getQueryStartTime();
             String user = hookContext.getUgi().getUserName();
             int numMrJobs = Utilities.getMRTasks(plan.getRootTasks()).size();
@@ -119,13 +117,13 @@
               JSONObject explainPlan = explain.getJSONPlan(null, null, rootTasks,
                    plan.getFetchTask(), true, false, false);
               fireAndForget(conf, createPreHookEvent(queryId, query,
-                   explainPlan, queryStartTime, user, numMrJobs, numTezJobs));
+                   explainPlan, queryStartTime, user, numMrJobs, numTezJobs, opId));
               break;
             case POST_EXEC_HOOK:
-              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, true));
+              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, true, opId));
               break;
             case ON_FAILURE_HOOK:
-              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, false));
+              fireAndForget(conf, createPostHookEvent(queryId, currentTime, user, false, opId));
               break;
             default:
               //ignore
@@ -139,7 +137,7 @@
   }
 
   TimelineEntity createPreHookEvent(String queryId, String query, JSONObject explainPlan,
-      long startTime, String user, int numMrJobs, int numTezJobs) throws Exception {
+      long startTime, String user, int numMrJobs, int numTezJobs, String opId) throws Exception {
 
     JSONObject queryObj = new JSONObject();
     queryObj.put("queryText", query);
@@ -148,12 +146,16 @@
     LOG.info("Received pre-hook notification for :" + queryId);
     if (LOG.isDebugEnabled()) {
       LOG.debug("Otherinfo: " + queryObj.toString());
+      LOG.debug("Operation id: <" + opId + ">");
     }
 
     TimelineEntity atsEntity = new TimelineEntity();
     atsEntity.setEntityId(queryId);
     atsEntity.setEntityType(EntityTypes.HIVE_QUERY_ID.name());
     atsEntity.addPrimaryFilter(PrimaryFilterTypes.user.name(), user);
+    if (opId != null) {
+      atsEntity.addPrimaryFilter(PrimaryFilterTypes.operationid.name(), opId);
+    }
 
     TimelineEvent startEvt = new TimelineEvent();
     startEvt.setEventType(EventTypes.QUERY_SUBMITTED.name());
@@ -166,13 +168,17 @@
     return atsEntity;
   }
 
-  TimelineEntity createPostHookEvent(String queryId, long stopTime, String user, boolean success) {
+  TimelineEntity createPostHookEvent(String queryId, long stopTime, String user, boolean success,
+      String opId) {
     LOG.info("Received post-hook notification for :" + queryId);
 
     TimelineEntity atsEntity = new TimelineEntity();
     atsEntity.setEntityId(queryId);
     atsEntity.setEntityType(EntityTypes.HIVE_QUERY_ID.name());
     atsEntity.addPrimaryFilter(PrimaryFilterTypes.user.name(), user);
+    if (opId != null) {
+      atsEntity.addPrimaryFilter(PrimaryFilterTypes.operationid.name(), opId);
+    }
 
     TimelineEvent stopEvt = new TimelineEvent();
     stopEvt.setEventType(EventTypes.QUERY_COMPLETED.name());
Index: ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java	(working copy)
@@ -29,7 +29,6 @@
 import org.apache.hadoop.hive.ql.QueryPlan;
 import org.apache.hadoop.hive.ql.exec.TaskRunner;
 import org.apache.hadoop.hive.ql.session.SessionState;
-import org.apache.hadoop.hive.shims.ShimLoader;
 import org.apache.hadoop.hive.shims.Utils;
 import org.apache.hadoop.security.UserGroupInformation;
 /**
@@ -53,9 +52,13 @@
   final private Map inputPathToContentSummary;
   private final String ipAddress;
   private final String userName;
+  // unique id set for operation when run from HS2, base64 encoded value of
+  // TExecuteStatementResp.TOperationHandle.THandleIdentifier.guid
+  private final String operationId;
 
   public HookContext(QueryPlan queryPlan, HiveConf conf,
-      Map inputPathToContentSummary, String userName, String ipAddress) throws Exception {
+      Map inputPathToContentSummary, String userName, String ipAddress,
+      String operationId) throws Exception {
     this.queryPlan = queryPlan;
     this.conf = conf;
     this.inputPathToContentSummary = inputPathToContentSummary;
@@ -67,8 +70,9 @@
     if(SessionState.get() != null){
       linfo = SessionState.get().getLineageState().getLineageInfo();
     }
+    this.userName = userName;
     this.ipAddress = ipAddress;
-    this.userName = userName;
+    this.operationId = operationId;
   }
 
   public QueryPlan getQueryPlan() {
@@ -154,4 +158,8 @@
   public String getUserName() {
     return this.userName;
   }
+
+  public String getOperationId() {
+    return operationId;
+  }
 }
Index: ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java	(working copy)
@@ -35,7 +35,7 @@
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.ql.metadata.HiveException;
 import org.apache.hadoop.hive.serde2.columnar.BytesRefWritable;
-import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe;
+import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters;
 import org.apache.hadoop.io.Text;
 import org.apache.hadoop.mapred.FileSplit;
 import org.apache.hadoop.mapred.JobConf;
@@ -144,7 +144,7 @@
     int firstEnd = 0;
     int i = 0;
     for (int index = 0; index < bytes.length; index++) {
-      if (bytes[index] == LazySimpleSerDe.DefaultSeparators[0]) {
+      if (bytes[index] == LazySerDeParameters.DefaultSeparators[0]) {
         i++;
         firstEnd = index;
       }
@@ -169,7 +169,7 @@
     int currentStart = firstEnd + 1;
     int currentEnd = firstEnd + 1;
     for (; currentEnd < bytes.length; currentEnd++) {
-      if (bytes[currentEnd] == LazySimpleSerDe.DefaultSeparators[1]) {
+      if (bytes[currentEnd] == LazySerDeParameters.DefaultSeparators[1]) {
         String one_offset = new String(bytes, currentStart, currentEnd
             - currentStart);
         Long offset = Long.parseLong(one_offset);
Index: ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java	(working copy)
@@ -204,7 +204,7 @@
   private ExprNodeDesc analyzeExpr(
     ExprNodeGenericFuncDesc expr,
     List searchConditions,
-    Object... nodeOutputs) {
+    Object... nodeOutputs) throws SemanticException {
 
     if (FunctionRegistry.isOpAnd(expr)) {
       assert(nodeOutputs.length == 2);
Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileDump.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileDump.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/FileDump.java	(working copy)
@@ -17,6 +17,14 @@
  */
 package org.apache.hadoop.hive.ql.io.orc;
 
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.text.DecimalFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
 import org.apache.commons.cli.CommandLine;
 import org.apache.commons.cli.GnuParser;
 import org.apache.commons.cli.HelpFormatter;
@@ -25,6 +33,7 @@
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.io.filters.BloomFilter;
 import org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex;
 import org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndexEntry;
 import org.apache.hadoop.hive.serde2.io.ByteWritable;
@@ -37,14 +46,6 @@
 import org.codehaus.jettison.json.JSONException;
 import org.codehaus.jettison.json.JSONWriter;
 
-import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.text.DecimalFormat;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
-
 /**
  * A tool for printing out the file structure of ORC files.
  */
@@ -114,6 +115,7 @@
         }
       }
       ColumnStatistics[] stats = reader.getStatistics();
+      int colCount = stats.length;
       System.out.println("\nFile Statistics:");
       for(int i=0; i < stats.length; ++i) {
         System.out.println("  Column " + i + ": " + stats[i].toString());
@@ -127,9 +129,10 @@
         OrcProto.StripeFooter footer = rows.readStripeFooter(stripe);
         long sectionStart = stripeStart;
         for(OrcProto.Stream section: footer.getStreamsList()) {
+          String kind = section.hasKind() ? section.getKind().name() : "UNKNOWN";
           System.out.println("    Stream: column " + section.getColumn() +
-            " section " + section.getKind() + " start: " + sectionStart +
-            " length " + section.getLength());
+              " section " + kind + " start: " + sectionStart +
+              " length " + section.getLength());
           sectionStart += section.getLength();
         }
         for (int i = 0; i < footer.getColumnsCount(); ++i) {
@@ -147,38 +150,20 @@
           }
           System.out.println(buf);
         }
-        if (rowIndexCols != null) {
-          RowIndex[] indices = rows.readRowIndex(stripeIx, null);
+        if (rowIndexCols != null && !rowIndexCols.isEmpty()) {
+          // include the columns that are specified, only if the columns are included, bloom filter
+          // will be read
+          boolean[] sargColumns = new boolean[colCount];
+          for (int colIdx : rowIndexCols) {
+            sargColumns[colIdx] = true;
+          }
+          RecordReaderImpl.Index indices = rows.readRowIndex(stripeIx, null, null, null, sargColumns);
           for (int col : rowIndexCols) {
             StringBuilder buf = new StringBuilder();
-            buf.append("    Row group index column ").append(col).append(":");
-            RowIndex index = null;
-            if ((col >= indices.length) || ((index = indices[col]) == null)) {
-              buf.append(" not found\n");
-              continue;
-            }
-            for (int entryIx = 0; entryIx < index.getEntryCount(); ++entryIx) {
-              buf.append("\n      Entry ").append(entryIx).append(":");
-              RowIndexEntry entry = index.getEntry(entryIx);
-              if (entry == null) {
-                buf.append("unknown\n");
-                continue;
-              }
-              OrcProto.ColumnStatistics colStats = entry.getStatistics();
-              if (colStats == null) {
-                buf.append("no stats at ");
-              } else {
-                ColumnStatistics cs = ColumnStatisticsImpl.deserialize(colStats);
-                buf.append(cs.toString());
-              }
-              buf.append(" positions: ");
-              for (int posIx = 0; posIx < entry.getPositionsCount(); ++posIx) {
-                if (posIx != 0) {
-                  buf.append(",");
-                }
-                buf.append(entry.getPositions(posIx));
-              }
-            }
+            String rowIdxString = getFormattedRowIndices(col, indices.getRowGroupIndex());
+            buf.append(rowIdxString);
+            String bloomFilString = getFormattedBloomFilters(col, indices.getBloomFilterIndex());
+            buf.append(bloomFilString);
             System.out.println(buf);
           }
         }
@@ -197,6 +182,82 @@
     }
   }
 
+  private static String getFormattedBloomFilters(int col,
+      OrcProto.BloomFilterIndex[] bloomFilterIndex) {
+    StringBuilder buf = new StringBuilder();
+    BloomFilter stripeLevelBF = null;
+    if (bloomFilterIndex != null && bloomFilterIndex[col] != null) {
+      int idx = 0;
+      buf.append("\n    Bloom filters for column ").append(col).append(":");
+      for (OrcProto.BloomFilter bf : bloomFilterIndex[col].getBloomFilterList()) {
+        BloomFilter toMerge = new BloomFilter(bf);
+        buf.append("\n      Entry ").append(idx++).append(":").append(getBloomFilterStats(toMerge));
+        if (stripeLevelBF == null) {
+          stripeLevelBF = toMerge;
+        } else {
+          stripeLevelBF.merge(toMerge);
+        }
+      }
+      String bloomFilterStats = getBloomFilterStats(stripeLevelBF);
+      buf.append("\n      Stripe level merge:").append(bloomFilterStats);
+    }
+    return buf.toString();
+  }
+
+  private static String getBloomFilterStats(BloomFilter bf) {
+    StringBuilder sb = new StringBuilder();
+    int bitCount = bf.getBitSize();
+    int popCount = 0;
+    for (long l : bf.getBitSet()) {
+      popCount += Long.bitCount(l);
+    }
+    int k = bf.getNumHashFunctions();
+    float loadFactor = (float) popCount / (float) bitCount;
+    float expectedFpp = (float) Math.pow(loadFactor, k);
+    DecimalFormat df = new DecimalFormat("###.####");
+    sb.append(" numHashFunctions: ").append(k);
+    sb.append(" bitCount: ").append(bitCount);
+    sb.append(" popCount: ").append(popCount);
+    sb.append(" loadFactor: ").append(df.format(loadFactor));
+    sb.append(" expectedFpp: ").append(expectedFpp);
+    return sb.toString();
+  }
+
+  private static String getFormattedRowIndices(int col, RowIndex[] rowGroupIndex) {
+    StringBuilder buf = new StringBuilder();
+    RowIndex index;
+    buf.append("    Row group indices for column ").append(col).append(":");
+    if (rowGroupIndex == null || (col >= rowGroupIndex.length) ||
+        ((index = rowGroupIndex[col]) == null)) {
+      buf.append(" not found\n");
+      return buf.toString();
+    }
+
+    for (int entryIx = 0; entryIx < index.getEntryCount(); ++entryIx) {
+      buf.append("\n      Entry ").append(entryIx).append(": ");
+      RowIndexEntry entry = index.getEntry(entryIx);
+      if (entry == null) {
+        buf.append("unknown\n");
+        continue;
+      }
+      OrcProto.ColumnStatistics colStats = entry.getStatistics();
+      if (colStats == null) {
+        buf.append("no stats at ");
+      } else {
+        ColumnStatistics cs = ColumnStatisticsImpl.deserialize(colStats);
+        buf.append(cs.toString());
+      }
+      buf.append(" positions: ");
+      for (int posIx = 0; posIx < entry.getPositionsCount(); ++posIx) {
+        if (posIx != 0) {
+          buf.append(",");
+        }
+        buf.append(entry.getPositions(posIx));
+      }
+    }
+    return buf.toString();
+  }
+
   private static long getTotalPaddingSize(Reader reader) throws IOException {
     long paddedBytes = 0;
     List stripes = reader.getStripes();
Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java	(working copy)
@@ -471,6 +471,15 @@
     }
   }
 
+  public static InStream create(String streamName,
+      ByteBuffer[] buffers,
+      long[] offsets,
+      long length,
+      CompressionCodec codec,
+      int bufferSize) throws IOException {
+    return create(null, streamName, buffers, offsets, length, codec, bufferSize);
+  }
+
   /**
    * Create an input stream from a list of buffers.
    * @param fileName name of the file
Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReader.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReader.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReader.java	(working copy)
@@ -19,6 +19,7 @@
 
 import java.io.IOException;
 import java.nio.ByteBuffer;
+import java.util.List;
 
 import org.apache.hadoop.fs.FSDataInputStream;
 import org.apache.hadoop.fs.FileSystem;
@@ -47,30 +48,58 @@
     this.typeCount = typeCount;
   }
 
-  public OrcProto.RowIndex[] readRowIndex(StripeInformation stripe, OrcProto.StripeFooter footer,
-      boolean[] included, OrcProto.RowIndex[] indexes) throws IOException {
+  public RecordReaderImpl.Index readRowIndex(StripeInformation stripe, OrcProto.StripeFooter footer,
+      boolean[] included, OrcProto.RowIndex[] indexes, boolean[] sargColumns,
+      OrcProto.BloomFilterIndex[] bloomFilterIndices) throws IOException {
     if (footer == null) {
       footer = readStripeFooter(stripe);
     }
     if (indexes == null) {
       indexes = new OrcProto.RowIndex[typeCount];
     }
+    if (bloomFilterIndices == null) {
+      bloomFilterIndices = new OrcProto.BloomFilterIndex[typeCount];
+    }
     long offset = stripe.getOffset();
-    for (OrcProto.Stream stream : footer.getStreamsList()) {
-      if (stream.getKind() == OrcProto.Stream.Kind.ROW_INDEX) {
-        int col = stream.getColumn();
+    List streams = footer.getStreamsList();
+    for (int i = 0; i < streams.size(); i++) {
+      OrcProto.Stream stream = streams.get(i);
+      OrcProto.Stream nextStream = null;
+      if (i < streams.size() - 1) {
+        nextStream = streams.get(i+1);
+      }
+      int col = stream.getColumn();
+      int len = (int) stream.getLength();
+      // row index stream and bloom filter are interlaced, check if the sarg column contains bloom
+      // filter and combine the io to read row index and bloom filters for that column together
+      if (stream.hasKind() && (stream.getKind() == OrcProto.Stream.Kind.ROW_INDEX)) {
+        boolean readBloomFilter = false;
+        if (sargColumns != null && sargColumns[col] &&
+            nextStream.getKind() == OrcProto.Stream.Kind.BLOOM_FILTER) {
+          len += nextStream.getLength();
+          i += 1;
+          readBloomFilter = true;
+        }
         if ((included == null || included[col]) && indexes[col] == null) {
-          byte[] buffer = new byte[(int) stream.getLength()];
+          byte[] buffer = new byte[len];
           file.seek(offset);
           file.readFully(buffer);
-          indexes[col] = OrcProto.RowIndex.parseFrom(InStream.create(null, "index",
-              Lists.newArrayList(new BufferChunk(ByteBuffer.wrap(buffer), 0)),
-              stream.getLength(), codec, bufferSize, null));
+          ByteBuffer[] bb = new ByteBuffer[] {ByteBuffer.wrap(buffer)};
+          indexes[col] = OrcProto.RowIndex.parseFrom(InStream.create("index",
+              bb, new long[]{0}, stream.getLength(), codec, bufferSize));
+          if (readBloomFilter) {
+            bb[0].position((int) stream.getLength());
+            bloomFilterIndices[col] = OrcProto.BloomFilterIndex.parseFrom(
+                InStream.create("bloom_filter", bb, new long[]{0}, nextStream.getLength(),
+                    codec, bufferSize));
+          }
         }
       }
-      offset += stream.getLength();
+      offset += len;
     }
-    return indexes;
+
+    RecordReaderImpl.Index index = new RecordReaderImpl.Index(indexes, bloomFilterIndices);
+    return index;
   }
 
   public OrcProto.StripeFooter readStripeFooter(StripeInformation stripe) throws IOException {
Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java	(working copy)
@@ -32,6 +32,7 @@
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
 import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.io.filters.BloomFilter;
 import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
 
 /**
@@ -154,7 +155,9 @@
     ROW_INDEX_STRIDE("orc.row.index.stride"),
     ENABLE_INDEXES("orc.create.index"),
     BLOCK_PADDING("orc.block.padding"),
-    ENCODING_STRATEGY("orc.encoding.strategy");
+    ENCODING_STRATEGY("orc.encoding.strategy"),
+    BLOOM_FILTER_COLUMNS("orc.bloom.filter.columns"),
+    BLOOM_FILTER_FPP("orc.bloom.filter.fpp");
 
     private final String propName;
 
@@ -262,6 +265,8 @@
     private EncodingStrategy encodingStrategy;
     private CompressionStrategy compressionStrategy;
     private float paddingTolerance;
+    private String bloomFilterColumns;
+    private double bloomFilterFpp;
 
     WriterOptions(Configuration conf) {
       configuration = conf;
@@ -294,9 +299,9 @@
         compressionStrategy = CompressionStrategy.valueOf(compString);
       }
 
-      paddingTolerance =
-          conf.getFloat(HiveConf.ConfVars.HIVE_ORC_BLOCK_PADDING_TOLERANCE.varname,
-              HiveConf.ConfVars.HIVE_ORC_BLOCK_PADDING_TOLERANCE.defaultFloatVal);
+      paddingTolerance = conf.getFloat(HiveConf.ConfVars.HIVE_ORC_BLOCK_PADDING_TOLERANCE.varname,
+          HiveConf.ConfVars.HIVE_ORC_BLOCK_PADDING_TOLERANCE.defaultFloatVal);
+      bloomFilterFpp = BloomFilter.DEFAULT_FPP;
     }
 
     /**
@@ -373,6 +378,24 @@
     }
 
     /**
+     * Comma separated values of column names for which bloom filter is to be created.
+     */
+    public WriterOptions bloomFilterColumns(String columns) {
+      bloomFilterColumns = columns;
+      return this;
+    }
+
+    /**
+     * Specify the false positive probability for bloom filter.
+     * @param fpp - false positive probability
+     * @return
+     */
+    public WriterOptions bloomFilterFpp(double fpp) {
+      bloomFilterFpp = fpp;
+      return this;
+    }
+
+    /**
      * Sets the generic compression that is used to compress the data.
      */
     public WriterOptions compress(CompressionKind value) {
@@ -444,8 +467,8 @@
                           opts.memoryManagerValue, opts.blockPaddingValue,
                           opts.versionValue, opts.callback,
                           opts.encodingStrategy, opts.compressionStrategy,
-                          opts.paddingTolerance,
-                          opts.blockSizeValue);
+                          opts.paddingTolerance, opts.blockSizeValue,
+                          opts.bloomFilterColumns, opts.bloomFilterFpp);
   }
 
   /**
Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
===================================================================
--- ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java	(revision 1661246)
+++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java	(working copy)
@@ -18,9 +18,18 @@
 
 package org.apache.hadoop.hive.ql.io.orc;
 
-import com.google.common.cache.Cache;
-import com.google.common.cache.CacheBuilder;
-import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.TreeMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
 
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
@@ -69,18 +78,9 @@
 import org.apache.hadoop.mapred.Reporter;
 import org.apache.hadoop.util.StringUtils;
 
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.NavigableMap;
-import java.util.TreeMap;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.atomic.AtomicInteger;
+import com.google.common.cache.Cache;
+import com.google.common.cache.CacheBuilder;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
 /**
  * A MapReduce/Hive input format for ORC files.
  * 

@@ -927,7 +927,7 @@ // column statistics at index 0 contains only the number of rows ColumnStatistics stats = stripeStatistics.getColumnStatistics()[filterColumns[pred]]; - truthValues[pred] = RecordReaderImpl.evaluatePredicate(stats, predLeaves.get(pred)); + truthValues[pred] = RecordReaderImpl.evaluatePredicate(stats, predLeaves.get(pred), null); } else { // parition column case. Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java (working copy) @@ -17,12 +17,17 @@ */ package org.apache.hadoop.hive.ql.io.orc; +import java.io.IOException; +import java.io.PrintStream; +import java.util.ArrayList; +import java.util.Properties; + import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.ql.io.AcidOutputFormat; import org.apache.hadoop.hive.ql.io.AcidUtils; +import org.apache.hadoop.hive.ql.io.RecordUpdater; import org.apache.hadoop.hive.ql.io.StatsProvidingRecordWriter; -import org.apache.hadoop.hive.ql.io.RecordUpdater; import org.apache.hadoop.hive.ql.io.orc.OrcFile.EncodingStrategy; import org.apache.hadoop.hive.ql.io.orc.OrcSerde.OrcSerdeRow; import org.apache.hadoop.hive.serde2.SerDeStats; @@ -41,11 +46,6 @@ import org.apache.hadoop.mapred.Reporter; import org.apache.hadoop.util.Progressable; -import java.io.IOException; -import java.io.PrintStream; -import java.util.ArrayList; -import java.util.Properties; - /** * A Hive OutputFormat for ORC files. */ @@ -170,6 +170,16 @@ options.encodingStrategy(EncodingStrategy.valueOf(propVal)); } + if ((propVal = getSettingFromPropsFallingBackToConf( + OrcFile.OrcTableProperties.BLOOM_FILTER_COLUMNS.getPropName(), props, conf)) != null) { + options.bloomFilterColumns(propVal); + } + + if ((propVal = getSettingFromPropsFallingBackToConf( + OrcFile.OrcTableProperties.BLOOM_FILTER_FPP.getPropName(), props, conf)) != null) { + options.bloomFilterFpp(Double.parseDouble(propVal)); + } + return options; } Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcStruct.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcStruct.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcStruct.java (working copy) @@ -346,10 +346,7 @@ @Override public Object getMapValueElement(Object map, Object key) { - if (map == null) { - return null; - } - return ((Map) map).get(key); + return ((map == null || key == null)? null : ((Map) map).get(key)); } @Override Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java (working copy) @@ -24,6 +24,7 @@ import java.math.BigDecimal; import java.math.BigInteger; import java.nio.ByteBuffer; +import java.sql.Date; import java.sql.Timestamp; import java.util.ArrayList; import java.util.Arrays; @@ -34,6 +35,7 @@ import java.util.Map; import org.apache.commons.lang.StringUtils; +import org.apache.commons.lang3.exception.ExceptionUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; @@ -54,6 +56,7 @@ import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch; import org.apache.hadoop.hive.ql.exec.vector.expressions.StringExpr; import org.apache.hadoop.hive.ql.io.orc.RecordReaderUtils.ByteBufferAllocatorPool; +import org.apache.hadoop.hive.ql.io.filters.BloomFilter; import org.apache.hadoop.hive.ql.io.sarg.PredicateLeaf; import org.apache.hadoop.hive.ql.io.sarg.SearchArgument; import org.apache.hadoop.hive.ql.io.sarg.SearchArgument.TruthValue; @@ -103,7 +106,10 @@ List bufferChunks = new ArrayList(0); private final TreeReader reader; private final OrcProto.RowIndex[] indexes; + private final OrcProto.BloomFilterIndex[] bloomFilterIndices; private final SargApplier sargApp; + // same as the above array, but indices are set to true + private final boolean[] sargColumns; // an array about which row groups aren't skipped private boolean[] includedRowGroups = null; private final Configuration conf; @@ -112,7 +118,28 @@ private final ByteBufferAllocatorPool pool = new ByteBufferAllocatorPool(); private final ZeroCopyReaderShim zcr; + public final static class Index { + OrcProto.RowIndex[] rowGroupIndex; + OrcProto.BloomFilterIndex[] bloomFilterIndex; + Index(OrcProto.RowIndex[] rgIndex, OrcProto.BloomFilterIndex[] bfIndex) { + this.rowGroupIndex = rgIndex; + this.bloomFilterIndex = bfIndex; + } + + public OrcProto.RowIndex[] getRowGroupIndex() { + return rowGroupIndex; + } + + public OrcProto.BloomFilterIndex[] getBloomFilterIndex() { + return bloomFilterIndex; + } + + public void setRowGroupIndex(OrcProto.RowIndex[] rowGroupIndex) { + this.rowGroupIndex = rowGroupIndex; + } + } + /** * Given a list of column names, find the given column and return the index. * @param columnNames the list of potential column names @@ -139,7 +166,7 @@ * result * @return an array mapping the sarg leaves to concrete column numbers */ - static int[] mapSargColumns(List sargLeaves, + public static int[] mapSargColumns(List sargLeaves, String[] columnNames, int rootColumn) { int[] result = new int[sargLeaves.size()]; @@ -172,9 +199,18 @@ this.metadata = new MetadataReader(file, codec, bufferSize, types.size()); SearchArgument sarg = options.getSearchArgument(); if (sarg != null && strideRate != 0) { - sargApp = new SargApplier(sarg, options.getColumnNames(), strideRate); + sargApp = new SargApplier(sarg, options.getColumnNames(), strideRate, types); + // included will not be null, row options will fill the array with trues if null + sargColumns = new boolean[included.length]; + for (int i : sargApp.filterColumns) { + // filter columns may have -1 as index which could be partition column in SARG. + if (i > 0) { + sargColumns[i] = true; + } + } } else { sargApp = null; + sargColumns = null; } long rows = 0; long skippedRows = 0; @@ -199,6 +235,7 @@ boolean skipCorrupt = HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ORC_SKIP_CORRUPT_DATA); reader = createTreeReader(0, types, included, skipCorrupt); indexes = new OrcProto.RowIndex[types.size()]; + bloomFilterIndices = new OrcProto.BloomFilterIndex[types.size()]; advanceToNextRow(reader, 0L, true); } @@ -1502,7 +1539,7 @@ } len -= bytesRead; offset += bytesRead; - } + } return allBytes; } @@ -1905,7 +1942,7 @@ } } } else { - if (result.noNulls){ + if (result.noNulls){ for (int i = 0; i < batchSize; i++) { adjustedDownLen = StringExpr.rightTrimAndTruncate(result.vector[i], result.start[i], result.length[i], maxLength); if (adjustedDownLen < result.length[i]) { @@ -1974,7 +2011,7 @@ } } } else { - if (result.noNulls){ + if (result.noNulls){ for (int i = 0; i < batchSize; i++) { adjustedDownLen = StringExpr.truncate(result.vector[i], result.start[i], result.length[i], maxLength); if (adjustedDownLen < result.length[i]) { @@ -2539,15 +2576,20 @@ * that is referenced in the predicate. * @param statsProto the statistics for the column mentioned in the predicate * @param predicate the leaf predicate we need to evaluation + * @param bloomFilter * @return the set of truth values that may be returned for the given * predicate. */ - static TruthValue evaluatePredicate(OrcProto.ColumnStatistics statsProto, - PredicateLeaf predicate) { + static TruthValue evaluatePredicateProto(OrcProto.ColumnStatistics statsProto, + PredicateLeaf predicate, OrcProto.BloomFilter bloomFilter) { ColumnStatistics cs = ColumnStatisticsImpl.deserialize(statsProto); Object minValue = getMin(cs); Object maxValue = getMax(cs); - return evaluatePredicateRange(predicate, minValue, maxValue, cs.hasNull()); + BloomFilter bf = null; + if (bloomFilter != null) { + bf = new BloomFilter(bloomFilter); + } + return evaluatePredicateRange(predicate, minValue, maxValue, cs.hasNull(), bf); } /** @@ -2559,14 +2601,14 @@ * predicate. */ static TruthValue evaluatePredicate(ColumnStatistics stats, - PredicateLeaf predicate) { + PredicateLeaf predicate, BloomFilter bloomFilter) { Object minValue = getMin(stats); Object maxValue = getMax(stats); - return evaluatePredicateRange(predicate, minValue, maxValue, stats.hasNull()); + return evaluatePredicateRange(predicate, minValue, maxValue, stats.hasNull(), bloomFilter); } static TruthValue evaluatePredicateRange(PredicateLeaf predicate, Object min, - Object max, boolean hasNull) { + Object max, boolean hasNull, BloomFilter bloomFilter) { // if we didn't have any values, everything must have been null if (min == null) { if (predicate.getOperator() == PredicateLeaf.Operator.IS_NULL) { @@ -2576,21 +2618,42 @@ } } - Location loc; + TruthValue result; + // Predicate object and stats object can be one of the following base types + // LONG, DOUBLE, STRING, DATE, DECIMAL + // Out of these DATE is not implicitly convertible to other types and rest + // others are implicitly convertible. In cases where DATE cannot be converted + // the stats object is converted to text and comparison is performed. + // When STRINGs are converted to other base types, NumberFormat exception + // can occur in which case TruthValue.YES_NO_NULL value is returned try { - // Predicate object and stats object can be one of the following base types - // LONG, DOUBLE, STRING, DATE, DECIMAL - // Out of these DATE is not implicitly convertible to other types and rest - // others are implicitly convertible. In cases where DATE cannot be converted - // the stats object is converted to text and comparison is performed. - // When STRINGs are converted to other base types, NumberFormat exception - // can occur in which case TruthValue.YES_NO_NULL value is returned Object baseObj = predicate.getLiteral(PredicateLeaf.FileFormat.ORC); Object minValue = getConvertedStatsObj(min, baseObj); Object maxValue = getConvertedStatsObj(max, baseObj); Object predObj = getBaseObjectForComparison(baseObj, minValue); - switch (predicate.getOperator()) { + result = evaluatePredicateMinMax(predicate, predObj, minValue, maxValue, hasNull); + if (bloomFilter != null && result != TruthValue.NO_NULL && result != TruthValue.NO) { + result = evaluatePredicateBloomFilter(predicate, predObj, bloomFilter, hasNull); + } + // in case failed conversion, return the default YES_NO_NULL truth value + } catch (NumberFormatException nfe) { + if (LOG.isWarnEnabled()) { + LOG.warn("NumberFormatException when type matching predicate object" + + " and statistics object. Exception: " + ExceptionUtils.getStackTrace(nfe)); + } + result = hasNull ? TruthValue.YES_NO_NULL : TruthValue.YES_NO; + } + return result; + } + + private static TruthValue evaluatePredicateMinMax(PredicateLeaf predicate, Object predObj, + Object minValue, + Object maxValue, + boolean hasNull) { + Location loc; + + switch (predicate.getOperator()) { case NULL_SAFE_EQUALS: loc = compareToRange((Comparable) predObj, minValue, maxValue); if (loc == Location.BEFORE || loc == Location.AFTER) { @@ -2675,12 +2738,81 @@ return hasNull ? TruthValue.YES_NO : TruthValue.NO; default: return hasNull ? TruthValue.YES_NO_NULL : TruthValue.YES_NO; + } + } + + private static TruthValue evaluatePredicateBloomFilter(PredicateLeaf predicate, Object predObj, + BloomFilter bloomFilter, boolean hasNull) { + switch (predicate.getOperator()) { + case NULL_SAFE_EQUALS: + // null safe equals does not return *_NULL variant. So set hasNull to false + return checkInBloomFilter(bloomFilter, predObj, false); + case EQUALS: + return checkInBloomFilter(bloomFilter, predObj, hasNull); + case IN: + for (Object arg : predicate.getLiteralList(PredicateLeaf.FileFormat.ORC)) { + // if atleast one value in IN list exist in bloom filter, qualify the row group/stripe + TruthValue result = checkInBloomFilter(bloomFilter, arg, hasNull); + if (result == TruthValue.YES_NO_NULL || result == TruthValue.YES_NO) { + return result; + } + } + return hasNull ? TruthValue.NO_NULL : TruthValue.NO; + default: + return hasNull ? TruthValue.YES_NO_NULL : TruthValue.YES_NO; + } + } + + private static TruthValue checkInBloomFilter(BloomFilter bf, Object predObj, boolean hasNull) { + TruthValue result = hasNull ? TruthValue.NO_NULL : TruthValue.NO; + + if (predObj instanceof Long) { + if (bf.testLong(((Long) predObj).longValue())) { + result = TruthValue.YES_NO_NULL; } + } else if (predObj instanceof Double) { + if (bf.testDouble(((Double) predObj).doubleValue())) { + result = TruthValue.YES_NO_NULL; + } + } else if (predObj instanceof String || predObj instanceof Text || + predObj instanceof HiveDecimal || predObj instanceof BigDecimal) { + if (bf.testString(predObj.toString())) { + result = TruthValue.YES_NO_NULL; + } + } else if (predObj instanceof Date) { + if (bf.testLong(DateWritable.dateToDays((Date) predObj))) { + result = TruthValue.YES_NO_NULL; + } + } else if (predObj instanceof DateWritable) { + if (bf.testLong(((DateWritable) predObj).getDays())) { + result = TruthValue.YES_NO_NULL; + } + } else if (predObj instanceof Timestamp) { + if (bf.testLong(((Timestamp) predObj).getTime())) { + result = TruthValue.YES_NO_NULL; + } + } else if (predObj instanceof TimestampWritable) { + if (bf.testLong(((TimestampWritable) predObj).getTimestamp().getTime())) { + result = TruthValue.YES_NO_NULL; + } + } else { + // if the predicate object is null and if hasNull says there are no nulls then return NO + if (predObj == null && !hasNull) { + result = TruthValue.NO; + } else { + result = TruthValue.YES_NO_NULL; + } + } - // in case failed conversion, return the default YES_NO_NULL truth value - } catch (NumberFormatException nfe) { - return hasNull ? TruthValue.YES_NO_NULL : TruthValue.YES_NO; + if (result == TruthValue.YES_NO_NULL && !hasNull) { + result = TruthValue.YES_NO; } + + if (LOG.isDebugEnabled()) { + LOG.debug("Bloom filter evaluation: " + result.toString()); + } + + return result; } private static Object getBaseObjectForComparison(Object predObj, Object statsObj) { @@ -2742,18 +2874,22 @@ private final List sargLeaves; private final int[] filterColumns; private final long rowIndexStride; + private final OrcProto.BloomFilterIndex[] bloomFilterIndices; - public SargApplier(SearchArgument sarg, String[] columnNames, long rowIndexStride) { + public SargApplier(SearchArgument sarg, String[] columnNames, long rowIndexStride, + List types) { this.sarg = sarg; sargLeaves = sarg.getLeaves(); filterColumns = mapSargColumns(sargLeaves, columnNames, 0); + bloomFilterIndices = new OrcProto.BloomFilterIndex[types.size()]; this.rowIndexStride = rowIndexStride; } /** * Pick the row groups that we need to load from the current stripe. + * * @return an array with a boolean for each row group or null if all of the - * row groups must be read. + * row groups must be read. * @throws IOException */ public boolean[] pickRowGroups( @@ -2762,12 +2898,16 @@ int groupsInStripe = (int) ((rowsInStripe + rowIndexStride - 1) / rowIndexStride); boolean[] result = new boolean[groupsInStripe]; // TODO: avoid alloc? TruthValue[] leafValues = new TruthValue[sargLeaves.size()]; - for(int rowGroup=0; rowGroup < result.length; ++rowGroup) { - for(int pred=0; pred < leafValues.length; ++pred) { + for (int rowGroup = 0; rowGroup < result.length; ++rowGroup) { + for (int pred = 0; pred < leafValues.length; ++pred) { if (filterColumns[pred] != -1) { OrcProto.ColumnStatistics stats = indexes[filterColumns[pred]].getEntry(rowGroup).getStatistics(); - leafValues[pred] = evaluatePredicate(stats, sargLeaves.get(pred)); + OrcProto.BloomFilter bf = null; + if (bloomFilterIndices[filterColumns[pred]] != null) { + bf = bloomFilterIndices[filterColumns[pred]].getBloomFilter(rowGroup); + } + leafValues[pred] = evaluatePredicateProto(stats, sargLeaves.get(pred), bf); if (LOG.isDebugEnabled()) { LOG.debug("Stats = " + stats); LOG.debug("Setting " + sargLeaves.get(pred) + " to " + @@ -2781,20 +2921,19 @@ result[rowGroup] = sarg.evaluate(leafValues).isNeeded(); if (LOG.isDebugEnabled()) { LOG.debug("Row group " + (rowIndexStride * rowGroup) + " to " + - (rowIndexStride * (rowGroup+1) - 1) + " is " + + (rowIndexStride * (rowGroup + 1) - 1) + " is " + (result[rowGroup] ? "" : "not ") + "included."); } } // if we found something to skip, use the array. otherwise, return null. - for (boolean b: result) { + for (boolean b : result) { if (!b) { return result; } } return null; } - } /** @@ -2994,7 +3133,10 @@ long length = stream.getLength(); int column = stream.getColumn(); OrcProto.Stream.Kind streamKind = stream.getKind(); - if (StreamName.getArea(streamKind) == StreamName.Area.DATA && includedColumns[column]) { + // since stream kind is optional, first check if it exists + if (stream.hasKind() && + (StreamName.getArea(streamKind) == StreamName.Area.DATA) && + includedColumns[column]) { // if we aren't filtering or it is a dictionary, load it. if (includedRowGroups == null || RecordReaderUtils.isDictionary(streamKind, encodings.get(column))) { @@ -3042,7 +3184,8 @@ for (OrcProto.Stream streamDesc: streamDescriptions) { int column = streamDesc.getColumn(); if ((includeColumn != null && !includeColumn[column]) || - StreamName.getArea(streamDesc.getKind()) != StreamName.Area.DATA) { + streamDesc.hasKind() && + (StreamName.getArea(streamDesc.getKind()) != StreamName.Area.DATA)) { streamOffset += streamDesc.getLength(); continue; } @@ -3247,21 +3390,22 @@ throw new IllegalArgumentException("Seek after the end of reader range"); } - OrcProto.RowIndex[] readRowIndex( - int stripeIndex, boolean[] included) throws IOException { - return readRowIndex(stripeIndex, included, null); + Index readRowIndex(int stripeIndex, boolean[] included) throws IOException { + return readRowIndex(stripeIndex, included, null, null, null); } - OrcProto.RowIndex[] readRowIndex( - int stripeIndex, boolean[] included, OrcProto.RowIndex[] indexes) throws IOException { + Index readRowIndex(int stripeIndex, boolean[] included, OrcProto.RowIndex[] indexes, + OrcProto.BloomFilterIndex[] bloomFilterIndex, boolean[] sargColumns) throws IOException { StripeInformation stripe = stripes.get(stripeIndex); OrcProto.StripeFooter stripeFooter = null; // if this is the current stripe, use the cached objects. if (stripeIndex == currentStripe) { stripeFooter = this.stripeFooter; indexes = indexes == null ? this.indexes : indexes; + bloomFilterIndex = bloomFilterIndex == null ? this.bloomFilterIndices : bloomFilterIndex; } - return metadata.readRowIndex(stripe, stripeFooter, included, indexes); + return metadata.readRowIndex(stripe, stripeFooter, included, indexes, sargColumns, + bloomFilterIndex); } private void seekToRowEntry(TreeReader reader, int rowEntry) throws IOException { @@ -3278,10 +3422,10 @@ public void seekToRow(long rowNumber) throws IOException { if (rowNumber < 0) { throw new IllegalArgumentException("Seek to a negative row number " + - rowNumber); + rowNumber); } else if (rowNumber < firstRow) { throw new IllegalArgumentException("Seek before reader range " + - rowNumber); + rowNumber); } // convert to our internal form (rows from the beginning of slice) rowNumber -= firstRow; Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderUtils.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderUtils.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderUtils.java (working copy) @@ -46,7 +46,7 @@ List streamList, List types) { boolean[] hasNull = new boolean[types.size()]; for(OrcProto.Stream stream: streamList) { - if (stream.getKind() == OrcProto.Stream.Kind.PRESENT) { + if (stream.hasKind() && (stream.getKind() == OrcProto.Stream.Kind.PRESENT)) { hasNull[stream.getColumn()] = true; } } Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/StreamName.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/orc/StreamName.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/StreamName.java (working copy) @@ -75,6 +75,7 @@ switch (kind) { case ROW_INDEX: case DICTIONARY_COUNT: + case BLOOM_FILTER: return Area.INDEX; default: return Area.DATA; Index: ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java (working copy) @@ -26,6 +26,7 @@ import java.nio.ByteBuffer; import java.sql.Timestamp; import java.util.ArrayList; +import java.util.Arrays; import java.util.EnumSet; import java.util.List; import java.util.Map; @@ -40,6 +41,7 @@ import org.apache.hadoop.hive.common.type.HiveDecimal; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.ql.io.IOConstants; +import org.apache.hadoop.hive.ql.io.filters.BloomFilter; import org.apache.hadoop.hive.ql.io.orc.CompressionCodec.Modifier; import org.apache.hadoop.hive.ql.io.orc.OrcFile.CompressionStrategy; import org.apache.hadoop.hive.ql.io.orc.OrcFile.EncodingStrategy; @@ -77,7 +79,9 @@ import org.apache.hadoop.io.Text; import com.google.common.annotations.VisibleForTesting; +import com.google.common.base.Joiner; import com.google.common.collect.Lists; +import com.google.common.primitives.Longs; import com.google.protobuf.ByteString; import com.google.protobuf.CodedOutputStream; @@ -145,23 +149,27 @@ private final OrcFile.WriterContext callbackContext; private final OrcFile.EncodingStrategy encodingStrategy; private final OrcFile.CompressionStrategy compressionStrategy; + private final boolean[] bloomFilterColumns; + private final double bloomFilterFpp; WriterImpl(FileSystem fs, - Path path, - Configuration conf, - ObjectInspector inspector, - long stripeSize, - CompressionKind compress, - int bufferSize, - int rowIndexStride, - MemoryManager memoryManager, - boolean addBlockPadding, - OrcFile.Version version, - OrcFile.WriterCallback callback, - OrcFile.EncodingStrategy encodingStrategy, - CompressionStrategy compressionStrategy, - float paddingTolerance, - long blockSizeValue) throws IOException { + Path path, + Configuration conf, + ObjectInspector inspector, + long stripeSize, + CompressionKind compress, + int bufferSize, + int rowIndexStride, + MemoryManager memoryManager, + boolean addBlockPadding, + OrcFile.Version version, + OrcFile.WriterCallback callback, + EncodingStrategy encodingStrategy, + CompressionStrategy compressionStrategy, + float paddingTolerance, + long blockSizeValue, + String bloomFilterColumnNames, + double bloomFilterFpp) throws IOException { this.fs = fs; this.path = path; this.conf = conf; @@ -190,7 +198,20 @@ this.memoryManager = memoryManager; buildIndex = rowIndexStride > 0; codec = createCodec(compress); - this.bufferSize = getEstimatedBufferSize(bufferSize); + String allColumns = conf.get(IOConstants.COLUMNS); + if (allColumns == null) { + allColumns = getColumnNamesFromInspector(inspector); + } + this.bufferSize = getEstimatedBufferSize(allColumns, bufferSize); + if (version == OrcFile.Version.V_0_11) { + /* do not write bloom filters for ORC v11 */ + this.bloomFilterColumns = + OrcUtils.includeColumns(null, allColumns, inspector); + } else { + this.bloomFilterColumns = + OrcUtils.includeColumns(bloomFilterColumnNames, allColumns, inspector); + } + this.bloomFilterFpp = bloomFilterFpp; treeWriter = createTreeWriter(inspector, streamFactory, false); if (buildIndex && rowIndexStride < MIN_ROW_INDEX_STRIDE) { throw new IllegalArgumentException("Row stride must be at least " + @@ -201,8 +222,25 @@ memoryManager.addWriter(path, stripeSize, this); } + private String getColumnNamesFromInspector(ObjectInspector inspector) { + List fieldNames = Lists.newArrayList(); + Joiner joiner = Joiner.on(","); + if (inspector instanceof StructObjectInspector) { + StructObjectInspector soi = (StructObjectInspector) inspector; + List fields = soi.getAllStructFieldRefs(); + for(StructField sf : fields) { + fieldNames.add(sf.getFieldName()); + } + } + return joiner.join(fieldNames); + } + + @VisibleForTesting int getEstimatedBufferSize(int bs) { - String colNames = conf.get(IOConstants.COLUMNS); + return getEstimatedBufferSize(conf.get(IOConstants.COLUMNS), bs); + } + + int getEstimatedBufferSize(String colNames, int bs) { long availableMem = getMemoryAvailableForORC(); if (colNames != null) { final int numCols = colNames.split(",").length; @@ -459,26 +497,27 @@ final EnumSet modifiers; switch (kind) { - case DATA: - case DICTIONARY_DATA: - if (getCompressionStrategy() == CompressionStrategy.SPEED) { - modifiers = EnumSet.of(Modifier.FAST, Modifier.TEXT); - } else { - modifiers = EnumSet.of(Modifier.DEFAULT, Modifier.TEXT); - } - break; - case LENGTH: - case DICTIONARY_COUNT: - case PRESENT: - case ROW_INDEX: - case SECONDARY: - // easily compressed using the fastest modes - modifiers = EnumSet.of(Modifier.FASTEST, Modifier.BINARY); - break; - default: - LOG.warn("Missing ORC compression modifiers for " + kind); - modifiers = null; - break; + case BLOOM_FILTER: + case DATA: + case DICTIONARY_DATA: + if (getCompressionStrategy() == CompressionStrategy.SPEED) { + modifiers = EnumSet.of(Modifier.FAST, Modifier.TEXT); + } else { + modifiers = EnumSet.of(Modifier.DEFAULT, Modifier.TEXT); + } + break; + case LENGTH: + case DICTIONARY_COUNT: + case PRESENT: + case ROW_INDEX: + case SECONDARY: + // easily compressed using the fastest modes + modifiers = EnumSet.of(Modifier.FASTEST, Modifier.BINARY); + break; + default: + LOG.warn("Missing ORC compression modifiers for " + kind); + modifiers = null; + break; } BufferedStream result = streams.get(name); @@ -499,6 +538,15 @@ } /** + * Get the current column id. After creating all tree writers this count should tell how many + * columns (including columns within nested complex objects) are created in total. + * @return current column id + */ + public int getCurrentColumnId() { + return columnCount; + } + + /** * Get the stride rate of the row index. */ public int getRowIndexStride() { @@ -538,6 +586,22 @@ } /** + * Get the bloom filter columns + * @return bloom filter columns + */ + public boolean[] getBloomFilterColumns() { + return bloomFilterColumns; + } + + /** + * Get bloom filter false positive percentage. + * @return fpp + */ + public double getBloomFilterFPP() { + return bloomFilterFpp; + } + + /** * Get the writer's configuration. * @return configuration */ @@ -572,6 +636,11 @@ private final OrcProto.RowIndex.Builder rowIndex; private final OrcProto.RowIndexEntry.Builder rowIndexEntry; private final PositionedOutputStream rowIndexStream; + private final PositionedOutputStream bloomFilterStream; + protected final BloomFilter bloomFilter; + protected final boolean createBloomFilter; + private final OrcProto.BloomFilterIndex.Builder bloomFilterIndex; + private final OrcProto.BloomFilter.Builder bloomFilterEntry; private boolean foundNulls; private OutStream isPresentOutStream; private final List stripeStatsBuilders; @@ -598,6 +667,7 @@ isPresent = null; } this.foundNulls = false; + createBloomFilter = streamFactory.getBloomFilterColumns()[columnId]; indexStatistics = ColumnStatisticsImpl.create(inspector); stripeColStatistics = ColumnStatisticsImpl.create(inspector); fileStatistics = ColumnStatisticsImpl.create(inspector); @@ -607,11 +677,22 @@ rowIndexPosition = new RowIndexPositionRecorder(rowIndexEntry); stripeStatsBuilders = Lists.newArrayList(); if (streamFactory.buildIndex()) { - rowIndexStream = streamFactory.createStream(id, - OrcProto.Stream.Kind.ROW_INDEX); + rowIndexStream = streamFactory.createStream(id, OrcProto.Stream.Kind.ROW_INDEX); } else { rowIndexStream = null; } + if (createBloomFilter) { + bloomFilterEntry = OrcProto.BloomFilter.newBuilder(); + bloomFilterIndex = OrcProto.BloomFilterIndex.newBuilder(); + bloomFilterStream = streamFactory.createStream(id, OrcProto.Stream.Kind.BLOOM_FILTER); + bloomFilter = new BloomFilter(streamFactory.getRowIndexStride(), + streamFactory.getBloomFilterFPP()); + } else { + bloomFilterEntry = null; + bloomFilterIndex = null; + bloomFilterStream = null; + bloomFilter = null; + } } protected OrcProto.RowIndex.Builder getRowIndex() { @@ -725,6 +806,14 @@ } rowIndex.clear(); rowIndexEntry.clear(); + + // write the bloom filter to out stream + if (bloomFilterStream != null) { + bloomFilterIndex.build().writeTo(bloomFilterStream); + bloomFilterStream.flush(); + bloomFilterIndex.clear(); + bloomFilterEntry.clear(); + } } private void writeStripeStatistics(OrcProto.StripeStatistics.Builder builder, @@ -763,12 +852,23 @@ indexStatistics.reset(); rowIndex.addEntry(rowIndexEntry); rowIndexEntry.clear(); + addBloomFilterEntry(); recordPosition(rowIndexPosition); for(TreeWriter child: childrenWriters) { child.createRowIndexEntry(); } } + void addBloomFilterEntry() { + if (createBloomFilter) { + bloomFilterEntry.setNumHashFunctions(bloomFilter.getNumHashFunctions()); + bloomFilterEntry.addAllBitset(Longs.asList(bloomFilter.getBitSet())); + bloomFilterIndex.addBloomFilter(bloomFilterEntry.build()); + bloomFilter.reset(); + bloomFilterEntry.clear(); + } + } + /** * Record the current position in each of this column's streams. * @param recorder where should the locations be recorded @@ -851,6 +951,9 @@ if (obj != null) { byte val = ((ByteObjectInspector) inspector).get(obj); indexStatistics.updateInteger(val); + if (createBloomFilter) { + bloomFilter.addLong(val); + } writer.write(val); } } @@ -926,6 +1029,10 @@ val = shortInspector.get(obj); } indexStatistics.updateInteger(val); + if (createBloomFilter) { + // integers are converted to longs in column statistics and during SARG evaluation + bloomFilter.addLong(val); + } writer.write(val); } } @@ -966,6 +1073,10 @@ if (obj != null) { float val = ((FloatObjectInspector) inspector).get(obj); indexStatistics.updateDouble(val); + if (createBloomFilter) { + // floats are converted to doubles in column statistics and during SARG evaluation + bloomFilter.addDouble(val); + } utils.writeFloat(stream, val); } } @@ -1006,6 +1117,9 @@ if (obj != null) { double val = ((DoubleObjectInspector) inspector).get(obj); indexStatistics.updateDouble(val); + if (createBloomFilter) { + bloomFilter.addDouble(val); + } utils.writeDouble(stream, val); } } @@ -1099,6 +1213,9 @@ directLengthOutput.write(val.getLength()); } indexStatistics.updateString(val); + if (createBloomFilter) { + bloomFilter.addBytes(val.getBytes(), val.getLength()); + } } } @@ -1258,6 +1375,7 @@ OrcProto.RowIndexEntry base = rowIndexEntry.build(); savedRowIndex.add(base); rowIndexEntry.clear(); + addBloomFilterEntry(); recordPosition(rowIndexPosition); rowIndexValueCount.add(Long.valueOf(rows.size())); if (strideDictionaryCheck) { @@ -1368,6 +1486,9 @@ stream.write(val.getBytes(), 0, val.getLength()); length.write(val.getLength()); indexStatistics.updateBinary(val); + if (createBloomFilter) { + bloomFilter.addBytes(val.getBytes(), val.getLength()); + } } } @@ -1430,6 +1551,9 @@ indexStatistics.updateTimestamp(val); seconds.write((val.getTime() / MILLIS_PER_SECOND) - BASE_TIMESTAMP); nanos.write(formatNanos(val.getNanos())); + if (createBloomFilter) { + bloomFilter.addLong(val.getTime()); + } } } @@ -1490,6 +1614,9 @@ DateWritable val = ((DateObjectInspector) inspector).getPrimitiveWritableObject(obj); indexStatistics.updateDate(val); writer.write(val.getDays()); + if (createBloomFilter) { + bloomFilter.addLong(val.getDays()); + } } } @@ -1558,6 +1685,9 @@ decimal.unscaledValue()); scaleStream.write(decimal.scale()); indexStatistics.updateDecimal(decimal); + if (createBloomFilter) { + bloomFilter.addString(decimal.toString()); + } } } @@ -1657,6 +1787,9 @@ ListObjectInspector insp = (ListObjectInspector) inspector; int len = insp.getListLength(obj); lengths.write(len); + if (createBloomFilter) { + bloomFilter.addLong(len); + } for(int i=0; i < len; ++i) { childrenWriters[0].write(insp.getListElement(obj, i)); } @@ -1721,6 +1854,9 @@ // accessor in the MapObjectInspector. Map valueMap = insp.getMap(obj); lengths.write(valueMap.size()); + if (createBloomFilter) { + bloomFilter.addLong(valueMap.size()); + } for(Map.Entry entry: valueMap.entrySet()) { childrenWriters[0].write(entry.getKey()); childrenWriters[1].write(entry.getValue()); @@ -1773,6 +1909,9 @@ UnionObjectInspector insp = (UnionObjectInspector) inspector; byte tag = insp.getTag(obj); tags.write(tag); + if (createBloomFilter) { + bloomFilter.addLong(tag); + } childrenWriters[tag].write(insp.getField(obj)); } } Index: ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/parquet/MapredParquetOutputFormat.java (working copy) @@ -21,7 +21,6 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.ql.io.HiveOutputFormat; @@ -29,10 +28,10 @@ import org.apache.hadoop.hive.ql.io.parquet.convert.HiveSchemaConverter; import org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport; import org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper; +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils; import org.apache.hadoop.hive.shims.ShimLoader; -import org.apache.hadoop.io.ArrayWritable; import org.apache.hadoop.io.Writable; import org.apache.hadoop.mapred.FileOutputFormat; import org.apache.hadoop.mapred.JobConf; @@ -41,27 +40,25 @@ import org.apache.hadoop.util.Progressable; import parquet.hadoop.ParquetOutputFormat; -import parquet.hadoop.metadata.CompressionCodecName; -import parquet.hadoop.util.ContextUtil; /** * * A Parquet OutputFormat for Hive (with the deprecated package mapred) * */ -public class MapredParquetOutputFormat extends FileOutputFormat implements - HiveOutputFormat { +public class MapredParquetOutputFormat extends FileOutputFormat implements + HiveOutputFormat { private static final Log LOG = LogFactory.getLog(MapredParquetOutputFormat.class); - protected ParquetOutputFormat realOutputFormat; + protected ParquetOutputFormat realOutputFormat; public MapredParquetOutputFormat() { - realOutputFormat = new ParquetOutputFormat(new DataWritableWriteSupport()); + realOutputFormat = new ParquetOutputFormat(new DataWritableWriteSupport()); } - public MapredParquetOutputFormat(final OutputFormat mapreduceOutputFormat) { - realOutputFormat = (ParquetOutputFormat) mapreduceOutputFormat; + public MapredParquetOutputFormat(final OutputFormat mapreduceOutputFormat) { + realOutputFormat = (ParquetOutputFormat) mapreduceOutputFormat; } @Override @@ -70,7 +67,7 @@ } @Override - public RecordWriter getRecordWriter( + public RecordWriter getRecordWriter( final FileSystem ignored, final JobConf job, final String name, @@ -119,7 +116,7 @@ } protected ParquetRecordWriterWrapper getParquerRecordWriterWrapper( - ParquetOutputFormat realOutputFormat, + ParquetOutputFormat realOutputFormat, JobConf jobConf, String finalOutPath, Progressable progress, Index: ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java (working copy) @@ -13,61 +13,31 @@ */ package org.apache.hadoop.hive.ql.io.parquet.serde; -import java.io.UnsupportedEncodingException; import java.util.ArrayList; import java.util.Arrays; import java.util.List; -import java.util.Map; -import java.util.Map.Entry; import java.util.Properties; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hive.common.type.HiveDecimal; import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.AbstractSerDe; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeSpec; import org.apache.hadoop.hive.serde2.SerDeStats; -import org.apache.hadoop.hive.serde2.io.ByteWritable; -import org.apache.hadoop.hive.serde2.io.DoubleWritable; -import org.apache.hadoop.hive.serde2.io.ShortWritable; -import org.apache.hadoop.hive.serde2.io.TimestampWritable; -import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.MapObjectInspector; +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector.Category; -import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.ByteObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.DoubleObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.FloatObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.HiveCharObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.HiveVarcharObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.IntObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.LongObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.ShortObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.StructField; import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; -import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo; import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils; import org.apache.hadoop.io.ArrayWritable; -import org.apache.hadoop.io.BooleanWritable; -import org.apache.hadoop.io.BytesWritable; -import org.apache.hadoop.io.FloatWritable; -import org.apache.hadoop.io.IntWritable; -import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.io.Writable; import parquet.hadoop.ParquetOutputFormat; import parquet.hadoop.ParquetWriter; -import parquet.io.api.Binary; /** * @@ -110,6 +80,13 @@ private long deserializedSize; private String compressionType; + private ParquetHiveRecord parquetRow; + + public ParquetHiveSerDe() { + parquetRow = new ParquetHiveRecord(); + stats = new SerDeStats(); + } + @Override public final void initialize(final Configuration conf, final Properties tbl) throws SerDeException { @@ -144,7 +121,6 @@ this.objInspector = new ArrayWritableObjectInspector((StructTypeInfo) rowTypeInfo); // Stats part - stats = new SerDeStats(); serializedSize = 0; deserializedSize = 0; status = LAST_OPERATION.UNKNOWN; @@ -169,7 +145,7 @@ @Override public Class getSerializedClass() { - return ArrayWritable.class; + return ParquetHiveRecord.class; } @Override @@ -178,156 +154,13 @@ if (!objInspector.getCategory().equals(Category.STRUCT)) { throw new SerDeException("Cannot serialize " + objInspector.getCategory() + ". Can only serialize a struct"); } - final ArrayWritable serializeData = createStruct(obj, (StructObjectInspector) objInspector); - serializedSize = serializeData.get().length; + serializedSize = ((StructObjectInspector)objInspector).getAllStructFieldRefs().size(); status = LAST_OPERATION.SERIALIZE; - return serializeData; + parquetRow.value = obj; + parquetRow.inspector= (StructObjectInspector)objInspector; + return parquetRow; } - private ArrayWritable createStruct(final Object obj, final StructObjectInspector inspector) - throws SerDeException { - final List fields = inspector.getAllStructFieldRefs(); - final Writable[] arr = new Writable[fields.size()]; - for (int i = 0; i < fields.size(); i++) { - final StructField field = fields.get(i); - final Object subObj = inspector.getStructFieldData(obj, field); - final ObjectInspector subInspector = field.getFieldObjectInspector(); - arr[i] = createObject(subObj, subInspector); - } - return new ArrayWritable(Writable.class, arr); - } - - private Writable createMap(final Object obj, final MapObjectInspector inspector) - throws SerDeException { - final Map sourceMap = inspector.getMap(obj); - final ObjectInspector keyInspector = inspector.getMapKeyObjectInspector(); - final ObjectInspector valueInspector = inspector.getMapValueObjectInspector(); - final List array = new ArrayList(); - - if (sourceMap != null) { - for (final Entry keyValue : sourceMap.entrySet()) { - final Writable key = createObject(keyValue.getKey(), keyInspector); - final Writable value = createObject(keyValue.getValue(), valueInspector); - if (key != null) { - Writable[] arr = new Writable[2]; - arr[0] = key; - arr[1] = value; - array.add(new ArrayWritable(Writable.class, arr)); - } - } - } - if (array.size() > 0) { - final ArrayWritable subArray = new ArrayWritable(ArrayWritable.class, - array.toArray(new ArrayWritable[array.size()])); - return new ArrayWritable(Writable.class, new Writable[] {subArray}); - } else { - return null; - } - } - - private ArrayWritable createArray(final Object obj, final ListObjectInspector inspector) - throws SerDeException { - final List sourceArray = inspector.getList(obj); - final ObjectInspector subInspector = inspector.getListElementObjectInspector(); - final List array = new ArrayList(); - if (sourceArray != null) { - for (final Object curObj : sourceArray) { - array.add(createObject(curObj, subInspector)); - } - } - if (array.size() > 0) { - final ArrayWritable subArray = new ArrayWritable(Writable.class, - array.toArray(new Writable[array.size()])); - return new ArrayWritable(Writable.class, new Writable[] {subArray}); - } else { - return null; - } - } - - private Writable createPrimitive(final Object obj, final PrimitiveObjectInspector inspector) - throws SerDeException { - if (obj == null) { - return null; - } - switch (inspector.getPrimitiveCategory()) { - case VOID: - return null; - case BOOLEAN: - return new BooleanWritable(((BooleanObjectInspector) inspector).get(obj) ? Boolean.TRUE : Boolean.FALSE); - case BYTE: - return new ByteWritable(((ByteObjectInspector) inspector).get(obj)); - case DOUBLE: - return new DoubleWritable(((DoubleObjectInspector) inspector).get(obj)); - case FLOAT: - return new FloatWritable(((FloatObjectInspector) inspector).get(obj)); - case INT: - return new IntWritable(((IntObjectInspector) inspector).get(obj)); - case LONG: - return new LongWritable(((LongObjectInspector) inspector).get(obj)); - case SHORT: - return new ShortWritable(((ShortObjectInspector) inspector).get(obj)); - case STRING: - String v = ((StringObjectInspector) inspector).getPrimitiveJavaObject(obj); - try { - return new BytesWritable(v.getBytes("UTF-8")); - } catch (UnsupportedEncodingException e) { - throw new SerDeException("Failed to encode string in UTF-8", e); - } - case DECIMAL: - HiveDecimal hd = (HiveDecimal)inspector.getPrimitiveJavaObject(obj); - DecimalTypeInfo decTypeInfo = (DecimalTypeInfo) inspector.getTypeInfo(); - int prec = decTypeInfo.precision(); - int scale = decTypeInfo.scale(); - byte[] src = hd.setScale(scale).unscaledValue().toByteArray(); - // Estimated number of bytes needed. - int bytes = PRECISION_TO_BYTE_COUNT[prec - 1]; - if (bytes == src.length) { - // No padding needed. - return new BytesWritable(src); - } - byte[] tgt = new byte[bytes]; - if ( hd.signum() == -1) { - // For negative number, initializing bits to 1 - for (int i = 0; i < bytes; i++) { - tgt[i] |= 0xFF; - } - } - System.arraycopy(src, 0, tgt, bytes - src.length, src.length); // Padding leading zeroes/ones. - return new BytesWritable(tgt); - case TIMESTAMP: - return new TimestampWritable(((TimestampObjectInspector) inspector).getPrimitiveJavaObject(obj)); - case CHAR: - String strippedValue = ((HiveCharObjectInspector) inspector).getPrimitiveJavaObject(obj).getStrippedValue(); - return new BytesWritable(Binary.fromString(strippedValue).getBytes()); - case VARCHAR: - String value = ((HiveVarcharObjectInspector) inspector).getPrimitiveJavaObject(obj).getValue(); - return new BytesWritable(Binary.fromString(value).getBytes()); - case BINARY: - return new BytesWritable(((BinaryObjectInspector) inspector).getPrimitiveJavaObject(obj)); - default: - throw new SerDeException("Unknown primitive : " + inspector.getPrimitiveCategory()); - } - } - - private Writable createObject(final Object obj, final ObjectInspector inspector) throws SerDeException { - if (obj == null) { - return null; - } - - switch (inspector.getCategory()) { - case STRUCT: - return createStruct(obj, (StructObjectInspector) inspector); - case LIST: - return createArray(obj, (ListObjectInspector) inspector); - case MAP: - return createMap(obj, (MapObjectInspector) inspector); - case PRIMITIVE: - return createPrimitive(obj, (PrimitiveObjectInspector) inspector); - default: - throw new SerDeException("Unknown data type" + inspector.getCategory()); - } - } - @Override public SerDeStats getSerDeStats() { // must be different Index: ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriteSupport.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriteSupport.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriteSupport.java (working copy) @@ -16,7 +16,7 @@ import java.util.HashMap; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.io.ArrayWritable; +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; import parquet.hadoop.api.WriteSupport; import parquet.io.api.RecordConsumer; @@ -28,7 +28,7 @@ * DataWritableWriteSupport is a WriteSupport for the DataWritableWriter * */ -public class DataWritableWriteSupport extends WriteSupport { +public class DataWritableWriteSupport extends WriteSupport { public static final String PARQUET_HIVE_SCHEMA = "parquet.hive.schema"; @@ -55,7 +55,7 @@ } @Override - public void write(final ArrayWritable record) { + public void write(final ParquetHiveRecord record) { writer.write(record); } } Index: ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/DataWritableWriter.java (working copy) @@ -13,37 +13,29 @@ */ package org.apache.hadoop.hive.ql.io.parquet.write; -import java.sql.Timestamp; - import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.hive.ql.io.parquet.timestamp.NanoTime; +import org.apache.hadoop.hive.common.type.HiveDecimal; +import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe; import org.apache.hadoop.hive.ql.io.parquet.timestamp.NanoTimeUtils; -import org.apache.hadoop.hive.serde2.io.ByteWritable; -import org.apache.hadoop.hive.serde2.io.DoubleWritable; -import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable; -import org.apache.hadoop.hive.serde2.io.ShortWritable; -import org.apache.hadoop.hive.serde2.io.TimestampWritable; -import org.apache.hadoop.io.ArrayWritable; -import org.apache.hadoop.io.BooleanWritable; -import org.apache.hadoop.io.BytesWritable; -import org.apache.hadoop.io.FloatWritable; -import org.apache.hadoop.io.IntWritable; -import org.apache.hadoop.io.LongWritable; -import org.apache.hadoop.io.Writable; - +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; +import org.apache.hadoop.hive.serde2.objectinspector.*; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.*; +import org.apache.hadoop.hive.serde2.typeinfo.DecimalTypeInfo; import parquet.io.api.Binary; import parquet.io.api.RecordConsumer; import parquet.schema.GroupType; import parquet.schema.OriginalType; import parquet.schema.Type; +import java.sql.Timestamp; +import java.util.List; +import java.util.Map; + /** * - * DataWritableWriter is a writer, - * that will read an ArrayWritable and give the data to parquet - * with the expected schema - * This is a helper class used by DataWritableWriteSupport class. + * DataWritableWriter is a writer that reads a ParquetWritable object and send the data to the Parquet + * API with the expected schema. This class is only used through DataWritableWriteSupport class. */ public class DataWritableWriter { private static final Log LOG = LogFactory.getLog(DataWritableWriter.class); @@ -57,13 +49,13 @@ /** * It writes all record values to the Parquet RecordConsumer. - * @param record Contains the record of values that are going to be written + * @param record Contains the record that are going to be written. */ - public void write(final ArrayWritable record) { + public void write(final ParquetHiveRecord record) { if (record != null) { recordConsumer.startMessage(); try { - writeGroupFields(record, schema); + writeGroupFields(record.getObject(), record.getObjectInspector(), schema); } catch (RuntimeException e) { String errorMessage = "Parquet record is malformed: " + e.getMessage(); LOG.error(errorMessage, e); @@ -76,19 +68,23 @@ /** * It writes all the fields contained inside a group to the RecordConsumer. * @param value The list of values contained in the group. + * @param inspector The object inspector used to get the correct value type. * @param type Type that contains information about the group schema. */ - public void writeGroupFields(final ArrayWritable value, final GroupType type) { + private void writeGroupFields(final Object value, final StructObjectInspector inspector, final GroupType type) { if (value != null) { + List fields = inspector.getAllStructFieldRefs(); + List fieldValuesList = inspector.getStructFieldsDataAsList(value); + for (int i = 0; i < type.getFieldCount(); i++) { Type fieldType = type.getType(i); String fieldName = fieldType.getName(); - Writable fieldValue = value.get()[i]; + Object fieldValue = fieldValuesList.get(i); - // Parquet does not write null elements if (fieldValue != null) { + ObjectInspector fieldInspector = fields.get(i).getFieldObjectInspector(); recordConsumer.startField(fieldName, i); - writeValue(fieldValue, fieldType); + writeValue(fieldValue, fieldInspector, fieldType); recordConsumer.endField(fieldName, i); } } @@ -96,68 +92,93 @@ } /** - * It writes the field value to the Parquet RecordConsumer. It detects the field type, and writes + * It writes the field value to the Parquet RecordConsumer. It detects the field type, and calls * the correct write function. * @param value The writable object that contains the value. + * @param inspector The object inspector used to get the correct value type. * @param type Type that contains information about the type schema. */ - private void writeValue(final Writable value, final Type type) { + private void writeValue(final Object value, final ObjectInspector inspector, final Type type) { if (type.isPrimitive()) { - writePrimitive(value); - } else if (value instanceof ArrayWritable) { + checkInspectorCategory(inspector, ObjectInspector.Category.PRIMITIVE); + writePrimitive(value, (PrimitiveObjectInspector)inspector); + } else { GroupType groupType = type.asGroupType(); OriginalType originalType = type.getOriginalType(); if (originalType != null && originalType.equals(OriginalType.LIST)) { - writeArray((ArrayWritable)value, groupType); + checkInspectorCategory(inspector, ObjectInspector.Category.LIST); + writeArray(value, (ListObjectInspector)inspector, groupType); } else if (originalType != null && originalType.equals(OriginalType.MAP)) { - writeMap((ArrayWritable)value, groupType); + checkInspectorCategory(inspector, ObjectInspector.Category.MAP); + writeMap(value, (MapObjectInspector)inspector, groupType); } else { - writeGroup((ArrayWritable) value, groupType); + checkInspectorCategory(inspector, ObjectInspector.Category.STRUCT); + writeGroup(value, (StructObjectInspector)inspector, groupType); } - } else { - throw new RuntimeException("Field value is not an ArrayWritable object: " + type); } } /** + * Checks that an inspector matches the category indicated as a parameter. + * @param inspector The object inspector to check + * @param category The category to match + * @throws IllegalArgumentException if inspector does not match the category + */ + private void checkInspectorCategory(ObjectInspector inspector, ObjectInspector.Category category) { + if (!inspector.getCategory().equals(category)) { + throw new IllegalArgumentException("Invalid data type: expected " + category + + " type, but found: " + inspector.getCategory()); + } + } + + /** * It writes a group type and all its values to the Parquet RecordConsumer. * This is used only for optional and required groups. - * @param value ArrayWritable object that contains the group values - * @param type Type that contains information about the group schema + * @param value Object that contains the group values. + * @param inspector The object inspector used to get the correct value type. + * @param type Type that contains information about the group schema. */ - private void writeGroup(final ArrayWritable value, final GroupType type) { + private void writeGroup(final Object value, final StructObjectInspector inspector, final GroupType type) { recordConsumer.startGroup(); - writeGroupFields(value, type); + writeGroupFields(value, inspector, type); recordConsumer.endGroup(); } /** - * It writes a map type and its key-pair values to the Parquet RecordConsumer. - * This is called when the original type (MAP) is detected by writeValue() - * @param value The list of map values that contains the repeated KEY_PAIR_VALUE group type - * @param type Type that contains information about the group schema + * It writes a list type and its array elements to the Parquet RecordConsumer. + * This is called when the original type (LIST) is detected by writeValue()/ + * This function assumes the following schema: + * optional group arrayCol (LIST) { + * repeated group array { + * optional TYPE array_element; + * } + * } + * @param value The object that contains the array values. + * @param inspector The object inspector used to get the correct value type. + * @param type Type that contains information about the group (LIST) schema. */ - private void writeMap(final ArrayWritable value, final GroupType type) { + private void writeArray(final Object value, final ListObjectInspector inspector, final GroupType type) { + // Get the internal array structure GroupType repeatedType = type.getType(0).asGroupType(); - ArrayWritable repeatedValue = (ArrayWritable)value.get()[0]; recordConsumer.startGroup(); recordConsumer.startField(repeatedType.getName(), 0); - Writable[] map_values = repeatedValue.get(); - for (int record = 0; record < map_values.length; record++) { - Writable key_value_pair = map_values[record]; - if (key_value_pair != null) { - // Hive wraps a map key-pair into an ArrayWritable - if (key_value_pair instanceof ArrayWritable) { - writeGroup((ArrayWritable)key_value_pair, repeatedType); - } else { - throw new RuntimeException("Map key-value pair is not an ArrayWritable object on record " + record); - } - } else { - throw new RuntimeException("Map key-value pair is null on record " + record); + List arrayValues = inspector.getList(value); + ObjectInspector elementInspector = inspector.getListElementObjectInspector(); + + Type elementType = repeatedType.getType(0); + String elementName = elementType.getName(); + + for (Object element : arrayValues) { + recordConsumer.startGroup(); + if (element != null) { + recordConsumer.startField(elementName, 0); + writeValue(element, elementInspector, elementType); + recordConsumer.endField(elementName, 0); } + recordConsumer.endGroup(); } recordConsumer.endField(repeatedType.getName(), 0); @@ -165,35 +186,53 @@ } /** - * It writes a list type and its array elements to the Parquet RecordConsumer. - * This is called when the original type (LIST) is detected by writeValue() - * @param array The list of array values that contains the repeated array group type - * @param type Type that contains information about the group schema + * It writes a map type and its key-pair values to the Parquet RecordConsumer. + * This is called when the original type (MAP) is detected by writeValue(). + * This function assumes the following schema: + * optional group mapCol (MAP) { + * repeated group map (MAP_KEY_VALUE) { + * required TYPE key; + * optional TYPE value; + * } + * } + * @param value The object that contains the map key-values. + * @param inspector The object inspector used to get the correct value type. + * @param type Type that contains information about the group (MAP) schema. */ - private void writeArray(final ArrayWritable array, final GroupType type) { + private void writeMap(final Object value, final MapObjectInspector inspector, final GroupType type) { + // Get the internal map structure (MAP_KEY_VALUE) GroupType repeatedType = type.getType(0).asGroupType(); - ArrayWritable repeatedValue = (ArrayWritable)array.get()[0]; recordConsumer.startGroup(); recordConsumer.startField(repeatedType.getName(), 0); - Writable[] array_values = repeatedValue.get(); - for (int record = 0; record < array_values.length; record++) { + Map mapValues = inspector.getMap(value); + + Type keyType = repeatedType.getType(0); + String keyName = keyType.getName(); + ObjectInspector keyInspector = inspector.getMapKeyObjectInspector(); + + Type valuetype = repeatedType.getType(1); + String valueName = valuetype.getName(); + ObjectInspector valueInspector = inspector.getMapValueObjectInspector(); + + for (Map.Entry keyValue : mapValues.entrySet()) { recordConsumer.startGroup(); + if (keyValue != null) { + // write key element + Object keyElement = keyValue.getKey(); + recordConsumer.startField(keyName, 0); + writeValue(keyElement, keyInspector, keyType); + recordConsumer.endField(keyName, 0); - // Null values must be wrapped into startGroup/endGroup - Writable element = array_values[record]; - if (element != null) { - for (int i = 0; i < type.getFieldCount(); i++) { - Type fieldType = repeatedType.getType(i); - String fieldName = fieldType.getName(); - - recordConsumer.startField(fieldName, i); - writeValue(element, fieldType); - recordConsumer.endField(fieldName, i); + // write value element + Object valueElement = keyValue.getValue(); + if (valueElement != null) { + recordConsumer.startField(valueName, 1); + writeValue(valueElement, valueInspector, valuetype); + recordConsumer.endField(valueName, 1); } } - recordConsumer.endGroup(); } @@ -203,36 +242,89 @@ /** * It writes the primitive value to the Parquet RecordConsumer. - * @param value The writable object that contains the primitive value. + * @param value The object that contains the primitive value. + * @param inspector The object inspector used to get the correct value type. */ - private void writePrimitive(final Writable value) { + private void writePrimitive(final Object value, final PrimitiveObjectInspector inspector) { if (value == null) { return; } - if (value instanceof DoubleWritable) { - recordConsumer.addDouble(((DoubleWritable) value).get()); - } else if (value instanceof BooleanWritable) { - recordConsumer.addBoolean(((BooleanWritable) value).get()); - } else if (value instanceof FloatWritable) { - recordConsumer.addFloat(((FloatWritable) value).get()); - } else if (value instanceof IntWritable) { - recordConsumer.addInteger(((IntWritable) value).get()); - } else if (value instanceof LongWritable) { - recordConsumer.addLong(((LongWritable) value).get()); - } else if (value instanceof ShortWritable) { - recordConsumer.addInteger(((ShortWritable) value).get()); - } else if (value instanceof ByteWritable) { - recordConsumer.addInteger(((ByteWritable) value).get()); - } else if (value instanceof HiveDecimalWritable) { - throw new UnsupportedOperationException("HiveDecimalWritable writing not implemented"); - } else if (value instanceof BytesWritable) { - recordConsumer.addBinary((Binary.fromByteArray(((BytesWritable) value).getBytes()))); - } else if (value instanceof TimestampWritable) { - Timestamp ts = ((TimestampWritable) value).getTimestamp(); - NanoTime nt = NanoTimeUtils.getNanoTime(ts, false); - nt.writeValue(recordConsumer); - } else { - throw new IllegalArgumentException("Unknown value type: " + value + " " + value.getClass()); + + switch (inspector.getPrimitiveCategory()) { + case VOID: + return; + case DOUBLE: + recordConsumer.addDouble(((DoubleObjectInspector) inspector).get(value)); + break; + case BOOLEAN: + recordConsumer.addBoolean(((BooleanObjectInspector) inspector).get(value)); + break; + case FLOAT: + recordConsumer.addFloat(((FloatObjectInspector) inspector).get(value)); + break; + case BYTE: + recordConsumer.addInteger(((ByteObjectInspector) inspector).get(value)); + break; + case INT: + recordConsumer.addInteger(((IntObjectInspector) inspector).get(value)); + break; + case LONG: + recordConsumer.addLong(((LongObjectInspector) inspector).get(value)); + break; + case SHORT: + recordConsumer.addInteger(((ShortObjectInspector) inspector).get(value)); + break; + case STRING: + String v = ((StringObjectInspector) inspector).getPrimitiveJavaObject(value); + recordConsumer.addBinary(Binary.fromString(v)); + break; + case CHAR: + String vChar = ((HiveCharObjectInspector) inspector).getPrimitiveJavaObject(value).getStrippedValue(); + recordConsumer.addBinary(Binary.fromString(vChar)); + break; + case VARCHAR: + String vVarchar = ((HiveVarcharObjectInspector) inspector).getPrimitiveJavaObject(value).getValue(); + recordConsumer.addBinary(Binary.fromString(vVarchar)); + break; + case BINARY: + byte[] vBinary = ((BinaryObjectInspector) inspector).getPrimitiveJavaObject(value); + recordConsumer.addBinary(Binary.fromByteArray(vBinary)); + break; + case TIMESTAMP: + Timestamp ts = ((TimestampObjectInspector) inspector).getPrimitiveJavaObject(value); + recordConsumer.addBinary(NanoTimeUtils.getNanoTime(ts, false).toBinary()); + break; + case DECIMAL: + HiveDecimal vDecimal = ((HiveDecimal)inspector.getPrimitiveJavaObject(value)); + DecimalTypeInfo decTypeInfo = (DecimalTypeInfo)inspector.getTypeInfo(); + recordConsumer.addBinary(decimalToBinary(vDecimal, decTypeInfo)); + break; + default: + throw new IllegalArgumentException("Unsupported primitive data type: " + inspector.getPrimitiveCategory()); } } + + private Binary decimalToBinary(final HiveDecimal hiveDecimal, final DecimalTypeInfo decimalTypeInfo) { + int prec = decimalTypeInfo.precision(); + int scale = decimalTypeInfo.scale(); + byte[] decimalBytes = hiveDecimal.setScale(scale).unscaledValue().toByteArray(); + + // Estimated number of bytes needed. + int precToBytes = ParquetHiveSerDe.PRECISION_TO_BYTE_COUNT[prec - 1]; + if (precToBytes == decimalBytes.length) { + // No padding needed. + return Binary.fromByteArray(decimalBytes); + } + + byte[] tgt = new byte[precToBytes]; + if (hiveDecimal.signum() == -1) { + // For negative number, initializing bits to 1 + for (int i = 0; i < precToBytes; i++) { + tgt[i] |= 0xFF; + } + } + + System.arraycopy(decimalBytes, 0, tgt, precToBytes - decimalBytes.length, decimalBytes.length); // Padding leading zeroes/ones. + return Binary.fromByteArray(tgt); + } } Index: ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/io/parquet/write/ParquetRecordWriterWrapper.java (working copy) @@ -20,7 +20,6 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.io.ArrayWritable; import org.apache.hadoop.io.Writable; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapreduce.JobContext; @@ -29,22 +28,23 @@ import org.apache.hadoop.mapreduce.OutputFormat; import org.apache.hadoop.mapreduce.TaskAttemptContext; import org.apache.hadoop.mapreduce.TaskAttemptID; +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; import org.apache.hadoop.util.Progressable; import parquet.hadoop.ParquetOutputFormat; import parquet.hadoop.metadata.CompressionCodecName; import parquet.hadoop.util.ContextUtil; -public class ParquetRecordWriterWrapper implements RecordWriter, +public class ParquetRecordWriterWrapper implements RecordWriter, org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter { public static final Log LOG = LogFactory.getLog(ParquetRecordWriterWrapper.class); - private final org.apache.hadoop.mapreduce.RecordWriter realWriter; + private final org.apache.hadoop.mapreduce.RecordWriter realWriter; private final TaskAttemptContext taskContext; public ParquetRecordWriterWrapper( - final OutputFormat realOutputFormat, + final OutputFormat realOutputFormat, final JobConf jobConf, final String name, final Progressable progress, Properties tableProperties) throws @@ -106,7 +106,7 @@ } @Override - public void write(final Void key, final ArrayWritable value) throws IOException { + public void write(final Void key, final ParquetHiveRecord value) throws IOException { try { realWriter.write(key, value); } catch (final InterruptedException e) { @@ -121,7 +121,7 @@ @Override public void write(final Writable w) throws IOException { - write(null, (ArrayWritable) w); + write(null, (ParquetHiveRecord) w); } } Index: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (working copy) @@ -29,7 +29,6 @@ import java.io.FileNotFoundException; import java.io.IOException; -import java.net.URI; import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; @@ -49,8 +48,6 @@ import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import org.apache.hadoop.fs.PathFilter; -import org.apache.hadoop.fs.permission.FsPermission; import org.apache.hadoop.hive.common.FileUtils; import org.apache.hadoop.hive.common.HiveStatsUtils; import org.apache.hadoop.hive.common.ObjectPair; @@ -73,7 +70,6 @@ import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj; import org.apache.hadoop.hive.metastore.api.CompactionType; import org.apache.hadoop.hive.metastore.api.Database; -import org.apache.hadoop.hive.metastore.api.EventRequestType; import org.apache.hadoop.hive.metastore.api.FieldSchema; import org.apache.hadoop.hive.metastore.api.FireEventRequest; import org.apache.hadoop.hive.metastore.api.FireEventRequestData; @@ -102,6 +98,8 @@ import org.apache.hadoop.hive.metastore.api.StorageDescriptor; import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants; import org.apache.hadoop.hive.ql.ErrorMsg; +import org.apache.hadoop.hive.ql.exec.FunctionRegistry; +import org.apache.hadoop.hive.ql.exec.FunctionTask; import org.apache.hadoop.hive.ql.exec.Utilities; import org.apache.hadoop.hive.ql.index.HiveIndexHandler; import org.apache.hadoop.hive.ql.io.AcidUtils; @@ -156,6 +154,31 @@ } }; + // register all permanent functions. need improvement + static { + try { + reloadFunctions(); + } catch (Exception e) { + LOG.warn("Failed to access metastore. This class should not accessed in runtime.",e); + } + } + + public static void reloadFunctions() throws HiveException { + Hive db = Hive.get(); + for (String dbName : db.getAllDatabases()) { + for (String functionName : db.getFunctions(dbName, "*")) { + Function function = db.getFunction(dbName, functionName); + try { + FunctionRegistry.registerPermanentFunction(functionName, function.getClassName(), false, + FunctionTask.toFunctionResource(function.getResourceUris())); + } catch (Exception e) { + LOG.warn("Failed to register persistent function " + + functionName + ":" + function.getClassName() + ". Ignore and continue."); + } + } + } + } + public static Hive get(Configuration c, Class clazz) throws HiveException { return get(c instanceof HiveConf ? (HiveConf)c : new HiveConf(c, clazz)); } @@ -1590,10 +1613,18 @@ throws HiveException { List newFiles = new ArrayList(); Table tbl = getTable(tableName); + HiveConf sessionConf = SessionState.getSessionConf(); if (replace) { - tbl.replaceFiles(loadPath, isSrcLocal); + Path tableDest = tbl.getPath(); + replaceFiles(tableDest, loadPath, tableDest, tableDest, sessionConf, isSrcLocal); } else { - tbl.copyFiles(loadPath, isSrcLocal, isAcid, newFiles); + FileSystem fs; + try { + fs = tbl.getDataLocation().getFileSystem(sessionConf); + copyFiles(sessionConf, loadPath, tbl.getPath(), fs, isSrcLocal, isAcid, newFiles); + } catch (IOException e) { + throw new HiveException("addFiles: filesystem error in check phase", e); + } tbl.getParameters().put(StatsSetupConst.STATS_GENERATED_VIA_STATS_TASK, "true"); } Index: ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java (working copy) @@ -43,6 +43,7 @@ import org.apache.hadoop.hive.ql.exec.Utilities; import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils; import org.apache.hadoop.hive.ql.io.HiveOutputFormat; +import org.apache.hadoop.hive.ql.session.SessionState; import org.apache.hadoop.hive.serde2.Deserializer; import org.apache.hadoop.mapred.InputFormat; import org.apache.hadoop.mapred.OutputFormat; @@ -246,10 +247,8 @@ final public Deserializer getDeserializer() { if (deserializer == null) { try { - deserializer = MetaStoreUtils.getDeserializer(Hive.get().getConf(), + deserializer = MetaStoreUtils.getDeserializer(SessionState.getSessionConf(), tPartition, table.getTTable()); - } catch (HiveException e) { - throw new RuntimeException(e); } catch (MetaException e) { throw new RuntimeException(e); } @@ -367,7 +366,7 @@ try { // Previously, this got the filesystem of the Table, which could be // different from the filesystem of the partition. - FileSystem fs = getDataLocation().getFileSystem(Hive.get().getConf()); + FileSystem fs = getDataLocation().getFileSystem(SessionState.getSessionConf()); String pathPattern = getDataLocation().toString(); if (getBucketCount() > 0) { pathPattern = pathPattern + "/*"; @@ -495,11 +494,11 @@ public List getCols() { try { - if (Table.hasMetastoreBasedSchema(Hive.get().getConf(), tPartition.getSd())) { + if (Table.hasMetastoreBasedSchema(SessionState.getSessionConf(), tPartition.getSd())) { return tPartition.getSd().getCols(); } - return Hive.getFieldsFromDeserializer(table.getTableName(), getDeserializer()); - } catch (HiveException e) { + return MetaStoreUtils.getFieldsFromDeserializer(table.getTableName(), getDeserializer()); + } catch (Exception e) { LOG.error("Unable to get cols from serde: " + tPartition.getSd().getSerdeInfo().getSerializationLib(), e); } Index: ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java (working copy) @@ -18,7 +18,6 @@ package org.apache.hadoop.hive.ql.metadata; -import java.io.IOException; import java.io.Serializable; import java.util.ArrayList; import java.util.Arrays; @@ -41,7 +40,6 @@ import org.apache.hadoop.hive.metastore.ProtectMode; import org.apache.hadoop.hive.metastore.TableType; import org.apache.hadoop.hive.metastore.api.FieldSchema; -import org.apache.hadoop.hive.metastore.api.Index; import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.metastore.api.Order; import org.apache.hadoop.hive.metastore.api.SerDeInfo; @@ -259,7 +257,7 @@ } final public Class getDeserializerClass() throws Exception { - return MetaStoreUtils.getDeserializerClass(Hive.get().getConf(), tTable); + return MetaStoreUtils.getDeserializerClass(SessionState.getSessionConf(), tTable); } final public Deserializer getDeserializer(boolean skipConfError) { @@ -271,11 +269,9 @@ final public Deserializer getDeserializerFromMetaStore(boolean skipConfError) { try { - return MetaStoreUtils.getDeserializer(Hive.get().getConf(), tTable, skipConfError); + return MetaStoreUtils.getDeserializer(SessionState.getSessionConf(), tTable, skipConfError); } catch (MetaException e) { throw new RuntimeException(e); - } catch (HiveException e) { - throw new RuntimeException(e); } } @@ -285,7 +281,7 @@ } try { storageHandler = HiveUtils.getStorageHandler( - Hive.get().getConf(), + SessionState.getSessionConf(), getProperty( org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE)); } catch (Exception e) { @@ -589,12 +585,12 @@ String serializationLib = getSerializationLib(); try { - if (hasMetastoreBasedSchema(Hive.get().getConf(), serializationLib)) { + if (hasMetastoreBasedSchema(SessionState.getSessionConf(), serializationLib)) { return tTable.getSd().getCols(); } else { - return Hive.getFieldsFromDeserializer(getTableName(), getDeserializer()); + return MetaStoreUtils.getFieldsFromDeserializer(getTableName(), getDeserializer()); } - } catch (HiveException e) { + } catch (Exception e) { LOG.error("Unable to get field from serde: " + serializationLib, e); } return new ArrayList(); @@ -625,45 +621,6 @@ return tTable.getSd().getNumBuckets(); } - /** - * Replaces the directory corresponding to the table by srcf. Works by - * deleting the table directory and renaming the source directory. - * - * @param srcf - * Source directory - * @param isSrcLocal - * If the source directory is LOCAL - */ - protected void replaceFiles(Path srcf, boolean isSrcLocal) - throws HiveException { - Path tableDest = getPath(); - Hive.replaceFiles(tableDest, srcf, tableDest, tableDest, Hive.get().getConf(), - isSrcLocal); - } - - /** - * Inserts files specified into the partition. Works by moving files - * - * @param srcf - * Files to be moved. Leaf directories or globbed file paths - * @param isSrcLocal - * If the source directory is LOCAL - * @param isAcid - * True if this is an ACID based insert, update, or delete - * @param newFiles optional list of paths. If non-null, then all files copyied to the table - * will be added to this list. - */ - protected void copyFiles(Path srcf, boolean isSrcLocal, boolean isAcid, List newFiles) - throws HiveException { - FileSystem fs; - try { - fs = getDataLocation().getFileSystem(Hive.get().getConf()); - Hive.copyFiles(Hive.get().getConf(), srcf, getPath(), fs, isSrcLocal, isAcid, newFiles); - } catch (IOException e) { - throw new HiveException("addFiles: filesystem error in check phase", e); - } - } - public void setInputFormatClass(String name) throws HiveException { if (name == null) { inputFormatClass = null; @@ -934,22 +891,12 @@ return dbName + "@" + tabName; } - /** - * @return List containing Indexes names if there are indexes on this table - * @throws HiveException - **/ - public List getAllIndexes(short max) throws HiveException { - Hive hive = Hive.get(); - return hive.getIndexes(getTTable().getDbName(), getTTable().getTableName(), max); - } - @SuppressWarnings("nls") public FileStatus[] getSortedPaths() { try { // Previously, this got the filesystem of the Table, which could be // different from the filesystem of the partition. - FileSystem fs = FileSystem.get(getPath().toUri(), Hive.get() - .getConf()); + FileSystem fs = FileSystem.get(getPath().toUri(), SessionState.getSessionConf()); String pathPattern = getPath().toString(); if (getNumBuckets() > 0) { pathPattern = pathPattern + "/*"; Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java (working copy) @@ -52,6 +52,7 @@ import org.apache.hadoop.hive.ql.lib.NodeProcessor; import org.apache.hadoop.hive.ql.lib.NodeProcessorCtx; import org.apache.hadoop.hive.ql.metadata.VirtualColumn; +import org.apache.hadoop.hive.ql.parse.RowResolver; import org.apache.hadoop.hive.ql.parse.SemanticException; import org.apache.hadoop.hive.ql.plan.AggregationDesc; import org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc; @@ -68,9 +69,11 @@ import org.apache.hadoop.hive.ql.plan.TableDesc; import org.apache.hadoop.hive.ql.plan.TableScanDesc; import org.apache.hadoop.hive.ql.plan.ptf.PTFExpressionDef; +import org.apache.hadoop.hive.ql.plan.ptf.PTFInputDef; +import org.apache.hadoop.hive.ql.plan.ptf.PartitionedTableFunctionDef; +import org.apache.hadoop.hive.ql.plan.ptf.ShapeDetails; import org.apache.hadoop.hive.ql.plan.ptf.WindowFunctionDef; import org.apache.hadoop.hive.ql.plan.ptf.WindowTableFunctionDef; -import org.apache.hadoop.hive.ql.udf.ptf.Noop; import org.apache.hadoop.hive.serde2.objectinspector.StructField; import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; @@ -260,81 +263,122 @@ PTFDesc conf = op.getConf(); //Since we cannot know what columns will be needed by a PTF chain, //we do not prune columns on PTFOperator for PTF chains. - if (!conf.forWindowing() && !Noop.class.isInstance(conf.getFuncDef().getTFunction())) { + PartitionedTableFunctionDef funcDef = conf.getFuncDef(); + List referencedColumns = funcDef.getReferencedColumns(); + if (!conf.forWindowing() && !conf.forNoop() && referencedColumns == null) { return super.process(nd, stack, cppCtx, nodeOutputs); } List prunedCols = cppCtx.getPrunedColList(op.getChildOperators().get(0)); - - WindowTableFunctionDef def = null; if (conf.forWindowing()) { - def = (WindowTableFunctionDef) conf.getFuncDef(); - prunedCols = prunedColumnsList(prunedCols, def); + WindowTableFunctionDef def = (WindowTableFunctionDef) funcDef; + prunedCols = Utilities.mergeUniqElems(getWindowFunctionColumns(def), prunedCols); + } else if (conf.forNoop()) { + prunedCols = new ArrayList(cppCtx.getPrunedColList(op.getChildOperators().get(0))); + } else { + prunedCols = referencedColumns; } + + List newRS = prunedColumnsList(prunedCols, op.getSchema(), funcDef); + + op.getSchema().setSignature(new ArrayList(newRS)); - RowSchema oldRS = op.getSchema(); - ArrayList sig = buildPrunedRR(prunedCols, oldRS); - op.getSchema().setSignature(sig); - - prunedCols = def == null ? prunedCols : prunedInputList(prunedCols, def); - cppCtx.getPrunedColLists().put(op, prunedCols); + ShapeDetails outputShape = funcDef.getStartOfChain().getInput().getOutputShape(); + cppCtx.getPrunedColLists().put(op, outputShape.getColumnNames()); return null; } - private static ArrayList buildPrunedRR(List prunedCols, - RowSchema oldRS) throws SemanticException{ + private List buildPrunedRS(List prunedCols, RowSchema oldRS) + throws SemanticException { ArrayList sig = new ArrayList(); HashSet prunedColsSet = new HashSet(prunedCols); - for(ColumnInfo cInfo : oldRS.getSignature()) { - if ( prunedColsSet.contains(cInfo.getInternalName())) { + for (ColumnInfo cInfo : oldRS.getSignature()) { + if (prunedColsSet.contains(cInfo.getInternalName())) { sig.add(cInfo); } } return sig; } + // always should be in this order (see PTFDeserializer#initializeWindowing) + private List getWindowFunctionColumns(WindowTableFunctionDef tDef) { + List columns = new ArrayList(); + if (tDef.getWindowFunctions() != null) { + for (WindowFunctionDef wDef : tDef.getWindowFunctions()) { + columns.add(wDef.getAlias()); + } + } + return columns; + } + + private RowResolver buildPrunedRR(List prunedCols, RowSchema oldRS) + throws SemanticException { + RowResolver resolver = new RowResolver(); + HashSet prunedColsSet = new HashSet(prunedCols); + for (ColumnInfo cInfo : oldRS.getSignature()) { + if (prunedColsSet.contains(cInfo.getInternalName())) { + resolver.put(cInfo.getTabAlias(), cInfo.getAlias(), cInfo); + } + } + return resolver; + } + /* * add any input columns referenced in WindowFn args or expressions. */ - private ArrayList prunedColumnsList(List prunedCols, - WindowTableFunctionDef tDef) { - //we create a copy of prunedCols to create a list of pruned columns for PTFOperator - ArrayList mergedColList = new ArrayList(prunedCols); - if ( tDef.getWindowFunctions() != null ) { - for(WindowFunctionDef wDef : tDef.getWindowFunctions() ) { - if ( wDef.getArgs() == null) { - continue; + private List prunedColumnsList(List prunedCols, RowSchema oldRS, + PartitionedTableFunctionDef pDef) throws SemanticException { + pDef.getOutputShape().setRr(null); + pDef.getOutputShape().setColumnNames(null); + if (pDef instanceof WindowTableFunctionDef) { + WindowTableFunctionDef tDef = (WindowTableFunctionDef) pDef; + if (tDef.getWindowFunctions() != null) { + for (WindowFunctionDef wDef : tDef.getWindowFunctions()) { + if (wDef.getArgs() == null) { + continue; + } + for (PTFExpressionDef arg : wDef.getArgs()) { + ExprNodeDesc exprNode = arg.getExprNode(); + Utilities.mergeUniqElems(prunedCols, exprNode.getCols()); + } } - for(PTFExpressionDef arg : wDef.getArgs()) { - ExprNodeDesc exprNode = arg.getExprNode(); - Utilities.mergeUniqElems(mergedColList, exprNode.getCols()); + } + if (tDef.getPartition() != null) { + for (PTFExpressionDef col : tDef.getPartition().getExpressions()) { + ExprNodeDesc exprNode = col.getExprNode(); + Utilities.mergeUniqElems(prunedCols, exprNode.getCols()); } } + if (tDef.getOrder() != null) { + for (PTFExpressionDef col : tDef.getOrder().getExpressions()) { + ExprNodeDesc exprNode = col.getExprNode(); + Utilities.mergeUniqElems(prunedCols, exprNode.getCols()); + } + } + } else { + pDef.getOutputShape().setRr(buildPrunedRR(prunedCols, oldRS)); } - if(tDef.getPartition() != null){ - for(PTFExpressionDef col : tDef.getPartition().getExpressions()){ - ExprNodeDesc exprNode = col.getExprNode(); - Utilities.mergeUniqElems(mergedColList, exprNode.getCols()); - } - } - if(tDef.getOrder() != null){ - for(PTFExpressionDef col : tDef.getOrder().getExpressions()){ - ExprNodeDesc exprNode = col.getExprNode(); - Utilities.mergeUniqElems(mergedColList, exprNode.getCols()); - } - } - return mergedColList; + + PTFInputDef input = pDef.getInput(); + if (input instanceof PartitionedTableFunctionDef) { + return prunedColumnsList(prunedCols, oldRS, (PartitionedTableFunctionDef)input); + } + + ArrayList inputColumns = prunedInputList(prunedCols, input); + input.getOutputShape().setRr(buildPrunedRR(inputColumns, oldRS)); + input.getOutputShape().setColumnNames(inputColumns); + + return buildPrunedRS(prunedCols, oldRS); } /* * from the prunedCols list filter out columns that refer to WindowFns or WindowExprs * the returned list is set as the prunedList needed by the PTFOp. */ - private ArrayList prunedInputList(List prunedCols, - WindowTableFunctionDef tDef) { + private ArrayList prunedInputList(List prunedCols, PTFInputDef tDef) { ArrayList prunedInputCols = new ArrayList(); - StructObjectInspector OI = tDef.getInput().getOutputShape().getOI(); + StructObjectInspector OI = tDef.getOutputShape().getOI(); for(StructField f : OI.getAllStructFieldRefs()) { String fName = f.getFieldName(); if ( prunedCols.contains(fName)) { Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java (working copy) @@ -90,16 +90,25 @@ QBParseInfo parseInfo = parseCtx.getQB().getParseInfo(); if (parseInfo.isAnalyzeCommand()) { - boolean partialScan = parseInfo.isPartialScanAnalyzeCommand(); - boolean noScan = parseInfo.isNoScanAnalyzeCommand(); - if (inputFormat.equals(OrcInputFormat.class) && (noScan || partialScan)) { - + if (inputFormat.equals(OrcInputFormat.class)) { + // For ORC, all the following statements are the same + // ANALYZE TABLE T [PARTITION (...)] COMPUTE STATISTICS // ANALYZE TABLE T [PARTITION (...)] COMPUTE STATISTICS partialscan; // ANALYZE TABLE T [PARTITION (...)] COMPUTE STATISTICS noscan; + // There will not be any MR or Tez job above this task StatsNoJobWork snjWork = new StatsNoJobWork(parseCtx.getQB().getParseInfo().getTableSpec()); snjWork.setStatsReliable(parseCtx.getConf().getBoolVar( HiveConf.ConfVars.HIVE_STATS_RELIABLE)); + // If partition is specified, get pruned partition list + Set confirmedParts = GenMapRedUtils.getConfirmedPartitionsForScan(parseInfo); + if (confirmedParts.size() > 0) { + Table source = parseCtx.getQB().getMetaData().getTableForAlias(alias); + List partCols = GenMapRedUtils.getPartitionColumns(parseInfo); + PrunedPartitionList partList = new PrunedPartitionList(source, confirmedParts, + partCols, false); + snjWork.setPrunedPartitionList(partList); + } Task snjTask = TaskFactory.get(snjWork, parseCtx.getConf()); ctx.setCurrTask(snjTask); ctx.setCurrTopOp(null); Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (working copy) @@ -90,7 +90,7 @@ import org.apache.hadoop.hive.ql.plan.FetchWork; import org.apache.hadoop.hive.ql.plan.FileMergeDesc; import org.apache.hadoop.hive.ql.plan.FileSinkDesc; -import org.apache.hadoop.hive.ql.plan.FilterDesc.sampleDesc; +import org.apache.hadoop.hive.ql.plan.FilterDesc.SampleDesc; import org.apache.hadoop.hive.ql.plan.LoadFileDesc; import org.apache.hadoop.hive.ql.plan.MapWork; import org.apache.hadoop.hive.ql.plan.MapredLocalWork; @@ -586,7 +586,7 @@ // Later the properties have to come from the partition as opposed // to from the table in order to support versioning. Path[] paths = null; - sampleDesc sampleDescr = parseCtx.getOpToSamplePruner().get(topOp); + SampleDesc sampleDescr = parseCtx.getOpToSamplePruner().get(topOp); // Lookup list bucketing pruner Map partToPruner = parseCtx.getOpToPartToSkewedPruner().get(topOp); Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java (working copy) @@ -190,7 +190,7 @@ List indexesOnTable; try { - indexesOnTable = baseTableMetaData.getAllIndexes((short) -1); // get all indexes + indexesOnTable = getAllIndexes(baseTableMetaData, (short) -1); // get all indexes } catch (HiveException e) { throw new SemanticException("Error accessing metastore", e); } @@ -204,6 +204,14 @@ return matchingIndexes; } + /** + * @return List containing Indexes names if there are indexes on this table + * @throws HiveException + **/ + public static List getAllIndexes(Table table, short max) throws HiveException { + Hive hive = Hive.get(); + return hive.getIndexes(table.getTTable().getDbName(), table.getTTable().getTableName(), max); + } public static Task createRootTask(HiveConf builderConf, Set inputs, Set outputs, StringBuilder command, Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/SamplePruner.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/SamplePruner.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/SamplePruner.java (working copy) @@ -48,7 +48,7 @@ import org.apache.hadoop.hive.ql.parse.ParseContext; import org.apache.hadoop.hive.ql.parse.SemanticException; import org.apache.hadoop.hive.ql.plan.FilterDesc; -import org.apache.hadoop.hive.ql.plan.FilterDesc.sampleDesc; +import org.apache.hadoop.hive.ql.plan.FilterDesc.SampleDesc; /** * The transformation step that does sample pruning. @@ -61,17 +61,17 @@ * */ public static class SamplePrunerCtx implements NodeProcessorCtx { - HashMap opToSamplePruner; + HashMap opToSamplePruner; public SamplePrunerCtx( - HashMap opToSamplePruner) { + HashMap opToSamplePruner) { this.opToSamplePruner = opToSamplePruner; } /** * @return the opToSamplePruner */ - public HashMap getOpToSamplePruner() { + public HashMap getOpToSamplePruner() { return opToSamplePruner; } @@ -80,7 +80,7 @@ * the opToSamplePruner to set */ public void setOpToSamplePruner( - HashMap opToSamplePruner) { + HashMap opToSamplePruner) { this.opToSamplePruner = opToSamplePruner; } } @@ -135,7 +135,7 @@ Object... nodeOutputs) throws SemanticException { FilterOperator filOp = (FilterOperator) nd; FilterDesc filOpDesc = filOp.getConf(); - sampleDesc sampleDescr = filOpDesc.getSampleDescr(); + SampleDesc sampleDescr = filOpDesc.getSampleDescr(); if ((sampleDescr == null) || !sampleDescr.getInputPruning()) { return null; @@ -182,7 +182,7 @@ * @throws SemanticException */ @SuppressWarnings("nls") - public static Path[] prune(Partition part, sampleDesc sampleDescr) + public static Path[] prune(Partition part, SampleDesc sampleDescr) throws SemanticException { int num = sampleDescr.getNumerator(); int den = sampleDescr.getDenominator(); Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java (working copy) @@ -180,7 +180,7 @@ } // We have valid pruning expressions, only retrieve qualifying partitions - ExprNodeDesc pruneExpr = pruneNode.accept(new ExprNodeConverter(getName(), getRowType(), true)); + ExprNodeDesc pruneExpr = pruneNode.accept(new ExprNodeConverter(getName(), getRowType(), true, getRelOptSchema().getTypeFactory())); partitionList = PartitionPruner.prune(hiveTblMetadata, pruneExpr, conf, getName(), partitionCache); } catch (HiveException he) { Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java (working copy) @@ -722,7 +722,7 @@ public ASTNode getUnionAllAST(ASTNode leftAST, ASTNode rightAST) { - ASTNode unionTokAST = ASTBuilder.construct(HiveParser.TOK_UNION, "TOK_UNION").add(leftAST) + ASTNode unionTokAST = ASTBuilder.construct(HiveParser.TOK_UNIONALL, "TOK_UNIONALL").add(leftAST) .add(rightAST).node(); return unionTokAST; Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ExprNodeConverter.java (working copy) @@ -33,12 +33,15 @@ import org.apache.hadoop.hive.ql.udf.generic.GenericUDF; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory; import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.rel.type.RelDataTypeFactory; import org.apache.calcite.rel.type.RelDataTypeField; import org.apache.calcite.rex.RexCall; import org.apache.calcite.rex.RexInputRef; import org.apache.calcite.rex.RexLiteral; import org.apache.calcite.rex.RexNode; import org.apache.calcite.rex.RexVisitorImpl; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.type.SqlTypeUtil; /* * convert a RexNode to an ExprNodeDesc @@ -48,8 +51,9 @@ RelDataType rType; String tabAlias; boolean partitioningExpr; + private final RelDataTypeFactory dTFactory; - public ExprNodeConverter(String tabAlias, RelDataType rType, boolean partitioningExpr) { + public ExprNodeConverter(String tabAlias, RelDataType rType, boolean partitioningExpr, RelDataTypeFactory dTFactory) { super(true); /* * hb: 6/25/14 for now we only support expressions that only contain @@ -62,6 +66,7 @@ this.tabAlias = tabAlias; this.rType = rType; this.partitioningExpr = partitioningExpr; + this.dTFactory = dTFactory; } @Override @@ -85,9 +90,15 @@ args.add(operand.accept(this)); } - // If Expr is flat (and[p,q,r,s] or[p,q,r,s]) then recursively build the - // exprnode - if (ASTConverter.isFlat(call)) { + // If Call is a redundant cast then bail out. Ex: cast(true)BOOLEAN + if (call.isA(SqlKind.CAST) + && (call.operands.size() == 1) + && SqlTypeUtil.equalSansNullability(dTFactory, call.getType(), + call.operands.get(0).getType())) { + return args.get(0); + } else if (ASTConverter.isFlat(call)) { + // If Expr is flat (and[p,q,r,s] or[p,q,r,s]) then recursively build the + // exprnode ArrayList tmpExprArgs = new ArrayList(); tmpExprArgs.addAll(args.subList(0, 2)); gfDesc = new ExprNodeGenericFuncDesc(TypeConverter.convert(call.getType()), Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java (working copy) @@ -64,7 +64,7 @@ import com.google.common.collect.Maps; public class SqlFunctionConverter { - private static final Log LOG = LogFactory.getLog(SqlFunctionConverter.class); + private static final Log LOG = LogFactory.getLog(SqlFunctionConverter.class); static final Map hiveToCalcite; static final Map calciteToHiveToken; @@ -79,7 +79,7 @@ public static SqlOperator getCalciteOperator(String funcTextName, GenericUDF hiveUDF, ImmutableList calciteArgTypes, RelDataType retType) - throws CalciteSemanticException { + throws SemanticException { // handle overloaded methods first if (hiveUDF instanceof GenericUDFOPNegative) { return SqlStdOperatorTable.UNARY_MINUS; @@ -182,7 +182,8 @@ } catch (UDFArgumentException e) { throw new RuntimeException(e); } - return new FunctionInfo(fi.isNative(), fi.getDisplayName(), (GenericUDF) udf); + return new FunctionInfo( + fi.isNative(), fi.getDisplayName(), (GenericUDF) udf, fi.getResources()); } // TODO: 1) handle Agg Func Name translation 2) is it correct to add func Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LlapDecider.java (working copy) @@ -68,8 +68,6 @@ import org.apache.hadoop.hive.ql.plan.Statistics; import org.apache.hadoop.hive.ql.plan.TezWork; -import com.google.common.base.Joiner; - /** * LlapDecider takes care of tagging certain vertices in the execution * graph as "llap", which in turn causes them to be submitted to an @@ -233,7 +231,7 @@ // return false; // } - if (!FunctionRegistry.isNativeFuncExpr((ExprNodeGenericFuncDesc)cur)) { + if (!FunctionRegistry.isBuiltInFuncExpr((ExprNodeGenericFuncDesc)cur)) { LOG.info("Not a built-in function: " + cur.getExprString()); return false; } Index: ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java (working copy) @@ -348,7 +348,7 @@ if (!(expr instanceof ExprNodeGenericFuncDesc)) { return false; } - if (!FunctionRegistry.isNativeFuncExpr((ExprNodeGenericFuncDesc)expr)) { + if (!FunctionRegistry.isBuiltInFuncExpr((ExprNodeGenericFuncDesc) expr)) { return true; } for (ExprNodeDesc child : expr.getChildren()) { Index: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (working copy) @@ -78,12 +78,14 @@ import org.apache.calcite.rel.type.RelDataTypeFactory; import org.apache.calcite.rel.type.RelDataTypeField; import org.apache.calcite.rex.RexBuilder; +import org.apache.calcite.rex.RexExecutorImpl; import org.apache.calcite.rex.RexFieldCollation; import org.apache.calcite.rex.RexInputRef; import org.apache.calcite.rex.RexNode; import org.apache.calcite.rex.RexUtil; import org.apache.calcite.rex.RexWindowBound; import org.apache.calcite.schema.SchemaPlus; +import org.apache.calcite.schema.Schemas; import org.apache.calcite.sql.SqlAggFunction; import org.apache.calcite.sql.SqlCall; import org.apache.calcite.sql.SqlExplainLevel; @@ -716,7 +718,7 @@ hepPgmBldr.addRuleInstance(ReduceExpressionsRule.JOIN_INSTANCE); hepPgmBldr.addRuleInstance(ReduceExpressionsRule.FILTER_INSTANCE); hepPgmBldr.addRuleInstance(ReduceExpressionsRule.PROJECT_INSTANCE); - hepPgmBldr.addRuleInstance(ProjectRemoveRule.INSTANCE); + hepPgmBldr.addRuleInstance(ProjectRemoveRule.NAME_CALC_INSTANCE); hepPgmBldr.addRuleInstance(UnionMergeRule.INSTANCE); hepPgm = hepPgmBldr.build(); @@ -790,7 +792,7 @@ RelFieldTrimmer fieldTrimmer = new RelFieldTrimmer(null, HiveProject.DEFAULT_PROJECT_FACTORY, HiveFilter.DEFAULT_FILTER_FACTORY, HiveJoin.HIVE_JOIN_FACTORY, RelFactories.DEFAULT_SEMI_JOIN_FACTORY, HiveSort.HIVE_SORT_REL_FACTORY, - HiveAggregate.HIVE_AGGR_REL_FACTORY, HiveUnion.UNION_REL_FACTORY); + HiveAggregate.HIVE_AGGR_REL_FACTORY, HiveUnion.UNION_REL_FACTORY, true); basePlan = fieldTrimmer.trim(basePlan); // 5. Rerun PPD through Project as column pruning would have introduced DT @@ -833,6 +835,11 @@ basePlan.getCluster().setMetadataProvider( new CachingRelMetadataProvider(chainedProvider, planner)); + // Executor is required for constant-reduction rules; see [CALCITE-566] + final RexExecutorImpl executor = + new RexExecutorImpl(Schemas.createDataContext(null)); + basePlan.getCluster().getPlanner().setExecutor(executor); + planner.setRoot(basePlan); optimizedRelNode = planner.findBestExp(); @@ -859,49 +866,30 @@ ASTNode tabref = getQB().getAliases().isEmpty() ? null : getQB().getParseInfo() .getSrcForAlias(getQB().getAliases().get(0)); - for (Map.Entry lEntry : leftmap.entrySet()) { + + // 3. construct Union Output RR using original left & right Input + RowResolver unionoutRR = new RowResolver(); + + Iterator> lIter = leftmap.entrySet().iterator(); + Iterator> rIter = rightmap.entrySet().iterator(); + while (lIter.hasNext()) { + Map.Entry lEntry = lIter.next(); + Map.Entry rEntry = rIter.next(); + ColumnInfo lInfo = lEntry.getValue(); + ColumnInfo rInfo = rEntry.getValue(); + String field = lEntry.getKey(); - ColumnInfo lInfo = lEntry.getValue(); - ColumnInfo rInfo = rightmap.get(field); - if (rInfo == null) { - throw new SemanticException(SemanticAnalyzer.generateErrorMessage(tabref, - "Schema of both sides of union should match. " + rightalias - + " does not have the field " + field)); - } - if (lInfo == null) { - throw new SemanticException(SemanticAnalyzer.generateErrorMessage(tabref, - "Schema of both sides of union should match. " + leftalias - + " does not have the field " + field)); - } - if (!lInfo.getInternalName().equals(rInfo.getInternalName())) { - throw new CalciteSemanticException(SemanticAnalyzer.generateErrorMessage( - tabref, - "Schema of both sides of union should match: field " + field + ":" - + " appears on the left side of the UNION at column position: " - + SemanticAnalyzer.getPositionFromInternalName(lInfo.getInternalName()) - + ", and on the right side of the UNION at column position: " - + SemanticAnalyzer.getPositionFromInternalName(rInfo.getInternalName()) - + ". Column positions should match for a UNION")); - } - // try widening coversion, otherwise fail union + // try widening conversion, otherwise fail union TypeInfo commonTypeInfo = FunctionRegistry.getCommonClassForUnionAll(lInfo.getType(), rInfo.getType()); if (commonTypeInfo == null) { - throw new CalciteSemanticException(SemanticAnalyzer.generateErrorMessage(tabref, - "Schema of both sides of union should match: Column " + field + " is of type " - + lInfo.getType().getTypeName() + " on first table and type " - + rInfo.getType().getTypeName() + " on second table")); + throw new SemanticException(generateErrorMessage(tabref, + "Schema of both sides of union should match: Column " + field + + " is of type " + lInfo.getType().getTypeName() + + " on first table and type " + rInfo.getType().getTypeName() + + " on second table")); } - } - - // 3. construct Union Output RR using original left & right Input - RowResolver unionoutRR = new RowResolver(); - for (Map.Entry lEntry : leftmap.entrySet()) { - String field = lEntry.getKey(); - ColumnInfo lInfo = lEntry.getValue(); - ColumnInfo rInfo = rightmap.get(field); ColumnInfo unionColInfo = new ColumnInfo(lInfo); - unionColInfo.setTabAlias(unionalias); unionColInfo.setType(FunctionRegistry.getCommonClassForUnionAll(lInfo.getType(), rInfo.getType())); unionoutRR.put(unionalias, field, unionColInfo); @@ -2637,6 +2625,7 @@ LOG.debug("Created Plan for Query Block " + qb.getId()); } + setQB(qb); return srcRel; } Index: ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java (working copy) @@ -28,6 +28,7 @@ import org.apache.hadoop.hive.metastore.api.ResourceType; import org.apache.hadoop.hive.metastore.api.ResourceUri; import org.apache.hadoop.hive.ql.ErrorMsg; +import org.apache.hadoop.hive.ql.exec.FunctionInfo; import org.apache.hadoop.hive.ql.exec.FunctionRegistry; import org.apache.hadoop.hive.ql.exec.FunctionUtils; import org.apache.hadoop.hive.ql.exec.TaskFactory; @@ -35,6 +36,7 @@ import org.apache.hadoop.hive.ql.hooks.WriteEntity; import org.apache.hadoop.hive.ql.metadata.HiveException; import org.apache.hadoop.hive.ql.plan.CreateFunctionDesc; +import org.apache.hadoop.hive.ql.plan.ReloadFunctionDesc; import org.apache.hadoop.hive.ql.plan.DropFunctionDesc; import org.apache.hadoop.hive.ql.plan.FunctionWork; import org.apache.hadoop.hive.ql.plan.PlanUtils; @@ -53,12 +55,13 @@ @Override public void analyzeInternal(ASTNode ast) throws SemanticException { - if (ast.getToken().getType() == HiveParser.TOK_CREATEFUNCTION) { + if (ast.getType() == HiveParser.TOK_CREATEFUNCTION) { analyzeCreateFunction(ast); + } else if (ast.getType() == HiveParser.TOK_DROPFUNCTION) { + analyzeDropFunction(ast); + } else if (ast.getType() == HiveParser.TOK_RELOADFUNCTION) { + rootTasks.add(TaskFactory.get(new FunctionWork(new ReloadFunctionDesc()), conf)); } - if (ast.getToken().getType() == HiveParser.TOK_DROPFUNCTION) { - analyzeDropFunction(ast); - } LOG.info("analyze done"); } @@ -93,13 +96,16 @@ boolean throwException = !ifExists && !HiveConf.getBoolVar(conf, ConfVars.DROPIGNORESNONEXISTENT); - if (FunctionRegistry.getFunctionInfo(functionName) == null) { + FunctionInfo info = FunctionRegistry.getFunctionInfo(functionName); + if (info == null) { if (throwException) { throw new SemanticException(ErrorMsg.INVALID_FUNCTION.getMsg(functionName)); } else { // Fail silently return; } + } else if (info.isBuiltIn()) { + throw new SemanticException(ErrorMsg.DROP_NATIVE_FUNCTION.getMsg(functionName)); } boolean isTemporaryFunction = (ast.getFirstChildWithType(HiveParser.TOK_TEMPORARY) != null); Index: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g (working copy) @@ -297,6 +297,7 @@ KW_AUTHORIZATION: 'AUTHORIZATION'; KW_CONF: 'CONF'; KW_VALUES: 'VALUES'; +KW_RELOAD: 'RELOAD'; // Operators // NOTE: if you add a new function/operator, add it to sysFuncNames so that describe function _FUNC_ will work. Index: ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g (working copy) @@ -85,7 +85,8 @@ TOK_CLUSTERBY; TOK_DISTRIBUTEBY; TOK_SORTBY; -TOK_UNION; +TOK_UNIONALL; +TOK_UNIONDISTINCT; TOK_JOIN; TOK_LEFTOUTERJOIN; TOK_RIGHTOUTERJOIN; @@ -215,6 +216,7 @@ TOK_CHARSETLITERAL; TOK_CREATEFUNCTION; TOK_DROPFUNCTION; +TOK_RELOADFUNCTION; TOK_CREATEMACRO; TOK_DROPMACRO; TOK_TEMPORARY; @@ -612,10 +614,12 @@ // counter to generate unique union aliases private int aliasCounter; - private String generateUnionAlias() { return "_u" + (++aliasCounter); } + private CommonTree throwSetOpException() throws RecognitionException { + throw new FailedPredicateException(input, "orderByClause clusterByClause distributeByClause sortByClause limitClause can only be applied to the whole union.", ""); + } } @rulecatch { @@ -699,6 +703,7 @@ | createIndexStatement | dropIndexStatement | dropFunctionStatement + | reloadFunctionStatement | dropMacroStatement | analyzeStatement | lockStatement @@ -1622,6 +1627,11 @@ -> ^(TOK_DROPFUNCTION functionIdentifier ifExists?) ; +reloadFunctionStatement +@init { pushMsg("reload function statement", state); } +@after { popMsg(state); } + : KW_RELOAD KW_FUNCTION -> ^(TOK_RELOADFUNCTION); + createMacroStatement @init { pushMsg("create macro statement", state); } @after { popMsg(state); } @@ -2043,7 +2053,8 @@ setOperator @init { pushMsg("set operator", state); } @after { popMsg(state); } - : KW_UNION KW_ALL -> ^(TOK_UNION) + : KW_UNION KW_ALL -> ^(TOK_UNIONALL) + | KW_UNION KW_DISTINCT? -> ^(TOK_UNIONDISTINCT) ; queryStatementExpression[boolean topLevel] @@ -2131,41 +2142,98 @@ selectStatement[topLevel] ; - selectStatement[boolean topLevel] - : (singleSelectStatement -> singleSelectStatement) - (u=setOperator b=singleSelectStatement - -> ^($u {$selectStatement.tree} $b) - )* - -> {u != null && topLevel}? ^(TOK_QUERY - ^(TOK_FROM - ^(TOK_SUBQUERY - {$selectStatement.tree} +selectStatement[boolean topLevel] + : + ( + s=selectClause + f=fromClause? + w=whereClause? + g=groupByClause? + h=havingClause? + o=orderByClause? + c=clusterByClause? + d=distributeByClause? + sort=sortByClause? + win=window_clause? + l=limitClause? + -> ^(TOK_QUERY $f? ^(TOK_INSERT ^(TOK_DESTINATION ^(TOK_DIR TOK_TMP_FILE)) + $s $w? $g? $h? $o? $c? + $d? $sort? $win? $l?)) + ) + (set=setOpSelectStatement[$selectStatement.tree, topLevel])? + -> {set == null}? + {$selectStatement.tree} + -> {o==null && c==null && d==null && sort==null && l==null}? + {$set.tree} + -> {throwSetOpException()} + ; + +setOpSelectStatement[CommonTree t, boolean topLevel] + : + (u=setOperator b=simpleSelectStatement + -> {$setOpSelectStatement.tree != null && u.tree.getType()==HiveParser.TOK_UNIONDISTINCT}? + ^(TOK_QUERY + ^(TOK_FROM + ^(TOK_SUBQUERY + ^(TOK_UNIONALL {$setOpSelectStatement.tree} $b) {adaptor.create(Identifier, generateUnionAlias())} ) ) - ^(TOK_INSERT + ^(TOK_INSERT + ^(TOK_DESTINATION ^(TOK_DIR TOK_TMP_FILE)) + ^(TOK_SELECTDI ^(TOK_SELEXPR TOK_ALLCOLREF)) + ) + ) + -> {$setOpSelectStatement.tree != null && u.tree.getType()!=HiveParser.TOK_UNIONDISTINCT}? + ^(TOK_UNIONALL {$setOpSelectStatement.tree} $b) + -> {$setOpSelectStatement.tree == null && u.tree.getType()==HiveParser.TOK_UNIONDISTINCT}? + ^(TOK_QUERY + ^(TOK_FROM + ^(TOK_SUBQUERY + ^(TOK_UNIONALL {$t} $b) + {adaptor.create(Identifier, generateUnionAlias())} + ) + ) + ^(TOK_INSERT ^(TOK_DESTINATION ^(TOK_DIR TOK_TMP_FILE)) - ^(TOK_SELECT ^(TOK_SELEXPR TOK_ALLCOLREF)) + ^(TOK_SELECTDI ^(TOK_SELEXPR TOK_ALLCOLREF)) + ) + ) + -> ^(TOK_UNIONALL {$t} $b) + )+ + o=orderByClause? + c=clusterByClause? + d=distributeByClause? + sort=sortByClause? + win=window_clause? + l=limitClause? + -> {o==null && c==null && d==null && sort==null && win==null && l==null && !topLevel}? + {$setOpSelectStatement.tree} + -> ^(TOK_QUERY + ^(TOK_FROM + ^(TOK_SUBQUERY + {$setOpSelectStatement.tree} + {adaptor.create(Identifier, generateUnionAlias())} + ) ) - ) - -> {$selectStatement.tree} - ; + ^(TOK_INSERT + ^(TOK_DESTINATION ^(TOK_DIR TOK_TMP_FILE)) + ^(TOK_SELECT ^(TOK_SELEXPR TOK_ALLCOLREF)) + $o? $c? $d? $sort? $win? $l? + ) + ) + ; -singleSelectStatement +simpleSelectStatement : selectClause fromClause? whereClause? groupByClause? havingClause? - orderByClause? - clusterByClause? - distributeByClause? - sortByClause? window_clause? - limitClause? -> ^(TOK_QUERY fromClause? ^(TOK_INSERT ^(TOK_DESTINATION ^(TOK_DIR TOK_TMP_FILE)) - selectClause whereClause? groupByClause? havingClause? orderByClause? clusterByClause? - distributeByClause? sortByClause? window_clause? limitClause?)) + -> ^(TOK_QUERY fromClause? ^(TOK_INSERT ^(TOK_DESTINATION ^(TOK_DIR TOK_TMP_FILE)) + selectClause whereClause? groupByClause? havingClause? window_clause?)) ; selectStatementWithCTE @@ -2216,8 +2284,8 @@ @after { popMsg(state); } : KW_INSERT KW_OVERWRITE destination ifNotExists? -> ^(TOK_DESTINATION destination ifNotExists?) - | KW_INSERT KW_INTO KW_TABLE? tableOrPartition - -> ^(TOK_INSERT_INTO tableOrPartition) + | KW_INSERT KW_INTO KW_TABLE? tableOrPartition (LPAREN targetCols=columnNameList RPAREN)? + -> ^(TOK_INSERT_INTO tableOrPartition $targetCols?) ; destination Index: ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g (working copy) @@ -572,5 +572,5 @@ nonReserved : - KW_TRUE | KW_FALSE | KW_LIKE | KW_EXISTS | KW_ASC | KW_DESC | KW_ORDER | KW_GROUP | KW_BY | KW_AS | KW_INSERT | KW_OVERWRITE | KW_OUTER | KW_LEFT | KW_RIGHT | KW_FULL | KW_PARTITION | KW_PARTITIONS | KW_TABLE | KW_TABLES | KW_COLUMNS | KW_INDEX | KW_INDEXES | KW_REBUILD | KW_FUNCTIONS | KW_SHOW | KW_MSCK | KW_REPAIR | KW_DIRECTORY | KW_LOCAL | KW_USING | KW_CLUSTER | KW_DISTRIBUTE | KW_SORT | KW_UNION | KW_LOAD | KW_EXPORT | KW_IMPORT | KW_DATA | KW_INPATH | KW_IS | KW_NULL | KW_CREATE | KW_EXTERNAL | KW_ALTER | KW_CHANGE | KW_FIRST | KW_AFTER | KW_DESCRIBE | KW_DROP | KW_RENAME | KW_IGNORE | KW_PROTECTION | KW_TO | KW_COMMENT | KW_BOOLEAN | KW_TINYINT | KW_SMALLINT | KW_INT | KW_BIGINT | KW_FLOAT | KW_DOUBLE | KW_DATE | KW_DATETIME | KW_TIMESTAMP | KW_DECIMAL | KW_STRING | KW_ARRAY | KW_STRUCT | KW_UNIONTYPE | KW_PARTITIONED | KW_CLUSTERED | KW_SORTED | KW_INTO | KW_BUCKETS | KW_ROW | KW_ROWS | KW_FORMAT | KW_DELIMITED | KW_FIELDS | KW_TERMINATED | KW_ESCAPED | KW_COLLECTION | KW_ITEMS | KW_KEYS | KW_KEY_TYPE | KW_LINES | KW_STORED | KW_FILEFORMAT | KW_INPUTFORMAT | KW_OUTPUTFORMAT | KW_INPUTDRIVER | KW_OUTPUTDRIVER | KW_OFFLINE | KW_ENABLE | KW_DISABLE | KW_READONLY | KW_NO_DROP | KW_LOCATION | KW_BUCKET | KW_OUT | KW_OF | KW_PERCENT | KW_ADD | KW_REPLACE | KW_RLIKE | KW_REGEXP | KW_TEMPORARY | KW_EXPLAIN | KW_FORMATTED | KW_PRETTY | KW_DEPENDENCY | KW_LOGICAL | KW_SERDE | KW_WITH | KW_DEFERRED | KW_SERDEPROPERTIES | KW_DBPROPERTIES | KW_LIMIT | KW_SET | KW_UNSET | KW_TBLPROPERTIES | KW_IDXPROPERTIES | KW_VALUE_TYPE | KW_ELEM_TYPE | KW_MAPJOIN | KW_STREAMTABLE | KW_HOLD_DDLTIME | KW_CLUSTERSTATUS | KW_UTC | KW_UTCTIMESTAMP | KW_LONG | KW_DELETE | KW_PLUS | KW_MINUS | KW_FETCH | KW_INTERSECT | KW_VIEW | KW_IN | KW_DATABASES | KW_MATERIALIZED | KW_SCHEMA | KW_SCHEMAS | KW_GRANT | KW_REVOKE | KW_SSL | KW_UNDO | KW_LOCK | KW_LOCKS | KW_UNLOCK | KW_SHARED | KW_EXCLUSIVE | KW_PROCEDURE | KW_UNSIGNED | KW_WHILE | KW_READ | KW_READS | KW_PURGE | KW_RANGE | KW_ANALYZE | KW_BEFORE | KW_BETWEEN | KW_BOTH | KW_BINARY | KW_CONTINUE | KW_CURSOR | KW_TRIGGER | KW_RECORDREADER | KW_RECORDWRITER | KW_SEMI | KW_LATERAL | KW_TOUCH | KW_ARCHIVE | KW_UNARCHIVE | KW_COMPUTE | KW_STATISTICS | KW_USE | KW_OPTION | KW_CONCATENATE | KW_SHOW_DATABASE | KW_UPDATE | KW_RESTRICT | KW_CASCADE | KW_SKEWED | KW_ROLLUP | KW_CUBE | KW_DIRECTORIES | KW_FOR | KW_GROUPING | KW_SETS | KW_TRUNCATE | KW_NOSCAN | KW_USER | KW_ROLE | KW_ROLES | KW_INNER | KW_DEFINED | KW_ADMIN | KW_JAR | KW_FILE | KW_OWNER | KW_PRINCIPALS | KW_ALL | KW_DEFAULT | KW_NONE | KW_COMPACT | KW_COMPACTIONS | KW_TRANSACTIONS | KW_REWRITE | KW_AUTHORIZATION | KW_VALUES | KW_URI | KW_SERVER + KW_TRUE | KW_FALSE | KW_LIKE | KW_EXISTS | KW_ASC | KW_DESC | KW_ORDER | KW_GROUP | KW_BY | KW_AS | KW_INSERT | KW_OVERWRITE | KW_OUTER | KW_LEFT | KW_RIGHT | KW_FULL | KW_PARTITION | KW_PARTITIONS | KW_TABLE | KW_TABLES | KW_COLUMNS | KW_INDEX | KW_INDEXES | KW_REBUILD | KW_FUNCTIONS | KW_SHOW | KW_MSCK | KW_REPAIR | KW_DIRECTORY | KW_LOCAL | KW_USING | KW_CLUSTER | KW_DISTRIBUTE | KW_SORT | KW_LOAD | KW_EXPORT | KW_IMPORT | KW_DATA | KW_INPATH | KW_IS | KW_NULL | KW_CREATE | KW_EXTERNAL | KW_ALTER | KW_CHANGE | KW_FIRST | KW_AFTER | KW_DESCRIBE | KW_DROP | KW_RENAME | KW_IGNORE | KW_PROTECTION | KW_TO | KW_COMMENT | KW_BOOLEAN | KW_TINYINT | KW_SMALLINT | KW_INT | KW_BIGINT | KW_FLOAT | KW_DOUBLE | KW_DATE | KW_DATETIME | KW_TIMESTAMP | KW_DECIMAL | KW_STRING | KW_ARRAY | KW_STRUCT | KW_UNIONTYPE | KW_PARTITIONED | KW_CLUSTERED | KW_SORTED | KW_INTO | KW_BUCKETS | KW_ROW | KW_ROWS | KW_FORMAT | KW_DELIMITED | KW_FIELDS | KW_TERMINATED | KW_ESCAPED | KW_COLLECTION | KW_ITEMS | KW_KEYS | KW_KEY_TYPE | KW_LINES | KW_STORED | KW_FILEFORMAT | KW_INPUTFORMAT | KW_OUTPUTFORMAT | KW_INPUTDRIVER | KW_OUTPUTDRIVER | KW_OFFLINE | KW_ENABLE | KW_DISABLE | KW_READONLY | KW_NO_DROP | KW_LOCATION | KW_BUCKET | KW_OUT | KW_OF | KW_PERCENT | KW_ADD | KW_REPLACE | KW_RLIKE | KW_REGEXP | KW_TEMPORARY | KW_EXPLAIN | KW_FORMATTED | KW_PRETTY | KW_DEPENDENCY | KW_LOGICAL | KW_SERDE | KW_WITH | KW_DEFERRED | KW_SERDEPROPERTIES | KW_DBPROPERTIES | KW_LIMIT | KW_SET | KW_UNSET | KW_TBLPROPERTIES | KW_IDXPROPERTIES | KW_VALUE_TYPE | KW_ELEM_TYPE | KW_MAPJOIN | KW_STREAMTABLE | KW_HOLD_DDLTIME | KW_CLUSTERSTATUS | KW_UTC | KW_UTCTIMESTAMP | KW_LONG | KW_DELETE | KW_PLUS | KW_MINUS | KW_FETCH | KW_INTERSECT | KW_VIEW | KW_IN | KW_DATABASES | KW_MATERIALIZED | KW_SCHEMA | KW_SCHEMAS | KW_GRANT | KW_REVOKE | KW_SSL | KW_UNDO | KW_LOCK | KW_LOCKS | KW_UNLOCK | KW_SHARED | KW_EXCLUSIVE | KW_PROCEDURE | KW_UNSIGNED | KW_WHILE | KW_READ | KW_READS | KW_PURGE | KW_RANGE | KW_ANALYZE | KW_BEFORE | KW_BETWEEN | KW_BOTH | KW_BINARY | KW_CONTINUE | KW_CURSOR | KW_TRIGGER | KW_RECORDREADER | KW_RECORDWRITER | KW_SEMI | KW_LATERAL | KW_TOUCH | KW_ARCHIVE | KW_UNARCHIVE | KW_COMPUTE | KW_STATISTICS | KW_USE | KW_OPTION | KW_CONCATENATE | KW_SHOW_DATABASE | KW_UPDATE | KW_RESTRICT | KW_CASCADE | KW_SKEWED | KW_ROLLUP | KW_CUBE | KW_DIRECTORIES | KW_FOR | KW_GROUPING | KW_SETS | KW_TRUNCATE | KW_NOSCAN | KW_USER | KW_ROLE | KW_ROLES | KW_INNER | KW_DEFINED | KW_ADMIN | KW_JAR | KW_FILE | KW_OWNER | KW_PRINCIPALS | KW_ALL | KW_DEFAULT | KW_NONE | KW_COMPACT | KW_COMPACTIONS | KW_TRANSACTIONS | KW_REWRITE | KW_AUTHORIZATION | KW_VALUES | KW_URI | KW_SERVER | KW_RELOAD ; Index: ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/IndexUpdater.java (working copy) @@ -30,6 +30,7 @@ import org.apache.hadoop.hive.ql.metadata.HiveException; import org.apache.hadoop.hive.ql.metadata.Partition; import org.apache.hadoop.hive.ql.metadata.Table; +import org.apache.hadoop.hive.ql.optimizer.IndexUtils; import org.apache.hadoop.hive.ql.plan.LoadTableDesc; import org.apache.hadoop.hive.ql.plan.TableDesc; @@ -69,7 +70,7 @@ for (LoadTableDesc ltd : loadTableWork) { TableDesc td = ltd.getTable(); Table srcTable = hive.getTable(td.getTableName()); - List tblIndexes = srcTable.getAllIndexes((short)-1); + List tblIndexes = IndexUtils.getAllIndexes(srcTable, (short)-1); Map partSpec = ltd.getPartitionSpec(); if (partSpec == null || partSpec.size() == 0) { //unpartitioned table, update whole index Index: ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/PTFTranslator.java (working copy) @@ -342,6 +342,7 @@ outColNames, outRR); def.setOutputShape(outputShape); + def.setReferencedColumns(tFn.getReferencedColumns()); return def; } Index: ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java (working copy) @@ -43,7 +43,7 @@ import org.apache.hadoop.hive.ql.optimizer.ppr.PartitionPruner; import org.apache.hadoop.hive.ql.optimizer.unionproc.UnionProcContext; import org.apache.hadoop.hive.ql.plan.ExprNodeDesc; -import org.apache.hadoop.hive.ql.plan.FilterDesc.sampleDesc; +import org.apache.hadoop.hive.ql.plan.FilterDesc.SampleDesc; import org.apache.hadoop.hive.ql.plan.LoadFileDesc; import org.apache.hadoop.hive.ql.plan.LoadTableDesc; import org.apache.hadoop.hive.ql.plan.MapJoinDesc; @@ -64,7 +64,7 @@ private QB qb; private HashMap opToPartPruner; private HashMap opToPartList; - private HashMap opToSamplePruner; + private HashMap opToSamplePruner; private Map> opToPartToSkewedPruner; private HashMap> topOps; private Set joinOps; @@ -152,7 +152,7 @@ Context ctx, HashMap idToTableNameMap, int destTableId, UnionProcContext uCtx, List> listMapJoinOpsNoReducer, Map prunedPartitions, - HashMap opToSamplePruner, + HashMap opToSamplePruner, GlobalLimitCtx globalLimitCtx, HashMap nameToSplitSample, HashSet semanticInputs, List> rootTasks, @@ -373,7 +373,7 @@ /** * @return the opToSamplePruner */ - public HashMap getOpToSamplePruner() { + public HashMap getOpToSamplePruner() { return opToSamplePruner; } @@ -382,7 +382,7 @@ * the opToSamplePruner to set */ public void setOpToSamplePruner( - HashMap opToSamplePruner) { + HashMap opToSamplePruner) { this.opToSamplePruner = opToSamplePruner; } Index: ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java (working copy) @@ -95,16 +95,25 @@ assert alias != null; TezWork tezWork = context.currentTask.getWork(); - boolean partialScan = parseInfo.isPartialScanAnalyzeCommand(); - boolean noScan = parseInfo.isNoScanAnalyzeCommand(); - if (inputFormat.equals(OrcInputFormat.class) && (noScan || partialScan)) { - + if (inputFormat.equals(OrcInputFormat.class)) { + // For ORC, all the following statements are the same + // ANALYZE TABLE T [PARTITION (...)] COMPUTE STATISTICS // ANALYZE TABLE T [PARTITION (...)] COMPUTE STATISTICS partialscan; // ANALYZE TABLE T [PARTITION (...)] COMPUTE STATISTICS noscan; + // There will not be any Tez job above this task StatsNoJobWork snjWork = new StatsNoJobWork(parseContext.getQB().getParseInfo().getTableSpec()); snjWork.setStatsReliable(parseContext.getConf().getBoolVar( HiveConf.ConfVars.HIVE_STATS_RELIABLE)); + // If partition is specified, get pruned partition list + Set confirmedParts = GenMapRedUtils.getConfirmedPartitionsForScan(parseInfo); + if (confirmedParts.size() > 0) { + Table source = parseContext.getQB().getMetaData().getTableForAlias(alias); + List partCols = GenMapRedUtils.getPartitionColumns(parseInfo); + PrunedPartitionList partList = new PrunedPartitionList(source, confirmedParts, + partCols, false); + snjWork.setPrunedPartitionList(partList); + } Task snjTask = TaskFactory.get(snjWork, parseContext.getConf()); snjTask.setParentTasks(null); context.rootTasks.remove(context.currentTask); Index: ql/src/java/org/apache/hadoop/hive/ql/parse/QBMetaData.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/QBMetaData.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/QBMetaData.java (working copy) @@ -103,6 +103,9 @@ return nameToDestType.get(alias.toLowerCase()); } + /** + * @param alias this is actually dest name, like insclause-0 + */ public Table getDestTableForAlias(String alias) { return nameToDestTable.get(alias.toLowerCase()); } Index: ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java (working copy) @@ -43,7 +43,15 @@ private ASTNode joinExpr; private ASTNode hints; private final HashMap aliasToSrc; + /** + * insclause-0 -> TOK_TAB ASTNode + */ private final HashMap nameToDest; + /** + * For 'insert into FOO(x,y) select ...' this stores the + * insclause-0 -> x,y mapping + */ + private final Map> nameToDestSchema; private final HashMap nameToSample; private final Map exprToColumnAlias; private final Map destToSelExpr; @@ -111,6 +119,7 @@ public QBParseInfo(String alias, boolean isSubQ) { aliasToSrc = new HashMap(); nameToDest = new HashMap(); + nameToDestSchema = new HashMap>(); nameToSample = new HashMap(); exprToColumnAlias = new HashMap(); destToLateralView = new HashMap(); @@ -234,6 +243,13 @@ nameToDest.put(clause, ast); } + List setDestSchemaForClause(String clause, List columnList) { + return nameToDestSchema.put(clause, columnList); + } + List getDestSchemaForClause(String clause) { + return nameToDestSchema.get(clause); + } + /** * Set the Cluster By AST for the clause. * Index: ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java (working copy) @@ -465,4 +465,15 @@ } return combinedRR; } + + public RowResolver duplicate() { + RowResolver resolver = new RowResolver(); + resolver.rowSchema = new RowSchema(rowSchema); + resolver.rslvMap.putAll(rslvMap); + resolver.invRslvMap.putAll(invRslvMap); + resolver.altInvRslvMap.putAll(altInvRslvMap); + resolver.expressionMap.putAll(expressionMap); + resolver.isExprResolver = isExprResolver; + return resolver; + } } Index: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java (working copy) @@ -152,7 +152,7 @@ import org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc; import org.apache.hadoop.hive.ql.plan.FileSinkDesc; import org.apache.hadoop.hive.ql.plan.FilterDesc; -import org.apache.hadoop.hive.ql.plan.FilterDesc.sampleDesc; +import org.apache.hadoop.hive.ql.plan.FilterDesc.SampleDesc; import org.apache.hadoop.hive.ql.plan.ForwardDesc; import org.apache.hadoop.hive.ql.plan.GroupByDesc; import org.apache.hadoop.hive.ql.plan.HiveOperation; @@ -244,7 +244,7 @@ private int destTableId; private UnionProcContext uCtx; List> listMapJoinOpsNoReducer; - private HashMap opToSamplePruner; + private HashMap opToSamplePruner; private final Map> opToPartToSkewedPruner; /** * a map for the split sampling, from alias to an instance of SplitSample @@ -300,7 +300,7 @@ super(conf); opToPartPruner = new HashMap(); opToPartList = new HashMap(); - opToSamplePruner = new HashMap(); + opToSamplePruner = new HashMap(); nameToSplitSample = new HashMap(); // Must be deterministic order maps - see HIVE-8707 topOps = new LinkedHashMap>(); @@ -420,7 +420,7 @@ qbexpr.setQB(qb); } break; - case HiveParser.TOK_UNION: { + case HiveParser.TOK_UNIONALL: { qbexpr.setOpcode(QBExpr.Opcode.UNION); // query 1 assert (ast.getChild(0) != null); @@ -430,7 +430,7 @@ qbexpr.setQBExpr1(qbexpr1); // query 2 - assert (ast.getChild(0) != null); + assert (ast.getChild(1) != null); QBExpr qbexpr2 = new QBExpr(alias + "-subquery2"); doPhase1QBExpr((ASTNode) ast.getChild(1), qbexpr2, id + "-subquery2", alias + "-subquery2"); @@ -557,7 +557,7 @@ public static String generateErrorMessage(ASTNode ast, String message) { StringBuilder sb = new StringBuilder(); if (ast == null) { - sb.append("The abstract syntax tree is null"); + sb.append(message).append(". Cannot tell the position of null AST."); return sb.toString(); } sb.append(ast.getLine()); @@ -1176,7 +1176,7 @@ } qbp.setDestForClause(ctx_1.dest, (ASTNode) ast.getChild(0)); - + handleInsertStatementSpecPhase1(ast, qbp, ctx_1); if (qbp.getClauseNamesForDest().size() > 1) { queryProperties.setMultiDestQuery(true); } @@ -1324,7 +1324,7 @@ break; - case HiveParser.TOK_UNION: + case HiveParser.TOK_UNIONALL: if (!qbp.getIsSubQ()) { // this shouldn't happen. The parser should have converted the union to be // contained in a subquery. Just in case, we keep the error as a fallback. @@ -1406,6 +1406,96 @@ return phase1Result; } + /** + * This is phase1 of supporting specifying schema in insert statement + * insert into foo(z,y) select a,b from bar; + * @see #handleInsertStatementSpec(java.util.List, String, RowResolver, RowResolver, QB, ASTNode) + * @throws SemanticException + */ + private void handleInsertStatementSpecPhase1(ASTNode ast, QBParseInfo qbp, Phase1Ctx ctx_1) throws SemanticException { + ASTNode tabColName = (ASTNode)ast.getChild(1); + if(ast.getType() == HiveParser.TOK_INSERT_INTO && tabColName != null && tabColName.getType() == HiveParser.TOK_TABCOLNAME) { + //we have "insert into foo(a,b)..."; parser will enforce that 1+ columns are listed if TOK_TABCOLNAME is present + List targetColNames = new ArrayList(); + for(Node col : tabColName.getChildren()) { + assert ((ASTNode)col).getType() == HiveParser.Identifier : + "expected token " + HiveParser.Identifier + " found " + ((ASTNode)col).getType(); + targetColNames.add(((ASTNode)col).getText()); + } + String fullTableName = getUnescapedName((ASTNode) ast.getChild(0).getChild(0), + SessionState.get().getCurrentDatabase()); + qbp.setDestSchemaForClause(ctx_1.dest, targetColNames); + Set targetColumns = new HashSet(); + targetColumns.addAll(targetColNames); + if(targetColNames.size() != targetColumns.size()) { + throw new SemanticException(generateErrorMessage(tabColName, + "Duplicate column name detected in " + fullTableName + " table schema specification")); + } + Table targetTable = null; + try { + targetTable = db.getTable(fullTableName, false); + } + catch (HiveException ex) { + LOG.error("Error processing HiveParser.TOK_DESTINATION: " + ex.getMessage(), ex); + throw new SemanticException(ex); + } + if(targetTable == null) { + throw new SemanticException(generateErrorMessage(ast, + "Unable to access metadata for table " + fullTableName)); + } + for(FieldSchema f : targetTable.getCols()) { + //parser only allows foo(a,b), not foo(foo.a, foo.b) + targetColumns.remove(f.getName()); + } + if(!targetColumns.isEmpty()) {//here we need to see if remaining columns are dynamic partition columns + /* We just checked the user specified schema columns among regular table column and found some which are not + 'regular'. Now check is they are dynamic partition columns + For dynamic partitioning, + Given "create table multipart(a int, b int) partitioned by (c int, d int);" + for "insert into multipart partition(c='1',d)(d,a) values(2,3);" we expect parse tree to look like this + (TOK_INSERT_INTO + (TOK_TAB + (TOK_TABNAME multipart) + (TOK_PARTSPEC + (TOK_PARTVAL c '1') + (TOK_PARTVAL d) + ) + ) + (TOK_TABCOLNAME d a) + )*/ + List dynamicPartitionColumns = new ArrayList(); + if(ast.getChild(0) != null && ast.getChild(0).getType() == HiveParser.TOK_TAB) { + ASTNode tokTab = (ASTNode)ast.getChild(0); + ASTNode tokPartSpec = (ASTNode)tokTab.getFirstChildWithType(HiveParser.TOK_PARTSPEC); + if(tokPartSpec != null) { + for(Node n : tokPartSpec.getChildren()) { + ASTNode tokPartVal = null; + if(n instanceof ASTNode) { + tokPartVal = (ASTNode)n; + } + if(tokPartVal != null && tokPartVal.getType() == HiveParser.TOK_PARTVAL && tokPartVal.getChildCount() == 1) { + assert tokPartVal.getChild(0).getType() == HiveParser.Identifier : + "Expected column name; found tokType=" + tokPartVal.getType(); + dynamicPartitionColumns.add(tokPartVal.getChild(0).getText()); + } + } + } + } + for(String colName : dynamicPartitionColumns) { + targetColumns.remove(colName); + } + if(!targetColumns.isEmpty()) { + //Found some columns in user specified schema which are neither regular not dynamic partition columns + throw new SemanticException(generateErrorMessage(tabColName, + "'" + (targetColumns.size() == 1 ? targetColumns.iterator().next() : targetColumns) + + "' in insert schema specification " + (targetColumns.size() == 1 ? "is" : "are") + + " not found among regular columns of " + + fullTableName + " nor dynamic partition columns.")); + } + } + } + } + private void getMetaData(QBExpr qbexpr, ReadEntity parentInput) throws SemanticException { if (qbexpr.getOpcode() == QBExpr.Opcode.NULLOP) { @@ -3493,7 +3583,7 @@ private Operator genSelectPlan(String dest, QB qb, Operator input, Operator inputForSelectStar) throws SemanticException { ASTNode selExprList = qb.getParseInfo().getSelForClause(dest); - Operator op = genSelectPlan(selExprList, qb, input, inputForSelectStar, false); + Operator op = genSelectPlan(dest, selExprList, qb, input, inputForSelectStar, false); if (LOG.isDebugEnabled()) { LOG.debug("Created Select Plan for clause: " + dest); @@ -3503,7 +3593,7 @@ } @SuppressWarnings("nls") - private Operator genSelectPlan(ASTNode selExprList, QB qb, Operator input, + private Operator genSelectPlan(String dest, ASTNode selExprList, QB qb, Operator input, Operator inputForSelectStar, boolean outerLV) throws SemanticException { if (LOG.isDebugEnabled()) { @@ -3741,6 +3831,8 @@ } selectStar = selectStar && exprList.getChildCount() == posn + 1; + handleInsertStatementSpec(col_list, dest, out_rwsch, inputRR, qb, selExprList); + ArrayList columnNames = new ArrayList(); Map colExprMap = new HashMap(); for (int i = 0; i < col_list.size(); i++) { @@ -3768,6 +3860,100 @@ return output; } + /** + * This modifies the Select projections when the Select is part of an insert statement and + * the insert statement specifies a column list for the target table, e.g. + * create table source (a int, b int); + * create table target (x int, y int, z int); + * insert into target(z,x) select * from source + * + * Once the * is resolved to 'a,b', this list needs to rewritten to 'b,null,a' so that it looks + * as if the original query was written as + * insert into target select b, null, a from source + * + * if target schema is not specified, this is no-op + * + * @see #handleInsertStatementSpecPhase1(ASTNode, QBParseInfo, org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.Phase1Ctx) + * @throws SemanticException + */ + private void handleInsertStatementSpec(List col_list, String dest, + RowResolver out_rwsch, RowResolver inputRR, QB qb, + ASTNode selExprList) throws SemanticException { + //(z,x) + List targetTableSchema = qb.getParseInfo().getDestSchemaForClause(dest);//specified in the query + if(targetTableSchema == null) { + //no insert schema was specified + return; + } + if(targetTableSchema.size() != col_list.size()) { + Table target = qb.getMetaData().getDestTableForAlias(dest); + Partition partition = target == null ? qb.getMetaData().getDestPartitionForAlias(dest) : null; + throw new SemanticException(generateErrorMessage(selExprList, + "Expected " + targetTableSchema.size() + " columns for " + dest + + (target != null ? "/" + target.getCompleteName() : (partition != null ? "/" + partition.getCompleteName() : "")) + + "; select produces " + col_list.size() + " columns")); + } + //e.g. map z->expr for a + Map targetCol2Projection = new HashMap(); + //e.g. map z->ColumnInfo for a + Map targetCol2ColumnInfo = new HashMap(); + int colListPos = 0; + for(String targetCol : targetTableSchema) { + targetCol2ColumnInfo.put(targetCol, out_rwsch.getColumnInfos().get(colListPos)); + targetCol2Projection.put(targetCol, col_list.get(colListPos++)); + } + Table target = qb.getMetaData().getDestTableForAlias(dest); + Partition partition = target == null ? qb.getMetaData().getDestPartitionForAlias(dest) : null; + if(target == null && partition == null) { + throw new SemanticException(generateErrorMessage(selExprList, + "No table/partition found in QB metadata for dest='" + dest + "'")); + } + ArrayList new_col_list = new ArrayList(); + ArrayList newSchema = new ArrayList(); + colListPos = 0; + List targetTableCols = target != null ? target.getCols() : partition.getCols(); + List targetTableColNames = new ArrayList(); + for(FieldSchema fs : targetTableCols) { + targetTableColNames.add(fs.getName()); + } + Map partSpec = qb.getMetaData().getPartSpecForAlias(dest); + if(partSpec != null) { + //find dynamic partition columns + //relies on consistent order via LinkedHashMap + for(Map.Entry partKeyVal : partSpec.entrySet()) { + if (partKeyVal.getValue() == null) { + targetTableColNames.add(partKeyVal.getKey());//these must be after non-partition cols + } + } + } + //now make the select produce , with + //where missing columns are NULL-filled + for(String f : targetTableColNames) { + if(targetCol2Projection.containsKey(f)) { + //put existing column in new list to make sure it is in the right position + new_col_list.add(targetCol2Projection.get(f)); + ColumnInfo ci = targetCol2ColumnInfo.get(f);//todo: is this OK? + ci.setInternalName(getColumnInternalName(colListPos)); + newSchema.add(ci); + } + else { + //add new 'synthetic' columns for projections not provided by Select + TypeCheckCtx tcCtx = new TypeCheckCtx(inputRR); + CommonToken t = new CommonToken(HiveParser.TOK_NULL); + t.setText("TOK_NULL"); + ExprNodeDesc exp = genExprNodeDesc(new ASTNode(t), inputRR, tcCtx); + new_col_list.add(exp); + final String tableAlias = "";//is this OK? this column doesn't come from any table + ColumnInfo colInfo = new ColumnInfo(getColumnInternalName(colListPos), + exp.getWritableObjectInspector(), tableAlias, false); + newSchema.add(colInfo); + } + colListPos++; + } + col_list.clear(); + col_list.addAll(new_col_list); + out_rwsch.setRowSchema(new RowSchema(newSchema)); + } String recommendName(ExprNodeDesc exp, String colAlias) { if (!colAlias.startsWith(autogenColAliasPrfxLbl)) { return null; @@ -5200,7 +5386,7 @@ // insert a select operator here used by the ColumnPruner to reduce // the data to shuffle - Operator select = insertSelectAllPlanForGroupBy(selectInput); + Operator select = genSelectAllDesc(selectInput); // Generate ReduceSinkOperator ReduceSinkOperator reduceSinkOperatorInfo = @@ -8324,8 +8510,7 @@ return type; } - private Operator insertSelectAllPlanForGroupBy(Operator input) - throws SemanticException { + private Operator genSelectAllDesc(Operator input) throws SemanticException { OpParseContext inputCtx = opParseCtx.get(input); RowResolver inputRR = inputCtx.getRowResolver(); ArrayList columns = inputRR.getColumnInfos(); @@ -8339,9 +8524,10 @@ columnNames.add(col.getInternalName()); columnExprMap.put(col.getInternalName(), new ExprNodeColumnDesc(col)); } + RowResolver outputRR = inputRR.duplicate(); Operator output = putOpInsertMap(OperatorFactory.getAndMakeChild( - new SelectDesc(colList, columnNames, true), new RowSchema(inputRR - .getColumnInfos()), input), inputRR); + new SelectDesc(colList, columnNames, true), + outputRR.getRowSchema(), input), outputRR); output.setColumnExprMap(columnExprMap); return output; } @@ -8587,7 +8773,7 @@ } // insert a select operator here used by the ColumnPruner to reduce // the data to shuffle - curr = insertSelectAllPlanForGroupBy(curr); + curr = genSelectAllDesc(curr); // Check and transform group by *. This will only happen for select distinct *. // Here the "genSelectPlan" is being leveraged. // The main benefits are (1) remove virtual columns that should @@ -8755,30 +8941,19 @@ if (leftmap.size() != rightmap.size()) { throw new SemanticException("Schema of both sides of union should match."); } - for (Map.Entry lEntry : leftmap.entrySet()) { - String field = lEntry.getKey(); + + RowResolver unionoutRR = new RowResolver(); + + Iterator> lIter = leftmap.entrySet().iterator(); + Iterator> rIter = rightmap.entrySet().iterator(); + while (lIter.hasNext()) { + Map.Entry lEntry = lIter.next(); + Map.Entry rEntry = rIter.next(); ColumnInfo lInfo = lEntry.getValue(); - ColumnInfo rInfo = rightmap.get(field); - if (rInfo == null) { - throw new SemanticException(generateErrorMessage(tabref, - "Schema of both sides of union should match. " + rightalias - + " does not have the field " + field)); - } - if (lInfo == null) { - throw new SemanticException(generateErrorMessage(tabref, - "Schema of both sides of union should match. " + leftalias - + " does not have the field " + field)); - } - if (!lInfo.getInternalName().equals(rInfo.getInternalName())) { - throw new SemanticException(generateErrorMessage(tabref, - "Schema of both sides of union should match: field " + field + ":" - + " appears on the left side of the UNION at column position: " + - getPositionFromInternalName(lInfo.getInternalName()) - + ", and on the right side of the UNION at column position: " + - getPositionFromInternalName(rInfo.getInternalName()) - + ". Column positions should match for a UNION")); - } - // try widening coversion, otherwise fail union + ColumnInfo rInfo = rEntry.getValue(); + + String field = lEntry.getKey(); // use left alias (~mysql, postgresql) + // try widening conversion, otherwise fail union TypeInfo commonTypeInfo = FunctionRegistry.getCommonClassForUnionAll(lInfo.getType(), rInfo.getType()); if (commonTypeInfo == null) { @@ -8788,25 +8963,27 @@ + " on first table and type " + rInfo.getType().getTypeName() + " on second table")); } - } - - // construct the forward operator - RowResolver unionoutRR = new RowResolver(); - for (Map.Entry lEntry : leftmap.entrySet()) { - String field = lEntry.getKey(); - ColumnInfo lInfo = lEntry.getValue(); - ColumnInfo rInfo = rightmap.get(field); ColumnInfo unionColInfo = new ColumnInfo(lInfo); unionColInfo.setType(FunctionRegistry.getCommonClassForUnionAll(lInfo.getType(), rInfo.getType())); unionoutRR.put(unionalias, field, unionColInfo); } - if (!(leftOp instanceof UnionOperator)) { + // For Spark, we rely on the generated SelectOperator to do the type casting. + // Consider: + // SEL_1 (int) SEL_2 (int) SEL_3 (double) + // If we first merge SEL_1 and SEL_2 into a UNION_1, and then merge UNION_1 + // with SEL_3 to get UNION_2, then no SelectOperator will be inserted. Hence error + // will happen afterwards. The solution here is to insert one after UNION_1, which + // cast int to double. + boolean isSpark = HiveConf.getVar(conf, + HiveConf.ConfVars.HIVE_EXECUTION_ENGINE).equals("spark"); + + if (isSpark || !(leftOp instanceof UnionOperator)) { leftOp = genInputSelectForUnion(leftOp, leftmap, leftalias, unionoutRR, unionalias); } - if (!(rightOp instanceof UnionOperator)) { + if (isSpark || !(rightOp instanceof UnionOperator)) { rightOp = genInputSelectForUnion(rightOp, rightmap, rightalias, unionoutRR, unionalias); } @@ -8921,17 +9098,22 @@ String origInputAlias, RowResolver unionoutRR, String unionalias) throws SemanticException { + HashMap fieldMap = unionoutRR.getFieldMap(unionalias); + + Iterator oIter = origInputFieldMap.values().iterator(); + Iterator uIter = fieldMap.values().iterator(); + List columns = new ArrayList(); boolean needsCast = false; - for (Map.Entry unionEntry : unionoutRR.getFieldMap(unionalias).entrySet()) { - String field = unionEntry.getKey(); - ColumnInfo lInfo = origInputFieldMap.get(field); - ExprNodeDesc column = new ExprNodeColumnDesc(lInfo.getType(), lInfo.getInternalName(), - lInfo.getTabAlias(), lInfo.getIsVirtualCol(), lInfo.isSkewedCol()); - if (!lInfo.getType().equals(unionEntry.getValue().getType())) { + while (oIter.hasNext()) { + ColumnInfo oInfo = oIter.next(); + ColumnInfo uInfo = uIter.next(); + ExprNodeDesc column = new ExprNodeColumnDesc(oInfo.getType(), oInfo.getInternalName(), + oInfo.getTabAlias(), oInfo.getIsVirtualCol(), oInfo.isSkewedCol()); + if (!oInfo.getType().equals(uInfo.getType())) { needsCast = true; column = ParseUtils.createConversionCast( - column, (PrimitiveTypeInfo)unionEntry.getValue().getType()); + column, (PrimitiveTypeInfo)uInfo.getType()); } columns.add(column); } @@ -9195,7 +9377,7 @@ ExprNodeDesc samplePredicate = genSamplePredicate(ts, tabBucketCols, colsEqual, alias, rwsch, qb.getMetaData(), null); tableOp = OperatorFactory.getAndMakeChild(new FilterDesc( - samplePredicate, true, new sampleDesc(ts.getNumerator(), ts + samplePredicate, true, new SampleDesc(ts.getNumerator(), ts .getDenominator(), tabBucketCols, true)), new RowSchema(rwsch.getColumnInfos()), top); } else { @@ -9236,7 +9418,7 @@ .getBucketCols(), true, alias, rwsch, qb.getMetaData(), null); tableOp = OperatorFactory .getAndMakeChild(new FilterDesc(samplePred, true, - new sampleDesc(tsSample.getNumerator(), tsSample + new SampleDesc(tsSample.getNumerator(), tsSample .getDenominator(), tab.getBucketCols(), true)), new RowSchema(rwsch.getColumnInfos()), top); LOG.info("No need for sample filter"); @@ -9624,7 +9806,7 @@ int allColumns = allPathRR.getColumnInfos().size(); // Get the UDTF Path QB blankQb = new QB(null, null, false); - Operator udtfPath = genSelectPlan((ASTNode) lateralViewTree + Operator udtfPath = genSelectPlan(null, (ASTNode) lateralViewTree .getChild(0), blankQb, lvForward, null, lateralViewTree.getType() == HiveParser.TOK_LATERAL_VIEW_OUTER); // add udtf aliases to QB @@ -11561,9 +11743,12 @@ List partColList = tabDef.getPartition().getExpressions(); for (PTFExpressionDef colDef : partColList) { - partCols.add(colDef.getExprNode()); - orderCols.add(colDef.getExprNode()); - orderString.append('+'); + ExprNodeDesc exprNode = colDef.getExprNode(); + if (ExprNodeDescUtils.indexOf(exprNode, partCols) < 0) { + partCols.add(exprNode); + orderCols.add(exprNode); + orderString.append('+'); + } } /* @@ -11576,13 +11761,14 @@ List orderColList = tabDef.getOrder().getExpressions(); for (int i = 0; i < orderColList.size(); i++) { OrderExpressionDef colDef = orderColList.get(i); - org.apache.hadoop.hive.ql.parse.PTFInvocationSpec.Order order = colDef.getOrder(); - if (order.name().equals("ASC")) { - orderString.append('+'); - } else { - orderString.append('-'); + char orderChar = colDef.getOrder() == PTFInvocationSpec.Order.ASC ? '+' : '-'; + int index = ExprNodeDescUtils.indexOf(colDef.getExprNode(), orderCols); + if (index >= 0) { + orderString.setCharAt(index, orderChar); + continue; } orderCols.add(colDef.getExprNode()); + orderString.append(orderChar); } } @@ -11680,6 +11866,7 @@ input = putOpInsertMap(OperatorFactory.getAndMakeChild(ptfDesc, new RowSchema(ptfOpRR.getColumnInfos()), input), ptfOpRR); + input = genSelectAllDesc(input); rr = ptfOpRR; } @@ -11695,20 +11882,24 @@ for (PartitionExpression partCol : spec.getQueryPartitionSpec().getExpressions()) { ExprNodeDesc partExpr = genExprNodeDesc(partCol.getExpression(), inputRR); - partCols.add(partExpr); - orderCols.add(partExpr); - order.append('+'); + if (ExprNodeDescUtils.indexOf(partExpr, partCols) < 0) { + partCols.add(partExpr); + orderCols.add(partExpr); + order.append('+'); + } } if (spec.getQueryOrderSpec() != null) { for (OrderExpression orderCol : spec.getQueryOrderSpec().getExpressions()) { - String orderString = orderCol.getOrder().name(); - if (orderString.equals("ASC")) { - order.append('+'); - } else { - order.append('-'); + ExprNodeDesc orderExpr = genExprNodeDesc(orderCol.getExpression(), inputRR); + char orderChar = orderCol.getOrder() == PTFInvocationSpec.Order.ASC ? '+' : '-'; + int index = ExprNodeDescUtils.indexOf(orderExpr, orderCols); + if (index >= 0) { + order.setCharAt(index, orderChar); + continue; } orderCols.add(genExprNodeDesc(orderCol.getExpression(), inputRR)); + order.append(orderChar); } } Index: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java (working copy) @@ -260,6 +260,7 @@ case HiveParser.TOK_CREATEFUNCTION: case HiveParser.TOK_DROPFUNCTION: + case HiveParser.TOK_RELOADFUNCTION: return new FunctionSemanticAnalyzer(conf); case HiveParser.TOK_ANALYZE: Index: ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java (working copy) @@ -216,7 +216,8 @@ * - A Window Specification with no Order and no Window Frame is interpreted as: ROW BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING */ - private void effectiveWindowFrame(WindowFunctionSpec wFn, WindowSpec wdwSpec) { + private void effectiveWindowFrame(WindowFunctionSpec wFn, WindowSpec wdwSpec) + throws SemanticException { WindowFunctionInfo wFnInfo = FunctionRegistry.getWindowFunctionInfo(wFn.getName()); boolean supportsWindowing = wFnInfo == null ? true : wFnInfo.isSupportsWindow(); Index: ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java (working copy) @@ -189,7 +189,15 @@ throws SemanticException { List> roots = new ArrayList>(); - roots.addAll(work.getAllRootOperators()); + + // For MapWork, getAllRootOperators is not suitable, since it checks + // getPathToAliases, and will return null if this is empty. Here we are + // replacing getAliasToWork, so should use that information instead. + if (work instanceof MapWork) { + roots.addAll(((MapWork) work).getAliasToWork().values()); + } else { + roots.addAll(work.getAllRootOperators()); + } if (work.getDummyOps() != null) { roots.addAll(work.getDummyOps()); } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java (working copy) @@ -31,7 +31,7 @@ /** * sampleDesc is used to keep track of the sampling descriptor. */ - public static class sampleDesc implements Cloneable { + public static class SampleDesc implements Cloneable { // The numerator of the TABLESAMPLE clause private int numerator; @@ -41,11 +41,11 @@ // Input files can be pruned private boolean inputPruning; - public sampleDesc() { + public SampleDesc() { } - public sampleDesc(int numerator, int denominator, - List tabBucketCols, boolean inputPruning) { + public SampleDesc(int numerator, int denominator, + List tabBucketCols, boolean inputPruning) { this.numerator = numerator; this.denominator = denominator; this.inputPruning = inputPruning; @@ -65,15 +65,19 @@ @Override public Object clone() { - sampleDesc desc = new sampleDesc(numerator, denominator, null, inputPruning); + SampleDesc desc = new SampleDesc(numerator, denominator, null, inputPruning); return desc; } + + public String toString() { + return inputPruning ? "BUCKET " + numerator + " OUT OF " + denominator: null; + } } private static final long serialVersionUID = 1L; private org.apache.hadoop.hive.ql.plan.ExprNodeDesc predicate; private boolean isSamplingPred; - private transient sampleDesc sampleDescr; + private transient SampleDesc sampleDescr; //Is this a filter that should perform a comparison for sorted searches private boolean isSortedFilter; @@ -90,7 +94,7 @@ public FilterDesc( final org.apache.hadoop.hive.ql.plan.ExprNodeDesc predicate, - boolean isSamplingPred, final sampleDesc sampleDescr) { + boolean isSamplingPred, final SampleDesc sampleDescr) { this.predicate = predicate; this.isSamplingPred = isSamplingPred; this.sampleDescr = sampleDescr; @@ -121,15 +125,19 @@ this.isSamplingPred = isSamplingPred; } - @Explain(displayName = "sampleDesc", normalExplain = false) - public sampleDesc getSampleDescr() { + public SampleDesc getSampleDescr() { return sampleDescr; } - public void setSampleDescr(final sampleDesc sampleDescr) { + public void setSampleDescr(final SampleDesc sampleDescr) { this.sampleDescr = sampleDescr; } + @Explain(displayName = "sampleDesc", normalExplain = false) + public String getSampleDescExpr() { + return sampleDescr == null ? null : sampleDescr.toString(); + } + public boolean isSortedFilter() { return isSortedFilter; } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/FunctionWork.java (working copy) @@ -28,6 +28,7 @@ private static final long serialVersionUID = 1L; private CreateFunctionDesc createFunctionDesc; private DropFunctionDesc dropFunctionDesc; + private ReloadFunctionDesc reloadFunctionDesc; private CreateMacroDesc createMacroDesc; private DropMacroDesc dropMacroDesc; @@ -45,6 +46,10 @@ this.dropFunctionDesc = dropFunctionDesc; } + public FunctionWork(ReloadFunctionDesc reloadFunctionDesc) { + this.reloadFunctionDesc = reloadFunctionDesc; + } + public FunctionWork(CreateMacroDesc createMacroDesc) { this.createMacroDesc = createMacroDesc; } @@ -69,6 +74,14 @@ this.dropFunctionDesc = dropFunctionDesc; } + public ReloadFunctionDesc getReloadFunctionDesc() { + return reloadFunctionDesc; + } + + public void setReloadFunctionDesc(ReloadFunctionDesc reloadFunctionDesc) { + this.reloadFunctionDesc = reloadFunctionDesc; + } + public CreateMacroDesc getCreateMacroDesc() { return createMacroDesc; } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/PTFDesc.java (working copy) @@ -22,14 +22,17 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.ql.exec.PTFUtils; import org.apache.hadoop.hive.ql.parse.LeadLagInfo; -import org.apache.hadoop.hive.ql.parse.PTFInvocationSpec.Order; -import org.apache.hadoop.hive.ql.parse.PTFInvocationSpec.PTFQueryInputType; +import org.apache.hadoop.hive.ql.plan.ptf.PTFInputDef; import org.apache.hadoop.hive.ql.plan.ptf.PartitionedTableFunctionDef; import org.apache.hadoop.hive.ql.plan.ptf.WindowTableFunctionDef; +import org.apache.hadoop.hive.ql.udf.ptf.Noop; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + @Explain(displayName = "PTF Operator") public class PTFDesc extends AbstractOperatorDesc { private static final long serialVersionUID = 1L; @@ -62,6 +65,19 @@ return funcDef == null ? null : funcDef.getStartOfChain(); } + @Explain(displayName = "Function definitions") + public List getFuncDefExplain() { + if (funcDef == null) { + return null; + } + List inputs = new ArrayList(); + for (PTFInputDef current = funcDef; current != null; current = current.getInput()) { + inputs.add(current); + } + Collections.reverse(inputs); + return inputs; + } + public LeadLagInfo getLlInfo() { return llInfo; } @@ -70,10 +86,23 @@ this.llInfo = llInfo; } + @Explain(displayName = "Lead/Lag information") + public String getLlInfoExplain() { + if (llInfo != null && llInfo.getLeadLagExprs() != null) { + return PlanUtils.getExprListString(llInfo.getLeadLagExprs()); + } + return null; + } + public boolean forWindowing() { - return funcDef != null && (funcDef instanceof WindowTableFunctionDef); + return funcDef instanceof WindowTableFunctionDef; } + public boolean forNoop() { + return funcDef.getTFunction() instanceof Noop; + } + + @Explain(displayName = "Map-side function", displayOnlyOnTrue = true) public boolean isMapSide() { return isMapSide; } @@ -89,5 +118,4 @@ public void setCfg(Configuration cfg) { this.cfg = cfg; } - } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java (working copy) @@ -62,6 +62,7 @@ import org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe; import org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe; import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory; @@ -287,7 +288,7 @@ tblDesc.getProperties().setProperty(serdeConstants.ESCAPE_CHAR, "\\"); //enable extended nesting levels tblDesc.getProperties().setProperty( - LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS, "true"); + LazySerDeParameters.SERIALIZATION_EXTEND_ADDITIONAL_NESTING_LEVELS, "true"); return tblDesc; } @@ -923,7 +924,7 @@ return null; } - public static String getExprListString(Collection exprs) { + public static String getExprListString(Collection exprs) { StringBuffer sb = new StringBuffer(); boolean first = true; for (ExprNodeDesc expr: exprs) { Index: ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java (working copy) @@ -21,6 +21,7 @@ import java.io.Serializable; import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.tableSpec; +import org.apache.hadoop.hive.ql.parse.PrunedPartitionList; /** * Client-side stats aggregator task. @@ -31,6 +32,7 @@ private tableSpec tableSpecs; private boolean statsReliable; + private PrunedPartitionList prunedPartitionList; public StatsNoJobWork() { } @@ -54,4 +56,12 @@ public void setStatsReliable(boolean statsReliable) { this.statsReliable = statsReliable; } + + public void setPrunedPartitionList(PrunedPartitionList prunedPartitionList) { + this.prunedPartitionList = prunedPartitionList; + } + + public PrunedPartitionList getPrunedPartitionList() { + return prunedPartitionList; + } } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/BoundaryDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/BoundaryDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/BoundaryDef.java (working copy) @@ -32,4 +32,10 @@ } public abstract int getAmt(); + + @Override + public String toString() { + return direction == null ? "" : + direction + "(" + (getAmt() == Integer.MAX_VALUE ? "MAX" : getAmt()) + ")"; + } } \ No newline at end of file Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFExpressionDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFExpressionDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFExpressionDef.java (working copy) @@ -20,6 +20,7 @@ import org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator; import org.apache.hadoop.hive.ql.exec.PTFUtils; +import org.apache.hadoop.hive.ql.plan.Explain; import org.apache.hadoop.hive.ql.plan.ExprNodeDesc; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; @@ -58,6 +59,11 @@ this.exprNode = exprNode; } + @Explain(displayName = "expr") + public String getExprNodeExplain() { + return exprNode == null ? null : exprNode.getExprString(); + } + public ExprNodeEvaluator getExprEvaluator() { return exprEvaluator; } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFInputDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFInputDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFInputDef.java (working copy) @@ -19,6 +19,10 @@ package org.apache.hadoop.hive.ql.plan.ptf; +import org.apache.hadoop.hive.ql.exec.RowSchema; +import org.apache.hadoop.hive.ql.plan.Explain; +import org.apache.hadoop.util.StringUtils; + public abstract class PTFInputDef { private String expressionTreeString; private ShapeDetails outputShape; @@ -36,9 +40,17 @@ return outputShape; } + @Explain(displayName = "output shape") + public String getOutputShapeExplain() { + RowSchema schema = outputShape.getRr().getRowSchema(); + return StringUtils.join(", ", schema.getSignature()); + } + public void setOutputShape(ShapeDetails outputShape) { this.outputShape = outputShape; } + + @Explain(displayName = "input alias") public String getAlias() { return alias; } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PTFQueryInputDef.java (working copy) @@ -19,11 +19,14 @@ package org.apache.hadoop.hive.ql.plan.ptf; import org.apache.hadoop.hive.ql.parse.PTFInvocationSpec.PTFQueryInputType; +import org.apache.hadoop.hive.ql.plan.Explain; +@Explain(displayName = "Input definition") public class PTFQueryInputDef extends PTFInputDef { private String destination; private PTFQueryInputType type; + @Explain(displayName = "destination") public String getDestination() { return destination; } @@ -40,6 +43,11 @@ this.type = type; } + @Explain(displayName = "type") + public String getTypeExplain() { + return type.name(); + } + @Override public PTFInputDef getInput() { return null; Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/PartitionedTableFunctionDef.java (working copy) @@ -21,8 +21,11 @@ import java.util.ArrayList; import java.util.List; +import org.apache.hadoop.hive.ql.parse.PTFInvocationSpec; +import org.apache.hadoop.hive.ql.plan.Explain; import org.apache.hadoop.hive.ql.udf.ptf.TableFunctionEvaluator; +@Explain(displayName = "Partition table definition") public class PartitionedTableFunctionDef extends PTFInputDef { private String name; private String resolverClassName; @@ -34,7 +37,10 @@ private OrderDef order; private TableFunctionEvaluator tFunction; boolean transformsRawInput; + + private transient List referencedColumns; + @Explain(displayName = "name") public String getName() { return name; } @@ -47,6 +53,11 @@ return rawInputShape; } + @Explain(displayName = "raw input shape") + public ShapeDetails getRawInputShapeExplain() { + return rawInputShape; + } + public void setRawInputShape(ShapeDetails rawInputShape) { this.rawInputShape = rawInputShape; } @@ -72,6 +83,21 @@ return partition; } + @Explain(displayName = "partition by") + public String getPartitionExplain() { + if (partition == null || partition.getExpressions() == null) { + return null; + } + StringBuilder builder = new StringBuilder(); + for (PTFExpressionDef expression : partition.getExpressions()) { + if (builder.length() > 0) { + builder.append(", "); + } + builder.append(expression.getExprNode().getExprString()); + } + return builder.toString(); + } + public void setPartition(PartitionDef partition) { this.partition = partition; } @@ -84,9 +110,28 @@ this.order = order; } + @Explain(displayName = "order by") + public String getOrderExplain() { + if (order == null || order.getExpressions() == null) { + return null; + } + StringBuilder builder = new StringBuilder(); + for (OrderExpressionDef expression : order.getExpressions()) { + if (builder.length() > 0) { + builder.append(", "); + } + builder.append(expression.getExprNode().getExprString()); + if (expression.getOrder() == PTFInvocationSpec.Order.DESC) { + builder.append("(DESC)"); + } + } + return builder.toString(); + } + public TableFunctionEvaluator getTFunction() { return tFunction; } + public void setTFunction(TableFunctionEvaluator tFunction) { this.tFunction = tFunction; } @@ -99,6 +144,21 @@ this.args = args; } + @Explain(displayName = "arguments") + public String getArgsExplain() { + if (args == null) { + return null; + } + StringBuilder builder = new StringBuilder(); + for (PTFExpressionDef expression : args) { + if (builder.length() > 0) { + builder.append(", "); + } + builder.append(expression.getExprNode().getExprString()); + } + return builder.toString(); + } + public void addArg(PTFExpressionDef arg) { args = args == null ? new ArrayList() : args; args.add(arg); @@ -111,6 +171,7 @@ return this; } + @Explain(displayName = "transforms raw input", displayOnlyOnTrue=true) public boolean isTransformsRawInput() { return transformsRawInput; } @@ -126,4 +187,13 @@ public void setResolverClassName(String resolverClassName) { this.resolverClassName = resolverClassName; } + + @Explain(displayName = "referenced columns") + public List getReferencedColumns() { + return referencedColumns; + } + + public void setReferencedColumns(List referencedColumns) { + this.referencedColumns = referencedColumns; + } } \ No newline at end of file Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowExpressionDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowExpressionDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowExpressionDef.java (working copy) @@ -19,15 +19,12 @@ package org.apache.hadoop.hive.ql.plan.ptf; -public class WindowExpressionDef extends PTFExpressionDef { +import org.apache.hadoop.hive.ql.plan.Explain; + +public abstract class WindowExpressionDef extends PTFExpressionDef { private String alias; - public WindowExpressionDef() {} - - public WindowExpressionDef(PTFExpressionDef eDef) { - super(eDef); - } - + @Explain(displayName = "alias") public String getAlias() { return alias; } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFrameDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFrameDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFrameDef.java (working copy) @@ -38,4 +38,9 @@ public void setEnd(BoundaryDef end) { this.end = end; } + + @Override + public String toString() { + return start + "~" + end; + } } \ No newline at end of file Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowFunctionDef.java (working copy) @@ -21,8 +21,10 @@ import java.util.ArrayList; import java.util.List; +import org.apache.hadoop.hive.ql.plan.Explain; import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator; +@Explain(displayName = "window function definition") public class WindowFunctionDef extends WindowExpressionDef { String name; boolean isStar; @@ -32,6 +34,7 @@ GenericUDAFEvaluator wFnEval; boolean pivotResult; + @Explain(displayName = "name") public String getName() { return name; } @@ -40,6 +43,7 @@ this.name = name; } + @Explain(displayName = "isStar", displayOnlyOnTrue = true) public boolean isStar() { return isStar; } @@ -48,6 +52,7 @@ this.isStar = isStar; } + @Explain(displayName = "isDistinct", displayOnlyOnTrue = true) public boolean isDistinct() { return isDistinct; } @@ -69,6 +74,21 @@ args.add(arg); } + @Explain(displayName = "arguments") + public String getArgsExplain() { + if (args == null) { + return null; + } + StringBuilder builder = new StringBuilder(); + for (PTFExpressionDef expression : args) { + if (builder.length() > 0) { + builder.append(", "); + } + builder.append(expression.getExprNode().getExprString()); + } + return builder.toString(); + } + public WindowFrameDef getWindowFrame() { return windowFrame; } @@ -77,6 +97,11 @@ this.windowFrame = windowFrame; } + @Explain(displayName = "window frame") + public String getWindowFrameExplain() { + return windowFrame == null ? null : windowFrame.toString(); + } + public GenericUDAFEvaluator getWFnEval() { return wFnEval; } @@ -85,6 +110,12 @@ this.wFnEval = wFnEval; } + @Explain(displayName = "window function") + public String getWFnEvalExplain() { + return wFnEval == null ? null : wFnEval.getClass().getSimpleName(); + } + + @Explain(displayName = "isPivotResult", displayOnlyOnTrue = true) public boolean isPivotResult() { return pivotResult; } Index: ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowTableFunctionDef.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowTableFunctionDef.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/plan/ptf/WindowTableFunctionDef.java (working copy) @@ -18,21 +18,26 @@ package org.apache.hadoop.hive.ql.plan.ptf; +import org.apache.hadoop.hive.ql.plan.Explain; + import java.util.List; - +@Explain(displayName = "Windowing table definition") public class WindowTableFunctionDef extends PartitionedTableFunctionDef { List windowFunctions; int rankLimit = -1; int rankLimitFunction; + @Explain(displayName = "window functions") public List getWindowFunctions() { return windowFunctions; } + public void setWindowFunctions(List windowFunctions) { this.windowFunctions = windowFunctions; } + public int getRankLimit() { return rankLimit; } Index: ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java (working copy) @@ -28,6 +28,7 @@ import java.sql.Timestamp; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collection; import java.util.HashMap; import java.util.HashSet; import java.util.LinkedHashMap; @@ -49,8 +50,8 @@ import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj; -import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.ql.MapRedStats; +import org.apache.hadoop.hive.ql.exec.Registry; import org.apache.hadoop.hive.ql.exec.Utilities; import org.apache.hadoop.hive.ql.exec.spark.session.SparkSession; import org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl; @@ -261,6 +262,8 @@ */ private final Set preReloadableAuxJars = new HashSet(); + private final Registry registry = new Registry(); + /** * CURRENT_TIMESTAMP value for query */ @@ -416,11 +419,33 @@ return hdfsEncryptionShim; } + // SessionState is not available in runtime and Hive.get().getConf() is not safe to call + private static class SessionStates { + private SessionState state; + private HiveConf conf; + private void attach(SessionState state) { + this.state = state; + attach(state.getConf()); + } + private void attach(HiveConf conf) { + this.conf = conf; + ClassLoader classLoader = conf.getClassLoader(); + if (classLoader != null) { + Thread.currentThread().setContextClassLoader(classLoader); + } + } + } + /** * Singleton Session object per thread. * **/ - private static ThreadLocal tss = new ThreadLocal(); + private static ThreadLocal tss = new ThreadLocal() { + @Override + protected SessionStates initialValue() { + return new SessionStates(); + } + }; /** * start a new session and set it to current session. @@ -434,8 +459,7 @@ * Sets the given session state in the thread local var for sessions. */ public static void setCurrentSessionState(SessionState startSs) { - tss.set(startSs); - Thread.currentThread().setContextClassLoader(startSs.getConf().getClassLoader()); + tss.get().attach(startSs); } public static void detachSession() { @@ -702,9 +726,6 @@ } catch (HiveException e) { LOG.error("Error setting up authorization: " + e.getMessage(), e); throw new RuntimeException(e); - } catch (MetaException e) { - LOG.error("Error setting up authorization: " + e.getMessage(), e); - throw new RuntimeException(e); } if(LOG.isDebugEnabled()){ @@ -714,7 +735,7 @@ return; } - private void setAuthorizerV2Config() throws MetaException, HiveException { + private void setAuthorizerV2Config() throws HiveException { // avoid processing the same config multiple times, check marker if (conf.get(CONFIG_AUTHZ_SETTINGS_APPLIED_MARKER, "").equals(Boolean.TRUE.toString())) { return; @@ -724,7 +745,13 @@ authorizerV2.applyAuthorizationConfigPolicy(conf); // update config in Hive thread local as well and init the metastore client - Hive.get(conf).getMSC(); + try { + Hive.get(conf).getMSC(); + } catch (Exception e) { + // catch-all due to some exec time dependencies on session state + // that would cause ClassNoFoundException otherwise + throw new HiveException(e.getMessage(), e); + } // set a marker that this conf has been processed. conf.set(CONFIG_AUTHZ_SETTINGS_APPLIED_MARKER, Boolean.TRUE.toString()); @@ -781,11 +808,32 @@ * get the current session. */ public static SessionState get() { - return tss.get(); + return tss.get().state; } + public static HiveConf getSessionConf() { + SessionStates state = tss.get(); + if (state.conf == null) { + state.attach(new HiveConf()); + } + return state.conf; + } + + public static Registry getRegistry() { + SessionState session = get(); + return session != null ? session.registry : null; + } + + public static Registry getRegistryForWrite() { + Registry registry = getRegistry(); + if (registry == null) { + throw new RuntimeException("Function registery for session is not initialized"); + } + return registry; + } + /** - * get hiveHitsory object which does structured logging. + * get hiveHistory object which does structured logging. * * @return The hive history object */ @@ -1080,13 +1128,13 @@ return added.get(0); } - public List add_resources(ResourceType t, List values) + public List add_resources(ResourceType t, Collection values) throws RuntimeException { // By default don't convert to unix return add_resources(t, values, false); } - public List add_resources(ResourceType t, List values, boolean convertToUnix) + public List add_resources(ResourceType t, Collection values, boolean convertToUnix) throws RuntimeException { Set resourceMap = getResourceMap(t); @@ -1300,6 +1348,7 @@ } public void close() throws IOException { + registry.clear();; if (txnMgr != null) txnMgr.closeTxnManager(); JavaUtils.closeClassLoadersTo(conf.getClassLoader(), parentLoader); File resourceDir = @@ -1388,9 +1437,8 @@ /** * If authorization mode is v2, then pass it through authorizer so that it can apply * any security configuration changes. - * @throws MetaException */ - public void applyAuthorizationPolicy() throws HiveException, MetaException { + public void applyAuthorizationPolicy() throws HiveException { setupAuth(); } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFEWAHBitmapBop.java (working copy) @@ -141,16 +141,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append(name); - sb.append("("); - for (int i = 0; i < children.length; i++) { - sb.append(children[i]); - if (i + 1 != children.length) { - sb.append(","); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString(name, children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/AbstractGenericUDFReflect.java (working copy) @@ -101,16 +101,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append(functionName()).append('('); - for (int i = 0; i < children.length; i++) { - if (i > 0) { - sb.append(','); - } - sb.append(children[i]); - } - sb.append(')'); - return sb.toString(); + return getStandardDisplayString(functionName(), children, ","); } protected abstract String functionName(); Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java (working copy) @@ -210,4 +210,23 @@ + " and " + newInstance.getClass().getName()); } } + + protected String getStandardDisplayString(String name, String[] children) { + return getStandardDisplayString(name, children, ", "); + } + + protected String getStandardDisplayString(String name, String[] children, String delim) { + StringBuilder sb = new StringBuilder(); + sb.append(name); + sb.append("("); + if (children.length > 0) { + sb.append(children[0]); + for (int i = 1; i < children.length; i++) { + sb.append(delim); + sb.append(children[i]); + } + } + sb.append(")"); + return sb.toString(); + } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAbs.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAbs.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAbs.java (working copy) @@ -150,17 +150,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("abs("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(","); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("abs", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAddMonths.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAddMonths.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAddMonths.java (working copy) @@ -73,11 +73,11 @@ } if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) { throw new UDFArgumentTypeException(0, "Only primitive type arguments are accepted but " - + arguments[0].getTypeName() + " is passed. as first arguments"); + + arguments[0].getTypeName() + " is passed as first arguments"); } if (arguments[1].getCategory() != ObjectInspector.Category.PRIMITIVE) { throw new UDFArgumentTypeException(1, "Only primitive type arguments are accepted but " - + arguments[2].getTypeName() + " is passed. as second arguments"); + + arguments[1].getTypeName() + " is passed as second arguments"); } inputType1 = ((PrimitiveObjectInspector) arguments[0]).getPrimitiveCategory(); ObjectInspector outputOI = PrimitiveObjectInspectorFactory.writableStringObjectInspector; @@ -100,14 +100,14 @@ PrimitiveObjectInspectorFactory.writableDateObjectInspector); break; default: - throw new UDFArgumentException( - " ADD_MONTHS() only takes STRING/TIMESTAMP/DATEWRITABLE types as first argument, got " + throw new UDFArgumentTypeException(0, + "ADD_MONTHS() only takes STRING/TIMESTAMP/DATEWRITABLE types as first argument, got " + inputType1); } inputType2 = ((PrimitiveObjectInspector) arguments[1]).getPrimitiveCategory(); if (inputType2 != PrimitiveCategory.INT) { - throw new UDFArgumentException(" ADD_MONTHS() only takes INT types as second argument, got " - + inputType2); + throw new UDFArgumentTypeException(1, + "ADD_MONTHS() only takes INT types as second argument, got " + inputType2); } intWritableConverter = ObjectInspectorConverters.getConverter( (PrimitiveObjectInspector) arguments[1], @@ -144,7 +144,7 @@ date = dw.get(); break; default: - throw new UDFArgumentException( + throw new UDFArgumentTypeException(0, "ADD_MONTHS() only takes STRING/TIMESTAMP/DATEWRITABLE types, got " + inputType1); } int numMonth = toBeAdded.get(); @@ -156,17 +156,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("add_months("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("add_months", children); } protected Calendar addMonth(Date d, int numMonths) { Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFArray.java (working copy) @@ -80,15 +80,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("array("); - for (int i = 0; i < children.length; i++) { - sb.append(children[i]); - if (i + 1 != children.length) { - sb.append(","); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("array", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrue.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrue.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrue.java (working copy) @@ -75,15 +75,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("assert_true("); - for (int i = 0; i < children.length; ++i) { - sb.append(children[i]); - if (i != children.length - 1) { - sb.append(", "); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("assert_true", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseNumeric.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseNumeric.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseNumeric.java (working copy) @@ -22,7 +22,6 @@ import java.util.List; import org.apache.hadoop.hive.common.type.HiveDecimal; -import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.ql.exec.Description; import org.apache.hadoop.hive.ql.exec.FunctionRegistry; import org.apache.hadoop.hive.ql.exec.NoMatchingMethodException; Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBasePad.java (working copy) @@ -18,16 +18,15 @@ package org.apache.hadoop.hive.ql.udf.generic; -import org.apache.commons.lang.StringUtils; import org.apache.hadoop.hive.ql.exec.UDFArgumentException; import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException; import org.apache.hadoop.hive.ql.metadata.HiveException; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.Converter; import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector.PrimitiveCategory; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory; -import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.Converter; -import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector.PrimitiveCategory; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; @@ -85,7 +84,7 @@ @Override public String getDisplayString(String[] children) { - return udfName + "(" + StringUtils.join(children, ", ") + ")"; + return getStandardDisplayString(udfName, children); } protected abstract void performOp(byte[] data, byte[] txt, byte[] padTxt, int len, Text str, Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFBaseTrim.java (working copy) @@ -78,7 +78,7 @@ @Override public String getDisplayString(String[] children) { - return udfName + "(" + StringUtils.join(children, ", ") + ")"; + return getStandardDisplayString(udfName, children); } protected abstract String performOp(String val); Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFCoalesce.java (working copy) @@ -69,17 +69,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("COALESCE("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(","); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("COALESCE", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcat.java (working copy) @@ -205,17 +205,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("concat("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("concat", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFConcatWS.java (working copy) @@ -142,12 +142,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length >= 2); - StringBuilder sb = new StringBuilder(); - sb.append("concat_ws("); - for (int i = 0; i < children.length - 1; i++) { - sb.append(children[i]).append(", "); - } - sb.append(children[children.length - 1]).append(")"); - return sb.toString(); + return getStandardDisplayString("concat_ws", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java (working copy) @@ -131,17 +131,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("to_date("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("to_date", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateAdd.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateAdd.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateAdd.java (working copy) @@ -89,7 +89,7 @@ if (arguments[1].getCategory() != ObjectInspector.Category.PRIMITIVE) { throw new UDFArgumentTypeException(1, "Only primitive type arguments are accepted but " - + arguments[2].getTypeName() + " is passed. as second arguments"); + + arguments[1].getTypeName() + " is passed. as second arguments"); } inputType1 = ((PrimitiveObjectInspector) arguments[0]).getPrimitiveCategory(); @@ -196,16 +196,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("date_add("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("date_add", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateDiff.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateDiff.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateDiff.java (working copy) @@ -96,17 +96,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("datediff("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("datediff", children); } private Date convertToDate(PrimitiveCategory inputType, Converter converter, DeferredObject argument) Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateSub.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateSub.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDateSub.java (working copy) @@ -89,7 +89,7 @@ if (arguments[1].getCategory() != ObjectInspector.Category.PRIMITIVE) { throw new UDFArgumentTypeException(1, "Only primitive type arguments are accepted but " - + arguments[2].getTypeName() + " is passed. as second arguments"); + + arguments[1].getTypeName() + " is passed. as second arguments"); } inputType1 = ((PrimitiveObjectInspector) arguments[0]).getPrimitiveCategory(); @@ -196,16 +196,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("date_sub("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("date_sub", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDecode.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDecode.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDecode.java (working copy) @@ -38,8 +38,8 @@ import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector.PrimitiveCategory; import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector; -import org.apache.hadoop.io.Text; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils; +import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.PrimitiveGrouping; @Description(name = "decode", value = "_FUNC_(bin, str) - Decode the first argument using the second argument character set", @@ -47,37 +47,44 @@ "'UTF-8', 'UTF-16BE', 'UTF-16LE', and 'UTF-16'. If either argument\n" + "is null, the result will also be null") public class GenericUDFDecode extends GenericUDF { - private transient CharsetDecoder decoder = null; - private transient BinaryObjectInspector bytesOI = null; - private transient StringObjectInspector charsetOI = null; + private transient CharsetDecoder decoder; + private transient BinaryObjectInspector bytesOI; + private transient PrimitiveObjectInspector charsetOI; @Override public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException { if (arguments.length != 2) { - throw new UDFArgumentLengthException("Encode() requires exactly two arguments"); + throw new UDFArgumentLengthException("Decode() requires exactly two arguments"); } if (arguments[0].getCategory() != Category.PRIMITIVE || ((PrimitiveObjectInspector)arguments[0]).getPrimitiveCategory() != PrimitiveCategory.BINARY){ - throw new UDFArgumentTypeException(0, "The first argument to Encode() must be a binary"); + throw new UDFArgumentTypeException(0, "The first argument to Decode() must be a binary"); } bytesOI = (BinaryObjectInspector) arguments[0]; - if (arguments[1].getCategory() != Category.PRIMITIVE || - ((PrimitiveObjectInspector)arguments[1]).getPrimitiveCategory() != PrimitiveCategory.STRING){ - throw new UDFArgumentTypeException(1, "The second argument to Encode() must be a string"); + if (arguments[1].getCategory() != Category.PRIMITIVE) { + throw new UDFArgumentTypeException(1, "The second argument to Decode() must be primitive"); } - charsetOI = (StringObjectInspector) arguments[1]; + charsetOI = (PrimitiveObjectInspector) arguments[1]; - // If the character set for encoding is constant, we can optimize that - StringObjectInspector charSetOI = (StringObjectInspector) arguments[1]; - if (charSetOI instanceof ConstantObjectInspector){ - String charSetName = ((Text) ((ConstantObjectInspector) charSetOI).getWritableConstantValue()).toString(); - decoder = Charset.forName(charSetName).newDecoder().onMalformedInput(CodingErrorAction.REPORT).onUnmappableCharacter(CodingErrorAction.REPORT); + if (PrimitiveGrouping.STRING_GROUP != PrimitiveObjectInspectorUtils + .getPrimitiveGrouping(charsetOI.getPrimitiveCategory())) { + throw new UDFArgumentTypeException(1, + "The second argument to Decode() must be from string group"); } + // If the character set for decoding is constant, we can optimize that + if (arguments[1] instanceof ConstantObjectInspector) { + String charSetName = ((ConstantObjectInspector) arguments[1]).getWritableConstantValue() + .toString(); + decoder = Charset.forName(charSetName).newDecoder() + .onMalformedInput(CodingErrorAction.REPORT) + .onUnmappableCharacter(CodingErrorAction.REPORT); + } + return (ObjectInspector) PrimitiveObjectInspectorFactory.javaStringObjectInspector; } @@ -97,7 +104,8 @@ throw new HiveException(e); } } else { - decoded = Charset.forName(charsetOI.getPrimitiveJavaObject(arguments[1].get())).decode(wrappedBytes); + String charSetName = PrimitiveObjectInspectorUtils.getString(arguments[1].get(), charsetOI); + decoded = Charset.forName(charSetName).decode(wrappedBytes); } return decoded.toString(); } @@ -105,10 +113,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - StringBuilder sb = new StringBuilder(); - sb.append("encode("); - sb.append(children[0]).append(","); - sb.append(children[1]).append(")"); - return sb.toString(); + return getStandardDisplayString("decode", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEWAHBitmapEmpty.java (working copy) @@ -95,15 +95,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("EWAH_BITMAP_EMPTY("); - for (int i = 0; i < children.length; i++) { - sb.append(children[i]); - if (i + 1 != children.length) { - sb.append(","); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("EWAH_BITMAP_EMPTY", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFElt.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFElt.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFElt.java (working copy) @@ -92,12 +92,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length >= 2); - StringBuilder sb = new StringBuilder(); - sb.append("elt("); - for (int i = 0; i < children.length - 1; i++) { - sb.append(children[i]).append(", "); - } - sb.append(children[children.length - 1]).append(")"); - return sb.toString(); + return getStandardDisplayString("elt", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEncode.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEncode.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEncode.java (working copy) @@ -35,13 +35,10 @@ import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector.Category; import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector.PrimitiveCategory; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.PrimitiveGrouping; -import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector; import org.apache.hadoop.io.BytesWritable; -import org.apache.hadoop.io.Text; @Description(name = "encode", value = "_FUNC_(str, str) - Encode the first argument using the second argument character set", @@ -116,10 +113,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - StringBuilder sb = new StringBuilder(); - sb.append("encode("); - sb.append(children[0]).append(","); - sb.append(children[1]).append(")"); - return sb.toString(); + return getStandardDisplayString("encode", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFField.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFField.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFField.java (working copy) @@ -90,12 +90,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length >= 2); - - final StringBuilder sb = new StringBuilder(); - sb.append("field("); - sb.append(StringUtils.join(children, ", ")); - sb.append(")"); - - return sb.toString(); + return getStandardDisplayString("field", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFloorCeilBase.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFloorCeilBase.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFloorCeilBase.java (working copy) @@ -128,7 +128,7 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 1); - return opDisplayName + "(" + children[0] + ")"; + return getStandardDisplayString(opDisplayName, children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFormatNumber.java (working copy) @@ -220,12 +220,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - StringBuilder sb = new StringBuilder(); - sb.append("format_number("); - for (int i = 0; i < children.length - 1; i++) { - sb.append(children[i]).append(", "); - } - sb.append(children[children.length - 1]).append(")"); - return sb.toString(); + return getStandardDisplayString("format_number", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java (working copy) @@ -32,11 +32,10 @@ import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter.TextConverter; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter.TimestampConverter; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory; -import org.apache.hadoop.io.Text; @Description(name = "from_utc_timestamp", value = "from_utc_timestamp(timestamp, string timezone) - " - + "Assumes given timestamp ist UTC and converts to given timezone (as of Hive 0.8.0)") + + "Assumes given timestamp is UTC and converts to given timezone (as of Hive 0.8.0)") public class GenericUDFFromUtcTimestamp extends GenericUDF { static final Log LOG = LogFactory.getLog(GenericUDFFromUtcTimestamp.class); @@ -48,17 +47,14 @@ @Override public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException { - if (arguments.length < 2) { - throw new UDFArgumentLengthException( - "The function " + getName() + " requires at least two " + if (arguments.length != 2) { + throw new UDFArgumentLengthException("The function " + getName() + " requires two " + "argument, got " + arguments.length); } try { argumentOIs = new PrimitiveObjectInspector[2]; argumentOIs[0] = (PrimitiveObjectInspector) arguments[0]; - if (arguments.length > 1) { - argumentOIs[1] = (PrimitiveObjectInspector) arguments[1]; - } + argumentOIs[1] = (PrimitiveObjectInspector) arguments[1]; } catch (ClassCastException e) { throw new UDFArgumentException( "The function " + getName() + " takes only primitive types"); @@ -73,20 +69,17 @@ @Override public Object evaluate(DeferredObject[] arguments) throws HiveException { Object o0 = arguments[0].get(); - TimeZone timezone = null; if (o0 == null) { return null; } - - if (arguments.length > 1 && arguments[1] != null) { - Text text = textConverter.convert(arguments[1].get()); - if (text != null) { - timezone = TimeZone.getTimeZone(text.toString()); - } - } else { + Object o1 = arguments[1].get(); + if (o1 == null) { return null; } + String tzStr = textConverter.convert(o1).toString(); + TimeZone timezone = TimeZone.getTimeZone(tzStr); + Timestamp timestamp = ((TimestampWritable) timestampConverter.convert(o0)) .getTimestamp(); Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFGreatest.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFGreatest.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFGreatest.java (working copy) @@ -94,17 +94,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append(getFuncName()).append("("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(","); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString(getFuncName(), children, ","); } protected String getFuncName() { Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFHash.java (working copy) @@ -58,7 +58,7 @@ @Override public String getDisplayString(String[] children) { - return "hash(" + StringUtils.join(children, ',') + ")"; + return getStandardDisplayString("hash", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIf.java (working copy) @@ -139,11 +139,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 3); - StringBuilder sb = new StringBuilder(); - sb.append("if("); - sb.append(children[0]).append(", "); - sb.append(children[1]).append(", "); - sb.append(children[2]).append(")"); - return sb.toString(); + return getStandardDisplayString("if", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInFile.java (working copy) @@ -165,6 +165,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - return "in_file(" + children[0] + ", " + children[1] + ")"; + return getStandardDisplayString("in_file", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInitCap.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInitCap.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInitCap.java (working copy) @@ -108,16 +108,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("initcap("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(","); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("initcap", children); } } \ No newline at end of file Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInstr.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInstr.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInstr.java (working copy) @@ -93,6 +93,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - return "instr(" + children[0] + children[1] + ")"; + return getStandardDisplayString("instr", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLastDay.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLastDay.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLastDay.java (working copy) @@ -68,7 +68,7 @@ } if (arguments[0].getCategory() != ObjectInspector.Category.PRIMITIVE) { throw new UDFArgumentTypeException(0, "Only primitive type arguments are accepted but " - + arguments[0].getTypeName() + " is passed. as first arguments"); + + arguments[0].getTypeName() + " is passed"); } inputType1 = ((PrimitiveObjectInspector) arguments[0]).getPrimitiveCategory(); ObjectInspector outputOI = PrimitiveObjectInspectorFactory.writableStringObjectInspector; @@ -91,9 +91,8 @@ PrimitiveObjectInspectorFactory.writableDateObjectInspector); break; default: - throw new UDFArgumentException( - " LAST_DAY() only takes STRING/TIMESTAMP/DATEWRITABLE types as first argument, got " - + inputType1); + throw new UDFArgumentTypeException(0, + "LAST_DAY() only takes STRING/TIMESTAMP/DATEWRITABLE types, got " + inputType1); } return outputOI; } @@ -112,23 +111,21 @@ } catch (ParseException e) { return null; } - lastDay(date); break; case TIMESTAMP: Timestamp ts = ((TimestampWritable) timestampConverter.convert(arguments[0].get())) .getTimestamp(); date = ts; - lastDay(date); break; case DATE: DateWritable dw = (DateWritable) dateWritableConverter.convert(arguments[0].get()); date = dw.get(); - lastDay(date); break; default: - throw new UDFArgumentException( + throw new UDFArgumentTypeException(0, "LAST_DAY() only takes STRING/TIMESTAMP/DATEWRITABLE types, got " + inputType1); } + lastDay(date); Date newDate = calendar.getTime(); output.set(formatter.format(newDate)); return output; @@ -136,17 +133,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("last_day("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("last_day", children); } protected Calendar lastDay(Date d) { Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLeadLag.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLeadLag.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLeadLag.java (working copy) @@ -165,14 +165,7 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - StringBuilder sb = new StringBuilder(); - sb.append(_getFnName()); - sb.append("("); - sb.append(children[0]); - sb.append(", "); - sb.append(children[1]); - sb.append(")"); - return sb.toString(); + return getStandardDisplayString(_getFnName(), children); } protected abstract String _getFnName(); Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLocate.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLocate.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLocate.java (working copy) @@ -109,7 +109,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2 || children.length == 3); - return "locate(" + children[0] + children[1] - + (children.length == 3 ? children[2] : "") + ")"; + return getStandardDisplayString("locate", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLower.java (working copy) @@ -105,17 +105,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("lower("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(","); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("lower", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMacro.java (working copy) @@ -128,17 +128,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append(macroName); - sb.append("("); - for (int i = 0; i < children.length; i++) { - sb.append(children[i]); - if (i + 1 < children.length) { - sb.append(", "); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString(macroName, children); } public void setMacroName(String macroName) { Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java (working copy) @@ -68,6 +68,6 @@ @Override public String getDisplayString(String[] children) { assert children.length == 1; - return "map_keys(" + children[0] + ")"; + return getStandardDisplayString("map_keys", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java (working copy) @@ -68,6 +68,6 @@ @Override public String getDisplayString(String[] children) { assert children.length == 1; - return "map_values(" + children[0] + ")"; + return getStandardDisplayString("map_values", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFNamedStruct.java (working copy) @@ -75,15 +75,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("named_struct("); - for (int i = 0; i < children.length; i++) { - if (i > 0) { - sb.append(','); - } - sb.append(children[i]); - } - sb.append(')'); - return sb.toString(); + return getStandardDisplayString("named_struct", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPower.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPower.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPower.java (working copy) @@ -41,7 +41,6 @@ @VectorizedExpressions({FuncPowerLongToDouble.class, FuncPowerDoubleToDouble.class}) public class GenericUDFPower extends GenericUDF { private final String opName; - private final String opDisplayName; private transient PrimitiveObjectInspector baseOI; private transient PrimitiveObjectInspector powerOI; @@ -54,7 +53,6 @@ public GenericUDFPower() { opName = getClass().getSimpleName(); - opDisplayName = "power"; resultOI = PrimitiveObjectInspectorFactory.writableDoubleObjectInspector; } @@ -103,7 +101,7 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - return opDisplayName + "(" + children[0] + ", " + children[1] + ")"; + return getStandardDisplayString("power", children); } @Override Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPrintf.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPrintf.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFPrintf.java (working copy) @@ -141,12 +141,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length >= 2); - StringBuilder sb = new StringBuilder(); - sb.append("printf("); - for (int i = 0; i < children.length - 1; i++) { - sb.append(children[i]).append(", "); - } - sb.append(children[children.length - 1]).append(")"); - return sb.toString(); + return getStandardDisplayString("printf", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java (working copy) @@ -22,6 +22,7 @@ import org.apache.hadoop.hive.ql.exec.Description; import org.apache.hadoop.hive.ql.exec.UDFArgumentException; import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException; +import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException; import org.apache.hadoop.hive.ql.exec.vector.VectorizedExpressions; import org.apache.hadoop.hive.ql.exec.vector.expressions.FuncRoundWithNumDigitsDecimalToDecimal; import org.apache.hadoop.hive.ql.exec.vector.expressions.RoundWithNumDigitsDoubleToDouble; @@ -83,38 +84,42 @@ "ROUND requires one or two argument, got " + arguments.length); } + if (arguments[0].getCategory() != Category.PRIMITIVE) { + throw new UDFArgumentTypeException(0, + "ROUND input only takes primitive types, got " + arguments[0].getTypeName()); + } inputOI = (PrimitiveObjectInspector) arguments[0]; - if (inputOI.getCategory() != Category.PRIMITIVE) { - throw new UDFArgumentException( - "ROUND input only takes primitive types, got " + inputOI.getTypeName()); - } if (arguments.length == 2) { + if (arguments[1].getCategory() != Category.PRIMITIVE) { + throw new UDFArgumentTypeException(1, + "ROUND second argument only takes primitive types, got " + arguments[1].getTypeName()); + } PrimitiveObjectInspector scaleOI = (PrimitiveObjectInspector) arguments[1]; switch (scaleOI.getPrimitiveCategory()) { case VOID: break; case BYTE: if (!(scaleOI instanceof WritableConstantByteObjectInspector)) { - throw new UDFArgumentException("ROUND second argument only takes constant"); + throw new UDFArgumentTypeException(1, "ROUND second argument only takes constant"); } scale = ((WritableConstantByteObjectInspector)scaleOI).getWritableConstantValue().get(); break; case SHORT: if (!(scaleOI instanceof WritableConstantShortObjectInspector)) { - throw new UDFArgumentException("ROUND second argument only takes constant"); + throw new UDFArgumentTypeException(1, "ROUND second argument only takes constant"); } scale = ((WritableConstantShortObjectInspector)scaleOI).getWritableConstantValue().get(); break; case INT: if (!(scaleOI instanceof WritableConstantIntObjectInspector)) { - throw new UDFArgumentException("ROUND second argument only takes constant"); + throw new UDFArgumentTypeException(1, "ROUND second argument only takes constant"); } scale = ((WritableConstantIntObjectInspector)scaleOI).getWritableConstantValue().get(); break; case LONG: if (!(scaleOI instanceof WritableConstantLongObjectInspector)) { - throw new UDFArgumentException("ROUND second argument only takes constant"); + throw new UDFArgumentTypeException(1, "ROUND second argument only takes constant"); } long l = ((WritableConstantLongObjectInspector)scaleOI).getWritableConstantValue().get(); if (l < Integer.MIN_VALUE || l > Integer.MAX_VALUE) { @@ -123,7 +128,7 @@ scale = (int)l; break; default: - throw new UDFArgumentException("ROUND second argument only takes integer constant"); + throw new UDFArgumentTypeException(1, "ROUND second argument only takes integer constant"); } } @@ -151,8 +156,9 @@ converterFromString = ObjectInspectorConverters.getConverter(inputOI, outputOI); break; default: - throw new UDFArgumentException("Only numeric data types are allowed for ROUND function. Got " + - inputType.name()); + throw new UDFArgumentTypeException(0, + "Only numeric or string group data types are allowed for ROUND function. Got " + + inputType.name()); } return outputOI; @@ -240,8 +246,9 @@ } return round(doubleValue, scale); default: - throw new UDFArgumentException("Only numeric data types are allowed for ROUND function. Got " + - inputType.name()); + throw new UDFArgumentTypeException(0, + "Only numeric or string group data types are allowed for ROUND function. Got " + + inputType.name()); } } @@ -256,17 +263,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("round("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(", "); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("round", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSentences.java (working copy) @@ -128,14 +128,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length >= 1 && children.length <= 3); - String display = "sentences(" + children[0]; - if(children.length > 1) { - display += ", " + children[1]; - if(children.length > 2) { - display += ", " + children[2]; - } - } - display += ")"; - return display; + return getStandardDisplayString("sentences", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSize.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSize.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSize.java (working copy) @@ -79,6 +79,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 1); - return "size(" + children[0] + ")"; + return getStandardDisplayString("size", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSortArray.java (working copy) @@ -116,6 +116,6 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 1); - return "sort_array(" + children[0] + ")"; + return getStandardDisplayString("sort_array", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSplit.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSplit.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSplit.java (working copy) @@ -82,7 +82,7 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 2); - return "split(" + children[0] + ", " + children[1] + ")"; + return getStandardDisplayString("split", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java (working copy) @@ -104,19 +104,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("str_to_map("); assert (children.length <= 3); - boolean firstChild = true; - for (String child : children) { - if (firstChild) { - firstChild = false; - } else { - sb.append(","); - } - sb.append(child); - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("str_to_map", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStruct.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStruct.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStruct.java (working copy) @@ -59,15 +59,6 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("struct("); - for (int i = 0; i < children.length; i++) { - if (i > 0) { - sb.append(','); - } - sb.append(children[i]); - } - sb.append(')'); - return sb.toString(); + return getStandardDisplayString("struct", children, ","); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTranslate.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTranslate.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTranslate.java (working copy) @@ -287,7 +287,7 @@ @Override public String getDisplayString(String[] children) { assert (children.length == 3); - return "translate(" + children[0] + ", " + children[1] + ", " + children[2] + ")"; + return getStandardDisplayString("translate", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFUpper.java (working copy) @@ -107,17 +107,7 @@ @Override public String getDisplayString(String[] children) { - StringBuilder sb = new StringBuilder(); - sb.append("upper("); - if (children.length > 0) { - sb.append(children[0]); - for (int i = 1; i < children.length; i++) { - sb.append(","); - sb.append(children[i]); - } - } - sb.append(")"); - return sb.toString(); + return getStandardDisplayString("upper", children); } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFInline.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFInline.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFInline.java (working copy) @@ -19,6 +19,7 @@ package org.apache.hadoop.hive.ql.udf.generic; import java.util.ArrayList; +import java.util.List; import org.apache.hadoop.hive.ql.exec.Description; import org.apache.hadoop.hive.ql.exec.UDFArgumentException; @@ -57,8 +58,11 @@ @Override public void process(Object[] os) throws HiveException { - for (Object row : new ArrayList(li.getList(os[0]))) { - forward(row); + List list = li.getList(os[0]); + if (list != null && !list.isEmpty()) { + for (Object row : list.toArray()) { + forward(row); + } } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java (working copy) @@ -35,18 +35,18 @@ * independent. As a consequence, the hash values will not distribute uniformly from 0 to 2^p-1 * thus introducing errors in the estimates. */ - private static final int bitVectorSize = 31; - private int numBitVectors; + private static final int BIT_VECTOR_SIZE = 31; + private final int numBitVectors; // Refer to Flajolet-Martin'86 for the value of phi - private final double phi = 0.77351; + private static final double PHI = 0.77351; - private int[] a; - private int[] b; - private FastBitSet[] bitVector = new FastBitSet[numBitVectors]; + private final int[] a; + private final int[] b; + private final FastBitSet[] bitVector; - private Random aValue; - private Random bValue; + private final Random aValue; + private final Random bValue; /* Create a new distinctValueEstimator */ @@ -54,7 +54,7 @@ this.numBitVectors = numBitVectors; bitVector = new FastBitSet[numBitVectors]; for (int i=0; i< numBitVectors; i++) { - bitVector[i] = new FastBitSet(bitVectorSize); + bitVector[i] = new FastBitSet(BIT_VECTOR_SIZE); } a = new int[numBitVectors]; @@ -98,23 +98,30 @@ b[i] = randVal; if (a[i] < 0) { - a[i] = a[i] + (1 << bitVectorSize - 1); + a[i] = a[i] + (1 << BIT_VECTOR_SIZE - 1); } if (b[i] < 0) { - b[i] = b[i] + (1 << bitVectorSize - 1); + b[i] = b[i] + (1 << BIT_VECTOR_SIZE - 1); } } } public NumDistinctValueEstimator(String s, int numBitVectors) { - FastBitSet b[] = deserialize(s, numBitVectors); + this.numBitVectors = numBitVectors; + FastBitSet bitVectorDeser[] = deserialize(s, numBitVectors); bitVector = new FastBitSet[numBitVectors]; for(int i=0; i > 1; @@ -277,8 +284,8 @@ int index; // Find the index of the least significant bit that is 1 - for (index=0; index> 1; @@ -321,13 +328,13 @@ for (int i=0; i < numBitVectors; i++) { int index = 0; - while (bitVector[i].get(index) && index < bitVectorSize) { + while (bitVector[i].get(index) && index < BIT_VECTOR_SIZE) { index = index + 1; } S = S + index; } - numDistinctValues = ((numBitVectors/phi) * Math.pow(2.0, S/numBitVectors)); + numDistinctValues = ((numBitVectors/PHI) * Math.pow(2.0, S/numBitVectors)); return ((long)numDistinctValues); } @@ -345,7 +352,7 @@ } avgLeastSigZero = - (double)(sumLeastSigZero/(numBitVectors * 1.0)) - (Math.log(phi)/Math.log(2.0)); + (double)(sumLeastSigZero/(numBitVectors * 1.0)) - (Math.log(PHI)/Math.log(2.0)); numDistinctValues = Math.pow(2.0, avgLeastSigZero); return ((long)(numDistinctValues)); } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/MatchPath.java (working copy) @@ -28,6 +28,7 @@ import org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory; import org.apache.hadoop.hive.ql.exec.PTFPartition; import org.apache.hadoop.hive.ql.exec.PTFPartition.PTFPartitionIterator; +import org.apache.hadoop.hive.ql.exec.Utilities; import org.apache.hadoop.hive.ql.metadata.HiveException; import org.apache.hadoop.hive.ql.parse.ASTNode; import org.apache.hadoop.hive.ql.parse.PTFTranslator; @@ -195,6 +196,20 @@ setOutputOI(OI); } + + @Override + public List getReferencedColumns() throws SemanticException { + MatchPath matchPath = (MatchPath) evaluator; + List columns = new ArrayList<>(); + for (ExprNodeDesc exprNode : matchPath.resultExprInfo.resultExprNodes) { + Utilities.mergeUniqElems(columns, exprNode.getCols()); + } + for (ExprNodeDesc exprNode : matchPath.symInfo.symbolExprsDecs) { + Utilities.mergeUniqElems(columns, exprNode.getCols()); + } + return columns; + } + /* * validate and setup patternStr */ @@ -356,6 +371,7 @@ static class SymbolsInfo { int sz; + ArrayList symbolExprsDecs; ArrayList symbolExprsEvaluators; ArrayList symbolExprsOIs; ArrayList symbolExprsNames; @@ -366,6 +382,7 @@ symbolExprsEvaluators = new ArrayList(sz); symbolExprsOIs = new ArrayList(sz); symbolExprsNames = new ArrayList(sz); + symbolExprsDecs = new ArrayList<>(sz); } void add(String name, PTFExpressionDef arg) @@ -373,6 +390,7 @@ symbolExprsNames.add(name); symbolExprsEvaluators.add(arg.getExprEvaluator()); symbolExprsOIs.add(arg.getOI()); + symbolExprsDecs.add(arg.getExprNode()); } } @@ -749,8 +767,7 @@ /* * create SelectListOI */ - selectListInputOI = (StructObjectInspector) - PTFTranslator.getStandardStructOI(selectListInputRowResolver); + selectListInputOI = PTFTranslator.getStandardStructOI(selectListInputRowResolver); } private void fixResultExprString() Index: ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/NoopWithMap.java (working copy) @@ -29,11 +29,6 @@ public class NoopWithMap extends Noop { - @Override - public PTFPartition execute(PTFPartition iPart) throws HiveException - { - return iPart; - } @Override protected PTFPartition _transformRawInput(PTFPartition iPart) throws HiveException Index: ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionEvaluator.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionEvaluator.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionEvaluator.java (working copy) @@ -30,7 +30,6 @@ import org.apache.hadoop.hive.ql.plan.PTFDesc; import org.apache.hadoop.hive.ql.plan.ptf.PartitionedTableFunctionDef; import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator; -import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; /* @@ -60,7 +59,7 @@ */ /** - * Based on Hive {@link GenericUDAFEvaluator}. Break up the responsibility of the old AsbtractTableFunction + * Based on Hive {@link GenericUDAFEvaluator}. Break up the responsibility of the old AbstractTableFunction * class into a Resolver and Evaluator. *

* The Evaluator also holds onto the {@link TableFunctionDef}. This provides information @@ -79,7 +78,7 @@ */ public abstract class TableFunctionEvaluator { /* - * how is this different from the OutpuShape set on the TableDef. + * how is this different from the OutputShape set on the TableDef. * This is the OI of the object coming out of the PTF. * It is put in an output Partition whose Serde is usually LazyBinarySerde. * So the next PTF (or Operator) in the chain gets a LazyBinaryStruct. Index: ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionResolver.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionResolver.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/TableFunctionResolver.java (working copy) @@ -60,7 +60,7 @@ /* * - called during translation. * - invokes createEvaluator which must be implemented by a subclass - * - sets up the evaluator with references to the TableDef, PartitionClass, PartitonMemsize and + * - sets up the evaluator with references to the TableDef, PartitionClass, PartitionMemsize and * the transformsRawInput boolean. */ public void initialize(HiveConf cfg, PTFDesc ptfDesc, PartitionedTableFunctionDef tDef) @@ -193,4 +193,14 @@ * a subclass must provide the {@link TableFunctionEvaluator} instance. */ protected abstract TableFunctionEvaluator createEvaluator(PTFDesc ptfDesc, PartitionedTableFunctionDef tDef); + + /** + * Provide referenced columns names to be used in partition function + * + * @return null for unknown (will get all columns from table including virtual columns) + * @throws SemanticException + */ + public List getReferencedColumns() throws SemanticException { + return null; + } } Index: ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java =================================================================== --- ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java (revision 1661246) +++ ql/src/java/org/apache/hadoop/hive/ql/udf/ptf/WindowingTableFunction.java (working copy) @@ -69,8 +69,7 @@ public void execute(PTFPartitionIterator pItr, PTFPartition outP) throws HiveException { ArrayList> oColumns = new ArrayList>(); PTFPartition iPart = pItr.getPartition(); - StructObjectInspector inputOI; - inputOI = (StructObjectInspector) iPart.getOutputOI(); + StructObjectInspector inputOI = iPart.getOutputOI(); WindowTableFunctionDef wTFnDef = (WindowTableFunctionDef) getTableDef(); Order order = wTFnDef.getOrder().getExpressions().get(0).getOrder(); @@ -145,7 +144,8 @@ return true; } - private boolean streamingPossible(Configuration cfg, WindowFunctionDef wFnDef) { + private boolean streamingPossible(Configuration cfg, WindowFunctionDef wFnDef) + throws HiveException { WindowFrameDef wdwFrame = wFnDef.getWindowFrame(); WindowFunctionInfo wFnInfo = FunctionRegistry.getWindowFunctionInfo(wFnDef .getName()); @@ -202,7 +202,7 @@ * So no Unbounded Preceding or Following. */ @SuppressWarnings("resource") - private int[] setCanAcceptInputAsStream(Configuration cfg) { + private int[] setCanAcceptInputAsStream(Configuration cfg) throws HiveException { canAcceptInputAsStream = false; @@ -514,7 +514,6 @@ i++; } - i=0; for(i=0; i < iPart.getOutputOI().getAllStructFieldRefs().size(); i++) { output.add(null); } Index: ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto =================================================================== --- ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto (revision 1661246) +++ ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto (working copy) @@ -16,8 +16,10 @@ * limitations under the License. */ -package org.apache.hadoop.hive.ql.io.orc; +package orc.proto; +option java_package = "org.apache.hadoop.hive.ql.io.orc"; + message IntegerStatistics { optional sint64 minimum = 1; optional sint64 maximum = 2; @@ -86,6 +88,15 @@ repeated RowIndexEntry entry = 1; } +message BloomFilter { + optional uint32 numHashFunctions = 1; + repeated fixed64 bitset = 2; +} + +message BloomFilterIndex { + repeated BloomFilter bloomFilter = 1; +} + message Stream { // if you add new index stream kinds, you need to make sure to update // StreamName to ensure it is added to the stripe in the right area @@ -97,8 +108,9 @@ DICTIONARY_COUNT = 4; SECONDARY = 5; ROW_INDEX = 6; + BLOOM_FILTER = 7; } - required Kind kind = 1; + optional Kind kind = 1; optional uint32 column = 2; optional uint64 length = 3; } @@ -110,7 +122,7 @@ DIRECT_V2 = 2; DICTIONARY_V2 = 3; } - required Kind kind = 1; + optional Kind kind = 1; optional uint32 dictionarySize = 2; } @@ -140,7 +152,7 @@ VARCHAR = 16; CHAR = 17; } - required Kind kind = 1; + optional Kind kind = 1; repeated uint32 subtypes = 2 [packed=true]; repeated string fieldNames = 3; optional uint32 maximumLength = 4; @@ -157,8 +169,8 @@ } message UserMetadataItem { - required string name = 1; - required bytes value = 2; + optional string name = 1; + optional bytes value = 2; } message StripeStatistics { Index: ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/metastore/TestMetastoreExpr.java (working copy) @@ -26,12 +26,9 @@ import junit.framework.TestCase; import org.apache.hadoop.hive.conf.HiveConf; -import org.apache.hadoop.hive.metastore.api.AlreadyExistsException; import org.apache.hadoop.hive.metastore.api.Database; import org.apache.hadoop.hive.metastore.api.FieldSchema; -import org.apache.hadoop.hive.metastore.api.InvalidObjectException; import org.apache.hadoop.hive.metastore.api.InvalidOperationException; -import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.metastore.api.NoSuchObjectException; import org.apache.hadoop.hive.metastore.api.Order; import org.apache.hadoop.hive.metastore.api.Partition; @@ -90,14 +87,14 @@ } } - private static void silentDropDatabase(String dbName) throws MetaException, TException { + private static void silentDropDatabase(String dbName) throws TException { try { for (String tableName : client.getTables(dbName, "*")) { client.dropTable(dbName, tableName); } client.dropDatabase(dbName); - } catch (NoSuchObjectException e) { - } catch (InvalidOperationException e) { + } catch (NoSuchObjectException ignore) { + } catch (InvalidOperationException ignore) { } } @@ -153,16 +150,16 @@ client.listPartitionsByExpr(dbName, tblName, new byte[] { 'f', 'o', 'o' }, null, (short)-1, new ArrayList()); fail("Should have thrown IncompatibleMetastoreException"); - } catch (IMetaStoreClient.IncompatibleMetastoreException ex) { + } catch (IMetaStoreClient.IncompatibleMetastoreException ignore) { } // Invalid expression => throw some exception, but not incompatible metastore. try { checkExpr(-1, dbName, tblName, e.val(31).intCol("p3").pred(">", 2).build()); fail("Should have thrown"); - } catch (IMetaStoreClient.IncompatibleMetastoreException ex) { + } catch (IMetaStoreClient.IncompatibleMetastoreException ignore) { fail("Should not have thrown IncompatibleMetastoreException"); - } catch (Exception ex) { + } catch (Exception ignore) { } } @@ -198,7 +195,7 @@ for (int i = 0; i < args; ++i) { children.add(stack.pop()); } - stack.push(new ExprNodeGenericFuncDesc(TypeInfoFactory.booleanTypeInfo, + stack.push(new ExprNodeGenericFuncDesc(ti, FunctionRegistry.getFunctionInfo(name).getGenericUDF(), children)); return this; } @@ -249,8 +246,7 @@ } private void addPartition(HiveMetaStoreClient client, Table table, - List vals, String location) throws InvalidObjectException, - AlreadyExistsException, MetaException, TException { + List vals, String location) throws TException { Partition part = new Partition(); part.setDbName(table.getDbName()); Index: ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestHiveKVResultCache.java (working copy) @@ -27,8 +27,6 @@ import java.util.List; import java.util.concurrent.LinkedBlockingQueue; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.ql.io.HiveKey; import org.apache.hadoop.io.BytesWritable; import org.junit.Test; @@ -42,8 +40,7 @@ @Test public void testSimple() throws Exception { // Create KV result cache object, add one (k,v) pair and retrieve them. - HiveConf conf = new HiveConf(); - HiveKVResultCache cache = new HiveKVResultCache(conf); + HiveKVResultCache cache = new HiveKVResultCache(); HiveKey key = new HiveKey("key".getBytes(), "key".hashCode()); BytesWritable value = new BytesWritable("value".getBytes()); @@ -60,10 +57,9 @@ @Test public void testSpilling() throws Exception { - HiveConf conf = new HiveConf(); - HiveKVResultCache cache = new HiveKVResultCache(conf); + HiveKVResultCache cache = new HiveKVResultCache(); - final int recordCount = HiveKVResultCache.IN_MEMORY_CACHE_SIZE * 3; + final int recordCount = HiveKVResultCache.IN_MEMORY_NUM_ROWS * 3; // Test using the same cache where first n rows are inserted then cache is cleared. // Next reuse the same cache and insert another m rows and verify the cache stores correctly. @@ -104,10 +100,18 @@ @Test public void testResultList() throws Exception { scanAndVerify(10000, 0, 0, "a", "b"); + scanAndVerify(10000, 511, 0, "a", "b"); + scanAndVerify(10000, 511 * 2, 0, "a", "b"); + scanAndVerify(10000, 511, 10, "a", "b"); + scanAndVerify(10000, 511 * 2, 10, "a", "b"); scanAndVerify(10000, 512, 0, "a", "b"); scanAndVerify(10000, 512 * 2, 0, "a", "b"); - scanAndVerify(10000, 512, 10, "a", "b"); - scanAndVerify(10000, 512 * 2, 10, "a", "b"); + scanAndVerify(10000, 512, 3, "a", "b"); + scanAndVerify(10000, 512 * 6, 10, "a", "b"); + scanAndVerify(10000, 512 * 7, 5, "a", "b"); + scanAndVerify(10000, 512 * 9, 19, "a", "b"); + scanAndVerify(10000, 1, 0, "a", "b"); + scanAndVerify(10000, 1, 1, "a", "b"); } private static void scanAndVerify( @@ -176,8 +180,8 @@ // A queue to notify separateRowGenerator to generate the next batch of rows. private LinkedBlockingQueue queue; - MyHiveFunctionResultList(Configuration conf, Iterator inputIterator) { - super(conf, inputIterator); + MyHiveFunctionResultList(Iterator inputIterator) { + super(inputIterator); } void init(long rows, int threshold, int separate, String p1, String p2) { @@ -258,8 +262,7 @@ private static long scanResultList(long rows, int threshold, int separate, List> output, String prefix1, String prefix2) { final long iteratorCount = threshold == 0 ? 1 : rows * (100 - separate) / 100 / threshold; - MyHiveFunctionResultList resultList = new MyHiveFunctionResultList( - new HiveConf(), new Iterator() { + MyHiveFunctionResultList resultList = new MyHiveFunctionResultList(new Iterator() { // Input record iterator, not used private int i = 0; @Override Index: ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFileDump.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFileDump.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestFileDump.java (working copy) @@ -21,19 +21,6 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNull; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.Path; -import org.apache.hadoop.hive.common.type.HiveChar; -import org.apache.hadoop.hive.common.type.HiveDecimal; -import org.apache.hadoop.hive.common.type.HiveVarchar; -import org.apache.hadoop.hive.conf.HiveConf; -import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; -import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory; -import org.apache.hive.common.util.HiveTestUtils; -import org.junit.Before; -import org.junit.Test; - import java.io.BufferedReader; import java.io.ByteArrayOutputStream; import java.io.File; @@ -49,6 +36,19 @@ import java.util.Map; import java.util.Random; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hive.common.type.HiveChar; +import org.apache.hadoop.hive.common.type.HiveDecimal; +import org.apache.hadoop.hive.common.type.HiveVarchar; +import org.apache.hadoop.hive.conf.HiveConf; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory; +import org.apache.hive.common.util.HiveTestUtils; +import org.junit.Before; +import org.junit.Test; + public class TestFileDump { Path workDir = new Path(System.getProperty("test.tmp.dir")); @@ -303,4 +303,101 @@ checkOutput(outputFilename, workDir + File.separator + outputFilename); } + + @Test + public void testBloomFilter() throws Exception { + ObjectInspector inspector; + synchronized (TestOrcFile.class) { + inspector = ObjectInspectorFactory.getReflectionObjectInspector + (MyRecord.class, ObjectInspectorFactory.ObjectInspectorOptions.JAVA); + } + conf.set(HiveConf.ConfVars.HIVE_ORC_ENCODING_STRATEGY.varname, "COMPRESSION"); + OrcFile.WriterOptions options = OrcFile.writerOptions(conf) + .fileSystem(fs) + .inspector(inspector) + .stripeSize(100000) + .compress(CompressionKind.ZLIB) + .bufferSize(10000) + .rowIndexStride(1000) + .bloomFilterColumns("s"); + Writer writer = OrcFile.createWriter(testFilePath, options); + Random r1 = new Random(1); + String[] words = new String[]{"It", "was", "the", "best", "of", "times,", + "it", "was", "the", "worst", "of", "times,", "it", "was", "the", "age", + "of", "wisdom,", "it", "was", "the", "age", "of", "foolishness,", "it", + "was", "the", "epoch", "of", "belief,", "it", "was", "the", "epoch", + "of", "incredulity,", "it", "was", "the", "season", "of", "Light,", + "it", "was", "the", "season", "of", "Darkness,", "it", "was", "the", + "spring", "of", "hope,", "it", "was", "the", "winter", "of", "despair,", + "we", "had", "everything", "before", "us,", "we", "had", "nothing", + "before", "us,", "we", "were", "all", "going", "direct", "to", + "Heaven,", "we", "were", "all", "going", "direct", "the", "other", + "way"}; + for(int i=0; i < 21000; ++i) { + writer.addRow(new MyRecord(r1.nextInt(), r1.nextLong(), + words[r1.nextInt(words.length)])); + } + writer.close(); + PrintStream origOut = System.out; + String outputFilename = "orc-file-dump-bloomfilter.out"; + FileOutputStream myOut = new FileOutputStream(workDir + File.separator + outputFilename); + + // replace stdout and run command + System.setOut(new PrintStream(myOut)); + FileDump.main(new String[]{testFilePath.toString(), "--rowindex=3"}); + System.out.flush(); + System.setOut(origOut); + + + checkOutput(outputFilename, workDir + File.separator + outputFilename); + } + + @Test + public void testBloomFilter2() throws Exception { + ObjectInspector inspector; + synchronized (TestOrcFile.class) { + inspector = ObjectInspectorFactory.getReflectionObjectInspector + (MyRecord.class, ObjectInspectorFactory.ObjectInspectorOptions.JAVA); + } + conf.set(HiveConf.ConfVars.HIVE_ORC_ENCODING_STRATEGY.varname, "COMPRESSION"); + OrcFile.WriterOptions options = OrcFile.writerOptions(conf) + .fileSystem(fs) + .inspector(inspector) + .stripeSize(100000) + .compress(CompressionKind.ZLIB) + .bufferSize(10000) + .rowIndexStride(1000) + .bloomFilterColumns("l") + .bloomFilterFpp(0.01); + Writer writer = OrcFile.createWriter(testFilePath, options); + Random r1 = new Random(1); + String[] words = new String[]{"It", "was", "the", "best", "of", "times,", + "it", "was", "the", "worst", "of", "times,", "it", "was", "the", "age", + "of", "wisdom,", "it", "was", "the", "age", "of", "foolishness,", "it", + "was", "the", "epoch", "of", "belief,", "it", "was", "the", "epoch", + "of", "incredulity,", "it", "was", "the", "season", "of", "Light,", + "it", "was", "the", "season", "of", "Darkness,", "it", "was", "the", + "spring", "of", "hope,", "it", "was", "the", "winter", "of", "despair,", + "we", "had", "everything", "before", "us,", "we", "had", "nothing", + "before", "us,", "we", "were", "all", "going", "direct", "to", + "Heaven,", "we", "were", "all", "going", "direct", "the", "other", + "way"}; + for(int i=0; i < 21000; ++i) { + writer.addRow(new MyRecord(r1.nextInt(), r1.nextLong(), + words[r1.nextInt(words.length)])); + } + writer.close(); + PrintStream origOut = System.out; + String outputFilename = "orc-file-dump-bloomfilter2.out"; + FileOutputStream myOut = new FileOutputStream(workDir + File.separator + outputFilename); + + // replace stdout and run command + System.setOut(new PrintStream(myOut)); + FileDump.main(new String[]{testFilePath.toString(), "--rowindex=2"}); + System.out.flush(); + System.setOut(origOut); + + + checkOutput(outputFilename, workDir + File.separator + outputFilename); + } } Index: ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java (working copy) @@ -515,6 +515,10 @@ Object row = rows.next(null); assertEquals(tslist.get(idx++).getNanos(), ((TimestampWritable) row).getNanos()); } + assertEquals(1, OrcUtils.getFlattenedColumnsCount(inspector)); + boolean[] expected = new boolean[] {false}; + boolean[] included = OrcUtils.includeColumns("", "ts", inspector); + assertEquals(true, Arrays.equals(expected, included)); } @Test @@ -538,6 +542,19 @@ Reader reader = OrcFile.createReader(testFilePath, OrcFile.readerOptions(conf).filesystem(fs)); + assertEquals(3, OrcUtils.getFlattenedColumnsCount(inspector)); + boolean[] expected = new boolean[] {false, false, true}; + boolean[] included = OrcUtils.includeColumns("string1", "bytes1,string1", inspector); + assertEquals(true, Arrays.equals(expected, included)); + + expected = new boolean[] {false, false, false}; + included = OrcUtils.includeColumns("", "bytes1,string1", inspector); + assertEquals(true, Arrays.equals(expected, included)); + + expected = new boolean[] {false, false, false}; + included = OrcUtils.includeColumns(null, "bytes1,string1", inspector); + assertEquals(true, Arrays.equals(expected, included)); + // check the stats ColumnStatistics[] stats = reader.getStatistics(); assertEquals(4, stats[0].getNumberOfValues()); @@ -634,6 +651,12 @@ writer.close(); Reader reader = OrcFile.createReader(testFilePath, OrcFile.readerOptions(conf).filesystem(fs)); + + assertEquals(3, OrcUtils.getFlattenedColumnsCount(inspector)); + boolean[] expected = new boolean[] {false, true, false}; + boolean[] included = OrcUtils.includeColumns("int1", "int1,string1", inspector); + assertEquals(true, Arrays.equals(expected, included)); + Metadata metadata = reader.getMetadata(); int numStripes = metadata.getStripeStatistics().size(); assertEquals(3, numStripes); @@ -672,7 +695,7 @@ assertEquals(5000, ((StringColumnStatistics)ss3.getColumnStatistics()[2]).getSum()); RecordReaderImpl recordReader = (RecordReaderImpl) reader.rows(); - OrcProto.RowIndex[] index = recordReader.readRowIndex(0, null); + OrcProto.RowIndex[] index = recordReader.readRowIndex(0, null).getRowGroupIndex(); assertEquals(3, index.length); List items = index[1].getEntryList(); assertEquals(1, items.size()); @@ -682,7 +705,7 @@ assertEquals(0, items.get(0).getPositions(2)); assertEquals(1, items.get(0).getStatistics().getIntStatistics().getMinimum()); - index = recordReader.readRowIndex(1, null); + index = recordReader.readRowIndex(1, null).getRowGroupIndex(); assertEquals(3, index.length); items = index[1].getEntryList(); assertEquals(2, @@ -715,6 +738,44 @@ Reader reader = OrcFile.createReader(testFilePath, OrcFile.readerOptions(conf).filesystem(fs)); + assertEquals(24, OrcUtils.getFlattenedColumnsCount(inspector)); + boolean[] expected = new boolean[] {false, false, false, false, false, + false, false, false, false, false, + false, false, false, false, false, + false, false, false, false, false, + false, false, false, false}; + boolean[] included = OrcUtils.includeColumns("", + "boolean1,byte1,short1,int1,long1,float1,double1,bytes1,string1,middle,list,map", inspector); + assertEquals(true, Arrays.equals(expected, included)); + + expected = new boolean[] {false, true, false, false, false, + false, false, false, false, true, + true, true, true, true, true, + false, false, false, false, true, + true, true, true, true}; + included = OrcUtils.includeColumns("boolean1,string1,middle,map", + "boolean1,byte1,short1,int1,long1,float1,double1,bytes1,string1,middle,list,map", inspector); + assertEquals(true, Arrays.equals(expected, included)); + + expected = new boolean[] {false, true, false, false, false, + false, false, false, false, true, + true, true, true, true, true, + false, false, false, false, true, + true, true, true, true}; + included = OrcUtils.includeColumns("boolean1,string1,middle,map", + "boolean1,byte1,short1,int1,long1,float1,double1,bytes1,string1,middle,list,map", inspector); + assertEquals(true, Arrays.equals(expected, included)); + + expected = new boolean[] {false, true, true, true, true, + true, true, true, true, true, + true, true, true, true, true, + true, true, true, true, true, + true, true, true, true}; + included = OrcUtils.includeColumns( + "boolean1,byte1,short1,int1,long1,float1,double1,bytes1,string1,middle,list,map", + "boolean1,byte1,short1,int1,long1,float1,double1,bytes1,string1,middle,list,map", inspector); + assertEquals(true, Arrays.equals(expected, included)); + Metadata metadata = reader.getMetadata(); // check the stats @@ -1183,6 +1244,20 @@ writer.close(); Reader reader = OrcFile.createReader(testFilePath, OrcFile.readerOptions(conf).filesystem(fs)); + + assertEquals(6, OrcUtils.getFlattenedColumnsCount(inspector)); + boolean[] expected = new boolean[] {false, false, false, false, false, false}; + boolean[] included = OrcUtils.includeColumns("", "time,union,decimal", inspector); + assertEquals(true, Arrays.equals(expected, included)); + + expected = new boolean[] {false, true, false, false, false, true}; + included = OrcUtils.includeColumns("time,decimal", "time,union,decimal", inspector); + assertEquals(true, Arrays.equals(expected, included)); + + expected = new boolean[] {false, false, true, true, true, false}; + included = OrcUtils.includeColumns("union", "time,union,decimal", inspector); + assertEquals(true, Arrays.equals(expected, included)); + assertEquals(false, reader.getMetadataKeys().iterator().hasNext()); assertEquals(5309, reader.getNumberOfRows()); DecimalColumnStatistics stats = Index: ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestRecordReaderImpl.java (working copy) @@ -23,6 +23,12 @@ import static org.junit.Assert.assertThat; import static org.junit.Assert.assertTrue; +import java.io.IOException; +import java.io.InputStream; +import java.sql.Timestamp; +import java.util.ArrayList; +import java.util.List; + import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FileStatus; @@ -32,6 +38,7 @@ import org.apache.hadoop.fs.Seekable; import org.apache.hadoop.hive.common.DiskRange; import org.apache.hadoop.hive.common.type.HiveDecimal; +import org.apache.hadoop.hive.ql.io.filters.BloomFilter; import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.Location; import org.apache.hadoop.hive.ql.io.sarg.PredicateLeaf; import org.apache.hadoop.hive.ql.io.sarg.SearchArgument.TruthValue; @@ -42,11 +49,6 @@ import org.mockito.MockSettings; import org.mockito.Mockito; -import java.io.IOException; -import java.io.InputStream; -import java.util.ArrayList; -import java.util.List; - public class TestRecordReaderImpl { // can add .verboseLogging() to cause Mockito to log invocations @@ -288,6 +290,13 @@ return OrcProto.ColumnStatistics.newBuilder().setDateStatistics(dateStats.build()).build(); } + private static OrcProto.ColumnStatistics createTimestampStats(int min, int max) { + OrcProto.TimestampStatistics.Builder tsStats = OrcProto.TimestampStatistics.newBuilder(); + tsStats.setMinimum(min); + tsStats.setMaximum(max); + return OrcProto.ColumnStatistics.newBuilder().setTimestampStatistics(tsStats.build()).build(); + } + private static OrcProto.ColumnStatistics createDecimalStats(String min, String max) { OrcProto.DecimalStatistics.Builder decStats = OrcProto.DecimalStatistics.newBuilder(); decStats.setMinimum(min); @@ -295,6 +304,15 @@ return OrcProto.ColumnStatistics.newBuilder().setDecimalStatistics(decStats.build()).build(); } + private static OrcProto.ColumnStatistics createDecimalStats(String min, String max, + boolean hasNull) { + OrcProto.DecimalStatistics.Builder decStats = OrcProto.DecimalStatistics.newBuilder(); + decStats.setMinimum(min); + decStats.setMaximum(max); + return OrcProto.ColumnStatistics.newBuilder().setDecimalStatistics(decStats.build()) + .setHasNull(hasNull).build(); + } + @Test public void testGetMax() throws Exception { assertEquals(100L, RecordReaderImpl.getMax(ColumnStatisticsImpl.deserialize(createIntStats(10L, 100L)))); @@ -321,27 +339,27 @@ PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.FLOAT, "x", 15.0, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "15", null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(15), null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DECIMAL, "x", HiveDecimal.create(15), null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10, 100), pred, null)); } @Test @@ -349,27 +367,27 @@ PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDoubleStats(10.0, 100.0), pred)); + RecordReaderImpl.evaluatePredicateProto(createDoubleStats(10.0, 100.0), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.FLOAT, "x", 15.0, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDoubleStats(10.0, 100.0), pred)); + RecordReaderImpl.evaluatePredicateProto(createDoubleStats(10.0, 100.0), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "15", null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDoubleStats(10.0, 100.0), pred)); + RecordReaderImpl.evaluatePredicateProto(createDoubleStats(10.0, 100.0), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(15), null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDoubleStats(10.0, 100.0), pred)); + RecordReaderImpl.evaluatePredicateProto(createDoubleStats(10.0, 100.0), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DECIMAL, "x", HiveDecimal.create(15), null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDoubleStats(10.0, 100.0), pred)); + RecordReaderImpl.evaluatePredicateProto(createDoubleStats(10.0, 100.0), pred, null)); } @Test @@ -377,27 +395,27 @@ PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.INTEGER, "x", 100, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("10", "1000"), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("10", "1000"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.FLOAT, "x", 100.0, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("10", "1000"), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("10", "1000"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "100", null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("10", "1000"), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("10", "1000"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(100), null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createStringStats("10", "1000"), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("10", "1000"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DECIMAL, "x", HiveDecimal.create(100), null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("10", "1000"), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("10", "1000"), pred, null)); } @Test @@ -405,57 +423,57 @@ PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.FLOAT, "x", 15.0, null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "15", null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "1970-01-11", null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "15.1", null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "__a15__1", null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "2000-01-16", null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "1970-01-16", null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(15), null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(150), null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DECIMAL, "x", HiveDecimal.create(15), null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDateStats(10, 100), pred)); + RecordReaderImpl.evaluatePredicateProto(createDateStats(10, 100), pred, null)); } @@ -464,27 +482,27 @@ PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDecimalStats("10.0", "100.0"), pred)); + RecordReaderImpl.evaluatePredicateProto(createDecimalStats("10.0", "100.0"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.FLOAT, "x", 15.0, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDecimalStats("10.0", "100.0"), pred)); + RecordReaderImpl.evaluatePredicateProto(createDecimalStats("10.0", "100.0"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "15", null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDecimalStats("10.0", "100.0"), pred)); + RecordReaderImpl.evaluatePredicateProto(createDecimalStats("10.0", "100.0"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(15), null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createDecimalStats("10.0", "100.0"), pred)); + RecordReaderImpl.evaluatePredicateProto(createDecimalStats("10.0", "100.0"), pred, null)); pred = TestSearchArgumentImpl.createPredicateLeaf(PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DECIMAL, "x", HiveDecimal.create(15), null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createDecimalStats("10.0", "100.0"), pred)); + RecordReaderImpl.evaluatePredicateProto(createDecimalStats("10.0", "100.0"), pred, null)); } @@ -494,17 +512,17 @@ (PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(20L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(20L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(15L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(15L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 15L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 15L), pred, null)); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(0L, 10L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(0L, 10L), pred, null)); assertEquals(TruthValue.YES, - RecordReaderImpl.evaluatePredicate(createIntStats(15L, 15L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(15L, 15L), pred, null)); } @Test @@ -513,17 +531,17 @@ (PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(20L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(20L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(15L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(15L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 15L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 15L), pred, null)); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(0L, 10L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(0L, 10L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(15L, 15L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(15L, 15L), pred, null)); } @Test @@ -532,15 +550,15 @@ (PredicateLeaf.Operator.LESS_THAN, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(20L, 30L), lessThan)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(20L, 30L), lessThan, null)); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(15L, 30L), lessThan)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(15L, 30L), lessThan, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 30L), lessThan)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 30L), lessThan, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 15L), lessThan)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 15L), lessThan, null)); assertEquals(TruthValue.YES, - RecordReaderImpl.evaluatePredicate(createIntStats(0L, 10L), lessThan)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(0L, 10L), lessThan, null)); } @Test @@ -549,15 +567,15 @@ (PredicateLeaf.Operator.LESS_THAN_EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(20L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(20L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(15L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(15L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 30L), pred, null)); assertEquals(TruthValue.YES, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 15L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 15L), pred, null)); assertEquals(TruthValue.YES, - RecordReaderImpl.evaluatePredicate(createIntStats(0L, 10L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(0L, 10L), pred, null)); } @Test @@ -569,13 +587,13 @@ (PredicateLeaf.Operator.IN, PredicateLeaf.Type.INTEGER, "x", null, args); assertEquals(TruthValue.YES, - RecordReaderImpl.evaluatePredicate(createIntStats(20L, 20L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(20L, 20L), pred, null)); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(30L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(30L, 30L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 30L), pred, null)); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(12L, 18L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(12L, 18L), pred, null)); } @Test @@ -587,19 +605,19 @@ (PredicateLeaf.Operator.BETWEEN, PredicateLeaf.Type.INTEGER, "x", null, args); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(0L, 5L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(0L, 5L), pred, null)); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(30L, 40L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(30L, 40L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(5L, 15L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(5L, 15L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(15L, 25L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(15L, 25L), pred, null)); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createIntStats(5L, 25L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(5L, 25L), pred, null)); assertEquals(TruthValue.YES, - RecordReaderImpl.evaluatePredicate(createIntStats(10L, 20L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(10L, 20L), pred, null)); assertEquals(TruthValue.YES, - RecordReaderImpl.evaluatePredicate(createIntStats(12L, 18L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(12L, 18L), pred, null)); } @Test @@ -608,7 +626,7 @@ (PredicateLeaf.Operator.IS_NULL, PredicateLeaf.Type.INTEGER, "x", null, null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createIntStats(20L, 30L), pred)); + RecordReaderImpl.evaluatePredicateProto(createIntStats(20L, 30L), pred, null)); } @@ -618,17 +636,17 @@ (PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.STRING, "x", "c", null); assertEquals(TruthValue.NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("d", "e", true), pred)); // before + RecordReaderImpl.evaluatePredicateProto(createStringStats("d", "e", true), pred, null)); // before assertEquals(TruthValue.NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("a", "b", true), pred)); // after + RecordReaderImpl.evaluatePredicateProto(createStringStats("a", "b", true), pred, null)); // after assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "c", true), pred)); // max + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "c", true), pred, null)); // max assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "d", true), pred)); // min + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "d", true), pred, null)); // min assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "d", true), pred)); // middle + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "d", true), pred, null)); // middle assertEquals(TruthValue.YES_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "c", true), pred)); // same + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "c", true), pred, null)); // same } @Test @@ -637,17 +655,17 @@ (PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "c", null); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createStringStats("d", "e", true), pred)); // before + RecordReaderImpl.evaluatePredicateProto(createStringStats("d", "e", true), pred, null)); // before assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createStringStats("a", "b", true), pred)); // after + RecordReaderImpl.evaluatePredicateProto(createStringStats("a", "b", true), pred, null)); // after assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "c", true), pred)); // max + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "c", true), pred, null)); // max assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "d", true), pred)); // min + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "d", true), pred, null)); // min assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "d", true), pred)); // middle + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "d", true), pred, null)); // middle assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "c", true), pred)); // same + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "c", true), pred, null)); // same } @Test @@ -656,17 +674,17 @@ (PredicateLeaf.Operator.LESS_THAN, PredicateLeaf.Type.STRING, "x", "c", null); assertEquals(TruthValue.NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("d", "e", true), pred)); // before + RecordReaderImpl.evaluatePredicateProto(createStringStats("d", "e", true), pred, null)); // before assertEquals(TruthValue.YES_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("a", "b", true), pred)); // after + RecordReaderImpl.evaluatePredicateProto(createStringStats("a", "b", true), pred, null)); // after assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "c", true), pred)); // max + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "c", true), pred, null)); // max assertEquals(TruthValue.NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "d", true), pred)); // min + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "d", true), pred, null)); // min assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "d", true), pred)); // middle + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "d", true), pred, null)); // middle assertEquals(TruthValue.NO_NULL, // min, same stats - RecordReaderImpl.evaluatePredicate(createStringStats("c", "c", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "c", true), pred, null)); } @Test @@ -675,17 +693,17 @@ (PredicateLeaf.Operator.LESS_THAN_EQUALS, PredicateLeaf.Type.STRING, "x", "c", null); assertEquals(TruthValue.NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("d", "e", true), pred)); // before + RecordReaderImpl.evaluatePredicateProto(createStringStats("d", "e", true), pred, null)); // before assertEquals(TruthValue.YES_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("a", "b", true), pred)); // after + RecordReaderImpl.evaluatePredicateProto(createStringStats("a", "b", true), pred, null)); // after assertEquals(TruthValue.YES_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "c", true), pred)); // max + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "c", true), pred, null)); // max assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "d", true), pred)); // min + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "d", true), pred, null)); // min assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "d", true), pred)); // middle + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "d", true), pred, null)); // middle assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "c", true), pred)); // same + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "c", true), pred, null)); // same } @Test @@ -697,17 +715,17 @@ (PredicateLeaf.Operator.IN, PredicateLeaf.Type.STRING, "x", null, args); assertEquals(TruthValue.NO_NULL, // before & after - RecordReaderImpl.evaluatePredicate(createStringStats("d", "e", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("d", "e", true), pred, null)); assertEquals(TruthValue.NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("a", "b", true), pred)); // after + RecordReaderImpl.evaluatePredicateProto(createStringStats("a", "b", true), pred, null)); // after assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("e", "f", true), pred)); // max + RecordReaderImpl.evaluatePredicateProto(createStringStats("e", "f", true), pred, null)); // max assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "d", true), pred)); // min + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "d", true), pred, null)); // min assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "d", true), pred)); // middle + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "d", true), pred, null)); // middle assertEquals(TruthValue.YES_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "c", true), pred)); // same + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "c", true), pred, null)); // same } @Test @@ -719,31 +737,31 @@ (PredicateLeaf.Operator.BETWEEN, PredicateLeaf.Type.STRING, "x", null, args); assertEquals(TruthValue.YES_NULL, // before & after - RecordReaderImpl.evaluatePredicate(createStringStats("d", "e", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("d", "e", true), pred, null)); assertEquals(TruthValue.YES_NULL, // before & max - RecordReaderImpl.evaluatePredicate(createStringStats("e", "f", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("e", "f", true), pred, null)); assertEquals(TruthValue.NO_NULL, // before & before - RecordReaderImpl.evaluatePredicate(createStringStats("h", "g", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("h", "g", true), pred, null)); assertEquals(TruthValue.YES_NO_NULL, // before & min - RecordReaderImpl.evaluatePredicate(createStringStats("f", "g", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("f", "g", true), pred, null)); assertEquals(TruthValue.YES_NO_NULL, // before & middle - RecordReaderImpl.evaluatePredicate(createStringStats("e", "g", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("e", "g", true), pred, null)); assertEquals(TruthValue.YES_NULL, // min & after - RecordReaderImpl.evaluatePredicate(createStringStats("c", "e", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "e", true), pred, null)); assertEquals(TruthValue.YES_NULL, // min & max - RecordReaderImpl.evaluatePredicate(createStringStats("c", "f", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "f", true), pred, null)); assertEquals(TruthValue.YES_NO_NULL, // min & middle - RecordReaderImpl.evaluatePredicate(createStringStats("c", "g", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "g", true), pred, null)); assertEquals(TruthValue.NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("a", "b", true), pred)); // after + RecordReaderImpl.evaluatePredicateProto(createStringStats("a", "b", true), pred, null)); // after assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("a", "c", true), pred)); // max + RecordReaderImpl.evaluatePredicateProto(createStringStats("a", "c", true), pred, null)); // max assertEquals(TruthValue.YES_NO_NULL, - RecordReaderImpl.evaluatePredicate(createStringStats("b", "d", true), pred)); // middle + RecordReaderImpl.evaluatePredicateProto(createStringStats("b", "d", true), pred, null)); // middle assertEquals(TruthValue.YES_NULL, // min & after, same stats - RecordReaderImpl.evaluatePredicate(createStringStats("c", "c", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "c", true), pred, null)); } @Test @@ -752,9 +770,9 @@ (PredicateLeaf.Operator.IS_NULL, PredicateLeaf.Type.STRING, "x", null, null); assertEquals(TruthValue.YES_NO, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "d", true), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "d", true), pred, null)); assertEquals(TruthValue.NO, - RecordReaderImpl.evaluatePredicate(createStringStats("c", "d", false), pred)); + RecordReaderImpl.evaluatePredicateProto(createStringStats("c", "d", false), pred, null)); } @Test @@ -1132,4 +1150,350 @@ 41000, 51000 + RecordReaderUtils.WORST_UNCOMPRESSED_SLOP, 51000, 95000, 95000, 97000, 97000, 100000))); } + + @Test + public void testIntNullSafeEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong(i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createIntStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong(15); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testIntEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.INTEGER, "x", 15L, null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong(i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createIntStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong(15); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testIntInBloomFilter() throws Exception { + List args = new ArrayList(); + args.add(15); + args.add(19); + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf + (PredicateLeaf.Operator.IN, PredicateLeaf.Type.INTEGER, + "x", null, args); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong(i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createIntStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong(19); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong(15); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDoubleNullSafeEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.FLOAT, "x", 15.0, null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addDouble(i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDoubleStats(10.0, 100.0)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addDouble(15.0); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDoubleEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.FLOAT, "x", 15.0, null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addDouble(i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDoubleStats(10.0, 100.0)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addDouble(15.0); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDoubleInBloomFilter() throws Exception { + List args = new ArrayList(); + args.add(15.0); + args.add(19.0); + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf + (PredicateLeaf.Operator.IN, PredicateLeaf.Type.FLOAT, + "x", null, args); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addDouble(i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDoubleStats(10.0, 100.0)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addDouble(19.0); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addDouble(15.0); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testStringNullSafeEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.STRING, "x", "str_15", null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addString("str_" + i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createStringStats("str_10", "str_200")); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString("str_15"); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testStringEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.STRING, "x", "str_15", null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addString("str_" + i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createStringStats("str_10", "str_200")); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString("str_15"); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testStringInBloomFilter() throws Exception { + List args = new ArrayList(); + args.add("str_15"); + args.add("str_19"); + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf + (PredicateLeaf.Operator.IN, PredicateLeaf.Type.STRING, + "x", null, args); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addString("str_" + i); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createStringStats("str_10", "str_200")); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString("str_19"); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString("str_15"); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDateWritableNullSafeEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(15), + null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong((new DateWritable(i)).getDays()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDateStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new DateWritable(15)).getDays()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDateWritableEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.DATE, "x", new DateWritable(15), null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong((new DateWritable(i)).getDays()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDateStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new DateWritable(15)).getDays()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDateWritableInBloomFilter() throws Exception { + List args = new ArrayList(); + args.add(new DateWritable(15)); + args.add(new DateWritable(19)); + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf + (PredicateLeaf.Operator.IN, PredicateLeaf.Type.DATE, + "x", null, args); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong((new DateWritable(i)).getDays()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDateStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new DateWritable(19)).getDays()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new DateWritable(15)).getDays()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testTimestampNullSafeEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.TIMESTAMP, "x", + new Timestamp(15), + null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong((new Timestamp(i)).getTime()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createTimestampStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new Timestamp(15)).getTime()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testTimestampEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.TIMESTAMP, "x", new Timestamp(15), null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong((new Timestamp(i)).getTime()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createTimestampStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new Timestamp(15)).getTime()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testTimestampInBloomFilter() throws Exception { + List args = new ArrayList(); + args.add(new Timestamp(15)); + args.add(new Timestamp(19)); + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf + (PredicateLeaf.Operator.IN, PredicateLeaf.Type.TIMESTAMP, + "x", null, args); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addLong((new Timestamp(i)).getTime()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createTimestampStats(10, 100)); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new Timestamp(19)).getTime()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addLong((new Timestamp(15)).getTime()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDecimalNullSafeEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.NULL_SAFE_EQUALS, PredicateLeaf.Type.DECIMAL, "x", + HiveDecimal.create(15), + null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addString(HiveDecimal.create(i).toString()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDecimalStats("10", "200")); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString(HiveDecimal.create(15).toString()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDecimalEqualsBloomFilter() throws Exception { + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf( + PredicateLeaf.Operator.EQUALS, PredicateLeaf.Type.DECIMAL, "x", HiveDecimal.create(15), + null); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addString(HiveDecimal.create(i).toString()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDecimalStats("10", "200")); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString(HiveDecimal.create(15).toString()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testDecimalInBloomFilter() throws Exception { + List args = new ArrayList(); + args.add(HiveDecimal.create(15)); + args.add(HiveDecimal.create(19)); + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf + (PredicateLeaf.Operator.IN, PredicateLeaf.Type.DECIMAL, + "x", null, args); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addString(HiveDecimal.create(i).toString()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDecimalStats("10", "200")); + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString(HiveDecimal.create(19).toString()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString(HiveDecimal.create(15).toString()); + assertEquals(TruthValue.YES_NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } + + @Test + public void testNullsInBloomFilter() throws Exception { + List args = new ArrayList(); + args.add(HiveDecimal.create(15)); + args.add(null); + args.add(HiveDecimal.create(19)); + PredicateLeaf pred = TestSearchArgumentImpl.createPredicateLeaf + (PredicateLeaf.Operator.IN, PredicateLeaf.Type.DECIMAL, + "x", null, args); + BloomFilter bf = new BloomFilter(10000); + for (int i = 20; i < 1000; i++) { + bf.addString(HiveDecimal.create(i).toString()); + } + ColumnStatistics cs = ColumnStatisticsImpl.deserialize(createDecimalStats("10", "200")); + // hasNull is false, so bloom filter should return NO + assertEquals(TruthValue.NO, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + cs = ColumnStatisticsImpl.deserialize(createDecimalStats("10", "200", true)); + // hasNull is true, so bloom filter should return YES_NO_NULL + assertEquals(TruthValue.YES_NO_NULL, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString(HiveDecimal.create(19).toString()); + assertEquals(TruthValue.YES_NO_NULL, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + + bf.addString(HiveDecimal.create(15).toString()); + assertEquals(TruthValue.YES_NO_NULL, RecordReaderImpl.evaluatePredicate(cs, pred, bf)); + } } Index: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestDataWritableWriter.java (working copy) @@ -13,9 +13,27 @@ */ package org.apache.hadoop.hive.ql.io.parquet; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.hive.ql.io.parquet.serde.ArrayWritableObjectInspector; +import org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe; import org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter; +import org.apache.hadoop.hive.serde2.io.ByteWritable; +import org.apache.hadoop.hive.serde2.io.ShortWritable; +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; +import org.apache.hadoop.hive.serde2.SerDeException; +import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.io.DoubleWritable; -import org.apache.hadoop.io.*; +import org.apache.hadoop.hive.serde2.typeinfo.StructTypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoFactory; +import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils; +import org.apache.hadoop.io.ArrayWritable; +import org.apache.hadoop.io.BooleanWritable; +import org.apache.hadoop.io.BytesWritable; +import org.apache.hadoop.io.FloatWritable; +import org.apache.hadoop.io.IntWritable; +import org.apache.hadoop.io.LongWritable; +import org.apache.hadoop.io.Writable; import org.junit.Before; import org.junit.Test; import org.mockito.InOrder; @@ -27,6 +45,10 @@ import parquet.schema.MessageTypeParser; import java.io.UnsupportedEncodingException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Properties; import static org.junit.Assert.*; import static org.mockito.Mockito.*; @@ -62,6 +84,10 @@ inOrder.verify(mockRecordConsumer).addInteger(value); } + private void addLong(int value) { + inOrder.verify(mockRecordConsumer).addLong(value); + } + private void addFloat(float value) { inOrder.verify(mockRecordConsumer).addFloat(value); } @@ -88,6 +114,12 @@ private Writable createNull() { return null; } + private ByteWritable createTinyInt(byte value) { return new ByteWritable(value); } + + private ShortWritable createSmallInt(short value) { return new ShortWritable(value); } + + private LongWritable createBigInt(long value) { return new LongWritable(value); } + private IntWritable createInt(int value) { return new IntWritable(value); } @@ -116,20 +148,68 @@ return new ArrayWritable(Writable.class, createGroup(values).get()); } - private void writeParquetRecord(String schemaStr, ArrayWritable record) { - MessageType schema = MessageTypeParser.parseMessageType(schemaStr); - DataWritableWriter hiveParquetWriter = new DataWritableWriter(mockRecordConsumer, schema); + private List createHiveColumnsFrom(final String columnNamesStr) { + List columnNames; + if (columnNamesStr.length() == 0) { + columnNames = new ArrayList(); + } else { + columnNames = Arrays.asList(columnNamesStr.split(",")); + } + + return columnNames; + } + + private List createHiveTypeInfoFrom(final String columnsTypeStr) { + List columnTypes; + + if (columnsTypeStr.length() == 0) { + columnTypes = new ArrayList(); + } else { + columnTypes = TypeInfoUtils.getTypeInfosFromTypeString(columnsTypeStr); + } + + return columnTypes; + } + + private ArrayWritableObjectInspector getObjectInspector(final String columnNames, final String columnTypes) { + List columnTypeList = createHiveTypeInfoFrom(columnTypes); + List columnNameList = createHiveColumnsFrom(columnNames); + StructTypeInfo rowTypeInfo = (StructTypeInfo) TypeInfoFactory.getStructTypeInfo(columnNameList, columnTypeList); + + return new ArrayWritableObjectInspector(rowTypeInfo); + } + + private ParquetHiveRecord getParquetWritable(String columnNames, String columnTypes, ArrayWritable record) throws SerDeException { + Properties recordProperties = new Properties(); + recordProperties.setProperty("columns", columnNames); + recordProperties.setProperty("columns.types", columnTypes); + + ParquetHiveSerDe serDe = new ParquetHiveSerDe(); + SerDeUtils.initializeSerDe(serDe, new Configuration(), recordProperties, null); + + return new ParquetHiveRecord(serDe.deserialize(record), getObjectInspector(columnNames, columnTypes)); + } + + private void writeParquetRecord(String schema, ParquetHiveRecord record) throws SerDeException { + MessageType fileSchema = MessageTypeParser.parseMessageType(schema); + DataWritableWriter hiveParquetWriter = new DataWritableWriter(mockRecordConsumer, fileSchema); hiveParquetWriter.write(record); } @Test public void testSimpleType() throws Exception { - String schemaStr = "message hive_schema {\n" + String columnNames = "int,double,boolean,float,string,tinyint,smallint,bigint"; + String columnTypes = "int,double,boolean,float,string,tinyint,smallint,bigint"; + + String fileSchema = "message hive_schema {\n" + " optional int32 int;\n" + " optional double double;\n" + " optional boolean boolean;\n" + " optional float float;\n" - + " optional binary string;\n" + + " optional binary string (UTF8);\n" + + " optional int32 tinyint;\n" + + " optional int32 smallint;\n" + + " optional int64 bigint;\n" + "}\n"; ArrayWritable hiveRecord = createGroup( @@ -137,11 +217,14 @@ createDouble(1.0), createBoolean(true), createFloat(1.0f), - createString("one") + createString("one"), + createTinyInt((byte)1), + createSmallInt((short)1), + createBigInt((long)1) ); // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); // Verify record was written correctly to Parquet startMessage(); @@ -160,12 +243,24 @@ startField("string", 4); addString("one"); endField("string", 4); + startField("tinyint", 5); + addInteger(1); + endField("tinyint", 5); + startField("smallint", 6); + addInteger(1); + endField("smallint", 6); + startField("bigint", 7); + addLong(1); + endField("bigint", 7); endMessage(); } @Test public void testStructType() throws Exception { - String schemaStr = "message hive_schema {\n" + String columnNames = "structCol"; + String columnTypes = "struct"; + + String fileSchema = "message hive_schema {\n" + " optional group structCol {\n" + " optional int32 a;\n" + " optional double b;\n" @@ -182,7 +277,7 @@ ); // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); // Verify record was written correctly to Parquet startMessage(); @@ -204,9 +299,12 @@ @Test public void testArrayType() throws Exception { - String schemaStr = "message hive_schema {\n" + String columnNames = "arrayCol"; + String columnTypes = "array"; + + String fileSchema = "message hive_schema {\n" + " optional group arrayCol (LIST) {\n" - + " repeated group bag {\n" + + " repeated group array {\n" + " optional int32 array_element;\n" + " }\n" + " }\n" @@ -223,13 +321,13 @@ ); // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); // Verify record was written correctly to Parquet startMessage(); startField("arrayCol", 0); startGroup(); - startField("bag", 0); + startField("array", 0); startGroup(); startField("array_element", 0); addInteger(1); @@ -242,7 +340,7 @@ addInteger(2); endField("array_element", 0); endGroup(); - endField("bag", 0); + endField("array", 0); endGroup(); endField("arrayCol", 0); endMessage(); @@ -250,7 +348,10 @@ @Test public void testMapType() throws Exception { - String schemaStr = "message hive_schema {\n" + String columnNames = "mapCol"; + String columnTypes = "map"; + + String fileSchema = "message hive_schema {\n" + " optional group mapCol (MAP) {\n" + " repeated group map (MAP_KEY_VALUE) {\n" + " required binary key;\n" @@ -279,7 +380,7 @@ ); // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); // Verify record was written correctly to Parquet startMessage(); @@ -315,12 +416,15 @@ @Test public void testArrayOfArrays() throws Exception { - String schemaStr = "message hive_schema {\n" + String columnNames = "array_of_arrays"; + String columnTypes = "array>"; + + String fileSchema = "message hive_schema {\n" + " optional group array_of_arrays (LIST) {\n" + " repeated group array {\n" - + " required group element (LIST) {\n" + + " optional group array_element (LIST) {\n" + " repeated group array {\n" - + " required int32 element;\n" + + " optional int32 array_element;\n" + " }\n" + " }\n" + " }\n" @@ -341,7 +445,7 @@ ); // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); // Verify record was written correctly to Parquet startMessage(); @@ -349,22 +453,22 @@ startGroup(); startField("array", 0); startGroup(); - startField("element", 0); + startField("array_element", 0); startGroup(); startField("array", 0); startGroup(); - startField("element", 0); + startField("array_element", 0); addInteger(1); - endField("element", 0); + endField("array_element", 0); endGroup(); startGroup(); - startField("element", 0); + startField("array_element", 0); addInteger(2); - endField("element", 0); + endField("array_element", 0); endGroup(); endField("array", 0); endGroup(); - endField("element", 0); + endField("array_element", 0); endGroup(); endField("array", 0); endGroup(); @@ -373,124 +477,63 @@ } @Test - public void testGroupFieldIsNotArrayWritable() throws Exception { - String schemaStr = "message hive_schema {\n" - + " optional group a {\n" - + " optional int32 b;\n" - + " }\n" - + "}\n"; + public void testExpectedStructTypeOnRecord() throws Exception { + String columnNames = "structCol"; + String columnTypes = "int"; ArrayWritable hiveRecord = createGroup( - createInt(1) + createInt(1) ); + String fileSchema = "message hive_schema {\n" + + " optional group structCol {\n" + + " optional int32 int;\n" + + " }\n" + + "}\n"; + try { - // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); fail(); } catch (RuntimeException e) { - assertEquals("Parquet record is malformed: Field value is not an ArrayWritable object: " + - "optional group a {\n optional int32 b;\n}", e.getMessage()); + assertEquals("Parquet record is malformed: Invalid data type: expected STRUCT type, but found: PRIMITIVE", e.getMessage()); } } @Test - public void testArrayGroupElementIsNotArrayWritable() throws Exception { - String schemaStr = "message hive_schema {\n" - + " optional group array_of_arrays (LIST) {\n" - + " repeated group array {\n" - + " required group element (LIST) {\n" - + " required int32 element;\n" - + " }\n" - + " }\n" - + " }\n" - + "}\n"; + public void testExpectedArrayTypeOnRecord() throws Exception { + String columnNames = "arrayCol"; + String columnTypes = "int"; ArrayWritable hiveRecord = createGroup( - createGroup( - createArray( - createInt(1) - ) - ) + createInt(1) ); - try { - // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); - fail(); - } catch (RuntimeException e) { - assertEquals("Parquet record is malformed: Field value is not an ArrayWritable object: " + - "required group element (LIST) {\n required int32 element;\n}", e.getMessage()); - } - } - - @Test - public void testMapElementIsNotArrayWritable() throws Exception { - String schemaStr = "message hive_schema {\n" - + " optional group mapCol (MAP) {\n" - + " repeated group map (MAP_KEY_VALUE) {\n" - + " required binary key;\n" - + " optional group value {\n" - + " required int32 value;" - + " }\n" + String fileSchema = "message hive_schema {\n" + + " optional group arrayCol (LIST) {\n" + + " repeated group bag {\n" + + " optional int32 array_element;\n" + " }\n" + " }\n" + "}\n"; - ArrayWritable hiveRecord = createGroup( - createGroup( - createArray( - createGroup( - createString("key1"), - createInt(1) - ) - ) - ) - ); - try { - // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); fail(); } catch (RuntimeException e) { - assertEquals( - "Parquet record is malformed: Field value is not an ArrayWritable object: " + - "optional group value {\n required int32 value;\n}", e.getMessage()); + assertEquals("Parquet record is malformed: Invalid data type: expected LIST type, but found: PRIMITIVE", e.getMessage()); } } @Test - public void testMapKeyValueIsNotArrayWritable() throws Exception { - String schemaStr = "message hive_schema {\n" - + " optional group mapCol (MAP) {\n" - + " repeated group map (MAP_KEY_VALUE) {\n" - + " required binary key;\n" - + " optional int32 value;\n" - + " }\n" - + " }\n" - + "}\n"; + public void testExpectedMapTypeOnRecord() throws Exception { + String columnNames = "mapCol"; + String columnTypes = "int"; ArrayWritable hiveRecord = createGroup( - createGroup( - createArray( - createString("key1"), - createInt(1) - ) - ) + createInt(1) ); - try { - // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); - fail(); - } catch (RuntimeException e) { - assertEquals("Parquet record is malformed: Map key-value pair is not an ArrayWritable object on record 0", e.getMessage()); - } - } - - @Test - public void testMapKeyValueIsNull() throws Exception { - String schemaStr = "message hive_schema {\n" + String fileSchema = "message hive_schema {\n" + " optional group mapCol (MAP) {\n" + " repeated group map (MAP_KEY_VALUE) {\n" + " required binary key;\n" @@ -499,20 +542,11 @@ + " }\n" + "}\n"; - ArrayWritable hiveRecord = createGroup( - createGroup( - createArray( - createNull() - ) - ) - ); - try { - // Write record to Parquet format - writeParquetRecord(schemaStr, hiveRecord); + writeParquetRecord(fileSchema, getParquetWritable(columnNames, columnTypes, hiveRecord)); fail(); } catch (RuntimeException e) { - assertEquals("Parquet record is malformed: Map key-value pair is null on record 0", e.getMessage()); + assertEquals("Parquet record is malformed: Invalid data type: expected MAP type, but found: PRIMITIVE", e.getMessage()); } } } Index: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestMapredParquetOutputFormat.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestMapredParquetOutputFormat.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestMapredParquetOutputFormat.java (working copy) @@ -24,7 +24,7 @@ import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport; import org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper; -import org.apache.hadoop.io.ArrayWritable; +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.util.Progressable; import org.junit.Test; @@ -41,7 +41,7 @@ @SuppressWarnings("unchecked") @Test public void testConstructorWithFormat() { - new MapredParquetOutputFormat((ParquetOutputFormat) mock(ParquetOutputFormat.class)); + new MapredParquetOutputFormat((ParquetOutputFormat) mock(ParquetOutputFormat.class)); } @Test @@ -62,7 +62,7 @@ tableProps.setProperty("columns.types", "int:int"); final Progressable mockProgress = mock(Progressable.class); - final ParquetOutputFormat outputFormat = (ParquetOutputFormat) mock(ParquetOutputFormat.class); + final ParquetOutputFormat outputFormat = (ParquetOutputFormat) mock(ParquetOutputFormat.class); JobConf jobConf = new JobConf(); @@ -70,7 +70,7 @@ new MapredParquetOutputFormat(outputFormat) { @Override protected ParquetRecordWriterWrapper getParquerRecordWriterWrapper( - ParquetOutputFormat realOutputFormat, + ParquetOutputFormat realOutputFormat, JobConf jobConf, String finalOutPath, Progressable progress, Index: ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetSerDe.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetSerDe.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetSerDe.java (working copy) @@ -23,6 +23,7 @@ import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.io.ByteWritable; import org.apache.hadoop.hive.serde2.io.DoubleWritable; +import org.apache.hadoop.hive.serde2.io.ParquetHiveRecord; import org.apache.hadoop.hive.serde2.io.ShortWritable; import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; import org.apache.hadoop.io.ArrayWritable; @@ -96,9 +97,9 @@ assertEquals("deserialization gives the wrong object", t, row); // Serialize - final ArrayWritable serializedArr = (ArrayWritable) serDe.serialize(row, oi); - assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(), serializedArr.get().length); - assertTrue("serialized object should be equal to starting object", arrayWritableEquals(t, serializedArr)); + final ParquetHiveRecord serializedArr = (ParquetHiveRecord) serDe.serialize(row, oi); + assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(), ((ArrayWritable)serializedArr.getObject()).get().length); + assertTrue("serialized object should be equal to starting object", arrayWritableEquals(t, (ArrayWritable)serializedArr.getObject())); } private Properties createProperties() { Index: ql/src/test/org/apache/hadoop/hive/ql/parse/TestIUD.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/parse/TestIUD.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/parse/TestIUD.java (working copy) @@ -207,7 +207,27 @@ "(TOK_WHERE (= (TOK_TABLE_OR_COL b) 9))))", ast.toStringTree()); } + /** + * same as testInsertIntoTableAsSelectFromNamedVirtTable but with column list on target table + * @throws ParseException + */ @Test + public void testInsertIntoTableAsSelectFromNamedVirtTableNamedCol() throws ParseException { + ASTNode ast = parse("insert into page_view(c1,c2) select a,b as c from (values (1,2),(3,4)) as VC(a,b) where b = 9"); + Assert.assertEquals("AST doesn't match", + "(TOK_QUERY " + + "(TOK_FROM " + + "(TOK_VIRTUAL_TABLE " + + "(TOK_VIRTUAL_TABREF (TOK_TABNAME VC) (TOK_COL_NAME a b)) " + + "(TOK_VALUES_TABLE (TOK_VALUE_ROW 1 2) (TOK_VALUE_ROW 3 4)))) " + + "(TOK_INSERT (TOK_INSERT_INTO (TOK_TAB (TOK_TABNAME page_view)) (TOK_TABCOLNAME c1 c2)) " + + "(TOK_SELECT " + + "(TOK_SELEXPR (TOK_TABLE_OR_COL a)) " + + "(TOK_SELEXPR (TOK_TABLE_OR_COL b) c)) " + + "(TOK_WHERE (= (TOK_TABLE_OR_COL b) 9))))", + ast.toStringTree()); + } + @Test public void testInsertIntoTableFromAnonymousTable1Row() throws ParseException { ASTNode ast = parse("insert into page_view values(1,2)"); Assert.assertEquals("AST doesn't match", @@ -220,7 +240,33 @@ "(TOK_SELECT (TOK_SELEXPR TOK_ALLCOLREF))))", ast.toStringTree()); } + /** + * Same as testInsertIntoTableFromAnonymousTable1Row but with column list on target table + * @throws ParseException + */ @Test + public void testInsertIntoTableFromAnonymousTable1RowNamedCol() throws ParseException { + ASTNode ast = parse("insert into page_view(a,b) values(1,2)"); + Assert.assertEquals("AST doesn't match", + "(TOK_QUERY " + + "(TOK_FROM " + + "(TOK_VIRTUAL_TABLE " + + "(TOK_VIRTUAL_TABREF TOK_ANONYMOUS) " + + "(TOK_VALUES_TABLE (TOK_VALUE_ROW 1 2))" + + ")" + + ") " + + "(TOK_INSERT " + + "(TOK_INSERT_INTO " + + "(TOK_TAB (TOK_TABNAME page_view)) " + + "(TOK_TABCOLNAME a b)" +//this is "extra" piece we get vs previous query + ") " + + "(TOK_SELECT " + + "(TOK_SELEXPR TOK_ALLCOLREF)" + + ")" + + ")" + + ")", ast.toStringTree()); + } + @Test public void testInsertIntoTableFromAnonymousTable() throws ParseException { ASTNode ast = parse("insert into table page_view values(-1,2),(3,+4)"); Assert.assertEquals("AST doesn't match", Index: ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/parse/TestMacroSemanticAnalyzer.java (working copy) @@ -66,12 +66,12 @@ @Test public void testDropMacroExistsDoNotIgnoreErrors() throws Exception { conf.setBoolVar(ConfVars.DROPIGNORESNONEXISTENT, false); - FunctionRegistry.registerGenericUDF(false, "SOME_MACRO", GenericUDFMacro.class); + FunctionRegistry.registerTemporaryUDF("SOME_MACRO", GenericUDFMacro.class); analyze(parse("DROP TEMPORARY MACRO SOME_MACRO")); } @Test public void testDropMacro() throws Exception { - FunctionRegistry.registerGenericUDF(false, "SOME_MACRO", GenericUDFMacro.class); + FunctionRegistry.registerTemporaryUDF("SOME_MACRO", GenericUDFMacro.class); analyze(parse("DROP TEMPORARY MACRO SOME_MACRO")); } @Test(expected = SemanticException.class) Index: ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAddMonths.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAddMonths.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFAddMonths.java (working copy) @@ -46,6 +46,7 @@ runAndVerify("2016-02-29", -12, "2015-02-28", udf); runAndVerify("2016-01-29", 1, "2016-02-29", udf); runAndVerify("2016-02-29", -1, "2016-01-31", udf); + runAndVerify("2014-01-32", 1, "2014-03-01", udf); } private void runAndVerify(String str, int months, String expResult, GenericUDF udf) Index: ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLastDay.java =================================================================== --- ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLastDay.java (revision 1661246) +++ ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLastDay.java (working copy) @@ -43,6 +43,8 @@ runAndVerify("2016-02-03", "2016-02-29", udf); runAndVerify("2016-02-28", "2016-02-29", udf); runAndVerify("2016-02-29", "2016-02-29", udf); + runAndVerify("01/14/2014", null, udf); + runAndVerify(null, null, udf); runAndVerify("2014-01-01 10:30:45", "2014-01-31", udf); runAndVerify("2014-01-14 10:30:45", "2014-01-31", udf); @@ -56,9 +58,9 @@ private void runAndVerify(String str, String expResult, GenericUDF udf) throws HiveException { - DeferredObject valueObj0 = new DeferredJavaObject(new Text(str)); + DeferredObject valueObj0 = new DeferredJavaObject(str != null ? new Text(str) : null); DeferredObject[] args = { valueObj0 }; Text output = (Text) udf.evaluate(args); - assertEquals("last_day() test ", expResult, output.toString()); + assertEquals("last_day() test ", expResult, output != null ? output.toString() : null); } } Index: ql/src/test/queries/clientnegative/drop_native_udf.q =================================================================== --- ql/src/test/queries/clientnegative/drop_native_udf.q (revision 1661246) +++ ql/src/test/queries/clientnegative/drop_native_udf.q (working copy) @@ -1 +1 @@ -DROP TEMPORARY FUNCTION max; +DROP FUNCTION max; Index: ql/src/test/queries/clientnegative/union3.q =================================================================== --- ql/src/test/queries/clientnegative/union3.q (revision 1661246) +++ ql/src/test/queries/clientnegative/union3.q (working copy) @@ -1,5 +0,0 @@ --- Ensure that UNION ALL columns are in the correct order on both sides --- Ensure that the appropriate error message is propagated -CREATE TABLE IF NOT EXISTS union3 (bar int, baz int); -SELECT * FROM ( SELECT f.bar, f.baz FROM union3 f UNION ALL SELECT b.baz, b.bar FROM union3 b ) c; -DROP TABLE union3; Index: ql/src/test/queries/clientpositive/cbo_union.q =================================================================== --- ql/src/test/queries/clientpositive/cbo_union.q (revision 1661246) +++ ql/src/test/queries/clientpositive/cbo_union.q (working copy) @@ -7,7 +7,7 @@ -- SORT_QUERY_RESULTS -- 11. Union All -select * from cbo_t1 order by key, c_boolean, value, dt union all select * from cbo_t2 order by key, c_boolean, value, dt; +select * from (select * from cbo_t1 order by key, c_boolean, value, dt)a union all select * from (select * from cbo_t2 order by key, c_boolean, value, dt)b; select key from (select key, c_int from (select * from cbo_t1 union all select * from cbo_t2 where cbo_t2.key >=0)r1 union all select key, c_int from cbo_t3)r2 where key >=0 order by key; select r2.key from (select key, c_int from (select key, c_int from cbo_t1 union all select key, c_int from cbo_t3 )r1 union all select key, c_int from cbo_t3)r2 join (select key, c_int from (select * from cbo_t1 union all select * from cbo_t2 where cbo_t2.key >=0)r1 union all select key, c_int from cbo_t3)r3 on r2.key=r3.key where r3.key >=0 order by r2.key; Index: ql/src/test/queries/clientpositive/explode_null.q =================================================================== --- ql/src/test/queries/clientpositive/explode_null.q (revision 1661246) +++ ql/src/test/queries/clientpositive/explode_null.q (working copy) @@ -1,10 +1,10 @@ SELECT explode(col) AS myCol FROM - (SELECT array(1,2,3) AS col FROM src LIMIT 1 + (select * from (SELECT array(1,2,3) AS col FROM src LIMIT 1)ea UNION ALL - SELECT IF(false, array(1,2,3), NULL) AS col FROM src LIMIT 1) a; + select * from (SELECT IF(false, array(1,2,3), NULL) AS col FROM src LIMIT 1)eb) a; SELECT explode(col) AS (myCol1,myCol2) FROM - (SELECT map(1,'one',2,'two',3,'three') AS col FROM src LIMIT 1 + (select * from (SELECT map(1,'one',2,'two',3,'three') AS col FROM src LIMIT 1)ea UNION ALL - SELECT IF(false, map(1,'one',2,'two',3,'three'), NULL) AS col FROM src LIMIT 1) a; + select * from (SELECT IF(false, map(1,'one',2,'two',3,'three'), NULL) AS col FROM src LIMIT 1)eb ) a; \ No newline at end of file Index: ql/src/test/queries/clientpositive/groupby3_map_skew.q =================================================================== --- ql/src/test/queries/clientpositive/groupby3_map_skew.q (revision 1661246) +++ ql/src/test/queries/clientpositive/groupby3_map_skew.q (working copy) @@ -29,5 +29,6 @@ variance(substr(src.value,5)), var_samp(substr(src.value,5)); -SELECT c1, c2, c3, c4, c5, c6, c7, ROUND(c8, 5), ROUND(c9, 5) FROM dest1; +SELECT ROUND(c1, 1), ROUND(c2, 3), ROUND(c3, 5), ROUND(c4, 1), ROUND(c5, 1), ROUND(c6, 5), +ROUND(c7,5), ROUND(c8, 5), ROUND(c9, 5) FROM dest1; Index: ql/src/test/queries/clientpositive/groupby7_noskew_multi_single_reducer.q =================================================================== --- ql/src/test/queries/clientpositive/groupby7_noskew_multi_single_reducer.q (revision 1661246) +++ ql/src/test/queries/clientpositive/groupby7_noskew_multi_single_reducer.q (working copy) @@ -12,8 +12,8 @@ EXPLAIN FROM SRC -INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 -INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10; +INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10; FROM SRC INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 Index: ql/src/test/queries/clientpositive/groupby_multi_single_reducer3.q =================================================================== --- ql/src/test/queries/clientpositive/groupby_multi_single_reducer3.q (revision 1661246) +++ ql/src/test/queries/clientpositive/groupby_multi_single_reducer3.q (working copy) @@ -1,4 +1,7 @@ -- HIVE-3849 Aliased column in where clause for multi-groupby single reducer cannot be resolved + +-- SORT_QUERY_RESULTS + create table e1 (key string, count int); create table e2 (key string, count int); Index: ql/src/test/queries/clientpositive/input25.q =================================================================== --- ql/src/test/queries/clientpositive/input25.q (revision 1661246) +++ ql/src/test/queries/clientpositive/input25.q (working copy) @@ -5,15 +5,15 @@ explain select * from ( - select * from tst x where x.d='2009-01-01' limit 10 + select * from (select * from tst x where x.d='2009-01-01' limit 10)a union all - select * from tst x where x.d='2009-02-02' limit 10 + select * from (select * from tst x where x.d='2009-02-02' limit 10)b ) subq; select * from ( - select * from tst x where x.d='2009-01-01' limit 10 + select * from (select * from tst x where x.d='2009-01-01' limit 10)a union all - select * from tst x where x.d='2009-02-02' limit 10 + select * from (select * from tst x where x.d='2009-02-02' limit 10)b ) subq; Index: ql/src/test/queries/clientpositive/input26.q =================================================================== --- ql/src/test/queries/clientpositive/input26.q (revision 1661246) +++ ql/src/test/queries/clientpositive/input26.q (working copy) @@ -1,12 +1,12 @@ explain select * from ( - select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5 + select * from (select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5)pa union all - select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5 + select * from (select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5)pb )subq; select * from ( - select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5 + select * from (select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5)pa union all - select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5 + select * from (select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5)pb )subq; Index: ql/src/test/queries/clientpositive/load_dyn_part14.q =================================================================== --- ql/src/test/queries/clientpositive/load_dyn_part14.q (revision 1661246) +++ ql/src/test/queries/clientpositive/load_dyn_part14.q (working copy) @@ -14,20 +14,20 @@ explain insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T; insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T; Index: ql/src/test/queries/clientpositive/metadataOnlyOptimizer.q =================================================================== --- ql/src/test/queries/clientpositive/metadataOnlyOptimizer.q (revision 1661246) +++ ql/src/test/queries/clientpositive/metadataOnlyOptimizer.q (working copy) @@ -33,9 +33,9 @@ SELECT key as key from src ) tab group by key; -select k from (SELECT '1' as k from src limit 0 union all select key as k from src limit 1)tab; +select k from (select * from (SELECT '1' as k from src limit 0)a union all select key as k from src limit 1)tab; -select k from (SELECT '1' as k from src limit 1 union all select key as k from src limit 0)tab; +select k from (select * from (SELECT '1' as k from src limit 1)a union all select key as k from src limit 0)tab; select max(ds) from srcpart; Index: ql/src/test/queries/clientpositive/orc_analyze.q =================================================================== --- ql/src/test/queries/clientpositive/orc_analyze.q (revision 1661246) +++ ql/src/test/queries/clientpositive/orc_analyze.q (working copy) @@ -30,8 +30,13 @@ INSERT OVERWRITE TABLE orc_create_people SELECT * FROM orc_create_people_staging ORDER BY id; set hive.stats.autogather = true; +analyze table orc_create_people compute statistics; +desc formatted orc_create_people; + analyze table orc_create_people compute statistics partialscan; +desc formatted orc_create_people; +analyze table orc_create_people compute statistics noscan; desc formatted orc_create_people; drop table orc_create_people; @@ -70,8 +75,15 @@ SELECT * FROM orc_create_people_staging ORDER BY id; set hive.stats.autogather = true; +analyze table orc_create_people partition(state) compute statistics; +desc formatted orc_create_people partition(state="Ca"); +desc formatted orc_create_people partition(state="Or"); + analyze table orc_create_people partition(state) compute statistics partialscan; +desc formatted orc_create_people partition(state="Ca"); +desc formatted orc_create_people partition(state="Or"); +analyze table orc_create_people partition(state) compute statistics noscan; desc formatted orc_create_people partition(state="Ca"); desc formatted orc_create_people partition(state="Or"); @@ -116,8 +128,15 @@ SELECT * FROM orc_create_people_staging ORDER BY id; set hive.stats.autogather = true; +analyze table orc_create_people partition(state) compute statistics; +desc formatted orc_create_people partition(state="Ca"); +desc formatted orc_create_people partition(state="Or"); + analyze table orc_create_people partition(state) compute statistics partialscan; +desc formatted orc_create_people partition(state="Ca"); +desc formatted orc_create_people partition(state="Or"); +analyze table orc_create_people partition(state) compute statistics noscan; desc formatted orc_create_people partition(state="Ca"); desc formatted orc_create_people partition(state="Or"); @@ -174,9 +193,16 @@ ALTER TABLE orc_create_people SET FILEFORMAT ORC; set hive.stats.autogather = true; -analyze table orc_create_people partition(state) compute statistics noscan; +analyze table orc_create_people partition(state) compute statistics; +desc formatted orc_create_people partition(state="Ca"); +desc formatted orc_create_people partition(state="OH"); +analyze table orc_create_people partition(state) compute statistics partialscan; desc formatted orc_create_people partition(state="Ca"); desc formatted orc_create_people partition(state="OH"); +analyze table orc_create_people partition(state) compute statistics noscan; +desc formatted orc_create_people partition(state="Ca"); +desc formatted orc_create_people partition(state="OH"); + drop table orc_create_people; Index: ql/src/test/queries/clientpositive/parallel_join0.q =================================================================== --- ql/src/test/queries/clientpositive/parallel_join0.q (revision 1661246) +++ ql/src/test/queries/clientpositive/parallel_join0.q (working copy) @@ -1,6 +1,6 @@ set mapreduce.job.reduces=4; --- SORT_BEFORE_DIFF +-- SORT_QUERY_RESULTS EXPLAIN SELECT src1.key as k1, src1.value as v1, Index: ql/src/test/queries/clientpositive/parquet_types.q =================================================================== --- ql/src/test/queries/clientpositive/parquet_types.q (revision 1661246) +++ ql/src/test/queries/clientpositive/parquet_types.q (working copy) @@ -55,8 +55,8 @@ MAX(cint), MIN(csmallint), COUNT(cstring1), - AVG(cfloat), - STDDEV_POP(cdouble) + ROUND(AVG(cfloat), 5), + ROUND(STDDEV_POP(cdouble),5) FROM parquet_types GROUP BY ctinyint ORDER BY ctinyint Index: ql/src/test/queries/clientpositive/ptf.q =================================================================== --- ql/src/test/queries/clientpositive/ptf.q (revision 1661246) +++ ql/src/test/queries/clientpositive/ptf.q (working copy) @@ -202,7 +202,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))); select p_mfgr, p_name, p_size, @@ -211,7 +211,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))); -- 13. testPTFAndWindowingInSubQ Index: ql/src/test/queries/clientpositive/ptf_matchpath.q =================================================================== --- ql/src/test/queries/clientpositive/ptf_matchpath.q (revision 1661246) +++ ql/src/test/queries/clientpositive/ptf_matchpath.q (working copy) @@ -15,6 +15,7 @@ -- SORT_QUERY_RESULTS -- 1. basic Matchpath test +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -25,7 +26,18 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ); +select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + distribute by fl_num + sort by year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ); + -- 2. Matchpath on 1 partition +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -36,7 +48,18 @@ ) where fl_num = 1142; +select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +where fl_num = 1142; + -- 3. empty partition. +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on (select * from flights_tiny where fl_num = -1142) flights_tiny @@ -46,3 +69,13 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ); + +select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + (select * from flights_tiny where fl_num = -1142) flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ); + Index: ql/src/test/queries/clientpositive/script_env_var1.q =================================================================== --- ql/src/test/queries/clientpositive/script_env_var1.q (revision 1661246) +++ ql/src/test/queries/clientpositive/script_env_var1.q (working copy) @@ -1,5 +1,5 @@ -- Verifies that script operator ID environment variables have unique values -- in each instance of the script operator. SELECT count(1) FROM -( SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key; +( SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)x UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)y ) a GROUP BY key; Index: ql/src/test/queries/clientpositive/script_env_var2.q =================================================================== --- ql/src/test/queries/clientpositive/script_env_var2.q (revision 1661246) +++ ql/src/test/queries/clientpositive/script_env_var2.q (working copy) @@ -1,5 +1,5 @@ set hive.script.operator.id.env.var = MY_ID; -- Same test as script_env_var1, but test setting the variable name SELECT count(1) FROM -( SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key; +( SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)a UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)b ) a GROUP BY key; Index: ql/src/test/queries/clientpositive/udaf_covar_pop.q =================================================================== --- ql/src/test/queries/clientpositive/udaf_covar_pop.q (revision 1661246) +++ ql/src/test/queries/clientpositive/udaf_covar_pop.q (working copy) @@ -11,6 +11,6 @@ SELECT covar_pop(b, c) FROM covar_tab WHERE a < 3; SELECT covar_pop(b, c) FROM covar_tab WHERE a = 3; SELECT a, covar_pop(b, c) FROM covar_tab GROUP BY a ORDER BY a; -SELECT covar_pop(b, c) FROM covar_tab; +SELECT ROUND(covar_pop(b, c), 5) FROM covar_tab; DROP TABLE covar_tab; Index: ql/src/test/queries/clientpositive/udaf_covar_samp.q =================================================================== --- ql/src/test/queries/clientpositive/udaf_covar_samp.q (revision 1661246) +++ ql/src/test/queries/clientpositive/udaf_covar_samp.q (working copy) @@ -11,6 +11,6 @@ SELECT covar_samp(b, c) FROM covar_tab WHERE a < 3; SELECT covar_samp(b, c) FROM covar_tab WHERE a = 3; SELECT a, covar_samp(b, c) FROM covar_tab GROUP BY a ORDER BY a; -SELECT covar_samp(b, c) FROM covar_tab; +SELECT ROUND(covar_samp(b, c), 5) FROM covar_tab; DROP TABLE covar_tab; Index: ql/src/test/queries/clientpositive/udf_add_months.q =================================================================== --- ql/src/test/queries/clientpositive/udf_add_months.q (revision 1661246) +++ ql/src/test/queries/clientpositive/udf_add_months.q (working copy) @@ -1,2 +1,43 @@ DESCRIBE FUNCTION add_months; DESCRIBE FUNCTION EXTENDED add_months; + +explain select add_months('2014-01-14', 1); + +select +add_months('2014-01-14', 1), +add_months('2014-01-31', 1), +add_months('2014-02-28', -1), +add_months('2014-02-28', 2), +add_months('2014-04-30', -2), +add_months('2015-02-28', 12), +add_months('2016-02-29', -12), +add_months('2016-01-29', 1), +add_months('2016-02-29', -1), +add_months('2014-01-32', 1), +add_months('01/14/2014', 1), +add_months(cast(null as string), 1), +add_months('2014-01-14', cast(null as int)); + +select +add_months('2014-01-14 10:30:00', 1), +add_months('2014-01-31 10:30:00', 1), +add_months('2014-02-28 10:30:00', -1), +add_months('2014-02-28 16:30:00', 2), +add_months('2014-04-30 10:30:00', -2), +add_months('2015-02-28 10:30:00', 12), +add_months('2016-02-29 10:30:00', -12), +add_months('2016-01-29 10:30:00', 1), +add_months('2016-02-29 10:30:00', -1), +add_months('2014-01-32 10:30:00', 1); + +select +add_months(cast('2014-01-14 10:30:00' as timestamp), 1), +add_months(cast('2014-01-31 10:30:00' as timestamp), 1), +add_months(cast('2014-02-28 10:30:00' as timestamp), -1), +add_months(cast('2014-02-28 16:30:00' as timestamp), 2), +add_months(cast('2014-04-30 10:30:00' as timestamp), -2), +add_months(cast('2015-02-28 10:30:00' as timestamp), 12), +add_months(cast('2016-02-29 10:30:00' as timestamp), -12), +add_months(cast('2016-01-29 10:30:00' as timestamp), 1), +add_months(cast('2016-02-29 10:30:00' as timestamp), -1), +add_months(cast(null as timestamp), 1); \ No newline at end of file Index: ql/src/test/queries/clientpositive/udf_last_day.q =================================================================== --- ql/src/test/queries/clientpositive/udf_last_day.q (revision 1661246) +++ ql/src/test/queries/clientpositive/udf_last_day.q (working copy) @@ -1,2 +1,52 @@ DESCRIBE FUNCTION last_day; DESCRIBE FUNCTION EXTENDED last_day; + +explain select last_day('2015-02-05'); + +select +last_day('2014-01-01'), +last_day('2014-01-14'), +last_day('2014-01-31'), +last_day('2014-02-02'), +last_day('2014-02-28'), +last_day('2016-02-03'), +last_day('2016-02-28'), +last_day('2016-02-29'), +last_day('2014-01-34'), +last_day(cast(null as string)), +last_day('01/29/2014'); + +select +last_day('2014-01-01 10:30:45'), +last_day('2014-01-14 10:30:45'), +last_day('2014-01-31 10:30:45'), +last_day('2014-02-02 10:30:45'), +last_day('2014-02-28 10:30:45'), +last_day('2016-02-03 10:30:45'), +last_day('2016-02-28 10:30:45'), +last_day('2016-02-29 10:30:45'), +last_day('2014-01-34 10:30:45'), +last_day(cast(null as string)), +last_day('01/29/2014 10:30:45'); + +select +last_day(cast('2014-01-01' as date)), +last_day(cast('2014-01-14' as date)), +last_day(cast('2014-01-31' as date)), +last_day(cast('2014-02-02' as date)), +last_day(cast('2014-02-28' as date)), +last_day(cast('2016-02-03' as date)), +last_day(cast('2016-02-28' as date)), +last_day(cast('2016-02-29' as date)), +last_day(cast(null as date)); + +select +last_day(cast('2014-01-01 10:30:45' as timestamp)), +last_day(cast('2014-01-14 10:30:45' as timestamp)), +last_day(cast('2014-01-31 10:30:45' as timestamp)), +last_day(cast('2014-02-02 10:30:45' as timestamp)), +last_day(cast('2014-02-28 10:30:45' as timestamp)), +last_day(cast('2016-02-03 10:30:45' as timestamp)), +last_day(cast('2016-02-28 10:30:45' as timestamp)), +last_day(cast('2016-02-29 10:30:45' as timestamp)), +last_day(cast(null as timestamp)); \ No newline at end of file Index: ql/src/test/queries/clientpositive/union3.q =================================================================== --- ql/src/test/queries/clientpositive/union3.q (revision 1661246) +++ ql/src/test/queries/clientpositive/union3.q (working copy) @@ -1,21 +1,20 @@ --- SORT_BEFORE_DIFF +-- SORT_QUERY_RESULTS explain SELECT * FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a; @@ -27,17 +26,16 @@ FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a; -select * from union_out cluster by id; +select * from union_out; Index: ql/src/test/queries/clientpositive/union4.q =================================================================== --- ql/src/test/queries/clientpositive/union4.q (revision 1661246) +++ ql/src/test/queries/clientpositive/union4.q (working copy) @@ -1,5 +1,7 @@ set hive.map.aggr = true; +-- SORT_QUERY_RESULTS + -- union case: both subqueries are map-reduce jobs on same input, followed by filesink Index: ql/src/test/queries/clientpositive/union_null.q =================================================================== --- ql/src/test/queries/clientpositive/union_null.q (revision 1661246) +++ ql/src/test/queries/clientpositive/union_null.q (working copy) @@ -1,7 +1,7 @@ -- SORT_BEFORE_DIFF -- HIVE-2901 -select x from (select value as x from src order by x limit 5 union all select NULL as x from src limit 5) a; +select x from (select * from (select value as x from src order by x limit 5)a union all select * from (select NULL as x from src limit 5)b )a; -- HIVE-4837 -select * from (select null as N from src1 group by key UNION ALL select null as N from src1 group by key ) a; +select * from (select * from (select null as N from src1 group by key)a UNION ALL select * from (select null as N from src1 group by key)b ) a; Index: ql/src/test/queries/clientpositive/union_remove_25.q =================================================================== --- ql/src/test/queries/clientpositive/union_remove_25.q (revision 1661246) +++ ql/src/test/queries/clientpositive/union_remove_25.q (working copy) @@ -51,17 +51,17 @@ insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a; insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a; show partitions outputTbl2; @@ -70,17 +70,17 @@ explain insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a; insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a; show partitions outputTbl3; Index: ql/src/test/queries/clientpositive/union_top_level.q =================================================================== --- ql/src/test/queries/clientpositive/union_top_level.q (revision 1661246) +++ ql/src/test/queries/clientpositive/union_top_level.q (working copy) @@ -2,42 +2,42 @@ -- top level explain -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; explain -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)a union all -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10; +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)b; -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)a union all -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10; +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)b; -- ctas explain create table union_top as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; create table union_top as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; select * from union_top; @@ -46,53 +46,53 @@ -- insert into explain insert into table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; insert into table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; select * from union_top; explain insert overwrite table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; insert overwrite table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; select * from union_top; -- create view explain create view union_top_view as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; create view union_top_view as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3; +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c; select * from union_top_view; Index: ql/src/test/queries/clientpositive/vector_multi_insert.q =================================================================== --- ql/src/test/queries/clientpositive/vector_multi_insert.q (revision 1661246) +++ ql/src/test/queries/clientpositive/vector_multi_insert.q (working copy) @@ -8,11 +8,11 @@ select rn from ( - select cast(1 as int) as rn from src limit 1 + select * from (select cast(1 as int) as rn from src limit 1)a union all - select cast(100 as int) as rn from src limit 1 + select * from (select cast(100 as int) as rn from src limit 1)b union all - select cast(10000 as int) as rn from src limit 1 + select * from (select cast(10000 as int) as rn from src limit 1)c ) t; create table orc_rn1 (rn int); Index: ql/src/test/queries/clientpositive/windowing_windowspec.q =================================================================== --- ql/src/test/queries/clientpositive/windowing_windowspec.q (revision 1661246) +++ ql/src/test/queries/clientpositive/windowing_windowspec.q (working copy) @@ -34,3 +34,7 @@ select s, i, round(avg(d) over (partition by s order by i) / 10.0 , 2) from over10k limit 7; select s, i, round((avg(d) over w1 + 10.0) - (avg(d) over w1 - 10.0),2) from over10k window w1 as (partition by s order by i) limit 7; + +set hive.cbo.enable=false; +-- HIVE-9228 +select s, i from ( select s, i, round((avg(d) over w1 + 10.0) - (avg(d) over w1 - 10.0),2) from over10k window w1 as (partition by s order by i)) X limit 7; Index: ql/src/test/resources/orc-file-dump-dictionary-threshold.out =================================================================== --- ql/src/test/resources/orc-file-dump-dictionary-threshold.out (revision 1661246) +++ ql/src/test/resources/orc-file-dump-dictionary-threshold.out (working copy) @@ -52,24 +52,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DIRECT_V2 - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2132329551 max: 2145911404 sum: 61941331718 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2138433136 max: 2145210552 sum: 14574030042 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2147115959 max: 2137805337 sum: -2032493169 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2137828953 max: 2145877119 sum: -3167202608 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2146452517 max: 2142394906 sum: 88361503212 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9206837518492372266 max: 9169230975203934579 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9188878639954124284 max: 9213664245516510068 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9211329013123260308 max: 9217851628057711416 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9185745718227889962 max: 9181722705210917931 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9216505819108477308 max: 9196474183833079923 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness,-230 max: worst-54-290-346-648-908-996 sum: 18442 positions: 0,0,0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966 sum: 46338 positions: 2777,8442,0,695,18 - Entry 2:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660 sum: 75448 positions: 13595,4780,0,1554,14 - Entry 3:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788 sum: 104868 positions: 31432,228,0,2372,90 - Entry 4:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744 sum: 136158 positions: 54111,5096,0,3354,108 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2132329551 max: 2145911404 sum: 61941331718 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2138433136 max: 2145210552 sum: 14574030042 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2147115959 max: 2137805337 sum: -2032493169 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2137828953 max: 2145877119 sum: -3167202608 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2146452517 max: 2142394906 sum: 88361503212 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9206837518492372266 max: 9169230975203934579 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9188878639954124284 max: 9213664245516510068 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9211329013123260308 max: 9217851628057711416 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9185745718227889962 max: 9181722705210917931 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9216505819108477308 max: 9196474183833079923 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness,-230 max: worst-54-290-346-648-908-996 sum: 18442 positions: 0,0,0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966 sum: 46338 positions: 2777,8442,0,695,18 + Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660 sum: 75448 positions: 13595,4780,0,1554,14 + Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788 sum: 104868 positions: 31432,228,0,2372,90 + Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744 sum: 136158 positions: 54111,5096,0,3354,108 Stripe: offset: 151897 data: 336358 rows: 5000 tail: 69 index: 954 Stream: column 0 section ROW_INDEX start: 151897 length 17 Stream: column 1 section ROW_INDEX start: 151914 length 153 @@ -83,24 +83,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DIRECT_V2 - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2146021688 max: 2146838901 sum: -50979197646 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2143569489 max: 2141223179 sum: 22810066834 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2140649392 max: 2146301701 sum: -31694882346 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2147390285 max: 2146299933 sum: 79371934221 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2145928262 max: 2147224606 sum: -34469378822 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9222178666167296739 max: 9191250610515369723 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9220148577547102875 max: 9213945522531717278 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9220818777591257749 max: 9221301751385928177 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9220031433030423388 max: 9207856144487414148 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9201438531577205959 max: 9212462124593119846 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726 sum: 166320 positions: 0,0,0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994 sum: 193436 positions: 37112,6320,0,967,90 - Entry 2:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988 sum: 224740 positions: 80822,9756,0,1945,222 - Entry 3:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984 sum: 252094 positions: 137149,4496,0,3268,48 - Entry 4:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938 sum: 281404 positions: 197972,6590,0,4064,342 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2146021688 max: 2146838901 sum: -50979197646 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2143569489 max: 2141223179 sum: 22810066834 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2140649392 max: 2146301701 sum: -31694882346 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2147390285 max: 2146299933 sum: 79371934221 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2145928262 max: 2147224606 sum: -34469378822 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9222178666167296739 max: 9191250610515369723 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9220148577547102875 max: 9213945522531717278 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9220818777591257749 max: 9221301751385928177 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9220031433030423388 max: 9207856144487414148 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9201438531577205959 max: 9212462124593119846 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726 sum: 166320 positions: 0,0,0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994 sum: 193436 positions: 37112,6320,0,967,90 + Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988 sum: 224740 positions: 80822,9756,0,1945,222 + Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984 sum: 252094 positions: 137149,4496,0,3268,48 + Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938 sum: 281404 positions: 197972,6590,0,4064,342 Stripe: offset: 489278 data: 558031 rows: 5000 tail: 69 index: 1181 Stream: column 0 section ROW_INDEX start: 489278 length 17 Stream: column 1 section ROW_INDEX start: 489295 length 166 @@ -114,24 +114,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DIRECT_V2 - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2138229212 max: 2144818981 sum: -22823642812 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2145842720 max: 2144179881 sum: -12562754334 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2143045885 max: 2146718321 sum: 82993638644 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2144745617 max: 2146570474 sum: 25138722367 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2140127150 max: 2135081620 sum: 68346511655 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9204340807292138409 max: 9208698732685326961 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9221963099397084326 max: 9222722740629726770 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9210480084701091299 max: 9207767402467343058 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9195038026813631215 max: 9199201928563274421 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9215483580266514322 max: 9220102792864959501 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876 sum: 313880 positions: 0,0,0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964 sum: 349542 positions: 80352,3880,0,1097,28 - Entry 2:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976 sum: 386538 positions: 170641,3422,0,2077,162 - Entry 3:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766 sum: 421660 positions: 268420,9960,0,3369,16 - Entry 4:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974 sum: 453606 positions: 377916,1620,0,4041,470 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2138229212 max: 2144818981 sum: -22823642812 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2145842720 max: 2144179881 sum: -12562754334 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2143045885 max: 2146718321 sum: 82993638644 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2144745617 max: 2146570474 sum: 25138722367 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2140127150 max: 2135081620 sum: 68346511655 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9204340807292138409 max: 9208698732685326961 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9221963099397084326 max: 9222722740629726770 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9210480084701091299 max: 9207767402467343058 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9195038026813631215 max: 9199201928563274421 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9215483580266514322 max: 9220102792864959501 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876 sum: 313880 positions: 0,0,0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964 sum: 349542 positions: 80352,3880,0,1097,28 + Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976 sum: 386538 positions: 170641,3422,0,2077,162 + Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766 sum: 421660 positions: 268420,9960,0,3369,16 + Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974 sum: 453606 positions: 377916,1620,0,4041,470 Stripe: offset: 1048559 data: 792850 rows: 5000 tail: 69 index: 1369 Stream: column 0 section ROW_INDEX start: 1048559 length 17 Stream: column 1 section ROW_INDEX start: 1048576 length 153 @@ -145,24 +145,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DIRECT_V2 - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2145319330 max: 2146998132 sum: -50856753363 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2134288866 max: 2147453086 sum: -17911019023 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2139010804 max: 2144727593 sum: -24993151857 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2145378214 max: 2144098933 sum: -18055164052 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2140494429 max: 2144595861 sum: -41863916235 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9172774601303513941 max: 9212917101275642143 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9218164880949195469 max: 9222919052987871506 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9196276654247395117 max: 9210639275226058005 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9197393848859294562 max: 9208134757538374043 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610 sum: 492916 positions: 0,0,0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936 sum: 527290 positions: 126968,2916,0,1077,140 - Entry 2:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878 sum: 568274 positions: 263111,206,0,1926,462 - Entry 3:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788 sum: 594578 positions: 407371,8480,0,3444,250 - Entry 4:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214-18444-18446-18724-18912-18952-19164 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904 sum: 631944 positions: 562094,3058,0,4643,292 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2145319330 max: 2146998132 sum: -50856753363 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2134288866 max: 2147453086 sum: -17911019023 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2139010804 max: 2144727593 sum: -24993151857 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2145378214 max: 2144098933 sum: -18055164052 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2140494429 max: 2144595861 sum: -41863916235 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9172774601303513941 max: 9212917101275642143 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9218164880949195469 max: 9222919052987871506 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9196276654247395117 max: 9210639275226058005 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9197393848859294562 max: 9208134757538374043 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610 sum: 492916 positions: 0,0,0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936 sum: 527290 positions: 126968,2916,0,1077,140 + Entry 2: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878 sum: 568274 positions: 263111,206,0,1926,462 + Entry 3: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788 sum: 594578 positions: 407371,8480,0,3444,250 + Entry 4: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214-18444-18446-18724-18912-18952-19164 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904 sum: 631944 positions: 562094,3058,0,4643,292 Stripe: offset: 1842847 data: 188033 rows: 1000 tail: 67 index: 841 Stream: column 0 section ROW_INDEX start: 1842847 length 12 Stream: column 1 section ROW_INDEX start: 1842859 length 38 @@ -176,12 +176,12 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DIRECT_V2 - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2143595397 max: 2136858458 sum: -22999664100 positions: 0,0,0 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9212379634781416464 max: 9197412874152820822 positions: 0,0,0 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214-18444-18446-18724-18912-18952-19164-19348-19400-19546-19776-19896-20084 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904-20390-20752-20936 sum: 670762 positions: 0,0,0,0,0 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2143595397 max: 2136858458 sum: -22999664100 positions: 0,0,0 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9212379634781416464 max: 9197412874152820822 positions: 0,0,0 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness,-230-368-488-586-862-930-1686-2044-2636-2652-2872-3108-3162-3192-3404-3442-3508-3542-3550-3712-3980-4146-4204-4336-4390-4418-4424-4490-4512-4650-4768-4924-4950-5210-5524-5630-5678-5710-5758-5952-6238-6252-6300-6366-6668-6712-6926-6942-7100-7194-7802-8030-8452-8608-8640-8862-8868-9134-9234-9412-9602-9608-9642-9678-9740-9780-10426-10510-10514-10706-10814-10870-10942-11028-11244-11326-11462-11496-11656-11830-12022-12178-12418-12832-13304-13448-13590-13618-13908-14188-14246-14340-14364-14394-14762-14850-14964-15048-15494-15674-15726-16006-16056-16180-16304-16332-16452-16598-16730-16810-16994-17210-17268-17786-17962-18214-18444-18446-18724-18912-18952-19164-19348-19400-19546-19776-19896-20084 max: worst-54-290-346-648-908-996-1038-1080-1560-1584-1620-1744-1770-1798-1852-1966-2162-2244-2286-2296-2534-2660-3114-3676-3788-4068-4150-4706-4744-5350-5420-5582-5696-5726-6006-6020-6024-6098-6184-6568-6636-6802-6994-7004-7318-7498-7758-7780-7798-7920-7952-7960-7988-8232-8256-8390-8416-8478-8620-8840-8984-9038-9128-9236-9248-9344-9594-9650-9714-9928-9938-10178-10368-10414-10502-10732-10876-11008-11158-11410-11722-11836-11964-12054-12096-12126-12136-12202-12246-12298-12616-12774-12782-12790-12802-12976-13216-13246-13502-13766-14454-14974-15004-15124-15252-15294-15356-15530-15610-16316-16936-17024-17122-17214-17310-17528-17682-17742-17870-17878-18010-18410-18524-18788-19204-19254-19518-19596-19786-19874-19904-20390-20752-20936 sum: 670762 positions: 0,0,0,0,0 File length: 2033643 bytes Padding length: 0 bytes Index: ql/src/test/resources/orc-file-dump.out =================================================================== --- ql/src/test/resources/orc-file-dump.out (revision 1661246) +++ ql/src/test/resources/orc-file-dump.out (working copy) @@ -53,24 +53,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DICTIONARY_V2[35] - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2145365268 max: 2135491313 sum: 7521792925 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2139452528 max: 2147223299 sum: -12923774313 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2142420586 max: 2143898386 sum: -25521983511 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2137233441 max: 2144267163 sum: 40993386199 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2146021688 max: 2146838901 sum: -9553628474 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9200577545527640566 max: 9175500305011173751 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9203618157670445774 max: 9208123824411178101 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9218592812243954469 max: 9221351515892923972 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9206585617947511272 max: 9167703224425685487 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9206645795733282496 max: 9221614132680747961 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness, max: worst sum: 3862 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness, max: worst sum: 3884 positions: 0,659,149 - Entry 2:count: 1000 hasNull: false min: Darkness, max: worst sum: 3893 positions: 0,1531,3 - Entry 3:count: 1000 hasNull: false min: Darkness, max: worst sum: 3798 positions: 0,2281,32 - Entry 4:count: 1000 hasNull: false min: Darkness, max: worst sum: 3843 positions: 0,3033,45 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2145365268 max: 2135491313 sum: 7521792925 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2139452528 max: 2147223299 sum: -12923774313 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2142420586 max: 2143898386 sum: -25521983511 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2137233441 max: 2144267163 sum: 40993386199 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2146021688 max: 2146838901 sum: -9553628474 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9200577545527640566 max: 9175500305011173751 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9203618157670445774 max: 9208123824411178101 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9218592812243954469 max: 9221351515892923972 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9206585617947511272 max: 9167703224425685487 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9206645795733282496 max: 9221614132680747961 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3862 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 3884 positions: 0,659,149 + Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3893 positions: 0,1531,3 + Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3798 positions: 0,2281,32 + Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 3843 positions: 0,3033,45 Stripe: offset: 64288 data: 63754 rows: 5000 tail: 79 index: 433 Stream: column 0 section ROW_INDEX start: 64288 length 17 Stream: column 1 section ROW_INDEX start: 64305 length 162 @@ -85,24 +85,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DICTIONARY_V2[35] - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2143799121 max: 2145249879 sum: -6966266181 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2146733128 max: 2147001622 sum: -35930106333 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2144302712 max: 2146299933 sum: 6944230435 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2145172948 max: 2144335014 sum: -29624404959 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2146428427 max: 2144067253 sum: 65584220465 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9218450653857701562 max: 9189819526332228512 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9220818777591257749 max: 9178821722829648113 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9220031433030423388 max: 9210838931786956852 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9208195729739635607 max: 9222259462014003839 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9174271499932339698 max: 9212277876771676916 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness, max: worst sum: 3923 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness, max: worst sum: 3869 positions: 0,761,12 - Entry 2:count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,1472,70 - Entry 3:count: 1000 hasNull: false min: Darkness, max: worst sum: 3931 positions: 0,2250,43 - Entry 4:count: 1000 hasNull: false min: Darkness, max: worst sum: 3964 positions: 0,2978,88 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2143799121 max: 2145249879 sum: -6966266181 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2146733128 max: 2147001622 sum: -35930106333 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2144302712 max: 2146299933 sum: 6944230435 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2145172948 max: 2144335014 sum: -29624404959 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2146428427 max: 2144067253 sum: 65584220465 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9218450653857701562 max: 9189819526332228512 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9220818777591257749 max: 9178821722829648113 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9220031433030423388 max: 9210838931786956852 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9208195729739635607 max: 9222259462014003839 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9174271499932339698 max: 9212277876771676916 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3923 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 3869 positions: 0,761,12 + Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,1472,70 + Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3931 positions: 0,2250,43 + Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 3964 positions: 0,2978,88 Stripe: offset: 128554 data: 63766 rows: 5000 tail: 79 index: 437 Stream: column 0 section ROW_INDEX start: 128554 length 17 Stream: column 1 section ROW_INDEX start: 128571 length 159 @@ -117,24 +117,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DICTIONARY_V2[35] - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2146993718 max: 2144179881 sum: -7829543271 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2144095505 max: 2144883384 sum: 51623839692 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2144113995 max: 2143773575 sum: 56574412741 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2146954065 max: 2146794873 sum: 4336083432 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2135511523 max: 2147378179 sum: 27955949957 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9211978436552246208 max: 9179058898902097152 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9195645160817780503 max: 9189147759444307708 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9202888157616520823 max: 9193561362676960747 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9216318198067839390 max: 9221286760675829363 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9218342074710552826 max: 9222303228623055266 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness, max: worst sum: 4008 positions: 0,634,174 - Entry 2:count: 1000 hasNull: false min: Darkness, max: worst sum: 3999 positions: 0,1469,69 - Entry 3:count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,2133,194 - Entry 4:count: 1000 hasNull: false min: Darkness, max: worst sum: 4000 positions: 0,3005,43 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2146993718 max: 2144179881 sum: -7829543271 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2144095505 max: 2144883384 sum: 51623839692 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2144113995 max: 2143773575 sum: 56574412741 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2146954065 max: 2146794873 sum: 4336083432 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2135511523 max: 2147378179 sum: 27955949957 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9211978436552246208 max: 9179058898902097152 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9195645160817780503 max: 9189147759444307708 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9202888157616520823 max: 9193561362676960747 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9216318198067839390 max: 9221286760675829363 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9218342074710552826 max: 9222303228623055266 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 4008 positions: 0,634,174 + Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3999 positions: 0,1469,69 + Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3817 positions: 0,2133,194 + Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 4000 positions: 0,3005,43 Stripe: offset: 192836 data: 63796 rows: 5000 tail: 79 index: 440 Stream: column 0 section ROW_INDEX start: 192836 length 17 Stream: column 1 section ROW_INDEX start: 192853 length 162 @@ -149,24 +149,24 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DICTIONARY_V2[35] - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2141355639 max: 2145520931 sum: 2726719912 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -2138324170 max: 2140167376 sum: -23606674002 positions: 0,2050,488 - Entry 2:count: 1000 hasNull: false min: -2146658006 max: 2144329742 sum: -41530109703 positions: 0,6150,464 - Entry 3:count: 1000 hasNull: false min: -2144207593 max: 2139456355 sum: 13559842458 positions: 10003,250,440 - Entry 4:count: 1000 hasNull: false min: -2145744719 max: 2145417153 sum: 57383770571 positions: 10003,4350,416 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: -9222758097219661129 max: 9221043130193737406 positions: 0,4098,488 - Entry 2:count: 1000 hasNull: false min: -9174483776261243438 max: 9208134757538374043 positions: 10003,2294,464 - Entry 3:count: 1000 hasNull: false min: -9174329712613510612 max: 9197412874152820822 positions: 20006,490,440 - Entry 4:count: 1000 hasNull: false min: -9221162005892422758 max: 9220625004936875965 positions: 20006,8686,416 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness, max: worst sum: 3901 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: Darkness, max: worst sum: 3900 positions: 0,431,431 - Entry 2:count: 1000 hasNull: false min: Darkness, max: worst sum: 3909 positions: 0,1485,52 - Entry 3:count: 1000 hasNull: false min: Darkness, max: worst sum: 3947 positions: 0,2196,104 - Entry 4:count: 1000 hasNull: false min: Darkness, max: worst sum: 3813 positions: 0,2934,131 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2141355639 max: 2145520931 sum: 2726719912 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -2138324170 max: 2140167376 sum: -23606674002 positions: 0,2050,488 + Entry 2: count: 1000 hasNull: false min: -2146658006 max: 2144329742 sum: -41530109703 positions: 0,6150,464 + Entry 3: count: 1000 hasNull: false min: -2144207593 max: 2139456355 sum: 13559842458 positions: 10003,250,440 + Entry 4: count: 1000 hasNull: false min: -2145744719 max: 2145417153 sum: 57383770571 positions: 10003,4350,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9222731174895935707 max: 9214167447015056056 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: -9222758097219661129 max: 9221043130193737406 positions: 0,4098,488 + Entry 2: count: 1000 hasNull: false min: -9174483776261243438 max: 9208134757538374043 positions: 10003,2294,464 + Entry 3: count: 1000 hasNull: false min: -9174329712613510612 max: 9197412874152820822 positions: 20006,490,440 + Entry 4: count: 1000 hasNull: false min: -9221162005892422758 max: 9220625004936875965 positions: 20006,8686,416 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3901 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: Darkness, max: worst sum: 3900 positions: 0,431,431 + Entry 2: count: 1000 hasNull: false min: Darkness, max: worst sum: 3909 positions: 0,1485,52 + Entry 3: count: 1000 hasNull: false min: Darkness, max: worst sum: 3947 positions: 0,2196,104 + Entry 4: count: 1000 hasNull: false min: Darkness, max: worst sum: 3813 positions: 0,2934,131 Stripe: offset: 257151 data: 12940 rows: 1000 tail: 71 index: 131 Stream: column 0 section ROW_INDEX start: 257151 length 12 Stream: column 1 section ROW_INDEX start: 257163 length 38 @@ -181,12 +181,12 @@ Encoding column 1: DIRECT_V2 Encoding column 2: DIRECT_V2 Encoding column 3: DICTIONARY_V2[35] - Row group index column 1: - Entry 0:count: 1000 hasNull: false min: -2146245500 max: 2146378640 sum: 51299706363 positions: 0,0,0 - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: -9208193203370316142 max: 9218567213558056476 positions: 0,0,0 - Row group index column 3: - Entry 0:count: 1000 hasNull: false min: Darkness, max: worst sum: 3866 positions: 0,0,0 + Row group indices for column 1: + Entry 0: count: 1000 hasNull: false min: -2146245500 max: 2146378640 sum: 51299706363 positions: 0,0,0 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: -9208193203370316142 max: 9218567213558056476 positions: 0,0,0 + Row group indices for column 3: + Entry 0: count: 1000 hasNull: false min: Darkness, max: worst sum: 3866 positions: 0,0,0 File length: 270838 bytes Padding length: 0 bytes Index: ql/src/test/resources/orc-file-has-null.out =================================================================== --- ql/src/test/resources/orc-file-has-null.out (revision 1661246) +++ ql/src/test/resources/orc-file-has-null.out (working copy) @@ -42,12 +42,12 @@ Encoding column 0: DIRECT Encoding column 1: DIRECT_V2 Encoding column 2: DICTIONARY_V2[2] - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: RG1 max: RG1 sum: 3000 positions: 0,0,0,0,0,0,0 - Entry 1:count: 0 hasNull: true positions: 0,0,125,0,0,66,488 - Entry 2:count: 1000 hasNull: false min: RG3 max: RG3 sum: 3000 positions: 0,2,125,0,0,66,488 - Entry 3:count: 0 hasNull: true positions: 0,4,125,0,0,136,488 - Entry 4:count: 0 hasNull: true positions: 0,6,125,0,0,136,488 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: RG1 max: RG1 sum: 3000 positions: 0,0,0,0,0,0,0 + Entry 1: count: 0 hasNull: true positions: 0,0,125,0,0,66,488 + Entry 2: count: 1000 hasNull: false min: RG3 max: RG3 sum: 3000 positions: 0,2,125,0,0,66,488 + Entry 3: count: 0 hasNull: true positions: 0,4,125,0,0,136,488 + Entry 4: count: 0 hasNull: true positions: 0,6,125,0,0,136,488 Stripe: offset: 424 data: 156 rows: 5000 tail: 55 index: 119 Stream: column 0 section ROW_INDEX start: 424 length 17 Stream: column 1 section ROW_INDEX start: 441 length 63 @@ -59,12 +59,12 @@ Encoding column 0: DIRECT Encoding column 1: DIRECT_V2 Encoding column 2: DICTIONARY_V2[0] - Row group index column 2: - Entry 0:count: 0 hasNull: true positions: 0,0,0,0,0,0,0 - Entry 1:count: 0 hasNull: true positions: 0,0,125,0,0,0,0 - Entry 2:count: 0 hasNull: true positions: 0,2,120,0,0,0,0 - Entry 3:count: 0 hasNull: true positions: 0,4,115,0,0,0,0 - Entry 4:count: 0 hasNull: true positions: 0,6,110,0,0,0,0 + Row group indices for column 2: + Entry 0: count: 0 hasNull: true positions: 0,0,0,0,0,0,0 + Entry 1: count: 0 hasNull: true positions: 0,0,125,0,0,0,0 + Entry 2: count: 0 hasNull: true positions: 0,2,120,0,0,0,0 + Entry 3: count: 0 hasNull: true positions: 0,4,115,0,0,0,0 + Entry 4: count: 0 hasNull: true positions: 0,6,110,0,0,0,0 Stripe: offset: 754 data: 186 rows: 5000 tail: 60 index: 148 Stream: column 0 section ROW_INDEX start: 754 length 17 Stream: column 1 section ROW_INDEX start: 771 length 63 @@ -77,12 +77,12 @@ Encoding column 0: DIRECT Encoding column 1: DIRECT_V2 Encoding column 2: DICTIONARY_V2[1] - Row group index column 2: - Entry 0:count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,0,0 - Entry 1:count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,66,488 - Entry 2:count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,198,464 - Entry 3:count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,330,440 - Entry 4:count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,462,416 + Row group indices for column 2: + Entry 0: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,0,0 + Entry 1: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,66,488 + Entry 2: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,198,464 + Entry 3: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,330,440 + Entry 4: count: 1000 hasNull: false min: STRIPE-3 max: STRIPE-3 sum: 8000 positions: 0,462,416 Stripe: offset: 1148 data: 156 rows: 5000 tail: 55 index: 119 Stream: column 0 section ROW_INDEX start: 1148 length 17 Stream: column 1 section ROW_INDEX start: 1165 length 63 @@ -94,12 +94,12 @@ Encoding column 0: DIRECT Encoding column 1: DIRECT_V2 Encoding column 2: DICTIONARY_V2[0] - Row group index column 2: - Entry 0:count: 0 hasNull: true positions: 0,0,0,0,0,0,0 - Entry 1:count: 0 hasNull: true positions: 0,0,125,0,0,0,0 - Entry 2:count: 0 hasNull: true positions: 0,2,120,0,0,0,0 - Entry 3:count: 0 hasNull: true positions: 0,4,115,0,0,0,0 - Entry 4:count: 0 hasNull: true positions: 0,6,110,0,0,0,0 + Row group indices for column 2: + Entry 0: count: 0 hasNull: true positions: 0,0,0,0,0,0,0 + Entry 1: count: 0 hasNull: true positions: 0,0,125,0,0,0,0 + Entry 2: count: 0 hasNull: true positions: 0,2,120,0,0,0,0 + Entry 3: count: 0 hasNull: true positions: 0,4,115,0,0,0,0 + Entry 4: count: 0 hasNull: true positions: 0,6,110,0,0,0,0 File length: 1728 bytes Padding length: 0 bytes Index: ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out =================================================================== --- ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out (revision 1661246) +++ ql/src/test/results/clientnegative/create_function_nonexistent_class.q.out (working copy) @@ -2,5 +2,5 @@ PREHOOK: type: CREATEFUNCTION PREHOOK: Output: database:default PREHOOK: Output: default.badfunc -FAILED: Class my.nonexistent.class not found +Failed to register default.badfunc using class my.nonexistent.class FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask Index: ql/src/test/results/clientnegative/create_function_nonudf_class.q.out =================================================================== --- ql/src/test/results/clientnegative/create_function_nonudf_class.q.out (revision 1661246) +++ ql/src/test/results/clientnegative/create_function_nonudf_class.q.out (working copy) @@ -2,5 +2,5 @@ PREHOOK: type: CREATEFUNCTION PREHOOK: Output: database:default PREHOOK: Output: default.badfunc -FAILED: Class java.lang.String does not implement UDF, GenericUDF, or UDAF +Failed to register default.badfunc using class java.lang.String FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask Index: ql/src/test/results/clientnegative/drop_native_udf.q.out =================================================================== --- ql/src/test/results/clientnegative/drop_native_udf.q.out (revision 1661246) +++ ql/src/test/results/clientnegative/drop_native_udf.q.out (working copy) @@ -1,4 +1 @@ -PREHOOK: query: DROP TEMPORARY FUNCTION max -PREHOOK: type: DROPFUNCTION -PREHOOK: Output: max -FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask +FAILED: SemanticException [Error 10301]: Cannot drop native function max Index: ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out =================================================================== --- ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out (revision 1661246) +++ ql/src/test/results/clientnegative/udf_nonexistent_resource.q.out (working copy) @@ -4,4 +4,5 @@ PREHOOK: Output: default.lookup #### A masked pattern was here #### nonexistent_file.txt does not exist -FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask. nonexistent_file.txt does not exist +Failed to register default.lookup using class org.apache.hadoop.hive.ql.udf.UDFFileLookup +FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask Index: ql/src/test/results/clientnegative/union3.q.out =================================================================== --- ql/src/test/results/clientnegative/union3.q.out (revision 1661246) +++ ql/src/test/results/clientnegative/union3.q.out (working copy) @@ -1,13 +0,0 @@ -PREHOOK: query: -- Ensure that UNION ALL columns are in the correct order on both sides --- Ensure that the appropriate error message is propagated -CREATE TABLE IF NOT EXISTS union3 (bar int, baz int) -PREHOOK: type: CREATETABLE -PREHOOK: Output: database:default -PREHOOK: Output: default@union3 -POSTHOOK: query: -- Ensure that UNION ALL columns are in the correct order on both sides --- Ensure that the appropriate error message is propagated -CREATE TABLE IF NOT EXISTS union3 (bar int, baz int) -POSTHOOK: type: CREATETABLE -POSTHOOK: Output: database:default -POSTHOOK: Output: default@union3 -FAILED: SemanticException 2:85 Schema of both sides of union should match: field bar: appears on the left side of the UNION at column position: 0, and on the right side of the UNION at column position: 1. Column positions should match for a UNION. Error encountered near token 'union3' Index: ql/src/test/results/clientpositive/alter_partition_coltype.q.out =================================================================== --- ql/src/test/results/clientpositive/alter_partition_coltype.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/alter_partition_coltype.q.out (working copy) @@ -255,7 +255,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -433,7 +433,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -590,7 +590,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/annotate_stats_part.q.out =================================================================== --- ql/src/test/results/clientpositive/annotate_stats_part.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/annotate_stats_part.q.out (working copy) @@ -109,14 +109,12 @@ analyze table loc_orc partition(year='2001') compute statistics PREHOOK: type: QUERY PREHOOK: Input: default@loc_orc -PREHOOK: Input: default@loc_orc@year=2001 PREHOOK: Output: default@loc_orc PREHOOK: Output: default@loc_orc@year=2001 POSTHOOK: query: -- partition level analyze statistics for specific parition analyze table loc_orc partition(year='2001') compute statistics POSTHOOK: type: QUERY POSTHOOK: Input: default@loc_orc -POSTHOOK: Input: default@loc_orc@year=2001 POSTHOOK: Output: default@loc_orc POSTHOOK: Output: default@loc_orc@year=2001 PREHOOK: query: -- basicStatState: PARTIAL colStatState: NONE @@ -158,11 +156,11 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 10 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 7 Data size: 678 Basic stats: PARTIAL Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 - Statistics: Num rows: 10 Data size: 1840 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 7 Data size: 678 Basic stats: PARTIAL Column stats: PARTIAL ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE @@ -181,19 +179,17 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7 Data size: 678 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), '2001' (type: string) outputColumnNames: _col0, _col1, _col2, _col3 - Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7 Data size: 678 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- partition level analyze statistics for all partitions analyze table loc_orc partition(year) compute statistics PREHOOK: type: QUERY PREHOOK: Input: default@loc_orc -PREHOOK: Input: default@loc_orc@year=2001 -PREHOOK: Input: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ PREHOOK: Output: default@loc_orc PREHOOK: Output: default@loc_orc@year=2001 PREHOOK: Output: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ @@ -201,8 +197,6 @@ analyze table loc_orc partition(year) compute statistics POSTHOOK: type: QUERY POSTHOOK: Input: default@loc_orc -POSTHOOK: Input: default@loc_orc@year=2001 -POSTHOOK: Input: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ POSTHOOK: Output: default@loc_orc POSTHOOK: Output: default@loc_orc@year=2001 POSTHOOK: Output: default@loc_orc@year=__HIVE_DEFAULT_PARTITION__ @@ -222,11 +216,11 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), '__HIVE_DEFAULT_PARTITION__' (type: string) outputColumnNames: _col0, _col1, _col2, _col3 - Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: NONE @@ -245,7 +239,7 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 @@ -268,7 +262,7 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 @@ -331,11 +325,11 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: zip (type: bigint) outputColumnNames: _col0 - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: PARTIAL @@ -354,7 +348,7 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string) outputColumnNames: _col0 @@ -377,7 +371,7 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: year (type: string) outputColumnNames: _col0 @@ -402,7 +396,7 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int) outputColumnNames: _col0, _col1 @@ -425,7 +419,7 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 7 Data size: 678 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: state (type: string), locid (type: int) outputColumnNames: _col0, _col1 @@ -448,11 +442,11 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: state (type: string), locid (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 1 Data size: 342 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- basicStatState: COMPLETE colStatState: PARTIAL @@ -471,7 +465,7 @@ Processor Tree: TableScan alias: loc_orc - Statistics: Num rows: 8 Data size: 767 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 8 Data size: 774 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: state (type: string), locid (type: int), zip (type: bigint), year (type: string) outputColumnNames: _col0, _col1, _col2, _col3 @@ -496,7 +490,7 @@ Map Operator Tree: TableScan alias: loc_orc - Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 7 Data size: 678 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (locid > 0) (type: boolean) Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE @@ -532,7 +526,7 @@ Map Operator Tree: TableScan alias: loc_orc - Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 7 Data size: 678 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (locid > 0) (type: boolean) Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE @@ -568,7 +562,7 @@ Map Operator Tree: TableScan alias: loc_orc - Statistics: Num rows: 7 Data size: 425 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 7 Data size: 678 Basic stats: COMPLETE Column stats: COMPLETE Filter Operator predicate: (locid > 0) (type: boolean) Statistics: Num rows: 2 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE Index: ql/src/test/results/clientpositive/annotate_stats_table.q.out =================================================================== --- ql/src/test/results/clientpositive/annotate_stats_table.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/annotate_stats_table.q.out (working copy) @@ -122,11 +122,11 @@ Processor Tree: TableScan alias: emp_orc - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: lastname (type: string), deptid (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: NONE ListSink PREHOOK: query: -- column level partial statistics @@ -155,7 +155,7 @@ Processor Tree: TableScan alias: emp_orc - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: PARTIAL + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: PARTIAL Select Operator expressions: lastname (type: string), deptid (type: int) outputColumnNames: _col0, _col1 @@ -180,7 +180,7 @@ Processor Tree: TableScan alias: emp_orc - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: deptid (type: int) outputColumnNames: _col0 @@ -213,7 +213,7 @@ Processor Tree: TableScan alias: emp_orc - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: lastname (type: string), deptid (type: int) outputColumnNames: _col0, _col1 @@ -236,7 +236,7 @@ Processor Tree: TableScan alias: emp_orc - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: lastname (type: string) outputColumnNames: _col0 @@ -259,7 +259,7 @@ Processor Tree: TableScan alias: emp_orc - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: deptid (type: int) outputColumnNames: _col0 @@ -282,7 +282,7 @@ Processor Tree: TableScan alias: emp_orc - Statistics: Num rows: 48 Data size: 384 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 48 Data size: 4512 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: lastname (type: string), deptid (type: int) outputColumnNames: _col0, _col1 Index: ql/src/test/results/clientpositive/auto_join_reordering_values.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_join_reordering_values.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_join_reordering_values.q.out (working copy) @@ -667,7 +667,7 @@ columns _col0,_col1 columns.types string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out (working copy) @@ -294,7 +294,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -514,7 +514,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -856,7 +856,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1173,7 +1173,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1329,7 +1329,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out (working copy) @@ -412,7 +412,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -762,7 +762,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1071,7 +1071,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1508,7 +1508,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out (working copy) @@ -719,7 +719,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out (working copy) @@ -274,7 +274,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -618,7 +618,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -935,7 +935,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1091,7 +1091,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out (working copy) @@ -225,7 +225,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -396,7 +396,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -782,7 +782,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1052,7 +1052,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1159,7 +1159,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out (working copy) @@ -241,7 +241,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -412,7 +412,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -798,7 +798,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1068,7 +1068,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1175,7 +1175,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out (working copy) @@ -202,7 +202,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -363,7 +363,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -590,7 +590,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -751,7 +751,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -854,7 +854,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out (working copy) @@ -307,7 +307,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -529,7 +529,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -966,7 +966,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1331,7 +1331,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1487,7 +1487,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out =================================================================== --- ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out (working copy) @@ -307,7 +307,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -529,7 +529,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -968,7 +968,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1333,7 +1333,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1489,7 +1489,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/ba_table_union.q.out =================================================================== --- ql/src/test/results/clientpositive/ba_table_union.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ba_table_union.q.out (working copy) @@ -49,9 +49,9 @@ 11 12 12 -128 -145 -146 +15 +15 +153 PREHOOK: query: drop table ba_test PREHOOK: type: DROPTABLE PREHOOK: Input: default@ba_test Index: ql/src/test/results/clientpositive/bucket_map_join_1.q.out =================================================================== --- ql/src/test/results/clientpositive/bucket_map_join_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucket_map_join_1.q.out (working copy) @@ -237,7 +237,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucket_map_join_2.q.out =================================================================== --- ql/src/test/results/clientpositive/bucket_map_join_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucket_map_join_2.q.out (working copy) @@ -237,7 +237,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out =================================================================== --- ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out (working copy) @@ -229,7 +229,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -633,7 +633,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_1.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_1.q.out (working copy) @@ -378,7 +378,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -603,7 +603,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_2.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_2.q.out (working copy) @@ -362,7 +362,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -587,7 +587,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_3.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_3.q.out (working copy) @@ -359,7 +359,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -535,7 +535,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_4.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_4.q.out (working copy) @@ -375,7 +375,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -551,7 +551,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_5.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_5.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_5.q.out (working copy) @@ -243,7 +243,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -409,7 +409,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_6.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_6.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_6.q.out (working copy) @@ -314,7 +314,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -537,7 +537,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_7.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_7.q.out (working copy) @@ -441,7 +441,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -668,7 +668,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketcontext_8.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketcontext_8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketcontext_8.q.out (working copy) @@ -441,7 +441,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -668,7 +668,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin10.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin10.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin10.q.out (working copy) @@ -463,7 +463,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin11.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin11.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin11.q.out (working copy) @@ -481,7 +481,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -869,7 +869,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin12.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin12.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin12.q.out (working copy) @@ -346,7 +346,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -622,7 +622,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin13.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin13.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin13.q.out (working copy) @@ -346,7 +346,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -626,7 +626,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -905,7 +905,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1186,7 +1186,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin7.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin7.q.out (working copy) @@ -330,7 +330,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin8.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin8.q.out (working copy) @@ -312,7 +312,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -605,7 +605,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin9.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin9.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin9.q.out (working copy) @@ -312,7 +312,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -630,7 +630,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out =================================================================== --- ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out (working copy) @@ -268,7 +268,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -457,7 +457,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -635,7 +635,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -816,7 +816,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -997,7 +997,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1178,7 +1178,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1359,7 +1359,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1540,7 +1540,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1721,7 +1721,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/cbo_union.q.out =================================================================== --- ql/src/test/results/clientpositive/cbo_union.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/cbo_union.q.out (working copy) @@ -1,7 +1,7 @@ PREHOOK: query: -- SORT_QUERY_RESULTS -- 11. Union All -select * from cbo_t1 order by key, c_boolean, value, dt union all select * from cbo_t2 order by key, c_boolean, value, dt +select * from (select * from cbo_t1 order by key, c_boolean, value, dt)a union all select * from (select * from cbo_t2 order by key, c_boolean, value, dt)b PREHOOK: type: QUERY PREHOOK: Input: default@cbo_t1 PREHOOK: Input: default@cbo_t1@dt=2014 @@ -11,7 +11,7 @@ POSTHOOK: query: -- SORT_QUERY_RESULTS -- 11. Union All -select * from cbo_t1 order by key, c_boolean, value, dt union all select * from cbo_t2 order by key, c_boolean, value, dt +select * from (select * from cbo_t1 order by key, c_boolean, value, dt)a union all select * from (select * from cbo_t2 order by key, c_boolean, value, dt)b POSTHOOK: type: QUERY POSTHOOK: Input: default@cbo_t1 POSTHOOK: Input: default@cbo_t1@dt=2014 Index: ql/src/test/results/clientpositive/char_union1.q.out =================================================================== --- ql/src/test/results/clientpositive/char_union1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/char_union1.q.out (working copy) @@ -95,9 +95,6 @@ POSTHOOK: Input: default@char_union1_ch1 #### A masked pattern was here #### 1 abc -1 abc -2 abc -3 abc PREHOOK: query: -- union char with different length char select * from ( select * from char_union1_ch1 @@ -119,9 +116,6 @@ POSTHOOK: Input: default@char_union1_ch2 #### A masked pattern was here #### 1 abc -1 abc -2 abc -3 abc PREHOOK: query: -- union char with string select * from ( select * from char_union1_ch1 @@ -143,9 +137,6 @@ POSTHOOK: Input: default@char_union1_str #### A masked pattern was here #### 1 abc -1 abc -2 abc -3 abc PREHOOK: query: drop table char_union1_ch1 PREHOOK: type: DROPTABLE PREHOOK: Input: default@char_union1_ch1 Index: ql/src/test/results/clientpositive/columnstats_partlvl.q.out =================================================================== --- ql/src/test/results/clientpositive/columnstats_partlvl.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/columnstats_partlvl.q.out (working copy) @@ -204,7 +204,7 @@ columns _col0,_col1 columns.types struct:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -404,7 +404,7 @@ columns _col0,_col1 columns.types struct:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/columnstats_tbllvl.q.out =================================================================== --- ql/src/test/results/clientpositive/columnstats_tbllvl.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/columnstats_tbllvl.q.out (working copy) @@ -191,7 +191,7 @@ columns _col0,_col1,_col2 columns.types struct:struct:struct escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/combine2.q.out =================================================================== --- ql/src/test/results/clientpositive/combine2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/combine2.q.out (working copy) @@ -593,7 +593,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/combine2_hadoop20.q.out =================================================================== --- ql/src/test/results/clientpositive/combine2_hadoop20.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/combine2_hadoop20.q.out (working copy) @@ -606,7 +606,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/combine2_win.q.out =================================================================== --- ql/src/test/results/clientpositive/combine2_win.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/combine2_win.q.out (working copy) @@ -603,7 +603,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 TotalFiles: 1 GatherStats: false Index: ql/src/test/results/clientpositive/constantPropagateForSubQuery.q.out =================================================================== --- ql/src/test/results/clientpositive/constantPropagateForSubQuery.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/constantPropagateForSubQuery.q.out (working copy) @@ -233,7 +233,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/correlationoptimizer12.q.out =================================================================== --- ql/src/test/results/clientpositive/correlationoptimizer12.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/correlationoptimizer12.q.out (working copy) @@ -32,8 +32,8 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ + key expressions: _col0 (type: string) + sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string) @@ -43,6 +43,24 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col0 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -103,8 +121,8 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ + key expressions: _col0 (type: string) + sort order: + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string) @@ -114,6 +132,24 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col0 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) Index: ql/src/test/results/clientpositive/ctas_colname.q.out =================================================================== --- ql/src/test/results/clientpositive/ctas_colname.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ctas_colname.q.out (working copy) @@ -189,6 +189,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _wcol0 (type: int) @@ -336,6 +355,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col0, 1 + name: lead + window function: GenericUDAFLeadEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _wcol0 (type: string) Index: ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out =================================================================== --- ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out (working copy) @@ -207,7 +207,7 @@ columns _col0,_col1,_col2 columns.types struct:struct:struct escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out =================================================================== --- ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/encrypted/encryption_join_unencrypted_tbl.q.out (working copy) @@ -738,7 +738,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/encrypted/encryption_join_with_different_encryption_keys.q.out =================================================================== --- ql/src/test/results/clientpositive/encrypted/encryption_join_with_different_encryption_keys.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/encrypted/encryption_join_with_different_encryption_keys.q.out (working copy) @@ -258,7 +258,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/explain_logical.q.out =================================================================== --- ql/src/test/results/clientpositive/explain_logical.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/explain_logical.q.out (working copy) @@ -229,7 +229,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF Index: ql/src/test/results/clientpositive/explode_null.q.out =================================================================== --- ql/src/test/results/clientpositive/explode_null.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/explode_null.q.out (working copy) @@ -1,14 +1,14 @@ PREHOOK: query: SELECT explode(col) AS myCol FROM - (SELECT array(1,2,3) AS col FROM src LIMIT 1 + (select * from (SELECT array(1,2,3) AS col FROM src LIMIT 1)ea UNION ALL - SELECT IF(false, array(1,2,3), NULL) AS col FROM src LIMIT 1) a + select * from (SELECT IF(false, array(1,2,3), NULL) AS col FROM src LIMIT 1)eb) a PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: SELECT explode(col) AS myCol FROM - (SELECT array(1,2,3) AS col FROM src LIMIT 1 + (select * from (SELECT array(1,2,3) AS col FROM src LIMIT 1)ea UNION ALL - SELECT IF(false, array(1,2,3), NULL) AS col FROM src LIMIT 1) a + select * from (SELECT IF(false, array(1,2,3), NULL) AS col FROM src LIMIT 1)eb) a POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### @@ -16,16 +16,16 @@ 2 3 PREHOOK: query: SELECT explode(col) AS (myCol1,myCol2) FROM - (SELECT map(1,'one',2,'two',3,'three') AS col FROM src LIMIT 1 + (select * from (SELECT map(1,'one',2,'two',3,'three') AS col FROM src LIMIT 1)ea UNION ALL - SELECT IF(false, map(1,'one',2,'two',3,'three'), NULL) AS col FROM src LIMIT 1) a + select * from (SELECT IF(false, map(1,'one',2,'two',3,'three'), NULL) AS col FROM src LIMIT 1)eb ) a PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: SELECT explode(col) AS (myCol1,myCol2) FROM - (SELECT map(1,'one',2,'two',3,'three') AS col FROM src LIMIT 1 + (select * from (SELECT map(1,'one',2,'two',3,'three') AS col FROM src LIMIT 1)ea UNION ALL - SELECT IF(false, map(1,'one',2,'two',3,'three'), NULL) AS col FROM src LIMIT 1) a + select * from (SELECT IF(false, map(1,'one',2,'two',3,'three'), NULL) AS col FROM src LIMIT 1)eb ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/filter_join_breaktask.q.out =================================================================== --- ql/src/test/results/clientpositive/filter_join_breaktask.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/filter_join_breaktask.q.out (working copy) @@ -379,7 +379,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/groupby3_map_skew.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby3_map_skew.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby3_map_skew.q.out (working copy) @@ -151,12 +151,14 @@ POSTHOOK: Lineage: dest1.c7 EXPRESSION [(src)src.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: dest1.c8 EXPRESSION [(src)src.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: dest1.c9 EXPRESSION [(src)src.FieldSchema(name:value, type:string, comment:default), ] -PREHOOK: query: SELECT c1, c2, c3, c4, c5, c6, c7, ROUND(c8, 5), ROUND(c9, 5) FROM dest1 +PREHOOK: query: SELECT ROUND(c1, 1), ROUND(c2, 3), ROUND(c3, 5), ROUND(c4, 1), ROUND(c5, 1), ROUND(c6, 5), +ROUND(c7,5), ROUND(c8, 5), ROUND(c9, 5) FROM dest1 PREHOOK: type: QUERY PREHOOK: Input: default@dest1 #### A masked pattern was here #### -POSTHOOK: query: SELECT c1, c2, c3, c4, c5, c6, c7, ROUND(c8, 5), ROUND(c9, 5) FROM dest1 +POSTHOOK: query: SELECT ROUND(c1, 1), ROUND(c2, 3), ROUND(c3, 5), ROUND(c4, 1), ROUND(c5, 1), ROUND(c6, 5), +ROUND(c7,5), ROUND(c8, 5), ROUND(c9, 5) FROM dest1 POSTHOOK: type: QUERY POSTHOOK: Input: default@dest1 #### A masked pattern was here #### -130091.0 260.182 256.10355987055016 98.0 0.0 142.9268095075238 143.06995106518906 20428.07288 20469.0109 +130091.0 260.182 256.10356 98.0 0.0 142.92681 143.06995 20428.07288 20469.0109 Index: ql/src/test/results/clientpositive/groupby7_noskew_multi_single_reducer.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby7_noskew_multi_single_reducer.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby7_noskew_multi_single_reducer.q.out (working copy) @@ -20,13 +20,13 @@ POSTHOOK: Output: default@DEST2 PREHOOK: query: EXPLAIN FROM SRC -INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 -INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 PREHOOK: type: QUERY POSTHOOK: query: EXPLAIN FROM SRC -INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 -INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-2 is a root stage @@ -63,44 +63,39 @@ mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE - Limit - Number of rows: 10 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: true - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + File Output Operator + compressed: true + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Group By Operator aggregations: sum(VALUE._col0) keys: KEY._col0 (type: string) mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE - Limit - Number of rows: 10 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: true - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + File Output Operator + compressed: true + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-3 Map Reduce Map Operator Tree: TableScan Reduce Output Operator - sort order: - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - value expressions: _col0 (type: string), _col1 (type: double) + key expressions: _col0 (type: string) + sort order: + + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE + value expressions: _col1 (type: double) Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: string), VALUE._col1 (type: double) + expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE @@ -135,14 +130,15 @@ Map Operator Tree: TableScan Reduce Output Operator - sort order: - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - value expressions: _col0 (type: string), _col1 (type: double) + key expressions: _col0 (type: string) + sort order: + + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE + value expressions: _col1 (type: double) Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: string), VALUE._col1 (type: double) + expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE Index: ql/src/test/results/clientpositive/groupby_grouping_window.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby_grouping_window.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby_grouping_window.q.out (working copy) @@ -93,6 +93,25 @@ outputColumnNames: _col0, _col2, _col3 Statistics: Num rows: 1 Data size: 6 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: int, _col2: int, _col3: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col3 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col3 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 1 Data size: 6 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: int), _col2 (type: int), _col3 (type: int), _wcol0 (type: int) Index: ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out (working copy) @@ -1,9 +1,15 @@ PREHOOK: query: -- HIVE-3849 Aliased column in where clause for multi-groupby single reducer cannot be resolved + +-- SORT_QUERY_RESULTS + create table e1 (key string, count int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@e1 POSTHOOK: query: -- HIVE-3849 Aliased column in where clause for multi-groupby single reducer cannot be resolved + +-- SORT_QUERY_RESULTS + create table e1 (key string, count int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default Index: ql/src/test/results/clientpositive/groupby_resolution.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby_resolution.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby_resolution.q.out (working copy) @@ -677,6 +677,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: bigint + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: 0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: bigint), _wcol0 (type: int) Index: ql/src/test/results/clientpositive/groupby_sort_1.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby_sort_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby_sort_1.q.out (working copy) @@ -4335,7 +4335,7 @@ columns _col0,_col1,_col2,_col3,_col4 columns.types string:bigint:string:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/groupby_sort_1_23.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby_sort_1_23.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby_sort_1_23.q.out (working copy) @@ -2406,7 +2406,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -2903,7 +2903,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -4150,7 +4150,7 @@ columns _col0,_col1,_col2,_col3,_col4 columns.types string:bigint:string:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out (working copy) @@ -4725,7 +4725,7 @@ columns _col0,_col1,_col2,_col3,_col4 columns.types string:bigint:string:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out =================================================================== --- ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out (working copy) @@ -2731,7 +2731,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -3228,7 +3228,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -4605,7 +4605,7 @@ columns _col0,_col1,_col2,_col3,_col4 columns.types string:bigint:string:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/input23.q.out =================================================================== --- ql/src/test/results/clientpositive/input23.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/input23.q.out (working copy) @@ -161,7 +161,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7 columns.types string:string:string:string:string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/input25.q.out =================================================================== --- ql/src/test/results/clientpositive/input25.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/input25.q.out (working copy) @@ -22,16 +22,16 @@ POSTHOOK: Output: default@tst@d=2009-02-02 PREHOOK: query: explain select * from ( - select * from tst x where x.d='2009-01-01' limit 10 + select * from (select * from tst x where x.d='2009-01-01' limit 10)a union all - select * from tst x where x.d='2009-02-02' limit 10 + select * from (select * from tst x where x.d='2009-02-02' limit 10)b ) subq PREHOOK: type: QUERY POSTHOOK: query: explain select * from ( - select * from tst x where x.d='2009-01-01' limit 10 + select * from (select * from tst x where x.d='2009-01-01' limit 10)a union all - select * from tst x where x.d='2009-02-02' limit 10 + select * from (select * from tst x where x.d='2009-02-02' limit 10)b ) subq POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -144,9 +144,9 @@ ListSink PREHOOK: query: select * from ( - select * from tst x where x.d='2009-01-01' limit 10 + select * from (select * from tst x where x.d='2009-01-01' limit 10)a union all - select * from tst x where x.d='2009-02-02' limit 10 + select * from (select * from tst x where x.d='2009-02-02' limit 10)b ) subq PREHOOK: type: QUERY PREHOOK: Input: default@tst @@ -154,9 +154,9 @@ PREHOOK: Input: default@tst@d=2009-02-02 #### A masked pattern was here #### POSTHOOK: query: select * from ( - select * from tst x where x.d='2009-01-01' limit 10 + select * from (select * from tst x where x.d='2009-01-01' limit 10)a union all - select * from tst x where x.d='2009-02-02' limit 10 + select * from (select * from tst x where x.d='2009-02-02' limit 10)b ) subq POSTHOOK: type: QUERY POSTHOOK: Input: default@tst Index: ql/src/test/results/clientpositive/input26.q.out =================================================================== --- ql/src/test/results/clientpositive/input26.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/input26.q.out (working copy) @@ -1,15 +1,15 @@ PREHOOK: query: explain select * from ( - select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5 + select * from (select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5)pa union all - select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5 + select * from (select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5)pb )subq PREHOOK: type: QUERY POSTHOOK: query: explain select * from ( - select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5 + select * from (select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5)pa union all - select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5 + select * from (select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5)pb )subq POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -113,18 +113,18 @@ ListSink PREHOOK: query: select * from ( - select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5 + select * from (select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5)pa union all - select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5 + select * from (select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5)pb )subq PREHOOK: type: QUERY PREHOOK: Input: default@srcpart PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 #### A masked pattern was here #### POSTHOOK: query: select * from ( - select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5 + select * from (select * from srcpart a where a.ds = '2008-04-08' and a.hr = '11' order by a.key limit 5)pa union all - select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5 + select * from (select * from srcpart b where b.ds = '2008-04-08' and b.hr = '14' limit 5)pb )subq POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart Index: ql/src/test/results/clientpositive/input42.q.out =================================================================== --- ql/src/test/results/clientpositive/input42.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/input42.q.out (working copy) @@ -1220,7 +1220,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1792,7 +1792,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/input_part7.q.out =================================================================== --- ql/src/test/results/clientpositive/input_part7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/input_part7.q.out (working copy) @@ -19,7 +19,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -283,7 +283,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/input_part9.q.out =================================================================== --- ql/src/test/results/clientpositive/input_part9.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/input_part9.q.out (working copy) @@ -71,7 +71,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/join34.q.out =================================================================== --- ql/src/test/results/clientpositive/join34.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/join34.q.out (working copy) @@ -40,7 +40,7 @@ TOK_FROM TOK_JOIN TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF Index: ql/src/test/results/clientpositive/join35.q.out =================================================================== --- ql/src/test/results/clientpositive/join35.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/join35.q.out (working copy) @@ -40,7 +40,7 @@ TOK_FROM TOK_JOIN TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF Index: ql/src/test/results/clientpositive/join_filters_overlap.q.out =================================================================== --- ql/src/test/results/clientpositive/join_filters_overlap.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/join_filters_overlap.q.out (working copy) @@ -229,7 +229,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -480,7 +480,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -745,7 +745,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1040,7 +1040,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7 columns.types int:int:int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1329,7 +1329,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7 columns.types int:int:int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/limit_pushdown.q.out =================================================================== --- ql/src/test/results/clientpositive/limit_pushdown.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/limit_pushdown.q.out (working copy) @@ -352,34 +352,34 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: double) mode: hash outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double) sort order: + Map-reduce partition columns: _col0 (type: double) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Reduce Operator Tree: Group By Operator keys: KEY._col0 (type: double) mode: mergepartial outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -435,22 +435,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cdouble (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(DISTINCT _col1) keys: _col0 (type: tinyint), _col1 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: double) sort order: ++ Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Reduce Operator Tree: Group By Operator @@ -458,13 +458,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -522,22 +522,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cstring1 (type: string), cstring2 (type: string) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(DISTINCT _col1), count(DISTINCT _col2) keys: _col0 (type: tinyint), _col1 (type: string), _col2 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: string), _col2 (type: string) sort order: +++ Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Reduce Operator Tree: Group By Operator @@ -545,13 +545,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/list_bucket_dml_1.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_1.q.out (working copy) @@ -466,7 +466,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_11.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_11.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_11.q.java1.7.out (working copy) @@ -338,7 +338,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.7.out (working copy) @@ -351,7 +351,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:string:string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -509,7 +509,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:string:string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_13.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_13.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_13.q.java1.7.out (working copy) @@ -351,7 +351,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:string:string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_14.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_14.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_14.q.out (working copy) @@ -309,7 +309,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_2.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_2.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_2.q.java1.7.out (working copy) @@ -437,7 +437,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_3.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_3.q.out (working copy) @@ -406,7 +406,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_4.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_4.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_4.q.java1.7.out (working copy) @@ -848,7 +848,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_5.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_5.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_5.q.java1.7.out (working copy) @@ -481,7 +481,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_dml_9.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_dml_9.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_dml_9.q.java1.7.out (working copy) @@ -848,7 +848,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_query_multiskew_1.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_query_multiskew_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_query_multiskew_1.q.out (working copy) @@ -189,7 +189,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -349,7 +349,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -502,7 +502,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -656,7 +656,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out (working copy) @@ -187,7 +187,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -386,7 +386,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -559,7 +559,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_query_multiskew_3.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_query_multiskew_3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_query_multiskew_3.q.out (working copy) @@ -305,7 +305,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -511,7 +511,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -614,7 +614,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -768,7 +768,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_query_oneskew_1.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_query_oneskew_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_query_oneskew_1.q.out (working copy) @@ -242,7 +242,7 @@ columns _col0 columns.types int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -389,7 +389,7 @@ columns _col0 columns.types int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -532,7 +532,7 @@ columns _col0 columns.types int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out (working copy) @@ -259,7 +259,7 @@ columns _col0 columns.types int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -426,7 +426,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -653,7 +653,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -845,7 +845,7 @@ columns _col0,_col1 columns.types int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/list_bucket_query_oneskew_3.q.out =================================================================== --- ql/src/test/results/clientpositive/list_bucket_query_oneskew_3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/list_bucket_query_oneskew_3.q.out (working copy) @@ -270,7 +270,7 @@ columns _col0 columns.types int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/load_dyn_part14.q.out =================================================================== --- ql/src/test/results/clientpositive/load_dyn_part14.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/load_dyn_part14.q.out (working copy) @@ -36,21 +36,21 @@ PREHOOK: query: explain insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T PREHOOK: type: QUERY POSTHOOK: query: explain insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -256,22 +256,22 @@ PREHOOK: query: insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@nzhang_part14 POSTHOOK: query: insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T POSTHOOK: type: QUERY POSTHOOK: Input: default@src Index: ql/src/test/results/clientpositive/louter_join_ppr.q.out =================================================================== --- ql/src/test/results/clientpositive/louter_join_ppr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/louter_join_ppr.q.out (working copy) @@ -327,7 +327,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -803,7 +803,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1180,7 +1180,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1557,7 +1557,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out =================================================================== --- ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out (working copy) @@ -159,7 +159,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/merge4.q.out =================================================================== --- ql/src/test/results/clientpositive/merge4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/merge4.q.out (working copy) @@ -2762,34 +2762,51 @@ select '1' as key, '1' as value, 'file,' as hr from src limit 1) s POSTHOOK: type: QUERY STAGE DEPENDENCIES: - Stage-9 is a root stage - Stage-2 depends on stages: Stage-9 - Stage-8 depends on stages: Stage-2 , consists of Stage-5, Stage-4, Stage-6 + Stage-1 is a root stage + Stage-7 depends on stages: Stage-1 , consists of Stage-4, Stage-3, Stage-5 + Stage-4 + Stage-0 depends on stages: Stage-4, Stage-3, Stage-6 + Stage-2 depends on stages: Stage-0 + Stage-3 Stage-5 - Stage-0 depends on stages: Stage-5, Stage-4, Stage-7 - Stage-3 depends on stages: Stage-0 - Stage-4 - Stage-6 - Stage-7 depends on stages: Stage-6 + Stage-6 depends on stages: Stage-5 STAGE PLANS: - Stage: Stage-9 + Stage: Stage-1 Map Reduce Map Operator Tree: TableScan + alias: srcpart + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: key (type: string), value (type: string), hr (type: string) + outputColumnNames: _col0, _col1, _col2 + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE + Union + Statistics: Num rows: 1500 Data size: 140124 Basic stats: COMPLETE Column stats: COMPLETE + Limit + Number of rows: 1 + Statistics: Num rows: 1 Data size: 259 Basic stats: COMPLETE Column stats: COMPLETE + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 259 Basic stats: COMPLETE Column stats: COMPLETE + value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string) + TableScan alias: src Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: '1' (type: string), '1' (type: string), 'file,' (type: string) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 500 Data size: 129500 Basic stats: COMPLETE Column stats: COMPLETE - Limit - Number of rows: 1 - Statistics: Num rows: 1 Data size: 259 Basic stats: COMPLETE Column stats: COMPLETE - Reduce Output Operator - sort order: + Union + Statistics: Num rows: 1500 Data size: 140124 Basic stats: COMPLETE Column stats: COMPLETE + Limit + Number of rows: 1 Statistics: Num rows: 1 Data size: 259 Basic stats: COMPLETE Column stats: COMPLETE - value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string) + Reduce Output Operator + sort order: + Statistics: Num rows: 1 Data size: 259 Basic stats: COMPLETE Column stats: COMPLETE + value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string) Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: string) @@ -2800,47 +2817,17 @@ Statistics: Num rows: 1 Data size: 259 Basic stats: COMPLETE Column stats: COMPLETE File Output Operator compressed: false + Statistics: Num rows: 1 Data size: 259 Basic stats: COMPLETE Column stats: COMPLETE table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + name: default.nzhang_part - Stage: Stage-2 - Map Reduce - Map Operator Tree: - TableScan - alias: srcpart - Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE - Select Operator - expressions: key (type: string), value (type: string), hr (type: string) - outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE - Union - Statistics: Num rows: 1001 Data size: 10883 Basic stats: COMPLETE Column stats: COMPLETE - File Output Operator - compressed: false - Statistics: Num rows: 1001 Data size: 10883 Basic stats: COMPLETE Column stats: COMPLETE - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - name: default.nzhang_part - TableScan - Union - Statistics: Num rows: 1001 Data size: 10883 Basic stats: COMPLETE Column stats: COMPLETE - File Output Operator - compressed: false - Statistics: Num rows: 1001 Data size: 10883 Basic stats: COMPLETE Column stats: COMPLETE - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - name: default.nzhang_part - - Stage: Stage-8 + Stage: Stage-7 Conditional Operator - Stage: Stage-5 + Stage: Stage-4 Move Operator files: hdfs directory: true @@ -2859,10 +2846,10 @@ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.nzhang_part - Stage: Stage-3 + Stage: Stage-2 Stats-Aggr Operator - Stage: Stage-4 + Stage: Stage-3 Map Reduce Map Operator Tree: TableScan @@ -2874,7 +2861,7 @@ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.nzhang_part - Stage: Stage-6 + Stage: Stage-5 Map Reduce Map Operator Tree: TableScan @@ -2886,7 +2873,7 @@ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe name: default.nzhang_part - Stage: Stage-7 + Stage: Stage-6 Move Operator files: hdfs directory: true @@ -2913,13 +2900,7 @@ POSTHOOK: Input: default@srcpart POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=11 -POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=12 POSTHOOK: Output: default@nzhang_part@ds=2010-08-15/hr=file, -POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] -POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=11).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] -POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] -POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=12).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).key EXPRESSION [(srcpart)srcpart.FieldSchema(name:key, type:string, comment:default), ] POSTHOOK: Lineage: nzhang_part PARTITION(ds=2010-08-15,hr=file,).value EXPRESSION [(srcpart)srcpart.FieldSchema(name:value, type:string, comment:default), ] PREHOOK: query: show partitions nzhang_part Index: ql/src/test/results/clientpositive/metadataOnlyOptimizer.q.out =================================================================== --- ql/src/test/results/clientpositive/metadataOnlyOptimizer.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/metadataOnlyOptimizer.q.out (working copy) @@ -1332,24 +1332,23 @@ 96 97 98 -PREHOOK: query: select k from (SELECT '1' as k from src limit 0 union all select key as k from src limit 1)tab +PREHOOK: query: select k from (select * from (SELECT '1' as k from src limit 0)a union all select key as k from src limit 1)tab PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### -POSTHOOK: query: select k from (SELECT '1' as k from src limit 0 union all select key as k from src limit 1)tab +POSTHOOK: query: select k from (select * from (SELECT '1' as k from src limit 0)a union all select key as k from src limit 1)tab POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### 238 -PREHOOK: query: select k from (SELECT '1' as k from src limit 1 union all select key as k from src limit 0)tab +PREHOOK: query: select k from (select * from (SELECT '1' as k from src limit 1)a union all select key as k from src limit 0)tab PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### -POSTHOOK: query: select k from (SELECT '1' as k from src limit 1 union all select key as k from src limit 0)tab +POSTHOOK: query: select k from (select * from (SELECT '1' as k from src limit 1)a union all select key as k from src limit 0)tab POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### -1 PREHOOK: query: select max(ds) from srcpart PREHOOK: type: QUERY PREHOOK: Input: default@srcpart Index: ql/src/test/results/clientpositive/metadataonly1.q.out =================================================================== --- ql/src/test/results/clientpositive/metadataonly1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/metadataonly1.q.out (working copy) @@ -57,7 +57,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -202,7 +202,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -343,7 +343,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -484,7 +484,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -934,7 +934,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1199,7 +1199,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1439,7 +1439,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1623,7 +1623,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1994,7 +1994,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/optimize_nullscan.q.out =================================================================== --- ql/src/test/results/clientpositive/optimize_nullscan.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/optimize_nullscan.q.out (working copy) @@ -67,7 +67,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -202,7 +202,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -656,7 +656,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -699,7 +699,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -865,7 +865,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -890,7 +890,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1630,7 +1630,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1674,7 +1674,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -1849,7 +1849,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1884,7 +1884,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2134,7 +2134,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/orc_analyze.q.out =================================================================== --- ql/src/test/results/clientpositive/orc_analyze.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/orc_analyze.q.out (working copy) @@ -71,6 +71,55 @@ POSTHOOK: Lineage: orc_create_people.salary SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:salary, type:decimal(10,0), comment:null), ] POSTHOOK: Lineage: orc_create_people.start_date SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:start_date, type:timestamp, comment:null), ] POSTHOOK: Lineage: orc_create_people.state SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:state, type:string, comment:null), ] +PREHOOK: query: analyze table orc_create_people compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +POSTHOOK: query: analyze table orc_create_people compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +PREHOOK: query: desc formatted orc_create_people +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp +state string + +# Detailed Table Information +Database: default +#### A masked pattern was here #### +Protect Mode: None +Retention: 0 +#### A masked pattern was here #### +Table Type: MANAGED_TABLE +Table Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 100 + rawDataSize 52600 + totalSize 3158 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people compute statistics partialscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people @@ -120,6 +169,55 @@ Sort Columns: [] Storage Desc Params: serialization.format 1 +PREHOOK: query: analyze table orc_create_people compute statistics noscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +POSTHOOK: query: analyze table orc_create_people compute statistics noscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +PREHOOK: query: desc formatted orc_create_people +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp +state string + +# Detailed Table Information +Database: default +#### A masked pattern was here #### +Protect Mode: None +Retention: 0 +#### A masked pattern was here #### +Table Type: MANAGED_TABLE +Table Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 100 + rawDataSize 52600 + totalSize 3158 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: drop table orc_create_people PREHOOK: type: DROPTABLE PREHOOK: Input: default@orc_create_people @@ -269,6 +367,108 @@ POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).last_name SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:last_name, type:string, comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).salary SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:salary, type:decimal(10,0), comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).start_date SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:start_date, type:timestamp, comment:null), ] +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people @@ -371,6 +571,108 @@ Sort Columns: [] Storage Desc Params: serialization.format 1 +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: drop table orc_create_people PREHOOK: type: DROPTABLE PREHOOK: Input: default@orc_create_people @@ -583,6 +885,108 @@ POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).last_name SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:last_name, type:string, comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).salary SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:salary, type:decimal(10,0), comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).start_date SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:start_date, type:timestamp, comment:null), ] +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people @@ -685,6 +1089,108 @@ Sort Columns: [Order(col:last_name, order:1)] Storage Desc Params: serialization.format 1 +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 PREHOOK: query: drop table orc_create_people PREHOOK: type: DROPTABLE PREHOOK: Input: default@orc_create_people @@ -946,6 +1452,214 @@ POSTHOOK: type: ALTERTABLE_FILEFORMAT POSTHOOK: Input: default@orc_create_people POSTHOOK: Output: default@orc_create_people +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=OH +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=OH +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="OH") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="OH") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [OH] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE false + numFiles 1 + numRows -1 + rawDataSize -1 + totalSize 5812 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe +InputFormat: org.apache.hadoop.mapred.TextInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=OH +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=OH +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="OH") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="OH") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [OH] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE false + numFiles 1 + numRows -1 + rawDataSize -1 + totalSize 5812 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe +InputFormat: org.apache.hadoop.mapred.TextInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people Index: ql/src/test/results/clientpositive/outer_join_ppr.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/outer_join_ppr.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/outer_join_ppr.q.java1.7.out (working copy) @@ -420,7 +420,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -801,7 +801,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/parallel_join0.q.out =================================================================== --- ql/src/test/results/clientpositive/parallel_join0.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/parallel_join0.q.out (working copy) @@ -1,5 +1,5 @@ Warning: Shuffle Join JOIN[8][tables = [src1, src2]] in Stage 'Stage-1:MAPRED' is a cross product -PREHOOK: query: -- SORT_BEFORE_DIFF +PREHOOK: query: -- SORT_QUERY_RESULTS EXPLAIN SELECT src1.key as k1, src1.value as v1, @@ -9,7 +9,7 @@ (SELECT * FROM src WHERE src.key < 10) src2 SORT BY k1, v1, k2, v2 PREHOOK: type: QUERY -POSTHOOK: query: -- SORT_BEFORE_DIFF +POSTHOOK: query: -- SORT_QUERY_RESULTS EXPLAIN SELECT src1.key as k1, src1.value as v1, @@ -138,101 +138,101 @@ #### A masked pattern was here #### 0 val_0 0 val_0 0 val_0 0 val_0 -0 val_0 2 val_2 -0 val_0 4 val_4 -0 val_0 5 val_5 -0 val_0 5 val_5 -0 val_0 8 val_8 -2 val_2 2 val_2 -2 val_2 5 val_5 -4 val_4 0 val_0 -4 val_4 4 val_4 -4 val_4 5 val_5 -4 val_4 9 val_9 -5 val_5 0 val_0 -5 val_5 0 val_0 -5 val_5 4 val_4 -5 val_5 5 val_5 -5 val_5 8 val_8 -5 val_5 8 val_8 -5 val_5 9 val_9 -5 val_5 9 val_9 -8 val_8 0 val_0 -8 val_8 5 val_5 -8 val_8 5 val_5 -8 val_8 5 val_5 -9 val_9 0 val_0 -9 val_9 0 val_0 -9 val_9 0 val_0 -9 val_9 2 val_2 -9 val_9 5 val_5 -9 val_9 5 val_5 0 val_0 0 val_0 0 val_0 0 val_0 0 val_0 0 val_0 -0 val_0 2 val_2 -0 val_0 5 val_5 -0 val_0 5 val_5 -0 val_0 8 val_8 -0 val_0 9 val_9 -2 val_2 0 val_0 -2 val_2 5 val_5 -4 val_4 5 val_5 -5 val_5 0 val_0 -5 val_5 2 val_2 -5 val_5 2 val_2 -5 val_5 4 val_4 -5 val_5 5 val_5 -5 val_5 5 val_5 -5 val_5 8 val_8 -5 val_5 9 val_9 -8 val_8 0 val_0 0 val_0 0 val_0 +0 val_0 0 val_0 +0 val_0 0 val_0 +0 val_0 0 val_0 0 val_0 2 val_2 +0 val_0 2 val_2 +0 val_0 2 val_2 0 val_0 4 val_4 0 val_0 4 val_4 +0 val_0 4 val_4 0 val_0 5 val_5 0 val_0 5 val_5 +0 val_0 5 val_5 +0 val_0 5 val_5 +0 val_0 5 val_5 +0 val_0 5 val_5 +0 val_0 5 val_5 +0 val_0 5 val_5 +0 val_0 5 val_5 0 val_0 8 val_8 +0 val_0 8 val_8 +0 val_0 8 val_8 0 val_0 9 val_9 0 val_0 9 val_9 +0 val_0 9 val_9 2 val_2 0 val_0 2 val_2 0 val_0 +2 val_2 0 val_0 +2 val_2 2 val_2 2 val_2 4 val_4 +2 val_2 5 val_5 +2 val_2 5 val_5 +2 val_2 5 val_5 2 val_2 8 val_8 2 val_2 9 val_9 4 val_4 0 val_0 +4 val_4 0 val_0 +4 val_4 0 val_0 +4 val_4 2 val_2 +4 val_4 4 val_4 +4 val_4 5 val_5 +4 val_4 5 val_5 +4 val_4 5 val_5 4 val_4 8 val_8 +4 val_4 9 val_9 5 val_5 0 val_0 5 val_5 0 val_0 5 val_5 0 val_0 +5 val_5 0 val_0 +5 val_5 0 val_0 +5 val_5 0 val_0 +5 val_5 0 val_0 +5 val_5 0 val_0 +5 val_5 0 val_0 5 val_5 2 val_2 +5 val_5 2 val_2 +5 val_5 2 val_2 5 val_5 4 val_4 +5 val_5 4 val_4 +5 val_5 4 val_4 5 val_5 5 val_5 5 val_5 5 val_5 5 val_5 5 val_5 +5 val_5 5 val_5 +5 val_5 5 val_5 +5 val_5 5 val_5 +5 val_5 5 val_5 +5 val_5 5 val_5 +5 val_5 5 val_5 +5 val_5 8 val_8 +5 val_5 8 val_8 +5 val_5 8 val_8 +5 val_5 9 val_9 +5 val_5 9 val_9 +5 val_5 9 val_9 8 val_8 0 val_0 +8 val_8 0 val_0 +8 val_8 0 val_0 +8 val_8 2 val_2 8 val_8 4 val_4 +8 val_8 5 val_5 +8 val_8 5 val_5 +8 val_8 5 val_5 +8 val_8 8 val_8 +8 val_8 9 val_9 +9 val_9 0 val_0 +9 val_9 0 val_0 +9 val_9 0 val_0 +9 val_9 2 val_2 9 val_9 4 val_4 9 val_9 5 val_5 +9 val_9 5 val_5 +9 val_9 5 val_5 9 val_9 8 val_8 9 val_9 9 val_9 -0 val_0 0 val_0 -0 val_0 0 val_0 -0 val_0 0 val_0 -0 val_0 5 val_5 -0 val_0 5 val_5 -0 val_0 5 val_5 -2 val_2 5 val_5 -4 val_4 0 val_0 -4 val_4 2 val_2 -4 val_4 5 val_5 -5 val_5 0 val_0 -5 val_5 0 val_0 -5 val_5 0 val_0 -5 val_5 5 val_5 -5 val_5 5 val_5 -5 val_5 5 val_5 -8 val_8 2 val_2 -8 val_8 8 val_8 -8 val_8 9 val_9 Index: ql/src/test/results/clientpositive/parquet_types.q.out =================================================================== --- ql/src/test/results/clientpositive/parquet_types.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/parquet_types.q.out (working copy) @@ -215,8 +215,8 @@ MAX(cint), MIN(csmallint), COUNT(cstring1), - AVG(cfloat), - STDDEV_POP(cdouble) + ROUND(AVG(cfloat), 5), + ROUND(STDDEV_POP(cdouble),5) FROM parquet_types GROUP BY ctinyint ORDER BY ctinyint @@ -229,17 +229,17 @@ MAX(cint), MIN(csmallint), COUNT(cstring1), - AVG(cfloat), - STDDEV_POP(cdouble) + ROUND(AVG(cfloat), 5), + ROUND(STDDEV_POP(cdouble),5) FROM parquet_types GROUP BY ctinyint ORDER BY ctinyint POSTHOOK: type: QUERY POSTHOOK: Input: default@parquet_types #### A masked pattern was here #### -1 121 1 8 1.1749999970197678 2.0621590627301285 -2 119 1 7 1.2142857142857142 1.8 -3 120 1 7 1.171428578240531 1.7999999999999996 +1 121 1 8 1.175 2.06216 +2 119 1 7 1.21429 1.8 +3 120 1 7 1.17143 1.8 PREHOOK: query: SELECT cfloat, count(*) FROM parquet_types GROUP BY cfloat ORDER BY cfloat PREHOOK: type: QUERY PREHOOK: Input: default@parquet_types Index: ql/src/test/results/clientpositive/pcr.q.out =================================================================== --- ql/src/test/results/clientpositive/pcr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/pcr.q.out (working copy) @@ -235,7 +235,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -498,7 +498,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -763,7 +763,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -998,7 +998,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1281,7 +1281,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1574,7 +1574,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1820,7 +1820,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2025,7 +2025,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2316,7 +2316,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2603,7 +2603,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2865,7 +2865,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:string:string:int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3208,7 +3208,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:string:string:int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3578,7 +3578,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3895,7 +3895,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5349,7 +5349,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5576,7 +5576,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5799,7 +5799,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/ppd_join_filter.q.out =================================================================== --- ql/src/test/results/clientpositive/ppd_join_filter.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ppd_join_filter.q.out (working copy) @@ -375,7 +375,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -792,7 +792,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1209,7 +1209,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1626,7 +1626,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/ppd_union_view.q.out =================================================================== --- ql/src/test/results/clientpositive/ppd_union_view.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ppd_union_view.q.out (working copy) @@ -353,7 +353,7 @@ columns _col0,_col1,_col2 columns.types string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -471,8 +471,8 @@ condition map: Inner Join 0 to 1 keys: - 0 _col0 (type: string), _col2 (type: string) - 1 _col1 (type: string), _col2 (type: string) + 0 _col0 (type: string) + 1 _col1 (type: string) outputColumnNames: _col1, _col3 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator @@ -524,7 +524,7 @@ columns _col0,_col1,_col2 columns.types string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -549,7 +549,7 @@ columns _col0,_col1,_col2 columns.types string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/ppd_vc.q.out =================================================================== --- ql/src/test/results/clientpositive/ppd_vc.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ppd_vc.q.out (working copy) @@ -63,7 +63,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -773,7 +773,7 @@ columns _col0,_col1,_col2,_col3,_col4 columns.types string:string:string:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/ppr_allchildsarenull.q.out =================================================================== --- ql/src/test/results/clientpositive/ppr_allchildsarenull.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ppr_allchildsarenull.q.out (working copy) @@ -94,7 +94,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -326,7 +326,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/ptf.q.out =================================================================== --- ql/src/test/results/clientpositive/ptf.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ptf.q.out (working copy) @@ -48,6 +48,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -72,6 +84,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -220,6 +264,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -244,6 +300,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -340,10 +415,22 @@ value expressions: p_size (type: int) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col3 (type: int) - outputColumnNames: _col2, _col1, _col5 + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int) + outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -449,6 +536,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -473,6 +572,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -586,6 +717,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -610,6 +753,39 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), (_col5 - _wcol2) (type: int) @@ -726,6 +902,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -781,6 +969,39 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col2, 1, _col2 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col2 (type: int), (_col2 - _wcol2) (type: int) @@ -889,6 +1110,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1028,6 +1261,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1160,7 +1405,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -1172,6 +1431,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1195,6 +1467,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -1289,7 +1580,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -1302,6 +1607,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1326,6 +1644,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1435,6 +1785,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1459,6 +1821,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1532,7 +1926,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) PREHOOK: type: QUERY POSTHOOK: query: -- 12. testFunctionChain @@ -1543,7 +1937,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -1560,20 +1954,53 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: -+ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -1586,17 +2013,37 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: -+ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1621,6 +2068,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1646,7 +2125,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) PREHOOK: type: QUERY PREHOOK: Input: default@part @@ -1657,7 +2136,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) POSTHOOK: type: QUERY POSTHOOK: Input: default@part @@ -1740,6 +2219,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1764,6 +2255,30 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol1 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: bigint), _wcol1 (type: double) @@ -1888,6 +2403,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1952,6 +2479,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -2063,6 +2635,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -2187,8 +2771,7 @@ STAGE DEPENDENCIES: Stage-1 is a root stage Stage-2 depends on stages: Stage-1 - Stage-3 depends on stages: Stage-2 - Stage-0 depends on stages: Stage-3 + Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-1 @@ -2210,7 +2793,7 @@ Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) sort order: ++ - Map-reduce partition columns: _col0 (type: string), _col1 (type: string) + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double) Reduce Operator Tree: @@ -2220,29 +2803,19 @@ mode: mergepartial outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - - Stage: Stage-2 - Map Reduce - Map Operator Tree: - TableScan - Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE - value expressions: _col1 (type: string), _col2 (type: double) - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1 (type: double) - outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: mfgr_price_view + output shape: _col0: string, _col1: string, _col2: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col0 + output shape: _col0: string, _col1: string, _col2: double + partition by: _col0 + raw input shape: Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -2251,7 +2824,7 @@ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - Stage: Stage-3 + Stage: Stage-2 Map Reduce Map Operator Tree: TableScan @@ -2267,6 +2840,24 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: double), _wcol0 (type: double) @@ -2429,6 +3020,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -2459,6 +3062,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -2502,30 +3137,91 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(5)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + Select Operator + expressions: _col1 (type: string), _col2 (type: string), _col5 (type: int), _wcol0 (type: bigint) + outputColumnNames: _col1, _col2, _col5, _wcol0 + Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-6 Map Reduce Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _wcol0 (type: bigint), _col5 (type: int) Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) + expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) outputColumnNames: _col0, _col2, _col3, _col6 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: bigint, _col2: string, _col3: string, _col6: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col3, _col2 + partition by: _col3 + raw input shape: + window functions: + window function definition + alias: _wcol1 + arguments: _col3, _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col3, _col2 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol3 + arguments: _col3, _col2 + name: cume_dist + window function: GenericUDAFCumeDistEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol4 + arguments: _col6, true + name: first_value + window function: GenericUDAFFirstValueEvaluator + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col3 (type: string), _col2 (type: string), _col6 (type: int), UDFToInteger(round(_col0, 1)) (type: int), _wcol1 (type: int), _wcol2 (type: int), _wcol3 (type: double), _wcol4 (type: int) @@ -2721,8 +3417,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -2732,9 +3428,49 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -2747,8 +3483,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2758,6 +3494,26 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -2771,8 +3527,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2782,6 +3538,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -2914,8 +3702,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -2925,6 +3713,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -2938,8 +3745,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2949,6 +3756,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -2962,8 +3781,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -2973,6 +3792,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -2997,6 +3828,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -3124,8 +3987,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3135,6 +3998,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3148,8 +4030,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3159,6 +4041,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3183,6 +4084,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -3311,8 +4244,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3322,6 +4255,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3335,8 +4287,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3346,9 +4298,35 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -3361,8 +4339,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3372,6 +4350,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3385,8 +4376,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3396,6 +4387,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -3525,8 +4548,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3536,9 +4559,42 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -3551,8 +4607,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3562,6 +4618,26 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3575,8 +4651,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3586,6 +4662,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -3709,8 +4817,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3720,9 +4828,42 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -3735,8 +4876,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3746,6 +4887,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3770,6 +4924,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) Index: ql/src/test/results/clientpositive/ptf_matchpath.q.out =================================================================== --- ql/src/test/results/clientpositive/ptf_matchpath.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ptf_matchpath.q.out (working copy) @@ -37,6 +37,7 @@ PREHOOK: query: -- SORT_QUERY_RESULTS -- 1. basic Matchpath test +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -47,11 +48,10 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) PREHOOK: type: QUERY -PREHOOK: Input: default@flights_tiny -#### A masked pattern was here #### POSTHOOK: query: -- SORT_QUERY_RESULTS -- 1. basic Matchpath test +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -62,6 +62,84 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-1 + Map Reduce + Map Operator Tree: + TableScan + alias: flights_tiny + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + key expressions: fl_num (type: string), year (type: int), month (type: int), day_of_month (type: int) + sort order: ++++ + Map-reduce partition columns: fl_num (type: string) + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + value expressions: origin_city_name (type: string), arr_delay (type: float) + Reduce Operator Tree: + Select Operator + expressions: VALUE._col0 (type: string), KEY.reducesinkkey1 (type: int), KEY.reducesinkkey2 (type: int), KEY.reducesinkkey3 (type: int), VALUE._col2 (type: float), KEY.reducesinkkey0 (type: string) + outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + PTF Operator + Function definitions: + Input definition + input alias: flights_tiny + output shape: + type: TABLE + Partition table definition + input alias: ptf_1 + arguments: 'LATE.LATE+', 'LATE', (_col5 > 15.0), 'origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath' + name: matchpath + order by: _col2, _col3, _col4 + output shape: tpath: int + partition by: _col6 + raw input shape: + referenced columns: _col0, _col6, _col2, _col3, _col4, tpath, _col5 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), fl_num (type: string), year (type: int), month (type: int), day_of_month (type: int), sz (type: int), tpath (type: int) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + distribute by fl_num + sort by year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +PREHOOK: type: QUERY +PREHOOK: Input: default@flights_tiny +#### A masked pattern was here #### +POSTHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + distribute by fl_num + sort by year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +POSTHOOK: type: QUERY POSTHOOK: Input: default@flights_tiny #### A masked pattern was here #### Baltimore 1142 2010 10 20 6 20 @@ -81,6 +159,7 @@ Chicago 897 2010 10 22 2 22 Washington 7291 2010 10 27 2 27 PREHOOK: query: -- 2. Matchpath on 1 partition +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -91,9 +170,8 @@ ) where fl_num = 1142 PREHOOK: type: QUERY -PREHOOK: Input: default@flights_tiny -#### A masked pattern was here #### POSTHOOK: query: -- 2. Matchpath on 1 partition +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -104,6 +182,87 @@ ) where fl_num = 1142 POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-1 + Map Reduce + Map Operator Tree: + TableScan + alias: flights_tiny + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + key expressions: 0 (type: int), fl_num (type: string), year (type: int), month (type: int), day_of_month (type: int) + sort order: +++++ + Map-reduce partition columns: 0 (type: int) + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + value expressions: origin_city_name (type: string), arr_delay (type: float) + Reduce Operator Tree: + Select Operator + expressions: VALUE._col0 (type: string), KEY.reducesinkkey2 (type: int), KEY.reducesinkkey3 (type: int), KEY.reducesinkkey4 (type: int), VALUE._col2 (type: float), KEY.reducesinkkey1 (type: string) + outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + PTF Operator + Function definitions: + Input definition + input alias: flights_tiny + output shape: + type: TABLE + Partition table definition + input alias: ptf_1 + arguments: 'LATE.LATE+', 'LATE', (_col5 > 15.0), 'origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath' + name: matchpath + order by: _col6, _col2, _col3, _col4 + output shape: tpath: int + partition by: 0 + raw input shape: + referenced columns: _col0, _col6, _col2, _col3, _col4, tpath, _col5 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (fl_num = 1142) (type: boolean) + Statistics: Num rows: 12 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), '1142' (type: string), year (type: int), month (type: int), day_of_month (type: int), sz (type: int), tpath (type: int) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 12 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 12 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +where fl_num = 1142 +PREHOOK: type: QUERY +PREHOOK: Input: default@flights_tiny +#### A masked pattern was here #### +POSTHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +where fl_num = 1142 +POSTHOOK: type: QUERY POSTHOOK: Input: default@flights_tiny #### A masked pattern was here #### Baltimore 1142 2010 10 20 6 20 @@ -112,6 +271,7 @@ Baltimore 1142 2010 10 25 3 25 Baltimore 1142 2010 10 26 2 26 PREHOOK: query: -- 3. empty partition. +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on (select * from flights_tiny where fl_num = -1142) flights_tiny @@ -121,9 +281,8 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) PREHOOK: type: QUERY -PREHOOK: Input: default@flights_tiny -#### A masked pattern was here #### POSTHOOK: query: -- 3. empty partition. +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on (select * from flights_tiny where fl_num = -1142) flights_tiny @@ -133,5 +292,88 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-1 + Map Reduce + Map Operator Tree: + TableScan + alias: flights_tiny + Statistics: Num rows: 44 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (fl_num = -1142) (type: boolean) + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), year (type: int), month (type: int), day_of_month (type: int), arr_delay (type: float) + outputColumnNames: _col0, _col2, _col3, _col4, _col5 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + key expressions: 0 (type: int), '-1142' (type: string), _col2 (type: int), _col3 (type: int), _col4 (type: int) + sort order: +++++ + Map-reduce partition columns: 0 (type: int) + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: string), _col5 (type: float) + Reduce Operator Tree: + Select Operator + expressions: VALUE._col0 (type: string), KEY.reducesinkkey2 (type: int), KEY.reducesinkkey3 (type: int), KEY.reducesinkkey4 (type: int), VALUE._col2 (type: float), '-1142' (type: string) + outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + PTF Operator + Function definitions: + Input definition + input alias: flights_tiny + output shape: + type: SUBQUERY + Partition table definition + input alias: ptf_1 + arguments: 'LATE.LATE+', 'LATE', (_col5 > 15.0), 'origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath' + name: matchpath + order by: _col6, _col2, _col3, _col4 + output shape: tpath: int + partition by: 0 + raw input shape: + referenced columns: _col0, _col6, _col2, _col3, _col4, tpath, _col5 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), '-1142' (type: string), year (type: int), month (type: int), day_of_month (type: int), sz (type: int), tpath (type: int) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + (select * from flights_tiny where fl_num = -1142) flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +PREHOOK: type: QUERY +PREHOOK: Input: default@flights_tiny +#### A masked pattern was here #### +POSTHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + (select * from flights_tiny where fl_num = -1142) flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +POSTHOOK: type: QUERY POSTHOOK: Input: default@flights_tiny #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/ptf_streaming.q.out =================================================================== --- ql/src/test/results/clientpositive/ptf_streaming.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/ptf_streaming.q.out (working copy) @@ -48,6 +48,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -72,6 +84,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -220,6 +264,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -244,6 +300,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -347,6 +422,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noopstreaming + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -475,7 +562,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -487,6 +588,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -510,6 +624,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -604,7 +737,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -617,6 +764,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -641,6 +801,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -742,20 +934,53 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -768,17 +993,37 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -803,6 +1048,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -906,20 +1183,53 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -932,17 +1242,37 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -967,6 +1297,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1070,20 +1432,53 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -1096,17 +1491,37 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1131,6 +1546,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1249,6 +1696,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noopstreaming + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1313,6 +1772,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -1436,8 +1940,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -1447,9 +1951,49 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -1462,8 +2006,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1473,6 +2017,26 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1486,8 +2050,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1497,6 +2061,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -1629,8 +2225,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -1640,6 +2236,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1653,8 +2268,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1664,6 +2279,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1677,8 +2304,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -1688,6 +2315,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1712,6 +2351,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -1837,8 +2508,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -1848,9 +2519,42 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false table: @@ -1863,8 +2567,8 @@ Map Operator Tree: TableScan Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1874,6 +2578,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1898,6 +2615,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) Index: ql/src/test/results/clientpositive/push_or.q.out =================================================================== --- ql/src/test/results/clientpositive/push_or.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/push_or.q.out (working copy) @@ -217,7 +217,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/quotedid_basic.q.out =================================================================== --- ql/src/test/results/clientpositive/quotedid_basic.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/quotedid_basic.q.out (working copy) @@ -198,6 +198,25 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _wcol0 (type: int) @@ -283,6 +302,25 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _wcol0 (type: int) Index: ql/src/test/results/clientpositive/rand_partitionpruner1.q.out =================================================================== --- ql/src/test/results/clientpositive/rand_partitionpruner1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/rand_partitionpruner1.q.out (working copy) @@ -60,7 +60,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/rand_partitionpruner3.q.out =================================================================== --- ql/src/test/results/clientpositive/rand_partitionpruner3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/rand_partitionpruner3.q.out (working copy) @@ -90,7 +90,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -254,7 +254,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/regexp_extract.q.out =================================================================== --- ql/src/test/results/clientpositive/regexp_extract.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/regexp_extract.q.out (working copy) @@ -196,7 +196,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -511,7 +511,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/router_join_ppr.q.out =================================================================== --- ql/src/test/results/clientpositive/router_join_ppr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/router_join_ppr.q.out (working copy) @@ -426,7 +426,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -807,7 +807,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1184,7 +1184,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1557,7 +1557,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/sample10.q.out =================================================================== --- ql/src/test/results/clientpositive/sample10.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sample10.q.out (working copy) @@ -97,6 +97,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 4 Statistics: Num rows: 20 Data size: 120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ds (type: string) @@ -399,7 +400,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/sample2.q.out =================================================================== --- ql/src/test/results/clientpositive/sample2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sample2.q.out (working copy) @@ -62,6 +62,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) Index: ql/src/test/results/clientpositive/sample4.q.out =================================================================== --- ql/src/test/results/clientpositive/sample4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sample4.q.out (working copy) @@ -64,6 +64,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) Index: ql/src/test/results/clientpositive/sample6.q.out =================================================================== --- ql/src/test/results/clientpositive/sample6.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sample6.q.out (working copy) @@ -62,6 +62,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 4 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -645,6 +646,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 3) (type: boolean) + sampleDesc: BUCKET 4 OUT OF 4 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -727,7 +729,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1042,6 +1044,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -1124,7 +1127,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1775,7 +1778,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2269,7 +2272,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2667,6 +2670,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -2796,7 +2800,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2994,6 +2998,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 1) (type: boolean) + sampleDesc: BUCKET 2 OUT OF 4 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -3076,7 +3081,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3210,7 +3215,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/sample8.q.out =================================================================== --- ql/src/test/results/clientpositive/sample8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sample8.q.out (working copy) @@ -346,7 +346,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/sample9.q.out =================================================================== --- ql/src/test/results/clientpositive/sample9.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sample9.q.out (working copy) @@ -58,6 +58,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -77,7 +78,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/script_env_var1.q.out =================================================================== --- ql/src/test/results/clientpositive/script_env_var1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/script_env_var1.q.out (working copy) @@ -1,16 +1,16 @@ PREHOOK: query: -- Verifies that script operator ID environment variables have unique values -- in each instance of the script operator. SELECT count(1) FROM -( SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)x UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)y ) a GROUP BY key PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- Verifies that script operator ID environment variables have unique values -- in each instance of the script operator. SELECT count(1) FROM -( SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)x UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)y ) a GROUP BY key POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/script_env_var2.q.out =================================================================== --- ql/src/test/results/clientpositive/script_env_var2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/script_env_var2.q.out (working copy) @@ -1,14 +1,14 @@ PREHOOK: query: -- Same test as script_env_var1, but test setting the variable name SELECT count(1) FROM -( SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)a UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)b ) a GROUP BY key PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- Same test as script_env_var1, but test setting the variable name SELECT count(1) FROM -( SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)a UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)b ) a GROUP BY key POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/serde_user_properties.q.out =================================================================== --- ql/src/test/results/clientpositive/serde_user_properties.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/serde_user_properties.q.out (working copy) @@ -140,7 +140,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -310,7 +310,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -488,7 +488,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/show_functions.q.out =================================================================== --- ql/src/test/results/clientpositive/show_functions.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/show_functions.q.out (working copy) @@ -105,6 +105,7 @@ lead least length +levenshtein like ln locate @@ -124,6 +125,7 @@ month named_struct negative +next_day ngrams noop noopstreaming Index: ql/src/test/results/clientpositive/smb_mapjoin_11.q.out =================================================================== --- ql/src/test/results/clientpositive/smb_mapjoin_11.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/smb_mapjoin_11.q.out (working copy) @@ -1863,6 +1863,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 16) = 1) (type: boolean) + sampleDesc: BUCKET 2 OUT OF 16 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string), ds (type: string) @@ -1882,7 +1883,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1987,6 +1988,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 16) = 1) (type: boolean) + sampleDesc: BUCKET 2 OUT OF 16 Statistics: Num rows: 514 Data size: 5484 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string), ds (type: string) @@ -2006,7 +2008,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/smb_mapjoin_13.q.out =================================================================== --- ql/src/test/results/clientpositive/smb_mapjoin_13.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/smb_mapjoin_13.q.out (working copy) @@ -226,7 +226,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -451,7 +451,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/smb_mapjoin_15.q.out =================================================================== --- ql/src/test/results/clientpositive/smb_mapjoin_15.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/smb_mapjoin_15.q.out (working copy) @@ -198,7 +198,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -454,7 +454,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:string:int:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -658,7 +658,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:string:int:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -901,7 +901,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:string:int:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/sort_merge_join_desc_5.q.out =================================================================== --- ql/src/test/results/clientpositive/sort_merge_join_desc_5.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sort_merge_join_desc_5.q.out (working copy) @@ -215,7 +215,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out =================================================================== --- ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out (working copy) @@ -299,7 +299,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out =================================================================== --- ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out (working copy) @@ -419,7 +419,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_join_reordering_values.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_join_reordering_values.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_join_reordering_values.q.out (working copy) @@ -601,7 +601,7 @@ columns _col0,_col1 columns.types string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out (working copy) @@ -305,7 +305,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -536,7 +536,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -858,7 +858,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out (working copy) @@ -382,7 +382,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -713,7 +713,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1043,7 +1043,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1515,7 +1515,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out (working copy) @@ -610,7 +610,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out (working copy) @@ -285,7 +285,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -609,7 +609,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out (working copy) @@ -236,7 +236,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -418,7 +418,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -740,7 +740,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out (working copy) @@ -252,7 +252,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -434,7 +434,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -756,7 +756,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out (working copy) @@ -213,7 +213,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -385,7 +385,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -644,7 +644,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out (working copy) @@ -318,7 +318,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -551,7 +551,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -924,7 +924,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out (working copy) @@ -318,7 +318,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -551,7 +551,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -926,7 +926,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucket_map_join_1.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucket_map_join_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucket_map_join_1.q.out (working copy) @@ -298,7 +298,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucket_map_join_2.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucket_map_join_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucket_map_join_2.q.out (working copy) @@ -298,7 +298,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucket_map_join_spark4.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucket_map_join_spark4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucket_map_join_spark4.q.out (working copy) @@ -348,7 +348,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -774,7 +774,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin10.q.out (working copy) @@ -480,7 +480,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin11.q.out (working copy) @@ -501,7 +501,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -909,7 +909,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin12.q.out (working copy) @@ -363,7 +363,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -653,7 +653,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin13.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin13.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin13.q.out (working copy) @@ -360,7 +360,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -657,7 +657,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -953,7 +953,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1251,7 +1251,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out (working copy) @@ -347,7 +347,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin8.q.out (working copy) @@ -329,7 +329,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -639,7 +639,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin9.q.out (working copy) @@ -326,7 +326,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -658,7 +658,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/bucketmapjoin_negative3.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/bucketmapjoin_negative3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/bucketmapjoin_negative3.q.out (working copy) @@ -320,7 +320,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -566,7 +566,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -803,7 +803,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1038,7 +1038,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1273,7 +1273,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1508,7 +1508,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1743,7 +1743,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1978,7 +1978,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2213,7 +2213,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/filter_join_breaktask.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/filter_join_breaktask.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/filter_join_breaktask.q.out (working copy) @@ -396,7 +396,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/groupby3_map_skew.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/groupby3_map_skew.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/groupby3_map_skew.q.out (working copy) @@ -147,12 +147,14 @@ POSTHOOK: Lineage: dest1.c7 EXPRESSION [(src)src.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: dest1.c8 EXPRESSION [(src)src.FieldSchema(name:value, type:string, comment:default), ] POSTHOOK: Lineage: dest1.c9 EXPRESSION [(src)src.FieldSchema(name:value, type:string, comment:default), ] -PREHOOK: query: SELECT c1, c2, c3, c4, c5, c6, c7, ROUND(c8, 5), ROUND(c9, 5) FROM dest1 +PREHOOK: query: SELECT ROUND(c1, 1), ROUND(c2, 3), ROUND(c3, 5), ROUND(c4, 1), ROUND(c5, 1), ROUND(c6, 5), +ROUND(c7,5), ROUND(c8, 5), ROUND(c9, 5) FROM dest1 PREHOOK: type: QUERY PREHOOK: Input: default@dest1 #### A masked pattern was here #### -POSTHOOK: query: SELECT c1, c2, c3, c4, c5, c6, c7, ROUND(c8, 5), ROUND(c9, 5) FROM dest1 +POSTHOOK: query: SELECT ROUND(c1, 1), ROUND(c2, 3), ROUND(c3, 5), ROUND(c4, 1), ROUND(c5, 1), ROUND(c6, 5), +ROUND(c7,5), ROUND(c8, 5), ROUND(c9, 5) FROM dest1 POSTHOOK: type: QUERY POSTHOOK: Input: default@dest1 #### A masked pattern was here #### -130091.0 260.182 256.10355987055016 98.0 0.0 142.9268095075238 143.06995106518906 20428.07288 20469.0109 +130091.0 260.182 256.10356 98.0 0.0 142.92681 143.06995 20428.07288 20469.0109 Index: ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out (working copy) @@ -20,13 +20,13 @@ POSTHOOK: Output: default@DEST2 PREHOOK: query: EXPLAIN FROM SRC -INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 -INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 PREHOOK: type: QUERY POSTHOOK: query: EXPLAIN FROM SRC -INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 -INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST1 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 +INSERT OVERWRITE TABLE DEST2 SELECT SRC.key, sum(SUBSTR(SRC.value,5)) GROUP BY SRC.key ORDER BY SRC.key limit 10 POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-2 is a root stage @@ -41,8 +41,8 @@ Edges: Reducer 5 <- Map 1 (SORT, 31) Reducer 6 <- Map 1 (SORT, 31) - Reducer 3 <- Reducer 5 (GROUP, 1) - Reducer 4 <- Reducer 6 (GROUP, 1) + Reducer 3 <- Reducer 5 (SORT, 1) + Reducer 4 <- Reducer 6 (SORT, 1) #### A masked pattern was here #### Vertices: Map 1 @@ -63,9 +63,9 @@ Reducer 3 Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: string), VALUE._col1 (type: double) + expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE @@ -84,9 +84,9 @@ Reducer 4 Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: string), VALUE._col1 (type: double) + expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE @@ -112,13 +112,11 @@ mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE - Limit - Number of rows: 10 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - sort order: - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - value expressions: _col0 (type: string), _col1 (type: double) + Reduce Output Operator + key expressions: _col0 (type: string) + sort order: + + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE + value expressions: _col1 (type: double) Reducer 6 Reduce Operator Tree: Forward @@ -129,13 +127,11 @@ mode: complete outputColumnNames: _col0, _col1 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE - Limit - Number of rows: 10 - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - sort order: - Statistics: Num rows: 10 Data size: 100 Basic stats: COMPLETE Column stats: NONE - value expressions: _col0 (type: string), _col1 (type: double) + Reduce Output Operator + key expressions: _col0 (type: string) + sort order: + + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE + value expressions: _col1 (type: double) Stage: Stage-0 Move Operator Index: ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer3.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer3.q.out (working copy) @@ -1,9 +1,15 @@ PREHOOK: query: -- HIVE-3849 Aliased column in where clause for multi-groupby single reducer cannot be resolved + +-- SORT_QUERY_RESULTS + create table e1 (key string, count int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@e1 POSTHOOK: query: -- HIVE-3849 Aliased column in where clause for multi-groupby single reducer cannot be resolved + +-- SORT_QUERY_RESULTS + create table e1 (key string, count int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default Index: ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out (working copy) @@ -1888,7 +1888,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -2241,7 +2241,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -3259,7 +3259,7 @@ columns _col0,_col1,_col2,_col3,_col4 columns.types string:bigint:string:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out (working copy) @@ -1960,7 +1960,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -2313,7 +2313,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -3350,7 +3350,7 @@ columns _col0,_col1,_col2,_col3,_col4 columns.types string:bigint:string:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/join34.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/join34.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/join34.q.out (working copy) @@ -40,7 +40,7 @@ TOK_FROM TOK_JOIN TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF Index: ql/src/test/results/clientpositive/spark/join35.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/join35.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/join35.q.out (working copy) @@ -40,7 +40,7 @@ TOK_FROM TOK_JOIN TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF Index: ql/src/test/results/clientpositive/spark/join_filters_overlap.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/join_filters_overlap.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/join_filters_overlap.q.out (working copy) @@ -337,7 +337,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -696,7 +696,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1069,7 +1069,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1523,7 +1523,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7 columns.types int:int:int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1971,7 +1971,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7 columns.types int:int:int:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/load_dyn_part14.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/load_dyn_part14.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/load_dyn_part14.q.out (working copy) @@ -36,21 +36,21 @@ PREHOOK: query: explain insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T PREHOOK: type: QUERY POSTHOOK: query: explain insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -175,22 +175,22 @@ PREHOOK: query: insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@nzhang_part14 POSTHOOK: query: insert overwrite table nzhang_part14 partition(value) select key, value from ( - select 'k1' as key, cast(null as string) as value from src limit 2 + select * from (select 'k1' as key, cast(null as string) as value from src limit 2)a union all - select 'k2' as key, '' as value from src limit 2 + select * from (select 'k2' as key, '' as value from src limit 2)b union all - select 'k3' as key, ' ' as value from src limit 2 + select * from (select 'k3' as key, ' ' as value from src limit 2)c ) T POSTHOOK: type: QUERY POSTHOOK: Input: default@src Index: ql/src/test/results/clientpositive/spark/louter_join_ppr.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/louter_join_ppr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/louter_join_ppr.q.out (working copy) @@ -331,7 +331,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -811,7 +811,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1290,7 +1290,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1675,7 +1675,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/mapjoin_mapjoin.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/mapjoin_mapjoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/mapjoin_mapjoin.q.out (working copy) @@ -264,7 +264,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/optimize_nullscan.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/optimize_nullscan.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/optimize_nullscan.q.out (working copy) @@ -134,7 +134,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -535,7 +535,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -595,7 +595,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -952,7 +952,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -979,7 +979,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1383,7 +1383,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1443,7 +1443,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -1532,7 +1532,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1674,7 +1674,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1910,7 +1910,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/outer_join_ppr.q.java1.7.out =================================================================== --- ql/src/test/results/clientpositive/spark/outer_join_ppr.q.java1.7.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/outer_join_ppr.q.java1.7.out (working copy) @@ -424,7 +424,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -895,7 +895,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/outer_join_ppr.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/outer_join_ppr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/outer_join_ppr.q.out (working copy) @@ -422,7 +422,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -893,7 +893,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/parallel_join0.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/parallel_join0.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/parallel_join0.q.out (working copy) @@ -1,5 +1,5 @@ Warning: Shuffle Join JOIN[8][tables = [src1, src2]] in Work 'Reducer 2' is a cross product -PREHOOK: query: -- SORT_BEFORE_DIFF +PREHOOK: query: -- SORT_QUERY_RESULTS EXPLAIN SELECT src1.key as k1, src1.value as v1, @@ -9,7 +9,7 @@ (SELECT * FROM src WHERE src.key < 10) src2 SORT BY k1, v1, k2, v2 PREHOOK: type: QUERY -POSTHOOK: query: -- SORT_BEFORE_DIFF +POSTHOOK: query: -- SORT_QUERY_RESULTS EXPLAIN SELECT src1.key as k1, src1.value as v1, Index: ql/src/test/results/clientpositive/spark/pcr.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/pcr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/pcr.q.out (working copy) @@ -241,7 +241,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -510,7 +510,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -781,7 +781,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1022,7 +1022,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1311,7 +1311,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1610,7 +1610,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1862,7 +1862,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2073,7 +2073,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2370,7 +2370,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2663,7 +2663,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2936,7 +2936,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:string:string:int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3244,7 +3244,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:string:string:int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3620,7 +3620,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3943,7 +3943,7 @@ columns _col0,_col1,_col2 columns.types int:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4717,7 +4717,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4950,7 +4950,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5179,7 +5179,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/ppd_join_filter.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/ppd_join_filter.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/ppd_join_filter.q.out (working copy) @@ -324,7 +324,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -698,7 +698,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1072,7 +1072,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1446,7 +1446,7 @@ columns _col0,_col1,_col2 columns.types string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/ptf.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/ptf.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/ptf.q.out (working copy) @@ -54,6 +54,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -68,6 +80,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -214,6 +258,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -228,6 +284,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -330,10 +405,22 @@ Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col3 (type: int) - outputColumnNames: _col2, _col1, _col5 + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int) + outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -445,6 +532,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -459,6 +558,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -578,6 +709,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -592,6 +735,39 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), (_col5 - _wcol2) (type: int) @@ -714,6 +890,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -749,6 +937,39 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col2, 1, _col2 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col2 (type: int), (_col2 - _wcol2) (type: int) @@ -876,6 +1097,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1034,6 +1267,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1129,7 +1374,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -1142,6 +1401,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -1155,6 +1427,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -1254,7 +1545,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -1268,6 +1573,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1282,6 +1600,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1397,6 +1747,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1411,6 +1773,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1484,7 +1878,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) PREHOOK: type: QUERY POSTHOOK: query: -- 12. testFunctionChain @@ -1495,7 +1889,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -1517,34 +1911,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: -+ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: -+ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1559,6 +2006,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1584,7 +2063,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) PREHOOK: type: QUERY PREHOOK: Input: default@part @@ -1595,7 +2074,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) POSTHOOK: type: QUERY POSTHOOK: Input: default@part @@ -1684,6 +2163,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1698,6 +2189,30 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol1 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: bigint), _wcol1 (type: double) @@ -1841,6 +2356,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1874,6 +2401,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -1991,6 +2563,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -2110,9 +2694,8 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (GROUP, 2) + Reducer 2 <- Map 1 (GROUP PARTITION-LEVEL SORT, 2) Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -2133,7 +2716,7 @@ Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) sort order: ++ - Map-reduce partition columns: _col0 (type: string), _col1 (type: string) + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double) Reducer 2 @@ -2144,19 +2727,19 @@ mode: mergepartial outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE - value expressions: _col1 (type: string), _col2 (type: double) - Reducer 3 - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1 (type: double) - outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: mfgr_price_view + output shape: _col0: string, _col1: string, _col2: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col0 + output shape: _col0: string, _col1: string, _col2: double + partition by: _col0 + raw input shape: Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) @@ -2164,13 +2747,31 @@ Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double) - Reducer 4 + Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col0 (type: double) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: double), _wcol0 (type: double) @@ -2340,6 +2941,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -2360,20 +2993,81 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(5)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ - Map-reduce partition columns: _col2 (type: string) + Select Operator + expressions: _col1 (type: string), _col2 (type: string), _col5 (type: int), _wcol0 (type: bigint) + outputColumnNames: _col1, _col2, _col5, _wcol0 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE - value expressions: _wcol0 (type: bigint), _col5 (type: int) + Reduce Output Operator + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ + Map-reduce partition columns: _col2 (type: string) + Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + value expressions: _wcol0 (type: bigint), _col5 (type: int) Reducer 5 Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) + expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) outputColumnNames: _col0, _col2, _col3, _col6 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: bigint, _col2: string, _col3: string, _col6: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col3, _col2 + partition by: _col3 + raw input shape: + window functions: + window function definition + alias: _wcol1 + arguments: _col3, _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col3, _col2 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol3 + arguments: _col3, _col2 + name: cume_dist + window function: GenericUDAFCumeDistEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol4 + arguments: _col6, true + name: first_value + window function: GenericUDAFFirstValueEvaluator + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col3 (type: string), _col2 (type: string), _col6 (type: int), UDFToInteger(round(_col0, 1)) (type: int), _wcol1 (type: int), _wcol2 (type: int), _wcol3 (type: double), _wcol4 (type: int) @@ -2394,6 +3088,16 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -2408,6 +3112,16 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col5 (type: int) @@ -2604,9 +3318,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -2615,8 +3329,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -2627,12 +3341,52 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2643,10 +3397,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2657,6 +3431,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -2782,9 +3588,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) Reducer 5 <- Reducer 4 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -2794,8 +3600,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -2806,10 +3612,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2820,10 +3645,22 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -2834,6 +3671,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -2848,6 +3697,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -2969,8 +3850,8 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -2980,8 +3861,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -2992,10 +3873,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3006,6 +3906,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3020,6 +3939,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -3141,10 +4092,10 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) - Reducer 5 <- Reducer 4 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) + Reducer 5 <- Reducer 4 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -3153,8 +4104,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3165,10 +4116,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3179,12 +4149,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3195,10 +4191,23 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3209,6 +4218,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -3332,9 +4373,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -3343,8 +4384,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3355,12 +4396,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3371,10 +4445,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3385,6 +4479,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -3502,8 +4628,8 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -3513,8 +4639,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3525,12 +4651,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3541,6 +4700,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3555,6 +4727,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) Index: ql/src/test/results/clientpositive/spark/ptf_matchpath.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/ptf_matchpath.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/ptf_matchpath.q.out (working copy) @@ -37,6 +37,7 @@ PREHOOK: query: -- SORT_QUERY_RESULTS -- 1. basic Matchpath test +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -47,11 +48,10 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) PREHOOK: type: QUERY -PREHOOK: Input: default@flights_tiny -#### A masked pattern was here #### POSTHOOK: query: -- SORT_QUERY_RESULTS -- 1. basic Matchpath test +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -62,6 +62,90 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-1 + Spark + Edges: + Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: flights_tiny + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + key expressions: fl_num (type: string), year (type: int), month (type: int), day_of_month (type: int) + sort order: ++++ + Map-reduce partition columns: fl_num (type: string) + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + value expressions: origin_city_name (type: string), arr_delay (type: float) + Reducer 2 + Reduce Operator Tree: + Select Operator + expressions: VALUE._col0 (type: string), KEY.reducesinkkey1 (type: int), KEY.reducesinkkey2 (type: int), KEY.reducesinkkey3 (type: int), VALUE._col2 (type: float), KEY.reducesinkkey0 (type: string) + outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + PTF Operator + Function definitions: + Input definition + input alias: flights_tiny + output shape: + type: TABLE + Partition table definition + input alias: ptf_1 + arguments: 'LATE.LATE+', 'LATE', (_col5 > 15.0), 'origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath' + name: matchpath + order by: _col2, _col3, _col4 + output shape: tpath: int + partition by: _col6 + raw input shape: + referenced columns: _col0, _col6, _col2, _col3, _col4, tpath, _col5 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), fl_num (type: string), year (type: int), month (type: int), day_of_month (type: int), sz (type: int), tpath (type: int) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + distribute by fl_num + sort by year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +PREHOOK: type: QUERY +PREHOOK: Input: default@flights_tiny +#### A masked pattern was here #### +POSTHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + distribute by fl_num + sort by year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +POSTHOOK: type: QUERY POSTHOOK: Input: default@flights_tiny #### A masked pattern was here #### Baltimore 1142 2010 10 20 6 20 @@ -81,6 +165,7 @@ Chicago 897 2010 10 22 2 22 Washington 7291 2010 10 27 2 27 PREHOOK: query: -- 2. Matchpath on 1 partition +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -91,9 +176,8 @@ ) where fl_num = 1142 PREHOOK: type: QUERY -PREHOOK: Input: default@flights_tiny -#### A masked pattern was here #### POSTHOOK: query: -- 2. Matchpath on 1 partition +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on flights_tiny @@ -104,6 +188,93 @@ ) where fl_num = 1142 POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-1 + Spark + Edges: + Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: flights_tiny + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + key expressions: 0 (type: int), fl_num (type: string), year (type: int), month (type: int), day_of_month (type: int) + sort order: +++++ + Map-reduce partition columns: 0 (type: int) + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + value expressions: origin_city_name (type: string), arr_delay (type: float) + Reducer 2 + Reduce Operator Tree: + Select Operator + expressions: VALUE._col0 (type: string), KEY.reducesinkkey2 (type: int), KEY.reducesinkkey3 (type: int), KEY.reducesinkkey4 (type: int), VALUE._col2 (type: float), KEY.reducesinkkey1 (type: string) + outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + PTF Operator + Function definitions: + Input definition + input alias: flights_tiny + output shape: + type: TABLE + Partition table definition + input alias: ptf_1 + arguments: 'LATE.LATE+', 'LATE', (_col5 > 15.0), 'origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath' + name: matchpath + order by: _col6, _col2, _col3, _col4 + output shape: tpath: int + partition by: 0 + raw input shape: + referenced columns: _col0, _col6, _col2, _col3, _col4, tpath, _col5 + Statistics: Num rows: 24 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (fl_num = 1142) (type: boolean) + Statistics: Num rows: 12 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), '1142' (type: string), year (type: int), month (type: int), day_of_month (type: int), sz (type: int), tpath (type: int) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 12 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 12 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +where fl_num = 1142 +PREHOOK: type: QUERY +PREHOOK: Input: default@flights_tiny +#### A masked pattern was here #### +POSTHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +where fl_num = 1142 +POSTHOOK: type: QUERY POSTHOOK: Input: default@flights_tiny #### A masked pattern was here #### Baltimore 1142 2010 10 20 6 20 @@ -112,6 +283,7 @@ Baltimore 1142 2010 10 25 3 25 Baltimore 1142 2010 10 26 2 26 PREHOOK: query: -- 3. empty partition. +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on (select * from flights_tiny where fl_num = -1142) flights_tiny @@ -121,9 +293,8 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) PREHOOK: type: QUERY -PREHOOK: Input: default@flights_tiny -#### A masked pattern was here #### POSTHOOK: query: -- 3. empty partition. +explain select origin_city_name, fl_num, year, month, day_of_month, sz, tpath from matchpath(on (select * from flights_tiny where fl_num = -1142) flights_tiny @@ -133,5 +304,94 @@ arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') ) POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 + +STAGE PLANS: + Stage: Stage-1 + Spark + Edges: + Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) +#### A masked pattern was here #### + Vertices: + Map 1 + Map Operator Tree: + TableScan + alias: flights_tiny + Statistics: Num rows: 44 Data size: 5379 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: (fl_num = -1142) (type: boolean) + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), year (type: int), month (type: int), day_of_month (type: int), arr_delay (type: float) + outputColumnNames: _col0, _col2, _col3, _col4, _col5 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Reduce Output Operator + key expressions: 0 (type: int), '-1142' (type: string), _col2 (type: int), _col3 (type: int), _col4 (type: int) + sort order: +++++ + Map-reduce partition columns: 0 (type: int) + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + value expressions: _col0 (type: string), _col5 (type: float) + Reducer 2 + Reduce Operator Tree: + Select Operator + expressions: VALUE._col0 (type: string), KEY.reducesinkkey2 (type: int), KEY.reducesinkkey3 (type: int), KEY.reducesinkkey4 (type: int), VALUE._col2 (type: float), '-1142' (type: string) + outputColumnNames: _col0, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + PTF Operator + Function definitions: + Input definition + input alias: flights_tiny + output shape: + type: SUBQUERY + Partition table definition + input alias: ptf_1 + arguments: 'LATE.LATE+', 'LATE', (_col5 > 15.0), 'origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath' + name: matchpath + order by: _col6, _col2, _col3, _col4 + output shape: tpath: int + partition by: 0 + raw input shape: + referenced columns: _col0, _col6, _col2, _col3, _col4, tpath, _col5 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: origin_city_name (type: string), '-1142' (type: string), year (type: int), month (type: int), day_of_month (type: int), sz (type: int), tpath (type: int) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 22 Data size: 2689 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + ListSink + +PREHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + (select * from flights_tiny where fl_num = -1142) flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +PREHOOK: type: QUERY +PREHOOK: Input: default@flights_tiny +#### A masked pattern was here #### +POSTHOOK: query: select origin_city_name, fl_num, year, month, day_of_month, sz, tpath +from matchpath(on + (select * from flights_tiny where fl_num = -1142) flights_tiny + sort by fl_num, year, month, day_of_month + arg1('LATE.LATE+'), + arg2('LATE'), arg3(arr_delay > 15), + arg4('origin_city_name, fl_num, year, month, day_of_month, size(tpath) as sz, tpath[0].day_of_month as tpath') + ) +POSTHOOK: type: QUERY POSTHOOK: Input: default@flights_tiny #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/spark/ptf_streaming.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/ptf_streaming.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/ptf_streaming.q.out (working copy) @@ -54,6 +54,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -68,6 +80,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -214,6 +258,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -228,6 +284,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -350,6 +425,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noopstreaming + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -462,7 +549,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -475,6 +576,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -488,6 +602,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -587,7 +720,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -601,6 +748,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -615,6 +775,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -721,34 +913,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -763,6 +1008,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -871,34 +1148,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -913,6 +1243,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1021,34 +1383,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1063,6 +1478,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1200,6 +1647,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noopstreaming + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1233,6 +1692,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -1350,9 +1854,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -1361,8 +1865,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -1373,12 +1877,52 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1389,10 +1933,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1403,6 +1967,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -1528,9 +2124,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) Reducer 5 <- Reducer 4 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -1540,8 +2136,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -1552,10 +2148,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1566,10 +2181,22 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -1580,6 +2207,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1594,6 +2233,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -1713,8 +2384,8 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -1724,8 +2395,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -1736,12 +2407,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1752,6 +2456,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1766,6 +2483,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) Index: ql/src/test/results/clientpositive/spark/router_join_ppr.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/router_join_ppr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/router_join_ppr.q.out (working copy) @@ -430,7 +430,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -815,7 +815,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1196,7 +1196,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1671,7 +1671,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/sample10.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sample10.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sample10.q.out (working copy) @@ -102,6 +102,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 4 Statistics: Num rows: 20 Data size: 120 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ds (type: string) @@ -358,7 +359,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/sample2.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sample2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sample2.q.out (working copy) @@ -60,6 +60,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) Index: ql/src/test/results/clientpositive/spark/sample4.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sample4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sample4.q.out (working copy) @@ -62,6 +62,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) Index: ql/src/test/results/clientpositive/spark/sample6.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sample6.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sample6.q.out (working copy) @@ -60,6 +60,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 4 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -495,6 +496,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 3) (type: boolean) + sampleDesc: BUCKET 4 OUT OF 4 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -578,7 +580,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -898,6 +900,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -981,7 +984,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1638,7 +1641,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2138,7 +2141,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2541,6 +2544,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -2671,7 +2675,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2874,6 +2878,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 4) = 1) (type: boolean) + sampleDesc: BUCKET 2 OUT OF 4 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) @@ -2957,7 +2962,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3097,7 +3102,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/sample8.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sample8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sample8.q.out (working copy) @@ -405,7 +405,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/sample9.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sample9.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sample9.q.out (working copy) @@ -58,6 +58,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 2) = 0) (type: boolean) + sampleDesc: BUCKET 1 OUT OF 2 Statistics: Num rows: 500 Data size: 5301 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string) Index: ql/src/test/results/clientpositive/spark/script_env_var1.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/script_env_var1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/script_env_var1.q.out (working copy) @@ -1,16 +1,16 @@ PREHOOK: query: -- Verifies that script operator ID environment variables have unique values -- in each instance of the script operator. SELECT count(1) FROM -( SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)x UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)y ) a GROUP BY key PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- Verifies that script operator ID environment variables have unique values -- in each instance of the script operator. SELECT count(1) FROM -( SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)x UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)y ) a GROUP BY key POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/spark/script_env_var2.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/script_env_var2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/script_env_var2.q.out (working copy) @@ -1,14 +1,14 @@ PREHOOK: query: -- Same test as script_env_var1, but test setting the variable name SELECT count(1) FROM -( SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)a UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)b ) a GROUP BY key PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- Same test as script_env_var1, but test setting the variable name SELECT count(1) FROM -( SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)a UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)b ) a GROUP BY key POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/spark/smb_mapjoin_11.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/smb_mapjoin_11.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/smb_mapjoin_11.q.out (working copy) @@ -1916,6 +1916,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 16) = 1) (type: boolean) + sampleDesc: BUCKET 2 OUT OF 16 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string), ds (type: string) @@ -2007,6 +2008,7 @@ Filter Operator isSamplingPred: true predicate: (((hash(key) & 2147483647) % 16) = 1) (type: boolean) + sampleDesc: BUCKET 2 OUT OF 16 Statistics: Num rows: 514 Data size: 5484 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: key (type: int), value (type: string), ds (type: string) Index: ql/src/test/results/clientpositive/spark/smb_mapjoin_13.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/smb_mapjoin_13.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/smb_mapjoin_13.q.out (working copy) @@ -238,7 +238,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -524,7 +524,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/smb_mapjoin_15.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/smb_mapjoin_15.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/smb_mapjoin_15.q.out (working copy) @@ -210,7 +210,7 @@ columns _col0,_col1,_col2,_col3 columns.types int:string:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -478,7 +478,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:string:int:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -694,7 +694,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:string:int:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1001,7 +1001,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types int:int:string:int:int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/sort_merge_join_desc_5.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sort_merge_join_desc_5.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sort_merge_join_desc_5.q.out (working copy) @@ -230,7 +230,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/sort_merge_join_desc_6.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sort_merge_join_desc_6.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sort_merge_join_desc_6.q.out (working copy) @@ -320,7 +320,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/sort_merge_join_desc_7.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/sort_merge_join_desc_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/sort_merge_join_desc_7.q.out (working copy) @@ -440,7 +440,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/subquery_in.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/subquery_in.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/subquery_in.q.out (working copy) @@ -314,6 +314,25 @@ outputColumnNames: _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -463,6 +482,25 @@ outputColumnNames: _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((_wcol0 <= 2) and _col2 is not null) (type: boolean) Index: ql/src/test/results/clientpositive/spark/transform_ppr1.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/transform_ppr1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/transform_ppr1.q.out (working copy) @@ -354,7 +354,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/transform_ppr2.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/transform_ppr2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/transform_ppr2.q.out (working copy) @@ -262,7 +262,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/union3.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/union3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/union3.q.out (working copy) @@ -1,41 +1,39 @@ -PREHOOK: query: -- SORT_BEFORE_DIFF +PREHOOK: query: -- SORT_QUERY_RESULTS explain SELECT * FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a PREHOOK: type: QUERY -POSTHOOK: query: -- SORT_BEFORE_DIFF +POSTHOOK: query: -- SORT_QUERY_RESULTS explain SELECT * FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -46,12 +44,11 @@ Stage: Stage-1 Spark Edges: - Reducer 10 <- Map 9 (GROUP, 1) Reducer 2 <- Map 1 (GROUP, 1) Reducer 5 <- Map 4 (GROUP, 1) - Reducer 8 <- Map 7 (GROUP, 1) - Reducer 3 <- Reducer 2 (SORT, 2) - Reducer 6 <- Reducer 5 (SORT, 2) + Reducer 7 <- Map 6 (GROUP, 1) + Reducer 9 <- Map 8 (GROUP, 1) + Reducer 3 <- Reducer 2 (SORT, 2), Reducer 5 (SORT, 2), Reducer 7 (SORT, 2), Reducer 9 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -80,7 +77,7 @@ Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Map 7 + Map 6 Map Operator Tree: TableScan alias: src @@ -93,7 +90,7 @@ Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Map 9 + Map 8 Map Operator Tree: TableScan alias: src @@ -106,36 +103,26 @@ Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reducer 10 + Reducer 2 Reduce Operator Tree: Limit Number of rows: 1 Select Operator - expressions: 4 (type: int) + expressions: 1 (type: int) outputColumnNames: _col0 - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - Reducer 2 - Reduce Operator Tree: - Limit - Number of rows: 1 - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reduce Output Operator - key expressions: 1 (type: int) - sort order: + - Map-reduce partition columns: 1 (type: int) - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: int) outputColumnNames: _col0 + Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE File Output Operator compressed: false + Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -144,36 +131,35 @@ Reduce Operator Tree: Limit Number of rows: 1 - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reduce Output Operator - key expressions: 2 (type: int) - sort order: + - Map-reduce partition columns: 2 (type: int) - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reducer 6 + Select Operator + expressions: 2 (type: int) + outputColumnNames: _col0 + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) + Reducer 7 Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: int) - outputColumnNames: _col0 - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - Reducer 8 - Reduce Operator Tree: Limit Number of rows: 1 Select Operator expressions: 3 (type: int) outputColumnNames: _col0 - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) + Reducer 9 + Reduce Operator Tree: + Limit + Number of rows: 1 + Select Operator + expressions: 4 (type: int) + outputColumnNames: _col0 + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) Stage: Stage-0 Fetch Operator @@ -194,17 +180,16 @@ FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a PREHOOK: type: QUERY PREHOOK: Input: default@src @@ -214,31 +199,30 @@ FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@union_out POSTHOOK: Lineage: union_out.id EXPRESSION [] -PREHOOK: query: select * from union_out cluster by id +PREHOOK: query: select * from union_out PREHOOK: type: QUERY PREHOOK: Input: default@union_out #### A masked pattern was here #### -POSTHOOK: query: select * from union_out cluster by id +POSTHOOK: query: select * from union_out POSTHOOK: type: QUERY POSTHOOK: Input: default@union_out #### A masked pattern was here #### -4 1 2 3 +4 Index: ql/src/test/results/clientpositive/spark/union4.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/union4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/union4.q.out (working copy) @@ -1,13 +1,17 @@ -PREHOOK: query: -- union case: both subqueries are map-reduce jobs on same input, followed by filesink +PREHOOK: query: -- SORT_QUERY_RESULTS +-- union case: both subqueries are map-reduce jobs on same input, followed by filesink + create table tmptable(key string, value int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@tmptable -POSTHOOK: query: -- union case: both subqueries are map-reduce jobs on same input, followed by filesink +POSTHOOK: query: -- SORT_QUERY_RESULTS +-- union case: both subqueries are map-reduce jobs on same input, followed by filesink + create table tmptable(key string, value int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default Index: ql/src/test/results/clientpositive/spark/union_null.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/union_null.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/union_null.q.out (working copy) @@ -1,34 +1,34 @@ PREHOOK: query: -- SORT_BEFORE_DIFF -- HIVE-2901 -select x from (select value as x from src order by x limit 5 union all select NULL as x from src limit 5) a +select x from (select * from (select value as x from src order by x limit 5)a union all select * from (select NULL as x from src limit 5)b )a PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- SORT_BEFORE_DIFF -- HIVE-2901 -select x from (select value as x from src order by x limit 5 union all select NULL as x from src limit 5) a +select x from (select * from (select value as x from src order by x limit 5)a union all select * from (select NULL as x from src limit 5)b )a POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### +val_0 +val_0 +val_0 +val_10 +val_100 NULL NULL NULL NULL NULL -val_0 -val_0 -val_0 -val_10 -val_100 PREHOOK: query: -- HIVE-4837 -select * from (select null as N from src1 group by key UNION ALL select null as N from src1 group by key ) a +select * from (select * from (select null as N from src1 group by key)a UNION ALL select * from (select null as N from src1 group by key)b ) a PREHOOK: type: QUERY PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: -- HIVE-4837 -select * from (select null as N from src1 group by key UNION ALL select null as N from src1 group by key ) a +select * from (select * from (select null as N from src1 group by key)a UNION ALL select * from (select null as N from src1 group by key)b ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@src1 #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/spark/union_ppr.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/union_ppr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/union_ppr.q.out (working copy) @@ -25,7 +25,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -374,7 +374,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/spark/union_remove_25.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/union_remove_25.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/union_remove_25.q.out (working copy) @@ -258,18 +258,18 @@ insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a PREHOOK: type: QUERY POSTHOOK: query: explain insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -366,9 +366,9 @@ PREHOOK: query: insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a PREHOOK: type: QUERY PREHOOK: Input: default@srcpart @@ -378,9 +378,9 @@ POSTHOOK: query: insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart @@ -440,17 +440,17 @@ PREHOOK: query: explain insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a PREHOOK: type: QUERY POSTHOOK: query: explain insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -548,9 +548,9 @@ PREHOOK: query: insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a PREHOOK: type: QUERY PREHOOK: Input: default@srcpart @@ -560,9 +560,9 @@ POSTHOOK: query: insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart Index: ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out =================================================================== --- ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out (working copy) @@ -304,6 +304,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -321,6 +333,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -340,7 +384,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -681,6 +725,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -698,6 +754,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -717,7 +792,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -903,10 +978,22 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col3 (type: int) - outputColumnNames: _col2, _col1, _col5 + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int) + outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -926,7 +1013,7 @@ columns _col0,_col1,_col2 columns.types string:string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1167,6 +1254,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1184,6 +1283,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1203,7 +1334,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1456,6 +1587,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1473,6 +1616,39 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), (_col5 - _wcol2) (type: int) @@ -1492,7 +1668,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1755,6 +1931,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -1797,6 +1985,39 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col2, 1, _col2 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col2 (type: int), (_col2 - _wcol2) (type: int) @@ -1816,7 +2037,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2091,6 +2312,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -2129,7 +2362,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types int:string:string:string:string:int:string:double:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2420,7 +2653,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types int:string:string:string:string:int:string:double:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2435,6 +2668,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -2588,7 +2833,21 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -2653,6 +2912,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -2669,6 +2941,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -2688,7 +2979,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2863,7 +3154,21 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -2929,6 +3234,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -2946,6 +3264,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -2965,7 +3315,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3209,6 +3559,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3226,6 +3588,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -3245,7 +3639,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3434,8 +3828,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -3495,16 +3889,49 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -3514,10 +3941,30 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3535,6 +3982,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -3554,7 +4033,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3828,6 +4307,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3845,6 +4336,30 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol1 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: bigint), _wcol1 (type: double) @@ -3864,7 +4379,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:bigint:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4282,6 +4797,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -4322,6 +4849,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -4341,7 +4913,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types string:string:int:int:bigint:double:double:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4542,6 +5114,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -4581,7 +5165,7 @@ columns _col0,_col1,_col2 columns.types string:string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4736,9 +5320,8 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (GROUP, 2) + Reducer 2 <- Map 1 (GROUP PARTITION-LEVEL SORT, 2) Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -4760,7 +5343,7 @@ Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) sort order: ++ - Map-reduce partition columns: _col0 (type: string), _col1 (type: string) + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 value expressions: _col2 (type: double) @@ -4824,23 +5407,19 @@ mode: mergepartial outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE - tag: -1 - value expressions: _col1 (type: string), _col2 (type: double) - auto parallelism: false - Execution mode: vectorized - Reducer 3 - Needs Tagging: false - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1 (type: double) - outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: mfgr_price_view + output shape: _col0: string, _col1: string, _col2: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col0 + output shape: _col0: string, _col1: string, _col2: double + partition by: _col0 + raw input shape: Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) @@ -4850,7 +5429,7 @@ tag: -1 value expressions: _col2 (type: double) auto parallelism: false - Reducer 4 + Reducer 3 Needs Tagging: false Reduce Operator Tree: Select Operator @@ -4858,6 +5437,24 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~ Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: double), _wcol0 (type: double) @@ -4877,7 +5474,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5288,6 +5885,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -5327,23 +5956,84 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(5)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ - Map-reduce partition columns: _col2 (type: string) + Select Operator + expressions: _col1 (type: string), _col2 (type: string), _col5 (type: int), _wcol0 (type: bigint) + outputColumnNames: _col1, _col2, _col5, _wcol0 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE - tag: -1 - value expressions: _wcol0 (type: bigint), _col5 (type: int) - auto parallelism: false + Reduce Output Operator + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ + Map-reduce partition columns: _col2 (type: string) + Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + tag: -1 + value expressions: _wcol0 (type: bigint), _col5 (type: int) + auto parallelism: false Reducer 5 Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) + expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) outputColumnNames: _col0, _col2, _col3, _col6 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: bigint, _col2: string, _col3: string, _col6: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col3, _col2 + partition by: _col3 + raw input shape: + window functions: + window function definition + alias: _wcol1 + arguments: _col3, _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col3, _col2 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol3 + arguments: _col3, _col2 + name: cume_dist + window function: GenericUDAFCumeDistEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol4 + arguments: _col6, true + name: first_value + window function: GenericUDAFFirstValueEvaluator + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col3 (type: string), _col2 (type: string), _col6 (type: int), UDFToInteger(round(_col0, 1)) (type: int), _wcol1 (type: int), _wcol2 (type: int), _wcol3 (type: double), _wcol4 (type: int) @@ -5383,6 +6073,16 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -5400,6 +6100,16 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col5 (type: int) @@ -5730,9 +6440,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -5742,8 +6452,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -5807,12 +6517,52 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -5826,10 +6576,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -5843,6 +6613,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -5862,7 +6664,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -6089,9 +6891,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) Reducer 5 <- Reducer 4 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -6102,8 +6904,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6167,10 +6969,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6184,10 +7005,22 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6201,6 +7034,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -6218,6 +7063,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -6237,7 +7114,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -6448,8 +7325,8 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -6460,8 +7337,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6525,10 +7402,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6542,6 +7438,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -6559,6 +7474,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -6578,7 +7525,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -6800,10 +7747,10 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) - Reducer 5 <- Reducer 4 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) + Reducer 5 <- Reducer 4 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -6813,8 +7760,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6878,10 +7825,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6895,12 +7861,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6914,10 +7906,23 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6931,6 +7936,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -6950,7 +7987,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -7204,9 +8241,9 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) - Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) + Reducer 4 <- Reducer 3 (SORT, 2) #### A masked pattern was here #### Vertices: Map 1 @@ -7216,8 +8253,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7281,12 +8318,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7300,10 +8370,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7317,6 +8407,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -7336,7 +8458,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:bigint:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -7556,8 +8678,8 @@ Stage: Stage-1 Spark Edges: - Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2) - Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2) + Reducer 2 <- Map 1 (SORT, 2) + Reducer 3 <- Reducer 2 (SORT, 2) Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2) #### A masked pattern was here #### Vertices: @@ -7568,8 +8690,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7633,12 +8755,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7652,6 +8807,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -7669,6 +8837,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -7688,7 +8888,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:bigint:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/subquery_in.q.out =================================================================== --- ql/src/test/results/clientpositive/subquery_in.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/subquery_in.q.out (working copy) @@ -269,6 +269,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -434,6 +453,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((_wcol0 <= 2) and _col0 is not null) (type: boolean) Index: ql/src/test/results/clientpositive/subquery_in_having.q.out =================================================================== --- ql/src/test/results/clientpositive/subquery_in_having.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/subquery_in_having.q.out (working copy) @@ -1348,6 +1348,24 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 15 Data size: 3173 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2 + partition by: _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col0 + name: first_value + window function: GenericUDAFFirstValueEvaluator + window frame: PRECEDING(MAX)~ Statistics: Num rows: 15 Data size: 3173 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _wcol0 is not null (type: boolean) Index: ql/src/test/results/clientpositive/subquery_notin.q.out =================================================================== --- ql/src/test/results/clientpositive/subquery_notin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/subquery_notin.q.out (working copy) @@ -285,7 +285,7 @@ 199 val_199 199 val_199 2 val_2 -Warning: Shuffle Join JOIN[30][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product +Warning: Shuffle Join JOIN[32][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product PREHOOK: query: -- non agg, corr explain select p_mfgr, b.p_name, p_size @@ -337,6 +337,25 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2 + partition by: _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((_wcol0 <= 2) and (_col0 is null or _col1 is null)) (type: boolean) @@ -477,6 +496,25 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2 + partition by: _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -498,7 +536,7 @@ Processor Tree: ListSink -Warning: Shuffle Join JOIN[30][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product +Warning: Shuffle Join JOIN[32][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product PREHOOK: query: select p_mfgr, b.p_name, p_size from part b where b.p_name not in @@ -537,7 +575,7 @@ Manufacturer#5 almond antique blue firebrick mint 31 Manufacturer#5 almond aquamarine dodger light gainsboro 46 Manufacturer#5 almond azure blanched chiffon midnight 23 -Warning: Shuffle Join JOIN[45][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product +Warning: Shuffle Join JOIN[47][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product PREHOOK: query: -- agg, non corr explain select p_name, p_size @@ -589,6 +627,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -721,6 +778,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -783,7 +859,7 @@ Processor Tree: ListSink -Warning: Shuffle Join JOIN[45][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product +Warning: Shuffle Join JOIN[47][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product PREHOOK: query: select p_name, p_size from part where part.p_size not in @@ -830,7 +906,7 @@ almond aquamarine yellow dodger mint 7 almond azure aquamarine papaya violet 12 almond azure blanched chiffon midnight 23 -Warning: Shuffle Join JOIN[42][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product +Warning: Shuffle Join JOIN[44][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product PREHOOK: query: -- agg, corr explain select p_mfgr, p_name, p_size @@ -881,6 +957,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -1057,6 +1152,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -1112,7 +1226,7 @@ Processor Tree: ListSink -Warning: Shuffle Join JOIN[42][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product +Warning: Shuffle Join JOIN[44][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product PREHOOK: query: select p_mfgr, p_name, p_size from part b where b.p_size not in (select min(p_size) Index: ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out =================================================================== --- ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out (working copy) @@ -222,6 +222,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: ((_wcol0 <= 2) and _col0 is not null) (type: boolean) @@ -375,6 +394,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((_wcol0 <= 2) and _col0 is not null) (type: boolean) @@ -751,7 +789,7 @@ Processor Tree: ListSink -Warning: Shuffle Join JOIN[30][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product +Warning: Shuffle Join JOIN[32][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product PREHOOK: query: -- non agg, corr explain select p_mfgr, b.p_name, p_size @@ -803,6 +841,25 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2 + partition by: _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((_wcol0 <= 2) and (_col0 is null or _col1 is null)) (type: boolean) @@ -943,6 +1000,25 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2 + partition by: _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) Index: ql/src/test/results/clientpositive/subquery_views.q.out =================================================================== --- ql/src/test/results/clientpositive/subquery_views.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/subquery_views.q.out (working copy) @@ -97,10 +97,10 @@ alias: b Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator - predicate: ((value > 'val_11') and (key is null or (value is null or key is null))) (type: boolean) - Statistics: Num rows: 249 Data size: 2645 Basic stats: COMPLETE Column stats: NONE + predicate: ((value > 'val_11') and (key is null or value is null)) (type: boolean) + Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE Select Operator - Statistics: Num rows: 249 Data size: 2645 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash @@ -252,10 +252,10 @@ alias: b Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Filter Operator - predicate: ((value > 'val_11') and (key is null or (value is null or key is null))) (type: boolean) - Statistics: Num rows: 249 Data size: 2645 Basic stats: COMPLETE Column stats: NONE + predicate: ((value > 'val_11') and (key is null or value is null)) (type: boolean) + Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE Select Operator - Statistics: Num rows: 249 Data size: 2645 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash Index: ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out =================================================================== --- ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out (working copy) @@ -215,7 +215,7 @@ columns _col0,_col1,_col2 columns.types struct:struct:struct escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_1.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_1.q.out (working copy) @@ -379,7 +379,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -684,7 +684,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -989,7 +989,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_11.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_11.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_11.q.out (working copy) @@ -372,7 +372,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -682,7 +682,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -991,7 +991,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1435,7 +1435,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out (working copy) @@ -598,7 +598,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_2.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_2.q.out (working copy) @@ -359,7 +359,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -666,7 +666,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_3.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_3.q.out (working copy) @@ -359,7 +359,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -664,7 +664,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -969,7 +969,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_4.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_4.q.out (working copy) @@ -375,7 +375,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -680,7 +680,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -985,7 +985,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_5.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_5.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_5.q.out (working copy) @@ -269,7 +269,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -497,7 +497,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -739,7 +739,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_7.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_7.q.out (working copy) @@ -441,7 +441,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -797,7 +797,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1153,7 +1153,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/auto_sortmerge_join_8.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/auto_sortmerge_join_8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/auto_sortmerge_join_8.q.out (working copy) @@ -441,7 +441,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -797,7 +797,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1155,7 +1155,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/cbo_union.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/cbo_union.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/cbo_union.q.out (working copy) @@ -1,7 +1,7 @@ PREHOOK: query: -- SORT_QUERY_RESULTS -- 11. Union All -select * from cbo_t1 order by key, c_boolean, value, dt union all select * from cbo_t2 order by key, c_boolean, value, dt +select * from (select * from cbo_t1 order by key, c_boolean, value, dt)a union all select * from (select * from cbo_t2 order by key, c_boolean, value, dt)b PREHOOK: type: QUERY PREHOOK: Input: default@cbo_t1 PREHOOK: Input: default@cbo_t1@dt=2014 @@ -11,7 +11,7 @@ POSTHOOK: query: -- SORT_QUERY_RESULTS -- 11. Union All -select * from cbo_t1 order by key, c_boolean, value, dt union all select * from cbo_t2 order by key, c_boolean, value, dt +select * from (select * from cbo_t1 order by key, c_boolean, value, dt)a union all select * from (select * from cbo_t2 order by key, c_boolean, value, dt)b POSTHOOK: type: QUERY POSTHOOK: Input: default@cbo_t1 POSTHOOK: Input: default@cbo_t1@dt=2014 Index: ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out (working copy) @@ -1773,6 +1773,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 #### A masked pattern was here #### 1000 +Warning: Shuffle Join MERGEJOIN[24][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product PREHOOK: query: -- parent is reduce tasks EXPLAIN select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY @@ -1796,17 +1797,13 @@ Map Operator Tree: TableScan alias: srcpart - filterExpr: ds is not null (type: boolean) - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + filterExpr: (ds = '2008-04-08') (type: boolean) + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: ds (type: string) - outputColumnNames: _col0 - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col0 (type: string) - sort order: + - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + sort order: + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Map 4 Map Operator Tree: TableScan @@ -1833,9 +1830,9 @@ condition map: Inner Join 0 to 1 keys: - 0 _col0 (type: string) - 1 _col0 (type: string) - Statistics: Num rows: 2200 Data size: 23372 Basic stats: COMPLETE Column stats: NONE + 0 + 1 + Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash @@ -1866,26 +1863,11 @@ mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string) - sort order: + - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: _col0 (type: string) - outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Group By Operator - keys: _col0 (type: string) - mode: hash - outputColumnNames: _col0 + Reduce Output Operator + sort order: Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Dynamic Partitioning Event Operator - Target Input: srcpart - Partition key expr: ds - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Target column: ds - Target Vertex: Map 1 Stage: Stage-0 Fetch Operator @@ -1893,21 +1875,18 @@ Processor Tree: ListSink +Warning: Shuffle Join MERGEJOIN[24][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: default@srcpart PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### POSTHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### 1000 PREHOOK: query: select count(*) from srcpart where ds = '2008-04-08' @@ -4082,6 +4061,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 #### A masked pattern was here #### 1000 +Warning: Map Join MAPJOIN[24][bigTable=?] in task 'Map 1' is a cross product PREHOOK: query: -- parent is reduce tasks EXPLAIN select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY @@ -4105,21 +4085,19 @@ Map Operator Tree: TableScan alias: srcpart - filterExpr: ds is not null (type: boolean) - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + filterExpr: (ds = '2008-04-08') (type: boolean) + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: ds (type: string) - outputColumnNames: _col0 - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: - 0 _col0 (type: string) - 1 _col0 (type: string) + 0 + 1 input vertices: 1 Reducer 4 - Statistics: Num rows: 2200 Data size: 23372 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash @@ -4170,26 +4148,11 @@ mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string) - sort order: + - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: _col0 (type: string) - outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Group By Operator - keys: _col0 (type: string) - mode: hash - outputColumnNames: _col0 + Reduce Output Operator + sort order: Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Dynamic Partitioning Event Operator - Target Input: srcpart - Partition key expr: ds - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Target column: ds - Target Vertex: Map 1 Stage: Stage-0 Fetch Operator @@ -4197,21 +4160,18 @@ Processor Tree: ListSink +Warning: Map Join MAPJOIN[24][bigTable=?] in task 'Map 1' is a cross product PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: default@srcpart PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### POSTHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### 1000 PREHOOK: query: select count(*) from srcpart where ds = '2008-04-08' Index: ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out (working copy) @@ -398,7 +398,7 @@ columns _col0,_col1 columns.types int:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/limit_pushdown.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/limit_pushdown.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/limit_pushdown.q.out (working copy) @@ -381,21 +381,21 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: double) mode: hash outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double) sort order: + Map-reduce partition columns: _col0 (type: double) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Reducer 2 Reduce Operator Tree: @@ -403,13 +403,13 @@ keys: KEY._col0 (type: double) mode: mergepartial outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -470,22 +470,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cdouble (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(DISTINCT _col1) keys: _col0 (type: tinyint), _col1 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: double) sort order: ++ Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Reducer 2 Reduce Operator Tree: @@ -494,13 +494,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -563,22 +563,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cstring1 (type: string), cstring2 (type: string) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(DISTINCT _col1), count(DISTINCT _col2) keys: _col0 (type: tinyint), _col1 (type: string), _col2 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: string), _col2 (type: string) sort order: +++ Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Reducer 2 Reduce Operator Tree: @@ -587,13 +587,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out (working copy) @@ -192,7 +192,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/metadataonly1.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/metadataonly1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/metadataonly1.q.out (working copy) @@ -63,7 +63,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -214,7 +214,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -361,7 +361,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -508,7 +508,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -859,7 +859,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1164,7 +1164,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1410,7 +1410,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1600,7 +1600,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1977,7 +1977,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/optimize_nullscan.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/optimize_nullscan.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/optimize_nullscan.q.out (working copy) @@ -131,7 +131,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -533,7 +533,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -593,7 +593,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -950,7 +950,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -977,7 +977,7 @@ columns _col0 columns.types bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1384,7 +1384,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1444,7 +1444,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -1534,7 +1534,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1680,7 +1680,7 @@ columns _col0 columns.types string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1919,7 +1919,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/orc_analyze.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/orc_analyze.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/orc_analyze.q.out (working copy) @@ -71,6 +71,55 @@ POSTHOOK: Lineage: orc_create_people.salary SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:salary, type:decimal(10,0), comment:null), ] POSTHOOK: Lineage: orc_create_people.start_date SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:start_date, type:timestamp, comment:null), ] POSTHOOK: Lineage: orc_create_people.state SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:state, type:string, comment:null), ] +PREHOOK: query: analyze table orc_create_people compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +POSTHOOK: query: analyze table orc_create_people compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +PREHOOK: query: desc formatted orc_create_people +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp +state string + +# Detailed Table Information +Database: default +#### A masked pattern was here #### +Protect Mode: None +Retention: 0 +#### A masked pattern was here #### +Table Type: MANAGED_TABLE +Table Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 100 + rawDataSize 52600 + totalSize 3158 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people compute statistics partialscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people @@ -120,6 +169,55 @@ Sort Columns: [] Storage Desc Params: serialization.format 1 +PREHOOK: query: analyze table orc_create_people compute statistics noscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +POSTHOOK: query: analyze table orc_create_people compute statistics noscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +PREHOOK: query: desc formatted orc_create_people +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp +state string + +# Detailed Table Information +Database: default +#### A masked pattern was here #### +Protect Mode: None +Retention: 0 +#### A masked pattern was here #### +Table Type: MANAGED_TABLE +Table Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 100 + rawDataSize 52600 + totalSize 3158 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: drop table orc_create_people PREHOOK: type: DROPTABLE PREHOOK: Input: default@orc_create_people @@ -269,6 +367,108 @@ POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).last_name SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:last_name, type:string, comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).salary SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:salary, type:decimal(10,0), comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).start_date SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:start_date, type:timestamp, comment:null), ] +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people @@ -371,6 +571,108 @@ Sort Columns: [] Storage Desc Params: serialization.format 1 +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: drop table orc_create_people PREHOOK: type: DROPTABLE PREHOOK: Input: default@orc_create_people @@ -583,6 +885,108 @@ POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).last_name SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:last_name, type:string, comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).salary SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:salary, type:decimal(10,0), comment:null), ] POSTHOOK: Lineage: orc_create_people PARTITION(state=Or).start_date SIMPLE [(orc_create_people_staging)orc_create_people_staging.FieldSchema(name:start_date, type:timestamp, comment:null), ] +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people @@ -685,6 +1089,108 @@ Sort Columns: [Order(col:last_name, order:1)] Storage Desc Params: serialization.format 1 +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="Or") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Or") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Or] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 22050 + totalSize 2071 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: 4 +Bucket Columns: [first_name] +Sort Columns: [Order(col:last_name, order:1)] +Storage Desc Params: + serialization.format 1 PREHOOK: query: drop table orc_create_people PREHOOK: type: DROPTABLE PREHOOK: Input: default@orc_create_people @@ -946,6 +1452,214 @@ POSTHOOK: type: ALTERTABLE_FILEFORMAT POSTHOOK: Input: default@orc_create_people POSTHOOK: Output: default@orc_create_people +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=OH +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=OH +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="OH") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="OH") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [OH] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE false + numFiles 1 + numRows -1 + rawDataSize -1 + totalSize 5812 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe +InputFormat: org.apache.hadoop.mapred.TextInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan +PREHOOK: type: QUERY +PREHOOK: Input: default@orc_create_people +PREHOOK: Output: default@orc_create_people +PREHOOK: Output: default@orc_create_people@state=Ca +PREHOOK: Output: default@orc_create_people@state=OH +PREHOOK: Output: default@orc_create_people@state=Or +POSTHOOK: query: analyze table orc_create_people partition(state) compute statistics partialscan +POSTHOOK: type: QUERY +POSTHOOK: Input: default@orc_create_people +POSTHOOK: Output: default@orc_create_people +POSTHOOK: Output: default@orc_create_people@state=Ca +POSTHOOK: Output: default@orc_create_people@state=OH +POSTHOOK: Output: default@orc_create_people@state=Or +PREHOOK: query: desc formatted orc_create_people partition(state="Ca") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="Ca") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [Ca] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE true + numFiles 1 + numRows 50 + rawDataSize 21950 + totalSize 2055 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde +InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 +PREHOOK: query: desc formatted orc_create_people partition(state="OH") +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@orc_create_people +POSTHOOK: query: desc formatted orc_create_people partition(state="OH") +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@orc_create_people +# col_name data_type comment + +id int +first_name string +last_name string +address string +salary decimal(10,0) +start_date timestamp + +# Partition Information +# col_name data_type comment + +state string + +# Detailed Partition Information +Partition Value: [OH] +Database: default +Table: orc_create_people +#### A masked pattern was here #### +Protect Mode: None +#### A masked pattern was here #### +Partition Parameters: + COLUMN_STATS_ACCURATE false + numFiles 1 + numRows -1 + rawDataSize -1 + totalSize 5812 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe +InputFormat: org.apache.hadoop.mapred.TextInputFormat +OutputFormat: org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + serialization.format 1 PREHOOK: query: analyze table orc_create_people partition(state) compute statistics noscan PREHOOK: type: QUERY PREHOOK: Input: default@orc_create_people Index: ql/src/test/results/clientpositive/tez/ptf.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/ptf.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/ptf.q.out (working copy) @@ -54,6 +54,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -68,6 +80,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -214,6 +258,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -228,6 +284,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -330,10 +405,22 @@ Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col3 (type: int) - outputColumnNames: _col2, _col1, _col5 + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int) + outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -445,6 +532,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -459,6 +558,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -578,6 +709,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -592,6 +735,39 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), (_col5 - _wcol2) (type: int) @@ -714,6 +890,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -749,6 +937,39 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col2, 1, _col2 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col2 (type: int), (_col2 - _wcol2) (type: int) @@ -876,6 +1097,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1034,6 +1267,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1129,7 +1374,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -1142,6 +1401,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -1155,6 +1427,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -1254,7 +1545,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -1268,6 +1573,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1282,6 +1600,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1397,6 +1747,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1411,6 +1773,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1484,7 +1878,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) PREHOOK: type: QUERY POSTHOOK: query: -- 12. testFunctionChain @@ -1495,7 +1889,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -1517,34 +1911,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: -+ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: -+ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2(DESC), _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1559,6 +2006,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1584,7 +2063,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) PREHOOK: type: QUERY PREHOOK: Input: default@part @@ -1595,7 +2074,7 @@ sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 from noop(on noopwithmap(on noop(on part partition by p_mfgr -order by p_mfgr, p_name +order by p_mfgr DESC, p_name ))) POSTHOOK: type: QUERY POSTHOOK: Input: default@part @@ -1684,6 +2163,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1698,6 +2189,30 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol1 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: bigint), _wcol1 (type: double) @@ -1841,6 +2356,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1874,6 +2401,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -1991,6 +2563,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -2112,7 +2696,6 @@ Edges: Reducer 2 <- Map 1 (SIMPLE_EDGE) Reducer 3 <- Reducer 2 (SIMPLE_EDGE) - Reducer 4 <- Reducer 3 (SIMPLE_EDGE) #### A masked pattern was here #### Vertices: Map 1 @@ -2133,7 +2716,7 @@ Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) sort order: ++ - Map-reduce partition columns: _col0 (type: string), _col1 (type: string) + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double) Reducer 2 @@ -2144,19 +2727,19 @@ mode: mergepartial outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE - value expressions: _col1 (type: string), _col2 (type: double) - Reducer 3 - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1 (type: double) - outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: mfgr_price_view + output shape: _col0: string, _col1: string, _col2: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col0 + output shape: _col0: string, _col1: string, _col2: double + partition by: _col0 + raw input shape: Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) @@ -2164,13 +2747,31 @@ Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double) - Reducer 4 + Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col0 (type: double) outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: double), _wcol0 (type: double) @@ -2340,6 +2941,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -2360,6 +2973,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -2380,20 +3025,81 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(5)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ - Map-reduce partition columns: _col2 (type: string) + Select Operator + expressions: _col1 (type: string), _col2 (type: string), _col5 (type: int), _wcol0 (type: bigint) + outputColumnNames: _col1, _col2, _col5, _wcol0 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE - value expressions: _wcol0 (type: bigint), _col5 (type: int) + Reduce Output Operator + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ + Map-reduce partition columns: _col2 (type: string) + Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + value expressions: _wcol0 (type: bigint), _col5 (type: int) Reducer 5 Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) + expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) outputColumnNames: _col0, _col2, _col3, _col6 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: bigint, _col2: string, _col3: string, _col6: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col3, _col2 + partition by: _col3 + raw input shape: + window functions: + window function definition + alias: _wcol1 + arguments: _col3, _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col3, _col2 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol3 + arguments: _col3, _col2 + name: cume_dist + window function: GenericUDAFCumeDistEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol4 + arguments: _col6, true + name: first_value + window function: GenericUDAFFirstValueEvaluator + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col3 (type: string), _col2 (type: string), _col6 (type: int), UDFToInteger(round(_col0, 1)) (type: int), _wcol1 (type: int), _wcol2 (type: int), _wcol3 (type: double), _wcol4 (type: int) @@ -2610,8 +3316,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -2622,12 +3328,52 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2638,10 +3384,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2652,6 +3418,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -2789,8 +3587,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -2801,10 +3599,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -2815,10 +3632,22 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -2829,6 +3658,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -2843,6 +3684,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -2975,8 +3848,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -2987,10 +3860,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3001,6 +3893,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3015,6 +3926,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -3148,8 +4091,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3160,10 +4103,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3174,12 +4136,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3190,10 +4178,23 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3204,6 +4205,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -3338,8 +4371,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3350,12 +4383,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -3366,10 +4432,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3380,6 +4466,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -3508,8 +4626,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -3520,12 +4638,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -3536,6 +4687,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3550,6 +4714,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) Index: ql/src/test/results/clientpositive/tez/ptf_streaming.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/ptf_streaming.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/ptf_streaming.q.out (working copy) @@ -54,6 +54,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -68,6 +80,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -214,6 +258,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -228,6 +284,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -350,6 +425,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noopstreaming + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -462,7 +549,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -475,6 +576,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -488,6 +602,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -587,7 +720,21 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -601,6 +748,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -615,6 +775,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -721,34 +913,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -763,6 +1008,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -871,34 +1148,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -913,6 +1243,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1021,34 +1383,87 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int), p_retailprice (type: double) Reducer 2 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int), _col7 (type: double) Reducer 3 Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1063,6 +1478,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1200,6 +1647,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noopstreaming + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: _col0 is not null (type: boolean) @@ -1233,6 +1692,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -1361,8 +1865,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -1373,12 +1877,52 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1389,10 +1933,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1403,6 +1967,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -1540,8 +2136,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_name (type: string), p_size (type: int) @@ -1552,10 +2148,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1566,10 +2181,22 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string), _col5 (type: int) @@ -1580,6 +2207,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1594,6 +2233,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -1724,8 +2395,8 @@ alias: part Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: p_size (type: int) @@ -1736,12 +2407,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: int) @@ -1752,6 +2456,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmapstreaming + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1766,6 +2483,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) Index: ql/src/test/results/clientpositive/tez/script_env_var1.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/script_env_var1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/script_env_var1.q.out (working copy) @@ -1,16 +1,16 @@ PREHOOK: query: -- Verifies that script operator ID environment variables have unique values -- in each instance of the script operator. SELECT count(1) FROM -( SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)x UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)y ) a GROUP BY key PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- Verifies that script operator ID environment variables have unique values -- in each instance of the script operator. SELECT count(1) FROM -( SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)x UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $HIVE_SCRIPT_OPERATOR_ID') USING 'sh' AS key FROM src LIMIT 1)y ) a GROUP BY key POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/tez/script_env_var2.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/script_env_var2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/script_env_var2.q.out (working copy) @@ -1,14 +1,14 @@ PREHOOK: query: -- Same test as script_env_var1, but test setting the variable name SELECT count(1) FROM -( SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)a UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)b ) a GROUP BY key PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- Same test as script_env_var1, but test setting the variable name SELECT count(1) FROM -( SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 UNION ALL - SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1 ) a GROUP BY key +( SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)a UNION ALL + SELECT * FROM (SELECT TRANSFORM('echo $MY_ID') USING 'sh' AS key FROM src LIMIT 1)b ) a GROUP BY key POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/tez/subquery_in.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/subquery_in.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/subquery_in.q.out (working copy) @@ -326,6 +326,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 <= 2) (type: boolean) @@ -483,6 +502,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((_wcol0 <= 2) and _col0 is not null) (type: boolean) Index: ql/src/test/results/clientpositive/tez/transform_ppr1.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/transform_ppr1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/transform_ppr1.q.out (working copy) @@ -354,7 +354,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/transform_ppr2.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/transform_ppr2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/transform_ppr2.q.out (working copy) @@ -262,7 +262,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/union3.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/union3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/union3.q.out (working copy) @@ -1,41 +1,39 @@ -PREHOOK: query: -- SORT_BEFORE_DIFF +PREHOOK: query: -- SORT_QUERY_RESULTS explain SELECT * FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a PREHOOK: type: QUERY -POSTHOOK: query: -- SORT_BEFORE_DIFF +POSTHOOK: query: -- SORT_QUERY_RESULTS explain SELECT * FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -46,12 +44,11 @@ Stage: Stage-1 Tez Edges: - Reducer 11 <- Map 10 (SIMPLE_EDGE), Union 4 (CONTAINS) - Reducer 2 <- Map 1 (SIMPLE_EDGE) - Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Union 4 (CONTAINS) - Reducer 6 <- Map 5 (SIMPLE_EDGE) - Reducer 7 <- Reducer 6 (SIMPLE_EDGE), Union 4 (CONTAINS) - Reducer 9 <- Map 8 (SIMPLE_EDGE), Union 4 (CONTAINS) + Reducer 10 <- Map 9 (SIMPLE_EDGE), Union 3 (CONTAINS) + Reducer 2 <- Map 1 (SIMPLE_EDGE), Union 3 (CONTAINS) + Reducer 4 <- Union 3 (SIMPLE_EDGE) + Reducer 6 <- Map 5 (SIMPLE_EDGE), Union 3 (CONTAINS) + Reducer 8 <- Map 7 (SIMPLE_EDGE), Union 3 (CONTAINS) #### A masked pattern was here #### Vertices: Map 1 @@ -67,7 +64,7 @@ Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Map 10 + Map 5 Map Operator Tree: TableScan alias: src @@ -80,7 +77,7 @@ Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Map 5 + Map 7 Map Operator Tree: TableScan alias: src @@ -93,7 +90,7 @@ Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Map 8 + Map 9 Map Operator Tree: TableScan alias: src @@ -106,36 +103,37 @@ Reduce Output Operator sort order: Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reducer 11 + Reducer 10 Reduce Operator Tree: Limit Number of rows: 1 Select Operator expressions: 4 (type: int) outputColumnNames: _col0 - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) Reducer 2 Reduce Operator Tree: Limit Number of rows: 1 - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reduce Output Operator - key expressions: 1 (type: int) - sort order: + - Map-reduce partition columns: 1 (type: int) - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reducer 3 + Select Operator + expressions: 1 (type: int) + outputColumnNames: _col0 + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) + Reducer 4 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: int) outputColumnNames: _col0 + Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE File Output Operator compressed: false + Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -144,38 +142,26 @@ Reduce Operator Tree: Limit Number of rows: 1 - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reduce Output Operator - key expressions: 2 (type: int) - sort order: + - Map-reduce partition columns: 2 (type: int) - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reducer 7 + Select Operator + expressions: 2 (type: int) + outputColumnNames: _col0 + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) + Reducer 8 Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: int) - outputColumnNames: _col0 - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - Reducer 9 - Reduce Operator Tree: Limit Number of rows: 1 Select Operator expressions: 3 (type: int) outputColumnNames: _col0 - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - Union 4 - Vertex: Union 4 + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) + Union 3 + Vertex: Union 3 Stage: Stage-0 Fetch Operator @@ -196,17 +182,16 @@ FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a PREHOOK: type: QUERY PREHOOK: Input: default@src @@ -216,27 +201,26 @@ FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@union_out POSTHOOK: Lineage: union_out.id EXPRESSION [] -PREHOOK: query: select * from union_out cluster by id +PREHOOK: query: select * from union_out PREHOOK: type: QUERY PREHOOK: Input: default@union_out #### A masked pattern was here #### -POSTHOOK: query: select * from union_out cluster by id +POSTHOOK: query: select * from union_out POSTHOOK: type: QUERY POSTHOOK: Input: default@union_out #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/tez/union4.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/union4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/union4.q.out (working copy) @@ -1,13 +1,17 @@ -PREHOOK: query: -- union case: both subqueries are map-reduce jobs on same input, followed by filesink +PREHOOK: query: -- SORT_QUERY_RESULTS +-- union case: both subqueries are map-reduce jobs on same input, followed by filesink + create table tmptable(key string, value int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@tmptable -POSTHOOK: query: -- union case: both subqueries are map-reduce jobs on same input, followed by filesink +POSTHOOK: query: -- SORT_QUERY_RESULTS +-- union case: both subqueries are map-reduce jobs on same input, followed by filesink + create table tmptable(key string, value int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default Index: ql/src/test/results/clientpositive/tez/vector_char_simple.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vector_char_simple.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vector_char_simple.q.out (working copy) @@ -271,17 +271,17 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator sort order: - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: int) Execution mode: vectorized Reducer 2 @@ -289,17 +289,17 @@ Select Operator expressions: VALUE._col0 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: CAST( _col0 AS CHAR(12) (type: char(12)) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat Index: ql/src/test/results/clientpositive/tez/vector_left_outer_join.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vector_left_outer_join.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vector_left_outer_join.q.out (working copy) @@ -32,11 +32,11 @@ Map Operator Tree: TableScan alias: c - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cint (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Left Outer Join0 to 1 @@ -46,7 +46,7 @@ outputColumnNames: _col0 input vertices: 1 Map 3 - Statistics: Num rows: 13516 Data size: 414960 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 13516 Data size: 2906160 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Left Outer Join0 to 1 @@ -55,7 +55,7 @@ 1 _col0 (type: tinyint) input vertices: 1 Map 4 - Statistics: Num rows: 14867 Data size: 456456 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 14867 Data size: 3196776 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash @@ -70,31 +70,31 @@ Map Operator Tree: TableScan alias: c - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + Map-reduce partition columns: _col0 (type: int) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Map 4 Map Operator Tree: TableScan alias: c - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint) sort order: + Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Reducer 2 Reduce Operator Tree: Index: ql/src/test/results/clientpositive/tez/vector_varchar_simple.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vector_varchar_simple.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vector_varchar_simple.q.out (working copy) @@ -271,17 +271,17 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator sort order: - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: int) Execution mode: vectorized Reducer 2 @@ -289,17 +289,17 @@ Select Operator expressions: VALUE._col0 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: CAST( _col0 AS varchar(25)) (type: varchar(25)) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_0.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_0.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_0.q.out (working copy) @@ -32,11 +32,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: ctinyint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: min(ctinyint), max(ctinyint), count(ctinyint), count() mode: hash @@ -126,11 +126,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash @@ -229,11 +229,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: ctinyint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(ctinyint), variance(ctinyint), var_pop(ctinyint), var_samp(ctinyint), std(ctinyint), stddev(ctinyint), stddev_pop(ctinyint), stddev_samp(ctinyint) mode: hash @@ -338,11 +338,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: cbigint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: min(cbigint), max(cbigint), count(cbigint), count() mode: hash @@ -432,11 +432,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash @@ -535,11 +535,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: cbigint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(cbigint), variance(cbigint), var_pop(cbigint), var_samp(cbigint), std(cbigint), stddev(cbigint), stddev_pop(cbigint), stddev_samp(cbigint) mode: hash @@ -644,11 +644,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float) outputColumnNames: cfloat - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: min(cfloat), max(cfloat), count(cfloat), count() mode: hash @@ -738,11 +738,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash @@ -841,11 +841,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float) outputColumnNames: cfloat - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(cfloat), variance(cfloat), var_pop(cfloat), var_samp(cfloat), std(cfloat), stddev(cfloat), stddev_pop(cfloat), stddev_samp(cfloat) mode: hash @@ -988,14 +988,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%b%') or ((79.553 <> UDFToDouble(cint)) or (UDFToDouble(cbigint) < cdouble))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint), cfloat (type: float), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0), stddev_pop(_col0), var_samp(_col0), count(), sum(_col1), min(_col2) mode: hash Index: ql/src/test/results/clientpositive/tez/vectorization_13.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_13.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_13.q.out (working copy) @@ -80,25 +80,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cfloat < 3569.0) and ((10.175 >= cdouble) and (cboolean1 <> 1))) or ((UDFToDouble(ctimestamp1) > 11.0) and ((UDFToDouble(ctimestamp2) <> 12.0) and (UDFToDouble(ctinyint) < 9763215.5639)))) (type: boolean) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col1), sum(_col3), stddev_pop(_col3), stddev_pop(_col1), max(_col3), min(_col1) keys: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ Map-reduce partition columns: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: double), _col7 (type: struct), _col8 (type: struct), _col9 (type: float), _col10 (type: tinyint) Execution mode: vectorized Reducer 2 @@ -108,28 +108,28 @@ keys: KEY._col0 (type: boolean), KEY._col1 (type: tinyint), KEY._col2 (type: timestamp), KEY._col3 (type: float), KEY._col4 (type: string) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: boolean), _col1 (type: tinyint), (- _col6) (type: double), (79.553 * UDFToDouble(_col3)) (type: double), _col7 (type: double), _col8 (type: double), (UDFToDouble(((- _col1) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col1) + _col5))) / UDFToDouble(_col1)) (type: double), _col2 (type: timestamp), _col10 (type: tinyint), _col3 (type: float), _col4 (type: string), (- _col1) (type: tinyint), _col5 (type: tinyint), ((- _col1) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col1) + _col5))) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: tinyint), _col7 (type: tinyint), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: double), _col14 (type: double), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: float), _col19 (type: double), _col20 (type: tinyint) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: boolean), KEY.reducesinkkey1 (type: tinyint), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey3 (type: float), KEY.reducesinkkey4 (type: string), VALUE._col0 (type: tinyint), VALUE._col1 (type: tinyint), VALUE._col2 (type: tinyint), VALUE._col3 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col5 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: float), VALUE._col13 (type: double), VALUE._col14 (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 40 - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -334,25 +334,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cfloat < 3569.0) and ((10.175 >= cdouble) and (cboolean1 <> 1))) or ((UDFToDouble(ctimestamp1) > -1.388) and ((UDFToDouble(ctimestamp2) <> -1.3359999999999999) and (UDFToDouble(ctinyint) < 9763215.5639)))) (type: boolean) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col1), sum(_col3), stddev_pop(_col3), stddev_pop(_col1), max(_col3), min(_col1) keys: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ Map-reduce partition columns: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: double), _col7 (type: struct), _col8 (type: struct), _col9 (type: float), _col10 (type: tinyint) Execution mode: vectorized Reducer 2 @@ -362,28 +362,28 @@ keys: KEY._col0 (type: boolean), KEY._col1 (type: tinyint), KEY._col2 (type: timestamp), KEY._col3 (type: float), KEY._col4 (type: string) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: boolean), _col1 (type: tinyint), (- _col6) (type: double), (79.553 * UDFToDouble(_col3)) (type: double), _col7 (type: double), _col8 (type: double), (UDFToDouble(((- _col1) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col1) + _col5))) / UDFToDouble(_col1)) (type: double), _col2 (type: timestamp), _col10 (type: tinyint), _col3 (type: float), _col4 (type: string), (- _col1) (type: tinyint), _col5 (type: tinyint), ((- _col1) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col1) + _col5))) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: tinyint), _col7 (type: tinyint), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: double), _col14 (type: double), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: float), _col19 (type: double), _col20 (type: tinyint) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: boolean), KEY.reducesinkkey1 (type: tinyint), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey3 (type: float), KEY.reducesinkkey4 (type: string), VALUE._col0 (type: tinyint), VALUE._col1 (type: tinyint), VALUE._col2 (type: tinyint), VALUE._col3 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col5 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: float), VALUE._col13 (type: double), VALUE._col14 (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 40 - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_14.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_14.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_14.q.out (working copy) @@ -80,25 +80,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((((UDFToLong(ctinyint) <= cbigint) and ((UDFToDouble(cint) <= cdouble) or (ctimestamp2 < ctimestamp1))) and (cdouble < UDFToDouble(ctinyint))) and ((cbigint > -257) or (cfloat < UDFToFloat(cint)))) (type: boolean) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string), cboolean1 (type: boolean), cdouble (type: double), (- (-26.28 + cdouble)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_samp(_col5), max(_col1), stddev_pop(_col1), count(_col1), var_pop(_col1), var_samp(_col1) keys: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double) sort order: +++++ Map-reduce partition columns: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: struct), _col6 (type: float), _col7 (type: struct), _col8 (type: bigint), _col9 (type: struct), _col10 (type: struct) Execution mode: vectorized Reducer 2 @@ -108,25 +108,25 @@ keys: KEY._col0 (type: timestamp), KEY._col1 (type: float), KEY._col2 (type: string), KEY._col3 (type: boolean), KEY._col4 (type: double) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double), (-26.28 + _col4) (type: double), (- (-26.28 + _col4)) (type: double), _col5 (type: double), (UDFToDouble(_col1) * -26.28) (type: double), _col6 (type: float), (- _col1) (type: float), (- _col6) (type: float), ((- (-26.28 + _col4)) / 10.175) (type: double), _col7 (type: double), _col8 (type: bigint), (- ((- (-26.28 + _col4)) / 10.175)) (type: double), (-1.389 % _col5) (type: double), (UDFToDouble(_col1) - _col4) (type: double), _col9 (type: double), (_col9 % 10.175) (type: double), _col10 (type: double), (- (UDFToDouble(_col1) - _col4)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: float), _col4 (type: double), _col0 (type: timestamp) sort order: ++++ - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE value expressions: _col3 (type: boolean), _col5 (type: double), _col6 (type: double), _col7 (type: double), _col8 (type: double), _col9 (type: float), _col10 (type: float), _col11 (type: float), _col12 (type: double), _col13 (type: double), _col14 (type: bigint), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: double), _col19 (type: double), _col20 (type: double), _col21 (type: double) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey3 (type: timestamp), KEY.reducesinkkey1 (type: float), KEY.reducesinkkey0 (type: string), VALUE._col0 (type: boolean), KEY.reducesinkkey2 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: float), VALUE._col6 (type: float), VALUE._col7 (type: float), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: bigint), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col14 (type: double), VALUE._col15 (type: double), VALUE._col16 (type: double), VALUE._col17 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_15.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_15.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_15.q.out (working copy) @@ -76,25 +76,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%ss%') or ((cstring1 like '10%') or ((cint >= -75) and ((UDFToShort(ctinyint) = csmallint) and (cdouble >= -3728.0))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float), cboolean1 (type: boolean), cdouble (type: double), cstring1 (type: string), ctinyint (type: tinyint), cint (type: int), ctimestamp1 (type: timestamp) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_samp(_col0), min(_col2), stddev_samp(_col4), var_pop(_col4), var_samp(_col5), stddev_pop(_col5) keys: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) sort order: +++++++ Map-reduce partition columns: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE value expressions: _col7 (type: struct), _col8 (type: double), _col9 (type: struct), _col10 (type: struct), _col11 (type: struct), _col12 (type: struct) Execution mode: vectorized Reducer 2 @@ -104,25 +104,25 @@ keys: KEY._col0 (type: float), KEY._col1 (type: boolean), KEY._col2 (type: double), KEY._col3 (type: string), KEY._col4 (type: tinyint), KEY._col5 (type: int), KEY._col6 (type: timestamp) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp), _col7 (type: double), (-26.28 - UDFToDouble(_col5)) (type: double), _col8 (type: double), (_col2 * 79.553) (type: double), (33.0 % _col0) (type: float), _col9 (type: double), _col10 (type: double), (-23.0 % _col2) (type: double), (- _col4) (type: tinyint), _col11 (type: double), (UDFToFloat(_col5) - _col0) (type: float), (-23 % UDFToInteger(_col4)) (type: int), (- (-26.28 - UDFToDouble(_col5))) (type: double), _col12 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) sort order: +++++++ - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE value expressions: _col7 (type: double), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: float), _col12 (type: double), _col13 (type: double), _col14 (type: double), _col15 (type: tinyint), _col16 (type: double), _col17 (type: float), _col18 (type: int), _col19 (type: double), _col20 (type: double) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: float), KEY.reducesinkkey1 (type: boolean), KEY.reducesinkkey2 (type: double), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: tinyint), KEY.reducesinkkey5 (type: int), KEY.reducesinkkey6 (type: timestamp), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col4 (type: float), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: tinyint), VALUE._col9 (type: double), VALUE._col10 (type: float), VALUE._col11 (type: int), VALUE._col12 (type: double), VALUE._col13 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_16.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_16.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_16.q.out (working copy) @@ -57,25 +57,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%b%') and ((cdouble >= -1.389) or (cstring1 < 'a'))) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cstring1 (type: string), cdouble (type: double), ctimestamp1 (type: timestamp) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col1), stddev_samp(_col1), min(_col1) keys: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) sort order: +++ Map-reduce partition columns: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE value expressions: _col3 (type: bigint), _col4 (type: struct), _col5 (type: double) Execution mode: vectorized Reducer 2 @@ -85,14 +85,14 @@ keys: KEY._col0 (type: string), KEY._col1 (type: double), KEY._col2 (type: timestamp) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp), (_col1 - 9763215.5639) (type: double), (- (_col1 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col1) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_7.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_7.q.out (working copy) @@ -63,20 +63,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((ctinyint <> 0) and ((UDFToDouble(ctimestamp1) <= 0.0) or ((UDFToInteger(ctinyint) = cint) or (cstring2 like 'ss')))) and ((988888.0 < cdouble) or ((UDFToDouble(ctimestamp2) > -15.0) and (3569.0 >= cdouble)))) (type: boolean) - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), cbigint (type: bigint), csmallint (type: smallint), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cstring1 (type: string), (cbigint + cbigint) (type: bigint), (UDFToInteger(csmallint) % -257) (type: int), (- csmallint) (type: smallint), (- ctinyint) (type: tinyint), (UDFToInteger((- ctinyint)) + 17) (type: int), (cbigint * UDFToLong((- csmallint))) (type: bigint), (cint % UDFToInteger(csmallint)) (type: int), (- ctinyint) (type: tinyint), ((- ctinyint) % ctinyint) (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -235,20 +235,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((ctinyint <> 0) and ((UDFToDouble(ctimestamp1) <= 0.0) or ((UDFToInteger(ctinyint) = cint) or (cstring2 like 'ss')))) and ((988888.0 < cdouble) or ((UDFToDouble(ctimestamp2) > 7.6850000000000005) and (3569.0 >= cdouble)))) (type: boolean) - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), cbigint (type: bigint), csmallint (type: smallint), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cstring1 (type: string), (cbigint + cbigint) (type: bigint), (UDFToInteger(csmallint) % -257) (type: int), (- csmallint) (type: smallint), (- ctinyint) (type: tinyint), (UDFToInteger((- ctinyint)) + 17) (type: int), (cbigint * UDFToLong((- csmallint))) (type: bigint), (cint % UDFToInteger(csmallint)) (type: int), (- ctinyint) (type: tinyint), ((- ctinyint) % ctinyint) (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_8.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_8.q.out (working copy) @@ -59,20 +59,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 is not null and ((UDFToDouble(ctimestamp1) <= 10.0) and (UDFToDouble(ctimestamp2) <> 16.0))) or ((cfloat < -6432.0) or (cboolean1 is not null and (cdouble = 988888.0)))) (type: boolean) - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cdouble (type: double), cboolean1 (type: boolean), cstring1 (type: string), cfloat (type: float), (- cdouble) (type: double), (-5638.15 - cdouble) (type: double), (cdouble * -257.0) (type: double), (UDFToFloat(cint) + cfloat) (type: float), ((- cdouble) + UDFToDouble(cbigint)) (type: double), (- cdouble) (type: double), (-1.389 - UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), ((-5638.15 - cdouble) + UDFToDouble((UDFToFloat(cint) + cfloat))) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13 - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -218,20 +218,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 is not null and ((UDFToDouble(ctimestamp1) <= 12.503) and (UDFToDouble(ctimestamp2) <> 11.998))) or ((cfloat < -6432.0) or (cboolean1 is not null and (cdouble = 988888.0)))) (type: boolean) - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cdouble (type: double), cboolean1 (type: boolean), cstring1 (type: string), cfloat (type: float), (- cdouble) (type: double), (-5638.15 - cdouble) (type: double), (cdouble * -257.0) (type: double), (UDFToFloat(cint) + cfloat) (type: float), ((- cdouble) + UDFToDouble(cbigint)) (type: double), (- cdouble) (type: double), (-1.389 - UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), ((-5638.15 - cdouble) + UDFToDouble((UDFToFloat(cint) + cfloat))) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13 - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_9.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_9.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_9.q.out (working copy) @@ -53,25 +53,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%b%') and ((cdouble >= -1.389) or (cstring1 < 'a'))) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cstring1 (type: string), cdouble (type: double), ctimestamp1 (type: timestamp) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col1), stddev_samp(_col1), min(_col1) keys: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) sort order: +++ Map-reduce partition columns: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE value expressions: _col3 (type: bigint), _col4 (type: struct), _col5 (type: double) Execution mode: vectorized Reducer 2 @@ -81,14 +81,14 @@ keys: KEY._col0 (type: string), KEY._col1 (type: double), KEY._col2 (type: timestamp) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp), (_col1 - 9763215.5639) (type: double), (- (_col1 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col1) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_div0.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_div0.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_div0.q.out (working copy) @@ -160,18 +160,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cbigint > 0) and (cbigint < 100000000)) (type: boolean) - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: (cbigint - 988888) (type: bigint), (cdouble / UDFToDouble((cbigint - 988888))) (type: double), (1.2 / UDFToDouble((cbigint - 988888))) (type: double) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: bigint), _col1 (type: double) sort order: ++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double) Execution mode: vectorized Reducer 2 @@ -179,13 +179,13 @@ Select Operator expressions: KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: double), VALUE._col0 (type: double) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 100 - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -335,18 +335,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cdouble >= -500.0) and (cdouble < -199.0)) (type: boolean) - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: (cdouble + 200.0) (type: double), (UDFToDouble(cbigint) / (cdouble + 200.0)) (type: double), ((cdouble + 200.0) / (cdouble + 200.0)) (type: double), (3.0 / (cdouble + 200.0)) (type: double), (1.2 / (cdouble + 200.0)) (type: double) outputColumnNames: _col0, _col1, _col2, _col4, _col5 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double), _col1 (type: double) sort order: ++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double), _col4 (type: double), _col5 (type: double) Execution mode: vectorized Reducer 2 @@ -354,13 +354,13 @@ Select Operator expressions: KEY.reducesinkkey0 (type: double), KEY.reducesinkkey1 (type: double), VALUE._col0 (type: double), KEY.reducesinkkey1 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 100 - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_limit.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_limit.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_limit.q.out (working copy) @@ -63,18 +63,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ctinyint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cdouble (type: double), csmallint (type: smallint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: double) sort order: ++ - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 value expressions: _col2 (type: smallint) Execution mode: vectorized @@ -83,13 +83,13 @@ Select Operator expressions: KEY.reducesinkkey0 (type: tinyint), KEY.reducesinkkey1 (type: double), VALUE._col0 (type: smallint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -153,22 +153,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), (cdouble + 1.0) (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col1) keys: _col0 (type: tinyint) mode: hash outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint) sort order: + Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 value expressions: _col1 (type: struct) Execution mode: vectorized @@ -179,13 +179,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -248,21 +248,21 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: tinyint) mode: hash outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint) sort order: + Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Execution mode: vectorized Reducer 2 @@ -271,13 +271,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -339,22 +339,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cdouble (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(DISTINCT _col1) keys: _col0 (type: tinyint), _col1 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: double) sort order: ++ Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Execution mode: vectorized Reducer 2 @@ -364,13 +364,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -460,25 +460,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ctinyint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double), ctinyint (type: tinyint) outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col1) keys: _col0 (type: double) mode: hash outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double) sort order: + Map-reduce partition columns: _col0 (type: double) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: bigint) Execution mode: vectorized Reducer 2 @@ -488,11 +488,11 @@ keys: KEY._col0 (type: double) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col1 (type: bigint), _col0 (type: double) sort order: ++ - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Execution mode: vectorized Reducer 3 @@ -500,13 +500,13 @@ Select Operator expressions: KEY.reducesinkkey1 (type: double), KEY.reducesinkkey0 (type: bigint) outputColumnNames: _col0, _col1 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorization_pushdown.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_pushdown.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_pushdown.q.out (working copy) @@ -19,14 +19,14 @@ TableScan alias: alltypesorc filterExpr: (UDFToDouble(cbigint) < cdouble) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (UDFToDouble(cbigint) < cdouble) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: _col0 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0) mode: hash Index: ql/src/test/results/clientpositive/tez/vectorization_short_regress.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorization_short_regress.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorization_short_regress.q.out (working copy) @@ -146,14 +146,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((762 = cbigint) or (((UDFToFloat(csmallint) < cfloat) and ((UDFToDouble(ctimestamp2) > -5.0) and (cdouble <> UDFToDouble(cint)))) or ((cstring1 = 'a') or ((UDFToDouble(cbigint) <= -1.389) and ((cstring2 <> 'a') and ((79.553 <> UDFToDouble(cint)) and (cboolean2 <> cboolean1))))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cdouble (type: double), csmallint (type: smallint), cfloat (type: float), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0), sum(_col1), stddev_pop(_col0), stddev_samp(_col2), var_samp(_col0), avg(_col3), stddev_samp(_col0), min(_col4), count(_col2) mode: hash @@ -358,14 +358,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cbigint <= 197) and (UDFToLong(cint) < cbigint)) or (((cdouble >= -26.28) and (UDFToDouble(csmallint) > cdouble)) or (((UDFToFloat(ctinyint) > cfloat) and (cstring1 rlike '.*ss.*')) or ((cfloat > 79.553) and (cstring2 like '10%'))))) (type: boolean) - Statistics: Num rows: 6826 Data size: 209555 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6826 Data size: 1467614 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cbigint (type: bigint), csmallint (type: smallint), cdouble (type: double), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 6826 Data size: 209555 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6826 Data size: 1467614 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col0), var_pop(_col1), stddev_pop(_col2), max(_col3), avg(_col4), min(_col0), min(_col3), stddev_samp(_col2), var_samp(_col0) mode: hash @@ -561,14 +561,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((ctimestamp1 = ctimestamp2) or ((762.0 = cfloat) or ((cstring1 = 'ss') or (((UDFToLong(csmallint) <= cbigint) and (1 = cboolean2)) or (cboolean1 is not null and (ctimestamp2 is not null and (cstring2 > 'a'))))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint), ctinyint (type: tinyint), csmallint (type: smallint), cint (type: int), cdouble (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: var_pop(_col0), count(), max(_col1), stddev_pop(_col2), max(_col3), stddev_samp(_col4), count(_col1), avg(_col1) mode: hash @@ -743,14 +743,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((ctimestamp2 <= ctimestamp1) and ((UDFToDouble(cbigint) <> cdouble) and ('ss' <= cstring1))) or (((csmallint < UDFToShort(ctinyint)) and (UDFToDouble(ctimestamp1) >= 0.0)) or (cfloat = 17.0))) (type: boolean) - Statistics: Num rows: 8874 Data size: 272428 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8874 Data size: 1907941 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cbigint (type: bigint), cint (type: int), cfloat (type: float) outputColumnNames: _col0, _col1, _col2, _col3 - Statistics: Num rows: 8874 Data size: 272428 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8874 Data size: 1907941 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0), max(_col1), stddev_samp(_col2), var_pop(_col2), var_pop(_col1), max(_col3) mode: hash @@ -933,18 +933,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cstring1 rlike 'a.*') and (cstring2 like '%ss%')) or (((1 <> cboolean2) and ((UDFToDouble(csmallint) < 79.553) and (-257 <> UDFToInteger(ctinyint)))) or (((cdouble > UDFToDouble(ctinyint)) and (cfloat >= UDFToFloat(cint))) or ((UDFToLong(cint) < cbigint) and (UDFToLong(ctinyint) > cbigint))))) (type: boolean) - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cdouble (type: double), ctimestamp2 (type: timestamp), cstring1 (type: string), cboolean2 (type: boolean), ctinyint (type: tinyint), cfloat (type: float), ctimestamp1 (type: timestamp), csmallint (type: smallint), cbigint (type: bigint), (-3728 * cbigint) (type: bigint), (- cint) (type: int), (-863.257 - UDFToDouble(cint)) (type: double), (- csmallint) (type: smallint), (csmallint - (- csmallint)) (type: smallint), ((csmallint - (- csmallint)) + (- csmallint)) (type: smallint), (UDFToDouble(cint) / UDFToDouble(cint)) (type: double), ((-863.257 - UDFToDouble(cint)) - -26.28) (type: double), (- cfloat) (type: float), (cdouble * -89010.0) (type: double), (UDFToDouble(ctinyint) / 988888.0) (type: double), (- ctinyint) (type: tinyint), (79.553 / UDFToDouble(ctinyint)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22 - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int), _col1 (type: double), _col2 (type: timestamp), _col3 (type: string), _col4 (type: boolean), _col5 (type: tinyint), _col6 (type: float), _col7 (type: timestamp), _col8 (type: smallint), _col9 (type: bigint) sort order: ++++++++++ - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE value expressions: _col10 (type: bigint), _col11 (type: int), _col12 (type: double), _col13 (type: smallint), _col14 (type: smallint), _col15 (type: smallint), _col16 (type: double), _col17 (type: double), _col18 (type: float), _col19 (type: double), _col20 (type: double), _col21 (type: tinyint), _col22 (type: double) Execution mode: vectorized Reducer 2 @@ -952,13 +952,13 @@ Select Operator expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: double), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: boolean), KEY.reducesinkkey5 (type: tinyint), KEY.reducesinkkey6 (type: float), KEY.reducesinkkey7 (type: timestamp), KEY.reducesinkkey8 (type: smallint), KEY.reducesinkkey9 (type: bigint), VALUE._col0 (type: bigint), VALUE._col1 (type: int), VALUE._col2 (type: double), VALUE._col3 (type: smallint), VALUE._col4 (type: smallint), VALUE._col5 (type: smallint), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: float), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: tinyint), VALUE._col12 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22 - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 50 - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1192,18 +1192,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((197.0 > UDFToDouble(ctinyint)) and (UDFToLong(cint) = cbigint)) or ((cbigint = 359) or ((cboolean1 < 0) or ((cstring1 like '%ss') and (cfloat <= UDFToFloat(ctinyint)))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cbigint (type: bigint), (UDFToDouble(cint) / UDFToDouble(cbigint)) (type: double), (UDFToDouble(cbigint) % 79.553) (type: double), (- (UDFToDouble(cint) / UDFToDouble(cbigint))) (type: double), (10.175 % UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), (cfloat - (- cfloat)) (type: float), ((cfloat - (- cfloat)) % -6432.0) (type: float), (cdouble * UDFToDouble(csmallint)) (type: double), (- cdouble) (type: double), (- cbigint) (type: bigint), cstring1 (type: string), (UDFToDouble(cfloat) - (UDFToDouble(cint) / UDFToDouble(cbigint))) (type: double), (- csmallint) (type: smallint), (3569 % cbigint) (type: bigint), (359.0 - cdouble) (type: double), cboolean1 (type: boolean), cfloat (type: float), cdouble (type: double), ctimestamp2 (type: timestamp), csmallint (type: smallint), cstring2 (type: string), cboolean2 (type: boolean) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col21, _col22, _col23, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int), _col1 (type: bigint), _col2 (type: string), _col3 (type: boolean), _col4 (type: float), _col5 (type: double), _col6 (type: timestamp), _col7 (type: smallint), _col8 (type: string), _col9 (type: boolean) sort order: ++++++++++ - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE value expressions: _col10 (type: double), _col11 (type: double), _col12 (type: double), _col13 (type: double), _col14 (type: float), _col15 (type: float), _col16 (type: float), _col17 (type: double), _col18 (type: double), _col19 (type: bigint), _col20 (type: double), _col21 (type: smallint), _col22 (type: bigint), _col23 (type: double) Execution mode: vectorized Reducer 2 @@ -1211,13 +1211,13 @@ Select Operator expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey3 (type: boolean), KEY.reducesinkkey4 (type: float), KEY.reducesinkkey5 (type: double), KEY.reducesinkkey6 (type: timestamp), KEY.reducesinkkey7 (type: smallint), KEY.reducesinkkey8 (type: string), KEY.reducesinkkey9 (type: boolean), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col4 (type: float), VALUE._col5 (type: float), VALUE._col6 (type: float), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: bigint), VALUE._col10 (type: double), VALUE._col11 (type: smallint), VALUE._col12 (type: bigint), VALUE._col13 (type: double), VALUE._col11 (type: smallint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1400,18 +1400,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((UDFToDouble(csmallint) > -26.28) and (cstring2 like 'ss')) or (((cdouble <= UDFToDouble(cbigint)) and ((cstring1 >= 'ss') and (UDFToDouble(cint) <> cdouble))) or ((UDFToInteger(ctinyint) = -89010) or ((UDFToFloat(cbigint) <= cfloat) and (-26.28 <= UDFToDouble(csmallint)))))) (type: boolean) - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cstring1 (type: string), cboolean2 (type: boolean), ctimestamp2 (type: timestamp), cdouble (type: double), cfloat (type: float), cbigint (type: bigint), csmallint (type: smallint), cboolean1 (type: boolean), (cint + UDFToInteger(csmallint)) (type: int), (cbigint - UDFToLong(ctinyint)) (type: bigint), (- cbigint) (type: bigint), (- cfloat) (type: float), ((cbigint - UDFToLong(ctinyint)) + cbigint) (type: bigint), (cdouble / cdouble) (type: double), (- cdouble) (type: double), (UDFToLong((cint + UDFToInteger(csmallint))) * (- cbigint)) (type: bigint), ((- cdouble) + UDFToDouble(cbigint)) (type: double), (-1.389 / UDFToDouble(ctinyint)) (type: double), (UDFToDouble(cbigint) % cdouble) (type: double), (- csmallint) (type: smallint), (UDFToInteger(csmallint) + (cint + UDFToInteger(csmallint))) (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col8 (type: boolean), _col1 (type: string), _col3 (type: timestamp), _col5 (type: float), _col6 (type: bigint), _col1 (type: string), _col4 (type: double), _col0 (type: int), _col7 (type: smallint), _col4 (type: double) sort order: ++++++++++ - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: boolean), _col9 (type: int), _col10 (type: bigint), _col11 (type: bigint), _col12 (type: float), _col13 (type: bigint), _col14 (type: double), _col15 (type: double), _col16 (type: bigint), _col17 (type: double), _col18 (type: double), _col19 (type: double), _col20 (type: smallint), _col21 (type: int) Execution mode: vectorized Reducer 2 @@ -1419,13 +1419,13 @@ Select Operator expressions: KEY.reducesinkkey7 (type: int), KEY.reducesinkkey1 (type: string), VALUE._col0 (type: boolean), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey6 (type: double), KEY.reducesinkkey3 (type: float), KEY.reducesinkkey4 (type: bigint), KEY.reducesinkkey8 (type: smallint), KEY.reducesinkkey0 (type: boolean), VALUE._col1 (type: int), VALUE._col2 (type: bigint), VALUE._col3 (type: bigint), VALUE._col4 (type: float), VALUE._col5 (type: bigint), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: bigint), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: smallint), VALUE._col13 (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 75 - Statistics: Num rows: 75 Data size: 2250 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 75 Data size: 16125 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 75 Data size: 2250 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 75 Data size: 16125 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1666,18 +1666,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((-1.389 >= UDFToDouble(cint)) and ((csmallint < UDFToShort(ctinyint)) and (-6432 > UDFToInteger(csmallint)))) or (((cdouble >= UDFToDouble(cfloat)) and (cstring2 <= 'a')) or ((cstring1 like 'ss%') and (10.175 > UDFToDouble(cbigint))))) (type: boolean) - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cstring2 (type: string), (cdouble * 10.175) (type: double), (UDFToDouble((-6432.0 * cfloat)) / UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), (cint % UDFToInteger(csmallint)) (type: int), (cdouble * (- cdouble)) (type: double), cdouble (type: double), cfloat (type: float), cbigint (type: bigint), csmallint (type: smallint), (UDFToDouble(cbigint) / 3569.0) (type: double), (-257 - UDFToInteger(csmallint)) (type: int), (-6432.0 * cfloat) (type: float), (- cdouble) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col15, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col5 (type: smallint), _col1 (type: string), _col2 (type: double) sort order: +++ - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: timestamp), _col3 (type: float), _col4 (type: bigint), _col6 (type: double), _col7 (type: int), _col8 (type: float), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: float), _col13 (type: int), _col15 (type: double) Execution mode: vectorized Reducer 2 @@ -1685,13 +1685,13 @@ Select Operator expressions: VALUE._col0 (type: timestamp), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: double), VALUE._col1 (type: float), VALUE._col2 (type: bigint), KEY.reducesinkkey0 (type: smallint), VALUE._col3 (type: double), VALUE._col4 (type: int), VALUE._col5 (type: float), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: float), VALUE._col10 (type: int), VALUE._col6 (type: double), VALUE._col11 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15 - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 45 - Statistics: Num rows: 45 Data size: 1350 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 45 Data size: 9675 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 45 Data size: 1350 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 45 Data size: 9675 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1874,25 +1874,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((UDFToInteger(csmallint) >= -257) and ((-6432 = UDFToInteger(csmallint)) or ((UDFToDouble(cint) >= cdouble) and (UDFToInteger(ctinyint) <= cint)))) (type: boolean) - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: csmallint (type: smallint), cbigint (type: bigint), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_samp(_col0), sum(_col1), var_pop(_col2), count() keys: _col0 (type: smallint) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: smallint) sort order: + Map-reduce partition columns: _col0 (type: smallint) - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: struct), _col2 (type: bigint), _col3 (type: struct), _col4 (type: bigint) Execution mode: vectorized Reducer 2 @@ -1902,28 +1902,28 @@ keys: KEY._col0 (type: smallint) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: smallint), (UDFToInteger(_col0) % -75) (type: int), _col1 (type: double), (-1.389 / UDFToDouble(_col0)) (type: double), _col2 (type: bigint), (UDFToDouble((UDFToInteger(_col0) % -75)) / UDFToDouble(_col2)) (type: double), (- (UDFToInteger(_col0) % -75)) (type: int), _col3 (type: double), (- (- (UDFToInteger(_col0) % -75))) (type: int), _col4 (type: bigint), (_col4 - -89010) (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: smallint) sort order: + - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: int), _col2 (type: double), _col3 (type: double), _col4 (type: bigint), _col5 (type: double), _col6 (type: int), _col7 (type: double), _col8 (type: int), _col9 (type: bigint), _col10 (type: bigint) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: smallint), VALUE._col0 (type: int), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: bigint), VALUE._col4 (type: double), VALUE._col5 (type: int), VALUE._col6 (type: double), VALUE._col7 (type: int), VALUE._col8 (type: bigint), VALUE._col9 (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -2082,25 +2082,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cdouble > 2563.58) and (((cbigint >= UDFToLong(cint)) and ((UDFToInteger(csmallint) < cint) and (UDFToDouble(cfloat) < -5638.15))) or ((cdouble <= UDFToDouble(cbigint)) and (-5638.15 > UDFToDouble(cbigint))))) (type: boolean) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double), cfloat (type: float) outputColumnNames: _col0, _col1 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: var_samp(_col0), count(_col1), sum(_col1), var_pop(_col0), stddev_pop(_col0), sum(_col0) keys: _col0 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double) sort order: + Map-reduce partition columns: _col0 (type: double) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: struct), _col2 (type: bigint), _col3 (type: double), _col4 (type: struct), _col5 (type: struct), _col6 (type: double) Execution mode: vectorized Reducer 2 @@ -2110,25 +2110,25 @@ keys: KEY._col0 (type: double) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: double), _col1 (type: double), _col5 (type: double), (_col0 + _col1) (type: double), (_col0 * 762.0) (type: double), _col6 (type: double), (-863.257 % (_col0 * 762.0)) (type: double), (2563.58 * _col1) (type: double), (- _col1) (type: double), _col2 (type: bigint), ((2563.58 * _col1) + -5638.15) (type: double), ((- _col1) * ((2563.58 * _col1) + -5638.15)) (type: double), _col3 (type: double), _col4 (type: double), (_col0 - (- _col1)) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double) sort order: + - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: double), _col2 (type: double), _col3 (type: double), _col4 (type: bigint), _col5 (type: double), _col6 (type: double), _col7 (type: double), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: double), _col13 (type: double), _col14 (type: double) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: double), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: bigint), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col12 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -2337,25 +2337,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((UDFToDouble(ctimestamp1) <> 0.0) and (((-257 <> UDFToInteger(ctinyint)) and (cboolean2 is not null and ((cstring1 rlike '.*ss') and (-3.0 < UDFToDouble(ctimestamp1))))) or ((UDFToDouble(ctimestamp2) = -5.0) or (((UDFToDouble(ctimestamp1) < 0.0) and (cstring2 like '%b%')) or ((cdouble = UDFToDouble(cint)) or (cboolean1 is null and (cfloat < UDFToFloat(cint)))))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cstring1 (type: string), cint (type: int), csmallint (type: smallint), ctinyint (type: tinyint), cfloat (type: float), cdouble (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_pop(_col2), avg(_col3), count(), min(_col4), var_samp(_col3), var_pop(_col5), avg(_col2), var_samp(_col5), avg(_col5), min(_col6), var_pop(_col3), stddev_pop(_col4), sum(_col2) keys: _col0 (type: timestamp), _col1 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: timestamp), _col1 (type: string) sort order: ++ Map-reduce partition columns: _col0 (type: timestamp), _col1 (type: string) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: struct), _col3 (type: struct), _col4 (type: bigint), _col5 (type: tinyint), _col6 (type: struct), _col7 (type: struct), _col8 (type: struct), _col9 (type: struct), _col10 (type: struct), _col11 (type: double), _col12 (type: struct), _col13 (type: struct), _col14 (type: bigint) Execution mode: vectorized Reducer 2 @@ -2365,28 +2365,28 @@ keys: KEY._col0 (type: timestamp), KEY._col1 (type: string) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: timestamp), _col1 (type: string), ((-26.28 - _col2) * (- _col2)) (type: double), _col5 (type: tinyint), (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4))) (type: double), (- (_col2 * 10.175)) (type: double), _col6 (type: double), (_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- _col2)) (type: double), (UDFToDouble((- _col4)) / _col2) (type: double), _col7 (type: double), (10.175 / _col3) (type: double), _col2 (type: double), _col8 (type: double), _col9 (type: double), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- (_col2 * 10.175))) (type: double), _col10 (type: double), (((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) * 10.175) (type: double), (10.175 % (10.175 / _col3)) (type: double), (- _col5) (type: tinyint), _col11 (type: double), _col12 (type: double), (_col2 * 10.175) (type: double), (- ((-26.28 - _col2) * (- _col2))) (type: double), ((- _col2) % _col10) (type: double), (-26.28 / UDFToDouble((- _col5))) (type: double), _col13 (type: double), _col14 (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) / _col7) (type: double), (- (- _col4)) (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) % -26.28) (type: double), (- _col2) (type: double), _col3 (type: double), (-26.28 - _col2) (type: double), _col4 (type: bigint), (- _col4) (type: bigint) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col3, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col38, _col4, _col5, _col7, _col8, _col9 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: timestamp), _col1 (type: string) sort order: ++ - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double), _col3 (type: double), _col4 (type: double), _col5 (type: double), _col7 (type: double), _col8 (type: bigint), _col9 (type: bigint), _col10 (type: double), _col11 (type: tinyint), _col12 (type: double), _col13 (type: double), _col14 (type: double), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: double), _col19 (type: double), _col20 (type: double), _col21 (type: double), _col22 (type: double), _col23 (type: double), _col24 (type: double), _col25 (type: double), _col26 (type: double), _col27 (type: tinyint), _col28 (type: double), _col29 (type: double), _col30 (type: double), _col31 (type: double), _col32 (type: double), _col33 (type: double), _col34 (type: bigint), _col35 (type: double), _col36 (type: bigint), _col38 (type: double) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: timestamp), KEY.reducesinkkey1 (type: string), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col2 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: bigint), VALUE._col6 (type: bigint), VALUE._col7 (type: double), VALUE._col8 (type: tinyint), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col14 (type: double), VALUE._col15 (type: double), VALUE._col16 (type: double), VALUE._col17 (type: double), VALUE._col18 (type: double), VALUE._col19 (type: double), VALUE._col20 (type: double), VALUE._col21 (type: double), VALUE._col22 (type: double), VALUE._col23 (type: double), VALUE._col24 (type: tinyint), VALUE._col25 (type: double), VALUE._col26 (type: double), VALUE._col27 (type: double), VALUE._col28 (type: double), VALUE._col29 (type: double), VALUE._col30 (type: double), VALUE._col31 (type: bigint), VALUE._col32 (type: double), VALUE._col33 (type: bigint), VALUE._col5 (type: bigint), VALUE._col34 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col37, _col38 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 50 - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -2670,25 +2670,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (cboolean1 is not null and (((cdouble < UDFToDouble(csmallint)) and ((cboolean2 = cboolean1) and (UDFToDouble(cbigint) <= -863.257))) or (((cint >= -257) and (cstring1 is not null and (cboolean1 >= 1))) or ((cstring2 rlike 'b') or ((csmallint >= UDFToShort(ctinyint)) and ctimestamp2 is null))))) (type: boolean) - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), cfloat (type: float), cbigint (type: bigint), cint (type: int), cdouble (type: double), ctinyint (type: tinyint), csmallint (type: smallint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col1), sum(_col2), var_samp(_col3), avg(_col4), min(_col2), var_pop(_col2), sum(_col3), stddev_samp(_col5), stddev_pop(_col6), avg(_col3) keys: _col0 (type: boolean) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean) sort order: + Map-reduce partition columns: _col0 (type: boolean) - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: float), _col2 (type: bigint), _col3 (type: struct), _col4 (type: struct), _col5 (type: bigint), _col6 (type: struct), _col7 (type: bigint), _col8 (type: struct), _col9 (type: struct), _col10 (type: struct) Execution mode: vectorized Reducer 2 @@ -2698,25 +2698,25 @@ keys: KEY._col0 (type: boolean) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: boolean), _col1 (type: float), ((UDFToDouble(_col2) - 10.175) + _col3) (type: double), _col5 (type: bigint), _col6 (type: double), (- (10.175 + UDFToDouble((- _col1)))) (type: double), (79.553 / _col6) (type: double), (_col3 % (79.553 / _col6)) (type: double), _col7 (type: bigint), _col8 (type: double), (-1.389 * UDFToDouble(_col5)) (type: double), (- _col1) (type: float), (UDFToDouble(_col7) - (-1.389 * UDFToDouble(_col5))) (type: double), _col9 (type: double), (- (UDFToDouble(_col7) - (-1.389 * UDFToDouble(_col5)))) (type: double), _col10 (type: double), (- _col10) (type: double), (_col10 * UDFToDouble(_col7)) (type: double), (-26.28 / UDFToDouble(_col1)) (type: double), _col2 (type: bigint), (UDFToDouble(_col2) - 10.175) (type: double), _col3 (type: double), (_col3 % UDFToDouble(_col1)) (type: double), (10.175 + UDFToDouble((- _col1))) (type: double), _col4 (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col15, _col17, _col18, _col19, _col2, _col20, _col21, _col22, _col23, _col24, _col25, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean) sort order: + - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: float), _col2 (type: float), _col3 (type: double), _col4 (type: bigint), _col5 (type: double), _col6 (type: double), _col7 (type: double), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: bigint), _col12 (type: double), _col13 (type: double), _col14 (type: double), _col15 (type: double), _col17 (type: bigint), _col18 (type: double), _col19 (type: double), _col20 (type: double), _col21 (type: double), _col22 (type: double), _col23 (type: double), _col24 (type: double), _col25 (type: double) Reducer 3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: boolean), VALUE._col0 (type: float), VALUE._col1 (type: float), VALUE._col2 (type: double), VALUE._col3 (type: bigint), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: bigint), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col14 (type: double), VALUE._col12 (type: double), VALUE._col15 (type: bigint), VALUE._col16 (type: double), VALUE._col17 (type: double), VALUE._col18 (type: double), VALUE._col19 (type: double), VALUE._col20 (type: double), VALUE._col21 (type: double), VALUE._col22 (type: double), VALUE._col23 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25 - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/tez/vectorized_distinct_gby.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorized_distinct_gby.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorized_distinct_gby.q.out (working copy) @@ -101,21 +101,21 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(DISTINCT _col0), count(DISTINCT _col0), avg(DISTINCT _col0), std(DISTINCT _col0) keys: _col0 (type: int) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Reducer 2 Reduce Operator Tree: Index: ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out (working copy) @@ -1793,6 +1793,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 #### A masked pattern was here #### 1000 +Warning: Shuffle Join MERGEJOIN[24][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product PREHOOK: query: -- parent is reduce tasks EXPLAIN select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY @@ -1816,17 +1817,13 @@ Map Operator Tree: TableScan alias: srcpart - filterExpr: ds is not null (type: boolean) - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + filterExpr: (ds = '2008-04-08') (type: boolean) + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: ds (type: string) - outputColumnNames: _col0 - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col0 (type: string) - sort order: + - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + sort order: + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Map 4 Map Operator Tree: TableScan @@ -1853,9 +1850,9 @@ condition map: Inner Join 0 to 1 keys: - 0 _col0 (type: string) - 1 _col0 (type: string) - Statistics: Num rows: 2200 Data size: 23372 Basic stats: COMPLETE Column stats: NONE + 0 + 1 + Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash @@ -1887,26 +1884,12 @@ mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string) - sort order: + - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: _col0 (type: string) - outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Group By Operator - keys: _col0 (type: string) - mode: hash - outputColumnNames: _col0 + Reduce Output Operator + sort order: Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Dynamic Partitioning Event Operator - Target Input: srcpart - Partition key expr: ds - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Target column: ds - Target Vertex: Map 1 + Execution mode: vectorized Stage: Stage-0 Fetch Operator @@ -1914,21 +1897,18 @@ Processor Tree: ListSink +Warning: Shuffle Join MERGEJOIN[24][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: default@srcpart PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### POSTHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### 1000 PREHOOK: query: select count(*) from srcpart where ds = '2008-04-08' @@ -4128,6 +4108,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 #### A masked pattern was here #### 1000 +Warning: Map Join MAPJOIN[24][bigTable=?] in task 'Map 1' is a cross product PREHOOK: query: -- parent is reduce tasks EXPLAIN select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY @@ -4151,21 +4132,19 @@ Map Operator Tree: TableScan alias: srcpart - filterExpr: ds is not null (type: boolean) - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + filterExpr: (ds = '2008-04-08') (type: boolean) + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: ds (type: string) - outputColumnNames: _col0 - Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 keys: - 0 _col0 (type: string) - 1 _col0 (type: string) + 0 + 1 input vertices: 1 Reducer 4 - Statistics: Num rows: 2200 Data size: 23372 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash @@ -4217,26 +4196,12 @@ mode: mergepartial outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string) - sort order: + - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE Select Operator - expressions: _col0 (type: string) - outputColumnNames: _col0 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Group By Operator - keys: _col0 (type: string) - mode: hash - outputColumnNames: _col0 + Reduce Output Operator + sort order: Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Dynamic Partitioning Event Operator - Target Input: srcpart - Partition key expr: ds - Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE - Target column: ds - Target Vertex: Map 1 + Execution mode: vectorized Stage: Stage-0 Fetch Operator @@ -4244,21 +4209,18 @@ Processor Tree: ListSink +Warning: Map Join MAPJOIN[24][bigTable=?] in task 'Map 1' is a cross product PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: default@srcpart PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -PREHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### POSTHOOK: query: select count(*) from srcpart join (select ds as ds, ds as date from srcpart group by ds) s on (srcpart.ds = s.ds) where s.date = '2008-04-08' POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11 POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11 -POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=12 #### A masked pattern was here #### 1000 PREHOOK: query: select count(*) from srcpart where ds = '2008-04-08' Index: ql/src/test/results/clientpositive/tez/vectorized_mapjoin.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorized_mapjoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorized_mapjoin.q.out (working copy) @@ -22,14 +22,14 @@ Map Operator Tree: TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -39,11 +39,11 @@ outputColumnNames: _col0, _col1 input vertices: 1 Map 3 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: int), _col1 (type: int), (_col0 + _col1) (type: int) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col0), max(_col1), min(_col0), avg(_col2) mode: hash @@ -58,19 +58,19 @@ Map Operator Tree: TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + Map-reduce partition columns: _col0 (type: int) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Reducer 2 Reduce Operator Tree: Index: ql/src/test/results/clientpositive/tez/vectorized_nested_mapjoin.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorized_nested_mapjoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorized_nested_mapjoin.q.out (working copy) @@ -18,33 +18,33 @@ Map Operator Tree: TableScan alias: v1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (ctinyint is not null and csmallint is not null) (type: boolean) - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), csmallint (type: smallint), cdouble (type: double) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint) sort order: + Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: smallint), _col2 (type: double) Execution mode: vectorized Map 2 Map Operator Tree: TableScan alias: v1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ctinyint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -54,11 +54,11 @@ outputColumnNames: _col1, _col2 input vertices: 0 Map 1 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col1 (type: smallint), _col2 (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -68,11 +68,11 @@ outputColumnNames: _col1 input vertices: 1 Map 4 - Statistics: Num rows: 7433 Data size: 228226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col1 (type: double) outputColumnNames: _col0 - Statistics: Num rows: 7433 Data size: 228226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash @@ -87,19 +87,19 @@ Map Operator Tree: TableScan alias: v1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: csmallint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: csmallint (type: smallint) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: smallint) sort order: + Map-reduce partition columns: _col0 (type: smallint) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Reducer 3 Reduce Operator Tree: Index: ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out (working copy) @@ -304,6 +304,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -321,6 +333,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -340,7 +384,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -682,6 +726,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -699,6 +755,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -718,7 +793,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -904,10 +979,22 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col3 (type: int) - outputColumnNames: _col2, _col1, _col5 + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int) + outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -927,7 +1014,7 @@ columns _col0,_col1,_col2 columns.types string:string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1168,6 +1255,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1185,6 +1284,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1204,7 +1335,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1457,6 +1588,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -1474,6 +1617,39 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), (_col5 - _wcol2) (type: int) @@ -1493,7 +1669,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1756,6 +1932,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -1798,6 +1986,39 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col2, 1, _col2 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col2 (type: int), (_col2 - _wcol2) (type: int) @@ -1817,7 +2038,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2092,6 +2313,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -2131,7 +2364,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types int:string:string:string:string:int:string:double:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2423,7 +2656,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types int:string:string:string:string:int:string:double:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2438,6 +2671,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -2591,7 +2836,21 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -2656,6 +2915,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -2672,6 +2944,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -2691,7 +2982,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2866,7 +3157,21 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -2932,6 +3237,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -2949,6 +3267,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -2968,7 +3318,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3212,6 +3562,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3229,6 +3591,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -3248,7 +3642,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3437,8 +3831,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -3498,16 +3892,49 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -3517,10 +3944,30 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3538,6 +3985,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -3557,7 +4036,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3831,6 +4310,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -3848,6 +4339,30 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol1 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: bigint), _wcol1 (type: double) @@ -3867,7 +4382,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:bigint:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4285,6 +4800,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -4326,6 +4853,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -4345,7 +4917,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types string:string:int:int:bigint:double:double:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4546,6 +5118,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -4585,7 +5169,7 @@ columns _col0,_col1,_col2 columns.types string:string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4742,7 +5326,6 @@ Edges: Reducer 2 <- Map 1 (SIMPLE_EDGE) Reducer 3 <- Reducer 2 (SIMPLE_EDGE) - Reducer 4 <- Reducer 3 (SIMPLE_EDGE) #### A masked pattern was here #### Vertices: Map 1 @@ -4764,7 +5347,7 @@ Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) sort order: ++ - Map-reduce partition columns: _col0 (type: string), _col1 (type: string) + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 value expressions: _col2 (type: double) @@ -4828,23 +5411,19 @@ mode: mergepartial outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE - tag: -1 - value expressions: _col1 (type: string), _col2 (type: double) - auto parallelism: true - Execution mode: vectorized - Reducer 3 - Needs Tagging: false - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1 (type: double) - outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: mfgr_price_view + output shape: _col0: string, _col1: string, _col2: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col0 + output shape: _col0: string, _col1: string, _col2: double + partition by: _col0 + raw input shape: Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) @@ -4854,7 +5433,7 @@ tag: -1 value expressions: _col2 (type: double) auto parallelism: true - Reducer 4 + Reducer 3 Needs Tagging: false Reduce Operator Tree: Select Operator @@ -4862,6 +5441,24 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~ Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: double), _wcol0 (type: double) @@ -4881,7 +5478,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5292,6 +5889,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -5317,6 +5926,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -5356,23 +5997,84 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(5)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE - Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ - Map-reduce partition columns: _col2 (type: string) + Select Operator + expressions: _col1 (type: string), _col2 (type: string), _col5 (type: int), _wcol0 (type: bigint) + outputColumnNames: _col1, _col2, _col5, _wcol0 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE - tag: -1 - value expressions: _wcol0 (type: bigint), _col5 (type: int) - auto parallelism: true + Reduce Output Operator + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ + Map-reduce partition columns: _col2 (type: string) + Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + tag: -1 + value expressions: _wcol0 (type: bigint), _col5 (type: int) + auto parallelism: true Reducer 5 Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) + expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) outputColumnNames: _col0, _col2, _col3, _col6 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: bigint, _col2: string, _col3: string, _col6: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col3, _col2 + partition by: _col3 + raw input shape: + window functions: + window function definition + alias: _wcol1 + arguments: _col3, _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col3, _col2 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol3 + arguments: _col3, _col2 + name: cume_dist + window function: GenericUDAFCumeDistEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol4 + arguments: _col6, true + name: first_value + window function: GenericUDAFFirstValueEvaluator + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col3 (type: string), _col2 (type: string), _col6 (type: int), UDFToInteger(round(_col0, 1)) (type: int), _wcol1 (type: int), _wcol2 (type: int), _wcol3 (type: double), _wcol4 (type: int) @@ -5740,8 +6442,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -5805,12 +6507,52 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -5824,10 +6566,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -5841,6 +6603,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -5860,7 +6654,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -6100,8 +6894,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6165,10 +6959,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6182,10 +6995,22 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6199,6 +7024,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -6216,6 +7053,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -6235,7 +7104,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -6458,8 +7327,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6523,10 +7392,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6540,6 +7428,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -6557,6 +7464,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -6576,7 +7515,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -6811,8 +7750,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6876,10 +7815,29 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6893,12 +7851,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6912,10 +7896,23 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6929,6 +7926,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -6948,7 +7977,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -7214,8 +8243,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7279,12 +8308,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7298,10 +8360,30 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7315,6 +8397,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -7334,7 +8448,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:bigint:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -7566,8 +8680,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7631,12 +8745,45 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7650,6 +8797,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: string) @@ -7667,6 +8827,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -7686,7 +8878,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:bigint:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/tez/vectorized_shufflejoin.q.out =================================================================== --- ql/src/test/results/clientpositive/tez/vectorized_shufflejoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/tez/vectorized_shufflejoin.q.out (working copy) @@ -22,37 +22,37 @@ Map Operator Tree: TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + Map-reduce partition columns: _col0 (type: int) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Map 4 Map Operator Tree: TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + Map-reduce partition columns: _col0 (type: int) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Reducer 2 Reduce Operator Tree: @@ -63,11 +63,11 @@ 0 _col0 (type: int) 1 _col0 (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: int), _col1 (type: int), (_col0 + _col1) (type: int) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col0), max(_col1), min(_col0), avg(_col2) mode: hash Index: ql/src/test/results/clientpositive/transform_ppr1.q.out =================================================================== --- ql/src/test/results/clientpositive/transform_ppr1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/transform_ppr1.q.out (working copy) @@ -348,7 +348,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/transform_ppr2.q.out =================================================================== --- ql/src/test/results/clientpositive/transform_ppr2.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/transform_ppr2.q.out (working copy) @@ -256,7 +256,7 @@ columns _col0,_col1 columns.types string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/truncate_column_list_bucket.q.out =================================================================== --- ql/src/test/results/clientpositive/truncate_column_list_bucket.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/truncate_column_list_bucket.q.out (working copy) @@ -127,7 +127,7 @@ columns _col0,_col1,_col2 columns.types string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -266,7 +266,7 @@ columns _col0,_col1,_col2 columns.types string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/udaf_covar_pop.q.out =================================================================== --- ql/src/test/results/clientpositive/udaf_covar_pop.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udaf_covar_pop.q.out (working copy) @@ -80,15 +80,15 @@ 4 0.0 5 0.0 6 0.0 -PREHOOK: query: SELECT covar_pop(b, c) FROM covar_tab +PREHOOK: query: SELECT ROUND(covar_pop(b, c), 5) FROM covar_tab PREHOOK: type: QUERY PREHOOK: Input: default@covar_tab #### A masked pattern was here #### -POSTHOOK: query: SELECT covar_pop(b, c) FROM covar_tab +POSTHOOK: query: SELECT ROUND(covar_pop(b, c), 5) FROM covar_tab POSTHOOK: type: QUERY POSTHOOK: Input: default@covar_tab #### A masked pattern was here #### -3.624999999999999 +3.625 PREHOOK: query: DROP TABLE covar_tab PREHOOK: type: DROPTABLE PREHOOK: Input: default@covar_tab Index: ql/src/test/results/clientpositive/udaf_covar_samp.q.out =================================================================== --- ql/src/test/results/clientpositive/udaf_covar_samp.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udaf_covar_samp.q.out (working copy) @@ -80,15 +80,15 @@ 4 0.0 5 0.0 6 0.0 -PREHOOK: query: SELECT covar_samp(b, c) FROM covar_tab +PREHOOK: query: SELECT ROUND(covar_samp(b, c), 5) FROM covar_tab PREHOOK: type: QUERY PREHOOK: Input: default@covar_tab #### A masked pattern was here #### -POSTHOOK: query: SELECT covar_samp(b, c) FROM covar_tab +POSTHOOK: query: SELECT ROUND(covar_samp(b, c), 5) FROM covar_tab POSTHOOK: type: QUERY POSTHOOK: Input: default@covar_tab #### A masked pattern was here #### -4.833333333333332 +4.83333 PREHOOK: query: DROP TABLE covar_tab PREHOOK: type: DROPTABLE PREHOOK: Input: default@covar_tab Index: ql/src/test/results/clientpositive/udf_add_months.q.out =================================================================== --- ql/src/test/results/clientpositive/udf_add_months.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udf_add_months.q.out (working copy) @@ -12,3 +12,118 @@ Example: > SELECT add_months('2009-08-31', 1) FROM src LIMIT 1; '2009-09-30' +PREHOOK: query: explain select add_months('2014-01-14', 1) +PREHOOK: type: QUERY +POSTHOOK: query: explain select add_months('2014-01-14', 1) +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-0 is a root stage + +STAGE PLANS: + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + TableScan + alias: _dummy_table + Row Limit Per Split: 1 + Statistics: Num rows: 0 Data size: 1 Basic stats: PARTIAL Column stats: COMPLETE + Select Operator + expressions: '2014-02-14' (type: string) + outputColumnNames: _col0 + Statistics: Num rows: 0 Data size: 1 Basic stats: PARTIAL Column stats: COMPLETE + ListSink + +PREHOOK: query: select +add_months('2014-01-14', 1), +add_months('2014-01-31', 1), +add_months('2014-02-28', -1), +add_months('2014-02-28', 2), +add_months('2014-04-30', -2), +add_months('2015-02-28', 12), +add_months('2016-02-29', -12), +add_months('2016-01-29', 1), +add_months('2016-02-29', -1), +add_months('2014-01-32', 1), +add_months('01/14/2014', 1), +add_months(cast(null as string), 1), +add_months('2014-01-14', cast(null as int)) +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +POSTHOOK: query: select +add_months('2014-01-14', 1), +add_months('2014-01-31', 1), +add_months('2014-02-28', -1), +add_months('2014-02-28', 2), +add_months('2014-04-30', -2), +add_months('2015-02-28', 12), +add_months('2016-02-29', -12), +add_months('2016-01-29', 1), +add_months('2016-02-29', -1), +add_months('2014-01-32', 1), +add_months('01/14/2014', 1), +add_months(cast(null as string), 1), +add_months('2014-01-14', cast(null as int)) +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +2014-02-14 2014-02-28 2014-01-31 2014-04-30 2014-02-28 2016-02-29 2015-02-28 2016-02-29 2016-01-31 2014-03-01 NULL NULL NULL +PREHOOK: query: select +add_months('2014-01-14 10:30:00', 1), +add_months('2014-01-31 10:30:00', 1), +add_months('2014-02-28 10:30:00', -1), +add_months('2014-02-28 16:30:00', 2), +add_months('2014-04-30 10:30:00', -2), +add_months('2015-02-28 10:30:00', 12), +add_months('2016-02-29 10:30:00', -12), +add_months('2016-01-29 10:30:00', 1), +add_months('2016-02-29 10:30:00', -1), +add_months('2014-01-32 10:30:00', 1) +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +POSTHOOK: query: select +add_months('2014-01-14 10:30:00', 1), +add_months('2014-01-31 10:30:00', 1), +add_months('2014-02-28 10:30:00', -1), +add_months('2014-02-28 16:30:00', 2), +add_months('2014-04-30 10:30:00', -2), +add_months('2015-02-28 10:30:00', 12), +add_months('2016-02-29 10:30:00', -12), +add_months('2016-01-29 10:30:00', 1), +add_months('2016-02-29 10:30:00', -1), +add_months('2014-01-32 10:30:00', 1) +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +2014-02-14 2014-02-28 2014-01-31 2014-04-30 2014-02-28 2016-02-29 2015-02-28 2016-02-29 2016-01-31 2014-03-01 +PREHOOK: query: select +add_months(cast('2014-01-14 10:30:00' as timestamp), 1), +add_months(cast('2014-01-31 10:30:00' as timestamp), 1), +add_months(cast('2014-02-28 10:30:00' as timestamp), -1), +add_months(cast('2014-02-28 16:30:00' as timestamp), 2), +add_months(cast('2014-04-30 10:30:00' as timestamp), -2), +add_months(cast('2015-02-28 10:30:00' as timestamp), 12), +add_months(cast('2016-02-29 10:30:00' as timestamp), -12), +add_months(cast('2016-01-29 10:30:00' as timestamp), 1), +add_months(cast('2016-02-29 10:30:00' as timestamp), -1), +add_months(cast(null as timestamp), 1) +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +POSTHOOK: query: select +add_months(cast('2014-01-14 10:30:00' as timestamp), 1), +add_months(cast('2014-01-31 10:30:00' as timestamp), 1), +add_months(cast('2014-02-28 10:30:00' as timestamp), -1), +add_months(cast('2014-02-28 16:30:00' as timestamp), 2), +add_months(cast('2014-04-30 10:30:00' as timestamp), -2), +add_months(cast('2015-02-28 10:30:00' as timestamp), 12), +add_months(cast('2016-02-29 10:30:00' as timestamp), -12), +add_months(cast('2016-01-29 10:30:00' as timestamp), 1), +add_months(cast('2016-02-29 10:30:00' as timestamp), -1), +add_months(cast(null as timestamp), 1) +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +2014-02-14 2014-02-28 2014-01-31 2014-04-30 2014-02-28 2016-02-29 2015-02-28 2016-02-29 2016-01-31 NULL Index: ql/src/test/results/clientpositive/udf_explode.q.out =================================================================== --- ql/src/test/results/clientpositive/udf_explode.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udf_explode.q.out (working copy) @@ -218,7 +218,7 @@ columns _col0,_col1 columns.types int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -492,7 +492,7 @@ columns _col0,_col1,_col2 columns.types int:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/udf_instr.q.out =================================================================== --- ql/src/test/results/clientpositive/udf_instr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udf_instr.q.out (working copy) @@ -56,7 +56,7 @@ Row Limit Per Split: 1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE Select Operator - expressions: 1 (type: int), 0 (type: int), 2 (type: int), 2 (type: int), 0 (type: int), 0 (type: int), 2 (type: int), 3 (type: int), 4 (type: int), 2 (type: int), 3 (type: int), instr(null'abc') (type: int), instr('abcd'null) (type: int) + expressions: 1 (type: int), 0 (type: int), 2 (type: int), 2 (type: int), 0 (type: int), 0 (type: int), 2 (type: int), 3 (type: int), 4 (type: int), 2 (type: int), 3 (type: int), instr(null, 'abc') (type: int), instr('abcd', null) (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 Statistics: Num rows: 500 Data size: 26000 Basic stats: COMPLETE Column stats: COMPLETE ListSink Index: ql/src/test/results/clientpositive/udf_last_day.q.out =================================================================== --- ql/src/test/results/clientpositive/udf_last_day.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udf_last_day.q.out (working copy) @@ -12,3 +12,141 @@ Example: > SELECT last_day('2009-01-12') FROM src LIMIT 1; '2009-01-31' +PREHOOK: query: explain select last_day('2015-02-05') +PREHOOK: type: QUERY +POSTHOOK: query: explain select last_day('2015-02-05') +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-0 is a root stage + +STAGE PLANS: + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + TableScan + alias: _dummy_table + Row Limit Per Split: 1 + Statistics: Num rows: 0 Data size: 1 Basic stats: PARTIAL Column stats: COMPLETE + Select Operator + expressions: '2015-02-28' (type: string) + outputColumnNames: _col0 + Statistics: Num rows: 0 Data size: 1 Basic stats: PARTIAL Column stats: COMPLETE + ListSink + +PREHOOK: query: select +last_day('2014-01-01'), +last_day('2014-01-14'), +last_day('2014-01-31'), +last_day('2014-02-02'), +last_day('2014-02-28'), +last_day('2016-02-03'), +last_day('2016-02-28'), +last_day('2016-02-29'), +last_day('2014-01-34'), +last_day(cast(null as string)), +last_day('01/29/2014') +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +POSTHOOK: query: select +last_day('2014-01-01'), +last_day('2014-01-14'), +last_day('2014-01-31'), +last_day('2014-02-02'), +last_day('2014-02-28'), +last_day('2016-02-03'), +last_day('2016-02-28'), +last_day('2016-02-29'), +last_day('2014-01-34'), +last_day(cast(null as string)), +last_day('01/29/2014') +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +2014-01-31 2014-01-31 2014-01-31 2014-02-28 2014-02-28 2016-02-29 2016-02-29 2016-02-29 2014-02-28 NULL NULL +PREHOOK: query: select +last_day('2014-01-01 10:30:45'), +last_day('2014-01-14 10:30:45'), +last_day('2014-01-31 10:30:45'), +last_day('2014-02-02 10:30:45'), +last_day('2014-02-28 10:30:45'), +last_day('2016-02-03 10:30:45'), +last_day('2016-02-28 10:30:45'), +last_day('2016-02-29 10:30:45'), +last_day('2014-01-34 10:30:45'), +last_day(cast(null as string)), +last_day('01/29/2014 10:30:45') +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +POSTHOOK: query: select +last_day('2014-01-01 10:30:45'), +last_day('2014-01-14 10:30:45'), +last_day('2014-01-31 10:30:45'), +last_day('2014-02-02 10:30:45'), +last_day('2014-02-28 10:30:45'), +last_day('2016-02-03 10:30:45'), +last_day('2016-02-28 10:30:45'), +last_day('2016-02-29 10:30:45'), +last_day('2014-01-34 10:30:45'), +last_day(cast(null as string)), +last_day('01/29/2014 10:30:45') +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +2014-01-31 2014-01-31 2014-01-31 2014-02-28 2014-02-28 2016-02-29 2016-02-29 2016-02-29 2014-02-28 NULL NULL +PREHOOK: query: select +last_day(cast('2014-01-01' as date)), +last_day(cast('2014-01-14' as date)), +last_day(cast('2014-01-31' as date)), +last_day(cast('2014-02-02' as date)), +last_day(cast('2014-02-28' as date)), +last_day(cast('2016-02-03' as date)), +last_day(cast('2016-02-28' as date)), +last_day(cast('2016-02-29' as date)), +last_day(cast(null as date)) +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +POSTHOOK: query: select +last_day(cast('2014-01-01' as date)), +last_day(cast('2014-01-14' as date)), +last_day(cast('2014-01-31' as date)), +last_day(cast('2014-02-02' as date)), +last_day(cast('2014-02-28' as date)), +last_day(cast('2016-02-03' as date)), +last_day(cast('2016-02-28' as date)), +last_day(cast('2016-02-29' as date)), +last_day(cast(null as date)) +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +2014-01-31 2014-01-31 2014-01-31 2014-02-28 2014-02-28 2016-02-29 2016-02-29 2016-02-29 NULL +PREHOOK: query: select +last_day(cast('2014-01-01 10:30:45' as timestamp)), +last_day(cast('2014-01-14 10:30:45' as timestamp)), +last_day(cast('2014-01-31 10:30:45' as timestamp)), +last_day(cast('2014-02-02 10:30:45' as timestamp)), +last_day(cast('2014-02-28 10:30:45' as timestamp)), +last_day(cast('2016-02-03 10:30:45' as timestamp)), +last_day(cast('2016-02-28 10:30:45' as timestamp)), +last_day(cast('2016-02-29 10:30:45' as timestamp)), +last_day(cast(null as timestamp)) +PREHOOK: type: QUERY +PREHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +POSTHOOK: query: select +last_day(cast('2014-01-01 10:30:45' as timestamp)), +last_day(cast('2014-01-14 10:30:45' as timestamp)), +last_day(cast('2014-01-31 10:30:45' as timestamp)), +last_day(cast('2014-02-02 10:30:45' as timestamp)), +last_day(cast('2014-02-28 10:30:45' as timestamp)), +last_day(cast('2016-02-03 10:30:45' as timestamp)), +last_day(cast('2016-02-28 10:30:45' as timestamp)), +last_day(cast('2016-02-29 10:30:45' as timestamp)), +last_day(cast(null as timestamp)) +POSTHOOK: type: QUERY +POSTHOOK: Input: _dummy_database@_dummy_table +#### A masked pattern was here #### +2014-01-31 2014-01-31 2014-01-31 2014-02-28 2014-02-28 2016-02-29 2016-02-29 2016-02-29 NULL Index: ql/src/test/results/clientpositive/udf_locate.q.out =================================================================== --- ql/src/test/results/clientpositive/udf_locate.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udf_locate.q.out (working copy) @@ -64,7 +64,7 @@ Row Limit Per Split: 1 Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE Select Operator - expressions: 1 (type: int), 0 (type: int), 2 (type: int), 2 (type: int), 4 (type: int), 4 (type: int), 0 (type: int), 0 (type: int), 2 (type: int), 3 (type: int), 4 (type: int), 2 (type: int), 3 (type: int), locate(null'abc') (type: int), locate('abc'null) (type: int), locate('abc''abcd'null) (type: int), 0 (type: int) + expressions: 1 (type: int), 0 (type: int), 2 (type: int), 2 (type: int), 4 (type: int), 4 (type: int), 0 (type: int), 0 (type: int), 2 (type: int), 3 (type: int), 4 (type: int), 2 (type: int), 3 (type: int), locate(null, 'abc') (type: int), locate('abc', null) (type: int), locate('abc', 'abcd', null) (type: int), 0 (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16 Statistics: Num rows: 500 Data size: 34000 Basic stats: COMPLETE Column stats: COMPLETE ListSink Index: ql/src/test/results/clientpositive/udtf_explode.q.out =================================================================== --- ql/src/test/results/clientpositive/udtf_explode.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/udtf_explode.q.out (working copy) @@ -284,7 +284,7 @@ columns _col0,_col1 columns.types int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -594,7 +594,7 @@ columns _col0,_col1,_col2 columns.types int:string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/union22.q.out =================================================================== --- ql/src/test/results/clientpositive/union22.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union22.q.out (working copy) @@ -83,7 +83,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF Index: ql/src/test/results/clientpositive/union24.q.out =================================================================== --- ql/src/test/results/clientpositive/union24.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union24.q.out (working copy) @@ -69,9 +69,9 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION - TOK_UNION - TOK_UNION + TOK_UNIONALL + TOK_UNIONALL + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -320,7 +320,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -355,7 +355,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -390,7 +390,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -415,7 +415,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -666,8 +666,8 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION - TOK_UNION + TOK_UNIONALL + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -969,7 +969,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1004,7 +1004,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1029,7 +1029,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1225,8 +1225,8 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION - TOK_UNION + TOK_UNIONALL + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -1598,7 +1598,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1633,7 +1633,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1658,7 +1658,7 @@ columns _col0,_col1 columns.types string:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/union3.q.out =================================================================== --- ql/src/test/results/clientpositive/union3.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union3.q.out (working copy) @@ -1,52 +1,48 @@ -PREHOOK: query: -- SORT_BEFORE_DIFF +PREHOOK: query: -- SORT_QUERY_RESULTS explain SELECT * FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a PREHOOK: type: QUERY -POSTHOOK: query: -- SORT_BEFORE_DIFF +POSTHOOK: query: -- SORT_QUERY_RESULTS explain SELECT * FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage - Stage-2 depends on stages: Stage-1 - Stage-3 depends on stages: Stage-2, Stage-5, Stage-6, Stage-7 + Stage-2 depends on stages: Stage-1, Stage-3, Stage-4, Stage-5 + Stage-3 is a root stage Stage-4 is a root stage - Stage-5 depends on stages: Stage-4 - Stage-6 is a root stage - Stage-7 is a root stage - Stage-0 depends on stages: Stage-3 + Stage-5 is a root stage + Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-1 @@ -67,79 +63,66 @@ Limit Number of rows: 1 Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + Select Operator + expressions: 1 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE + File Output Operator + compressed: false + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe Stage: Stage-2 Map Reduce Map Operator Tree: TableScan - Reduce Output Operator - key expressions: 1 (type: int) - sort order: + - Map-reduce partition columns: 1 (type: int) - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: int) - outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - - Stage: Stage-3 - Map Reduce - Map Operator Tree: - TableScan Union Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe TableScan Union Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe TableScan Union Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe TableScan Union Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false + Reduce Output Operator + key expressions: _col0 (type: int) + sort order: + + Map-reduce partition columns: _col0 (type: int) Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - table: - input format: org.apache.hadoop.mapred.TextInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat - serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Reduce Operator Tree: + Select Operator + expressions: KEY.reducesinkkey0 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE + File Output Operator + compressed: false + Statistics: Num rows: 4 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE + table: + input format: org.apache.hadoop.mapred.TextInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat + serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe - Stage: Stage-4 + Stage: Stage-3 Map Reduce Map Operator Tree: TableScan @@ -157,38 +140,21 @@ Limit Number of rows: 1 Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + Select Operator + expressions: 2 (type: int) + outputColumnNames: _col0 + Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE + File Output Operator + compressed: false + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - Stage: Stage-5 + Stage: Stage-4 Map Reduce Map Operator Tree: TableScan - Reduce Output Operator - key expressions: 2 (type: int) - sort order: + - Map-reduce partition columns: 2 (type: int) - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: int) - outputColumnNames: _col0 - Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE - File Output Operator - compressed: false - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - - Stage: Stage-6 - Map Reduce - Map Operator Tree: - TableScan alias: src Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE Select Operator @@ -214,7 +180,7 @@ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - Stage: Stage-7 + Stage: Stage-5 Map Reduce Map Operator Tree: TableScan @@ -262,17 +228,16 @@ FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a PREHOOK: type: QUERY PREHOOK: Input: default@src @@ -282,27 +247,26 @@ FROM ( SELECT 1 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 2 AS id FROM (SELECT * FROM src LIMIT 1) s1 - CLUSTER BY id UNION ALL SELECT 3 AS id FROM (SELECT * FROM src LIMIT 1) s2 UNION ALL SELECT 4 AS id FROM (SELECT * FROM src LIMIT 1) s2 + CLUSTER BY id ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@union_out POSTHOOK: Lineage: union_out.id EXPRESSION [] -PREHOOK: query: select * from union_out cluster by id +PREHOOK: query: select * from union_out PREHOOK: type: QUERY PREHOOK: Input: default@union_out #### A masked pattern was here #### -POSTHOOK: query: select * from union_out cluster by id +POSTHOOK: query: select * from union_out POSTHOOK: type: QUERY POSTHOOK: Input: default@union_out #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/union4.q.out =================================================================== --- ql/src/test/results/clientpositive/union4.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union4.q.out (working copy) @@ -1,13 +1,17 @@ -PREHOOK: query: -- union case: both subqueries are map-reduce jobs on same input, followed by filesink +PREHOOK: query: -- SORT_QUERY_RESULTS +-- union case: both subqueries are map-reduce jobs on same input, followed by filesink + create table tmptable(key string, value int) PREHOOK: type: CREATETABLE PREHOOK: Output: database:default PREHOOK: Output: default@tmptable -POSTHOOK: query: -- union case: both subqueries are map-reduce jobs on same input, followed by filesink +POSTHOOK: query: -- SORT_QUERY_RESULTS +-- union case: both subqueries are map-reduce jobs on same input, followed by filesink + create table tmptable(key string, value int) POSTHOOK: type: CREATETABLE POSTHOOK: Output: database:default Index: ql/src/test/results/clientpositive/union_null.q.out =================================================================== --- ql/src/test/results/clientpositive/union_null.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union_null.q.out (working copy) @@ -1,14 +1,14 @@ PREHOOK: query: -- SORT_BEFORE_DIFF -- HIVE-2901 -select x from (select value as x from src order by x limit 5 union all select NULL as x from src limit 5) a +select x from (select * from (select value as x from src order by x limit 5)a union all select * from (select NULL as x from src limit 5)b )a PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### POSTHOOK: query: -- SORT_BEFORE_DIFF -- HIVE-2901 -select x from (select value as x from src order by x limit 5 union all select NULL as x from src limit 5) a +select x from (select * from (select value as x from src order by x limit 5)a union all select * from (select NULL as x from src limit 5)b )a POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### @@ -23,12 +23,12 @@ NULL NULL PREHOOK: query: -- HIVE-4837 -select * from (select null as N from src1 group by key UNION ALL select null as N from src1 group by key ) a +select * from (select * from (select null as N from src1 group by key)a UNION ALL select * from (select null as N from src1 group by key)b ) a PREHOOK: type: QUERY PREHOOK: Input: default@src1 #### A masked pattern was here #### POSTHOOK: query: -- HIVE-4837 -select * from (select null as N from src1 group by key UNION ALL select null as N from src1 group by key ) a +select * from (select * from (select null as N from src1 group by key)a UNION ALL select * from (select null as N from src1 group by key)b ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@src1 #### A masked pattern was here #### Index: ql/src/test/results/clientpositive/union_ppr.q.out =================================================================== --- ql/src/test/results/clientpositive/union_ppr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union_ppr.q.out (working copy) @@ -25,7 +25,7 @@ TOK_QUERY TOK_FROM TOK_SUBQUERY - TOK_UNION + TOK_UNIONALL TOK_QUERY TOK_FROM TOK_TABREF @@ -282,7 +282,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:string:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/union_remove_25.q.out =================================================================== --- ql/src/test/results/clientpositive/union_remove_25.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union_remove_25.q.out (working copy) @@ -257,18 +257,18 @@ insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a PREHOOK: type: QUERY POSTHOOK: query: explain insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -368,9 +368,9 @@ PREHOOK: query: insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a PREHOOK: type: QUERY PREHOOK: Input: default@srcpart @@ -380,9 +380,9 @@ POSTHOOK: query: insert overwrite table outputTbl2 partition(ds) SELECT * FROM ( - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)a UNION ALL - SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500 + select * from (SELECT key, value, ds from srcpart where ds='2008-04-08' limit 500)b ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart @@ -442,17 +442,17 @@ PREHOOK: query: explain insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a PREHOOK: type: QUERY POSTHOOK: query: explain insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a POSTHOOK: type: QUERY STAGE DEPENDENCIES: @@ -553,9 +553,9 @@ PREHOOK: query: insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a PREHOOK: type: QUERY PREHOOK: Input: default@srcpart @@ -565,9 +565,9 @@ POSTHOOK: query: insert overwrite table outputTbl3 partition(ds, hr) SELECT * FROM ( - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)a UNION ALL - SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000 + select * from (SELECT key, value, ds, hr from srcpart where ds='2008-04-08' limit 1000)b ) a POSTHOOK: type: QUERY POSTHOOK: Input: default@srcpart Index: ql/src/test/results/clientpositive/union_remove_6_subq.q.out =================================================================== --- ql/src/test/results/clientpositive/union_remove_6_subq.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union_remove_6_subq.q.out (working copy) @@ -546,6 +546,24 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: bigint + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: avg + window function: GenericUDAFAverageEvaluatorDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _wcol0 (type: double) Index: ql/src/test/results/clientpositive/union_top_level.q.out =================================================================== --- ql/src/test/results/clientpositive/union_top_level.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/union_top_level.q.out (working copy) @@ -2,21 +2,21 @@ -- top level explain -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: QUERY POSTHOOK: query: -- SORT_QUERY_RESULTS -- top level explain -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage @@ -171,19 +171,19 @@ Processor Tree: ListSink -PREHOOK: query: select key, 0 as value from src where key % 3 == 0 limit 3 +PREHOOK: query: select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### -POSTHOOK: query: select key, 0 as value from src where key % 3 == 0 limit 3 +POSTHOOK: query: select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### @@ -197,14 +197,14 @@ 484 1 86 2 PREHOOK: query: explain -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)a union all -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)b PREHOOK: type: QUERY POSTHOOK: query: explain -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)a union all -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)b POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage @@ -405,15 +405,15 @@ Processor Tree: ListSink -PREHOOK: query: select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +PREHOOK: query: select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)a union all -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)b PREHOOK: type: QUERY PREHOOK: Input: default@src #### A masked pattern was here #### -POSTHOOK: query: select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +POSTHOOK: query: select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)a union all -select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10 +select * from (select s1.key as k, s2.value as v from src s1 join src s2 on (s1.key = s2.key) limit 10)b POSTHOOK: type: QUERY POSTHOOK: Input: default@src #### A masked pattern was here #### @@ -440,20 +440,20 @@ PREHOOK: query: -- ctas explain create table union_top as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: CREATETABLE_AS_SELECT POSTHOOK: query: -- ctas explain create table union_top as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: CREATETABLE_AS_SELECT STAGE DEPENDENCIES: Stage-1 is a root stage @@ -670,21 +670,21 @@ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe PREHOOK: query: create table union_top as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: CREATETABLE_AS_SELECT PREHOOK: Input: default@src PREHOOK: Output: database:default PREHOOK: Output: default@union_top POSTHOOK: query: create table union_top as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: CREATETABLE_AS_SELECT POSTHOOK: Input: default@src POSTHOOK: Output: database:default @@ -715,20 +715,20 @@ PREHOOK: query: -- insert into explain insert into table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: QUERY POSTHOOK: query: -- insert into explain insert into table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage @@ -939,20 +939,20 @@ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe PREHOOK: query: insert into table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@union_top POSTHOOK: query: insert into table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@union_top @@ -977,19 +977,19 @@ 86 2 PREHOOK: query: explain insert overwrite table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: QUERY POSTHOOK: query: explain insert overwrite table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: QUERY STAGE DEPENDENCIES: Stage-1 is a root stage @@ -1200,20 +1200,20 @@ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe PREHOOK: query: insert overwrite table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: QUERY PREHOOK: Input: default@src PREHOOK: Output: default@union_top POSTHOOK: query: insert overwrite table union_top -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: QUERY POSTHOOK: Input: default@src POSTHOOK: Output: default@union_top @@ -1239,20 +1239,20 @@ PREHOOK: query: -- create view explain create view union_top_view as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: CREATEVIEW POSTHOOK: query: -- create view explain create view union_top_view as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: CREATEVIEW STAGE DEPENDENCIES: Stage-0 is a root stage @@ -1263,34 +1263,34 @@ Create View or replace: false columns: key string, value int - expanded text: select `src`.`key`, 0 as `value` from `default`.`src` where `src`.`key` % 3 == 0 limit 3 + expanded text: select `a`.`key`, `a`.`value` from (select `src`.`key`, 0 as `value` from `default`.`src` where `src`.`key` % 3 == 0 limit 3)`a` union all -select `src`.`key`, 1 as `value` from `default`.`src` where `src`.`key` % 3 == 1 limit 3 +select `b`.`key`, `b`.`value` from (select `src`.`key`, 1 as `value` from `default`.`src` where `src`.`key` % 3 == 1 limit 3)`b` union all -select `src`.`key`, 2 as `value` from `default`.`src` where `src`.`key` % 3 == 2 limit 3 +select `c`.`key`, `c`.`value` from (select `src`.`key`, 2 as `value` from `default`.`src` where `src`.`key` % 3 == 2 limit 3)`c` name: default.union_top_view - original text: select key, 0 as value from src where key % 3 == 0 limit 3 + original text: select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: query: create view union_top_view as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c PREHOOK: type: CREATEVIEW PREHOOK: Input: default@src PREHOOK: Output: database:default PREHOOK: Output: default@union_top_view POSTHOOK: query: create view union_top_view as -select key, 0 as value from src where key % 3 == 0 limit 3 +select * from (select key, 0 as value from src where key % 3 == 0 limit 3)a union all -select key, 1 as value from src where key % 3 == 1 limit 3 +select * from (select key, 1 as value from src where key % 3 == 1 limit 3)b union all -select key, 2 as value from src where key % 3 == 2 limit 3 +select * from (select key, 2 as value from src where key % 3 == 2 limit 3)c POSTHOOK: type: CREATEVIEW POSTHOOK: Input: default@src POSTHOOK: Output: database:default Index: ql/src/test/results/clientpositive/varchar_union1.q.out =================================================================== --- ql/src/test/results/clientpositive/varchar_union1.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/varchar_union1.q.out (working copy) @@ -95,9 +95,6 @@ POSTHOOK: Input: default@varchar_union1_vc1 #### A masked pattern was here #### 1 abc -1 abc -2 abc -3 abc PREHOOK: query: -- union varchar with different length varchar select * from ( select * from varchar_union1_vc1 @@ -119,9 +116,6 @@ POSTHOOK: Input: default@varchar_union1_vc2 #### A masked pattern was here #### 1 abc -1 abc -2 abc -3 abc PREHOOK: query: -- union varchar with string select * from ( select * from varchar_union1_vc1 @@ -143,9 +137,6 @@ POSTHOOK: Input: default@varchar_union1_vc1 #### A masked pattern was here #### 1 abc -1 abc -2 abc -3 abc PREHOOK: query: drop table varchar_union1_vc1 PREHOOK: type: DROPTABLE PREHOOK: Input: default@varchar_union1_vc1 Index: ql/src/test/results/clientpositive/vector_char_simple.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_char_simple.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_char_simple.q.out (working copy) @@ -251,34 +251,34 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator sort order: - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: int) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: CAST( _col0 AS CHAR(12) (type: char(12)) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat Index: ql/src/test/results/clientpositive/vector_coalesce.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_coalesce.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_coalesce.q.out (working copy) @@ -16,20 +16,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cdouble is null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: null (type: void), cstring1 (type: string), cint (type: int), cfloat (type: float), csmallint (type: smallint), COALESCE(null,cstring1,cint,cfloat,csmallint) (type: string) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -82,20 +82,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ctinyint is null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: null (type: void), cdouble (type: double), cint (type: int), COALESCE((UDFToInteger(null) + 10),(cdouble + log2(cint)),0) (type: double) outputColumnNames: _col0, _col1, _col2, _col3 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -148,20 +148,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (cfloat is null and cbigint is null) (type: boolean) - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: null (type: void), null (type: void), COALESCE(null,null,0) (type: float) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -214,20 +214,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (ctimestamp1 is not null or ctimestamp2 is not null) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), ctimestamp2 (type: timestamp), COALESCE(ctimestamp1,ctimestamp2) (type: timestamp) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vector_decimal_cast.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_decimal_cast.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_decimal_cast.q.out (working copy) @@ -12,20 +12,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cdouble is not null and cint is not null) and cboolean1 is not null) and ctimestamp1 is not null) (type: boolean) - Statistics: Num rows: 768 Data size: 23577 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 768 Data size: 165122 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double), cint (type: int), cboolean1 (type: boolean), ctimestamp1 (type: timestamp), CAST( cdouble AS decimal(20,10)) (type: decimal(20,10)), CAST( cint AS decimal(23,14)) (type: decimal(23,14)), CAST( cboolean1 AS decimal(5,2)) (type: decimal(5,2)), CAST( ctimestamp1 AS decimal(15,0)) (type: decimal(15,0)) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7 - Statistics: Num rows: 768 Data size: 23577 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 768 Data size: 165122 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vector_elt.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_elt.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_elt.q.out (working copy) @@ -16,20 +16,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (ctinyint > 0) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ((UDFToInteger(ctinyint) % 2) + 1) (type: int), cstring1 (type: string), cint (type: int), elt(((UDFToInteger(ctinyint) % 2) + 1), cstring1, cint) (type: string) outputColumnNames: _col0, _col1, _col2, _col3 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -100,7 +100,7 @@ Processor Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: COMPLETE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: COMPLETE Select Operator expressions: 'defg' (type: string), 'cc' (type: string), 'abc' (type: string), '2' (type: string), '12345' (type: string), '123456789012' (type: string), '1.25' (type: string), '16.0' (type: string), null (type: void), null (type: void) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9 Index: ql/src/test/results/clientpositive/vector_if_expr.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_if_expr.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_if_expr.q.out (working copy) @@ -14,28 +14,28 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (cboolean1 is not null and cboolean1) (type: boolean) - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), if(cboolean1, 'first', 'second') (type: string) outputColumnNames: _col0, _col1 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean) sort order: + - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: string) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: boolean), VALUE._col0 (type: string) outputColumnNames: _col0, _col1 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vector_left_outer_join.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_left_outer_join.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_left_outer_join.q.out (working copy) @@ -35,11 +35,11 @@ $hdt$_0:$hdt$_1:c TableScan alias: c - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col1 (type: int) @@ -47,11 +47,11 @@ $hdt$_0:$hdt$_2:c TableScan alias: c - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: tinyint) @@ -62,11 +62,11 @@ Map Operator Tree: TableScan alias: c - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cint (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Left Outer Join0 to 1 @@ -74,14 +74,14 @@ 0 _col1 (type: int) 1 _col0 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 13516 Data size: 414960 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 13516 Data size: 2906160 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Left Outer Join0 to 1 keys: 0 _col0 (type: tinyint) 1 _col0 (type: tinyint) - Statistics: Num rows: 14867 Data size: 456456 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 14867 Data size: 3196776 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count() mode: hash Index: ql/src/test/results/clientpositive/vector_multi_insert.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_multi_insert.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_multi_insert.q.out (working copy) @@ -5,11 +5,11 @@ select rn from ( - select cast(1 as int) as rn from src limit 1 + select * from (select cast(1 as int) as rn from src limit 1)a union all - select cast(100 as int) as rn from src limit 1 + select * from (select cast(100 as int) as rn from src limit 1)b union all - select cast(10000 as int) as rn from src limit 1 + select * from (select cast(10000 as int) as rn from src limit 1)c ) t PREHOOK: type: CREATETABLE_AS_SELECT PREHOOK: Input: default@src @@ -22,11 +22,11 @@ select rn from ( - select cast(1 as int) as rn from src limit 1 + select * from (select cast(1 as int) as rn from src limit 1)a union all - select cast(100 as int) as rn from src limit 1 + select * from (select cast(100 as int) as rn from src limit 1)b union all - select cast(10000 as int) as rn from src limit 1 + select * from (select cast(10000 as int) as rn from src limit 1)c ) t POSTHOOK: type: CREATETABLE_AS_SELECT POSTHOOK: Input: default@src Index: ql/src/test/results/clientpositive/vector_varchar_simple.q.out =================================================================== --- ql/src/test/results/clientpositive/vector_varchar_simple.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vector_varchar_simple.q.out (working copy) @@ -251,34 +251,34 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator sort order: - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: int) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: int) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 10 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: CAST( _col0 AS varchar(25)) (type: varchar(25)) outputColumnNames: _col0 - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 10 Data size: 300 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10 Data size: 2150 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat output format: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat Index: ql/src/test/results/clientpositive/vectorization_0.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_0.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_0.q.out (working copy) @@ -27,11 +27,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: ctinyint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: min(ctinyint), max(ctinyint), count(ctinyint), count() mode: hash @@ -123,11 +123,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash @@ -228,11 +228,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: ctinyint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(ctinyint), variance(ctinyint), var_pop(ctinyint), var_samp(ctinyint), std(ctinyint), stddev(ctinyint), stddev_pop(ctinyint), stddev_samp(ctinyint) mode: hash @@ -340,11 +340,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: cbigint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: min(cbigint), max(cbigint), count(cbigint), count() mode: hash @@ -436,11 +436,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash @@ -541,11 +541,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: cbigint - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(cbigint), variance(cbigint), var_pop(cbigint), var_samp(cbigint), std(cbigint), stddev(cbigint), stddev_pop(cbigint), stddev_samp(cbigint) mode: hash @@ -653,11 +653,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float) outputColumnNames: cfloat - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: min(cfloat), max(cfloat), count(cfloat), count() mode: hash @@ -749,11 +749,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash @@ -854,11 +854,11 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float) outputColumnNames: cfloat - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(cfloat), variance(cfloat), var_pop(cfloat), var_samp(cfloat), std(cfloat), stddev(cfloat), stddev_pop(cfloat), stddev_samp(cfloat) mode: hash @@ -1004,14 +1004,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%b%') or ((79.553 <> UDFToDouble(cint)) or (UDFToDouble(cbigint) < cdouble))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint), cfloat (type: float), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0), stddev_pop(_col0), var_samp(_col0), count(), sum(_col1), min(_col2) mode: hash Index: ql/src/test/results/clientpositive/vectorization_13.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_13.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_13.q.out (working copy) @@ -75,25 +75,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cfloat < 3569.0) and ((10.175 >= cdouble) and (cboolean1 <> 1))) or ((UDFToDouble(ctimestamp1) > 11.0) and ((UDFToDouble(ctimestamp2) <> 12.0) and (UDFToDouble(ctinyint) < 9763215.5639)))) (type: boolean) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col1), sum(_col3), stddev_pop(_col3), stddev_pop(_col1), max(_col3), min(_col1) keys: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ Map-reduce partition columns: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: double), _col7 (type: struct), _col8 (type: struct), _col9 (type: float), _col10 (type: tinyint) Execution mode: vectorized Reduce Operator Tree: @@ -102,11 +102,11 @@ keys: KEY._col0 (type: boolean), KEY._col1 (type: tinyint), KEY._col2 (type: timestamp), KEY._col3 (type: float), KEY._col4 (type: string) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: boolean), _col1 (type: tinyint), (- _col6) (type: double), (79.553 * UDFToDouble(_col3)) (type: double), _col7 (type: double), _col8 (type: double), (UDFToDouble(((- _col1) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col1) + _col5))) / UDFToDouble(_col1)) (type: double), _col2 (type: timestamp), _col10 (type: tinyint), _col3 (type: float), _col4 (type: string), (- _col1) (type: tinyint), _col5 (type: tinyint), ((- _col1) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col1) + _col5))) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -121,19 +121,19 @@ Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: tinyint), _col7 (type: tinyint), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: double), _col14 (type: double), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: float), _col19 (type: double), _col20 (type: tinyint) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: boolean), KEY.reducesinkkey1 (type: tinyint), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey3 (type: float), KEY.reducesinkkey4 (type: string), VALUE._col0 (type: tinyint), VALUE._col1 (type: tinyint), VALUE._col2 (type: tinyint), VALUE._col3 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col5 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: float), VALUE._col13 (type: double), VALUE._col14 (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 40 - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -332,25 +332,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cfloat < 3569.0) and ((10.175 >= cdouble) and (cboolean1 <> 1))) or ((UDFToDouble(ctimestamp1) > -1.388) and ((UDFToDouble(ctimestamp2) <> -1.3359999999999999) and (UDFToDouble(ctinyint) < 9763215.5639)))) (type: boolean) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col1), sum(_col3), stddev_pop(_col3), stddev_pop(_col1), max(_col3), min(_col1) keys: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ Map-reduce partition columns: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) - Statistics: Num rows: 2730 Data size: 83809 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: double), _col7 (type: struct), _col8 (type: struct), _col9 (type: float), _col10 (type: tinyint) Execution mode: vectorized Reduce Operator Tree: @@ -359,11 +359,11 @@ keys: KEY._col0 (type: boolean), KEY._col1 (type: tinyint), KEY._col2 (type: timestamp), KEY._col3 (type: float), KEY._col4 (type: string) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: boolean), _col1 (type: tinyint), (- _col6) (type: double), (79.553 * UDFToDouble(_col3)) (type: double), _col7 (type: double), _col8 (type: double), (UDFToDouble(((- _col1) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col1) + _col5))) / UDFToDouble(_col1)) (type: double), _col2 (type: timestamp), _col10 (type: tinyint), _col3 (type: float), _col4 (type: string), (- _col1) (type: tinyint), _col5 (type: tinyint), ((- _col1) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col1) + _col5))) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -378,19 +378,19 @@ Reduce Output Operator key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string) sort order: +++++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: tinyint), _col6 (type: tinyint), _col7 (type: tinyint), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: double), _col14 (type: double), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: float), _col19 (type: double), _col20 (type: tinyint) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: boolean), KEY.reducesinkkey1 (type: tinyint), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey3 (type: float), KEY.reducesinkkey4 (type: string), VALUE._col0 (type: tinyint), VALUE._col1 (type: tinyint), VALUE._col2 (type: tinyint), VALUE._col3 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col5 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: float), VALUE._col13 (type: double), VALUE._col14 (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 40 - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 40 Data size: 1200 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 40 Data size: 8600 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_14.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_14.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_14.q.out (working copy) @@ -75,25 +75,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((((UDFToLong(ctinyint) <= cbigint) and ((UDFToDouble(cint) <= cdouble) or (ctimestamp2 < ctimestamp1))) and (cdouble < UDFToDouble(ctinyint))) and ((cbigint > -257) or (cfloat < UDFToFloat(cint)))) (type: boolean) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string), cboolean1 (type: boolean), cdouble (type: double), (- (-26.28 + cdouble)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_samp(_col5), max(_col1), stddev_pop(_col1), count(_col1), var_pop(_col1), var_samp(_col1) keys: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double) sort order: +++++ Map-reduce partition columns: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE value expressions: _col5 (type: struct), _col6 (type: float), _col7 (type: struct), _col8 (type: bigint), _col9 (type: struct), _col10 (type: struct) Execution mode: vectorized Reduce Operator Tree: @@ -102,11 +102,11 @@ keys: KEY._col0 (type: timestamp), KEY._col1 (type: float), KEY._col2 (type: string), KEY._col3 (type: boolean), KEY._col4 (type: double) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: timestamp), _col1 (type: float), _col2 (type: string), _col3 (type: boolean), _col4 (type: double), (-26.28 + _col4) (type: double), (- (-26.28 + _col4)) (type: double), _col5 (type: double), (UDFToDouble(_col1) * -26.28) (type: double), _col6 (type: float), (- _col1) (type: float), (- _col6) (type: float), ((- (-26.28 + _col4)) / 10.175) (type: double), _col7 (type: double), _col8 (type: bigint), (- ((- (-26.28 + _col4)) / 10.175)) (type: double), (-1.389 % _col5) (type: double), (UDFToDouble(_col1) - _col4) (type: double), _col9 (type: double), (_col9 % 10.175) (type: double), _col10 (type: double), (- (UDFToDouble(_col1) - _col4)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -121,16 +121,16 @@ Reduce Output Operator key expressions: _col2 (type: string), _col1 (type: float), _col4 (type: double), _col0 (type: timestamp) sort order: ++++ - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE value expressions: _col3 (type: boolean), _col5 (type: double), _col6 (type: double), _col7 (type: double), _col8 (type: double), _col9 (type: float), _col10 (type: float), _col11 (type: float), _col12 (type: double), _col13 (type: double), _col14 (type: bigint), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: double), _col19 (type: double), _col20 (type: double), _col21 (type: double) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey3 (type: timestamp), KEY.reducesinkkey1 (type: float), KEY.reducesinkkey0 (type: string), VALUE._col0 (type: boolean), KEY.reducesinkkey2 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: float), VALUE._col6 (type: float), VALUE._col7 (type: float), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: bigint), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col14 (type: double), VALUE._col15 (type: double), VALUE._col16 (type: double), VALUE._col17 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_15.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_15.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_15.q.out (working copy) @@ -71,25 +71,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%ss%') or ((cstring1 like '10%') or ((cint >= -75) and ((UDFToShort(ctinyint) = csmallint) and (cdouble >= -3728.0))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cfloat (type: float), cboolean1 (type: boolean), cdouble (type: double), cstring1 (type: string), ctinyint (type: tinyint), cint (type: int), ctimestamp1 (type: timestamp) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_samp(_col0), min(_col2), stddev_samp(_col4), var_pop(_col4), var_samp(_col5), stddev_pop(_col5) keys: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) sort order: +++++++ Map-reduce partition columns: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE value expressions: _col7 (type: struct), _col8 (type: double), _col9 (type: struct), _col10 (type: struct), _col11 (type: struct), _col12 (type: struct) Execution mode: vectorized Reduce Operator Tree: @@ -98,11 +98,11 @@ keys: KEY._col0 (type: float), KEY._col1 (type: boolean), KEY._col2 (type: double), KEY._col3 (type: string), KEY._col4 (type: tinyint), KEY._col5 (type: int), KEY._col6 (type: timestamp) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp), _col7 (type: double), (-26.28 - UDFToDouble(_col5)) (type: double), _col8 (type: double), (_col2 * 79.553) (type: double), (33.0 % _col0) (type: float), _col9 (type: double), _col10 (type: double), (-23.0 % _col2) (type: double), (- _col4) (type: tinyint), _col11 (type: double), (UDFToFloat(_col5) - _col0) (type: float), (-23 % UDFToInteger(_col4)) (type: int), (- (-26.28 - UDFToDouble(_col5))) (type: double), _col12 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -117,16 +117,16 @@ Reduce Output Operator key expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp) sort order: +++++++ - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE value expressions: _col7 (type: double), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: float), _col12 (type: double), _col13 (type: double), _col14 (type: double), _col15 (type: tinyint), _col16 (type: double), _col17 (type: float), _col18 (type: int), _col19 (type: double), _col20 (type: double) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: float), KEY.reducesinkkey1 (type: boolean), KEY.reducesinkkey2 (type: double), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: tinyint), KEY.reducesinkkey5 (type: int), KEY.reducesinkkey6 (type: timestamp), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col4 (type: float), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: tinyint), VALUE._col9 (type: double), VALUE._col10 (type: float), VALUE._col11 (type: int), VALUE._col12 (type: double), VALUE._col13 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_16.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_16.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_16.q.out (working copy) @@ -52,25 +52,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%b%') and ((cdouble >= -1.389) or (cstring1 < 'a'))) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cstring1 (type: string), cdouble (type: double), ctimestamp1 (type: timestamp) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col1), stddev_samp(_col1), min(_col1) keys: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) sort order: +++ Map-reduce partition columns: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE value expressions: _col3 (type: bigint), _col4 (type: struct), _col5 (type: double) Execution mode: vectorized Reduce Operator Tree: @@ -79,14 +79,14 @@ keys: KEY._col0 (type: string), KEY._col1 (type: double), KEY._col2 (type: timestamp) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp), (_col1 - 9763215.5639) (type: double), (- (_col1 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col1) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_7.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_7.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_7.q.out (working copy) @@ -60,20 +60,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((ctinyint <> 0) and ((UDFToDouble(ctimestamp1) <= 0.0) or ((UDFToInteger(ctinyint) = cint) or (cstring2 like 'ss')))) and ((988888.0 < cdouble) or ((UDFToDouble(ctimestamp2) > -15.0) and (3569.0 >= cdouble)))) (type: boolean) - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), cbigint (type: bigint), csmallint (type: smallint), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cstring1 (type: string), (cbigint + cbigint) (type: bigint), (UDFToInteger(csmallint) % -257) (type: int), (- csmallint) (type: smallint), (- ctinyint) (type: tinyint), (UDFToInteger((- ctinyint)) + 17) (type: int), (cbigint * UDFToLong((- csmallint))) (type: bigint), (cint % UDFToInteger(csmallint)) (type: int), (- ctinyint) (type: tinyint), ((- ctinyint) % ctinyint) (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -229,20 +229,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((ctinyint <> 0) and ((UDFToDouble(ctimestamp1) <= 0.0) or ((UDFToInteger(ctinyint) = cint) or (cstring2 like 'ss')))) and ((988888.0 < cdouble) or ((UDFToDouble(ctimestamp2) > 7.6850000000000005) and (3569.0 >= cdouble)))) (type: boolean) - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), cbigint (type: bigint), csmallint (type: smallint), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cstring1 (type: string), (cbigint + cbigint) (type: bigint), (UDFToInteger(csmallint) % -257) (type: int), (- csmallint) (type: smallint), (- ctinyint) (type: tinyint), (UDFToInteger((- ctinyint)) + 17) (type: int), (cbigint * UDFToLong((- csmallint))) (type: bigint), (cint % UDFToInteger(csmallint)) (type: int), (- ctinyint) (type: tinyint), ((- ctinyint) % ctinyint) (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 7281 Data size: 223523 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7281 Data size: 1565441 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_8.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_8.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_8.q.out (working copy) @@ -56,20 +56,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 is not null and ((UDFToDouble(ctimestamp1) <= 10.0) and (UDFToDouble(ctimestamp2) <> 16.0))) or ((cfloat < -6432.0) or (cboolean1 is not null and (cdouble = 988888.0)))) (type: boolean) - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cdouble (type: double), cboolean1 (type: boolean), cstring1 (type: string), cfloat (type: float), (- cdouble) (type: double), (-5638.15 - cdouble) (type: double), (cdouble * -257.0) (type: double), (UDFToFloat(cint) + cfloat) (type: float), ((- cdouble) + UDFToDouble(cbigint)) (type: double), (- cdouble) (type: double), (-1.389 - UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), ((-5638.15 - cdouble) + UDFToDouble((UDFToFloat(cint) + cfloat))) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13 - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -212,20 +212,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 is not null and ((UDFToDouble(ctimestamp1) <= 12.503) and (UDFToDouble(ctimestamp2) <> 11.998))) or ((cfloat < -6432.0) or (cboolean1 is not null and (cdouble = 988888.0)))) (type: boolean) - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cdouble (type: double), cboolean1 (type: boolean), cstring1 (type: string), cfloat (type: float), (- cdouble) (type: double), (-5638.15 - cdouble) (type: double), (cdouble * -257.0) (type: double), (UDFToFloat(cint) + cfloat) (type: float), ((- cdouble) + UDFToDouble(cbigint)) (type: double), (- cdouble) (type: double), (-1.389 - UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), ((-5638.15 - cdouble) + UDFToDouble((UDFToFloat(cint) + cfloat))) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13 - Statistics: Num rows: 9216 Data size: 282927 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9216 Data size: 1981473 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_9.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_9.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_9.q.out (working copy) @@ -48,25 +48,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cstring2 like '%b%') and ((cdouble >= -1.389) or (cstring1 < 'a'))) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cstring1 (type: string), cdouble (type: double), ctimestamp1 (type: timestamp) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col1), stddev_samp(_col1), min(_col1) keys: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) sort order: +++ Map-reduce partition columns: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE value expressions: _col3 (type: bigint), _col4 (type: struct), _col5 (type: double) Execution mode: vectorized Reduce Operator Tree: @@ -75,14 +75,14 @@ keys: KEY._col0 (type: string), KEY._col1 (type: double), KEY._col2 (type: timestamp) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp), (_col1 - 9763215.5639) (type: double), (- (_col1 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col1) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_div0.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_div0.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_div0.q.out (working copy) @@ -16,17 +16,17 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: (cdouble / 0.0) (type: double) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 100 - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -171,31 +171,31 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cbigint > 0) and (cbigint < 100000000)) (type: boolean) - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: (cbigint - 988888) (type: bigint), (cdouble / UDFToDouble((cbigint - 988888))) (type: double), (1.2 / UDFToDouble((cbigint - 988888))) (type: double) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: bigint), _col1 (type: double) sort order: ++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: bigint), KEY.reducesinkkey1 (type: double), VALUE._col0 (type: double) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 100 - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -339,31 +339,31 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cdouble >= -500.0) and (cdouble < -199.0)) (type: boolean) - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: (cdouble + 200.0) (type: double), (UDFToDouble(cbigint) / (cdouble + 200.0)) (type: double), ((cdouble + 200.0) / (cdouble + 200.0)) (type: double), (3.0 / (cdouble + 200.0)) (type: double), (1.2 / (cdouble + 200.0)) (type: double) outputColumnNames: _col0, _col1, _col2, _col4, _col5 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double), _col1 (type: double) sort order: ++ - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double), _col4 (type: double), _col5 (type: double) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: double), KEY.reducesinkkey1 (type: double), VALUE._col0 (type: double), KEY.reducesinkkey1 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 100 - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 100 Data size: 3000 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 100 Data size: 21500 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_limit.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_limit.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_limit.q.out (working copy) @@ -13,20 +13,20 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((UDFToDouble(cbigint) < cdouble) and (cint > 0)) (type: boolean) - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint), cdouble (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 1365 Data size: 41904 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 7 - Statistics: Num rows: 7 Data size: 210 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7 Data size: 1505 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 7 Data size: 210 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7 Data size: 1505 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -75,18 +75,18 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ctinyint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cdouble (type: double), csmallint (type: smallint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: double) sort order: ++ - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 value expressions: _col2 (type: smallint) Execution mode: vectorized @@ -94,13 +94,13 @@ Select Operator expressions: KEY.reducesinkkey0 (type: tinyint), KEY.reducesinkkey1 (type: double), VALUE._col0 (type: smallint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -158,22 +158,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), (cdouble + 1.0) (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col1) keys: _col0 (type: tinyint) mode: hash outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint) sort order: + Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 value expressions: _col1 (type: struct) Execution mode: vectorized @@ -183,13 +183,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -247,21 +247,21 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator keys: _col0 (type: tinyint) mode: hash outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint) sort order: + Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Execution mode: vectorized Reduce Operator Tree: @@ -269,13 +269,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -331,22 +331,22 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cdouble (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(DISTINCT _col1) keys: _col0 (type: tinyint), _col1 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: double) sort order: ++ Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Execution mode: vectorized Reduce Operator Tree: @@ -355,13 +355,13 @@ keys: KEY._col0 (type: tinyint) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -446,25 +446,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ctinyint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double), ctinyint (type: tinyint) outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col1) keys: _col0 (type: double) mode: hash outputColumnNames: _col0, _col1 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double) sort order: + Map-reduce partition columns: _col0 (type: double) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: bigint) Execution mode: vectorized Reduce Operator Tree: @@ -473,7 +473,7 @@ keys: KEY._col0 (type: double) mode: mergepartial outputColumnNames: _col0, _col1 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -488,19 +488,19 @@ Reduce Output Operator key expressions: _col1 (type: bigint), _col0 (type: double) sort order: ++ - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.3 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey1 (type: double), KEY.reducesinkkey0 (type: bigint) outputColumnNames: _col0, _col1 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorization_pushdown.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_pushdown.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_pushdown.q.out (working copy) @@ -14,14 +14,14 @@ TableScan alias: alltypesorc filterExpr: (UDFToDouble(cbigint) < cdouble) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (UDFToDouble(cbigint) < cdouble) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint) outputColumnNames: _col0 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0) mode: hash Index: ql/src/test/results/clientpositive/vectorization_short_regress.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorization_short_regress.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorization_short_regress.q.out (working copy) @@ -141,14 +141,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((762 = cbigint) or (((UDFToFloat(csmallint) < cfloat) and ((UDFToDouble(ctimestamp2) > -5.0) and (cdouble <> UDFToDouble(cint)))) or ((cstring1 = 'a') or ((UDFToDouble(cbigint) <= -1.389) and ((cstring2 <> 'a') and ((79.553 <> UDFToDouble(cint)) and (cboolean2 <> cboolean1))))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cdouble (type: double), csmallint (type: smallint), cfloat (type: float), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0), sum(_col1), stddev_pop(_col0), stddev_samp(_col2), var_samp(_col0), avg(_col3), stddev_samp(_col0), min(_col4), count(_col2) mode: hash @@ -347,14 +347,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cbigint <= 197) and (UDFToLong(cint) < cbigint)) or (((cdouble >= -26.28) and (UDFToDouble(csmallint) > cdouble)) or (((UDFToFloat(ctinyint) > cfloat) and (cstring1 rlike '.*ss.*')) or ((cfloat > 79.553) and (cstring2 like '10%'))))) (type: boolean) - Statistics: Num rows: 6826 Data size: 209555 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6826 Data size: 1467614 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cbigint (type: bigint), csmallint (type: smallint), cdouble (type: double), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 6826 Data size: 209555 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6826 Data size: 1467614 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col0), var_pop(_col1), stddev_pop(_col2), max(_col3), avg(_col4), min(_col0), min(_col3), stddev_samp(_col2), var_samp(_col0) mode: hash @@ -544,14 +544,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((ctimestamp1 = ctimestamp2) or ((762.0 = cfloat) or ((cstring1 = 'ss') or (((UDFToLong(csmallint) <= cbigint) and (1 = cboolean2)) or (cboolean1 is not null and (ctimestamp2 is not null and (cstring2 > 'a'))))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cbigint (type: bigint), ctinyint (type: tinyint), csmallint (type: smallint), cint (type: int), cdouble (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: var_pop(_col0), count(), max(_col1), stddev_pop(_col2), max(_col3), stddev_samp(_col4), count(_col1), avg(_col1) mode: hash @@ -720,14 +720,14 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((ctimestamp2 <= ctimestamp1) and ((UDFToDouble(cbigint) <> cdouble) and ('ss' <= cstring1))) or (((csmallint < UDFToShort(ctinyint)) and (UDFToDouble(ctimestamp1) >= 0.0)) or (cfloat = 17.0))) (type: boolean) - Statistics: Num rows: 8874 Data size: 272428 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8874 Data size: 1907941 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cbigint (type: bigint), cint (type: int), cfloat (type: float) outputColumnNames: _col0, _col1, _col2, _col3 - Statistics: Num rows: 8874 Data size: 272428 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 8874 Data size: 1907941 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: avg(_col0), max(_col1), stddev_samp(_col2), var_pop(_col2), var_pop(_col1), max(_col3) mode: hash @@ -904,31 +904,31 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cstring1 rlike 'a.*') and (cstring2 like '%ss%')) or (((1 <> cboolean2) and ((UDFToDouble(csmallint) < 79.553) and (-257 <> UDFToInteger(ctinyint)))) or (((cdouble > UDFToDouble(ctinyint)) and (cfloat >= UDFToFloat(cint))) or ((UDFToLong(cint) < cbigint) and (UDFToLong(ctinyint) > cbigint))))) (type: boolean) - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cdouble (type: double), ctimestamp2 (type: timestamp), cstring1 (type: string), cboolean2 (type: boolean), ctinyint (type: tinyint), cfloat (type: float), ctimestamp1 (type: timestamp), csmallint (type: smallint), cbigint (type: bigint), (-3728 * cbigint) (type: bigint), (- cint) (type: int), (-863.257 - UDFToDouble(cint)) (type: double), (- csmallint) (type: smallint), (csmallint - (- csmallint)) (type: smallint), ((csmallint - (- csmallint)) + (- csmallint)) (type: smallint), (UDFToDouble(cint) / UDFToDouble(cint)) (type: double), ((-863.257 - UDFToDouble(cint)) - -26.28) (type: double), (- cfloat) (type: float), (cdouble * -89010.0) (type: double), (UDFToDouble(ctinyint) / 988888.0) (type: double), (- ctinyint) (type: tinyint), (79.553 / UDFToDouble(ctinyint)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22 - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int), _col1 (type: double), _col2 (type: timestamp), _col3 (type: string), _col4 (type: boolean), _col5 (type: tinyint), _col6 (type: float), _col7 (type: timestamp), _col8 (type: smallint), _col9 (type: bigint) sort order: ++++++++++ - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE value expressions: _col10 (type: bigint), _col11 (type: int), _col12 (type: double), _col13 (type: smallint), _col14 (type: smallint), _col15 (type: smallint), _col16 (type: double), _col17 (type: double), _col18 (type: float), _col19 (type: double), _col20 (type: double), _col21 (type: tinyint), _col22 (type: double) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: double), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey3 (type: string), KEY.reducesinkkey4 (type: boolean), KEY.reducesinkkey5 (type: tinyint), KEY.reducesinkkey6 (type: float), KEY.reducesinkkey7 (type: timestamp), KEY.reducesinkkey8 (type: smallint), KEY.reducesinkkey9 (type: bigint), VALUE._col0 (type: bigint), VALUE._col1 (type: int), VALUE._col2 (type: double), VALUE._col3 (type: smallint), VALUE._col4 (type: smallint), VALUE._col5 (type: smallint), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: float), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: tinyint), VALUE._col12 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22 - Statistics: Num rows: 9898 Data size: 303864 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 9898 Data size: 2128105 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 50 - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1156,31 +1156,31 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((197.0 > UDFToDouble(ctinyint)) and (UDFToLong(cint) = cbigint)) or ((cbigint = 359) or ((cboolean1 < 0) or ((cstring1 like '%ss') and (cfloat <= UDFToFloat(ctinyint)))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cbigint (type: bigint), (UDFToDouble(cint) / UDFToDouble(cbigint)) (type: double), (UDFToDouble(cbigint) % 79.553) (type: double), (- (UDFToDouble(cint) / UDFToDouble(cbigint))) (type: double), (10.175 % UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), (cfloat - (- cfloat)) (type: float), ((cfloat - (- cfloat)) % -6432.0) (type: float), (cdouble * UDFToDouble(csmallint)) (type: double), (- cdouble) (type: double), (- cbigint) (type: bigint), cstring1 (type: string), (UDFToDouble(cfloat) - (UDFToDouble(cint) / UDFToDouble(cbigint))) (type: double), (- csmallint) (type: smallint), (3569 % cbigint) (type: bigint), (359.0 - cdouble) (type: double), cboolean1 (type: boolean), cfloat (type: float), cdouble (type: double), ctimestamp2 (type: timestamp), csmallint (type: smallint), cstring2 (type: string), cboolean2 (type: boolean) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col21, _col22, _col23, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int), _col1 (type: bigint), _col2 (type: string), _col3 (type: boolean), _col4 (type: float), _col5 (type: double), _col6 (type: timestamp), _col7 (type: smallint), _col8 (type: string), _col9 (type: boolean) sort order: ++++++++++ - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE value expressions: _col10 (type: double), _col11 (type: double), _col12 (type: double), _col13 (type: double), _col14 (type: float), _col15 (type: float), _col16 (type: float), _col17 (type: double), _col18 (type: double), _col19 (type: bigint), _col20 (type: double), _col21 (type: smallint), _col22 (type: bigint), _col23 (type: double) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey3 (type: boolean), KEY.reducesinkkey4 (type: float), KEY.reducesinkkey5 (type: double), KEY.reducesinkkey6 (type: timestamp), KEY.reducesinkkey7 (type: smallint), KEY.reducesinkkey8 (type: string), KEY.reducesinkkey9 (type: boolean), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col4 (type: float), VALUE._col5 (type: float), VALUE._col6 (type: float), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: bigint), VALUE._col10 (type: double), VALUE._col11 (type: smallint), VALUE._col12 (type: bigint), VALUE._col13 (type: double), VALUE._col11 (type: smallint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 25 - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 25 Data size: 750 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 25 Data size: 5375 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1357,31 +1357,31 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((UDFToDouble(csmallint) > -26.28) and (cstring2 like 'ss')) or (((cdouble <= UDFToDouble(cbigint)) and ((cstring1 >= 'ss') and (UDFToDouble(cint) <> cdouble))) or ((UDFToInteger(ctinyint) = -89010) or ((UDFToFloat(cbigint) <= cfloat) and (-26.28 <= UDFToDouble(csmallint)))))) (type: boolean) - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int), cstring1 (type: string), cboolean2 (type: boolean), ctimestamp2 (type: timestamp), cdouble (type: double), cfloat (type: float), cbigint (type: bigint), csmallint (type: smallint), cboolean1 (type: boolean), (cint + UDFToInteger(csmallint)) (type: int), (cbigint - UDFToLong(ctinyint)) (type: bigint), (- cbigint) (type: bigint), (- cfloat) (type: float), ((cbigint - UDFToLong(ctinyint)) + cbigint) (type: bigint), (cdouble / cdouble) (type: double), (- cdouble) (type: double), (UDFToLong((cint + UDFToInteger(csmallint))) * (- cbigint)) (type: bigint), ((- cdouble) + UDFToDouble(cbigint)) (type: double), (-1.389 / UDFToDouble(ctinyint)) (type: double), (UDFToDouble(cbigint) % cdouble) (type: double), (- csmallint) (type: smallint), (UDFToInteger(csmallint) + (cint + UDFToInteger(csmallint))) (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col8 (type: boolean), _col1 (type: string), _col3 (type: timestamp), _col5 (type: float), _col6 (type: bigint), _col1 (type: string), _col4 (type: double), _col0 (type: int), _col7 (type: smallint), _col4 (type: double) sort order: ++++++++++ - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: boolean), _col9 (type: int), _col10 (type: bigint), _col11 (type: bigint), _col12 (type: float), _col13 (type: bigint), _col14 (type: double), _col15 (type: double), _col16 (type: bigint), _col17 (type: double), _col18 (type: double), _col19 (type: double), _col20 (type: smallint), _col21 (type: int) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey7 (type: int), KEY.reducesinkkey1 (type: string), VALUE._col0 (type: boolean), KEY.reducesinkkey2 (type: timestamp), KEY.reducesinkkey6 (type: double), KEY.reducesinkkey3 (type: float), KEY.reducesinkkey4 (type: bigint), KEY.reducesinkkey8 (type: smallint), KEY.reducesinkkey0 (type: boolean), VALUE._col1 (type: int), VALUE._col2 (type: bigint), VALUE._col3 (type: bigint), VALUE._col4 (type: float), VALUE._col5 (type: bigint), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: bigint), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: smallint), VALUE._col13 (type: int) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21 - Statistics: Num rows: 10922 Data size: 335301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 10922 Data size: 2348269 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 75 - Statistics: Num rows: 75 Data size: 2250 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 75 Data size: 16125 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 75 Data size: 2250 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 75 Data size: 16125 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1616,31 +1616,31 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((-1.389 >= UDFToDouble(cint)) and ((csmallint < UDFToShort(ctinyint)) and (-6432 > UDFToInteger(csmallint)))) or (((cdouble >= UDFToDouble(cfloat)) and (cstring2 <= 'a')) or ((cstring1 like 'ss%') and (10.175 > UDFToDouble(cbigint))))) (type: boolean) - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cstring2 (type: string), (cdouble * 10.175) (type: double), (UDFToDouble((-6432.0 * cfloat)) / UDFToDouble(cfloat)) (type: double), (- cfloat) (type: float), (cint % UDFToInteger(csmallint)) (type: int), (cdouble * (- cdouble)) (type: double), cdouble (type: double), cfloat (type: float), cbigint (type: bigint), csmallint (type: smallint), (UDFToDouble(cbigint) / 3569.0) (type: double), (-257 - UDFToInteger(csmallint)) (type: int), (-6432.0 * cfloat) (type: float), (- cdouble) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col15, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col5 (type: smallint), _col1 (type: string), _col2 (type: double) sort order: +++ - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE value expressions: _col0 (type: timestamp), _col3 (type: float), _col4 (type: bigint), _col6 (type: double), _col7 (type: int), _col8 (type: float), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: float), _col13 (type: int), _col15 (type: double) Execution mode: vectorized Reduce Operator Tree: Select Operator expressions: VALUE._col0 (type: timestamp), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey2 (type: double), VALUE._col1 (type: float), VALUE._col2 (type: bigint), KEY.reducesinkkey0 (type: smallint), VALUE._col3 (type: double), VALUE._col4 (type: int), VALUE._col5 (type: float), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: float), VALUE._col10 (type: int), VALUE._col6 (type: double), VALUE._col11 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15 - Statistics: Num rows: 3868 Data size: 118746 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3868 Data size: 831633 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 45 - Statistics: Num rows: 45 Data size: 1350 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 45 Data size: 9675 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 45 Data size: 1350 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 45 Data size: 9675 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -1817,25 +1817,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((UDFToInteger(csmallint) >= -257) and ((-6432 = UDFToInteger(csmallint)) or ((UDFToDouble(cint) >= cdouble) and (UDFToInteger(ctinyint) <= cint)))) (type: boolean) - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: csmallint (type: smallint), cbigint (type: bigint), ctinyint (type: tinyint) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_samp(_col0), sum(_col1), var_pop(_col2), count() keys: _col0 (type: smallint) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: smallint) sort order: + Map-reduce partition columns: _col0 (type: smallint) - Statistics: Num rows: 2503 Data size: 76841 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: struct), _col2 (type: bigint), _col3 (type: struct), _col4 (type: bigint) Execution mode: vectorized Reduce Operator Tree: @@ -1844,11 +1844,11 @@ keys: KEY._col0 (type: smallint) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: smallint), (UDFToInteger(_col0) % -75) (type: int), _col1 (type: double), (-1.389 / UDFToDouble(_col0)) (type: double), _col2 (type: bigint), (UDFToDouble((UDFToInteger(_col0) % -75)) / UDFToDouble(_col2)) (type: double), (- (UDFToInteger(_col0) % -75)) (type: int), _col3 (type: double), (- (- (UDFToInteger(_col0) % -75))) (type: int), _col4 (type: bigint), (_col4 - -89010) (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -1863,19 +1863,19 @@ Reduce Output Operator key expressions: _col0 (type: smallint) sort order: + - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: int), _col2 (type: double), _col3 (type: double), _col4 (type: bigint), _col5 (type: double), _col6 (type: int), _col7 (type: double), _col8 (type: int), _col9 (type: bigint), _col10 (type: bigint) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: smallint), VALUE._col0 (type: int), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: bigint), VALUE._col4 (type: double), VALUE._col5 (type: int), VALUE._col6 (type: double), VALUE._col7 (type: int), VALUE._col8 (type: bigint), VALUE._col9 (type: bigint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 1251 Data size: 38405 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1251 Data size: 268968 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 20 Data size: 600 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 20 Data size: 4300 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -2028,25 +2028,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cdouble > 2563.58) and (((cbigint >= UDFToLong(cint)) and ((UDFToInteger(csmallint) < cint) and (UDFToDouble(cfloat) < -5638.15))) or ((cdouble <= UDFToDouble(cbigint)) and (-5638.15 > UDFToDouble(cbigint))))) (type: boolean) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double), cfloat (type: float) outputColumnNames: _col0, _col1 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: var_samp(_col0), count(_col1), sum(_col1), var_pop(_col0), stddev_pop(_col0), sum(_col0) keys: _col0 (type: double) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: double) sort order: + Map-reduce partition columns: _col0 (type: double) - Statistics: Num rows: 606 Data size: 18603 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 606 Data size: 130292 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: struct), _col2 (type: bigint), _col3 (type: double), _col4 (type: struct), _col5 (type: struct), _col6 (type: double) Execution mode: vectorized Reduce Operator Tree: @@ -2055,11 +2055,11 @@ keys: KEY._col0 (type: double) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: double), _col1 (type: double), _col5 (type: double), (_col0 + _col1) (type: double), (_col0 * 762.0) (type: double), _col6 (type: double), (-863.257 % (_col0 * 762.0)) (type: double), (2563.58 * _col1) (type: double), (- _col1) (type: double), _col2 (type: bigint), ((2563.58 * _col1) + -5638.15) (type: double), ((- _col1) * ((2563.58 * _col1) + -5638.15)) (type: double), _col3 (type: double), _col4 (type: double), (_col0 - (- _col1)) (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -2074,16 +2074,16 @@ Reduce Output Operator key expressions: _col0 (type: double) sort order: + - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: double), _col2 (type: double), _col3 (type: double), _col4 (type: bigint), _col5 (type: double), _col6 (type: double), _col7 (type: double), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: double), _col12 (type: double), _col13 (type: double), _col14 (type: double) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: double), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: bigint), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col12 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15 - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 303 Data size: 9301 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 303 Data size: 65146 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -2286,25 +2286,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((UDFToDouble(ctimestamp1) <> 0.0) and (((-257 <> UDFToInteger(ctinyint)) and (cboolean2 is not null and ((cstring1 rlike '.*ss') and (-3.0 < UDFToDouble(ctimestamp1))))) or ((UDFToDouble(ctimestamp2) = -5.0) or (((UDFToDouble(ctimestamp1) < 0.0) and (cstring2 like '%b%')) or ((cdouble = UDFToDouble(cint)) or (cboolean1 is null and (cfloat < UDFToFloat(cint)))))))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctimestamp1 (type: timestamp), cstring1 (type: string), cint (type: int), csmallint (type: smallint), ctinyint (type: tinyint), cfloat (type: float), cdouble (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: stddev_pop(_col2), avg(_col3), count(), min(_col4), var_samp(_col3), var_pop(_col5), avg(_col2), var_samp(_col5), avg(_col5), min(_col6), var_pop(_col3), stddev_pop(_col4), sum(_col2) keys: _col0 (type: timestamp), _col1 (type: string) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: timestamp), _col1 (type: string) sort order: ++ Map-reduce partition columns: _col0 (type: timestamp), _col1 (type: string) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: struct), _col3 (type: struct), _col4 (type: bigint), _col5 (type: tinyint), _col6 (type: struct), _col7 (type: struct), _col8 (type: struct), _col9 (type: struct), _col10 (type: struct), _col11 (type: double), _col12 (type: struct), _col13 (type: struct), _col14 (type: bigint) Execution mode: vectorized Reduce Operator Tree: @@ -2313,11 +2313,11 @@ keys: KEY._col0 (type: timestamp), KEY._col1 (type: string) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: timestamp), _col1 (type: string), ((-26.28 - _col2) * (- _col2)) (type: double), _col5 (type: tinyint), (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4))) (type: double), (- (_col2 * 10.175)) (type: double), _col6 (type: double), (_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- _col2)) (type: double), (UDFToDouble((- _col4)) / _col2) (type: double), _col7 (type: double), (10.175 / _col3) (type: double), _col2 (type: double), _col8 (type: double), _col9 (type: double), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- (_col2 * 10.175))) (type: double), _col10 (type: double), (((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) * 10.175) (type: double), (10.175 % (10.175 / _col3)) (type: double), (- _col5) (type: tinyint), _col11 (type: double), _col12 (type: double), (_col2 * 10.175) (type: double), (- ((-26.28 - _col2) * (- _col2))) (type: double), ((- _col2) % _col10) (type: double), (-26.28 / UDFToDouble((- _col5))) (type: double), _col13 (type: double), _col14 (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) / _col7) (type: double), (- (- _col4)) (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) % -26.28) (type: double), (- _col2) (type: double), _col3 (type: double), (-26.28 - _col2) (type: double), _col4 (type: bigint), (- _col4) (type: bigint) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col2, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col3, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col38, _col4, _col5, _col7, _col8, _col9 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -2332,19 +2332,19 @@ Reduce Output Operator key expressions: _col0 (type: timestamp), _col1 (type: string) sort order: ++ - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE value expressions: _col2 (type: double), _col3 (type: double), _col4 (type: double), _col5 (type: double), _col7 (type: double), _col8 (type: bigint), _col9 (type: bigint), _col10 (type: double), _col11 (type: tinyint), _col12 (type: double), _col13 (type: double), _col14 (type: double), _col15 (type: double), _col16 (type: double), _col17 (type: double), _col18 (type: double), _col19 (type: double), _col20 (type: double), _col21 (type: double), _col22 (type: double), _col23 (type: double), _col24 (type: double), _col25 (type: double), _col26 (type: double), _col27 (type: tinyint), _col28 (type: double), _col29 (type: double), _col30 (type: double), _col31 (type: double), _col32 (type: double), _col33 (type: double), _col34 (type: bigint), _col35 (type: double), _col36 (type: bigint), _col38 (type: double) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: timestamp), KEY.reducesinkkey1 (type: string), VALUE._col0 (type: double), VALUE._col1 (type: double), VALUE._col2 (type: double), VALUE._col3 (type: double), VALUE._col2 (type: double), VALUE._col4 (type: double), VALUE._col5 (type: bigint), VALUE._col6 (type: bigint), VALUE._col7 (type: double), VALUE._col8 (type: tinyint), VALUE._col9 (type: double), VALUE._col10 (type: double), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col14 (type: double), VALUE._col15 (type: double), VALUE._col16 (type: double), VALUE._col17 (type: double), VALUE._col18 (type: double), VALUE._col19 (type: double), VALUE._col20 (type: double), VALUE._col21 (type: double), VALUE._col22 (type: double), VALUE._col23 (type: double), VALUE._col24 (type: tinyint), VALUE._col25 (type: double), VALUE._col26 (type: double), VALUE._col27 (type: double), VALUE._col28 (type: double), VALUE._col29 (type: double), VALUE._col30 (type: double), VALUE._col31 (type: bigint), VALUE._col32 (type: double), VALUE._col33 (type: bigint), VALUE._col5 (type: bigint), VALUE._col34 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col37, _col38 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 50 - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 50 Data size: 1500 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 50 Data size: 10750 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat @@ -2622,25 +2622,25 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (cboolean1 is not null and (((cdouble < UDFToDouble(csmallint)) and ((cboolean2 = cboolean1) and (UDFToDouble(cbigint) <= -863.257))) or (((cint >= -257) and (cstring1 is not null and (cboolean1 >= 1))) or ((cstring2 rlike 'b') or ((csmallint >= UDFToShort(ctinyint)) and ctimestamp2 is null))))) (type: boolean) - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cboolean1 (type: boolean), cfloat (type: float), cbigint (type: bigint), cint (type: int), cdouble (type: double), ctinyint (type: tinyint), csmallint (type: smallint) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6 - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: max(_col1), sum(_col2), var_samp(_col3), avg(_col4), min(_col2), var_pop(_col2), sum(_col3), stddev_samp(_col5), stddev_pop(_col6), avg(_col3) keys: _col0 (type: boolean) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: boolean) sort order: + Map-reduce partition columns: _col0 (type: boolean) - Statistics: Num rows: 4778 Data size: 146682 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: float), _col2 (type: bigint), _col3 (type: struct), _col4 (type: struct), _col5 (type: bigint), _col6 (type: struct), _col7 (type: bigint), _col8 (type: struct), _col9 (type: struct), _col10 (type: struct) Execution mode: vectorized Reduce Operator Tree: @@ -2649,11 +2649,11 @@ keys: KEY._col0 (type: boolean) mode: mergepartial outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10 - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: boolean), _col1 (type: float), ((UDFToDouble(_col2) - 10.175) + _col3) (type: double), _col5 (type: bigint), _col6 (type: double), (- (10.175 + UDFToDouble((- _col1)))) (type: double), (79.553 / _col6) (type: double), (_col3 % (79.553 / _col6)) (type: double), _col7 (type: bigint), _col8 (type: double), (-1.389 * UDFToDouble(_col5)) (type: double), (- _col1) (type: float), (UDFToDouble(_col7) - (-1.389 * UDFToDouble(_col5))) (type: double), _col9 (type: double), (- (UDFToDouble(_col7) - (-1.389 * UDFToDouble(_col5)))) (type: double), _col10 (type: double), (- _col10) (type: double), (_col10 * UDFToDouble(_col7)) (type: double), (-26.28 / UDFToDouble(_col1)) (type: double), _col2 (type: bigint), (UDFToDouble(_col2) - 10.175) (type: double), _col3 (type: double), (_col3 % UDFToDouble(_col1)) (type: double), (10.175 + UDFToDouble((- _col1))) (type: double), _col4 (type: double) outputColumnNames: _col0, _col1, _col10, _col11, _col12, _col13, _col14, _col15, _col17, _col18, _col19, _col2, _col20, _col21, _col22, _col23, _col24, _col25, _col3, _col4, _col5, _col6, _col7, _col8, _col9 - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false table: @@ -2668,16 +2668,16 @@ Reduce Output Operator key expressions: _col0 (type: boolean) sort order: + - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE value expressions: _col1 (type: float), _col2 (type: float), _col3 (type: double), _col4 (type: bigint), _col5 (type: double), _col6 (type: double), _col7 (type: double), _col8 (type: double), _col9 (type: double), _col10 (type: double), _col11 (type: bigint), _col12 (type: double), _col13 (type: double), _col14 (type: double), _col15 (type: double), _col17 (type: bigint), _col18 (type: double), _col19 (type: double), _col20 (type: double), _col21 (type: double), _col22 (type: double), _col23 (type: double), _col24 (type: double), _col25 (type: double) Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: boolean), VALUE._col0 (type: float), VALUE._col1 (type: float), VALUE._col2 (type: double), VALUE._col3 (type: bigint), VALUE._col4 (type: double), VALUE._col5 (type: double), VALUE._col6 (type: double), VALUE._col7 (type: double), VALUE._col8 (type: double), VALUE._col9 (type: double), VALUE._col10 (type: bigint), VALUE._col11 (type: double), VALUE._col12 (type: double), VALUE._col13 (type: double), VALUE._col14 (type: double), VALUE._col12 (type: double), VALUE._col15 (type: bigint), VALUE._col16 (type: double), VALUE._col17 (type: double), VALUE._col18 (type: double), VALUE._col19 (type: double), VALUE._col20 (type: double), VALUE._col21 (type: double), VALUE._col22 (type: double), VALUE._col23 (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25 - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 2389 Data size: 73341 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2389 Data size: 513643 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorized_case.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_case.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_case.q.out (working copy) @@ -44,17 +44,17 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((csmallint = 418) or ((csmallint = 12205) or (csmallint = 10583))) (type: boolean) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: csmallint (type: smallint), CASE WHEN ((csmallint = 418)) THEN ('a') WHEN ((csmallint = 12205)) THEN ('b') ELSE ('c') END (type: string), CASE (csmallint) WHEN (418) THEN ('a') WHEN (12205) THEN ('b') ELSE ('c') END (type: string) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorized_casts.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_casts.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_casts.q.out (working copy) @@ -156,17 +156,17 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((cbigint % 250) = 0) (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: UDFToBoolean(ctinyint) (type: boolean), UDFToBoolean(csmallint) (type: boolean), UDFToBoolean(cint) (type: boolean), UDFToBoolean(cbigint) (type: boolean), UDFToBoolean(cfloat) (type: boolean), UDFToBoolean(cdouble) (type: boolean), cboolean1 (type: boolean), UDFToBoolean((cbigint * 0)) (type: boolean), UDFToBoolean(ctimestamp1) (type: boolean), UDFToBoolean(cstring1) (type: boolean), UDFToInteger(ctinyint) (type: int), UDFToInteger(csmallint) (type: int), cint (type: int), UDFToInteger(cbigint) (type: int), UDFToInteger(cfloat) (type: int), UDFToInteger(cdouble) (type: int), UDFToInteger(cboolean1) (type: int), UDFToInteger(ctimestamp1) (type: int), UDFToInteger(cstring1) (type: int), UDFToInteger(substr(cstring1, 1, 1)) (type: int), UDFToByte(cfloat) (type: tinyint), UDFToShort(cfloat) (type: smallint), UDFToLong(cfloat) (type: bigint), UDFToDouble(ctinyint) (type: double), UDFToDouble(csmallint) (type: double), UDFToDouble(cint) (type: double), UDFToDouble(cbigint) (type: double), UDFToDouble(cfloat) (type: double), cdouble (type: double), UDFToDouble(cboolean1) (type: double), UDFToDouble(ctimestamp1) (type: double), UDFToDouble(cstring1) (type: double), UDFToDouble(substr(cstring1, 1, 1)) (type: double), UDFToFloat(cint) (type: float), UDFToFloat(cdouble) (type: float), CAST( ctinyint AS TIMESTAMP) (type: timestamp), CAST( csmallint AS TIMESTAMP) (type: timestamp), CAST( cint AS TIMESTAMP) (type: timestamp), CAST( cbigint AS TIMESTAMP) (type: timestamp), CAST( cfloat AS TIMESTAMP) (type: timestamp), CAST( cdouble AS TIMESTAMP) (type: timestamp), CAST( cboolean1 AS TIMESTAMP) (type: timestamp), CAST( (cbigint * 0) AS TIMESTAMP) (type: timestamp), ctimestamp1 (type: timestamp), CAST( cstring1 AS TIMESTAMP) (type: timestamp), CAST( substr(cstring1, 1, 1) AS TIMESTAMP) (type: timestamp), UDFToString(ctinyint) (type: string), UDFToString(csmallint) (type: string), UDFToString(cint) (type: string), UDFToString(cbigint) (type: string), UDFToString(cfloat) (type: string), UDFToString(cdouble) (type: string), UDFToString(cboolean1) (type: string), UDFToString((cbigint * 0)) (type: string), UDFToString(ctimestamp1) (type: string), cstring1 (type: string), UDFToFloat(UDFToInteger(cfloat)) (type: float), UDFToDouble((cint * 2)) (type: double), UDFToString(sin(cfloat)) (type: string), (UDFToDouble(UDFToFloat(cint)) + UDFToDouble(cboolean1)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col37, _col38, _col39, _col40, _col41, _col42, _col43, _col44, _col45, _col46, _col47, _col48, _col49, _col50, _col51, _col52, _col53, _col54, _col55, _col56, _col57, _col58, _col59 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorized_distinct_gby.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_distinct_gby.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_distinct_gby.q.out (working copy) @@ -90,21 +90,21 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(DISTINCT _col0), count(DISTINCT _col0), avg(DISTINCT _col0), std(DISTINCT _col0) keys: _col0 (type: int) mode: hash outputColumnNames: _col0, _col1, _col2, _col3, _col4 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Reduce Operator Tree: Group By Operator Index: ql/src/test/results/clientpositive/vectorized_mapjoin.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_mapjoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_mapjoin.q.out (working copy) @@ -22,14 +22,14 @@ $hdt$_0:$hdt$_0:t1 TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: int) @@ -40,14 +40,14 @@ Map Operator Tree: TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -55,11 +55,11 @@ 0 _col0 (type: int) 1 _col0 (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: int), _col1 (type: int), (_col0 + _col1) (type: int) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col0), max(_col1), min(_col0), avg(_col2) mode: hash Index: ql/src/test/results/clientpositive/vectorized_math_funcs.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_math_funcs.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_math_funcs.q.out (working copy) @@ -116,17 +116,17 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (((cbigint % 500) = 0) and (sin(cfloat) >= -1.0)) (type: boolean) - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cdouble (type: double), round(cdouble, 2) (type: double), floor(cdouble) (type: bigint), ceil(cdouble) (type: bigint), rand() (type: double), rand(98007) (type: double), exp(ln(cdouble)) (type: double), ln(cdouble) (type: double), ln(cfloat) (type: double), log10(cdouble) (type: double), log2(cdouble) (type: double), log2((cdouble - 15601.0)) (type: double), log2(cfloat) (type: double), log2(cbigint) (type: double), log2(cint) (type: double), log2(csmallint) (type: double), log2(ctinyint) (type: double), log(2.0, cdouble) (type: double), power(log2(cdouble), 2.0) (type: double), power(log2(cdouble), 2.0) (type: double), sqrt(cdouble) (type: double), sqrt(cbigint) (type: double), bin(cbigint) (type: string), hex(cdouble) (type: string), conv(cbigint, 10, 16) (type: string), abs(cdouble) (type: double), abs(ctinyint) (type: int), (cint pmod 3) (type: int), sin(cdouble) (type: double), asin(cdouble) (type: double), cos(cdouble) (type: double), acos(cdouble) (type: double), atan(cdouble) (type: double), degrees(cdouble) (type: double), radians(cdouble) (type: double), cdouble (type: double), cbigint (type: bigint), (- cdouble) (type: double), sign(cdouble) (type: double), sign(cbigint) (type: double), cos(((- sin(log(cdouble))) + 3.14159)) (type: double) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col37, _col38, _col39, _col40 - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 2048 Data size: 62872 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/vectorized_nested_mapjoin.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_nested_mapjoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_nested_mapjoin.q.out (working copy) @@ -21,14 +21,14 @@ $hdt$_0:$hdt$_0:$hdt$_0:v1 TableScan alias: v1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (ctinyint is not null and csmallint is not null) (type: boolean) - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), csmallint (type: smallint), cdouble (type: double) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 3072 Data size: 660491 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: tinyint) @@ -36,14 +36,14 @@ $hdt$_0:$hdt$_1:v1 TableScan alias: v1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: csmallint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: csmallint (type: smallint) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE HashTable Sink Operator keys: 0 _col0 (type: smallint) @@ -54,14 +54,14 @@ Map Operator Tree: TableScan alias: v1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ctinyint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -69,11 +69,11 @@ 0 _col0 (type: tinyint) 1 _col0 (type: tinyint) outputColumnNames: _col1, _col2 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col1 (type: smallint), _col2 (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Map Join Operator condition map: Inner Join 0 to 1 @@ -81,11 +81,11 @@ 0 _col0 (type: smallint) 1 _col0 (type: smallint) outputColumnNames: _col1 - Statistics: Num rows: 7433 Data size: 228226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col1 (type: double) outputColumnNames: _col0 - Statistics: Num rows: 7433 Data size: 228226 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: sum(_col0) mode: hash Index: ql/src/test/results/clientpositive/vectorized_ptf.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_ptf.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_ptf.q.out (working copy) @@ -298,6 +298,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -362,6 +374,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -381,7 +425,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -711,6 +755,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: j + output shape: _col1: string, _col2: string, _col5: int + type: SUBQUERY + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -775,6 +831,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), (_col5 - _wcol0) (type: int) @@ -794,7 +869,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -974,10 +1049,22 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: string), VALUE._col3 (type: int) - outputColumnNames: _col2, _col1, _col5 + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int) + outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -997,7 +1084,7 @@ columns _col0,_col1,_col2 columns.types string:string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1232,6 +1319,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1296,6 +1395,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -1315,7 +1446,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1562,6 +1693,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -1626,6 +1769,39 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), (_col5 - _wcol2) (type: int) @@ -1645,7 +1821,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -1902,6 +2078,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -2037,6 +2225,39 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col2, 1, _col2 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: int), _wcol0 (type: int), _wcol1 (type: int), _col2 (type: int), (_col2 - _wcol2) (type: int) @@ -2056,7 +2277,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2258,6 +2479,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -2403,7 +2636,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types int:string:string:string:string:int:string:double:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2600,6 +2833,18 @@ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col3: string, _col4: string, _col5: int, _col6: string, _col7: double, _col8: string + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -2749,7 +2994,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types int:string:string:string:string:int:string:double:string escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -2891,7 +3136,21 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: p_name: string, p_mfgr: string, p_size: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name, p_size(DESC) + output shape: p_name: string, p_mfgr: string, p_size: int + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string), p_size (type: int) sort order: ++- @@ -2955,6 +3214,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1, _col5(DESC) + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3018,6 +3290,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1, _col5(DESC) + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1, _col5 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int) @@ -3037,7 +3328,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3207,7 +3498,21 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: p_name + output shape: p_name: string, p_mfgr: string, p_size: int, p_retailprice: double + partition by: p_mfgr + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true Reduce Output Operator key expressions: p_mfgr (type: string), p_name (type: string) sort order: ++ @@ -3272,6 +3577,19 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3336,6 +3654,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -3355,7 +3705,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3593,6 +3943,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -3657,6 +4019,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -3676,7 +4070,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -3860,8 +4254,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string), p_name (type: string) - sort order: +++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -3920,13 +4314,46 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false GlobalTableId: 0 @@ -3951,8 +4378,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -3986,10 +4413,30 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) + expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col3 (type: int), VALUE._col5 (type: double) outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -4054,6 +4501,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -4073,7 +4552,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4341,6 +4820,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -4405,6 +4896,30 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol1 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: bigint), _wcol1 (type: double) @@ -4424,7 +4939,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:bigint:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -4769,6 +5284,18 @@ outputColumnNames: _col0, _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: abc + name: noop + order by: _col1 + output shape: _col0: int, _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Filter Operator isSamplingPred: false @@ -4963,6 +5490,51 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col1 + name: count + window function: GenericUDAFCountEvaluator + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol3 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ + window function definition + alias: _wcol4 + arguments: _col5, 1, _col5 + name: lag + window function: GenericUDAFLagEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 14 Data size: 8823 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: bigint), _col7 (type: double), _wcol3 (type: double), _col5 (type: int), (_col5 - _wcol4) (type: int) @@ -4982,7 +5554,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8 columns.types string:string:int:int:bigint:double:double:int:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5177,6 +5749,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int) @@ -5263,7 +5847,7 @@ columns _col0,_col1,_col2 columns.types string:string:int escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -5412,8 +5996,7 @@ STAGE DEPENDENCIES: Stage-1 is a root stage Stage-2 depends on stages: Stage-1 - Stage-3 depends on stages: Stage-2 - Stage-0 depends on stages: Stage-3 + Stage-0 depends on stages: Stage-2 STAGE PLANS: Stage: Stage-1 @@ -5436,7 +6019,7 @@ Reduce Output Operator key expressions: _col0 (type: string), _col1 (type: string) sort order: ++ - Map-reduce partition columns: _col0 (type: string), _col1 (type: string) + Map-reduce partition columns: _col0 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 value expressions: _col2 (type: double) @@ -5499,69 +6082,19 @@ mode: mergepartial outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: false - GlobalTableId: 0 -#### A masked pattern was here #### - NumFilesPerFileSink: 1 - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - properties: - columns _col0,_col1,_col2 - columns.types string,string,double - escape.delim \ - serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - TotalFiles: 1 - GatherStats: false - MultiFileSpray: false - - Stage: Stage-2 - Map Reduce - Map Operator Tree: - TableScan - GatherStats: false - Reduce Output Operator - key expressions: _col0 (type: string), _col0 (type: string) - sort order: ++ - Map-reduce partition columns: _col0 (type: string) - Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE - tag: -1 - value expressions: _col1 (type: string), _col2 (type: double) - auto parallelism: false - Path -> Alias: -#### A masked pattern was here #### - Path -> Partition: -#### A masked pattern was here #### - Partition - base file name: -mr-10002 - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - properties: - columns _col0,_col1,_col2 - columns.types string,string,double - escape.delim \ - serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - properties: - columns _col0,_col1,_col2 - columns.types string,string,double - escape.delim \ - serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - Truncated Path -> Alias: -#### A masked pattern was here #### - Needs Tagging: false - Reduce Operator Tree: - Select Operator - expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string), VALUE._col1 (type: double) - outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: mfgr_price_view + output shape: _col0: string, _col1: string, _col2: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col0 + output shape: _col0: string, _col1: string, _col2: double + partition by: _col0 + raw input shape: Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -5581,7 +6114,7 @@ GatherStats: false MultiFileSpray: false - Stage: Stage-3 + Stage: Stage-2 Map Reduce Map Operator Tree: TableScan @@ -5599,7 +6132,7 @@ Path -> Partition: #### A masked pattern was here #### Partition - base file name: -mr-10003 + base file name: -mr-10002 input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat properties: @@ -5626,6 +6159,24 @@ outputColumnNames: _col0, _col1, _col2 Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string, _col2: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(2)~ Statistics: Num rows: 13 Data size: 8021 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: string), _col1 (type: string), _col2 (type: double), _wcol0 (type: double) @@ -5645,7 +6196,7 @@ columns _col0,_col1,_col2,_col3 columns.types string:string:double:double escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -6049,6 +6600,18 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col1 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -6130,6 +6693,38 @@ outputColumnNames: _col1, _col2, _col5, _col7 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int, _col7: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col7 + name: sum + window function: GenericUDAFSumDouble + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), _wcol0 (type: int), _wcol1 (type: int), _wcol2 (type: double) @@ -6233,24 +6828,46 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col5 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(5)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE - File Output Operator - compressed: false - GlobalTableId: 0 + Select Operator + expressions: _col1 (type: string), _col2 (type: string), _col5 (type: int), _wcol0 (type: bigint) + outputColumnNames: _col1, _col2, _col5, _wcol0 + Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + GlobalTableId: 0 #### A masked pattern was here #### - NumFilesPerFileSink: 1 - table: - input format: org.apache.hadoop.mapred.SequenceFileInputFormat - output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat - properties: - columns _wcol0,_col1,_col2,_col5 - columns.types bigint,string,string,int - escape.delim \ - serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe - TotalFiles: 1 - GatherStats: false - MultiFileSpray: false + NumFilesPerFileSink: 1 + table: + input format: org.apache.hadoop.mapred.SequenceFileInputFormat + output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat + properties: + columns _col1,_col2,_col5,_wcol0 + columns.types string,string,int,bigint + escape.delim \ + serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe + TotalFiles: 1 + GatherStats: false + MultiFileSpray: false Stage: Stage-6 Map Reduce @@ -6258,8 +6875,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string), _col1 (type: string) - sort order: +++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6274,8 +6891,8 @@ input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat properties: - columns _wcol0,_col1,_col2,_col5 - columns.types bigint,string,string,int + columns _col1,_col2,_col5,_wcol0 + columns.types string,string,int,bigint escape.delim \ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe @@ -6283,8 +6900,8 @@ input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat properties: - columns _wcol0,_col1,_col2,_col5 - columns.types bigint,string,string,int + columns _col1,_col2,_col5,_wcol0 + columns.types string,string,int,bigint escape.delim \ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe @@ -6293,10 +6910,49 @@ Needs Tagging: false Reduce Operator Tree: Select Operator - expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey2 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) + expressions: VALUE._col0 (type: bigint), KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string), VALUE._col4 (type: int) outputColumnNames: _col0, _col2, _col3, _col6 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: bigint, _col2: string, _col3: string, _col6: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col3, _col2 + partition by: _col3 + raw input shape: + window functions: + window function definition + alias: _wcol1 + arguments: _col3, _col2 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col3, _col2 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol3 + arguments: _col3, _col2 + name: cume_dist + window function: GenericUDAFCumeDistEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol4 + arguments: _col6, true + name: first_value + window function: GenericUDAFFirstValueEvaluator + window frame: PRECEDING(2)~FOLLOWING(2) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col3 (type: string), _col2 (type: string), _col6 (type: int), UDFToInteger(round(_col0, 1)) (type: int), _wcol1 (type: int), _wcol2 (type: int), _wcol3 (type: double), _wcol4 (type: int) @@ -6630,8 +7286,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6694,9 +7350,49 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false GlobalTableId: 0 @@ -6721,8 +7417,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6760,6 +7456,26 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -6785,8 +7501,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -6824,6 +7540,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -6843,7 +7591,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -7078,8 +7826,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_mfgr (type: string) - sort order: ++ + key expressions: p_mfgr (type: string) + sort order: + Map-reduce partition columns: p_mfgr (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7142,6 +7890,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -7167,8 +7934,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7206,6 +7973,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -7231,8 +8010,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7270,6 +8049,18 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -7334,6 +8125,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -7353,7 +8176,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -7571,8 +8394,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7635,6 +8458,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -7660,8 +8502,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -7699,6 +8541,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -7763,6 +8624,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -7782,7 +8675,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -8012,8 +8905,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -8076,6 +8969,25 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -8101,8 +9013,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -8140,9 +9052,35 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false GlobalTableId: 0 @@ -8167,8 +9105,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -8206,6 +9144,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -8231,8 +9182,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -8270,6 +9221,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint) @@ -8289,7 +9272,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5 columns.types string:string:int:int:int:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -8550,8 +9533,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -8614,9 +9597,42 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false GlobalTableId: 0 @@ -8641,8 +9657,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col2 (type: string) - sort order: ++ + key expressions: _col2 (type: string) + sort order: + Map-reduce partition columns: _col2 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -8680,6 +9696,26 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: + transforms raw input: true + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -8705,8 +9741,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -8744,6 +9780,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col2, _col1 + partition by: _col2, _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col2, _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col2, _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -8763,7 +9831,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:bigint:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe @@ -8990,8 +10058,8 @@ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE GatherStats: false Reduce Output Operator - key expressions: p_mfgr (type: string), p_name (type: string), p_mfgr (type: string), p_name (type: string) - sort order: ++++ + key expressions: p_mfgr (type: string), p_name (type: string) + sort order: ++ Map-reduce partition columns: p_mfgr (type: string), p_name (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -9054,9 +10122,42 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: part_orc + output shape: _col1: string, _col2: string, _col5: int + type: TABLE + Partition table definition + input alias: ptf_1 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + Partition table definition + input alias: ptf_2 + name: noop + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE + Map-side function: true File Output Operator compressed: false GlobalTableId: 0 @@ -9081,8 +10182,8 @@ TableScan GatherStats: false Reduce Output Operator - key expressions: _col2 (type: string), _col1 (type: string), _col2 (type: string), _col1 (type: string) - sort order: ++++ + key expressions: _col2 (type: string), _col1 (type: string) + sort order: ++ Map-reduce partition columns: _col2 (type: string), _col1 (type: string) Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE tag: -1 @@ -9120,6 +10221,19 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: PTFCOMPONENT + Partition table definition + input alias: ptf_1 + name: noopwithmap + order by: _col2, _col1 + output shape: _col1: string, _col2: string, _col5: int + partition by: _col2, _col1 + raw input shape: + transforms raw input: true Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false @@ -9184,6 +10298,38 @@ outputColumnNames: _col1, _col2, _col5 Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col1: string, _col2: string, _col5: int + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col2 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol1 + arguments: _col1 + name: dense_rank + window function: GenericUDAFDenseRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + window function definition + alias: _wcol2 + arguments: _col5 + name: sum + window function: GenericUDAFSumLong + window frame: PRECEDING(MAX)~ Statistics: Num rows: 26 Data size: 16042 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col2 (type: string), _col1 (type: string), _wcol0 (type: int), _wcol1 (type: int), _col5 (type: int), _wcol2 (type: bigint), _wcol2 (type: bigint) @@ -9203,7 +10349,7 @@ columns _col0,_col1,_col2,_col3,_col4,_col5,_col6 columns.types string:string:int:int:int:bigint:bigint escape.delim \ - hive.serialization.extend.nesting.levels true + hive.serialization.extend.additional.nesting.levels true serialization.format 1 serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Index: ql/src/test/results/clientpositive/vectorized_shufflejoin.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_shufflejoin.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_shufflejoin.q.out (working copy) @@ -17,34 +17,34 @@ Map Operator Tree: TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + Map-reduce partition columns: _col0 (type: int) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE TableScan alias: t1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: cint is not null (type: boolean) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: cint (type: int) outputColumnNames: _col0 - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: int) sort order: + Map-reduce partition columns: _col0 (type: int) - Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE Execution mode: vectorized Reduce Operator Tree: Join Operator @@ -54,11 +54,11 @@ 0 _col0 (type: int) 1 _col0 (type: int) outputColumnNames: _col0, _col1 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: int), _col1 (type: int), (_col0 + _col1) (type: int) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 6758 Data size: 1453080 Basic stats: COMPLETE Column stats: NONE Group By Operator aggregations: count(_col0), max(_col1), min(_col0), avg(_col2) mode: hash Index: ql/src/test/results/clientpositive/vectorized_string_funcs.q.out =================================================================== --- ql/src/test/results/clientpositive/vectorized_string_funcs.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/vectorized_string_funcs.q.out (working copy) @@ -56,17 +56,17 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((((cbigint % 237) = 0) and (length(substr(cstring1, 1, 2)) <= 2)) and (cstring1 like '%')) (type: boolean) - Statistics: Num rows: 1024 Data size: 31436 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1024 Data size: 220163 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: substr(cstring1, 1, 2) (type: string), substr(cstring1, 2) (type: string), lower(cstring1) (type: string), upper(cstring1) (type: string), upper(cstring1) (type: string), length(cstring1) (type: int), trim(cstring1) (type: string), ltrim(cstring1) (type: string), rtrim(cstring1) (type: string), concat(cstring1, cstring2) (type: string), concat('>', cstring1) (type: string), concat(cstring1, '<') (type: string), concat(substr(cstring1, 1, 2), substr(cstring2, 1, 2)) (type: string) outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12 - Statistics: Num rows: 1024 Data size: 31436 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1024 Data size: 220163 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 1024 Data size: 31436 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 1024 Data size: 220163 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/windowing_streaming.q.out =================================================================== --- ql/src/test/results/clientpositive/windowing_streaming.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/windowing_streaming.q.out (working copy) @@ -80,6 +80,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col0 + partition by: _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col0 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col1 (type: string), _wcol0 (type: int) @@ -136,6 +155,25 @@ outputColumnNames: _col0, _col1 Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE PTF Operator + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: string, _col1: string + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col0 + partition by: _col1 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col0 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 < 4) (type: boolean) @@ -273,34 +311,53 @@ Map Operator Tree: TableScan alias: alltypesorc - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: ctinyint (type: tinyint), cdouble (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Reduce Output Operator key expressions: _col0 (type: tinyint), _col1 (type: double) sort order: ++ Map-reduce partition columns: _col0 (type: tinyint) - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE TopN Hash Memory Usage: 0.8 Reduce Operator Tree: Select Operator expressions: KEY.reducesinkkey0 (type: tinyint), KEY.reducesinkkey1 (type: double) outputColumnNames: _col0, _col1 - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE PTF Operator - Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE + Function definitions: + Input definition + input alias: ptf_0 + output shape: _col0: tinyint, _col1: double + type: WINDOWING + Windowing table definition + input alias: ptf_1 + name: windowingtablefunction + order by: _col1 + partition by: _col0 + raw input shape: + window functions: + window function definition + alias: _wcol0 + arguments: _col1 + name: rank + window function: GenericUDAFRankEvaluator + window frame: PRECEDING(MAX)~FOLLOWING(MAX) + isPivotResult: true + Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: (_wcol0 < 5) (type: boolean) - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: _col0 (type: tinyint), _col1 (type: double), _wcol0 (type: int) outputColumnNames: _col0, _col1, _col2 - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false - Statistics: Num rows: 4096 Data size: 125745 Basic stats: COMPLETE Column stats: NONE + Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat Index: ql/src/test/results/clientpositive/windowing_windowspec.q.out =================================================================== --- ql/src/test/results/clientpositive/windowing_windowspec.q.out (revision 1661246) +++ ql/src/test/results/clientpositive/windowing_windowspec.q.out (working copy) @@ -830,3 +830,20 @@ alice allen 65662 20.0 alice allen 65670 20.0 alice allen 65720 20.0 +PREHOOK: query: -- HIVE-9228 +select s, i from ( select s, i, round((avg(d) over w1 + 10.0) - (avg(d) over w1 - 10.0),2) from over10k window w1 as (partition by s order by i)) X limit 7 +PREHOOK: type: QUERY +PREHOOK: Input: default@over10k +#### A masked pattern was here #### +POSTHOOK: query: -- HIVE-9228 +select s, i from ( select s, i, round((avg(d) over w1 + 10.0) - (avg(d) over w1 - 10.0),2) from over10k window w1 as (partition by s order by i)) X limit 7 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@over10k +#### A masked pattern was here #### +alice allen 65545 +alice allen 65557 +alice allen 65600 +alice allen 65609 +alice allen 65662 +alice allen 65670 +alice allen 65720 Index: serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/DelimitedJSONSerDe.java (working copy) @@ -24,6 +24,7 @@ import org.apache.commons.logging.LogFactory; import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector.Category; import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory; @@ -53,7 +54,7 @@ @Override protected void serializeField(ByteStream.Output out, Object obj, ObjectInspector objInspector, - SerDeParameters serdeParams) throws SerDeException { + LazySerDeParameters serdeParams) throws SerDeException { if (!objInspector.getCategory().equals(Category.PRIMITIVE) || (objInspector.getTypeName().equalsIgnoreCase(serdeConstants.BINARY_TYPE_NAME))) { //do this for all complex types and binary try { Index: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java (working copy) @@ -342,7 +342,8 @@ currentFileSchema = fileSchema; } } - return worker(datum, currentFileSchema, schema, SchemaToTypeInfo.generateTypeInfo(schema)); + return worker(datum, currentFileSchema, schema, + SchemaToTypeInfo.generateTypeInfo(schema, null)); } Index: serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java (working copy) @@ -23,6 +23,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Set; import org.apache.avro.Schema; import org.apache.avro.Schema.Field; @@ -152,10 +153,12 @@ final InstanceCache> enums = new InstanceCache>() { @Override - protected InstanceCache makeInstance(final Schema schema) { + protected InstanceCache makeInstance(final Schema schema, + Set seenSchemas) { return new InstanceCache() { @Override - protected GenericEnumSymbol makeInstance(Object seed) { + protected GenericEnumSymbol makeInstance(Object seed, + Set seenSchemas) { return new GenericData.EnumSymbol(schema, seed.toString()); } }; Index: serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java (working copy) @@ -21,6 +21,7 @@ import org.apache.commons.logging.LogFactory; import java.util.HashMap; +import java.util.Set; /** * Cache for objects whose creation only depends on some other set of objects @@ -41,6 +42,15 @@ * SeedObject */ public Instance retrieve(SeedObject hv) throws AvroSerdeException { + return retrieve(hv, null); + } + + /** + * Retrieve (or create if it doesn't exist) the correct Instance for this + * SeedObject using 'seenSchemas' to resolve circular references + */ + public Instance retrieve(SeedObject hv, + Set seenSchemas) throws AvroSerdeException { if(LOG.isDebugEnabled()) LOG.debug("Checking for hv: " + hv.toString()); if(cache.containsKey(hv.hashCode())) { @@ -50,10 +60,11 @@ if(LOG.isDebugEnabled()) LOG.debug("Creating new instance and storing in cache"); - Instance instance = makeInstance(hv); + Instance instance = makeInstance(hv, seenSchemas); cache.put(hv.hashCode(), instance); return instance; } - protected abstract Instance makeInstance(SeedObject hv) throws AvroSerdeException; + protected abstract Instance makeInstance(SeedObject hv, + Set seenSchemas) throws AvroSerdeException; } Index: serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java (working copy) @@ -30,8 +30,10 @@ import java.util.ArrayList; import java.util.Collections; import java.util.Hashtable; +import java.util.IdentityHashMap; import java.util.List; import java.util.Map; +import java.util.Set; import org.apache.avro.Schema; import org.apache.hadoop.hive.serde2.typeinfo.HiveDecimalUtils; @@ -82,12 +84,28 @@ * @throws AvroSerdeException for problems during conversion. */ public static List generateColumnTypes(Schema schema) throws AvroSerdeException { + return generateColumnTypes (schema, null); + } + + /** + * Generate a list of of TypeInfos from an Avro schema. This method is + * currently public due to some weirdness in deserializing unions, but + * will be made private once that is resolved. + * @param schema Schema to generate field types for + * @param seenSchemas stores schemas processed in the parsing done so far, + * helping to resolve circular references in the schema + * @return List of TypeInfos, each element of which is a TypeInfo derived + * from the schema. + * @throws AvroSerdeException for problems during conversion. + */ + public static List generateColumnTypes(Schema schema, + Set seenSchemas) throws AvroSerdeException { List fields = schema.getFields(); List types = new ArrayList(fields.size()); for (Schema.Field field : fields) { - types.add(generateTypeInfo(field.schema())); + types.add(generateTypeInfo(field.schema(), seenSchemas)); } return types; @@ -95,17 +113,22 @@ static InstanceCache typeInfoCache = new InstanceCache() { @Override - protected TypeInfo makeInstance(Schema s) throws AvroSerdeException { - return generateTypeInfoWorker(s); + protected TypeInfo makeInstance(Schema s, + Set seenSchemas) + throws AvroSerdeException { + return generateTypeInfoWorker(s, seenSchemas); } }; /** * Convert an Avro Schema into an equivalent Hive TypeInfo. * @param schema to record. Must be of record type. + * @param seenSchemas stores schemas processed in the parsing done so far, + * helping to resolve circular references in the schema * @return TypeInfo matching the Avro schema * @throws AvroSerdeException for any problems during conversion. */ - public static TypeInfo generateTypeInfo(Schema schema) throws AvroSerdeException { + public static TypeInfo generateTypeInfo(Schema schema, + Set seenSchemas) throws AvroSerdeException { // For bytes type, it can be mapped to decimal. Schema.Type type = schema.getType(); if (type == BYTES && AvroSerDe.DECIMAL_TYPE_NAME @@ -160,14 +183,16 @@ return TypeInfoFactory.timestampTypeInfo; } - return typeInfoCache.retrieve(schema); + return typeInfoCache.retrieve(schema, seenSchemas); } - private static TypeInfo generateTypeInfoWorker(Schema schema) throws AvroSerdeException { + private static TypeInfo generateTypeInfoWorker(Schema schema, + Set seenSchemas) throws AvroSerdeException { // Avro requires NULLable types to be defined as unions of some type T // and NULL. This is annoying and we're going to hide it from the user. if(AvroSerdeUtils.isNullableType(schema)) { - return generateTypeInfo(AvroSerdeUtils.getOtherTypeFromNullableType(schema)); + return generateTypeInfo( + AvroSerdeUtils.getOtherTypeFromNullableType(schema), seenSchemas); } Schema.Type type = schema.getType(); @@ -176,25 +201,33 @@ } switch(type) { - case RECORD: return generateRecordTypeInfo(schema); - case MAP: return generateMapTypeInfo(schema); - case ARRAY: return generateArrayTypeInfo(schema); - case UNION: return generateUnionTypeInfo(schema); + case RECORD: return generateRecordTypeInfo(schema, seenSchemas); + case MAP: return generateMapTypeInfo(schema, seenSchemas); + case ARRAY: return generateArrayTypeInfo(schema, seenSchemas); + case UNION: return generateUnionTypeInfo(schema, seenSchemas); case ENUM: return generateEnumTypeInfo(schema); default: throw new AvroSerdeException("Do not yet support: " + schema); } } - private static TypeInfo generateRecordTypeInfo(Schema schema) throws AvroSerdeException { + private static TypeInfo generateRecordTypeInfo(Schema schema, + Set seenSchemas) throws AvroSerdeException { assert schema.getType().equals(Schema.Type.RECORD); + if (seenSchemas == null) { + seenSchemas = Collections.newSetFromMap(new IdentityHashMap()); + } else if (seenSchemas.contains(schema)) { + return primitiveTypeToTypeInfo.get(Schema.Type.NULL); + } + seenSchemas.add(schema); + List fields = schema.getFields(); List fieldNames = new ArrayList(fields.size()); List typeInfos = new ArrayList(fields.size()); for(int i = 0; i < fields.size(); i++) { fieldNames.add(i, fields.get(i).name()); - typeInfos.add(i, generateTypeInfo(fields.get(i).schema())); + typeInfos.add(i, generateTypeInfo(fields.get(i).schema(), seenSchemas)); } return TypeInfoFactory.getStructTypeInfo(fieldNames, typeInfos); @@ -204,23 +237,26 @@ * Generate a TypeInfo for an Avro Map. This is made slightly simpler in that * Avro only allows maps with strings for keys. */ - private static TypeInfo generateMapTypeInfo(Schema schema) throws AvroSerdeException { + private static TypeInfo generateMapTypeInfo(Schema schema, + Set seenSchemas) throws AvroSerdeException { assert schema.getType().equals(Schema.Type.MAP); Schema valueType = schema.getValueType(); - TypeInfo ti = generateTypeInfo(valueType); + TypeInfo ti = generateTypeInfo(valueType, seenSchemas); return TypeInfoFactory.getMapTypeInfo(TypeInfoFactory.getPrimitiveTypeInfo("string"), ti); } - private static TypeInfo generateArrayTypeInfo(Schema schema) throws AvroSerdeException { + private static TypeInfo generateArrayTypeInfo(Schema schema, + Set seenSchemas) throws AvroSerdeException { assert schema.getType().equals(Schema.Type.ARRAY); Schema itemsType = schema.getElementType(); - TypeInfo itemsTypeInfo = generateTypeInfo(itemsType); + TypeInfo itemsTypeInfo = generateTypeInfo(itemsType, seenSchemas); return TypeInfoFactory.getListTypeInfo(itemsTypeInfo); } - private static TypeInfo generateUnionTypeInfo(Schema schema) throws AvroSerdeException { + private static TypeInfo generateUnionTypeInfo(Schema schema, + Set seenSchemas) throws AvroSerdeException { assert schema.getType().equals(Schema.Type.UNION); List types = schema.getTypes(); @@ -228,7 +264,7 @@ List typeInfos = new ArrayList(types.size()); for(Schema type : types) { - typeInfos.add(generateTypeInfo(type)); + typeInfos.add(generateTypeInfo(type, seenSchemas)); } return TypeInfoFactory.getUnionTypeInfo(typeInfos); Index: serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java (working copy) @@ -29,14 +29,12 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.ColumnProjectionUtils; -import org.apache.hadoop.hive.serde2.SerDe; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeSpec; import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.lazy.LazyFactory; import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; -import org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyObjectInspectorParametersImpl; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector.Category; import org.apache.hadoop.hive.serde2.objectinspector.StructField; @@ -60,7 +58,9 @@ serdeConstants.SERIALIZATION_LAST_COLUMN_TAKES_REST, serdeConstants.ESCAPE_CHAR, serdeConstants.SERIALIZATION_ENCODING, - LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS}) + LazySerDeParameters.SERIALIZATION_EXTEND_NESTING_LEVELS, + LazySerDeParameters.SERIALIZATION_EXTEND_ADDITIONAL_NESTING_LEVELS + }) public class ColumnarSerDe extends ColumnarSerDeBase { @Override @@ -82,7 +82,7 @@ public ColumnarSerDe() throws SerDeException { } - protected SerDeParameters serdeParams = null; + protected LazySerDeParameters serdeParams = null; /** * Initialize the SerDe given the parameters. @@ -92,7 +92,7 @@ @Override public void initialize(Configuration conf, Properties tbl) throws SerDeException { - serdeParams = LazySimpleSerDe.initSerdeParams(conf, tbl, getClass().getName()); + serdeParams = new LazySerDeParameters(conf, tbl, getClass().getName()); // Create the ObjectInspectors for the fields. Note: Currently // ColumnarObject uses same ObjectInpector as LazyStruct Index: serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java (working copy) @@ -26,8 +26,7 @@ import org.apache.hadoop.hive.serde2.ColumnProjectionUtils; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeSpec; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; -import org.apache.hadoop.hive.serde2.lazy.LazyUtils; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryFactory; import org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; @@ -63,8 +62,8 @@ @Override public void initialize(Configuration conf, Properties tbl) throws SerDeException { - SerDeParameters serdeParams = new SerDeParameters(); - LazyUtils.extractColumnInfo(tbl, serdeParams, getClass().getName()); + LazySerDeParameters serdeParams = new LazySerDeParameters(conf, tbl, getClass().getName()); + columnNames = serdeParams.getColumnNames(); columnTypes = serdeParams.getColumnTypes(); Index: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java (working copy) @@ -220,9 +220,9 @@ */ @Deprecated public static ObjectInspector createLazyObjectInspector(TypeInfo typeInfo, - byte[] separator, int separatorIndex, Text nullSequence, boolean escaped, + byte[] separators, int separatorIndex, Text nullSequence, boolean escaped, byte escapeChar, ObjectInspectorOptions option) throws SerDeException { - return createLazyObjectInspector(typeInfo, separator, separatorIndex, nullSequence, + return createLazyObjectInspector(typeInfo, separators, separatorIndex, nullSequence, escaped, escapeChar, false, option); } @@ -245,9 +245,9 @@ */ @Deprecated public static ObjectInspector createLazyObjectInspector(TypeInfo typeInfo, - byte[] separator, int separatorIndex, Text nullSequence, boolean escaped, + byte[] separators, int separatorIndex, Text nullSequence, boolean escaped, byte escapeChar) throws SerDeException { - return createLazyObjectInspector(typeInfo, separator, separatorIndex, nullSequence, + return createLazyObjectInspector(typeInfo, separators, separatorIndex, nullSequence, escaped, escapeChar, false, ObjectInspectorOptions.JAVA); } @@ -267,9 +267,9 @@ */ @Deprecated public static ObjectInspector createLazyObjectInspector(TypeInfo typeInfo, - byte[] separator, int separatorIndex, Text nullSequence, boolean escaped, + byte[] separators, int separatorIndex, Text nullSequence, boolean escaped, byte escapeChar, boolean extendedBooleanLiteral) throws SerDeException { - return createLazyObjectInspector(typeInfo, separator, separatorIndex, nullSequence, escaped, + return createLazyObjectInspector(typeInfo, separators, separatorIndex, nullSequence, escaped, escapeChar, extendedBooleanLiteral, ObjectInspectorOptions.JAVA); } @@ -289,10 +289,10 @@ */ @Deprecated public static ObjectInspector createLazyObjectInspector(TypeInfo typeInfo, - byte[] separator, int separatorIndex, Text nullSequence, boolean escaped, + byte[] separators, int separatorIndex, Text nullSequence, boolean escaped, byte escapeChar, boolean extendedBooleanLiteral, ObjectInspectorOptions option) throws SerDeException { LazyObjectInspectorParametersImpl lazyParams = new LazyObjectInspectorParametersImpl( - escaped, escapeChar, extendedBooleanLiteral, null, separator, nullSequence); + escaped, escapeChar, extendedBooleanLiteral, null, separators, nullSequence); return createLazyObjectInspector(typeInfo, separatorIndex, lazyParams, option); } @@ -332,7 +332,7 @@ .getListElementTypeInfo(), separatorIndex + 1, lazyParams, option), LazyUtils.getSeparator(lazyParams.getSeparators(), separatorIndex), lazyParams); - case STRUCT: + case STRUCT: StructTypeInfo structTypeInfo = (StructTypeInfo) typeInfo; List fieldNames = structTypeInfo.getAllStructFieldNames(); List fieldTypeInfos = structTypeInfo @@ -347,7 +347,7 @@ fieldNames, fieldObjectInspectors, null, LazyUtils.getSeparator(lazyParams.getSeparators(), separatorIndex), lazyParams, option); - case UNION: + case UNION: UnionTypeInfo unionTypeInfo = (UnionTypeInfo) typeInfo; List lazyOIs = new ArrayList(); for (TypeInfo uti : unionTypeInfo.getAllUnionObjectTypeInfos()) { @@ -357,7 +357,7 @@ return LazyObjectInspectorFactory.getLazyUnionObjectInspector(lazyOIs, LazyUtils.getSeparator(lazyParams.getSeparators(), separatorIndex), lazyParams); - } + } throw new RuntimeException("Hive LazySerDe Internal error."); } @@ -396,7 +396,7 @@ */ @Deprecated public static ObjectInspector createLazyStructInspector( - List columnNames, List typeInfos, byte[] separators, + List columnNames, List typeInfos, byte[] separators, Text nullSequence, boolean lastColumnTakesRest, boolean escaped, byte escapeChar, boolean extendedBooleanLiteral) throws SerDeException { LazyObjectInspectorParametersImpl lazyParams = new LazyObjectInspectorParametersImpl( Index: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java (working copy) @@ -19,13 +19,14 @@ package org.apache.hadoop.hive.serde2.lazy; import java.io.IOException; -import java.nio.charset.Charset; import java.util.ArrayList; import java.util.Arrays; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Properties; +import org.apache.commons.lang.ArrayUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; @@ -33,7 +34,6 @@ import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe; import org.apache.hadoop.hive.serde2.ByteStream; -import org.apache.hadoop.hive.serde2.SerDe; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeSpec; import org.apache.hadoop.hive.serde2.SerDeStats; @@ -58,6 +58,7 @@ import org.apache.hadoop.io.Writable; import org.apache.hive.common.util.HiveStringUtils; + /** * LazySimpleSerDe can be used to read the same data format as * MetadataTypedColumnsetSerDe and TCTLSeparatedProtocol. @@ -75,17 +76,16 @@ serdeConstants.SERIALIZATION_LAST_COLUMN_TAKES_REST, serdeConstants.ESCAPE_CHAR, serdeConstants.SERIALIZATION_ENCODING, - LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS}) + LazySerDeParameters.SERIALIZATION_EXTEND_NESTING_LEVELS, + LazySerDeParameters.SERIALIZATION_EXTEND_ADDITIONAL_NESTING_LEVELS + }) public class LazySimpleSerDe extends AbstractEncodingAwareSerDe { public static final Log LOG = LogFactory.getLog(LazySimpleSerDe.class .getName()); - public static final String SERIALIZATION_EXTEND_NESTING_LEVELS - = "hive.serialization.extend.nesting.levels"; + private LazySerDeParameters serdeParams = null; - public static final byte[] DefaultSeparators = {(byte) 1, (byte) 2, (byte) 3}; - private ObjectInspector cachedObjectInspector; private long serializedSize; @@ -97,11 +97,11 @@ public String toString() { return getClass().toString() + "[" - + Arrays.asList(serdeParams.separators) + + Arrays.asList(serdeParams.getSeparators()) + ":" - + ((StructTypeInfo) serdeParams.rowTypeInfo).getAllStructFieldNames() + + ((StructTypeInfo) serdeParams.getRowTypeInfo()).getAllStructFieldNames() + ":" - + ((StructTypeInfo) serdeParams.rowTypeInfo) + + ((StructTypeInfo) serdeParams.getRowTypeInfo()) .getAllStructFieldTypeInfos() + "]"; } @@ -109,100 +109,6 @@ } /** - * Return the byte value of the number string. - * - * @param altValue - * The string containing a number. - * @param defaultVal - * If the altValue does not represent a number, return the - * defaultVal. - */ - public static byte getByte(String altValue, byte defaultVal) { - if (altValue != null && altValue.length() > 0) { - try { - return Byte.valueOf(altValue).byteValue(); - } catch (NumberFormatException e) { - return (byte) altValue.charAt(0); - } - } - return defaultVal; - } - - /** - * SerDeParameters. - * - */ - public static class SerDeParameters implements LazyObjectInspectorParameters { - byte[] separators = DefaultSeparators; - String nullString; - Text nullSequence; - TypeInfo rowTypeInfo; - boolean lastColumnTakesRest; - List columnNames; - List columnTypes; - - boolean escaped; - byte escapeChar; - boolean[] needsEscape; - - boolean extendedBooleanLiteral; - List timestampFormats; - - public SerDeParameters() { - } - - public List getColumnTypes() { - return columnTypes; - } - - public List getColumnNames() { - return columnNames; - } - - public byte[] getSeparators() { - return separators; - } - - public String getNullString() { - return nullString; - } - - public Text getNullSequence() { - return nullSequence; - } - - public TypeInfo getRowTypeInfo() { - return rowTypeInfo; - } - - public boolean isLastColumnTakesRest() { - return lastColumnTakesRest; - } - - public boolean isEscaped() { - return escaped; - } - - public byte getEscapeChar() { - return escapeChar; - } - - public boolean[] getNeedsEscape() { - return needsEscape; - } - - public boolean isExtendedBooleanLiteral() { - return extendedBooleanLiteral; - } - - public List getTimestampFormats() { - return timestampFormats; - } - } - - SerDeParameters serdeParams = null; - - /** * Initialize the SerDe given the parameters. serialization.format: separator * char or byte code (only supports byte-value up to 127) columns: * ","-separated column names columns.types: ",", ":", or ";"-separated column @@ -216,8 +122,7 @@ super.initialize(job, tbl); - serdeParams = LazySimpleSerDe.initSerdeParams(job, tbl, getClass() - .getName()); + serdeParams = new LazySerDeParameters(job, tbl, getClass().getName()); // Create the ObjectInspectors for the fields cachedObjectInspector = LazyFactory.createLazyStructInspector(serdeParams @@ -227,10 +132,10 @@ .createLazyObject(cachedObjectInspector); LOG.debug(getClass().getName() + " initialized with: columnNames=" - + serdeParams.columnNames + " columnTypes=" + serdeParams.columnTypes - + " separator=" + Arrays.asList(serdeParams.separators) - + " nullstring=" + serdeParams.nullString + " lastColumnTakesRest=" - + serdeParams.lastColumnTakesRest + " timestampFormats=" + serdeParams.timestampFormats); + + serdeParams.getColumnNames() + " columnTypes=" + serdeParams.getColumnTypes() + + " separator=" + Arrays.asList(serdeParams.getSeparators()) + + " nullstring=" + serdeParams.getNullString() + " lastColumnTakesRest=" + + serdeParams.isLastColumnTakesRest() + " timestampFormats=" + serdeParams.getTimestampFormats()); serializedSize = 0; stats = new SerDeStats(); @@ -238,115 +143,6 @@ lastOperationDeserialize = false; } - public static SerDeParameters initSerdeParams(Configuration job, - Properties tbl, String serdeName) throws SerDeException { - SerDeParameters serdeParams = new SerDeParameters(); - // Read the separators: We use 8 levels of separators by default, - // and 24 if SERIALIZATION_EXTEND_NESTING_LEVELS is set to true - // The levels possible are the set of control chars that we can use as - // special delimiters, ie they should absent in the data or escaped. - // To increase this level further, we need to stop relying - // on single control chars delimiters - - serdeParams.separators = new byte[8]; - serdeParams.separators[0] = getByte(tbl.getProperty(serdeConstants.FIELD_DELIM, - tbl.getProperty(serdeConstants.SERIALIZATION_FORMAT)), DefaultSeparators[0]); - serdeParams.separators[1] = getByte(tbl - .getProperty(serdeConstants.COLLECTION_DELIM), DefaultSeparators[1]); - serdeParams.separators[2] = getByte( - tbl.getProperty(serdeConstants.MAPKEY_DELIM), DefaultSeparators[2]); - String extendedNesting = - tbl.getProperty(SERIALIZATION_EXTEND_NESTING_LEVELS); - if(extendedNesting == null || !extendedNesting.equalsIgnoreCase("true")){ - //use the default smaller set of separators for backward compatibility - for (int i = 3; i < serdeParams.separators.length; i++) { - serdeParams.separators[i] = (byte) (i + 1); - } - } - else{ - //If extended nesting is enabled, set the extended set of separator chars - - final int MAX_CTRL_CHARS = 29; - byte[] extendedSeparators = new byte[MAX_CTRL_CHARS]; - int extendedSeparatorsIdx = 0; - - //get the first 3 separators that have already been set (defaults to 1,2,3) - for(int i = 0; i < 3; i++){ - extendedSeparators[extendedSeparatorsIdx++] = serdeParams.separators[i]; - } - - for (byte asciival = 4; asciival <= MAX_CTRL_CHARS; asciival++) { - - //use only control chars that are very unlikely to be part of the string - // the following might/likely to be used in text files for strings - // 9 (horizontal tab, HT, \t, ^I) - // 10 (line feed, LF, \n, ^J), - // 12 (form feed, FF, \f, ^L), - // 13 (carriage return, CR, \r, ^M), - // 27 (escape, ESC, \e [GCC only], ^[). - - //reserving the following values for future dynamic level impl - // 30 - // 31 - - switch(asciival){ - case 9: - case 10: - case 12: - case 13: - case 27: - continue; - } - extendedSeparators[extendedSeparatorsIdx++] = asciival; - } - - serdeParams.separators = - Arrays.copyOfRange(extendedSeparators, 0, extendedSeparatorsIdx); - } - - serdeParams.nullString = tbl.getProperty( - serdeConstants.SERIALIZATION_NULL_FORMAT, "\\N"); - serdeParams.nullSequence = new Text(serdeParams.nullString); - - String lastColumnTakesRestString = tbl - .getProperty(serdeConstants.SERIALIZATION_LAST_COLUMN_TAKES_REST); - serdeParams.lastColumnTakesRest = (lastColumnTakesRestString != null && lastColumnTakesRestString - .equalsIgnoreCase("true")); - - LazyUtils.extractColumnInfo(tbl, serdeParams, serdeName); - - // Create the LazyObject for storing the rows - serdeParams.rowTypeInfo = TypeInfoFactory.getStructTypeInfo( - serdeParams.columnNames, serdeParams.columnTypes); - - // Get the escape information - String escapeProperty = tbl.getProperty(serdeConstants.ESCAPE_CHAR); - serdeParams.escaped = (escapeProperty != null); - if (serdeParams.escaped) { - serdeParams.escapeChar = getByte(escapeProperty, (byte) '\\'); - } - if (serdeParams.escaped) { - serdeParams.needsEscape = new boolean[128]; - for (int i = 0; i < 128; i++) { - serdeParams.needsEscape[i] = false; - } - serdeParams.needsEscape[serdeParams.escapeChar] = true; - for (int i = 0; i < serdeParams.separators.length; i++) { - serdeParams.needsEscape[serdeParams.separators[i]] = true; - } - } - - serdeParams.extendedBooleanLiteral = job == null ? false : - job.getBoolean(ConfVars.HIVE_LAZYSIMPLE_EXTENDED_BOOLEAN_LITERAL.varname, false); - - String[] timestampFormatsArray = - HiveStringUtils.splitAndUnEscape(tbl.getProperty(serdeConstants.TIMESTAMP_FORMATS)); - if (timestampFormatsArray != null) { - serdeParams.timestampFormats = Arrays.asList(timestampFormatsArray); - } - return serdeParams; - } - // The object for storing row data LazyStruct cachedLazyStruct; @@ -420,7 +216,7 @@ StructObjectInspector soi = (StructObjectInspector) objInspector; List fields = soi.getAllStructFieldRefs(); List list = soi.getStructFieldsDataAsList(obj); - List declaredFields = (serdeParams.rowTypeInfo != null && ((StructTypeInfo) serdeParams.rowTypeInfo) + List declaredFields = (serdeParams.getRowTypeInfo() != null && ((StructTypeInfo) serdeParams.getRowTypeInfo()) .getAllStructFieldNames().size() > 0) ? ((StructObjectInspector) getObjectInspector()) .getAllStructFieldRefs() : null; @@ -432,7 +228,7 @@ for (int i = 0; i < fields.size(); i++) { // Append the separator if needed. if (i > 0) { - serializeStream.write(serdeParams.separators[0]); + serializeStream.write(serdeParams.getSeparators()[0]); } // Get the field objectInspector and the field object. ObjectInspector foi = fields.get(i).getFieldObjectInspector(); @@ -441,7 +237,7 @@ if (declaredFields != null && i >= declaredFields.size()) { throw new SerDeException("Error: expecting " + declaredFields.size() + " but asking for field " + i + "\n" + "data=" + obj + "\n" - + "tableType=" + serdeParams.rowTypeInfo.toString() + "\n" + + "tableType=" + serdeParams.getRowTypeInfo().toString() + "\n" + "dataType=" + TypeInfoUtils.getTypeInfoFromObjectInspector(objInspector)); } @@ -460,10 +256,10 @@ } protected void serializeField(ByteStream.Output out, Object obj, ObjectInspector objInspector, - SerDeParameters serdeParams) throws SerDeException { + LazySerDeParameters serdeParams) throws SerDeException { try { - serialize(out, obj, objInspector, serdeParams.separators, 1, serdeParams.nullSequence, - serdeParams.escaped, serdeParams.escapeChar, serdeParams.needsEscape); + serialize(out, obj, objInspector, serdeParams.getSeparators(), 1, serdeParams.getNullSequence(), + serdeParams.isEscaped(), serdeParams.getEscapeChar(), serdeParams.getNeedsEscape()); } catch (IOException e) { throw new SerDeException(e); } @@ -489,9 +285,7 @@ * @param escapeChar * Which char to use as the escape char, e.g. '\\' * @param needsEscape - * Which chars needs to be escaped. This array should have size of - * 128. Negative byte values (or byte values >= 128) are never - * escaped. + * Which byte needs to be escaped for 256 bytes. * @throws IOException * @throws SerDeException */ Index: serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java (working copy) @@ -23,16 +23,13 @@ import java.io.OutputStream; import java.nio.ByteBuffer; import java.nio.charset.CharacterCodingException; -import java.util.ArrayList; import java.util.Arrays; -import java.util.Properties; +import java.util.Map; import org.apache.commons.codec.binary.Base64; -import org.apache.hadoop.hive.serde.serdeConstants; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.io.HiveCharWritable; import org.apache.hadoop.hive.serde2.io.HiveVarcharWritable; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector; @@ -48,7 +45,6 @@ import org.apache.hadoop.hive.serde2.objectinspector.primitive.ShortObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector; import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector; -import org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils; import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.Text; @@ -140,10 +136,10 @@ * @param escaped * Whether the data should be written out in an escaped way. * @param escapeChar - * if escaped, the char for prefixing special characters. + * If escaped, the char for prefixing special characters. * @param needsEscape - * if escaped, whether a specific character needs escaping. This - * array should have size of 128. + * If escaped, whether a specific character needs escaping. This + * array should have size of 256. */ public static void writeEscaped(OutputStream out, byte[] bytes, int start, int len, boolean escaped, byte escapeChar, boolean[] needsEscape) @@ -151,7 +147,7 @@ if (escaped) { int end = start + len; for (int i = start; i <= end; i++) { - if (i == end || (bytes[i] >= 0 && needsEscape[bytes[i]])) { + if (i == end || needsEscape[bytes[i] & 0xFF]) { // Converts negative byte to positive index if (i > start) { out.write(bytes, start, i - start); } @@ -176,8 +172,7 @@ * @param o * The primitive Object * @param needsEscape - * Whether a character needs escaping. This array should have size of - * 128. + * Whether a character needs escaping. */ public static void writePrimitiveUTF8(OutputStream out, Object o, PrimitiveObjectInspector oi, boolean escaped, byte escapeChar, @@ -341,43 +336,8 @@ return hash; } - public static void extractColumnInfo(Properties tbl, SerDeParameters serdeParams, - String serdeName) throws SerDeException { - // Read the configuration parameters - String columnNameProperty = tbl.getProperty(serdeConstants.LIST_COLUMNS); - // NOTE: if "columns.types" is missing, all columns will be of String type - String columnTypeProperty = tbl.getProperty(serdeConstants.LIST_COLUMN_TYPES); - // Parse the configuration parameters - if (columnNameProperty != null && columnNameProperty.length() > 0) { - serdeParams.columnNames = Arrays.asList(columnNameProperty.split(",")); - } else { - serdeParams.columnNames = new ArrayList(); - } - if (columnTypeProperty == null) { - // Default type: all string - StringBuilder sb = new StringBuilder(); - for (int i = 0; i < serdeParams.columnNames.size(); i++) { - if (i > 0) { - sb.append(":"); - } - sb.append(serdeConstants.STRING_TYPE_NAME); - } - columnTypeProperty = sb.toString(); - } - - serdeParams.columnTypes = TypeInfoUtils - .getTypeInfosFromTypeString(columnTypeProperty); - - if (serdeParams.columnNames.size() != serdeParams.columnTypes.size()) { - throw new SerDeException(serdeName + ": columns has " - + serdeParams.columnNames.size() - + " elements while columns.types has " - + serdeParams.columnTypes.size() + " elements!"); - } - } - /** * gets a byte[] with copy of data from source BytesWritable * @param sourceBw - source BytesWritable @@ -404,10 +364,15 @@ String msg = "Number of levels of nesting supported for " + "LazySimpleSerde is " + (separators.length - 1) + " Unable to work with level " + level; + + String txt = ". Use %s serde property for tables using LazySimpleSerde."; + if(separators.length < 9){ - msg += ". Use " + LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS + - " serde property for tables using LazySimpleSerde."; + msg += String.format(txt, LazySerDeParameters.SERIALIZATION_EXTEND_NESTING_LEVELS); + } else if (separators.length < 25) { + msg += String.format(txt, LazySerDeParameters.SERIALIZATION_EXTEND_ADDITIONAL_NESTING_LEVELS); } + throw new SerDeException(msg, e); } } @@ -450,6 +415,26 @@ } } + /** + * Return the byte value of the number string. + * + * @param altValue + * The string containing a number. + * @param defaultVal + * If the altValue does not represent a number, return the + * defaultVal. + */ + public static byte getByte(String altValue, byte defaultVal) { + if (altValue != null && altValue.length() > 0) { + try { + return Byte.valueOf(altValue).byteValue(); + } catch (NumberFormatException e) { + return (byte) altValue.charAt(0); + } + } + return defaultVal; + } + private LazyUtils() { // prevent instantiation } Index: serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyMapObjectInspector.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyMapObjectInspector.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyMapObjectInspector.java (working copy) @@ -101,10 +101,7 @@ @Override public Object getMapValueElement(Object data, Object key) { - if (data == null) { - return null; - } - return ((LazyMap) data).getMapValueElement(key); + return ((data==null || key == null)? null : ((LazyMap) data).getMapValueElement(key)); } @Override Index: serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryArray.java =================================================================== --- serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryArray.java (revision 1661246) +++ serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryArray.java (working copy) @@ -214,7 +214,7 @@ /** * cachedList is reused every time getList is called. Different - * LazyBianryArray instances cannot share the same cachedList. + * LazyBinaryArray instances cannot share the same cachedList. */ ArrayList cachedList; Index: serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java =================================================================== --- serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java (revision 1661246) +++ serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java (working copy) @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hive.serde2.avro; +import org.apache.avro.reflect.ReflectData; import org.apache.avro.Schema; import org.apache.avro.generic.GenericData; import org.apache.avro.generic.GenericRecord; @@ -486,4 +487,47 @@ assertArrayEquals(fixed.bytes(), ((GenericData.Fixed) r.get("fixed1")).bytes()); } + @Test + public void canSerializeCyclesInSchema() throws SerDeException, IOException { + // Create parent-child avro-record and avro-schema + AvroCycleParent parent = new AvroCycleParent(); + AvroCycleChild child = new AvroCycleChild(); + parent.setChild (child); + Schema parentS = ReflectData.AllowNull.get().getSchema(AvroCycleParent.class); + GenericData.Record parentRec = new GenericData.Record(parentS); + Schema childS = ReflectData.AllowNull.get().getSchema(AvroCycleChild.class); + GenericData.Record childRec = new GenericData.Record(childS); + parentRec.put("child", childRec); + + // Initialize Avro SerDe + AvroSerializer as = new AvroSerializer(); + AvroDeserializer ad = new AvroDeserializer(); + AvroObjectInspectorGenerator aoig = new AvroObjectInspectorGenerator(parentS); + ObjectInspector oi = aoig.getObjectInspector(); + List columnNames = aoig.getColumnNames(); + List columnTypes = aoig.getColumnTypes(); + + // Check serialization and deserialization + AvroGenericRecordWritable agrw = Utils.serializeAndDeserializeRecord(parentRec); + Object obj = ad.deserialize(columnNames, columnTypes, agrw, parentS); + + Writable result = as.serialize(obj, oi, columnNames, columnTypes, parentS); + assertTrue(result instanceof AvroGenericRecordWritable); + GenericRecord r2 = ((AvroGenericRecordWritable) result).getRecord(); + assertEquals(parentS, r2.getSchema()); + } + + private static class AvroCycleParent { + AvroCycleChild child; + public AvroCycleChild getChild () {return child;} + public void setChild (AvroCycleChild child) {this.child = child;} + } + + private static class AvroCycleChild { + AvroCycleParent parent; + AvroCycleChild next; + Map map; + public AvroCycleParent getParent () {return parent;} + public void setParent (AvroCycleParent parent) {this.parent = parent;} + } } Index: serde/src/test/org/apache/hadoop/hive/serde2/avro/TestInstanceCache.java =================================================================== --- serde/src/test/org/apache/hadoop/hive/serde2/avro/TestInstanceCache.java (revision 1661246) +++ serde/src/test/org/apache/hadoop/hive/serde2/avro/TestInstanceCache.java (working copy) @@ -17,6 +17,7 @@ */ package org.apache.hadoop.hive.serde2.avro; +import java.util.Set; import org.junit.Test; import static org.junit.Assert.assertSame; @@ -41,18 +42,19 @@ public void instanceCachesOnlyCreateOneInstance() throws AvroSerdeException { InstanceCache> ic = new InstanceCache>() { @Override - protected Wrapper makeInstance(Foo hv) { + protected Wrapper makeInstance(Foo hv, + Set seenSchemas) { return new Wrapper(hv); } }; Foo f1 = new Foo(); - Wrapper fc = ic.retrieve(f1); + Wrapper fc = ic.retrieve(f1, null); assertSame(f1, fc.wrapped); // Our original foo should be in the wrapper Foo f2 = new Foo(); // Different instance, same value - Wrapper fc2 = ic.retrieve(f2); + Wrapper fc2 = ic.retrieve(f2, null); assertSame(fc2,fc); // Since equiv f, should get back first container assertSame(fc2.wrapped, f1); } @@ -60,19 +62,20 @@ @Test public void instanceCacheReturnsCorrectInstances() throws AvroSerdeException { InstanceCache> ic = new InstanceCache>() { - @Override - protected Wrapper makeInstance(String hv) { - return new Wrapper(hv); - } - }; + @Override + protected Wrapper makeInstance( + String hv, Set seenSchemas) { + return new Wrapper(hv); + } + }; - Wrapper one = ic.retrieve("one"); - Wrapper two = ic.retrieve("two"); + Wrapper one = ic.retrieve("one", null); + Wrapper two = ic.retrieve("two", null); - Wrapper anotherOne = ic.retrieve("one"); + Wrapper anotherOne = ic.retrieve("one", null); assertSame(one, anotherOne); - Wrapper anotherTwo = ic.retrieve("two"); + Wrapper anotherTwo = ic.retrieve("two", null); assertSame(two, anotherTwo); } } Index: serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java =================================================================== --- serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java (revision 1661246) +++ serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyArrayMapStruct.java (working copy) @@ -638,7 +638,7 @@ private void testNestedinArrayAtLevelExtended(int nestingLevel, ObjectInspector.Category dtype) throws SerDeException { Properties tableProp = new Properties(); - tableProp.setProperty(LazySimpleSerDe.SERIALIZATION_EXTEND_NESTING_LEVELS, "true"); + tableProp.setProperty(LazySerDeParameters.SERIALIZATION_EXTEND_NESTING_LEVELS, "true"); testNestedinArrayAtLevel(nestingLevel, dtype, tableProp); } @@ -693,9 +693,10 @@ tableProp.setProperty("columns", "narray"); tableProp.setProperty("columns.types", schema.toString()); SerDeUtils.initializeSerDe(serDe, conf, tableProp, null); - + LazySerDeParameters serdeParams = new LazySerDeParameters(conf, tableProp, LazySimpleSerDe.class.getName()); + //create the serialized string for type - byte[] separators = serDe.serdeParams.getSeparators(); + byte[] separators = serdeParams.getSeparators(); System.err.println("Using separator " + (char)separators[nestingLevel]); byte [] serializedRow = null; switch(dtype){ Index: serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java =================================================================== --- serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java (revision 1661246) +++ serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazySimpleSerDe.java (working copy) @@ -17,6 +17,8 @@ */ package org.apache.hadoop.hive.serde2.lazy; +import java.io.IOException; +import java.util.ArrayList; import java.util.List; import java.util.Properties; @@ -25,13 +27,20 @@ import org.apache.commons.codec.binary.Base64; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hive.serde.serdeConstants; +import org.apache.hadoop.hive.serde2.ByteStream; import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.io.ByteWritable; import org.apache.hadoop.hive.serde2.io.DoubleWritable; import org.apache.hadoop.hive.serde2.io.ShortWritable; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils; +import org.apache.hadoop.hive.serde2.objectinspector.SimpleMapEqualComparer; import org.apache.hadoop.hive.serde2.objectinspector.StructField; import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; +import org.apache.hadoop.hive.serde2.objectinspector.TestSimpleMapEqualComparer.TextStringMapHolder; import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; @@ -79,42 +88,8 @@ } } - private void deserializeAndSerialize(LazySimpleSerDe serDe, Text t, String s, - Object[] expectedFieldsData) throws SerDeException { - // Get the row structure - StructObjectInspector oi = (StructObjectInspector) serDe - .getObjectInspector(); - List fieldRefs = oi.getAllStructFieldRefs(); - assertEquals(expectedFieldsData.length, fieldRefs.size()); - // Deserialize - Object row = serDe.deserialize(t); - for (int i = 0; i < fieldRefs.size(); i++) { - Object fieldData = oi.getStructFieldData(row, fieldRefs.get(i)); - if (fieldData != null) { - fieldData = ((LazyPrimitive) fieldData).getWritableObject(); - } - assertEquals("Field " + i, expectedFieldsData[i], fieldData); - } - // Serialize - assertEquals(Text.class, serDe.getSerializedClass()); - Text serializedText = (Text) serDe.serialize(row, oi); - assertEquals("Serialized data", s, serializedText.toString()); - } - private Properties createProperties() { - Properties tbl = new Properties(); - - // Set the configuration parameters - tbl.setProperty(serdeConstants.SERIALIZATION_FORMAT, "9"); - tbl.setProperty("columns", - "abyte,ashort,aint,along,adouble,astring,anullint,anullstring"); - tbl.setProperty("columns.types", - "tinyint:smallint:int:bigint:double:string:int:string"); - tbl.setProperty(serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL"); - return tbl; - } - /** * Test the LazySimpleSerDe class with LastColumnTakesRest option. */ @@ -199,5 +174,53 @@ throw e; } } + + Object serializeAndDeserialize(List o1, StructObjectInspector oi1, + LazySimpleSerDe serde, + LazySerDeParameters serdeParams) throws IOException, SerDeException { + ByteStream.Output serializeStream = new ByteStream.Output(); + LazySimpleSerDe.serialize(serializeStream, o1, oi1, serdeParams + .getSeparators(), 0, serdeParams.getNullSequence(), serdeParams + .isEscaped(), serdeParams.getEscapeChar(), serdeParams + .getNeedsEscape()); + Text t = new Text(serializeStream.toByteArray()); + return serde.deserialize(t); + } + + + private void deserializeAndSerialize(LazySimpleSerDe serDe, Text t, String s, + Object[] expectedFieldsData) throws SerDeException { + // Get the row structure + StructObjectInspector oi = (StructObjectInspector) serDe + .getObjectInspector(); + List fieldRefs = oi.getAllStructFieldRefs(); + assertEquals(expectedFieldsData.length, fieldRefs.size()); + // Deserialize + Object row = serDe.deserialize(t); + for (int i = 0; i < fieldRefs.size(); i++) { + Object fieldData = oi.getStructFieldData(row, fieldRefs.get(i)); + if (fieldData != null) { + fieldData = ((LazyPrimitive) fieldData).getWritableObject(); + } + assertEquals("Field " + i, expectedFieldsData[i], fieldData); + } + // Serialize + assertEquals(Text.class, serDe.getSerializedClass()); + Text serializedText = (Text) serDe.serialize(row, oi); + assertEquals("Serialized data", s, serializedText.toString()); + } + + private Properties createProperties() { + Properties tbl = new Properties(); + + // Set the configuration parameters + tbl.setProperty(serdeConstants.SERIALIZATION_FORMAT, "9"); + tbl.setProperty("columns", + "abyte,ashort,aint,along,adouble,astring,anullint,anullstring"); + tbl.setProperty("columns.types", + "tinyint:smallint:int:bigint:double:string:int:string"); + tbl.setProperty(serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL"); + return tbl; + } } Index: serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestCrossMapEqualComparer.java =================================================================== --- serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestCrossMapEqualComparer.java (revision 1661246) +++ serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestCrossMapEqualComparer.java (working copy) @@ -28,7 +28,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; import org.apache.hadoop.io.Text; @@ -78,7 +78,7 @@ Object serializeAndDeserialize(TextStringMapHolder o1, StructObjectInspector oi1, LazySimpleSerDe serde, - SerDeParameters serdeParams) throws IOException, SerDeException { + LazySerDeParameters serdeParams) throws IOException, SerDeException { ByteStream.Output serializeStream = new ByteStream.Output(); LazySimpleSerDe.serialize(serializeStream, o1, oi1, serdeParams .getSeparators(), 0, serdeParams.getNullSequence(), serdeParams @@ -99,8 +99,7 @@ Properties tbl = new Properties(); tbl.setProperty(serdeConstants.LIST_COLUMNS, ObjectInspectorUtils.getFieldNames(oi1)); tbl.setProperty(serdeConstants.LIST_COLUMN_TYPES, ObjectInspectorUtils.getFieldTypes(oi1)); - SerDeParameters serdeParams = LazySimpleSerDe.initSerdeParams(conf, tbl, - LazySimpleSerDe.class.getName()); + LazySerDeParameters serdeParams = new LazySerDeParameters(conf, tbl, LazySimpleSerDe.class.getName()); SerDeUtils.initializeSerDe(serde, conf, tbl, null); ObjectInspector oi2 = serde.getObjectInspector(); @@ -132,7 +131,7 @@ Object serializeAndDeserialize(StringTextMapHolder o1, StructObjectInspector oi1, LazySimpleSerDe serde, - SerDeParameters serdeParams) throws IOException, SerDeException { + LazySerDeParameters serdeParams) throws IOException, SerDeException { ByteStream.Output serializeStream = new ByteStream.Output(); LazySimpleSerDe.serialize(serializeStream, o1, oi1, serdeParams .getSeparators(), 0, serdeParams.getNullSequence(), serdeParams @@ -153,8 +152,7 @@ Properties tbl = new Properties(); tbl.setProperty(serdeConstants.LIST_COLUMNS, ObjectInspectorUtils.getFieldNames(oi1)); tbl.setProperty(serdeConstants.LIST_COLUMN_TYPES, ObjectInspectorUtils.getFieldTypes(oi1)); - SerDeParameters serdeParams = LazySimpleSerDe.initSerdeParams(conf, tbl, - LazySimpleSerDe.class.getName()); + LazySerDeParameters serdeParams = new LazySerDeParameters(conf, tbl, LazySimpleSerDe.class.getName()); SerDeUtils.initializeSerDe(serde, conf, tbl, null); ObjectInspector oi2 = serde.getObjectInspector(); Index: serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestSimpleMapEqualComparer.java =================================================================== --- serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestSimpleMapEqualComparer.java (revision 1661246) +++ serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestSimpleMapEqualComparer.java (working copy) @@ -28,7 +28,7 @@ import org.apache.hadoop.hive.serde2.SerDeException; import org.apache.hadoop.hive.serde2.SerDeUtils; import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe; -import org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.SerDeParameters; +import org.apache.hadoop.hive.serde2.lazy.LazySerDeParameters; import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.ObjectInspectorOptions; import org.apache.hadoop.io.Text; @@ -78,7 +78,7 @@ Object serializeAndDeserialize(TextStringMapHolder o1, StructObjectInspector oi1, LazySimpleSerDe serde, - SerDeParameters serdeParams) throws IOException, SerDeException { + LazySerDeParameters serdeParams) throws IOException, SerDeException { ByteStream.Output serializeStream = new ByteStream.Output(); LazySimpleSerDe.serialize(serializeStream, o1, oi1, serdeParams .getSeparators(), 0, serdeParams.getNullSequence(), serdeParams @@ -99,8 +99,7 @@ Properties tbl = new Properties(); tbl.setProperty(serdeConstants.LIST_COLUMNS, ObjectInspectorUtils.getFieldNames(oi1)); tbl.setProperty(serdeConstants.LIST_COLUMN_TYPES, ObjectInspectorUtils.getFieldTypes(oi1)); - SerDeParameters serdeParams = LazySimpleSerDe.initSerdeParams(conf, tbl, - LazySimpleSerDe.class.getName()); + LazySerDeParameters serdeParams = new LazySerDeParameters(conf, tbl, LazySimpleSerDe.class.getName()); SerDeUtils.initializeSerDe(serde, conf, tbl, null); ObjectInspector oi2 = serde.getObjectInspector(); @@ -132,7 +131,7 @@ Object serializeAndDeserialize(StringTextMapHolder o1, StructObjectInspector oi1, LazySimpleSerDe serde, - SerDeParameters serdeParams) throws IOException, SerDeException { + LazySerDeParameters serdeParams) throws IOException, SerDeException { ByteStream.Output serializeStream = new ByteStream.Output(); LazySimpleSerDe.serialize(serializeStream, o1, oi1, serdeParams .getSeparators(), 0, serdeParams.getNullSequence(), serdeParams @@ -153,8 +152,7 @@ Properties tbl = new Properties(); tbl.setProperty(serdeConstants.LIST_COLUMNS, ObjectInspectorUtils.getFieldNames(oi1)); tbl.setProperty(serdeConstants.LIST_COLUMN_TYPES, ObjectInspectorUtils.getFieldTypes(oi1)); - SerDeParameters serdeParams = LazySimpleSerDe.initSerdeParams(conf, tbl, - LazySimpleSerDe.class.getName()); + LazySerDeParameters serdeParams = new LazySerDeParameters(conf, tbl, LazySimpleSerDe.class.getName()); SerDeUtils.initializeSerDe(serde, conf, tbl, null); ObjectInspector oi2 = serde.getObjectInspector(); Index: service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java =================================================================== --- service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java (revision 1661246) +++ service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java (working copy) @@ -18,7 +18,6 @@ package org.apache.hive.service.auth; import java.io.IOException; -import java.net.InetAddress; import java.net.InetSocketAddress; import java.net.UnknownHostException; import java.util.ArrayList; @@ -33,6 +32,9 @@ import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.conf.HiveConf.ConfVars; +import org.apache.hadoop.hive.metastore.HiveMetaStore; +import org.apache.hadoop.hive.metastore.HiveMetaStore.HMSHandler; +import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.shims.HadoopShims.KerberosNameShim; import org.apache.hadoop.hive.shims.ShimLoader; import org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge; @@ -108,8 +110,11 @@ conf.getVar(ConfVars.HIVE_SERVER2_KERBEROS_PRINCIPAL)); // start delegation token manager try { - saslServer.startDelegationTokenSecretManager(conf, null, ServerMode.HIVESERVER2); - } catch (IOException e) { + HMSHandler baseHandler = new HiveMetaStore.HMSHandler( + "new db based metaserver", conf, true); + saslServer.startDelegationTokenSecretManager(conf, baseHandler.getMS(), ServerMode.HIVESERVER2); + } + catch (MetaException|IOException e) { throw new TTransportException("Failed to start token manager", e); } } Index: service/src/java/org/apache/hive/service/cli/CLIService.java =================================================================== --- service/src/java/org/apache/hive/service/cli/CLIService.java (revision 1661246) +++ service/src/java/org/apache/hive/service/cli/CLIService.java (working copy) @@ -78,12 +78,6 @@ @Override public synchronized void init(HiveConf hiveConf) { - try { - applyAuthorizationConfigPolicy(hiveConf); - } catch (Exception e) { - throw new RuntimeException("Error applying authorization policy on hive configuration: " - + e.getMessage(), e); - } this.hiveConf = hiveConf; sessionManager = new SessionManager(hiveServer2); addService(sessionManager); @@ -113,6 +107,13 @@ } } } + // creates connection to HMS and thus *must* occur after kerberos login above + try { + applyAuthorizationConfigPolicy(hiveConf); + } catch (Exception e) { + throw new RuntimeException("Error applying authorization policy on hive configuration: " + + e.getMessage(), e); + } setupBlockedUdfs(); super.init(hiveConf); } Index: service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java =================================================================== --- service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java (revision 1661246) +++ service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java (working copy) @@ -85,7 +85,8 @@ } else { // in non verbose mode, show only select logger messages String[] inclLoggerNames = { "org.apache.hadoop.mapreduce.JobSubmitter", - "org.apache.hadoop.mapreduce.Job", "SessionState", Task.class.getName() }; + "org.apache.hadoop.mapreduce.Job", "SessionState", Task.class.getName(), + "org.apache.hadoop.hive.ql.exec.spark.status.SparkJobMonitor"}; addFilter(new NameFilter(false, inclLoggerNames)); } } Index: service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java =================================================================== --- service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java (revision 1661246) +++ service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java (working copy) @@ -30,6 +30,7 @@ import java.util.concurrent.Future; import java.util.concurrent.RejectedExecutionException; +import org.apache.commons.codec.binary.Base64; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.metastore.api.FieldSchema; import org.apache.hadoop.hive.metastore.api.Schema; @@ -92,6 +93,14 @@ try { driver = new Driver(sqlOperationConf, getParentSession().getUserName()); + + // set the operation handle information in Driver, so that thrift API users + // can use the operation handle they receive, to lookup query information in + // Yarn ATS + String guid64 = Base64.encodeBase64URLSafeString(getHandle().getHandleIdentifier() + .toTHandleIdentifier().getGuid()).trim(); + driver.setOperationId(guid64); + // In Hive server mode, we are not able to retry in the FetchTask // case, when calling fetch queries since execute() has returned. // For now, we disable the test attempts. Index: service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java =================================================================== --- service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java (revision 1661246) +++ service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java (working copy) @@ -20,7 +20,7 @@ import java.util.Arrays; import java.util.concurrent.ExecutorService; -import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.SynchronousQueue; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; @@ -63,9 +63,10 @@ httpServer = new org.eclipse.jetty.server.Server(); // Server thread pool + // Start with minWorkerThreads, expand till maxWorkerThreads and reject subsequent requests String threadPoolName = "HiveServer2-HttpHandler-Pool"; ExecutorService executorService = new ThreadPoolExecutor(minWorkerThreads, maxWorkerThreads, - workerKeepAliveTime, TimeUnit.SECONDS, new LinkedBlockingQueue(), + workerKeepAliveTime, TimeUnit.SECONDS, new SynchronousQueue(), new ThreadFactoryWithGarbageCleanup(threadPoolName)); ExecutorThreadPool threadPool = new ExecutorThreadPool(executorService); httpServer.setThreadPool(threadPool); Index: shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java =================================================================== --- shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java (revision 1661246) +++ shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java (working copy) @@ -39,6 +39,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.crypto.key.KeyProvider; import org.apache.hadoop.crypto.key.KeyProvider.Options; +import org.apache.hadoop.crypto.key.KeyProviderCryptoExtension; import org.apache.hadoop.crypto.key.KeyProviderFactory; import org.apache.hadoop.fs.BlockLocation; import org.apache.hadoop.fs.DefaultFileAccess; @@ -511,8 +512,10 @@ // Need to set the client's KeyProvider to the NN's for JKS, // else the updates do not get flushed properly - miniDFSCluster.getFileSystem().getClient().setKeyProvider( - miniDFSCluster.getNameNode().getNamesystem().getProvider()); + KeyProviderCryptoExtension keyProvider = miniDFSCluster.getNameNode().getNamesystem().getProvider(); + if (keyProvider != null) { + miniDFSCluster.getFileSystem().getClient().setKeyProvider(keyProvider); + } cluster = new MiniDFSShim(miniDFSCluster); return cluster; Index: shims/common/pom.xml =================================================================== --- shims/common/pom.xml (revision 1661246) +++ shims/common/pom.xml (working copy) @@ -74,7 +74,6 @@ org.apache.curator curator-framework - ${curator.version} org.apache.zookeeper Index: spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java =================================================================== --- spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java (revision 1661246) +++ spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java (working copy) @@ -18,7 +18,7 @@ package org.apache.hive.spark.client; import java.io.Serializable; -import java.net.URL; +import java.net.URI; import java.util.concurrent.Future; import org.apache.hadoop.hive.common.classification.InterfaceAudience; @@ -68,10 +68,10 @@ * in cluster mode, it may reside on a different host, meaning "file:" URLs have to exist * on that node (and not on the client machine). * - * @param url The location of the jar file. + * @param uri The location of the jar file. * @return A future that can be used to monitor the operation. */ - Future addJar(URL url); + Future addJar(URI uri); /** * Adds a file to the running remote context. @@ -80,10 +80,10 @@ * in cluster mode, it may reside on a different host, meaning "file:" URLs have to exist * on that node (and not on the client machine). * - * @param url The location of the file. + * @param uri The location of the file. * @return A future that can be used to monitor the operation. */ - Future addFile(URL url); + Future addFile(URI uri); /** * Get the count of executors. Index: spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java =================================================================== --- spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java (revision 1661246) +++ spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java (working copy) @@ -17,6 +17,14 @@ package org.apache.hive.spark.client; +import com.google.common.base.Charsets; +import com.google.common.base.Joiner; +import com.google.common.base.Strings; +import com.google.common.base.Preconditions; +import com.google.common.base.Throwables; +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + import io.netty.channel.ChannelHandlerContext; import io.netty.util.concurrent.GenericFutureListener; import io.netty.util.concurrent.Promise; @@ -30,14 +38,12 @@ import java.io.OutputStreamWriter; import java.io.Serializable; import java.io.Writer; -import java.net.URL; +import java.net.URI; import java.util.List; import java.util.Map; import java.util.Properties; import java.util.UUID; import java.util.concurrent.Future; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicInteger; import org.apache.hadoop.hive.conf.HiveConf; @@ -49,14 +55,6 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import com.google.common.base.Charsets; -import com.google.common.base.Joiner; -import com.google.common.base.Strings; -import com.google.common.base.Preconditions; -import com.google.common.base.Throwables; -import com.google.common.collect.Lists; -import com.google.common.collect.Maps; - class SparkClientImpl implements SparkClient { private static final long serialVersionUID = 1L; @@ -65,6 +63,8 @@ private static final long DEFAULT_SHUTDOWN_TIMEOUT = 10000; // In milliseconds private static final String OSX_TEST_OPTS = "SPARK_OSX_TEST_OPTS"; + private static final String SPARK_HOME_ENV = "SPARK_HOME"; + private static final String SPARK_HOME_KEY = "spark.home"; private static final String DRIVER_OPTS_KEY = "spark.driver.extraJavaOptions"; private static final String EXECUTOR_OPTS_KEY = "spark.executor.extraJavaOptions"; private static final String DRIVER_EXTRA_CLASSPATH = "spark.driver.extraClassPath"; @@ -153,13 +153,13 @@ } @Override - public Future addJar(URL url) { - return run(new AddJarJob(url.toString())); + public Future addJar(URI uri) { + return run(new AddJarJob(uri.toString())); } @Override - public Future addFile(URL url) { - return run(new AddFileJob(url.toString())); + public Future addFile(URI uri) { + return run(new AddFileJob(uri.toString())); } @Override @@ -213,10 +213,13 @@ // If a Spark installation is provided, use the spark-submit script. Otherwise, call the // SparkSubmit class directly, which has some caveats (like having to provide a proper // version of Guava on the classpath depending on the deploy mode). - String sparkHome = conf.get("spark.home"); + String sparkHome = conf.get(SPARK_HOME_KEY); if (sparkHome == null) { - sparkHome = System.getProperty("spark.home"); + sparkHome = System.getenv(SPARK_HOME_ENV); } + if (sparkHome == null) { + sparkHome = System.getProperty(SPARK_HOME_KEY); + } String sparkLogDir = conf.get("hive.spark.log.dir"); if (sparkLogDir == null) { if (sparkHome == null) { Index: spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java =================================================================== --- spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java (revision 1661246) +++ spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java (working copy) @@ -22,7 +22,7 @@ import java.io.FileOutputStream; import java.io.InputStream; import java.io.Serializable; -import java.net.URL; +import java.net.URI; import java.util.Arrays; import java.util.HashMap; import java.util.Map; @@ -204,7 +204,7 @@ jarFile.closeEntry(); jarFile.close(); - client.addJar(new URL("file:" + jar.getAbsolutePath())) + client.addJar(new URI("file:" + jar.getAbsolutePath())) .get(TIMEOUT, TimeUnit.SECONDS); // Need to run a Spark job to make sure the jar is added to the class loader. Monitoring @@ -220,7 +220,7 @@ fileStream.write("test file".getBytes("UTF-8")); fileStream.close(); - client.addJar(new URL("file:" + file.getAbsolutePath())) + client.addJar(new URI("file:" + file.getAbsolutePath())) .get(TIMEOUT, TimeUnit.SECONDS); // The same applies to files added with "addFile". They're only guaranteed to be available Index: testlibs/ant-contrib-1.0b3.jar =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Index: testlibs/ant-contrib.LICENSE.txt =================================================================== --- testlibs/ant-contrib.LICENSE.txt (revision 1661246) +++ testlibs/ant-contrib.LICENSE.txt (working copy) @@ -1,47 +0,0 @@ -/* - * The Apache Software License, Version 1.1 - * - * Copyright (c) 2001-2003 Ant-Contrib project. All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * - * 3. The end-user documentation included with the redistribution, if - * any, must include the following acknowlegement: - * "This product includes software developed by the - * Ant-Contrib project (http://sourceforge.net/projects/ant-contrib)." - * Alternately, this acknowlegement may appear in the software itself, - * if and wherever such third-party acknowlegements normally appear. - * - * 4. The name Ant-Contrib must not be used to endorse or promote products - * derived from this software without prior written permission. For - * written permission, please contact - * ant-contrib-developers@lists.sourceforge.net. - * - * 5. Products derived from this software may not be called "Ant-Contrib" - * nor may "Ant-Contrib" appear in their names without prior written - * permission of the Ant-Contrib project. - * - * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESSED OR IMPLIED - * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES - * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - * DISCLAIMED. IN NO EVENT SHALL THE ANT-CONTRIB PROJECT OR ITS - * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF - * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, - * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT - * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - * ==================================================================== - */