Carl SteinbachThanks for the questions and I would definitely appreciate some feedback on how to appropriately document the test strategy I used here.
In regards to you question about magic numbers in the queries, the values of effectively random, but they are important. If you look at ql/src/test/org/apache/hadoop/hive/ql/exec/vector/util/OrcFileGenerator.java which is the data generation class you'll see that those values are specified in the initializeFixedPointValues for each data type. When I created the queries I used those values where I needed scalar values to ensure that when the queries executed their predicates would be filtering on values that are guaranteed to exist.
Beyond those values, all the other data in the alltypesorc file is random, but there is a specific pattern to the data that is important for coverage. In orc and subsequently vectorization there are a number of optimizations for certain data patterns: AllValues, NoNulls, RepeatingValue, RepeatingNull. The data in alltypesorc is generated such that each column has exactly 3 batches of each data pattern. This gives us coverage for the vector expression optimizations and ensure the metadata in appropriately set on the row batch object which are reused across batches.
For the queries themselves in order to efficiently cover as much of the new vectorization functionality as I could I used a number of different techniques to create the vectorization_*.q test suites, primarily equivalence classes, and pairwise combinations.
First I divided the search space into a number of dimensions such as type, aggregate function, filter operation, arithmetic operation, etc. The types were explored as equivalence classes of long, double, time, string, and bool. Also, rather than creating a very large number of small queries the resulting vectors were grouped by compatible dimensions to reduce the number of queries.
It wouldn't be to much work to add comments into the .q files that summarize the coverage they provide based on the vectors used to create each scenario.