Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-7292 Hive on Spark
  3. HIVE-9335

Address review items on HIVE-9257 [Spark Branch]

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: spark-branch
    • Fix Version/s: 1.1.0
    • Component/s: Spark
    • Labels:
      None

      Description

      I made a pass through HIVE-9257 and found the following issues:

      HashTableSinkOperator.java
      The fields EMPTY_OBJECT_ARRAY and EMPTY_ROW_CONTAINER are no longer constants and should not be in upper case.

      HivePairFlatMapFunction.java
      We share NumberFormat accross threads and it's not thread safe.

      KryoSerializer.java
      we eat the stack trace in deserializeJobConf

      SparkMapRecordHandler
      in processRow we should not be using StringUtils.stringifyException since LOG can handle stack traces.
      in close:

              // signal new failure to map-reduce
              LOG.error("Hit error while closing operators - failing tree");
              throw new IllegalStateException("Error while closing operators", e);
      

      Should be:

               String msg = "Error while closing operators: " + e;
              throw new IllegalStateException(msg, e);
      

      SparkSessionManagerImpl - the method canReuseSession is useless

      GenSparkSkewJoinProcessor

      +      // keep it as reference in case we need fetch work
      +//        localPlan.getAliasToFetchWork().put(small_alias.toString(),
      +//            new FetchWork(tblDir, tableDescList.get(small_alias)));
      

      GenSparkWorkWalker trim ws

      SparkCompiler remote init

      SparkEdgeProperty trim ws

      CounterStatsPublisher eat exception

      Hadoop23Shims unused import of ResourceBundles

        Attachments

        1. HIVE-9335.1-spark.patch
          16 kB
          Brock Noland
        2. HIVE-9335.2-spark.patch
          13 kB
          Brock Noland

          Issue Links

            Activity

              People

              • Assignee:
                brocknoland Brock Noland
                Reporter:
                brocknoland Brock Noland
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: