Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17498

Does hive have mr-nativetask support refer to MAPREDUCE-2841

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Hive
    • Labels:
      None

      Description

      I try to implement a HivePlatform extends org.apache.hadoop.mapred.nativetask.Platform.

      /**
       * Licensed to the Apache Software Foundation (ASF) under one
       * or more contributor license agreements.  See the NOTICE file
       * distributed with this work for additional information
       * regarding copyright ownership.  The ASF licenses this file
       * to you under the Apache License, Version 2.0 (the
       * "License"); you may not use this file except in compliance
       * with the License.  You may obtain a copy of the License at
       *
       *     http://www.apache.org/licenses/LICENSE-2.0
       *
       * Unless required by applicable law or agreed to in writing, software
       * distributed under the License is distributed on an "AS IS" BASIS,
       * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       * See the License for the specific language governing permissions and
       * limitations under the License.
       */
      package org.apache.hadoop.mapred.nativetask;
      
      import org.apache.hadoop.hive.ql.io.HiveKey;
      import org.apache.hadoop.mapred.JobConf;
      import org.apache.hadoop.mapred.nativetask.serde.INativeSerializer;
      import org.apache.log4j.Logger;
      
      import java.io.DataInput;
      import java.io.DataOutput;
      import java.io.IOException;
      
      public class HivePlatform extends Platform {
      
        private static final Logger LOG = Logger.getLogger(HivePlatform.class);
      
        public HivePlatform() {
        }
      
        @Override
        public void init() throws IOException {
          registerKey("org.apache.hadoop.hive.ql.io.HiveKey", HiveKeySerializer.class);
          LOG.info("Hive platform inited");
        }
      
        @Override
        public String name() {
          return "Hive";
        }
      
        @Override
        public boolean support(String keyClassName, INativeSerializer serializer, JobConf job) {
          if (keyClassNames.contains(keyClassName) && serializer instanceof INativeComparable) {
            String nativeComparator = Constants.NATIVE_MAPOUT_KEY_COMPARATOR + "." + keyClassName;
            job.set(nativeComparator, "HivePlatform.HivePlatform::HiveKeyComparator");
            if (job.get(Constants.NATIVE_CLASS_LIBRARY_BUILDIN) == null) {
              job.set(Constants.NATIVE_CLASS_LIBRARY_BUILDIN, "HivePlatform=libnativetask.so");
            }
            return true;
          } else {
            return false;
          }
        }
      
        @Override
        public boolean define(Class comparatorClass) {
          return false;
        }
      
        public static class HiveKeySerializer implements INativeComparable, INativeSerializer<HiveKey> {
      
          public HiveKeySerializer() throws ClassNotFoundException, SecurityException, NoSuchMethodException {
          }
      
          @Override
          public int getLength(HiveKey w) throws IOException {
            return 4 + w.getLength();
          }
      
          @Override
          public void serialize(HiveKey w, DataOutput out) throws IOException {
            w.write(out);
          }
      
          @Override
          public void deserialize(DataInput in, int length, HiveKey w ) throws IOException {
            w.readFields(in);
          }
        }
      }
      

      and throws exceptions:

      Error: java.io.IOException: Initialization of all the collectors failed. Error in last collector was :Native output collector cannot be loaded; at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:415) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:442) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1700) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.io.IOException: Native output collector cannot be loaded; at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:165) at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402) ... 7 more Caused by: java.io.IOException: /PartitionBucket.h:56:pool is NULL, or comparator is not set /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(_ZN10NativeTask15HadoopExceptionC2ERKSs+0x76) [0x7ffdcbba6436] /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(_ZN10NativeTask18MapOutputCollector4initEjjPFiPKcjS2_jEPNS_14ICombineRunnerE+0x36a) [0x7ffdcbb9ad6a] /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(_ZN10NativeTask18MapOutputCollector9configureEPNS_6ConfigE+0x24a) [0x7ffdcbb9b37a] /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(_ZN10NativeTask23MCollectorOutputHandler9configureEPNS_6ConfigE+0x80) [0x7ffdcbb91b40] /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(_ZN10NativeTask12BatchHandler7onSetupEPNS_6ConfigEPcjS3_j+0xe9) [0x7ffdcbb90b29] /usr/local/hadoop-2.7.3-yarn/lib/native/libnativetask.so.1.0.0(Java_org_apache_hadoop_mapred_nativetask_NativeBatchProcessor_setupHandler+0x1fe) [0x7ffdcbb90e0e] [0x7ffde9017a34] at org.apache.hadoop.mapred.nativetask.NativeBatchProcessor.setupHandler(Native Method) at org.apache.hadoop.mapred.nativetask.NativeBatchProcessor.init(NativeBatchProcessor.java:122) at org.apache.hadoop.mapred.nativetask.NativeBatchProcessor.create(NativeBatchProcessor.java:91) at org.apache.hadoop.mapred.nativetask.handlers.NativeCollectorOnlyHandler.create(NativeCollectorOnlyHandler.java:85) at org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:161) ... 8 more
      

      My environments are:
      hadoop 2.7.3 compile base on:

           Linux x.x.x.x 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
      

      compile command:

          mvn package -DskipTests=true -Dmaven.javadoc.skip=true -Pdist,native -Dtar -Drequire.snappy  -Dbundle.snappy -Dsnappy.lib=/usr/local/lib
      

      hive running parameter:

          set mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
          set mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator;
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Feng Yuan Feng Yuan
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: