Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3930

ArrayIndexOutOfBoundsException when building

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: all
    • Fix Version/s: v2.6.2
    • Component/s: Job Engine
    • Labels:
      None

      Description

      ArrayIndexOutOfBoundsException when building.

      I hive a cube building error with kylin-2.5.0:

      2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
      java.lang.ArrayIndexOutOfBoundsException
              at java.lang.System.arraycopy(Native Method)
              at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
              at org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
              at org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
              at org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
              at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
              at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
              at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
              at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
              at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      

      I checked the code of "NDCuboidBuilder.buildKeyInternal" method

      private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid, ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
              RowKeyEncoder rowkeyEncoder = rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
      
              // rowkey columns
              long mask = Long.highestOneBit(parentCuboid.getId());
              long parentCuboidId = parentCuboid.getId();
              long childCuboidId = childCuboid.getId();
              long parentCuboidIdActualLength = (long)Long.SIZE - Long.numberOfLeadingZeros(parentCuboid.getId());
              int index = rowKeySplitter.getBodySplitOffset(); // skip shard and cuboidId
              int offset = RowConstants.ROWKEY_SHARDID_LEN + RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
              for (int i = 0; i < parentCuboidIdActualLength; i++) {
                  if ((mask & parentCuboidId) > 0) {// if the this bit position equals
                      // 1
                      if ((mask & childCuboidId) > 0) {// if the child cuboid has this
                          // column
                          System.arraycopy(splitBuffers[index].array(), splitBuffers[index].offset(), newKeyBodyBuf.array(), offset, splitBuffers[index].length());
                          offset += splitBuffers[index].length();
                      }
                      index++;
                  }
                  mask = mask >> 1;
              }
      
              rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
          }
      

      Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube is not sharding. In my case my cube's storage type is 0, which means it is not sharding.

      So, I set offset according to cube sharding, like below:

      int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
      

      After modifying building succeeds in my environment.

        Attachments

        1. KYLIN-3930.master.01.patch
          3 kB
          Jacky Woo

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jackyWoo Jacky Woo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: