Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13377 AliyunOSS: improvements for stabilization and optimization
  3. HADOOP-15607

AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.1.0, 2.10.0, 2.9.1, 3.2.0, 3.0.3
    • 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2
    • None
    • None

    Description

      When I generated data with hive-tpcds tool, I got exception below:

      2018-07-16 14:50:43,680 INFO mapreduce.Job: Task Id : attempt_1531723399698_0001_m_000052_0, Status : FAILED
      Error: com.aliyun.oss.OSSException: The list of parts was not in ascending order. Parts list must specified in order by part number.
      [ErrorCode]: InvalidPartOrder
      [RequestId]: 5B4C40425FCC208D79D1EAF5
      [HostId]: 100.103.0.137
      [ResponseError]:
      <?xml version="1.0" encoding="UTF-8"?>
      <Error>
      <Code>InvalidPartOrder</Code>
      <Message>The list of parts was not in ascending order. Parts list must specified in order by part number.</Message>
      <RequestId>5B4C40425FCC208D79D1EAF5</RequestId>
      <HostId>xx.xx.xx.xx</HostId>
      <ErrorDetail>current PartNumber 3, you given part number 3is not in ascending order</ErrorDetail>
      </Error>

      at com.aliyun.oss.common.utils.ExceptionFactory.createOSSException(ExceptionFactory.java:99)
      at com.aliyun.oss.internal.OSSErrorResponseHandler.handle(OSSErrorResponseHandler.java:69)
      at com.aliyun.oss.common.comm.ServiceClient.handleResponse(ServiceClient.java:248)
      at com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:130)
      at com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:68)
      at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:94)
      at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:149)
      at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:113)
      at com.aliyun.oss.internal.OSSMultipartOperation.completeMultipartUpload(OSSMultipartOperation.java:185)
      at com.aliyun.oss.OSSClient.completeMultipartUpload(OSSClient.java:790)
      at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.completeMultipartUpload(AliyunOSSFileSystemStore.java:643)
      at org.apache.hadoop.fs.aliyun.oss.AliyunOSSBlockOutputStream.close(AliyunOSSBlockOutputStream.java:120)
      at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
      at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
      at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat$LineRecordWriter.close(TextOutputFormat.java:106)
      at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.close(MultipleOutputs.java:574)
      at org.notmysock.tpcds.GenTable$DSDGen.cleanup(GenTable.java:169)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)

       

      I reviewed code below, 

      blockId 

      has thread synchronization problem

      // code placeholder
      private void uploadCurrentPart() throws IOException {
        blockFiles.add(blockFile);
        blockStream.flush();
        blockStream.close();
        if (blockId == 0) {
          uploadId = store.getUploadId(key);
        }
        ListenableFuture<PartETag> partETagFuture =
            executorService.submit(() -> {
              PartETag partETag = store.uploadPart(blockFile, key, uploadId,
                  blockId + 1);
              return partETag;
            });
        partETagsFutures.add(partETagFuture);
        blockFile = newBlockFile();
        blockId++;
        blockStream = new BufferedOutputStream(new FileOutputStream(blockFile));
      }
      

      Attachments

        1. HADOOP-15607-branch-2.001.patch
          7 kB
          wujinhu
        2. HADOOP-15607.004.patch
          7 kB
          wujinhu
        3. HADOOP-15607.003.patch
          7 kB
          wujinhu
        4. HADOOP-15607.002.patch
          4 kB
          wujinhu
        5. HADOOP-15607.001.patch
          4 kB
          wujinhu

        Issue Links

          Activity

            People

              wujinhu wujinhu
              wujinhu wujinhu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: