Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1166

S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT

    Details

      Description

      mvn clean install -DskipTests -Dhadoop.version=2.6.0-SNAPSHOT
      

      The above command will cause the following error:

      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[26,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[27,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[28,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[29,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[38,44] error: cannot find symbol
      [ERROR]   symbol: class S3FileSystem
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[42,10] error: cannot find symbol
      [ERROR]   symbol:   class FileSystemStore
        location: class SmallBlockS3FileSystem
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[23,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[24,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[25,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[36,48] error: cannot find symbol
      [ERROR]   symbol: class FileSystemStore
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java:[23,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[58,26] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class InMemoryFileSystemStore
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[78,28] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class InMemoryFileSystemStore
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[139,25] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class InMemoryFileSystemStore
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java:[48,10] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class INode
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java:[50,34] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class INode
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java:[58,9] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class INode
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[49,31] error: cannot find symbol
      [ERROR]   symbol:   class FileSystemStore
        location: class Holder
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/TestFileSystems.java:[24,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[27,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[28,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[29,30] error: package org.apache.hadoop.fs.s3 does not exist
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[48,10] error: cannot find symbol
      [ERROR]   symbol:   class FileSystemStore
        location: class S3OutputStream
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[70,15] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class S3OutputStream
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[72,10] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class S3OutputStream
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[78,44] error: cannot find symbol
      [ERROR]   symbol:   class FileSystemStore
        location: class S3OutputStream
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[72,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[76,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[87,4] error: cannot find symbol
      [ERROR]   symbol:   method setConf(Configuration)
        location: class SmallBlockS3FileSystem
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[90,58] error: no suitable method found for makeQualified(SmallBlockS3FileSystem)
      [ERROR]     method Path.makeQualified(URI,Path) is not applicable
            (actual and formal argument lists differ in length)
          method Path.makeQualified(FileSystem) is not applicable
            (actual argument SmallBlockS3FileSystem cannot be converted to FileSystem by method invocation conversion)
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[81,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[92,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[121,28] error: no suitable method found for makeQualified(SmallBlockS3FileSystem)
      [ERROR]     method Path.makeQualified(URI,Path) is not applicable
            (actual and formal argument lists differ in length)
          method Path.makeQualified(FileSystem) is not applicable
            (actual argument SmallBlockS3FileSystem cannot be converted to FileSystem by method invocation conversion)
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[126,29] error: no suitable method found for makeQualified(SmallBlockS3FileSystem)
      [ERROR]     method Path.makeQualified(URI,Path) is not applicable
            (actual and formal argument lists differ in length)
          method Path.makeQualified(FileSystem) is not applicable
            (actual argument SmallBlockS3FileSystem cannot be converted to FileSystem by method invocation conversion)
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[112,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[146,13] error: method mkdirs in class SmallBlockS3FileSystem cannot be applied to given types;
      [ERROR]   required: Path,FsPermission
        found: Path
        reason: actual and formal argument lists differ in length
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[152,28] error: cannot find symbol
      [ERROR]   symbol:   method getConf()
        location: class SmallBlockS3FileSystem
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[154,12] error: cannot find symbol
      [ERROR]   symbol:   variable statistics
        location: class SmallBlockS3FileSystem
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[130,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[156,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[172,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[177,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[236,11] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class SmallBlockS3FileSystem
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[227,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[270,29] error: no suitable method found for makeQualified(SmallBlockS3FileSystem)
      [ERROR]     method Path.makeQualified(URI,Path) is not applicable
            (actual and formal argument lists differ in length)
          method Path.makeQualified(FileSystem) is not applicable
            (actual argument SmallBlockS3FileSystem cannot be converted to FileSystem by method invocation conversion)
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[264,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[301,13] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class S3FileStatus
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java:[310,12] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class S3FileStatus
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[42,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[47,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[52,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[57,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[62,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[67,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[72,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[77,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[103,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[116,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[133,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[138,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[164,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java:[170,2] error: method does not override or implement a method from a supertype
      [ERROR] /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java:[111,8] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class INode
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java:[111,29] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class INode
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java:[115,26] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class INode
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/TestFileSystems.java:[64,21] error: cannot find symbol
      [ERROR]   symbol:   class S3FileSystem
        location: class TestFileSystems
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/TestFileSystems.java:[86,9] error: incompatible types
      [ERROR]   required: Object
        found:    SmallBlockS3FileSystem
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[70,45] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class S3OutputStream
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[185,4] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class S3OutputStream
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[185,22] error: cannot find symbol
      [ERROR]   symbol:   class Block
        location: class S3OutputStream
      /Users/hyunsik/Code/tajo/tajo-9/tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java:[206,20] error: cannot find symbol
      

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #106 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/106/)
        TAJO-1166: S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT. (jaehwa) (blrunner: rev 3d485ecb0112af12258a5a2bdc4a400b8df4fae8)

        • tajo-core/pom.xml
        • tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/SubQueryContainerAllocationEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactory.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/QueryUnitAttemptScheduleEvent.java
        • tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/impl/pb/TajoContainerIdPBImpl.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
        • tajo-core/src/main/proto/ContainerProtocol.proto
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java
        • tajo-core/src/main/proto/TajoWorkerProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java
        • tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java
        • tajo-core/src/main/proto/ResourceTrackerProtocol.proto
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java
        • tajo-storage/src/main/java/org/apache/tajo/storage/StorageManager.java
        • tajo-core/src/main/proto/TajoMasterProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryPBImpl.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecords.java
        • tajo-core/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
        • tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java
        • tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java
        • tajo-core/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterService.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java
        • tajo-core/src/main/proto/QueryMasterProtocol.proto
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryProvider.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java
        • tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/TestFileSystems.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #106 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/106/ ) TAJO-1166 : S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT. (jaehwa) (blrunner: rev 3d485ecb0112af12258a5a2bdc4a400b8df4fae8) tajo-core/pom.xml tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/master/event/SubQueryContainerAllocationEvent.java tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactory.java tajo-core/src/main/java/org/apache/tajo/master/event/QueryUnitAttemptScheduleEvent.java tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java tajo-core/src/main/java/org/apache/tajo/master/container/impl/pb/TajoContainerIdPBImpl.java tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java tajo-core/src/main/proto/ContainerProtocol.proto tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java tajo-core/src/main/proto/ResourceTrackerProtocol.proto CHANGES tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java tajo-storage/src/main/java/org/apache/tajo/storage/StorageManager.java tajo-core/src/main/proto/TajoMasterProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryPBImpl.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecords.java tajo-core/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterService.java tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java tajo-core/src/main/proto/QueryMasterProtocol.proto tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryProvider.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java tajo-storage/src/test/java/org/apache/tajo/storage/TestFileSystems.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #464 (See https://builds.apache.org/job/Tajo-master-build/464/)
        TAJO-1166: S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT. (jaehwa) (blrunner: rev 3d485ecb0112af12258a5a2bdc4a400b8df4fae8)

        • tajo-core/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java
        • tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java
        • CHANGES
        • tajo-core/pom.xml
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryPBImpl.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/impl/pb/TajoContainerIdPBImpl.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/QueryUnitAttemptScheduleEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
        • tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java
        • tajo-core/src/main/proto/ResourceTrackerProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/event/SubQueryContainerAllocationEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java
        • tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecords.java
        • tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java
        • tajo-core/src/main/proto/TajoWorkerProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java
        • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java
        • tajo-core/src/main/proto/TajoMasterProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java
        • tajo-storage/src/main/java/org/apache/tajo/storage/StorageManager.java
        • tajo-core/src/main/proto/QueryMasterProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactory.java
        • tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java
        • tajo-storage/src/test/java/org/apache/tajo/storage/TestFileSystems.java
        • tajo-core/src/main/proto/ContainerProtocol.proto
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java
        • tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java
        • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterService.java
        • tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryProvider.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #464 (See https://builds.apache.org/job/Tajo-master-build/464/ ) TAJO-1166 : S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT. (jaehwa) (blrunner: rev 3d485ecb0112af12258a5a2bdc4a400b8df4fae8) tajo-core/src/main/java/org/apache/tajo/master/DefaultTaskScheduler.java tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java CHANGES tajo-core/pom.xml tajo-storage/src/test/java/org/apache/tajo/storage/s3/INode.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java tajo-storage/src/test/java/org/apache/tajo/storage/s3/S3OutputStream.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryPBImpl.java tajo-core/src/main/java/org/apache/tajo/master/container/impl/pb/TajoContainerIdPBImpl.java tajo-storage/src/test/java/org/apache/tajo/storage/s3/InMemoryFileSystemStore.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/event/QueryUnitAttemptScheduleEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java tajo-core/src/main/java/org/apache/tajo/master/LazyTaskScheduler.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java tajo-core/src/main/proto/ResourceTrackerProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/event/SubQueryContainerAllocationEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecords.java tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-storage/src/test/java/org/apache/tajo/storage/s3/SmallBlockS3FileSystem.java tajo-core/src/main/proto/TajoMasterProtocol.proto tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java tajo-storage/src/main/java/org/apache/tajo/storage/StorageManager.java tajo-core/src/main/proto/QueryMasterProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactory.java tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java tajo-storage/src/test/java/org/apache/tajo/storage/TestFileSystems.java tajo-core/src/main/proto/ContainerProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterService.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoRecordFactoryProvider.java
        Hide
        blrunner Jaehwa Jung added a comment -

        Jinho Kim, thanks for your quick review.
        I've just committed it to master branch.

        Show
        blrunner Jaehwa Jung added a comment - Jinho Kim , thanks for your quick review. I've just committed it to master branch.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/262

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/262
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/262#issuecomment-64763244

        +1 Looks good to me.
        Ship it!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/262#issuecomment-64763244 +1 Looks good to me. Ship it!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/262#issuecomment-64750332

        Hi @jinossy

        Thanks for your review. I added a comment to indicate the source for yarn and I fixed a few document bugs.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/262#issuecomment-64750332 Hi @jinossy Thanks for your review. I added a comment to indicate the source for yarn and I fixed a few document bugs.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/262#issuecomment-64747249

        @blrunner
        Could you add comment to record that we borrow this class from YARN? It would be useful for other guys to maintain the code in the future.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/262#issuecomment-64747249 @blrunner Could you add comment to record that we borrow this class from YARN? It would be useful for other guys to maintain the code in the future.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/262#issuecomment-63771106

        For reference, I also tested some TPC-H queries on a real cluster.
        In addition, all test has passed successfully.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/262#issuecomment-63771106 For reference, I also tested some TPC-H queries on a real cluster. In addition, all test has passed successfully.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/262#issuecomment-63769261

        As I mentioned, I replaced S3FileSystem to LocalFileSystem for unit test. But in this case, we can't test split size because of result data size is less than HDFS default block size. So, I added "fs.local.block.size" to storage-site.xml for unit test. I found that this branch build had succeed on hadoop-2.5.1 and hadoop-2.6.0-SNAPSHOT.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/262#issuecomment-63769261 As I mentioned, I replaced S3FileSystem to LocalFileSystem for unit test. But in this case, we can't test split size because of result data size is less than HDFS default block size. So, I added "fs.local.block.size" to storage-site.xml for unit test. I found that this branch build had succeed on hadoop-2.5.1 and hadoop-2.6.0-SNAPSHOT.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user blrunner opened a pull request:

        https://github.com/apache/tajo/pull/262

        TAJO-1166: S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT

        I implemented for this issue as follows.

        • Replace S3 to LocalFileSystem for unit test cases
        • Added container package (borrowed from hadoop-2.5.1)

        You can build tajo on both hadoop-2.5.1 and hadoop-2.6.0-SNAPSHOT.

        • mvn clean install -Dhadoop.version=2.6.0-SNAPSHOT
        • mvn clean install -Dhadoop.version=2.5.1

        I think that we're able to bump up hadoop version easily for this patch.
        But, we need to implement our own container in the long-term.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/blrunner/tajo TAJO-1166_3

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/262.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #262


        commit 0f48003e4d8464d5da45dce3869aea0bfe3c1606
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-19T17:02:44Z

        Initial Commit


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/262 TAJO-1166 : S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT I implemented for this issue as follows. Replace S3 to LocalFileSystem for unit test cases Added container package (borrowed from hadoop-2.5.1) You can build tajo on both hadoop-2.5.1 and hadoop-2.6.0-SNAPSHOT. mvn clean install -Dhadoop.version=2.6.0-SNAPSHOT mvn clean install -Dhadoop.version=2.5.1 I think that we're able to bump up hadoop version easily for this patch. But, we need to implement our own container in the long-term. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1166 _3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/262.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #262 commit 0f48003e4d8464d5da45dce3869aea0bfe3c1606 Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-19T17:02:44Z Initial Commit
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner closed the pull request at:

        https://github.com/apache/tajo/pull/260

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner closed the pull request at: https://github.com/apache/tajo/pull/260
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/260#issuecomment-63632041

        I found that commit logs are mixed with another patch.
        I'll create another request.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/260#issuecomment-63632041 I found that commit logs are mixed with another patch. I'll create another request.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user blrunner opened a pull request:

        https://github.com/apache/tajo/pull/260

        TAJO-1166: S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT

        I implemented for this issue as follows.

        • Replace S3 to LocalFileSystem for unit test cases
        • Added container package (borrowed from hadoop-2.5.1)

        You can build tajo on both hadoop-2.5.1 and hadoop-2.6.0-SNAPSHOT.

        mvn clean install -Dhadoop.version=2.6.0-SNAPSHOT
        mvn clean install -Dhadoop.version=2.5.1

        For reference, we need to implement our own container in the long-term.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/blrunner/tajo TAJO-1166_2

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/260.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #260


        commit 1e5d3629d58fc2af1dd3507591d4726e903f1c7a
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-12T06:04:33Z

        TAJO-1166: S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT

        commit 4d4d4f577d7cb5ea5287a305ce9bf896a121624b
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-14T06:09:24Z

        Added aws-sdk dependency.

        commit 35c55ec54ce51cad3ec09316188118e1ba513695
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-14T08:39:11Z

        Remove aws-sdk dependency

        commit 67bdc71b17b54da731eaf01b9fac51ad8983353b
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-19T06:46:47Z

        TAJO-1195: Remove unused CachedDNSResolver Class. (DaeMyung Kang via jaehwa)

        Closes #253

        commit 1b96f07b9129076a5be21bbe33c56eedc829f80e
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-19T08:49:47Z

        Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1166

        commit 85407e7b548a5953fa377d1848d25ed03b9c34a5
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2014-11-19T11:54:15Z

        Added container package.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/260 TAJO-1166 : S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT I implemented for this issue as follows. Replace S3 to LocalFileSystem for unit test cases Added container package (borrowed from hadoop-2.5.1) You can build tajo on both hadoop-2.5.1 and hadoop-2.6.0-SNAPSHOT. mvn clean install -Dhadoop.version=2.6.0-SNAPSHOT mvn clean install -Dhadoop.version=2.5.1 For reference, we need to implement our own container in the long-term. You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-1166 _2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/260.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #260 commit 1e5d3629d58fc2af1dd3507591d4726e903f1c7a Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-12T06:04:33Z TAJO-1166 : S3 related storage causes compilation error in Hadoop 2.6.0-SNAPSHOT commit 4d4d4f577d7cb5ea5287a305ce9bf896a121624b Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-14T06:09:24Z Added aws-sdk dependency. commit 35c55ec54ce51cad3ec09316188118e1ba513695 Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-14T08:39:11Z Remove aws-sdk dependency commit 67bdc71b17b54da731eaf01b9fac51ad8983353b Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-19T06:46:47Z TAJO-1195 : Remove unused CachedDNSResolver Class. (DaeMyung Kang via jaehwa) Closes #253 commit 1b96f07b9129076a5be21bbe33c56eedc829f80e Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-19T08:49:47Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1166 commit 85407e7b548a5953fa377d1848d25ed03b9c34a5 Author: JaeHwa Jung <blrunner@apache.org> Date: 2014-11-19T11:54:15Z Added container package.
        Hide
        blrunner Jaehwa Jung added a comment -

        I've just removed aws-sdk dependencies.
        And I replaced S3FileSystem to LocalFileSystem for S3 unit test case in temporary.

        Show
        blrunner Jaehwa Jung added a comment - I've just removed aws-sdk dependencies. And I replaced S3FileSystem to LocalFileSystem for S3 unit test case in temporary.
        Hide
        blrunner Jaehwa Jung added a comment -

        Hi guys.

        Thanks for your ideas and I also agree with yours. And I found two problems fixing this issue.
        The first is aws dependency. As Hyunsik said, S3 module separated to hadoop-aws.
        The second is yarn containerId primitive change. In hadoop-2.6.0, containerId primtive type changed from int to long.
        Thus, this issue need to separate S3 dependency improvement and hadoop-2.6.0 bump up.
        For the first, I'll improve S3 dependency with netflix contribution sooner or later.

        For reference, I uploaded small patch to my private repository as follows.

        You can build it as follows:

        mvn clean install -Dhadoop.version=2.6.0-SNAPSHOT 
        
        Show
        blrunner Jaehwa Jung added a comment - Hi guys. Thanks for your ideas and I also agree with yours. And I found two problems fixing this issue. The first is aws dependency. As Hyunsik said, S3 module separated to hadoop-aws. The second is yarn containerId primitive change. In hadoop-2.6.0, containerId primtive type changed from int to long. Thus, this issue need to separate S3 dependency improvement and hadoop-2.6.0 bump up. For the first, I'll improve S3 dependency with netflix contribution sooner or later. For reference, I uploaded small patch to my private repository as follows. https://github.com/blrunner/tajo/tree/TAJO-1166 You can build it as follows: mvn clean install -Dhadoop.version=2.6.0-SNAPSHOT
        Hide
        hyunsik Hyunsik Choi added a comment -

        +1 for Jinho's idea.

        In Hadoop 2.6.0-SNAPSHOT, S3 related classes seem to be moved to a separate maven module, named hadoop-aws. It may be hard for Tajo to support multiple Hadoop versions. Actually, we can solve this problem if we use both maven profile (-P) and maven -D option. But, instead, as Jinho said, it would be great to have our own S3 implementation based on some other implementation. Netflix contributions look promising.

        Show
        hyunsik Hyunsik Choi added a comment - +1 for Jinho's idea. In Hadoop 2.6.0-SNAPSHOT, S3 related classes seem to be moved to a separate maven module, named hadoop-aws. It may be hard for Tajo to support multiple Hadoop versions. Actually, we can solve this problem if we use both maven profile ( -P ) and maven -D option. But, instead, as Jinho said, it would be great to have our own S3 implementation based on some other implementation. Netflix contributions look promising.
        Hide
        jhkim Jinho Kim added a comment -

        It seems that the s3 packages moved to sub-level
        HADOOP-11074, HADOOP-10400

        I found the netflix contribution. it looks good to me
        http://www.slideshare.net/zhenxiao/presto-atnetflix
        http://techblog.netflix.com/2014/10/using-presto-in-our-big-data-platform.html

        Show
        jhkim Jinho Kim added a comment - It seems that the s3 packages moved to sub-level HADOOP-11074 , HADOOP-10400 I found the netflix contribution. it looks good to me http://www.slideshare.net/zhenxiao/presto-atnetflix http://techblog.netflix.com/2014/10/using-presto-in-our-big-data-platform.html

          People

          • Assignee:
            blrunner Jaehwa Jung
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development