Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4298

Issue with shrunken dictionary on S3

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • v3.0.0-alpha2
    • v3.1.0
    • None
    • None

    Description

      We have run into an issue when using Kylin on S3. After moving kylin.env.hdfs-working-dir to S3, we got this exception:

      Error: java.lang.IllegalArgumentException: Wrong FS: s3://kylin-XXXXX/kylin-test/hdfs-rootdir/kylin_metadata/kylin-330f6073-7123-75f6-ea28-09daab247d0a/vds_crosswalks/dictionary_shrunken/OBJECT_MOVEMENT_EVENTS_OLAP_VIEW.OBJECT_ID, expected: hdfs://ip-24-0-2-235.us-west-2.compute.internal:8020 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:669) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1440) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1437) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1452) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1440) at org.apache.kylin.engine.mr.steps.ExtractDictionaryFromGlobalMapper.doCleanup(ExtractDictionaryFromGlobalMapper.java:142) at org.apache.kylin.engine.mr.KylinMapper.cleanup(KylinMapper.java:103) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)

      (I masked the real S3 bucket name in the above exception)

      The problem ceased after disabling the shrunken dictionary feature (setting kylin.dictionary.shrunken-from-global-enabled=false)

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            hit_lacus Xiaoxiang Yu
            ainagy Andras Istvan Nagy
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment