Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Storage
    • Labels:
      None

      Description

      This is an example of the first draft of new storage configuration file for generic storage. It allows users to specify:

      • table spaces
      • space handlers
      • data formats
      • relationships among them

      More detailed explanation is as follows:

      • /storages specifies all storage types.
      • /storages/* specifies which handler deals with which uri scheme.
      • /formats can specify each data format or file format
      • /formats/* specifies which format is available in which storage types.
      • /formats/*/handler specifies a handler class to allows clients to get scanner or appender.
      {
        "spaces": {
          "default": {
            "uri": "hdfs://localhost:8020/tajo/warehouse",
            "configs": [
              {"x": "y"},
              {"x": "y"}
            ]
          },
          "ssd1": {
            "uri": "hdfs://localhost:8020/....",
            "configs": [
              {"x": "y"},
              {"x": "y"}
            ]
          },
          "mysql": {
            "uri": "jdbc://....",
            "configs": [
              {"x": "y"},
              {"x": "y"}
            ]
          }
        },
        
        "storages": {
          "hdfs": {
            "handler": "org.apache.tajo.storage.HdfsTablespace",
            "default-format": "text"
          },
          "file": {
            "handler": "org.apache.tajo.storage.FileTablespace",
            "default-format": "text"
          },
          "hbase": {
            "handler": "org.apache.tajo.storage.hbase.HBaseTablespace",
            "default-format": "hbase"
          },
          "s3": {
            "handler": "org.apache.tajo.storage.S3Tablespace",
            "default-format": "text"
          },
          "jdbc": {
            "handler": "org.apache.tajo.storage.JdbcTablespace",
            "default-format": "rowstore"
          }
        },
        
        "formats": {
          
          "avro": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.AvroHandler"
          },
          "csv": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.CSVHandler"
          },
          "text": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.TextHandler"
          },
          "parquet": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.ParquetHandler"
          },
          "raw": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.RawHandler"
          },
          "rcfile": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.RCFileHandler"
          },
          "row": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.RowHandler"
          },
          "sequencefile": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.SeqFileHandler"
          },
          
          
          "hbase": {
            "storage-support": ["hbase"],
            "handler": "org.apache.tajo.storage.HbaseHandler"
          },
          "kyro": {
            "storage-support": ["hbase"],
            "handler": "org.apache.tajo.storage.KyroHandler"
          },
          
          "rowstore": {
            "storage-support": ["jdbc"],
            "handler": "org.apache.tajo.storage.RowStoreHandler"
          }
        }
      }
      

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        Hi Jihoon Son,

        > Is this able to be difined for storage and file types?

        Yes, it can. If a user implements their own storage and file type, they can customize this file too. I'll embed some default config file to Tajo source and also allows users to specific their own file, which overrides the default config.

        > do you have any plans to provide customizable properties or only predefined ones?

        Except for some common properties like timezone, I'll move most predefined properties to custom properties.

        Show
        hyunsik Hyunsik Choi added a comment - Hi Jihoon Son , > Is this able to be difined for storage and file types? Yes, it can. If a user implements their own storage and file type, they can customize this file too. I'll embed some default config file to Tajo source and also allows users to specific their own file, which overrides the default config. > do you have any plans to provide customizable properties or only predefined ones? Except for some common properties like timezone, I'll move most predefined properties to custom properties.
        Hide
        jihoonson Jihoon Son added a comment -

        Hi Hyunsik Choi, thanks for the nice proposal.
        It looks sufficiently general to represent wide range of storage and file types.
        I have two questions. According to your proposal, properties cab be defined for each specific table space types. Is this able to be difined for storage and file types?
        Next, do you have any plans to provide customizable properties or only predefined ones?

        Show
        jihoonson Jihoon Son added a comment - Hi Hyunsik Choi , thanks for the nice proposal. It looks sufficiently general to represent wide range of storage and file types. I have two questions. According to your proposal, properties cab be defined for each specific table space types. Is this able to be difined for storage and file types? Next, do you have any plans to provide customizable properties or only predefined ones?

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development