Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-337 Tajo Generic Storage Support
  3. TAJO-1614

Configuration format proposal for generic storage support

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Storage
    • Labels:
      None

      Description

      This is an example of the first draft of new storage configuration file for generic storage. It allows users to specify:

      • table spaces
      • space handlers
      • data formats
      • relationships among them

      More detailed explanation is as follows:

      • /storages specifies all storage types.
      • /storages/* specifies which handler deals with which uri scheme.
      • /formats can specify each data format or file format
      • /formats/* specifies which format is available in which storage types.
      • /formats/*/handler specifies a handler class to allows clients to get scanner or appender.
      {
        "spaces": {
          "default": {
            "uri": "hdfs://localhost:8020/tajo/warehouse",
            "configs": [
              {"x": "y"},
              {"x": "y"}
            ]
          },
          "ssd1": {
            "uri": "hdfs://localhost:8020/....",
            "configs": [
              {"x": "y"},
              {"x": "y"}
            ]
          },
          "mysql": {
            "uri": "jdbc://....",
            "configs": [
              {"x": "y"},
              {"x": "y"}
            ]
          }
        },
        
        "storages": {
          "hdfs": {
            "handler": "org.apache.tajo.storage.HdfsTablespace",
            "default-format": "text"
          },
          "file": {
            "handler": "org.apache.tajo.storage.FileTablespace",
            "default-format": "text"
          },
          "hbase": {
            "handler": "org.apache.tajo.storage.hbase.HBaseTablespace",
            "default-format": "hbase"
          },
          "s3": {
            "handler": "org.apache.tajo.storage.S3Tablespace",
            "default-format": "text"
          },
          "jdbc": {
            "handler": "org.apache.tajo.storage.JdbcTablespace",
            "default-format": "rowstore"
          }
        },
        
        "formats": {
          
          "avro": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.AvroHandler"
          },
          "csv": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.CSVHandler"
          },
          "text": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.TextHandler"
          },
          "parquet": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.ParquetHandler"
          },
          "raw": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.RawHandler"
          },
          "rcfile": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.RCFileHandler"
          },
          "row": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.RowHandler"
          },
          "sequencefile": {
            "storage-support": ["hdfs", "file", "s3"],
            "handler": "org.apache.tajo.storage.SeqFileHandler"
          },
          
          
          "hbase": {
            "storage-support": ["hbase"],
            "handler": "org.apache.tajo.storage.HbaseHandler"
          },
          "kyro": {
            "storage-support": ["hbase"],
            "handler": "org.apache.tajo.storage.KyroHandler"
          },
          
          "rowstore": {
            "storage-support": ["jdbc"],
            "handler": "org.apache.tajo.storage.RowStoreHandler"
          }
        }
      }
      

        Attachments

          Activity

            People

            • Assignee:
              hyunsik Hyunsik Choi
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: