Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Catalog
    • Labels:
      None

      Description

      This patch add a MetadataProvider interface and LinkedMetadataManager;

      • MetadataProvider provides a way to access linked meta data store through an unified API.
      • LinkedMetadataManager manages all MetadataProvider instances.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user hyunsik opened a pull request:

          https://github.com/apache/tajo/pull/673

          TAJO-1735: Implement MetadataProvider and LinkedMetadataManager.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/hyunsik/tajo TAJO-1734

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/673.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #673


          commit 1cc4fb87fd218fd592934112e6709c2e81c0c0e6
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2015-07-31T09:25:46Z

          TAJO-1735: Implement MetadataProvider and LinkedMetadataManager.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/673 TAJO-1735 : Implement MetadataProvider and LinkedMetadataManager. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-1734 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/673.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #673 commit 1cc4fb87fd218fd592934112e6709c2e81c0c0e6 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2015-07-31T09:25:46Z TAJO-1735 : Implement MetadataProvider and LinkedMetadataManager.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/673#issuecomment-127482793

          Hi @hyunsik

          Could you rebase it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/673#issuecomment-127482793 Hi @hyunsik Could you rebase it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/673#issuecomment-127507990

          Overall, it looks good to me. And if you add more unit test cases for InsufficientPrivilegeException, it would be more nice.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/673#issuecomment-127507990 Overall, it looks good to me. And if you add more unit test cases for InsufficientPrivilegeException, it would be more nice.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/673#discussion_r36164948

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -97,19 +103,17 @@
          public CatalogServer() throws IOException

          { super(CatalogServer.class.getName()); this.handler = new CatalogProtocolHandler(); + this.linkedMetadataManager = new LinkedMetadataManager(Collections.EMPTY_LIST); this.builtingFuncs = new ArrayList<FunctionDesc>(); }
          • public CatalogServer(Collection<FunctionDesc> sqlFuncs) throws IOException {
          • this();
            + public CatalogServer(Set<MetadataProvider> metadataProviders, Collection<FunctionDesc> sqlFuncs) throws IOException {
              • End diff –

          Why don't you implement a constructor which needs only MetadataProvider?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/673#discussion_r36164948 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -97,19 +103,17 @@ public CatalogServer() throws IOException { super(CatalogServer.class.getName()); this.handler = new CatalogProtocolHandler(); + this.linkedMetadataManager = new LinkedMetadataManager(Collections.EMPTY_LIST); this.builtingFuncs = new ArrayList<FunctionDesc>(); } public CatalogServer(Collection<FunctionDesc> sqlFuncs) throws IOException { this(); + public CatalogServer(Set<MetadataProvider> metadataProviders, Collection<FunctionDesc> sqlFuncs) throws IOException { End diff – Why don't you implement a constructor which needs only MetadataProvider?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/673#discussion_r36164997

          — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java —
          @@ -24,6 +24,7 @@
          import org.apache.tajo.catalog.proto.CatalogProtos.*;
          import org.apache.tajo.common.TajoDataTypes.DataType;

          +import javax.annotation.Nullable;
          — End diff –

          It looks like unused package.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/673#discussion_r36164997 — Diff: tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogService.java — @@ -24,6 +24,7 @@ import org.apache.tajo.catalog.proto.CatalogProtos.*; import org.apache.tajo.common.TajoDataTypes.DataType; +import javax.annotation.Nullable; — End diff – It looks like unused package.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/673#discussion_r36165105

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -304,9 +295,25 @@ public GetTablespaceListResponse getAllTablespaces(RpcController controller, Nul
          throws ServiceException {
          rlock.lock();
          try {
          +
          + // retrieves tablespaces from catalog store
          + final List<TablespaceProto> tableSpaces = Lists.newArrayList(store.getTablespaces());
          +
          + // retrieves tablespaces from linked meta data
          + tableSpaces.addAll(Collections2.transform(linkedMetadataManager.getTablespaces(),
          — End diff –

          Could you explain why you use Collections2?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/673#discussion_r36165105 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -304,9 +295,25 @@ public GetTablespaceListResponse getAllTablespaces(RpcController controller, Nul throws ServiceException { rlock.lock(); try { + + // retrieves tablespaces from catalog store + final List<TablespaceProto> tableSpaces = Lists.newArrayList(store.getTablespaces()); + + // retrieves tablespaces from linked meta data + tableSpaces.addAll(Collections2.transform(linkedMetadataManager.getTablespaces(), — End diff – Could you explain why you use Collections2?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/673#discussion_r36165634

          — Diff: tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java —
          @@ -0,0 +1,270 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.catalog;
          +
          +import com.google.common.base.Function;
          +import com.google.common.collect.Collections2;
          +import com.google.common.collect.Lists;
          +import com.google.common.collect.Sets;
          +import org.apache.hadoop.fs.Path;
          +import org.apache.tajo.TajoConstants;
          +import org.apache.tajo.catalog.exception.UndefinedTablespaceException;
          +import org.apache.tajo.catalog.proto.CatalogProtos;
          +import org.apache.tajo.common.TajoDataTypes.Type;
          +import org.apache.tajo.conf.TajoConf;
          +import org.apache.tajo.error.Errors;
          +import org.apache.tajo.exception.TajoInternalError;
          +import org.apache.tajo.util.CommonTestingUtil;
          +import org.apache.tajo.util.KeyValueSet;
          +import org.junit.AfterClass;
          +import org.junit.BeforeClass;
          +import org.junit.Test;
          +
          +import javax.annotation.Nullable;
          +import java.io.IOException;
          +import java.net.URI;
          +import java.util.Collection;
          +import java.util.Collections;
          +
          +import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME;
          +import static org.junit.Assert.*;
          +
          +public class TestLinkedMetadataManager {
          +
          + static TableDesc TABLE1 = new TableDesc(
          + "table1",
          + new Schema(new Column[]

          {new Column("c1", Type.INT8)}),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table1")
          + );
          +
          + static TableDesc TABLE2 = new TableDesc(
          + "table2",
          + new Schema(new Column[]{new Column("c1", Type.INT8)}

          ),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table2")
          + );
          +
          + static TableDesc TABLE3 = new TableDesc(
          + "table3",
          + new Schema(new Column[]

          {new Column("c1", Type.INT8)}),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table3")
          + );
          +
          + static TableDesc TABLE4 = new TableDesc(
          + "table4",
          + new Schema(new Column[]{new Column("c1", Type.INT8)}

          ),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table4")
          + );
          +
          + static class MockupMetadataProvider1 implements MetadataProvider {
          +
          + @Override
          + public String getTablespaceName()

          { + return "space1"; + }
          +
          + @Override
          + public URI getTablespaceUri() { + return URI.create("http://space1/x"); + }
          +
          + @Override
          + public String getDatabaseName() { + return "space1"; + }

          +
          + @Override
          + public Collection<String> getCatalogs()

          { + return Lists.newArrayList("cat1", "cat2"); + }

          +
          + @Override
          + public Collection<String> getTables(@Nullable String catalog)

          { + return Lists.newArrayList("table1", "table2"); + }

          +
          + @Override
          + public TableDesc getTableDescriptor(String catalogName, String tableName) {
          + if (tableName.equals("table1"))

          { + return TABLE1; + }

          else if (tableName.equals("table2"))

          { + return TABLE2; + }

          +
          + throw new UndefinedTablespaceException(tableName);
          + }
          + }
          +
          + static class MockupMetadataProvider2 implements MetadataProvider {
          +
          + @Override
          + public String getTablespaceName()

          { + return "space2"; + }
          +
          + @Override
          + public URI getTablespaceUri() { + return URI.create("http://space2/y"); + }
          +
          + @Override
          + public String getDatabaseName() { + return "space2"; + }

          +
          + @Override
          + public Collection<String> getCatalogs()

          { + return Lists.newArrayList("cat3", "cat4"); + }

          +
          + @Override
          + public Collection<String> getTables(@Nullable String catalog)

          { + return Lists.newArrayList("table3", "table4"); + }

          +
          + @Override
          + public TableDesc getTableDescriptor(String catalogName, String tableName) {
          + if (tableName.equals("table3"))

          { + return TABLE3; + }

          else if (tableName.equals("table4"))

          { + return TABLE4; + }

          +
          + throw new UndefinedTablespaceException(tableName);
          + }
          + }
          +
          + static CatalogServer server;
          + static CatalogService catalog;
          +
          + @BeforeClass
          + public static void setUp() throws IOException {
          + TajoConf conf = new TajoConf();
          + conf.setVar(TajoConf.ConfVars.CATALOG_ADDRESS, "127.0.0.1:0");
          +
          + server = new CatalogServer(
          + Sets.newHashSet(new MockupMetadataProvider1(), new MockupMetadataProvider2()), Collections.EMPTY_LIST);
          + server.init(TestCatalog.newTajoConfForCatalogTest());
          + server.start();
          + catalog = new LocalCatalogWrapper(server);
          +
          + Path defaultTableSpace = CommonTestingUtil.getTestDir();
          +
          + if (!catalog.existTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME))

          { + catalog.createTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME, defaultTableSpace.toUri().toString()); + }

          + if (!catalog.existDatabase(DEFAULT_DATABASE_NAME))

          { + catalog.createDatabase(DEFAULT_DATABASE_NAME, TajoConstants.DEFAULT_TABLESPACE_NAME); + }

          + }
          +
          + @AfterClass
          + public static void tearDown() throws IOException

          { + server.stop(); + }

          +
          + @Test
          + public void testGetTablespaceNames() throws Exception

          { + assertEquals(Sets.newHashSet("space1", "space2", "default"), Sets.newHashSet(catalog.getAllTablespaceNames())); + }

          +
          + @Test
          + public void testGetTablespace() throws Exception

          { + CatalogProtos.TablespaceProto space1 = catalog.getTablespace("space1"); + assertEquals("space1", space1.getSpaceName()); + assertEquals("http://space1/x", space1.getUri()); + + CatalogProtos.TablespaceProto space2 = catalog.getTablespace("space2"); + assertEquals("space2", space2.getSpaceName()); + assertEquals("http://space2/y", space2.getUri()); + }

          +
          + @Test
          + public void testGetTablespaces() throws Exception {
          + Collection<String> names = Collections2.transform(catalog.getAllTablespaces(), new Function<CatalogProtos.TablespaceProto, String>() {
          — End diff –

          Exceeded character limit for one line.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/673#discussion_r36165634 — Diff: tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java — @@ -0,0 +1,270 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.catalog; + +import com.google.common.base.Function; +import com.google.common.collect.Collections2; +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; +import org.apache.hadoop.fs.Path; +import org.apache.tajo.TajoConstants; +import org.apache.tajo.catalog.exception.UndefinedTablespaceException; +import org.apache.tajo.catalog.proto.CatalogProtos; +import org.apache.tajo.common.TajoDataTypes.Type; +import org.apache.tajo.conf.TajoConf; +import org.apache.tajo.error.Errors; +import org.apache.tajo.exception.TajoInternalError; +import org.apache.tajo.util.CommonTestingUtil; +import org.apache.tajo.util.KeyValueSet; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; + +import javax.annotation.Nullable; +import java.io.IOException; +import java.net.URI; +import java.util.Collection; +import java.util.Collections; + +import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME; +import static org.junit.Assert.*; + +public class TestLinkedMetadataManager { + + static TableDesc TABLE1 = new TableDesc( + "table1", + new Schema(new Column[] {new Column("c1", Type.INT8)}), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table1") + ); + + static TableDesc TABLE2 = new TableDesc( + "table2", + new Schema(new Column[]{new Column("c1", Type.INT8)} ), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table2") + ); + + static TableDesc TABLE3 = new TableDesc( + "table3", + new Schema(new Column[] {new Column("c1", Type.INT8)}), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table3") + ); + + static TableDesc TABLE4 = new TableDesc( + "table4", + new Schema(new Column[]{new Column("c1", Type.INT8)} ), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table4") + ); + + static class MockupMetadataProvider1 implements MetadataProvider { + + @Override + public String getTablespaceName() { + return "space1"; + } + + @Override + public URI getTablespaceUri() { + return URI.create("http://space1/x"); + } + + @Override + public String getDatabaseName() { + return "space1"; + } + + @Override + public Collection<String> getCatalogs() { + return Lists.newArrayList("cat1", "cat2"); + } + + @Override + public Collection<String> getTables(@Nullable String catalog) { + return Lists.newArrayList("table1", "table2"); + } + + @Override + public TableDesc getTableDescriptor(String catalogName, String tableName) { + if (tableName.equals("table1")) { + return TABLE1; + } else if (tableName.equals("table2")) { + return TABLE2; + } + + throw new UndefinedTablespaceException(tableName); + } + } + + static class MockupMetadataProvider2 implements MetadataProvider { + + @Override + public String getTablespaceName() { + return "space2"; + } + + @Override + public URI getTablespaceUri() { + return URI.create("http://space2/y"); + } + + @Override + public String getDatabaseName() { + return "space2"; + } + + @Override + public Collection<String> getCatalogs() { + return Lists.newArrayList("cat3", "cat4"); + } + + @Override + public Collection<String> getTables(@Nullable String catalog) { + return Lists.newArrayList("table3", "table4"); + } + + @Override + public TableDesc getTableDescriptor(String catalogName, String tableName) { + if (tableName.equals("table3")) { + return TABLE3; + } else if (tableName.equals("table4")) { + return TABLE4; + } + + throw new UndefinedTablespaceException(tableName); + } + } + + static CatalogServer server; + static CatalogService catalog; + + @BeforeClass + public static void setUp() throws IOException { + TajoConf conf = new TajoConf(); + conf.setVar(TajoConf.ConfVars.CATALOG_ADDRESS, "127.0.0.1:0"); + + server = new CatalogServer( + Sets.newHashSet(new MockupMetadataProvider1(), new MockupMetadataProvider2()), Collections.EMPTY_LIST); + server.init(TestCatalog.newTajoConfForCatalogTest()); + server.start(); + catalog = new LocalCatalogWrapper(server); + + Path defaultTableSpace = CommonTestingUtil.getTestDir(); + + if (!catalog.existTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME)) { + catalog.createTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME, defaultTableSpace.toUri().toString()); + } + if (!catalog.existDatabase(DEFAULT_DATABASE_NAME)) { + catalog.createDatabase(DEFAULT_DATABASE_NAME, TajoConstants.DEFAULT_TABLESPACE_NAME); + } + } + + @AfterClass + public static void tearDown() throws IOException { + server.stop(); + } + + @Test + public void testGetTablespaceNames() throws Exception { + assertEquals(Sets.newHashSet("space1", "space2", "default"), Sets.newHashSet(catalog.getAllTablespaceNames())); + } + + @Test + public void testGetTablespace() throws Exception { + CatalogProtos.TablespaceProto space1 = catalog.getTablespace("space1"); + assertEquals("space1", space1.getSpaceName()); + assertEquals("http://space1/x", space1.getUri()); + + CatalogProtos.TablespaceProto space2 = catalog.getTablespace("space2"); + assertEquals("space2", space2.getSpaceName()); + assertEquals("http://space2/y", space2.getUri()); + } + + @Test + public void testGetTablespaces() throws Exception { + Collection<String> names = Collections2.transform(catalog.getAllTablespaces(), new Function<CatalogProtos.TablespaceProto, String>() { — End diff – Exceeded character limit for one line.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/673#discussion_r36168885

          — Diff: tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java —
          @@ -0,0 +1,270 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.catalog;
          +
          +import com.google.common.base.Function;
          +import com.google.common.collect.Collections2;
          +import com.google.common.collect.Lists;
          +import com.google.common.collect.Sets;
          +import org.apache.hadoop.fs.Path;
          +import org.apache.tajo.TajoConstants;
          +import org.apache.tajo.catalog.exception.UndefinedTablespaceException;
          +import org.apache.tajo.catalog.proto.CatalogProtos;
          +import org.apache.tajo.common.TajoDataTypes.Type;
          +import org.apache.tajo.conf.TajoConf;
          +import org.apache.tajo.error.Errors;
          +import org.apache.tajo.exception.TajoInternalError;
          +import org.apache.tajo.util.CommonTestingUtil;
          +import org.apache.tajo.util.KeyValueSet;
          +import org.junit.AfterClass;
          +import org.junit.BeforeClass;
          +import org.junit.Test;
          +
          +import javax.annotation.Nullable;
          +import java.io.IOException;
          +import java.net.URI;
          +import java.util.Collection;
          +import java.util.Collections;
          +
          +import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME;
          +import static org.junit.Assert.*;
          +
          +public class TestLinkedMetadataManager {
          +
          + static TableDesc TABLE1 = new TableDesc(
          + "table1",
          + new Schema(new Column[]

          {new Column("c1", Type.INT8)}),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table1")
          + );
          +
          + static TableDesc TABLE2 = new TableDesc(
          + "table2",
          + new Schema(new Column[]{new Column("c1", Type.INT8)}

          ),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table2")
          + );
          +
          + static TableDesc TABLE3 = new TableDesc(
          + "table3",
          + new Schema(new Column[]

          {new Column("c1", Type.INT8)}),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table3")
          + );
          +
          + static TableDesc TABLE4 = new TableDesc(
          + "table4",
          + new Schema(new Column[]{new Column("c1", Type.INT8)}

          ),
          + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table4")
          + );
          +
          + static class MockupMetadataProvider1 implements MetadataProvider {
          +
          + @Override
          + public String getTablespaceName()

          { + return "space1"; + }
          +
          + @Override
          + public URI getTablespaceUri() { + return URI.create("http://space1/x"); + }
          +
          + @Override
          + public String getDatabaseName() { + return "space1"; + }

          +
          + @Override
          + public Collection<String> getCatalogs()

          { + return Lists.newArrayList("cat1", "cat2"); + }

          +
          + @Override
          + public Collection<String> getTables(@Nullable String catalog)

          { + return Lists.newArrayList("table1", "table2"); + }

          +
          + @Override
          + public TableDesc getTableDescriptor(String catalogName, String tableName) {
          + if (tableName.equals("table1"))

          { + return TABLE1; + }

          else if (tableName.equals("table2"))

          { + return TABLE2; + }

          +
          + throw new UndefinedTablespaceException(tableName);
          + }
          + }
          +
          + static class MockupMetadataProvider2 implements MetadataProvider {
          +
          + @Override
          + public String getTablespaceName()

          { + return "space2"; + }
          +
          + @Override
          + public URI getTablespaceUri() { + return URI.create("http://space2/y"); + }
          +
          + @Override
          + public String getDatabaseName() { + return "space2"; + }

          +
          + @Override
          + public Collection<String> getCatalogs()

          { + return Lists.newArrayList("cat3", "cat4"); + }

          +
          + @Override
          + public Collection<String> getTables(@Nullable String catalog)

          { + return Lists.newArrayList("table3", "table4"); + }

          +
          + @Override
          + public TableDesc getTableDescriptor(String catalogName, String tableName) {
          + if (tableName.equals("table3"))

          { + return TABLE3; + }

          else if (tableName.equals("table4"))

          { + return TABLE4; + }

          +
          + throw new UndefinedTablespaceException(tableName);
          + }
          + }
          +
          + static CatalogServer server;
          + static CatalogService catalog;
          +
          + @BeforeClass
          + public static void setUp() throws IOException {
          + TajoConf conf = new TajoConf();
          + conf.setVar(TajoConf.ConfVars.CATALOG_ADDRESS, "127.0.0.1:0");
          +
          + server = new CatalogServer(
          + Sets.newHashSet(new MockupMetadataProvider1(), new MockupMetadataProvider2()), Collections.EMPTY_LIST);
          + server.init(TestCatalog.newTajoConfForCatalogTest());
          + server.start();
          + catalog = new LocalCatalogWrapper(server);
          +
          + Path defaultTableSpace = CommonTestingUtil.getTestDir();
          +
          + if (!catalog.existTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME))

          { + catalog.createTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME, defaultTableSpace.toUri().toString()); + }

          + if (!catalog.existDatabase(DEFAULT_DATABASE_NAME))

          { + catalog.createDatabase(DEFAULT_DATABASE_NAME, TajoConstants.DEFAULT_TABLESPACE_NAME); + }

          + }
          +
          + @AfterClass
          + public static void tearDown() throws IOException

          { + server.stop(); + }

          +
          + @Test
          + public void testGetTablespaceNames() throws Exception

          { + assertEquals(Sets.newHashSet("space1", "space2", "default"), Sets.newHashSet(catalog.getAllTablespaceNames())); + }

          +
          + @Test
          + public void testGetTablespace() throws Exception

          { + CatalogProtos.TablespaceProto space1 = catalog.getTablespace("space1"); + assertEquals("space1", space1.getSpaceName()); + assertEquals("http://space1/x", space1.getUri()); + + CatalogProtos.TablespaceProto space2 = catalog.getTablespace("space2"); + assertEquals("space2", space2.getSpaceName()); + assertEquals("http://space2/y", space2.getUri()); + }

          +
          + @Test
          + public void testGetTablespaces() throws Exception {
          + Collection<String> names = Collections2.transform(catalog.getAllTablespaces(), new Function<CatalogProtos.TablespaceProto, String>() {
          — End diff –

          fixed

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/673#discussion_r36168885 — Diff: tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java — @@ -0,0 +1,270 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.catalog; + +import com.google.common.base.Function; +import com.google.common.collect.Collections2; +import com.google.common.collect.Lists; +import com.google.common.collect.Sets; +import org.apache.hadoop.fs.Path; +import org.apache.tajo.TajoConstants; +import org.apache.tajo.catalog.exception.UndefinedTablespaceException; +import org.apache.tajo.catalog.proto.CatalogProtos; +import org.apache.tajo.common.TajoDataTypes.Type; +import org.apache.tajo.conf.TajoConf; +import org.apache.tajo.error.Errors; +import org.apache.tajo.exception.TajoInternalError; +import org.apache.tajo.util.CommonTestingUtil; +import org.apache.tajo.util.KeyValueSet; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; + +import javax.annotation.Nullable; +import java.io.IOException; +import java.net.URI; +import java.util.Collection; +import java.util.Collections; + +import static org.apache.tajo.TajoConstants.DEFAULT_DATABASE_NAME; +import static org.junit.Assert.*; + +public class TestLinkedMetadataManager { + + static TableDesc TABLE1 = new TableDesc( + "table1", + new Schema(new Column[] {new Column("c1", Type.INT8)}), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table1") + ); + + static TableDesc TABLE2 = new TableDesc( + "table2", + new Schema(new Column[]{new Column("c1", Type.INT8)} ), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table2") + ); + + static TableDesc TABLE3 = new TableDesc( + "table3", + new Schema(new Column[] {new Column("c1", Type.INT8)}), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table3") + ); + + static TableDesc TABLE4 = new TableDesc( + "table4", + new Schema(new Column[]{new Column("c1", Type.INT8)} ), + "TEXT", new KeyValueSet(), URI.create("http://space1/x/table4") + ); + + static class MockupMetadataProvider1 implements MetadataProvider { + + @Override + public String getTablespaceName() { + return "space1"; + } + + @Override + public URI getTablespaceUri() { + return URI.create("http://space1/x"); + } + + @Override + public String getDatabaseName() { + return "space1"; + } + + @Override + public Collection<String> getCatalogs() { + return Lists.newArrayList("cat1", "cat2"); + } + + @Override + public Collection<String> getTables(@Nullable String catalog) { + return Lists.newArrayList("table1", "table2"); + } + + @Override + public TableDesc getTableDescriptor(String catalogName, String tableName) { + if (tableName.equals("table1")) { + return TABLE1; + } else if (tableName.equals("table2")) { + return TABLE2; + } + + throw new UndefinedTablespaceException(tableName); + } + } + + static class MockupMetadataProvider2 implements MetadataProvider { + + @Override + public String getTablespaceName() { + return "space2"; + } + + @Override + public URI getTablespaceUri() { + return URI.create("http://space2/y"); + } + + @Override + public String getDatabaseName() { + return "space2"; + } + + @Override + public Collection<String> getCatalogs() { + return Lists.newArrayList("cat3", "cat4"); + } + + @Override + public Collection<String> getTables(@Nullable String catalog) { + return Lists.newArrayList("table3", "table4"); + } + + @Override + public TableDesc getTableDescriptor(String catalogName, String tableName) { + if (tableName.equals("table3")) { + return TABLE3; + } else if (tableName.equals("table4")) { + return TABLE4; + } + + throw new UndefinedTablespaceException(tableName); + } + } + + static CatalogServer server; + static CatalogService catalog; + + @BeforeClass + public static void setUp() throws IOException { + TajoConf conf = new TajoConf(); + conf.setVar(TajoConf.ConfVars.CATALOG_ADDRESS, "127.0.0.1:0"); + + server = new CatalogServer( + Sets.newHashSet(new MockupMetadataProvider1(), new MockupMetadataProvider2()), Collections.EMPTY_LIST); + server.init(TestCatalog.newTajoConfForCatalogTest()); + server.start(); + catalog = new LocalCatalogWrapper(server); + + Path defaultTableSpace = CommonTestingUtil.getTestDir(); + + if (!catalog.existTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME)) { + catalog.createTablespace(TajoConstants.DEFAULT_TABLESPACE_NAME, defaultTableSpace.toUri().toString()); + } + if (!catalog.existDatabase(DEFAULT_DATABASE_NAME)) { + catalog.createDatabase(DEFAULT_DATABASE_NAME, TajoConstants.DEFAULT_TABLESPACE_NAME); + } + } + + @AfterClass + public static void tearDown() throws IOException { + server.stop(); + } + + @Test + public void testGetTablespaceNames() throws Exception { + assertEquals(Sets.newHashSet("space1", "space2", "default"), Sets.newHashSet(catalog.getAllTablespaceNames())); + } + + @Test + public void testGetTablespace() throws Exception { + CatalogProtos.TablespaceProto space1 = catalog.getTablespace("space1"); + assertEquals("space1", space1.getSpaceName()); + assertEquals("http://space1/x", space1.getUri()); + + CatalogProtos.TablespaceProto space2 = catalog.getTablespace("space2"); + assertEquals("space2", space2.getSpaceName()); + assertEquals("http://space2/y", space2.getUri()); + } + + @Test + public void testGetTablespaces() throws Exception { + Collection<String> names = Collections2.transform(catalog.getAllTablespaces(), new Function<CatalogProtos.TablespaceProto, String>() { — End diff – fixed
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/673#discussion_r36168924

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -304,9 +295,25 @@ public GetTablespaceListResponse getAllTablespaces(RpcController controller, Nul
          throws ServiceException {
          rlock.lock();
          try {
          +
          + // retrieves tablespaces from catalog store
          + final List<TablespaceProto> tableSpaces = Lists.newArrayList(store.getTablespaces());
          +
          + // retrieves tablespaces from linked meta data
          + tableSpaces.addAll(Collections2.transform(linkedMetadataManager.getTablespaces(),
          — End diff –

          Some functional methods like transform and filter are in guava Collections2 class.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/673#discussion_r36168924 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -304,9 +295,25 @@ public GetTablespaceListResponse getAllTablespaces(RpcController controller, Nul throws ServiceException { rlock.lock(); try { + + // retrieves tablespaces from catalog store + final List<TablespaceProto> tableSpaces = Lists.newArrayList(store.getTablespaces()); + + // retrieves tablespaces from linked meta data + tableSpaces.addAll(Collections2.transform(linkedMetadataManager.getTablespaces(), — End diff – Some functional methods like transform and filter are in guava Collections2 class.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/673#discussion_r36169035

          — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java —
          @@ -97,19 +103,17 @@
          public CatalogServer() throws IOException

          { super(CatalogServer.class.getName()); this.handler = new CatalogProtocolHandler(); + this.linkedMetadataManager = new LinkedMetadataManager(Collections.EMPTY_LIST); this.builtingFuncs = new ArrayList<FunctionDesc>(); }
          • public CatalogServer(Collection<FunctionDesc> sqlFuncs) throws IOException {
          • this();
            + public CatalogServer(Set<MetadataProvider> metadataProviders, Collection<FunctionDesc> sqlFuncs) throws IOException {
              • End diff –

          It is because the constructor is unnecessary.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/673#discussion_r36169035 — Diff: tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java — @@ -97,19 +103,17 @@ public CatalogServer() throws IOException { super(CatalogServer.class.getName()); this.handler = new CatalogProtocolHandler(); + this.linkedMetadataManager = new LinkedMetadataManager(Collections.EMPTY_LIST); this.builtingFuncs = new ArrayList<FunctionDesc>(); } public CatalogServer(Collection<FunctionDesc> sqlFuncs) throws IOException { this(); + public CatalogServer(Set<MetadataProvider> metadataProviders, Collection<FunctionDesc> sqlFuncs) throws IOException { End diff – It is because the constructor is unnecessary.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/673#issuecomment-127533418

          rebased and reflected the comments.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/673#issuecomment-127533418 rebased and reflected the comments.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/673#issuecomment-127791640

          +1
          LGTM.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/673#issuecomment-127791640 +1 LGTM.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/673

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/673
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #431 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/431/)
          TAJO-1735: Implement MetadataProvider and LinkedMetadataManager. (hyunsik: rev f868c0e237f7523f429d0196e67652e980ca5907)

          • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/MetadataProvider.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/InsufficientPrivilegeException.java
          • CHANGES
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/LinkedMetadataManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #431 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/431/ ) TAJO-1735 : Implement MetadataProvider and LinkedMetadataManager. (hyunsik: rev f868c0e237f7523f429d0196e67652e980ca5907) tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/MetadataProvider.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/InsufficientPrivilegeException.java CHANGES tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/LinkedMetadataManager.java
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed.

          Show
          hyunsik Hyunsik Choi added a comment - committed.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-build #793 (See https://builds.apache.org/job/Tajo-master-build/793/)
          TAJO-1735: Implement MetadataProvider and LinkedMetadataManager. (hyunsik: rev f868c0e237f7523f429d0196e67652e980ca5907)

          • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/LinkedMetadataManager.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java
          • CHANGES
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/MetadataProvider.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/InsufficientPrivilegeException.java
          • tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #793 (See https://builds.apache.org/job/Tajo-master-build/793/ ) TAJO-1735 : Implement MetadataProvider and LinkedMetadataManager. (hyunsik: rev f868c0e237f7523f429d0196e67652e980ca5907) tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/LinkedMetadataManager.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestLinkedMetadataManager.java CHANGES tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/MetadataProvider.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/exception/InsufficientPrivilegeException.java tajo-catalog/tajo-catalog-client/src/main/java/org/apache/tajo/catalog/AbstractCatalogClient.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java

            People

            • Assignee:
              hyunsik Hyunsik Choi
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development