Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Storage
    • Labels:

      Description

      If a text file is uncompressed, scanner is able to read from direct buffer
      Other cases, it read from input stream.

      This patch includes some refactoring as follows:

      • rename properties
        • csvfile.delimiter to text.delimiter
        • csvfile.null to text.null
      • Add DelimitedLineReader
        • it will be easier to implement text-format variations.
      • Use the java Unsafe for split line and field
        • It will be better performance
      • Remove seekable interface
        • We use the CSVFile to just debugging

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jinossy opened a pull request:

          https://github.com/apache/tajo/pull/221

          TAJO-1149: Implement direct read of TextFile scanner

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jinossy/tajo TAJO-1149

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/221.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #221


          commit aed406357e016e602b11dbaed3d20994dd651a17
          Author: jhkim <jhkim@apache.org>
          Date: 2014-10-31T12:10:08Z

          TAJO-1149: Implement direct read of TextFile scanner


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/221 TAJO-1149 : Implement direct read of TextFile scanner You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1149 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/221.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #221 commit aed406357e016e602b11dbaed3d20994dd651a17 Author: jhkim <jhkim@apache.org> Date: 2014-10-31T12:10:08Z TAJO-1149 : Implement direct read of TextFile scanner
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/221#issuecomment-61458728

          I didn't change the CSVFile properties in test case, because it can test backward compatibility

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/221#issuecomment-61458728 I didn't change the CSVFile properties in test case, because it can test backward compatibility
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r19776277

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java —
          @@ -194,8 +194,8 @@ public boolean existTable(final String databaseName, final String tableName) thr
          storeType = CatalogUtil.getStoreType(HCatalogUtil.getStoreType(fileOutputformat));

          if (storeType.equals(CatalogProtos.StoreType.CSV)) {
          — End diff –

          It seems to need the condition branch for TEXT storage type.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r19776277 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java — @@ -194,8 +194,8 @@ public boolean existTable(final String databaseName, final String tableName) thr storeType = CatalogUtil.getStoreType(HCatalogUtil.getStoreType(fileOutputformat)); if (storeType.equals(CatalogProtos.StoreType.CSV)) { — End diff – It seems to need the condition branch for TEXT storage type.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r19785760

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java —
          @@ -194,8 +194,8 @@ public boolean existTable(final String databaseName, final String tableName) thr
          storeType = CatalogUtil.getStoreType(HCatalogUtil.getStoreType(fileOutputformat));

          if (storeType.equals(CatalogProtos.StoreType.CSV)) {
          — End diff –

          @hyunsik
          We store the HiveIgnoreKeyTextOutputFormat in hive meta. on the contrary to this, We should choose csv or text format.
          In my opinion, text format is good. because hivemeta of tajo is not allow the custom serder

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r19785760 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java — @@ -194,8 +194,8 @@ public boolean existTable(final String databaseName, final String tableName) thr storeType = CatalogUtil.getStoreType(HCatalogUtil.getStoreType(fileOutputformat)); if (storeType.equals(CatalogProtos.StoreType.CSV)) { — End diff – @hyunsik We store the HiveIgnoreKeyTextOutputFormat in hive meta. on the contrary to this, We should choose csv or text format. In my opinion, text format is good. because hivemeta of tajo is not allow the custom serder
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r19788115

          — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java —
          @@ -194,8 +194,8 @@ public boolean existTable(final String databaseName, final String tableName) thr
          storeType = CatalogUtil.getStoreType(HCatalogUtil.getStoreType(fileOutputformat));

          if (storeType.equals(CatalogProtos.StoreType.CSV)) {
          — End diff –

          Sounds reasonable. Please go ahead!

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r19788115 — Diff: tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java — @@ -194,8 +194,8 @@ public boolean existTable(final String databaseName, final String tableName) thr storeType = CatalogUtil.getStoreType(HCatalogUtil.getStoreType(fileOutputformat)); if (storeType.equals(CatalogProtos.StoreType.CSV)) { — End diff – Sounds reasonable. Please go ahead!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/221#issuecomment-61606007

          changed to textfile

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/221#issuecomment-61606007 changed to textfile
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r20055854

          — Diff: tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java —
          @@ -0,0 +1,173 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.storage;
          +
          +import org.apache.tajo.datum.Datum;
          +import org.apache.tajo.datum.NullDatum;
          +import org.apache.tajo.datum.ProtobufDatum;
          +
          — End diff –

          Could you give some brief description about this tuple implementation?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r20055854 — Diff: tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java — @@ -0,0 +1,173 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.storage; + +import org.apache.tajo.datum.Datum; +import org.apache.tajo.datum.NullDatum; +import org.apache.tajo.datum.ProtobufDatum; + — End diff – Could you give some brief description about this tuple implementation?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r20055859

          — Diff: tajo-common/src/main/java/org/apache/tajo/storage/StorageConstants.java —
          @@ -23,15 +23,24 @@
          public static final String COMPRESSION_CODEC = "compression.codec";
          public static final String COMPRESSION_TYPE = "compression.type";

          + @Deprecated
          public static final String CSVFILE_DELIMITER = "csvfile.delimiter";
          + @Deprecated
          public static final String CSVFILE_NULL = "csvfile.null";
          + @Deprecated
          public static final String CSVFILE_SERDE = "csvfile.serde";

          + public static final String TEXT_DELIMITER = "text.delimiter";
          — End diff –

          It looks better. We should had done it before

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r20055859 — Diff: tajo-common/src/main/java/org/apache/tajo/storage/StorageConstants.java — @@ -23,15 +23,24 @@ public static final String COMPRESSION_CODEC = "compression.codec"; public static final String COMPRESSION_TYPE = "compression.type"; + @Deprecated public static final String CSVFILE_DELIMITER = "csvfile.delimiter"; + @Deprecated public static final String CSVFILE_NULL = "csvfile.null"; + @Deprecated public static final String CSVFILE_SERDE = "csvfile.serde"; + public static final String TEXT_DELIMITER = "text.delimiter"; — End diff – It looks better. We should had done it before
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r20066678

          — Diff: tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java —
          @@ -0,0 +1,173 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.storage;
          +
          +import org.apache.tajo.datum.Datum;
          +import org.apache.tajo.datum.NullDatum;
          +import org.apache.tajo.datum.ProtobufDatum;
          +
          — End diff –

          if selected column is zero, a vtuple should allocate the unnecessary array
          ex) select count from table. Have you better idea ?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r20066678 — Diff: tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java — @@ -0,0 +1,173 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.storage; + +import org.apache.tajo.datum.Datum; +import org.apache.tajo.datum.NullDatum; +import org.apache.tajo.datum.ProtobufDatum; + — End diff – if selected column is zero, a vtuple should allocate the unnecessary array ex) select count from table. Have you better idea ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r20068950

          — Diff: tajo-storage/src/test/java/org/apache/tajo/storage/TestLineReader.java —
          @@ -0,0 +1,163 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.storage;
          +
          +import io.netty.buffer.ByteBuf;
          +import org.apache.hadoop.fs.FileStatus;
          +import org.apache.hadoop.fs.FileSystem;
          +import org.apache.hadoop.fs.Path;
          +import org.apache.hadoop.io.IOUtils;
          +import org.apache.hadoop.io.compress.DeflateCodec;
          +import org.apache.tajo.catalog.CatalogUtil;
          +import org.apache.tajo.catalog.Schema;
          +import org.apache.tajo.catalog.TableMeta;
          +import org.apache.tajo.catalog.proto.CatalogProtos.StoreType;
          +import org.apache.tajo.common.TajoDataTypes.Type;
          +import org.apache.tajo.conf.TajoConf;
          +import org.apache.tajo.datum.DatumFactory;
          +import org.apache.tajo.datum.NullDatum;
          +import org.apache.tajo.storage.fragment.FileFragment;
          +import org.apache.tajo.storage.text.ByteBufLineReader;
          +import org.apache.tajo.storage.text.DelimitedTextFile;
          +import org.apache.tajo.storage.text.DelimitedLineReader;
          +import org.apache.tajo.util.CommonTestingUtil;
          +import org.junit.Test;
          +
          +import java.io.IOException;
          +import java.util.concurrent.atomic.AtomicInteger;
          +
          +import static org.junit.Assert.*;
          +
          +public class TestLineReader {
          + private static String TEST_PATH = "target/test-data/TestLineReader";
          +
          + @Test
          + public void testByteBufLineReader() throws IOException {
          + TajoConf conf = new TajoConf();
          + Path testDir = CommonTestingUtil.getTestDir(TEST_PATH);
          + FileSystem fs = testDir.getFileSystem(conf);
          +
          + Schema schema = new Schema();
          + schema.addColumn("id", Type.INT4);
          + schema.addColumn("age", Type.INT8);
          + schema.addColumn("comment", Type.TEXT);
          + schema.addColumn("comment2", Type.TEXT);
          +
          + TableMeta meta = CatalogUtil.newTableMeta(StoreType.TEXTFILE);
          + Path tablePath = new Path(testDir, "line.data");
          + FileAppender appender = (FileAppender) StorageManager.getStorageManager(conf).getAppender(meta, schema,
          + tablePath);
          + appender.enableStats();
          + appender.init();
          + int tupleNum = 10000;
          + VTuple vTuple;
          +
          + for (int i = 0; i < tupleNum; i++) {
          + vTuple = new VTuple(4);
          + vTuple.put(0, DatumFactory.createInt4(i + 1));
          + vTuple.put(1, DatumFactory.createInt8(25l));
          + vTuple.put(2, DatumFactory.createText("emiya muljomdao"));
          — End diff –

          looks interesting.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r20068950 — Diff: tajo-storage/src/test/java/org/apache/tajo/storage/TestLineReader.java — @@ -0,0 +1,163 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.storage; + +import io.netty.buffer.ByteBuf; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.io.IOUtils; +import org.apache.hadoop.io.compress.DeflateCodec; +import org.apache.tajo.catalog.CatalogUtil; +import org.apache.tajo.catalog.Schema; +import org.apache.tajo.catalog.TableMeta; +import org.apache.tajo.catalog.proto.CatalogProtos.StoreType; +import org.apache.tajo.common.TajoDataTypes.Type; +import org.apache.tajo.conf.TajoConf; +import org.apache.tajo.datum.DatumFactory; +import org.apache.tajo.datum.NullDatum; +import org.apache.tajo.storage.fragment.FileFragment; +import org.apache.tajo.storage.text.ByteBufLineReader; +import org.apache.tajo.storage.text.DelimitedTextFile; +import org.apache.tajo.storage.text.DelimitedLineReader; +import org.apache.tajo.util.CommonTestingUtil; +import org.junit.Test; + +import java.io.IOException; +import java.util.concurrent.atomic.AtomicInteger; + +import static org.junit.Assert.*; + +public class TestLineReader { + private static String TEST_PATH = "target/test-data/TestLineReader"; + + @Test + public void testByteBufLineReader() throws IOException { + TajoConf conf = new TajoConf(); + Path testDir = CommonTestingUtil.getTestDir(TEST_PATH); + FileSystem fs = testDir.getFileSystem(conf); + + Schema schema = new Schema(); + schema.addColumn("id", Type.INT4); + schema.addColumn("age", Type.INT8); + schema.addColumn("comment", Type.TEXT); + schema.addColumn("comment2", Type.TEXT); + + TableMeta meta = CatalogUtil.newTableMeta(StoreType.TEXTFILE); + Path tablePath = new Path(testDir, "line.data"); + FileAppender appender = (FileAppender) StorageManager.getStorageManager(conf).getAppender(meta, schema, + tablePath); + appender.enableStats(); + appender.init(); + int tupleNum = 10000; + VTuple vTuple; + + for (int i = 0; i < tupleNum; i++) { + vTuple = new VTuple(4); + vTuple.put(0, DatumFactory.createInt4(i + 1)); + vTuple.put(1, DatumFactory.createInt8(25l)); + vTuple.put(2, DatumFactory.createText("emiya muljomdao")); — End diff – looks interesting.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/221#discussion_r20068981

          — Diff: tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java —
          @@ -0,0 +1,173 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.storage;
          +
          +import org.apache.tajo.datum.Datum;
          +import org.apache.tajo.datum.NullDatum;
          +import org.apache.tajo.datum.ProtobufDatum;
          +
          — End diff –

          It seems to be reasonable. Could you add the description to the class? It is hard to imagine its purpose.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/221#discussion_r20068981 — Diff: tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java — @@ -0,0 +1,173 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.storage; + +import org.apache.tajo.datum.Datum; +import org.apache.tajo.datum.NullDatum; +import org.apache.tajo.datum.ProtobufDatum; + — End diff – It seems to be reasonable. Could you add the description to the class? It is hard to imagine its purpose.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/221#issuecomment-62348743

          +1

          It's awesome work. Ship it. Please add some description for EmptyTuple in the EmptyTuple class before committing.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/221#issuecomment-62348743 +1 It's awesome work. Ship it. Please add some description for EmptyTuple in the EmptyTuple class before committing.
          Hide
          jhkim Jinho Kim added a comment -

          Committed it. Thank you for the review!!

          Show
          jhkim Jinho Kim added a comment - Committed it. Thank you for the review!!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #440 (See https://builds.apache.org/job/Tajo-master-build/440/)
          TAJO-1149: Implement direct read of DelimitedTextFile. (jinho) (jhkim: rev 3e305b15fa94b1ed76dfa0e769825137ba1dfc1a)

          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForExternalTable.result
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestSplitProcessor.java
          • tajo-core/src/test/java/org/apache/tajo/client/TestTajoClient.java
          • tajo-storage/src/test/resources/storage-default.xml
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinBroadcast.java
          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForBaseTable.result
          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName2.result
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedLineReader.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
          • tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java
          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName1.result
          • tajo-storage/pom.xml
          • tajo-storage/src/main/java/org/apache/tajo/storage/FieldSerializerDeserializer.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/LineSplitProcessor.java
          • tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/ByteBufLineReader.java
          • tajo-storage/src/main/resources/storage-default.xml
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestLineReader.java
          • tajo-project/pom.xml
          • tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/ByteBufInputChannel.java
          • tajo-core/src/test/resources/results/TestTajoCli/testDescTable.result
          • tajo-docs/src/main/sphinx/table_management/csv.rst
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
          • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/BufferPool.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/SerializerDeserializer.java
          • tajo-docs/src/main/sphinx/getting_started/first_query.rst
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestNullValues.java
          • tajo-core/src/main/java/org/apache/tajo/benchmark/TPCH.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/TextSerializerDeserializer.java
          • tajo-core/src/test/java/org/apache/tajo/TpchTestBase.java
          • CHANGES
          • tajo-core/src/test/resources/results/TestTajoDump/testDump1.result
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/BinarySerializerDeserializer.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java
          • tajo-docs/src/main/sphinx/sql_language/ddl.rst
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/FieldSplitProcessor.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalPlanUtil.java
          • tajo-docs/src/main/sphinx/tsql/variables.rst
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
          • tajo-common/src/main/java/org/apache/tajo/storage/StorageConstants.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java
          • tajo-core/src/test/java/org/apache/tajo/cli/tools/TestDDLBuilder.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #440 (See https://builds.apache.org/job/Tajo-master-build/440/ ) TAJO-1149 : Implement direct read of DelimitedTextFile. (jinho) (jhkim: rev 3e305b15fa94b1ed76dfa0e769825137ba1dfc1a) tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForExternalTable.result tajo-storage/src/test/java/org/apache/tajo/storage/TestSplitProcessor.java tajo-core/src/test/java/org/apache/tajo/client/TestTajoClient.java tajo-storage/src/test/resources/storage-default.xml tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinBroadcast.java tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForBaseTable.result tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName2.result tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedLineReader.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName1.result tajo-storage/pom.xml tajo-storage/src/main/java/org/apache/tajo/storage/FieldSerializerDeserializer.java tajo-storage/src/main/java/org/apache/tajo/storage/text/LineSplitProcessor.java tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java tajo-storage/src/main/java/org/apache/tajo/storage/text/ByteBufLineReader.java tajo-storage/src/main/resources/storage-default.xml tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java tajo-storage/src/test/java/org/apache/tajo/storage/TestLineReader.java tajo-project/pom.xml tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java tajo-storage/src/main/java/org/apache/tajo/storage/ByteBufInputChannel.java tajo-core/src/test/resources/results/TestTajoCli/testDescTable.result tajo-docs/src/main/sphinx/table_management/csv.rst tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java tajo-storage/src/main/java/org/apache/tajo/storage/BufferPool.java tajo-storage/src/main/java/org/apache/tajo/storage/SerializerDeserializer.java tajo-docs/src/main/sphinx/getting_started/first_query.rst tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestNullValues.java tajo-core/src/main/java/org/apache/tajo/benchmark/TPCH.java tajo-storage/src/main/java/org/apache/tajo/storage/TextSerializerDeserializer.java tajo-core/src/test/java/org/apache/tajo/TpchTestBase.java CHANGES tajo-core/src/test/resources/results/TestTajoDump/testDump1.result tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-storage/src/main/java/org/apache/tajo/storage/BinarySerializerDeserializer.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java tajo-docs/src/main/sphinx/sql_language/ddl.rst tajo-storage/src/main/java/org/apache/tajo/storage/text/FieldSplitProcessor.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalPlanUtil.java tajo-docs/src/main/sphinx/tsql/variables.rst tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-common/src/main/java/org/apache/tajo/storage/StorageConstants.java tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java tajo-core/src/test/java/org/apache/tajo/cli/tools/TestDDLBuilder.java tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #82 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/82/)
          TAJO-1149: Implement direct read of DelimitedTextFile. (jinho) (jhkim: rev 3e305b15fa94b1ed76dfa0e769825137ba1dfc1a)

          • tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-docs/src/main/sphinx/tsql/variables.rst
          • tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForBaseTable.result
          • tajo-project/pom.xml
          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForExternalTable.result
          • tajo-docs/src/main/sphinx/sql_language/ddl.rst
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
          • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestNullValues.java
          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName2.result
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-common/src/main/java/org/apache/tajo/storage/StorageConstants.java
          • tajo-docs/src/main/sphinx/getting_started/first_query.rst
          • tajo-core/src/test/java/org/apache/tajo/cli/tools/TestDDLBuilder.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedLineReader.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/FieldSerializerDeserializer.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/BinarySerializerDeserializer.java
          • tajo-core/src/test/java/org/apache/tajo/TpchTestBase.java
          • tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName1.result
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java
          • tajo-storage/pom.xml
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalPlanUtil.java
          • tajo-core/src/test/resources/results/TestTajoDump/testDump1.result
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/LineSplitProcessor.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/SerializerDeserializer.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestSplitProcessor.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/ByteBufLineReader.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/BufferPool.java
          • tajo-core/src/test/resources/results/TestTajoCli/testDescTable.result
          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/benchmark/TPCH.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/text/FieldSplitProcessor.java
          • tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java
          • tajo-storage/src/test/resources/storage-default.xml
          • tajo-storage/src/main/java/org/apache/tajo/storage/TextSerializerDeserializer.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
          • tajo-storage/src/main/resources/storage-default.xml
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/ByteBufInputChannel.java
          • tajo-storage/src/test/java/org/apache/tajo/storage/TestLineReader.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
          • tajo-core/src/test/java/org/apache/tajo/client/TestTajoClient.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinBroadcast.java
          • tajo-docs/src/main/sphinx/table_management/csv.rst
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #82 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/82/ ) TAJO-1149 : Implement direct read of DelimitedTextFile. (jinho) (jhkim: rev 3e305b15fa94b1ed76dfa0e769825137ba1dfc1a) tajo-common/src/main/java/org/apache/tajo/storage/EmptyTuple.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-docs/src/main/sphinx/tsql/variables.rst tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForBaseTable.result tajo-project/pom.xml tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLForExternalTable.result tajo-docs/src/main/sphinx/sql_language/ddl.rst tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestNullValues.java tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName2.result tajo-storage/src/test/java/org/apache/tajo/storage/TestStorages.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-common/src/main/java/org/apache/tajo/storage/StorageConstants.java tajo-docs/src/main/sphinx/getting_started/first_query.rst tajo-core/src/test/java/org/apache/tajo/cli/tools/TestDDLBuilder.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestCTASQuery.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogUtil.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedLineReader.java tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java tajo-storage/src/main/java/org/apache/tajo/storage/FieldSerializerDeserializer.java tajo-storage/src/main/java/org/apache/tajo/storage/BinarySerializerDeserializer.java tajo-core/src/test/java/org/apache/tajo/TpchTestBase.java tajo-core/src/test/resources/results/testDDLBuilder/testBuildDDLQuotedTableName1.result tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestTablePartitions.java tajo-storage/pom.xml tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalPlanUtil.java tajo-core/src/test/resources/results/TestTajoDump/testDump1.result tajo-storage/src/main/java/org/apache/tajo/storage/text/LineSplitProcessor.java tajo-storage/src/main/java/org/apache/tajo/storage/SerializerDeserializer.java tajo-storage/src/test/java/org/apache/tajo/storage/TestCompressionStorages.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-storage/src/test/java/org/apache/tajo/storage/TestSplitProcessor.java tajo-storage/src/main/java/org/apache/tajo/storage/text/ByteBufLineReader.java tajo-storage/src/main/java/org/apache/tajo/storage/text/TextFieldSerializerDeserializer.java tajo-storage/src/main/java/org/apache/tajo/storage/text/DelimitedTextFile.java tajo-storage/src/main/java/org/apache/tajo/storage/BufferPool.java tajo-core/src/test/resources/results/TestTajoCli/testDescTable.result CHANGES tajo-core/src/main/java/org/apache/tajo/benchmark/TPCH.java tajo-storage/src/main/java/org/apache/tajo/storage/text/FieldSplitProcessor.java tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java tajo-storage/src/test/resources/storage-default.xml tajo-storage/src/main/java/org/apache/tajo/storage/TextSerializerDeserializer.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-storage/src/main/resources/storage-default.xml tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java tajo-storage/src/main/java/org/apache/tajo/storage/ByteBufInputChannel.java tajo-storage/src/test/java/org/apache/tajo/storage/TestLineReader.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java tajo-core/src/test/java/org/apache/tajo/client/TestTajoClient.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinBroadcast.java tajo-docs/src/main/sphinx/table_management/csv.rst tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/221

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/221

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              jhkim Jinho Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development