Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5658

[JAVA] Provide ability to resync VectorSchemaRoot if types change

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Component/s: FlightRPC, Java
    • Environment:
      java8 arrow-java 0.13.0

      Description

      I can't transfer using apache arrow-flihgt. Contains listvector data. The problem description is as follows:

      1. I parse an xml file and convert it to an arrow format and finally convert it to a parquet data format. The address of the .xml file data is url http://www.w3school.com.cn/example/xmle/cd_catalog.xml
      2. I created a schema that uses listvector.
        code show as below:
        List<FiledVector> list = childrenBuilder.add(ListVector.empty(column.getId().toString(),allocator));
        VectorSchemaRoot root = VectorSchemaRoot.of(inVector)
      3. Parse the xml file to get the list data in "cd". Use api use listvector.
        `ListVector listVector = (ListVector) valueVectors;
                        List<Column> columns = column.getColumns();
                        Column column1 = columns.get(0);
                        String name = column1.getId().toString();
                        UnionListWriter writer = listVector.getWriter();
                        Writer.allocate();
                        For (int j = 0; j < column1.getColumns().size();j++) {

                          writer.setPosition(j);
                          writer.startList();
                          Writer.list().startList();
                          Column column2 = column1.getColumns().get(j);
                          List<Map<String, String>> lst = (List<Map<String, String>>) ((Map) val).get(name);

                          For (int k = 0; k < lst.size(); k++) {
                              Map<String, String> stringStringMap = lst.get(k);
                              String value = stringStringMap.get(column2.getId().toString());
                              Switch (column2.getType())

      Unknown macro: {                             Case FLOAT}

                          }
                          Writer.list().endList();
                          writer.endList();
                      }`

       4. 

      After the write is complete, I will send to the arrow-flight server. server code :

      @Override
      public Callable<Flight.PutResult> acceptPut(FlightStream flightStream) {

      return () -> {

      try (VectorSchemaRoot root = flightStream.getRoot()) {

      while (flightStream.next()) {
      VectorSchemaRoot other = null;
      try

      Unknown macro: { logger.info(" Receive message ...... size}

      catch (Exception e)

      Unknown macro: { logger.error(e.getMessage(), e); }

      }
      }

      return Flight.PutResult.parseFrom("ok".getBytes());
      };

      }

       But the server did not receive any information.!! it is error .

      client code :

      root = message.getRoot();
      //client.close();
      FlightClient.ClientStreamListener listener =
      client.startPut(FlightDescriptor.path(message.getFilename()), root);
      listener.putNext();
      listener.completed();
      client.close();
      listener.putNext();
      listener.completed();
      Flight.PutResult result =
      listener.getResult();
      String s = new String(result.toByteArray());
      System.out.println(s);

        Attachments

        1. ClientStart.java
          4 kB
          luckily
        2. ServerStart.java
          2 kB
          luckily
        3. pom.xml
          3 kB
          luckily
        4. image-2019-06-25-17-58-09-038.png
          4 kB
          Liya Fan
        5. image-2019-06-25-17-59-07-352.png
          21 kB
          Liya Fan
        6. ClientStart.java
          4 kB
          Liya Fan
        7. image-2019-06-26-16-07-23-366.png
          75 kB
          luckily
        8. image-2019-06-26-16-09-30-275.png
          130 kB
          luckily
        9. image-2019-06-27-09-38-52-735.png
          76 kB
          luckily
        10. image-2019-06-27-09-39-18-597.png
          97 kB
          luckily
        11. ClientStart.java
          5 kB
          luckily
        12. ServerStart.java
          3 kB
          luckily
        13. ServerStart.java
          3 kB
          luckily
        14. ServerStart.java
          2 kB
          Liya Fan

          Issue Links

            Activity

              People

              • Assignee:
                fan_li_ya Liya Fan
                Reporter:
                liaotian1005 luckily
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 5.5h
                  5.5h