Thrift
  1. Thrift
  2. THRIFT-847

Test Framework harmonization across all languages

    Details

      Description

      Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.

      • C++ Test and JavaScript Test Server written in Java have different responses for the same services
      • C# and Java Test Server have different responses for testException as C++

      I propose the following steps:

      • identify the language with the reference implementation (well defined return values for all test cases)
      • update the ThriftTest.thrift with details about the required return values that have to be implemented
      • update test implementations and move language tests into their appropriate library directory (THRIFT-35)
      • a public test server that supports multiple protocols and transports could be another enhancement for testing purposes

      I'm ready to help preparing patches, just tell me what you need!

      1. THRIFT-847_add__make_cross__build_target.patch
        0.2 kB
        Roger Meier
      2. test.sh
        4 kB
        Roger Meier
      3. build.xml
        3 kB
        Roger Meier
      4. v1-WORK_IN_PROGRESS-unified_tests.tar.gz
        6 kB
        Christian Lavoie

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Thrift #1121 (See https://builds.apache.org/job/Thrift/1121/)
          THRIFT-847: Test Framework harmonization across all languages (roger: rev 5829a2c64a47bb122a4c7e6ddf93acd6b41dfd7d)

          • test/test.sh
          Show
          Hudson added a comment - SUCCESS: Integrated in Thrift #1121 (See https://builds.apache.org/job/Thrift/1121/ ) THRIFT-847 : Test Framework harmonization across all languages (roger: rev 5829a2c64a47bb122a4c7e6ddf93acd6b41dfd7d) test/test.sh
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Thrift #1065 (See https://builds.apache.org/job/Thrift/1065/)
          THRIFT-847 Test Framework harmonization across all languages (roger: rev f85fdd7a3af3255ea2464118930c10f582caf035)

          • test/README.md
          Show
          Hudson added a comment - SUCCESS: Integrated in Thrift #1065 (See https://builds.apache.org/job/Thrift/1065/ ) THRIFT-847 Test Framework harmonization across all languages (roger: rev f85fdd7a3af3255ea2464118930c10f582caf035) test/README.md
          Hide
          Roger Meier added a comment -

          The goal of this GSoC task is:
          • Cover any possible language, protocol, transport, type, etc. vice versa(Just a few are integrated right now)
          • Port cross language test suite to non linux systems
          • Improve test and functionality reporting towards features list for the web site
          • Fix documentation
          • Quality in general
          • Help to make it perfect… I’m sure there is enough to do

          as soon as we have candidates we can setup a new ticket or sub tasks for this.

          Show
          Roger Meier added a comment - The goal of this GSoC task is: • Cover any possible language, protocol, transport, type, etc. vice versa(Just a few are integrated right now) • Port cross language test suite to non linux systems • Improve test and functionality reporting towards features list for the web site • Fix documentation • Quality in general • Help to make it perfect… I’m sure there is enough to do as soon as we have candidates we can setup a new ticket or sub tasks for this.
          Hide
          Roger Meier added a comment -

          README.md committed.

          Show
          Roger Meier added a comment - README.md committed.
          Hide
          Jake Farrell added a comment -

          Roger Meier objections to closing this specific issue since we have the Test Suite component and this will be something that we will always be working on and subtasks/depends will always be part of Test Suite

          Show
          Jake Farrell added a comment - Roger Meier objections to closing this specific issue since we have the Test Suite component and this will be something that we will always be working on and subtasks/depends will always be part of Test Suite
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Thrift #1056 (See https://builds.apache.org/job/Thrift/1056/)
          THRIFT-847 Test Framework harmonization across all languages (roger: rev 7e10329f7982f2602d6dbdcb2b45e843b85170c4)

          • Makefile.am
          Show
          Hudson added a comment - SUCCESS: Integrated in Thrift #1056 (See https://builds.apache.org/job/Thrift/1056/ ) THRIFT-847 Test Framework harmonization across all languages (roger: rev 7e10329f7982f2602d6dbdcb2b45e843b85170c4) Makefile.am
          Hide
          Roger Meier added a comment -

          committed make cross
          ;-r

          Show
          Roger Meier added a comment - committed make cross ;-r
          Hide
          Roger Meier added a comment -

          THRIFT-847_add_make_cross_build_target.patch adds a *make cross* target to run the cross language test suite

          Show
          Roger Meier added a comment - THRIFT-847_add_ make_cross _build_target.patch adds a * make cross * target to run the cross language test suite
          Hide
          Hudson added a comment -

          Integrated in Thrift #615 (See https://builds.apache.org/job/Thrift/615/)
          THRIFT-847 Test Framework harmonization across all languages (Revision f42ce2a8f49cf09e695974e6cd3c434b8dda61ab)

          Result = FAILURE
          roger : https://git-wip-us.apache.org/repos/asf?p=thrift.git&a=commit&h=f42ce2a8f49cf09e695974e6cd3c434b8dda61ab
          Files :

          • lib/java/build.xml
          • test/test.sh
          • lib/java/test/org/apache/thrift/test/TestServer.java
          • lib/java/test/org/apache/thrift/test/TestClient.java
          Show
          Hudson added a comment - Integrated in Thrift #615 (See https://builds.apache.org/job/Thrift/615/ ) THRIFT-847 Test Framework harmonization across all languages (Revision f42ce2a8f49cf09e695974e6cd3c434b8dda61ab) Result = FAILURE roger : https://git-wip-us.apache.org/repos/asf?p=thrift.git&a=commit&h=f42ce2a8f49cf09e695974e6cd3c434b8dda61ab Files : lib/java/build.xml test/test.sh lib/java/test/org/apache/thrift/test/TestServer.java lib/java/test/org/apache/thrift/test/TestClient.java
          Hide
          Hudson added a comment -

          Integrated in Thrift #580 (See https://builds.apache.org/job/Thrift/580/)
          THRIFT-847 Test Framework harmonization across all languages
          add NODE_PATH
          remove Unix Domain Socket used by tests (Revision 1404889)
          THRIFT-1745 Python JSON protocol
          TJSONProtocol.py: Frederic Delbos

          THRIFT-847 Test Framework harmonization across all languages
          Integration into py lib and test suite (Revision 1404838)

          Result = FAILURE
          roger : http://svn.apache.org/viewvc/?view=rev&rev=1404889
          Files :

          • /thrift/trunk/test/nodejs/Makefile.am
          • /thrift/trunk/test/test.sh

          roger : http://svn.apache.org/viewvc/?view=rev&rev=1404838
          Files :

          • /thrift/trunk/lib/py/src/protocol/TJSONProtocol.py
          • /thrift/trunk/lib/py/src/protocol/_init_.py
          • /thrift/trunk/test/py/RunClientServer.py
          • /thrift/trunk/test/py/TestClient.py
          • /thrift/trunk/test/py/TestServer.py
          • /thrift/trunk/test/test.sh
          Show
          Hudson added a comment - Integrated in Thrift #580 (See https://builds.apache.org/job/Thrift/580/ ) THRIFT-847 Test Framework harmonization across all languages add NODE_PATH remove Unix Domain Socket used by tests (Revision 1404889) THRIFT-1745 Python JSON protocol TJSONProtocol.py: Frederic Delbos THRIFT-847 Test Framework harmonization across all languages Integration into py lib and test suite (Revision 1404838) Result = FAILURE roger : http://svn.apache.org/viewvc/?view=rev&rev=1404889 Files : /thrift/trunk/test/nodejs/Makefile.am /thrift/trunk/test/test.sh roger : http://svn.apache.org/viewvc/?view=rev&rev=1404838 Files : /thrift/trunk/lib/py/src/protocol/TJSONProtocol.py /thrift/trunk/lib/py/src/protocol/_ init _.py /thrift/trunk/test/py/RunClientServer.py /thrift/trunk/test/py/TestClient.py /thrift/trunk/test/py/TestServer.py /thrift/trunk/test/test.sh
          Hide
          Hudson added a comment -

          Integrated in Thrift #393 (See https://builds.apache.org/job/Thrift/393/)
          THRIFT-847 Test Framework harmonization across all languages
          add php TestClient to the testsuite

          roger : http://svn.apache.org/viewvc/?view=rev&rev=1234292
          Files :

          • /thrift/trunk/.gitignore
          • /thrift/trunk/configure.ac
          • /thrift/trunk/test/Makefile.am
          • /thrift/trunk/test/php/Makefile
          • /thrift/trunk/test/php/Makefile.am
          • /thrift/trunk/test/test.sh
          Show
          Hudson added a comment - Integrated in Thrift #393 (See https://builds.apache.org/job/Thrift/393/ ) THRIFT-847 Test Framework harmonization across all languages add php TestClient to the testsuite roger : http://svn.apache.org/viewvc/?view=rev&rev=1234292 Files : /thrift/trunk/.gitignore /thrift/trunk/configure.ac /thrift/trunk/test/Makefile.am /thrift/trunk/test/php/Makefile /thrift/trunk/test/php/Makefile.am /thrift/trunk/test/test.sh
          Hide
          Roger Meier added a comment -

          see:
          https://builds.apache.org/view/S-Z/view/Thrift/job/Thrift/lastSuccessfulBuild/artifact/thrift/test/test.log

          yes, we have some issues... but I'm sure we will fix them! patches are welcome!
          -r

          Show
          Roger Meier added a comment - see: https://builds.apache.org/view/S-Z/view/Thrift/job/Thrift/lastSuccessfulBuild/artifact/thrift/test/test.log yes, we have some issues... but I'm sure we will fix them! patches are welcome! -r
          Hide
          Hudson added a comment -

          Integrated in Thrift #384 (See https://builds.apache.org/job/Thrift/384/)
          THRIFT-847 Test Framework harmonization across all languages
          perl fixes:

          • include test/perl if perl is enabled

          roger : http://svn.apache.org/viewvc/?view=rev&rev=1229764
          Files :

          • /thrift/trunk/test/Makefile.am
          Show
          Hudson added a comment - Integrated in Thrift #384 (See https://builds.apache.org/job/Thrift/384/ ) THRIFT-847 Test Framework harmonization across all languages perl fixes: include test/perl if perl is enabled roger : http://svn.apache.org/viewvc/?view=rev&rev=1229764 Files : /thrift/trunk/test/Makefile.am
          Hide
          Hudson added a comment -

          Integrated in Thrift #383 (See https://builds.apache.org/job/Thrift/383/)
          THRIFT-847 Test Framework harmonization across all languages
          perl fixes:

          • generate Makfile for perl tests
          • add inc path to test

          roger : http://svn.apache.org/viewvc/?view=rev&rev=1229745
          Files :

          • /thrift/trunk/.gitignore
          • /thrift/trunk/configure.ac
          • /thrift/trunk/test/perl/Makefile
          • /thrift/trunk/test/perl/Makefile.am
          • /thrift/trunk/test/test.sh
          Show
          Hudson added a comment - Integrated in Thrift #383 (See https://builds.apache.org/job/Thrift/383/ ) THRIFT-847 Test Framework harmonization across all languages perl fixes: generate Makfile for perl tests add inc path to test roger : http://svn.apache.org/viewvc/?view=rev&rev=1229745 Files : /thrift/trunk/.gitignore /thrift/trunk/configure.ac /thrift/trunk/test/perl/Makefile /thrift/trunk/test/perl/Makefile.am /thrift/trunk/test/test.sh
          Hide
          Roger Meier added a comment -

          committed test/test.sh and added to the CI job
          see https://builds.apache.org/view/S-Z/view/Thrift/job/Thrift/lastSuccessfulBuild/artifact/thrift/test/test.log

          however, timeout which is usually a part of coreutils is not installed on the build servers
          I will request that now.

          Show
          Roger Meier added a comment - committed test/test.sh and added to the CI job see https://builds.apache.org/view/S-Z/view/Thrift/job/Thrift/lastSuccessfulBuild/artifact/thrift/test/test.log however, timeout which is usually a part of coreutils is not installed on the build servers I will request that now.
          Hide
          Hudson added a comment -

          Integrated in Thrift #378 (See https://builds.apache.org/job/Thrift/378/)
          THRIFT-847 Test Framework harmonization across all languages
          add test.sh as a first approach

          roger : http://svn.apache.org/viewvc/?view=rev&rev=1229359
          Files :

          • /thrift/trunk/.gitignore
          • /thrift/trunk/test/test.sh
          Show
          Hudson added a comment - Integrated in Thrift #378 (See https://builds.apache.org/job/Thrift/378/ ) THRIFT-847 Test Framework harmonization across all languages add test.sh as a first approach roger : http://svn.apache.org/viewvc/?view=rev&rev=1229359 Files : /thrift/trunk/.gitignore /thrift/trunk/test/test.sh
          Hide
          Henrique Mendonça added a comment -

          Awesome Roger, thanks a lot!
          As you said, perhaps we could create a separate make target for that, or use 'check', so it would handle to dependencies and read the configuration (e.g. test all, and only, configured languages...)
          but that's a lot more work, and this is already working perfectly
          Cheers!

          Show
          Henrique Mendonça added a comment - Awesome Roger, thanks a lot! As you said, perhaps we could create a separate make target for that, or use 'check', so it would handle to dependencies and read the configuration (e.g. test all, and only, configured languages...) but that's a lot more work, and this is already working perfectly Cheers!
          Hide
          Roger Meier added a comment -

          I did some rework and have a first version for a test/test.sh which can be executed independently on the ci server.

          further work is adding much more tests and integrate test suites into autoconf build procedure

          What do you think about that?

          Show
          Roger Meier added a comment - I did some rework and have a first version for a test/ test.sh which can be executed independently on the ci server. further work is adding much more tests and integrate test suites into autoconf build procedure What do you think about that?
          Hide
          Roger Meier added a comment -

          What do you think abou something like that build.xml?
          using the regular test server and clients and do interaction tests across languages?
          this build.xml just does c++ tests, but can be extended if we harmonize the parameters for the test suites across languages.

          Show
          Roger Meier added a comment - What do you think abou something like that build.xml ? using the regular test server and clients and do interaction tests across languages? this build.xml just does c++ tests, but can be extended if we harmonize the parameters for the test suites across languages.
          Hide
          Roger Meier added a comment -

          THRIFT-960 provides Java TestServer and Client

          Show
          Roger Meier added a comment - THRIFT-960 provides Java TestServer and Client
          Hide
          Roger Meier added a comment -

          Thanks Christian for your inputs.

          I had the following things in mind:

          1. identify the language with the reference implementation (well defined return values for all test cases) based on ThriftTest.thrift, probably Java (lib/java/test/org/apache/thrift/server/ServerTestBase.java) or cpp (test/cpp/src/TestClient.cpp test/cpp/src/TestServer.cpp)
          2. update the ThriftTest.thrift with details about the required return values that have to be implemented
          3. update test implementations
          4. add some wrapper scripts to automate the cross language tests (as Christian did)
          5. move language tests into their appropriate library directory (THRIFT-35)
          6. a public test server that supports multiple protocols and transports could be another enhancement for testing purposes

          Today neary every language has already Test Server and Test Clients based on ThriftTest.thrift , we just have to modify them a bit so that they have exactly the same behavior. I think it is not worth to add another SimpleTest.thrift, the target should be extending ThriftTest.thrift.

          Show
          Roger Meier added a comment - Thanks Christian for your inputs. I had the following things in mind: identify the language with the reference implementation (well defined return values for all test cases) based on ThriftTest.thrift, probably Java (lib/java/test/org/apache/thrift/server/ServerTestBase.java) or cpp (test/cpp/src/TestClient.cpp test/cpp/src/TestServer.cpp) update the ThriftTest.thrift with details about the required return values that have to be implemented update test implementations add some wrapper scripts to automate the cross language tests (as Christian did) move language tests into their appropriate library directory ( THRIFT-35 ) a public test server that supports multiple protocols and transports could be another enhancement for testing purposes Today neary every language has already Test Server and Test Clients based on ThriftTest.thrift , we just have to modify them a bit so that they have exactly the same behavior. I think it is not worth to add another SimpleTest.thrift, the target should be extending ThriftTest.thrift.
          Hide
          Christian Lavoie added a comment -

          So, here's a very rough draft of unfinished cross language tests. To run, cd into a working thrift tree, extract the tar, it'll create a crosstest/ subdir with a few shell scripts:

          $ ls
          SimpleTest.cpp		SimpleTest.thrift	java-build.sh
          SimpleTest.hs		cpp-build.sh		java-run.sh
          SimpleTest.java		hs-build.sh
          

          To build the tests:

          $ ./java-build.sh && ./cpp-build.sh && ./hs-build.sh 
          Note: Some input files use unchecked or unsafe operations.
          Note: Recompile with -Xlint:unchecked for details.
          [ 1 of 14] Compiling Thrift.Transport ( ../lib/hs/src/Thrift/Transport.hs, build-hs/Thrift/Transport.o )
          [ 2 of 14] Compiling Thrift.Transport.Handle ( ../lib/hs/src/Thrift/Transport/Handle.hs, build-hs/Thrift/Transport/Handle.o )
          [ 3 of 14] Compiling Thrift.Protocol  ( ../lib/hs/src/Thrift/Protocol.hs, build-hs/Thrift/Protocol.o )
          [ 4 of 14] Compiling Thrift.Protocol.Binary ( ../lib/hs/src/Thrift/Protocol/Binary.hs, build-hs/Thrift/Protocol/Binary.o )
          [ 5 of 14] Compiling Thrift           ( ../lib/hs/src/Thrift.hs, build-hs/Thrift.o )
          [ 6 of 14] Compiling Thrift.Server    ( ../lib/hs/src/Thrift/Server.hs, build-hs/Thrift/Server.o )
          [ 7 of 14] Compiling SimpleTest_Types ( gen-hs/SimpleTest_Types.hs, build-hs/SimpleTest_Types.o )
          [ 8 of 14] Compiling Base_Iface       ( gen-hs/Base_Iface.hs, build-hs/Base_Iface.o )
          [ 9 of 14] Compiling Simple_Iface     ( gen-hs/Simple_Iface.hs, build-hs/Simple_Iface.o )
          [10 of 14] Compiling Base             ( gen-hs/Base.hs, build-hs/Base.o )
          [11 of 14] Compiling Simple           ( gen-hs/Simple.hs, build-hs/Simple.o )
          [12 of 14] Compiling Base_Client      ( gen-hs/Base_Client.hs, build-hs/Base_Client.o )
          [13 of 14] Compiling Simple_Client    ( gen-hs/Simple_Client.hs, build-hs/Simple_Client.o )
          [14 of 14] Compiling Main             ( SimpleTest.hs, build-hs/Main.o )
          Linking hs-test ...
          

          And to run the tests in both mode (each language is run against itself, but through a TCP socket):

          $ ./cpp-simple-test both && ./java-run.sh both && ./hs-test both
          Running in both client-and-server mode
          Running byte tests
          arg: both
          Starting server on port 9090 ...
          Running byte numeric tests
          Running i16 numeric tests
          Running i32 numeric tests
          Running i64 numeric tests
          Running double numeric tests
          SUCCESS
          Runing in both client and server mode
          Testing Simple.addBytes...
          

          You can also run each pair of executables with one in server and one in client mode:

          $ ./java-run.sh server & sleep 5 && ./cpp-simple-test client && ./hs-test client && sleep 1 && kill $(jobs -p)
          [1] 12018
          arg: server
          Starting server on port 9090 ...
          Running in client-only mode
          Running byte tests
          Runing in client-only mode
          Testing Simple.addBytes...
          

          On my Mac machine, I can get all pairs (and all three tests in both modes) to run with 0.5.0, but NOT with HEAD; the cpp client seems to be broken since dreiss' facebook patches; just guessing here, but the error message is telling:

          $ ./hs-test server & sleep 3 && ./cpp-simple-test client
          [1] 22775
          Runing in server-only mode
          Running in client-only mode
          Running byte tests
          terminate called after throwing an instance of 'apache::thrift::transport::TTransportException'
            what():  Base TTransport cannot write.
          Abort trap
          

          So, here's my plan, and my RFC:

          1. Finish the simplest possible tests, and implement a handful of clients (I suspect I'll get perl, haskell, c++ and java done) and a handful of servers (haskell, c++ and java; most likely)
          2. Get this hooked to the autotools build system and make check
          3. Define a set of base tests: you aren't a proper thrift bindings unless you pass those
          4. Define different set of advanced tests: you match specific features exactly as the reference languages do (== java, cpp, I think). In particular, I introduced a bug where Haskell consider byte to be unsigned, and java/cpp don't. Under this definition, Haskell would fail the advanced tests that check for numerical type implementation details
          5. Create some sort of score card for each language – to see how closely they match the reference implementation behaviour

          Please comment on the approach, I'd like some feedback before I spend too much time implementing tests nobody will like.

          Show
          Christian Lavoie added a comment - So, here's a very rough draft of unfinished cross language tests. To run, cd into a working thrift tree, extract the tar, it'll create a crosstest/ subdir with a few shell scripts: $ ls SimpleTest.cpp SimpleTest.thrift java-build.sh SimpleTest.hs cpp-build.sh java-run.sh SimpleTest.java hs-build.sh To build the tests: $ ./java-build.sh && ./cpp-build.sh && ./hs-build.sh Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. [ 1 of 14] Compiling Thrift.Transport ( ../lib/hs/src/Thrift/Transport.hs, build-hs/Thrift/Transport.o ) [ 2 of 14] Compiling Thrift.Transport.Handle ( ../lib/hs/src/Thrift/Transport/Handle.hs, build-hs/Thrift/Transport/Handle.o ) [ 3 of 14] Compiling Thrift.Protocol ( ../lib/hs/src/Thrift/Protocol.hs, build-hs/Thrift/Protocol.o ) [ 4 of 14] Compiling Thrift.Protocol.Binary ( ../lib/hs/src/Thrift/Protocol/Binary.hs, build-hs/Thrift/Protocol/Binary.o ) [ 5 of 14] Compiling Thrift ( ../lib/hs/src/Thrift.hs, build-hs/Thrift.o ) [ 6 of 14] Compiling Thrift.Server ( ../lib/hs/src/Thrift/Server.hs, build-hs/Thrift/Server.o ) [ 7 of 14] Compiling SimpleTest_Types ( gen-hs/SimpleTest_Types.hs, build-hs/SimpleTest_Types.o ) [ 8 of 14] Compiling Base_Iface ( gen-hs/Base_Iface.hs, build-hs/Base_Iface.o ) [ 9 of 14] Compiling Simple_Iface ( gen-hs/Simple_Iface.hs, build-hs/Simple_Iface.o ) [10 of 14] Compiling Base ( gen-hs/Base.hs, build-hs/Base.o ) [11 of 14] Compiling Simple ( gen-hs/Simple.hs, build-hs/Simple.o ) [12 of 14] Compiling Base_Client ( gen-hs/Base_Client.hs, build-hs/Base_Client.o ) [13 of 14] Compiling Simple_Client ( gen-hs/Simple_Client.hs, build-hs/Simple_Client.o ) [14 of 14] Compiling Main ( SimpleTest.hs, build-hs/Main.o ) Linking hs-test ... And to run the tests in both mode (each language is run against itself, but through a TCP socket): $ ./cpp-simple-test both && ./java-run.sh both && ./hs-test both Running in both client-and-server mode Running byte tests arg: both Starting server on port 9090 ... Running byte numeric tests Running i16 numeric tests Running i32 numeric tests Running i64 numeric tests Running double numeric tests SUCCESS Runing in both client and server mode Testing Simple.addBytes... You can also run each pair of executables with one in server and one in client mode: $ ./java-run.sh server & sleep 5 && ./cpp-simple-test client && ./hs-test client && sleep 1 && kill $(jobs -p) [1] 12018 arg: server Starting server on port 9090 ... Running in client-only mode Running byte tests Runing in client-only mode Testing Simple.addBytes... On my Mac machine, I can get all pairs (and all three tests in both modes) to run with 0.5.0, but NOT with HEAD; the cpp client seems to be broken since dreiss' facebook patches; just guessing here, but the error message is telling: $ ./hs-test server & sleep 3 && ./cpp-simple-test client [1] 22775 Runing in server-only mode Running in client-only mode Running byte tests terminate called after throwing an instance of 'apache::thrift::transport::TTransportException' what(): Base TTransport cannot write. Abort trap So, here's my plan, and my RFC: Finish the simplest possible tests, and implement a handful of clients (I suspect I'll get perl, haskell, c++ and java done) and a handful of servers (haskell, c++ and java; most likely) Get this hooked to the autotools build system and make check Define a set of base tests: you aren't a proper thrift bindings unless you pass those Define different set of advanced tests: you match specific features exactly as the reference languages do (== java, cpp, I think). In particular, I introduced a bug where Haskell consider byte to be unsigned, and java/cpp don't. Under this definition, Haskell would fail the advanced tests that check for numerical type implementation details Create some sort of score card for each language – to see how closely they match the reference implementation behaviour Please comment on the approach, I'd like some feedback before I spend too much time implementing tests nobody will like.
          Hide
          Bryan Duxbury added a comment -

          This is a really cool idea, but I don't think this is going to make it into TRUNK before the 0.4 branch gets cut.

          Show
          Bryan Duxbury added a comment - This is a really cool idea, but I don't think this is going to make it into TRUNK before the 0.4 branch gets cut.

            People

            • Assignee:
              Roger Meier
              Reporter:
              Roger Meier
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development