Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      As some of you might remember, I started working on D programming language support for Thrift during the 2011 Google Summer of Code. After many unexpected delays, the time has finally come where both the code is stable - that is, ready for upstream merging, I hope - and I can spare enough time to focus any further issues that should pop up.

      The library generally tries to follow the precedent set by C++ and Java in design and, in some places, implementation. What is pretty unique, though, is that most of the actual code generation is done at D compile time, the Thrift compiler module for D does little more than providing a direct translation of the IDL file(s) into D code.

      What is done:

      • Binary/Compact/JSON protocols
      • Socket, SSL, HTTP and file transports (plus your familiar helpers, i.e. buffered/framed/memory-buffer/piped/zlib...)
      • Several single- and multithreaded server variants (including a libevent-based non-blocking implementation)
      • Both synchronous and asynchronous client implementations
      • Reasonably good test suite coverage

      Wish/Todo list:

      • Unix domain sockets - the D standard library will start to properly support them in the next release
      • More real-world battle testing (most testing has been done using synthetic test cases similar to the other languages; and while I know of a few people intending to use Thrift/D in the wild, I still need to inquire about the current status of their projects)

      I have been developing the project in my own fork over at GitHub, attached is a (large) patch against current trunk. Using a current D compiler, the test suite should pass on Linux x86/x86_64, OS X x86/x86_64 and Windows x86. The necessary dependencies are detected by configure. At the GitHub project wiki, I put together an introduction and build instructions, which could e.g. be moved to the Thrift wiki. A recent build of the API documentation can currently be found at my personal website.

      Please let me know about the best way to go forward with this, and feel free to let me know of any issues with the code you might come across (the build system integration could particularly use a look, Autotools isn't exactly my strength). If the code will make its way into trunk, I will also be happy to fill out any necessary legal paperwork, I'm just not quite sure what is required by the ASF in that regard.

      1. thrift-d.patch
        719 kB
        David Nadlinger

        Activity

        Hide
        Jake Farrell added a comment -

        David, thanks for starting the work on this. The major items that will be necessary are generator, client lib, test cases (server/client), tutorial, documentation. As long as you have included the patch with asf inclusion selected and this is your own work and not under another license then you have satisfied ASF compliance. I'll take a look through this and let you know what will be necessary to get this moving into trunk

        Show
        Jake Farrell added a comment - David, thanks for starting the work on this. The major items that will be necessary are generator, client lib, test cases (server/client), tutorial, documentation. As long as you have included the patch with asf inclusion selected and this is your own work and not under another license then you have satisfied ASF compliance. I'll take a look through this and let you know what will be necessary to get this moving into trunk
        Hide
        David Nadlinger added a comment -

        Thanks for your quick response. The basic blocks you mentioned should all be in place. I submitted the patch with ASF inclusion enabled and all source files feature the ASF license header, so I guess this should be fine then.

        Show
        David Nadlinger added a comment - Thanks for your quick response. The basic blocks you mentioned should all be in place. I submitted the patch with ASF inclusion enabled and all source files feature the ASF license header, so I guess this should be fine then.
        Hide
        David Nadlinger added a comment -

        During the last few weeks, I did some more work on the few remaining items on my todo list. I updated the attached patch to reflect the latest changes (and in the process accidentally removed the old one completely, let me know if you need it or see the Git tag). A summary of what changed since the first revision (the GitHub comparison view also shows upstream changes in the same period):

        • Updated the socket code to handle IPv6 and multiple addresses on name lookup correctly.
        • Implemented experimental SSL support for the async client. This entailed some refactoring of the SSL support code (end-user visible: TSSLSocketFactory -> TSSLContext, TSSLSocket now constructed directly).
        • A long standing bug in the D compiler which sometimes made symbols visible when they should not be has been fixed in the development tree, which necessitated some minor changes to the code. The updated version works with both the current (2.057) as well as the upcoming DMD release.
        • Test suite: Dummy OpenSSL certificates are automatically generated and most stand-alone test cases run as part of make check.

        I took the opportunity to make some last-minute breaking API changes as long as relatively few people are using the code, and I don't foresee this to be necessary again soon. Compared to the first patch, no changes outside lib/d/ were made.

        Show
        David Nadlinger added a comment - During the last few weeks, I did some more work on the few remaining items on my todo list. I updated the attached patch to reflect the latest changes (and in the process accidentally removed the old one completely, let me know if you need it or see the Git tag ). A summary of what changed since the first revision (the GitHub comparison view also shows upstream changes in the same period): Updated the socket code to handle IPv6 and multiple addresses on name lookup correctly. Implemented experimental SSL support for the async client. This entailed some refactoring of the SSL support code (end-user visible: TSSLSocketFactory -> TSSLContext , TSSLSocket now constructed directly). A long standing bug in the D compiler which sometimes made symbols visible when they should not be has been fixed in the development tree, which necessitated some minor changes to the code. The updated version works with both the current (2.057) as well as the upcoming DMD release. Test suite: Dummy OpenSSL certificates are automatically generated and most stand-alone test cases run as part of make check . I took the opportunity to make some last-minute breaking API changes as long as relatively few people are using the code, and I don't foresee this to be necessary again soon. Compared to the first patch, no changes outside lib/d/ were made.
        Hide
        Jake Farrell added a comment -

        David, this is a great patch, thank you for including the test cases and tutorials along with this. Anyone familiar with D want to comment on this

        Show
        Jake Farrell added a comment - David, this is a great patch, thank you for including the test cases and tutorials along with this. Anyone familiar with D want to comment on this
        Hide
        David Nadlinger added a comment -

        Rebased patch against latest trunk and applied a few fixes (a test was missing clean target, future-proofed code for a small change in struct literal semantics in future D releases).

        Show
        David Nadlinger added a comment - Rebased patch against latest trunk and applied a few fixes (a test was missing clean target, future-proofed code for a small change in struct literal semantics in future D releases).
        Hide
        Dave Watson added a comment -

        ping

        We also built/used this for some things internally at facebook. I saw no major issues when I looked it over. We don't have any major services that use it actively though.

        Would be nice to get this in

        Show
        Dave Watson added a comment - ping We also built/used this for some things internally at facebook. I saw no major issues when I looked it over. We don't have any major services that use it actively though. Would be nice to get this in
        Hide
        Jake Farrell added a comment -

        Sorry this took me awhile to review.

        David thank you for taking the time to put together D support, there was a lot of work that went behind this and it showed. This patch is a great example of everything necessary when adding a new library.

        Committed

        Show
        Jake Farrell added a comment - Sorry this took me awhile to review. David thank you for taking the time to put together D support, there was a lot of work that went behind this and it showed. This patch is a great example of everything necessary when adding a new library. Committed
        Hide
        Hudson added a comment -

        Integrated in Thrift #429 (See https://builds.apache.org/job/Thrift/429/)
        THRIFT-1500: d programming language support
        Client: D
        Patch: David Nadlinger

        D program language library and additions (Revision 1304085)

        Result = SUCCESS
        jfarrell : http://svn.apache.org/viewvc/?view=rev&rev=1304085
        Files :

        • /thrift/trunk/aclocal/ax_check_openssl.m4
        • /thrift/trunk/aclocal/ax_dmd.m4
        • /thrift/trunk/compiler/cpp/Makefile.am
        • /thrift/trunk/compiler/cpp/src/generate/t_d_generator.cc
        • /thrift/trunk/configure.ac
        • /thrift/trunk/lib/Makefile.am
        • /thrift/trunk/lib/d
        • /thrift/trunk/lib/d/Makefile.am
        • /thrift/trunk/lib/d/README
        • /thrift/trunk/lib/d/src
        • /thrift/trunk/lib/d/src/thrift
        • /thrift/trunk/lib/d/src/thrift/async
        • /thrift/trunk/lib/d/src/thrift/async/base.d
        • /thrift/trunk/lib/d/src/thrift/async/libevent.d
        • /thrift/trunk/lib/d/src/thrift/async/socket.d
        • /thrift/trunk/lib/d/src/thrift/async/ssl.d
        • /thrift/trunk/lib/d/src/thrift/base.d
        • /thrift/trunk/lib/d/src/thrift/codegen
        • /thrift/trunk/lib/d/src/thrift/codegen/async_client.d
        • /thrift/trunk/lib/d/src/thrift/codegen/async_client_pool.d
        • /thrift/trunk/lib/d/src/thrift/codegen/base.d
        • /thrift/trunk/lib/d/src/thrift/codegen/client.d
        • /thrift/trunk/lib/d/src/thrift/codegen/client_pool.d
        • /thrift/trunk/lib/d/src/thrift/codegen/idlgen.d
        • /thrift/trunk/lib/d/src/thrift/codegen/processor.d
        • /thrift/trunk/lib/d/src/thrift/index.d
        • /thrift/trunk/lib/d/src/thrift/internal
        • /thrift/trunk/lib/d/src/thrift/internal/algorithm.d
        • /thrift/trunk/lib/d/src/thrift/internal/codegen.d
        • /thrift/trunk/lib/d/src/thrift/internal/ctfe.d
        • /thrift/trunk/lib/d/src/thrift/internal/endian.d
        • /thrift/trunk/lib/d/src/thrift/internal/resource_pool.d
        • /thrift/trunk/lib/d/src/thrift/internal/socket.d
        • /thrift/trunk/lib/d/src/thrift/internal/ssl.d
        • /thrift/trunk/lib/d/src/thrift/internal/ssl_bio.d
        • /thrift/trunk/lib/d/src/thrift/internal/test
        • /thrift/trunk/lib/d/src/thrift/internal/test/protocol.d
        • /thrift/trunk/lib/d/src/thrift/internal/test/server.d
        • /thrift/trunk/lib/d/src/thrift/internal/traits.d
        • /thrift/trunk/lib/d/src/thrift/protocol
        • /thrift/trunk/lib/d/src/thrift/protocol/base.d
        • /thrift/trunk/lib/d/src/thrift/protocol/binary.d
        • /thrift/trunk/lib/d/src/thrift/protocol/compact.d
        • /thrift/trunk/lib/d/src/thrift/protocol/json.d
        • /thrift/trunk/lib/d/src/thrift/protocol/processor.d
        • /thrift/trunk/lib/d/src/thrift/server
        • /thrift/trunk/lib/d/src/thrift/server/base.d
        • /thrift/trunk/lib/d/src/thrift/server/nonblocking.d
        • /thrift/trunk/lib/d/src/thrift/server/simple.d
        • /thrift/trunk/lib/d/src/thrift/server/taskpool.d
        • /thrift/trunk/lib/d/src/thrift/server/threaded.d
        • /thrift/trunk/lib/d/src/thrift/server/transport
        • /thrift/trunk/lib/d/src/thrift/server/transport/base.d
        • /thrift/trunk/lib/d/src/thrift/server/transport/socket.d
        • /thrift/trunk/lib/d/src/thrift/server/transport/ssl.d
        • /thrift/trunk/lib/d/src/thrift/transport
        • /thrift/trunk/lib/d/src/thrift/transport/base.d
        • /thrift/trunk/lib/d/src/thrift/transport/buffered.d
        • /thrift/trunk/lib/d/src/thrift/transport/file.d
        • /thrift/trunk/lib/d/src/thrift/transport/framed.d
        • /thrift/trunk/lib/d/src/thrift/transport/http.d
        • /thrift/trunk/lib/d/src/thrift/transport/memory.d
        • /thrift/trunk/lib/d/src/thrift/transport/piped.d
        • /thrift/trunk/lib/d/src/thrift/transport/range.d
        • /thrift/trunk/lib/d/src/thrift/transport/socket.d
        • /thrift/trunk/lib/d/src/thrift/transport/ssl.d
        • /thrift/trunk/lib/d/src/thrift/transport/zlib.d
        • /thrift/trunk/lib/d/src/thrift/util
        • /thrift/trunk/lib/d/src/thrift/util/awaitable.d
        • /thrift/trunk/lib/d/src/thrift/util/cancellation.d
        • /thrift/trunk/lib/d/src/thrift/util/future.d
        • /thrift/trunk/lib/d/src/thrift/util/hashset.d
        • /thrift/trunk/lib/d/test
        • /thrift/trunk/lib/d/test/Makefile.am
        • /thrift/trunk/lib/d/test/async_test.d
        • /thrift/trunk/lib/d/test/async_test_runner.sh
        • /thrift/trunk/lib/d/test/client_pool_test.d
        • /thrift/trunk/lib/d/test/openssl.test.cnf
        • /thrift/trunk/lib/d/test/serialization_benchmark.d
        • /thrift/trunk/lib/d/test/stress_test_server.d
        • /thrift/trunk/lib/d/test/test_utils.d
        • /thrift/trunk/lib/d/test/thrift_test_client.d
        • /thrift/trunk/lib/d/test/thrift_test_common.d
        • /thrift/trunk/lib/d/test/thrift_test_runner.sh
        • /thrift/trunk/lib/d/test/thrift_test_server.d
        • /thrift/trunk/lib/d/test/transport_test.d
        • /thrift/trunk/test/StressTest.thrift
        • /thrift/trunk/tutorial/d
        • /thrift/trunk/tutorial/d/Makefile
        • /thrift/trunk/tutorial/d/async_client.d
        • /thrift/trunk/tutorial/d/client.d
        • /thrift/trunk/tutorial/d/server.d
        • /thrift/trunk/tutorial/shared.thrift
        • /thrift/trunk/tutorial/tutorial.thrift
        Show
        Hudson added a comment - Integrated in Thrift #429 (See https://builds.apache.org/job/Thrift/429/ ) THRIFT-1500 : d programming language support Client: D Patch: David Nadlinger D program language library and additions (Revision 1304085) Result = SUCCESS jfarrell : http://svn.apache.org/viewvc/?view=rev&rev=1304085 Files : /thrift/trunk/aclocal/ax_check_openssl.m4 /thrift/trunk/aclocal/ax_dmd.m4 /thrift/trunk/compiler/cpp/Makefile.am /thrift/trunk/compiler/cpp/src/generate/t_d_generator.cc /thrift/trunk/configure.ac /thrift/trunk/lib/Makefile.am /thrift/trunk/lib/d /thrift/trunk/lib/d/Makefile.am /thrift/trunk/lib/d/README /thrift/trunk/lib/d/src /thrift/trunk/lib/d/src/thrift /thrift/trunk/lib/d/src/thrift/async /thrift/trunk/lib/d/src/thrift/async/base.d /thrift/trunk/lib/d/src/thrift/async/libevent.d /thrift/trunk/lib/d/src/thrift/async/socket.d /thrift/trunk/lib/d/src/thrift/async/ssl.d /thrift/trunk/lib/d/src/thrift/base.d /thrift/trunk/lib/d/src/thrift/codegen /thrift/trunk/lib/d/src/thrift/codegen/async_client.d /thrift/trunk/lib/d/src/thrift/codegen/async_client_pool.d /thrift/trunk/lib/d/src/thrift/codegen/base.d /thrift/trunk/lib/d/src/thrift/codegen/client.d /thrift/trunk/lib/d/src/thrift/codegen/client_pool.d /thrift/trunk/lib/d/src/thrift/codegen/idlgen.d /thrift/trunk/lib/d/src/thrift/codegen/processor.d /thrift/trunk/lib/d/src/thrift/index.d /thrift/trunk/lib/d/src/thrift/internal /thrift/trunk/lib/d/src/thrift/internal/algorithm.d /thrift/trunk/lib/d/src/thrift/internal/codegen.d /thrift/trunk/lib/d/src/thrift/internal/ctfe.d /thrift/trunk/lib/d/src/thrift/internal/endian.d /thrift/trunk/lib/d/src/thrift/internal/resource_pool.d /thrift/trunk/lib/d/src/thrift/internal/socket.d /thrift/trunk/lib/d/src/thrift/internal/ssl.d /thrift/trunk/lib/d/src/thrift/internal/ssl_bio.d /thrift/trunk/lib/d/src/thrift/internal/test /thrift/trunk/lib/d/src/thrift/internal/test/protocol.d /thrift/trunk/lib/d/src/thrift/internal/test/server.d /thrift/trunk/lib/d/src/thrift/internal/traits.d /thrift/trunk/lib/d/src/thrift/protocol /thrift/trunk/lib/d/src/thrift/protocol/base.d /thrift/trunk/lib/d/src/thrift/protocol/binary.d /thrift/trunk/lib/d/src/thrift/protocol/compact.d /thrift/trunk/lib/d/src/thrift/protocol/json.d /thrift/trunk/lib/d/src/thrift/protocol/processor.d /thrift/trunk/lib/d/src/thrift/server /thrift/trunk/lib/d/src/thrift/server/base.d /thrift/trunk/lib/d/src/thrift/server/nonblocking.d /thrift/trunk/lib/d/src/thrift/server/simple.d /thrift/trunk/lib/d/src/thrift/server/taskpool.d /thrift/trunk/lib/d/src/thrift/server/threaded.d /thrift/trunk/lib/d/src/thrift/server/transport /thrift/trunk/lib/d/src/thrift/server/transport/base.d /thrift/trunk/lib/d/src/thrift/server/transport/socket.d /thrift/trunk/lib/d/src/thrift/server/transport/ssl.d /thrift/trunk/lib/d/src/thrift/transport /thrift/trunk/lib/d/src/thrift/transport/base.d /thrift/trunk/lib/d/src/thrift/transport/buffered.d /thrift/trunk/lib/d/src/thrift/transport/file.d /thrift/trunk/lib/d/src/thrift/transport/framed.d /thrift/trunk/lib/d/src/thrift/transport/http.d /thrift/trunk/lib/d/src/thrift/transport/memory.d /thrift/trunk/lib/d/src/thrift/transport/piped.d /thrift/trunk/lib/d/src/thrift/transport/range.d /thrift/trunk/lib/d/src/thrift/transport/socket.d /thrift/trunk/lib/d/src/thrift/transport/ssl.d /thrift/trunk/lib/d/src/thrift/transport/zlib.d /thrift/trunk/lib/d/src/thrift/util /thrift/trunk/lib/d/src/thrift/util/awaitable.d /thrift/trunk/lib/d/src/thrift/util/cancellation.d /thrift/trunk/lib/d/src/thrift/util/future.d /thrift/trunk/lib/d/src/thrift/util/hashset.d /thrift/trunk/lib/d/test /thrift/trunk/lib/d/test/Makefile.am /thrift/trunk/lib/d/test/async_test.d /thrift/trunk/lib/d/test/async_test_runner.sh /thrift/trunk/lib/d/test/client_pool_test.d /thrift/trunk/lib/d/test/openssl.test.cnf /thrift/trunk/lib/d/test/serialization_benchmark.d /thrift/trunk/lib/d/test/stress_test_server.d /thrift/trunk/lib/d/test/test_utils.d /thrift/trunk/lib/d/test/thrift_test_client.d /thrift/trunk/lib/d/test/thrift_test_common.d /thrift/trunk/lib/d/test/thrift_test_runner.sh /thrift/trunk/lib/d/test/thrift_test_server.d /thrift/trunk/lib/d/test/transport_test.d /thrift/trunk/test/StressTest.thrift /thrift/trunk/tutorial/d /thrift/trunk/tutorial/d/Makefile /thrift/trunk/tutorial/d/async_client.d /thrift/trunk/tutorial/d/client.d /thrift/trunk/tutorial/d/server.d /thrift/trunk/tutorial/shared.thrift /thrift/trunk/tutorial/tutorial.thrift

          People

          • Assignee:
            David Nadlinger
            Reporter:
            David Nadlinger
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development