Uploaded image for project: 'REEF'
  1. REEF
  2. REEF-334

Make sure REEF.NET builds and runs on Linux

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: REEF.NET
    • Labels:
      None

      Description

      .NET is being open sourced and ported to Linux and OSX. That opens the door for REEF.NET to also be available on those platforms. This JIRA serves as the umbrella for work towards that goal.

        Issue Links

        1.
        Figure out testing for .NET on Linux Sub-task Resolved Unassigned  
         
        2.
        Fix CoreCLR incompatibilities in Tang Tools Sub-task Closed Mario Inchiosa  
         
        3.
        Fix core .NET compatibility issues in REEF.Driver Sub-task Resolved Doug Service  
         
        4.
        Fix CoreCLR incompatibility for isDefined function in Tang Reflection Utilities Sub-task Closed Shiv Sundram

        0%

        Original Estimate - 24h
        Remaining Estimate - 24h
         
        5.
        Fix CoreCLR incompatibilities in Tang related to System.IO and Application Exceptions Sub-task Closed Shiv Sundram  
         
        6.
        Fix CoreCLR incompatibilities in Tang Sub-task Closed Unassigned  
         
        7.
        Fix remaining Core .NET incompatibilities in REEF.Driver Sub-task Resolved Doug Service  
         
        8.
        Fix .NET Core System.Type.get_Assembly incompatibility in Tang.Tests Sub-task Closed Mario Inchiosa  
         
        9.
        Fix .NET Core System.Text.Encoding.get_Default incompatibility in Tang Sub-task Closed Mario Inchiosa  
         
        10.
        Fix CoreCLR incompatibilities in Tang related to String.StartsWith method Sub-task Closed Shiv Sundram  
         
        11.
        Fix CoreCLR incompatibilities related to use of SystemException Sub-task Closed Mario Inchiosa  
         
        12.
        Fix comments in Tang.NET referring to ApplicationException Sub-task Closed Mario Inchiosa  
         
        13.
        Fix .NET Core compatibility issues in REEF.Wake Sub-task Resolved Doug Service  
         
        14. Update Newtonsoft Json.NET to version 10.0.3 Sub-task In Progress Julia  
         
        15.
        Create build of CoreCLR compatible projects Sub-task Closed Scott Inglis  
         
        16.
        Fix CoreCLR issues in Org.Apache.REEF.Wake.Tests Sub-task Resolved Unassigned  
         
        17.
        Fix CoreCLR issues in Org.Apache.REEF.Wake Sub-task Closed Scott Inglis  
         
        18.
        Fix CoreCLR issues in Org.Apache.REEF.Utilities Sub-task Closed Scott Inglis  
         
        19.
        Fix CoreCLR issues in Org.Apache.REEF.Tests Sub-task Resolved Scott Inglis  
         
        20.
        Fix CoreCLR issues in Org.Apache.REEF.Tang Sub-task Closed Scott Inglis  
         
        21.
        Fix CoreCLR issues in Org.Apache.REEF.Network.Tests Sub-task Resolved Unassigned  
         
        22.
        Fix CoreCLR issues in Org.Apache.REEF.Network.Examples.Client Sub-task Resolved Rogan Carr  
         
        23.
        Upgrade REEF.NET to require .NET 4.5.1 Sub-task Resolved Saikat Kanjilal  
         
        24.
        Fix CoreCLR issues in Org.Apache.REEF.IO Sub-task Resolved Rogan Carr  
         
        25.
        Fix CoreCLR issues in Org.Apache.REEF.IMRU Sub-task Resolved Rogan Carr  
         
        26.
        Fix CoreCLR issues in Org.Apache.REEF.Evaluator Sub-task Resolved Rogan Carr  
         
        27.
        Fix CoreCLR issues in Org.Apache.REEF.Driver, Sub-task Resolved Saikat Kanjilal  
         
        28.
        Fix CoreCLR issues in Org.Apache.REEF.Client Sub-task Resolved Rogan Carr  
         
        29.
        Fix CoreCLR issues in Org.Apache.REEF.Driver Sub-task Closed Unassigned  
         
        30.
        Fix CoreCLR issues in Org.Apache.REEF.Client Sub-task Closed Unassigned  
         
        31.
        Migrate Wake projects to VS2017 csproj files Sub-task Resolved Scott Inglis  
         
        32.
        Migrate all tang projects to VS2017 Sub-task Resolved Scott Inglis  
         
        33.
        Migrate IO and Evaluator to VS2017 Sub-task Closed Scott Inglis  
         
        34.
        Migrate remaining projects to .net core. Sub-task Resolved Scott Inglis  
         
        35.
        Enable VS 2017 command line builds by adding nuget.org to lang/cs/nuget.config Sub-task Resolved Doug Service  
         
        36.
        Upgrade nuget.exe to version 4.1. Sub-task Resolved Doug Service  
         
        37.
        Port PerformanceCounters to a CoreCLR-Compliant System Sub-task Resolved Rogan Carr  
         
        38.
        Upgrade System.Reactive.* to 3x for CoreCLR Compatibility Sub-task Resolved Rogan Carr  
         
        39.
        Make REEF.Common Exceptions CoreCLR Compliant Sub-task Closed Rogan Carr  
         
        40.
        Create a script to analyze CLR Compliance Sub-task Resolved Rogan Carr  
         
        41.
        Update Protobuf to version 2.1.0 Sub-task Closed Scott Inglis  
         
        42.
        Fix CoreCLR incompatibilities in REEF Network Sub-task Resolved Rogan Carr  
         
        43.
        Add netcoreapp2.0 to target framework for Org.Apache.Utilities Sub-task Closed Unassigned  
         
        44.
        Add netcoreapp2.0 and netstandard2.0 to target framework for Org.Apache.REEF.Tang Sub-task Resolved Unassigned  
         
        45. Replace Microsoft.Hadoop.Avro to Microsoft.Avro.Core Sub-task Open Unassigned  
         
        46.
        Add netcoreapp2.0 to some projects Sub-task Closed Scott Inglis  
         
        47. Update Build Documentation for dot net core projects Sub-task Open Scott Inglis  
         
        48.
        Update .net core projects to support .net core 2.0 Sub-task Resolved Scott Inglis  
         
        49. Remove .net46 and .net451 targets. Sub-task Open Unassigned  
         
        50. .NetCore: Unsupported Operation StreamingNetworkServicetests.TestStreamingCodecFunctionCache Sub-task Open Unassigned  
         
        51. Network.Tests aborts the active test run Sub-task Open Unassigned  
         
        52. Avro Custom Task needs to be moved to a class for BridgeCLR project Sub-task Open Unassigned  
         

          Activity

          Hide
          rogan Rogan Carr added a comment -

          Note that we are at the stage where the majority of code fixes for REEF to target .NET Standard 2.0 are in or are in progress. The next major steps are to

          • Finish the DotNet build files
          • Fixing all the build issues that crop up
          Show
          rogan Rogan Carr added a comment - Note that we are at the stage where the majority of code fixes for REEF to target .NET Standard 2.0 are in or are in progress. The next major steps are to Finish the DotNet build files Fixing all the build issues that crop up
          Hide
          rogan Rogan Carr added a comment - - edited

          For reference, if you want to build for .NET Standard 2.0, you can use the dotnet command-line tool.

          e.g. For Org.Apache.REEF.Utilities:
          dotnet build --framework netstandard2.0 Org.Apache.REEF.Utilities.DotNet.csproj

          (This successfully builds a .NET Standard 2.0 library on Bash on Ubuntu on Windows. I haven't tested on a GNU/Linux environment yet.}

          Show
          rogan Rogan Carr added a comment - - edited For reference, if you want to build for .NET Standard 2.0 , you can use the dotnet command-line tool. e.g. For Org.Apache.REEF.Utilities: dotnet build --framework netstandard2.0 Org.Apache.REEF.Utilities.DotNet.csproj (This successfully builds a .NET Standard 2.0 library on Bash on Ubuntu on Windows. I haven't tested on a GNU/Linux environment yet.}
          Hide
          markus.weimer Markus Weimer added a comment -

          Unlikely. This is pretty involved work redoing the bridge, our protocols and such. At the same time, .NET Core is a moving target.

          Show
          markus.weimer Markus Weimer added a comment - Unlikely. This is pretty involved work redoing the bridge, our protocols and such. At the same time, .NET Core is a moving target.
          Hide
          bgchun Byung-Gon Chun added a comment -

          Can this issue be a GSoC topic?

          Show
          bgchun Byung-Gon Chun added a comment - Can this issue be a GSoC topic?
          Hide
          shravanmn Shravan Matthur Narayanamurthy added a comment -

          I see a couple of things here:
          Pros:
          1. Bridge porting need not be done
          2. Bridge code can be eliminated and no longer maintained
          3. We get the benefits of .Net on the evaluator side where it matters
          Cons:
          1. We have to migrate all existing .Net drivers
          2. The developer experience is a bit weird as the driver needs to developed in Java while task code is in .Net. Not sure how we will manage build dependencies on either side.

          I am not sure how to weigh these trade-offs and it probably depends on the longer term vision of whether we want to have .Net on the driver side.

          Show
          shravanmn Shravan Matthur Narayanamurthy added a comment - I see a couple of things here: Pros: 1. Bridge porting need not be done 2. Bridge code can be eliminated and no longer maintained 3. We get the benefits of .Net on the evaluator side where it matters Cons: 1. We have to migrate all existing .Net drivers 2. The developer experience is a bit weird as the driver needs to developed in Java while task code is in .Net. Not sure how we will manage build dependencies on either side. I am not sure how to weigh these trade-offs and it probably depends on the longer term vision of whether we want to have .Net on the driver side.
          Hide
          markus.weimer Markus Weimer added a comment -

          At the very least, this would be a good milestone. However, we have some apps we'd like to run on Linux which have a .NET Driver...

          Show
          markus.weimer Markus Weimer added a comment - At the very least, this would be a good milestone. However, we have some apps we'd like to run on Linux which have a .NET Driver...
          Hide
          shravanmn Shravan Matthur Narayanamurthy added a comment -

          +1

          This gives us a solution that has the advantages of .Net where it matters and the existence of almost bug-free driver part of the code in Java. The only thing that we might have wanted additionally from the Java side is data access which would still be a pain to do with .Net.

          Show
          shravanmn Shravan Matthur Narayanamurthy added a comment - +1 This gives us a solution that has the advantages of .Net where it matters and the existence of almost bug-free driver part of the code in Java. The only thing that we might have wanted additionally from the Java side is data access which would still be a pain to do with .Net.
          Hide
          anupam128 Anupam added a comment -

          Markus Weimer Is this just a good to have or are there concrete motivations driving this effort.
          If my understanding is right, most of the complex changes for CoreCLR compatibility will be isolated to .NET driver. In that case another approach for the user could be to use Java driver with CoreCLR compatible evaluator. Not entirely sure of the feasibility of this idea, just putting it out there.

          Show
          anupam128 Anupam added a comment - Markus Weimer Is this just a good to have or are there concrete motivations driving this effort. If my understanding is right, most of the complex changes for CoreCLR compatibility will be isolated to .NET driver. In that case another approach for the user could be to use Java driver with CoreCLR compatible evaluator. Not entirely sure of the feasibility of this idea, just putting it out there.
          Hide
          MariiaMykhailova Mariia Mykhailova added a comment -

          Shiv Sundram Do you have permissions to close redundant JIRA as duplicate? Or you can edit JIRA descriptions so that they address different issues.

          Show
          MariiaMykhailova Mariia Mykhailova added a comment - Shiv Sundram Do you have permissions to close redundant JIRA as duplicate? Or you can edit JIRA descriptions so that they address different issues.
          Hide
          shsundr Shiv Sundram added a comment - - edited

          When I built the task, I clicked the "create" button twice: the first time nothing happened, so I clicked it again, and it worked...However, there must have just been some latency the first time I clicked, so two identical tasks were created.

          Show
          shsundr Shiv Sundram added a comment - - edited When I built the task, I clicked the "create" button twice: the first time nothing happened, so I clicked it again, and it worked...However, there must have just been some latency the first time I clicked, so two identical tasks were created.
          Hide
          MariiaMykhailova Mariia Mykhailova added a comment -

          Shiv Sundram Could you please clarify what is the difference between REEF-1500 and REEF-1501? They both say "Fix CoreCLR incompatibilities in Tang project"

          Show
          MariiaMykhailova Mariia Mykhailova added a comment - Shiv Sundram Could you please clarify what is the difference between REEF-1500 and REEF-1501 ? They both say "Fix CoreCLR incompatibilities in Tang project"
          Hide
          markus.weimer Markus Weimer added a comment -

          One challenge will be to make sure that future changes will always be compatible with CoreCLR. One way way would be to add a build server using CoreCLR on Linux.

          Show
          markus.weimer Markus Weimer added a comment - One challenge will be to make sure that future changes will always be compatible with CoreCLR. One way way would be to add a build server using CoreCLR on Linux.
          Hide
          markus.weimer Markus Weimer added a comment -

          Beysim Sezgin, what's the status on this work thread? Are you looking into it or should you un-assign yourself?

          Show
          markus.weimer Markus Weimer added a comment - Beysim Sezgin , what's the status on this work thread? Are you looking into it or should you un-assign yourself?
          Hide
          markus.weimer Markus Weimer added a comment -

          I added a link to the .NET Portability analyzer. Scanning its output, we seem to be in the 90%+ range of portability to .NET Core. Next step would probably be to go through the reports in more detail and then to change our code accordingly to only use APIs that are in .NET Core.

          Show
          markus.weimer Markus Weimer added a comment - I added a link to the .NET Portability analyzer. Scanning its output, we seem to be in the 90%+ range of portability to .NET Core. Next step would probably be to go through the reports in more detail and then to change our code accordingly to only use APIs that are in .NET Core.
          Hide
          markus.weimer Markus Weimer added a comment - - edited

          There is an article on MSDN on Porting to .NET Core.

          Show
          markus.weimer Markus Weimer added a comment - - edited There is an article on MSDN on Porting to .NET Core .
          Hide
          markus.weimer Markus Weimer added a comment -

          One way to attack this is to start with already built executables and check whether they run on Mono / CoreCLR. Given the C++/C#/Java nature of the Driver, that seems a less likely success in the beginning. Maybe we should start with the Evaluator.exe first.

          Show
          markus.weimer Markus Weimer added a comment - One way to attack this is to start with already built executables and check whether they run on Mono / CoreCLR. Given the C++/C#/Java nature of the Driver, that seems a less likely success in the beginning. Maybe we should start with the Evaluator.exe first.

            People

            • Assignee:
              Unassigned
              Reporter:
              markus.weimer Markus Weimer
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development