[STATISTICS-7] Stream-based Java statistical processing

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: descriptive
• Labels: GSoC2019 gsoc2019 statistics streams
• Assignee: Unassigned
• Reporter: Eric Barnhill
• Created: 12/Mar/19 17:59
• Updated: 11/Nov/22 12:54

Description

The new component aims to be a library of commons statistics functions synchronized with the latest developments in the Java language, in particular Java's functional programming syntax.

The library will make commonly used statistical functions available to an end user through a simple grammar comparable to commons-math-statistics or scikit-learn, while under the hood will implement Java's mapping, streaming, and other producer and consumer functions to ensure the statistical methods run optimally in new Java implementations.

As functional programming grows increasingly central to big data applications we believe these libraries will play an important function in the data engineering ecosystem. In particular, data engineering is widely done with Java, then passed to other languages for data-scientific analyses; however, the common availability of functionally implemented statistical mapping and reductions in Java could prove very useful at the interface of data science and engineering, by enabling teams to more easily perform reductions on the engineering side before handing off to the analysis side.

Developers working on the project will have the opportunity to demonstrate Java programming, functional programming, algorithm design, and data science skills and receive authorship on a commons project that is likely to be widely used.

The ideal contributor will also be able to help with important architectural decision making. The old source of these libraries, commons-math, grew too large, hierarchically complex and interdependent for the commons mission. The developers on this project need to make architectural choices that will enable the statiscal code to be lightweight and reusable, with a minimum of outside dependencies while avoiding redundancy.

Issue Links

• is related to: STATISTICS-5 Port codes from Commons Math
• relates to: MATH-1491 Percentile computational accuracy issue
• relates to: MATH-928 Alternative approach to matrix implementations
• relates to: STATISTICS-54 [GSoC] Summary statistics API for Java 8 streams
• relates to: STATISTICS-8 Implementation of regression libraries within common-statistics framework
• links to: GitHub Pull Request #7
• links to: GitHub Pull Request #17

Sub-Tasks

• Class Diagram of "math4.stat.descriptive" module & development flow: Open
• Port Percentile class from Commons-Maths: Reopened
• GSoC-Progress Tracker for "Commons-Statistics-Descriptive": Closed