[RFC]: Expanding array-based statistical computation in stdlib

[gururaj1512]

Full name

Gururaj Gurram

University status

Yes

University name

Walchand Institute of Technology

University program

Computer Science & Engineering

Expected graduation

2027

Short biography

I am a second-year undergraduate student pursuing a B.Tech degree in Computer Science and Engineering. My academic journey has equipped me with a strong foundation in Data Structures and Algorithms, Computer Networks, linear algebra, and Mathematics. My technical expertise spans Python, C, JavaScript, TypeScript, and frameworks/tools like React, Docker, Firebase, etc.

My journey began with web development, subsequently expanding to encompass full-stack development, machine learning, and generative AI. These skills proved invaluable during the development of various academic projects and internships. Furthermore, my passion for collaborative development has driven me to participate in numerous hackathons and build innovative solutions.

Beyond academics, I am passionate about developing applications, exploring emerging technologies and contributing to open-source projects. I always try to learn and build cool things, which helps me to keep getting better as a student and a developer.

Timezone

Indian Standard Time (GMT+5:30)

Contact details

email: gururajgurram1512@gmail.com, github: gururaj1512

Platform

Windows

Editor

I use Visual Studio Code as my preferred code editor because of its user-friendly interface, and wide variety of available extensions. Paired with Windows Subsystem for Linux (WSL), it allows me to seamlessly run a Linux environment on my Windows machine, making development more efficient. The built-in Git integration, and support for extensions like EditorConfig and GitHub Copilot enhance my workrate, ensuring consistent code quality and faster development.

Programming experience

Although my programming journey began with HTML, CSS, and C in high school, I delved deeper into it in college by expanding my knowledge of JavaScript and web development. Later, I explored backend technologies, specifically the MERN stack, along with databases and shell scripting, while working on various academic projects, hackathons, and open-source contributions. Some of the projects I have worked on include:

• CureConnect : A web application that delivers a seamless healthcare experience through a telemedicine platform. It offers video consultations, real-time chat, AI-driven ECG and X-ray analysis, personalized health tips, an emergency hospital locator, and an intuitive appointment dashboard—all in one place. (The provided link leads to documentation, as the project is currently in its pre-seed stage.)
• NyayaSetu : An intelligent chatbot designed to streamline and enhance the user experience for individuals accessing the Department of Justice's services. A Smart India Hackathon 2024 finalist, this chatbot offers:
  • Case Status Updates: Provides real-time information on legal case statuses.
  • eFiling Support: Guides users step-by-step through electronic case and document filing.
  • Tele-Law Services: Connects users with legal aid and advisory services for timely support.
  • Data Integration: Scrapes the DOJ website to provide users with comprehensive, up-to-date legal information.
• Horizon : Horizon is full stack banking app developed with NEXT.js for frontend, Appwrite for backend, dwolla & shadcn

JavaScript experience

JavaScript is the language I am most comfortable with, as I have used it extensively in almost all of my projects, especially during my contributions to stdlib. Working with JavaScript has allowed me to explore its versatility, from building backend services and APIs to designing extraordinary frontends. My experience with stdlib has further deepened my understanding of JavaScript’s structure, flow, and best practices, enabling me to write clean and maintainable code.

One of JavaScript’s biggest advantages is its rich ecosystem, which includes a wide range of frameworks and libraries for frontend, backend, and even current AI/ML development. Its asynchronous programming model, with features like promises and async/await, makes handling complex tasks seamless. However, JavaScript’s weak typing and performance limitations can be drawbacks, especially in larger codebases. Despite this, its flexibility and widespread adoption make it an indispensable tool for modern development.

Node.js experience

My experience with Node.js comes from working on various projects and contributions that have helped me understand its core functionalities and ecosystem. I have used Node.js to build backend services, develop APIs, and perform general-purpose scripting, as it is a widely used runtime environment for JavaScript. Through these projects, I have gained hands-on experience with frameworks and libraries like Express.js and Mongoose, as well as working with databases and authentication systems. Additionally, my contributions to open-source projects, such as stdlib-js, have further deepened my understanding of Node.js, particularly in implementing efficient routines and handling complex tasks.

C/Fortran experience

My experience with C began during my academic journey, as it was the first programming language I learned. Through coursework and projects, I developed a solid understanding of its core concepts, including data structures and memory management. Additionally, I have applied my C knowledge in various scenarios, such as implementing blas, stats, and math routines while contributing to stdlib, further strengthening my skills.

While my experience with Fortran is still basic and in the learning phase, I have gained familiarity with it while working on a PR for blas/base routine. Translating Fortran 77 reference implementations into C has helped me understand its syntax and structure. Although I am still building my proficiency in Fortran, my strong foundation and adaptability enable me to effectively work with it as needed.

Interest in stdlib

I started my open-source journey with stdlib because of its alignment with my technical knowledge and my interest in mathematics. Initially, I questioned:

Necessity of stdlib, Due to widespread use and development of Python libraries like NumPy and SciPy for mathematical and scientific computing.

However, as I continued contributing and opening PRs, I was amazed by the vision of bringing NumPy and SciPy-like functionalities directly into the JavaScript ecosystem. The idea of enabling complex mathematical operations in the browser without relying on Python libraries is both innovative and practical. Additionally, stdlib's inclusion of LAPACK and BLAS routines under one roof further highlights its comprehensive approach to technical computing. Resources like this podcast and this video helped me gain a deeper understanding of stdlib's mission and its potential impact.

What makes stdlib truly special is its welcoming and supportive community. The maintainers are approachable and always ready to guide contributors, making it easy for newcomers to get started. The collaborative environment fosters learning and innovation, ensuring that every contribution is a valuable experience. The detailed review process and strong emphasis on maintaining a high-quality codebase have helped me improve my skills and appreciate the importance of writing clean, consistent code. Being part of such a positive and inspiring community as both a contributor and a member motivates me to continue contributing and growing as a developer.

Version control

Yes

Contributions to stdlib

I have contributed to stdlib extensively across various categories. A comprehensive list of all my pull requests can be found here:

Merged Work

• Migrated @stdlib/math/base/ops/* packages to @stdlib/number/* and @stdlib/complex/*.
• Added C and JS implementations for math/base/special functions.
• Added C implementations for stats/base/dists functions.
• Updated documentation for blas/base packages.
• Improved test coverage.
• Refactored existing code to follow current coding conventions.
• Performed cleanup, refactored benchmarks, and applied fixes.

Open Work

• My open work under review includes PRs related to ndarray, blas/base packages, and C and JS implementations for math/base/special functions.

stdlib showcase

stdlib-showcase: A project demonstrating the capabilities of stdlib, with a focus on ndarray operations and mathematical functions.

• NDArray Page: Includes an interactive visualizer for exploring multi-dimensional arrays.
• Math Page: Highlights matrix operations such as dot products and matrix multiplication.

This is a simple project created to explore the use of stdlib functions.

health-risk-predictor: A basic attempt at anomaly detection using statistical functions, inspired by my work on CureConnect. While not a comprehensive health-risk prediction tool, it explores related concepts.

Goals

Abstract

This project aims to improve the stats routines by introducing array-friendly wrappers for all 1D strided APIs, alongside implementing specialized ndarray kernels tailored for various statistical computations and wasm builds. These wrappers will simplify access to statistical functions for common use cases, while the ndarray kernels will boost performance, scalability, and adaptability. Additionally, these enhancements will make the library more efficient and user-friendly for developers working with multidimensional data.

Main Goals:

• Introduce array-friendly wrappers for all 1D strided APIs in the stats/strided/* namespace to simplify access to statistical functions for common use cases.
• Develop specialized ndarray kernels for various statistical operations, such as mean, variance, and correlation, to serve as foundational building blocks for higher-level routines.
• Add C ndarray interfaces and refactor all single and double-precision packages, migrating them from the stats/base/* namespace to the stats/strided/* namespace.
• Refactor and optimize existing implementations to align with stdlib's coding standards and improve maintainability.
• Implement WebAssembly (WASM) versions of some single and double-precision stats/strided/* packages to enhance performance and scalability with low priority.

Supporting Goals:

• Enhance overall stats packages.
• Ensure all new implementations are thoroughly tested with relevant benchmarks, unit tests, and achieve 100% test coverage for all stats/strided packages.
• Update documentation for all modified and newly implemented packages to maintain clarity, usability, and alignment with stdlib's conventions.
• Provide detailed documentation and usage guides for ndarray kernels to help users understand their functionality.
• Collaborate with mentors and contributors to ensure the implementations meet stdlib's high-performance and quality standards.
• If time permits, contribute to improving test coverage and refactoring related packages for better performance and maintainability.

Approach

1. Enhancing & Refactoring
   The first step will involve completing the prerequisites for the project. This includes adding C ndarray interfaces and refactoring packages by migrating them from the stats/base/* namespace to the stats/strided/* namespace.

2. Adding Array-Friendly Wrappers
   Array-friendly wrappers simplify the usage of 1D strided APIs, making them more accessible for common use cases.

   These wrappers provide a more intuitive interface for users. For example, the current mean function is used as follows:

   mean( N, x, strideX );

   This can be adapted for typical use cases with a wrapper like this:

   // Package: stats/array/mean
   
   var strided = require( '@stdlib/stats/strided/mean' );
   
   function mean( x ) {
        return strided( x.length, x, 1 );
   }

   The wrapper allows users to call the function in a simpler way:

   mean( x ); // x => Array-like object

   This approach enhances usability by abstracting away the need to manually specify parameters like N and strideX for common scenarios.

3. Develop Lower-Level NDArray Kernels for Statistical Operations
   In parallel, I will work on implementing lower-level ndarray kernels for statistical operations such as mean, variance, standard deviation, and correlation. These kernels will efficiently handle multi-dimensional data and act as foundational components for advanced statistical routines.

   Currently, the statistics packages are limited to one-dimensional arrays (e.g., Float64Array, Float32Array, etc.). To enable more versatile usage, ndarray kernels can be developed to support statistical operations on multi-dimensional arrays.

   Below is an illustrative example (for conceptual purposes only):

   var ndarray = require( '@stdlib/ndarray/ctor' );
   var ndarray2array = require( '@stdlib/ndarray/to-array' );
   var mean = require( '@stdlib/stats/base/mean' ); // ndarray kernel
   
   var buffer = [ 1, 2, 3, 4 ];
   var shape = [ 2, 2 ];
   var order = 'row-major';
   var strides = [ 2, 1 ];
   var offset = 0;
   
   var x = ndarray( 'generic', buffer, shape, strides, offset, order );
   // returns <ndarray>
   
   var out = ndarray2array( x );
   // returns [ [ 1, 2 ], [ 3, 4 ] ]
   
   var v = mean( N, x, 1, ... ); // arguments may vary based on requirements
   // returns 2.5

Project Outcome

By the end of this project, the stats/strided/* routines will include array-friendly wrappers, making statistical functions more accessible for common use cases. Additionally, lower-level ndarray kernels for statistical operations, such as mean, variance, and correlation, will be developed to efficiently handle multi-dimensional data and serve as foundational building blocks for advanced routines.

I will continue to assist stdlib in the near future by contributing to various areas, including adding WebAssembly implementations for stats/strided/* packages. My goal is to support the library's growth and help enhance its performance and usability wherever possible.

Why this project?

The proposed project offers an exciting opportunity to enhance numerical computing routines by introducing convenience wrappers and developing specialized ndarray kernels for statistical operations. These improvements aim to make statistical functions more accessible and efficient for developers working with multi-dimensional data.

My interest stems from working with stats and ndarray routines in stdlib, which are fundamental to scientific and engineering computations. Developing ndarray kernels for statistical operations excites me, as they will enable efficient handling of multi-dimensional data and serve as foundational building blocks for advanced statistical routines. Additionally, creating convenience wrappers for 1D strided APIs will simplify their usage, making them more intuitive and user-friendly for common scenarios.

My primary focus is on improving usability and performance through these wrappers and kernels. The challenge of designing efficient, modular solutions that align with stdlib's mission to make high-performance numerical computing accessible in JavaScript is both inspiring and rewarding.

By contributing to this project, I aim to deepen my understanding of ndarray structures and statistical operations while making meaningful contributions to the open-source community.

Qualifications

This project requires proficiency in JavaScript and C, along with a solid grasp of numerical computing and the structure of stdlib. I am confident in meeting these requirements, as I have developed a strong foundation in JavaScript and C through academic coursework, hands-on projects, and contributions to stdlib.

My contributions to stdlib have provided valuable insights into its architecture, coding standards, and review process, equipping me to write high-quality code. With my technical skills, adaptability, and eagerness to learn, I am well-prepared to take on this project and contribute effectively to stdlib.

Prior art

A significant portion of the single-precision and double-precision packages has already been updated with C ndarray interfaces and migrated to the stats/strided/* namespace, significantly reducing the workload. However, while there are advanced packages for statistical operations, they need to be simplified for general use cases to improve accessibility. Additionally, although the stats packages support array-like objects, they currently do not support ndarrays and no work has been done until now in this direction. Introducing multidimensional support for these packages would greatly enhance their usability and accessibility.

Commitment

Given my familiarity with the codebase and a strong understanding of the proposed work, I plan to start working on the project before the GSoC period and utilize the community bonding period to begin completing some requisites. Although I have examinations scheduled from May 20th to May 31st during the community bonding period, I do not have any other major commitments this summer.

Once the coding period begins on June 1st, I plan to dedicate approximately 40 hours per week until July 15th. After that, I will continue working at a slower pace, dedicating around 20 hours per week.

Schedule

Assuming a 12-week schedule:

• Community Bonding Period:
  Given my familiarity with the codebase and availability, I will begin by refactoring and adding C ndarray interfaces to stats packages that currently lack them and parallely adding convenience wrappers. My primary focus will be on collaborating with mentors to deepen my understanding of ndarray kernels for statistical operations, their functionality, usability, and related research.

• Week 1 - Week 4:
  I will start working on convenience wrappers for 1D strided APIs to simplify their usage for common scenarios. Concurrently, I will implement WebAssembly (WASM) builds for packages that significantly vary from each other, such as dmean, dvariance, and dstdev. This phase will focus on laying the groundwork for the project by addressing distinct packages first.

• Week 5 - Week 8:
  After conducting thorough research on ndarray kernels, I will begin implementing them for statistical operations such as mean, variance, and standard deviation. Concurrently, I will continue working on convenience wrappers for 1D strided APIs to ensure their usability and accessibility.

• Week 9 - Week 12:
  During this phase, I will focus on completing the most critical tasks, including finalizing ndarray kernels, achieving 100% test coverage, and writing benchmarks. Additionally, I will work on detailed documentation and examples to ensure clarity and usability for all implemented features.

• Post GSoC:
  After the GSoC period, I plan to continue contributing to stdlib as an active contributor. My focus will be on improving existing implementations, enhancing test coverage, and contributing to new features. As someone who loves stdlib and its mission, I am excited to remain a part of this vibrant open-source community and contribute to its growth and success.

Notes:

• The community bonding period is a 3 week period built into GSoC to help you get to know the project community and participate in project discussion. This is an opportunity for you to setup your local development environment, learn how the project's source control works, refine your project plan, read any necessary documentation, and otherwise prepare to execute on your project project proposal.
• Usually, even week 1 deliverables include some code.
• By week 6, you need enough done at this point for your mentor to evaluate your progress and pass you. Usually, you want to be a bit more than halfway done.
• By week 11, you may want to "code freeze" and focus on completing any tests and/or documentation.
• During the final week, you'll be submitting your project.

Checklist

• I have read and understood the Code of Conduct.
• I have read and understood the application materials found in this repository.
• I understand that plagiarism will not be tolerated, and I have authored this application in my own words.
• I have read and understood the patch requirement which is necessary for my application to be considered for acceptance.
• I have read and understood the stdlib showcase requirement which is necessary for my application to be considered for acceptance.
• The issue name begins with [RFC]: and succinctly describes your proposal.
• I understand that, in order to apply to be a GSoC contributor, I must submit my final application to https://summerofcode.withgoogle.com/ before the submission deadline.

[kgryte]

@gururaj1512 Thank you for opening this RFC. A few initial comments/questions:

1. Do you have a sense of timeline for your proposed activities?
2. Given the breadth of proposed work, how are you planning on prioritizing tasks?
3. I wonder if we could modify this proposal a bit such that the first half is finishing off some of the stats/base/* clean-up work, including C ndarray interfaces and migration, along with WebAssembly builds. And the second half continues that work at a slower pace while working on lower level ndarray kernels, similar to those found in ndarray/base/*. However, this is somewhat dependent on your availability/commitment over the summer and beyond. Regardless, were the focus of the proposal to change/widen a bit, you'd want to rename your proposal and reframe its goals.

[gururaj1512]

@kgryte

1. If we focus sololy on ndarray interfaces, refactoring implementations, and migrations along with WebAssembly (WASM) builds for double-precision packages, the project can certainly be completed within 350+ hours, though some additional time may be needed.
2. However, your suggested approach seems preety much practical—refactoring, migrating, and implementing WASM packages. Since WASM packages can be replicated, and once the initial package PRs are merged, they won’t be too time-consuming.
3. I am considering to reshape the proposal accoring to your suggstion and include the work on lower level ndarray/base/* kernels. I have got the basic idea of this, such as adding ndarray/base/reinterpret-complex and addressing missing feature support in some packages. Could you elaborate on this further so I can properly outline it in my proposal? I've also been researching and exploring to identify irregularities.

[gururaj1512]

@kgryte I have reformatted the goals. You can have a look

[kgryte]

And the second half continues that work at a slower pace while working on lower level ndarray kernels, similar to those found in ndarray/base/*.

@gururaj1512 By lower level ndarray kernels, I have specifically in mind stats/kernels. These would be similar to ndarray/base/*, but not the same. In particular, I'd be interested in seeing the development of ndarray kernels for various statistical operations (e.g., mean, variance, pcorr, etc).

[kgryte]

The ndarray/base/* feature parity work should be considered a separate project and not mixed up in this one.

[gururaj1512]

The ndarray/base/* feature parity work should be considered a separate project and not mixed up in this one.

Ahhhhh actually got it...
I thought you were suggesting adding missing ndarray/base/* implementations which were suggested by you previously at gitter.
I'll refactor it for stats/kernels