Dependencies of ML projects

[nielsbauman]

We could try to identify how dependencies affect the maintainability of an ML project.

Possible approaches:

• Looking at one project - look at the history of the dependencies and relate it to the maintainability of the project.
• Comparing multiple projects - see when developers are adding/removing dependencies.
• Try to find a correlation between certain dependencies and maintainability.
• Look at the history of the dependency requirements.

[rlogothetis01]

Motivation: can we improve maintainability and robustness relating to dependencies

[rlogothetis01]

Data:

• data set
• ML repos
• gitprojects with a few commits about 100
• python projects

[DungLai]

Motivating example: Lots of machine learning projects and paper implementation are written in python2. However, python2 is no longer supported. It is difficult to change to python 3, difficult to install dependency for python2. If there is a security vulnerability in the code, it's difficult to maintain or fix the existing systems that use depreciated dependencies.

[luiscruz]

Tool to extract module dependencies
https://pypi.org/project/findimports/

[DungLai]

List of projects can be found in this list of over 4000 repositories from this Microsoft paper

[rlogothetis01]

Methodology:

• identify depencies by looking at statements or write a script (tool) [AST]
• dependencies change over time (when added and when removed - if removed)
• look at size of projects: define size - features, no. files etc.

[rlogothetis01]

• do a comparative analysis
• identify if there is a correlation between dependency & change over time?
• what changes were made over time?

[rlogothetis01]

Dependencies
Look at practicality (what is the problem we are solving)
Too many dependencies - impact on deployment etc.
DS pulls in as many dependencies to make things work
SE remove dependencies
Example: pandas may not be needed but is commonly pulled in in DS projects but can find ways to not include
Evolution aspect
From build to deployment
Dependencies bw ML micro services (cross boundary issues)