Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: Use inner/outer splitter, analogize to zip/product #773

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

effigies
Copy link
Contributor

Additional minor wording changes for clarity.

@effigies effigies requested review from tclose and djarecka March 28, 2025 19:18
@tclose
Copy link
Contributor

tclose commented Mar 31, 2025

I think the only issue with this description is that @djarecka was using "inner splitter" to refer to the split that occurs on a specific node and then combined on that node (i.e. not propagated to downstream nodes). So there may be places in the comments and variable names where it could get a little confusing

@tclose
Copy link
Contributor

tclose commented Mar 31, 2025

I was thinking that we could do a search/replace for inner and replace it with internal and then use inner to refer to the scalar/inner/zip split you are suggesting here. Could still be a little confusing but since "internal" would only be used internally (i.e. not exposed in the public API), it is probably ok if it is clearly explained in the glossary

@tclose
Copy link
Contributor

tclose commented Mar 31, 2025

Actually looking through the use of inner splitter, it just seems to refer to a split of a node within a workflow. @djarecka is that right?

@tclose tclose changed the base branch from develop to master April 1, 2025 01:00
Additional minor wording changes for clarity.
Copy link

codecov bot commented Apr 1, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.66%. Comparing base (455e16f) to head (d962cc5).
Report is 3 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #773      +/-   ##
==========================================
+ Coverage   88.64%   88.66%   +0.02%     
==========================================
  Files          85       85              
  Lines       17719    17719              
  Branches     1538     1538              
==========================================
+ Hits        15707    15711       +4     
+ Misses       1644     1640       -4     
  Partials      368      368              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@effigies
Copy link
Contributor Author

effigies commented Apr 1, 2025

It's possible that I've misunderstood previous conversations, but I don't recall "inner splitter" ever referring to something internal to a node or a workflow, so much as an analogy to an inner-product / inner-join.

I would generally expect most splits to be combined before the final node in a workflow, which is frequently the way it's done with iterables (and always with MapNodes) in Nipype 1, but that doesn't seem like a thing that really needs a particular name. If we do, I would probably just figure out how we want to describe nodes in terms of their modification of the splitting state; workflows are just nodes, and the terminology can be used in both cases.

To throw out some terms:

  1. "Splitting" node/workflow: A node or workflow that strictly expands the state space without combining. It may split and combine internally, so long as the end result does not combine over pre-existing state variables.
  2. "Combining" node/workflow: A node or workflow that strictly contracts the state space without splitting. It may split and combine internally, so long as the end result does not split over new state variables.
  3. "Reshaping" node/workflow: A node or workflow that modifies the state space, splitting over at least one new variable and combining over at least one pre-existing state variable. This is a catch-all for nodes that are neither purely splitting nor combining.

Examples (apologies if this gets the new syntax wrong):

# Splitting node, splits over x
a = Task1().split(x=xList, y=yList).combine('y')

# Combining node, combines over x
b = Task2(var1=a.out).split(var2=var2List).combine('var2', 'a.x')

# Reshaping node, combines over x, splits over z
c = Task3(var1=a.out).split(x=zList).combine('a.x')

The idea is that what matters is the impact on the state space for downstream nodes, not the entire sequence of manipulations that only that node and the workflow writer ever see.


Anyway, that's getting off-topic for this PR. @satra @djarecka Would be good to have your understanding of how "scalar"/"inner" have been used.

@satra
Copy link
Contributor

satra commented Apr 1, 2025

but I don't recall "inner splitter" ever referring to something internal to a node or a workflow, so much as an analogy to an inner-product / inner-join.

perhaps semantics are at play here and inner and nested splits are being conflated. if i remember somewhat i believe that nested splits (i.e. any level of splits set inside a workflow), need to be combined before one steps out of a workflow. i cannot remember now how a workflow (as a cacheable task object) deals with splits inside it (whether inner/outer), and especially if the workflow also has splits set on it's inputs. there were certain circumstances under which an output state could not be easily generated/tracked, and we forced that combines needed to happen. i think it was this nested condition.

@tclose tclose added the to consider suggesting changes that require more discussion label Apr 2, 2025
@tclose
Copy link
Contributor

tclose commented Apr 7, 2025

@djarecka would you be happy with these semantics if we did a search replace on "inner" and replaced it with "internal" to remove references to the old nomenclature?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
to consider suggesting changes that require more discussion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants