Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reevaluate mc6 model directionality flag #323

Open
madison-feshuk opened this issue Jan 13, 2025 · 3 comments
Open

Reevaluate mc6 model directionality flag #323

madison-feshuk opened this issue Jan 13, 2025 · 3 comments

Comments

@madison-feshuk
Copy link
Collaborator

madison-feshuk commented Jan 13, 2025

Review current logic for the model directionality questionable flag: https://github.com/USEPA/CompTox-ToxCast-tcpl/blob/dev/R/mc6_mthds.R#L78

Inactives are getting flagged due to 1 data point surpassing cutoff. Ex:
image
image

Consider updated logic to 1) filter so flag only applies to actives (where active is defined as absolute value of hitc >= 0.9) AND/OR 2) require minimum 2 data points surpassing cutoff in opposite direction of winning model top

@Kelly-Carstens-EPA
Copy link
Contributor

Kelly-Carstens-EPA commented Jan 13, 2025

The flag description currently describes: "Flag series if model directionality is questionable, i.e. if the winning model direction was opposite, more responses (resp) would have exceeded the cutoff (coff). If loss was winning directionality (top < 0), flag if count(resp < -1coff) < 2count(resp > coff). If gain was winning directionality (top > 0), flag if count(resp > coff) < 2count(resp < -1coff)."

The mc6 methods code does not reflect this definition. However, I think this is the intent of the flag and we should update the code to consider only responses that exceed cutoff in the 'opposite' direction (opposite sign from the top of the winning model). Presently, the code considers all response in the opposite direction, regardless of whether they exceed cutoff.

@ElizabethGilson
Copy link
Contributor

Out of endpoints that are a hit and have the model directionality flag the majority have a winning model of poly2. The percent break-down is as follows

  • poly2: 71.0%
  • exp2: 16.1%
  • exp5: 6.45%
  • poly1: 6.45%

Here are some examples plots of endpoints with the flag and with a winning model of poly2:
tcplPlotDirectionality3_11746628_11746628
tcplPlotDirectionality3_11750000_11750000

@Kelly-Carstens-EPA
Copy link
Contributor

Interesting examples. I have a couple questions about these endpoints. 1) Is this a biologically relevant gain-loss behavior? 2) If not, is this flag flagging something concerning about model fit? It seems like a good fit. I think this is flagged because the top of the model is positive (or maybe the it's negative if defined by max conc tested) and the majority of the responses are in the negative direction? It might be helpful to look into the sign of the top for these two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants