On the different "causal models" of SSVC and CVSS

[j---]

As raised by @jchester in his post:

The other area where SSVC arguably falls short is that it has a simpler implied causal model than CVSS does. In the critique by Risk Based Security discussed above, it was argued that CVSS was not detailed enough. But SSVC goes from CVSS's eight variables down to just four. It might be argued that three of CVSS's variables (Confidentiality, Integrity and Availability) can be usefully compressed into the SSVC Mission & Well-Being variable. That still leaves a substantial gap in the input permutations between the two schemes.

I think it is possible that within the context of this paragraph, different stakeholders have different "causal models."
If that is the case, SSVC and CVSS are not directly comparable. I'd like to explore if this is a reasonable basis to understand the "causal model" idea, because I did not so much think of SSVC as having a causal model. And at least, insofar as it does, the "stakeholder specific" thing means I think it might have multiple.

I think this might suggest some changes to the visual display of the calculator, also. In the section of the paper that talks about relationship to other systems and CVSS specifically, we sketch how technical impact is related to the CVSS v3 impact metrics. That the post relates the CVSS impact metrics to mission impact and safety impact indicates we need to either message this better or change our minds. One way to message better would be to pull the CVSS vector string in for vuls that have one and use our suggested mapping to make a suggestion for the technical impact value.
@sei-vsarvepalli , how hard would it be to pull the CVSS vector string values for C/I/A impact and Scope if a user enters a CVE-ID into the calculator?

[sei-vsarvepalli]

Hey @j---

I had noticed this till just now. It is very easy to pull CVSS vector strings. May be will do this once the open Bug tickets are fixed for the demo site.

Vijay

[jchester]

/CCing my dayjob alterego, @jchestershopify.

[j---]

@sei-vsarvepalli , let's handle the calculator engineering under #195 . This issue will be closed once the discussion of causal modes and the comparison to CVSS is handled in the PDF document / documentation.

[ahouseholder]

I converted this from an issue #186 to a discussion because it is not clear what change(s) is/are needed.

In particular, the causal model concept that @j--- mentions is unclear to me. While I am somewhat familiar with what Judea Pearl means by it, I'm missing a connection here.

[j---]

So in my STS work, I would roughly equate "causal model" to "model of the mechanistic explanation of how the vulnerability leads to a successful attack".
Mechanistic explanation would be fleshed out in what we did here: https://link.springer.com/article/10.1007/s13347-018-0329-z
I suppose one question is, if this question is important to discuss, then I would have to find some correct level of detail with which to justify the choice SSVC makes. I think the choice we made, FWIW, is that a coarse mechanistic explanation of how the vul leads to a successful attack was not only sufficient to make the actual decisions at hand but also that a coarser target level of explanation makes gather adequate evidence easier and automation easier to validate.
Is that sentence enough, or does it need to get fleshed out further? I don't think we have independently verified what the "best" level of coarseness in level of detail is for our goals. I'm not sure how we would study that?

[jchester]

If I had to pick a kind of causality I had in mind, it would be one of necessity and sufficiency. "These are the factors that must be true in order to achieve an exploitation" and "these factors being true alone mean an exploit is possible". But I didn't think of that explicitly, because I don't feel like I had a firm grasp on the structure of such a model for CVSS. It was more that CVSS has an unexamined or implicit such model in order to decide which factors to include or exclude; the same would go for SSVC.

[ahouseholder]

So based on your "factors in order to achieve exploitation" point, I wonder if you're describing a different problem than what we were going after with SSVC. (I can't really comment on what CVSS was trying to do.)

But here's what I mean:
SSVC isn't speculating as much on what the vulnerability does or doesn't do. It's not trying to structure data about the vulnerability. It's trying to inform specific decisions in specific contexts for specific stakeholders. It's looking at things we can know, and those depend on where you sit on the find/fix/deploy pipeline. So for a deployer, the decision is basically "deploy an available fix now or later", for a few different values of "later". We expect deployers to know more about

• the relationship of the affected software to the mission that it supports (human impact)
• the relationship of the affected software to the infrastructure in which it resides, (exposure)
• an adversary's work factor to leverage the vulnerability against you
  • is there an exploit public? is it already being exploited? (exploitation)
  • what's the ROI for attackers per successful exploitation (utility)

So really, only a few technical details of the vulnerability show up as part of utility, whereas everything else is either (in CVSS terms) environmental or temporal attributes that have nothing to do with the technical details of the vul. Also, at the deployer end of the pipeline, you can probably get the utility info from the vendor, because (if they're also using SSVC) they've already thought about that since:

For suppliers (vendors), who sit earlier in the fix deployment pipeline and are thus making a more speculative assessment of what might happen, we add in the technical impact (which is really about how much control an attacker gains from successful exploitation), and keep exploitation and utility. Vendors know less about deployment contexts so we don't ask them for as much detail there, instead just generalizing public safety impact. Vendors are also making a different decision: instead of "should I deploy an available patch now or later?", their question is "should I develop a patch now or later?"

So we're not trying to model an exploit developer's difficulty level. We're just saying "attackers have some level of incentive (utility), and some level of known capability (exploitation), and some level of control (technical impact) when they use that capability", and mapping that onto "and if they used it against you, it might affect you because (exposure, human impact, public safety impact)".

[j---]

The way that "necessary and sufficient" as the definition of causation gets problematized in the PhilSci literature is by asking the question "at what level of generality?". Necessary and sufficient for my specific machine today with me sitting at it will have different necessary and sufficient causes, in the details, because we may have ASLR that has put the vulnerable code in different memory locations and the attacker needs to know the memory location. They also might need to know our IP, and if I'm behind a NAT gateway, etc., and the specific necessary details are different across some very narrow level of generality. And then we have to ask "well how general is general enough to actually be useful but not so general as to be super vague?".
The structure of mechanistic explanation provides hooks to answer these questions (that's why I linked https://link.springer.com/article/10.1007/s13347-018-0329-z). But it's not the only way to answer those questions. I'm offering it in the spirit of making the design goals behind SSVC explicit, so we can talk about them and decide if they're adequate / better than CVSS's implicit ones / etc.
The way the mechanisms view answers "what is the right level of generality to describe the causes" is by saying it is based on the goal the modeler wants to achieve (basically the phenomenon being modeled in a scientific setting). SSVC is very explicit about it's goal being to help vulnerability managers make the actual decision they have to make in their job. Different stakeholders have different decisions -- compare defer/scheduled/etc for deployers versus publish/don't publish for coordinators. This explains why those different goals have different causal models -- that is, different decision points.
Assuming that vul managers have a secondary goal of being efficient with their time (not asking more questions than is necessary to actually make the decision within their organization's risk tolerance for accuracy), then more questions (metrics in the case of CVSS) is not necessarily more causal fidelity. It might just be noise.
Herb Simon's "Sciences of the Artificial" coins the term "satisficing" for this. SSVC is not looking for the optimal or most detailed causal description of how the vul works. It is looking for the first satisfactory description of how the vul works that meets the needs of the organization making the decision.

That description, with some polish, should perhaps go in the SSVCv2.1 doc just because it explains where we currently are. However, there should be further discussion as to whether that is the right place for SSVC to be as far as it's underlying causal model and goals for how that model is used.

[ahouseholder]

Is there a particular spot you had in mind for something like this to go?

[j---]

Section 3 briefly mentions the satisficing point, but I think it could go somewhere like the Scope section in https://github.com/CERTCC/SSVC/blob/main/doc/md_src_files/040_stakeholders-scope.md.
Causal importance is mentioned in the Reasoning Steps Forward paragraph, so maybe this could expand that subsection?

[ahouseholder]

Noting for situational awareness: after recent file refactoring, the scope section is now in https://github.com/CERTCC/SSVC/blob/main/doc/md_src_files/04_06_scope.md

[ahouseholder]

This is, for me at least, the most confusing discussion thread in this project (so far at least). I don't understand the initial point, nor do I understand the commentary that followed. As a result, I have no idea at all what the diff to the doc would look like that would resolve this thread. With no insult intended, it seems to me that if we have to appeal to PhilSci literature to explain why SSVC is simpler than CVSS then we've lost the plot.

Are we overthinking this? Is this just the point that "You don't need a megabyte of input to adequately model a two-bit output?"

[jeroenh]

I still have difficulty understanding what the initial point was with respect to "causal model".
What I do get from the rest of the discussion and my understanding of SSVC (and CVSS) that indeed SSVC is much simpler than CVSS to come to a decision. and a good summary of that point is indeed the point you mention ("You don't need a megabyte of input to adequately model a two-bit output?").

But then the point is: how much input do you need to find the appropriate vulnerability management decision?

And then, my impression is that SSVC is trying to address that problem explicitly. There are numerous decision points that can be taken into account, some can be defined in a strict way, and some depend on context. And then this document explains what that context is and tries to find ways to take that into account so that you can come to a reasonable decision.
Whereas CVSS uses some calculations to come to a "value" that seems scientific, but does not actually help you all that much with your vulnerability management decision. CVSS 4 is better at taking context into account, but still not in the way as SSVC does.

[jeroenh]

Also, with regards to the later point: SSVC does not aim to be complete, but rather aims to be practical. SSVC does not strive to be provably perfect, but give you the best decision most of the time.
And the SSVC model does give anyone involved in the process a better way of evaluating a decision should the outcome not have been correct in retrospect.
Whereas with CVSS that evaluation really hard to do.

[j---]

So, putting all my philosophy of science PhD aside.
I think the question amounts to, quite coarsely, "CVSS has more options, doesn't that make it better?"
The concrete change we might want to make in the SSVC documentation is to explain why having more options is not always better.
There are two answers here. One is sometimes simpler is better. The other is that SSVC is hiding some complexity in some places for people.

For the first, see https://en.wikipedia.org/wiki/Information_bias_(psychology) and https://en.wikipedia.org/wiki/Conjunction_fallacy and https://en.wikipedia.org/wiki/Precision_bias. There are also details about this in the PhilSci modeling literature if anyone cares, but these wikipedia summaries of common decision making errors are probably enough.

For the second, for example, although Automatable is just a yes/no answer, it has four component questions for each kill chain stage. Although 4 yes/no questions is less than the number of options in CVSS v3 "Exploitability metric group" in the base score (with attack vector, user interaction, privileges required, and attack complexity being 432*2 options), they require a similar amount of work. It's just that SSVC has a strong claim on what is important (adversary can automate exploitation events). This makes the apparent options (yes/no) look more simple than the work it takes to get there (answer all four kill chain steps). (CVSS v4 adds more options to this group)

This discussion could go in https://github.com/CERTCC/SSVC/blob/main/doc/md_src_files/03_representing_information.md, though I'm open to other options.

[jeroenh]

I like that idea.
I would also add that it is still an open question how much and which information you need to make the "best" decision in a practical sense. It is simply a design choice that SSVC chooses to do it in this way and it has lead to good results so far.

[j---]

It's a good point that we could / should include some more emphasis on the fact that SSVC asks folks to measure what counts as best for them and refine. Or at least, we're trying to do that at the SSVC spec level and also encouraging folks who can to do that both for themselves and to feed that information up as appropriate.