ch_framework.Rmd

# Framework

This paper ...

## Types of Questions to Ask

- Differences in documentation
- Differences in modeling assumptions
- Differences in default values
- Differences in available parameters 
- Edge cases
- Can default call in one language be replicated in another
    + If yes, do we have numerical equivalence within an acceptable level
- Can a common/historical specification in one language be replicated in another
    + If yes, do we have numerical equivalence within an acceptable level
    + If no, is the common/historical specification the statistically preferred method or should it be abandoned?
- Can the preferred specification in one language be replicated in another?
    + If yes, do we have numerical equivalence within an acceptable level
- Differences in available output statistics
- Differences ‘deep’ in the implementation which are the result of decisions developers make when implementing a particular algorithm
    + May not be clear in the documentation at the level exposed to the user
    
## Possible Solutions

- Select a method for the analysis plan – be transparent on known alternatives and known differences
- Are there multiple solutions within a given language to consider?