-
Notifications
You must be signed in to change notification settings - Fork 25
Benchmarking
We wanted to verify that Tellurium/Roadrunner is faster than competing simulations software. This page lists our results and description of our methods to test the speeds of Tellurium and others under different circumstances.
We did most of our testing with a rather large model that would put adequate stress on the software, namely this model by Millard from 2016 of E. coli
As with any GUI tool, explaining how to navigate it can be difficult. Here I’ll explain the significance of some of the important features, how to run a simulation, and get the output to benchmark the speed of COPASI
There are four sections to COPASI: Model, Tasks, Output Specifications, and Functions. Since we are really just benchmarking by importing SBML, the last section isn’t super important. On to the other three!
Okay so if you really are just doing benchmarking this one isn’t super important either, since it’ll be all set up for you by the SBML. There is actually a really nice video on this section by the COPASI people on their website, so go check it out
This is where you tell COPASI how you want it to be modeled. Fair warning, I’m not sure what most of this stuff does in here, but I got benchmarking to work, so I’ll explain that. Under tasks it says "Time Course". As I understand it, this is the equiv- alent of Tellurium’s simulate(). On here you can specify the duration of the test, and most importantly, how many intervals. In the bottom right corner there are two buttons "Report" and "Output Assistant" For benchmarking, "Report" is most important. You will first have to create an approriate benchmarking report template, which is covered in the next section. After you have your report template, you can click the "Report" button and select it in the drop-down menu. After that, specify where you want it to be saved, and if you want it to append to the file or write over it. When you’re setting up the tests, it makes sense to leave "Append" unchecked because some of the tests may not be valid e.g. incorrect interval ammount, etc. However, it can be a handy tool if you want to run several tests and then process them all to get a distribution of times.
The most important part of this section is to create a benchmarking re- port template, so click on "Report Templates" under Output Specifications. There should be a "New Report" box under the rest of the report templates. Click on it and give it an approriate name like "Benchmark" Now it will appear in the hierarchy under "Report Templates". Click on it, and you will see a button labeled "Item" to the left. This will add the columns to your output template. First, we want the real time so that we can benchmark it, so click on the time section->"real time". It should be placed at the top of table meaning that it will be the leftmost column You can add other columns, like the species to be able to verify that the test went correctly compared to other tests.
This is a MATLAB extension, and although it has a GUI that can be used to create and test models, we need to use the console interface to benchmark it.
The timing tools we are using here are MathWorks' own commands tic and toc. Tic starts the timer, and toc displays the current time.
Another useful MATLAB tool is linspace(start,end,n) which generates a vector of n items starting from start
until end evenly spaced
By default when you run a statement on the MATLAB shell, it will printout the value. It turns out that IO is very slow and skews timing results, especially on programs that run for less than a second. To suppress IO in MATLAB, end a statement with a semicolon
sbiosimulate(modelObj) % Prints out everything
sbiosimulate(modelObj); % Does not printSim Biology has a few important concepts, some of which are borrowed from another MathWorks modeling tool: Simulink.
The Model Object represents the species and reactions. It can be created in the GUI, in code, or by loading a file. We can load our SBML in with the sbmlimport command which will return to us the model object. Note that sbmlimport takes a character vector denoted by the apostrophes, not the quotation marks, which create a string.
modelObj = sbmlimport('C:\millard.xml')The Configuration Set is just as it sounds, a set of options that determines how the model will be performed e.g. start time, end time, integrator type, etc. The configuration set can be obtained from the model, and any changes to it will automatically be seen by the model.
cs = getConfigSet(modelObj, 'active')