RFE: Create combined logs

[redsaz]

Some of the reports have a long list of logs to compare. Some of the logs are repeated runs of certain settings. For example:

• log A, B, and C all ran against version 1.0.0 of a web app. This is the control (or baseline) group.
• log D, E, and F all ran against version 2.0.0 of a web app. This is an experimental group and gets compared against the baseline group.
• log G, H, and I all ran against version 2.0.0 of a web app, but with different settings toggled. This is another experimental group and also gets compared against the baseline group.

In this scenario, I run these multiple tests because the environment the tests run in are just unstable enough that just one run from each group is not statistically significant enough to determine if performance improved, stayed the same, or got worse.

In a difference scenario, sometimes I need to be able to run multiple instances of jmeter at the same time, but not using the client/server mode of jmeter, which creates multiple jtl files. Sure, I could combine these multiple jtl files myself before I upload them to lognition, or have a script do it, but it'd be nice if I could upload these independent logs and then somehow indicate I'd like a single combined log of all those logs, and use that combined log in the reports.

In a THIRD scenario, sometimes I need to do both! That is, I run multiple instances of jmeter at the same time for one test run. Then, some time later, I run multiple instance of jmeter at the same time for another test run. And repeat, until I'm satisfied I made enough runs.

So, instead of only being able to import logs, I'd like to be able to create what are called "combined logs", a.k.a. "merged logs", a.k.a. "composite logs", a.k.a "unified logs", whatever. These combined logs are similar to reports in that they use selectors on the logs in order to determine which logs get combined into the one log. This combined log has all of the features of a normal log:

• I can view histograms, percentiles, all normal statistics.
• All of the percentiles are TRUE percentiles (not averaged percentiles like seen elsewhere)
• the "over time" graphs still work (except in the case of traffic over time, see notes below
• the log has its own tag list, so that I can include it in reports like I would any other log.
• the log, even though it was created in combination of other logs, can be included in another combined log.

This last point may be worth a different RFE, regarding how timestamps and sample counts over time are combined. That is, take the scenarios above:

1. Several runs of the jmeter test are run sequentially, with an irregular and possibly lengthy gap of time between these runs (maybe only a couple minutes of gap, but also maybe a few hours, days, or weeks of gap).
2. Several jmeter tests are run at the same time
3. Several jmeter tests run at the same time. This can be repeatedly, with potentially the same gaps inbetween as scenario 1.

So, for scenario 1:

• When I look at the "over time" graphs, I don't want to see all these gaps. Instead, I'd like them all to be "relativized" so that the first sample from each log starts at timestamp 0.
• When I look at the total number of samples, broken down by call type, I want to see the averaged count of samples. Example: for GET /example call type, log A made 999 samples for the entire run, log B made 1000 samples for the entire run, and log C made 1001 samples for the entire run. I want to see the average value across all of these be reported: 1000 samples. This is because I'm doing these multiple runs to smooth out some of the variance I see in my testing environment.
• When I look at the "traffic over time" graph, I want to see the averaged traffic per timeslice, NOT the total traffic per timeslice. (That is, say in the first 5 second window, log A has 99 samples, log B has 100 samples, and log C has 101 samples. I would want this timeslice to show up in the graph as the average, which is 100, NOT the total, which is 300.) This is for the same reason as the above point.

For scenario 2:

• When I look at the "over time" graphs, since I'm running multiple jmeter instances at the same time, I don't want to use the "relativized" time, but an absolute time. This is because I perhaps didn't start some of my jmeter instances at the same time as the other jmeter instances (possibly to stagger the load against the app-under-test.)
• When I look at the total number of samples, broken down by call type, I want to see the total samples. That is, all of the samples of that call type, across all of the logs, are added together to get this number. Not the average. For example, for GET /example call type, log A made 999 samples for the entire run, log B made 1000 samples for the entire run, and log C made 1001 samples. I want to see the total across all those logs to be reported: 3000 samples. This is because all these logs are a portion of a single run, and so the entire traffic needs taken into account.
• When I look at the "traffic over time" graph, I want to see the total traffic per timeslice, not the average. This is for the same reason as the above point.

For scenario 3:

• This can be solved by creating a "combined log" for each run. Then, I can create a "combined log" that is comprised of the "combined logs".
• We'd need to make sure that we cannot create cyclic combined logs of combined logs. For example, combined-log Z is a combination of combined-log Y and combined-log X. Combined-log X is a combination of log c, log d, and combined-log Z. Because Z needs X in order to be computed, and X needs Z in order to be computed, nothing can happen.
• Combined logs probably shouldn't combine the same logs more than once, right? For example, combined-log X is a combination of log A, log B, and log C. Combined-log Y is a combination of log B, log C, and log D. Combined-log Z is a combination of combined-log X and combined-log Y. Because X and Y both included log B and log C, we wouldn't want to double count the entries from those logs.

I'm also not sure how best to present this option of "relatively combine log entries and average traffic", or "combine log entries by absolute timestamp and use total traffic" to the end-user.

I also don't know if this should be done like how reports are currently done, in that the combined log uses tag selectors to choose which logs to combine, and will automatically update whenever a new log comes in. Seems like it is more confusing than anything. Maybe it is better if the user must explicitly pick each log to combine? Or could we do both? Sounds like another RFE.