v3.2 User Guide

Configuration
1.1. Configuration Syntax
1.2. CLI Arguments Aliasing
1.3. Configuration Pattern Values
Items
2.1. Item Types
2.1.1. Data Items
2.1.1.1. Fixed Size Data Items
2.1.1.1.1. Empty Data Items
2.1.1.1.2. Small Data Items (1B-100KB)
2.1.1.1.3. Intermediate Size Data Items (100KB-10MB)
2.1.1.1.4. Big Data Items (10MB-100MB)
2.1.1.1.5. Very Big Data Items (100MB-10GB)
2.1.1.1.6. Huge Data Items (10GB-1TB)
2.1.1.1.7. Ultimate Data Items (>=1TB)
2.1.1.2. Random Size Data Items
2.1.1.3. Biased Random Size Data Items
2.1.2. Path Items
2.1.3. Token Items
2.2. Items Input
2.2.1. Items Input File
2.2.2. Items Path Listing Input
2.2.3. New Items Input
2.2.3.1. Random Item Ids
2.2.3.2. Ascending Item Ids
2.2.3.3. Descending Item Ids
2.2.3.4. Items Id Prefix
2.2.3.5. Items Id Radix
2.2.3.6. Items Id Offset
2.2.3.7. Items Id Length
2.3. Items Output
2.3.1. Items Output Delay
2.3.2. Items Output File
2.3.3. Items Destination Path
2.3.3.1. Constant Items Destination Path
2.3.3.2. Pattern Items Destination Path
Content
3.1. Uniform Random Data Payload
3.2. Payload From the External File
Concurrency
4.1. Default Concurrency Level (1)
4.2. Small Concurrency Level (2-10)
4.3. Medium Concurrency Level (11-100)
4.4. High Concurrency Level (101-1K)
4.5. Very High Concurrency Level (1K-10K)
4.6. Huge Concurrency Level (10K-100K)
4.7. Ultimate Concurrency Level (100K-1M)
Circularity
Load Jobs
6.1. Load Jobs Naming
6.2. Load Jobs Limitation
6.2.1. Load Jobs Are Infinite by Default
6.2.2. Limit Load Job by Processed Item Count
6.2.3. Limit Load Job by Rate
6.2.4. Limit Load Job by Processed Data Size
6.2.5. Limit Load Job by Time
6.2.6. Limit Load Job by End of Items Input
Metrics Reporting
7.1. Metrics Periodic Reporting
7.2. Metrics Reporting is Suppressed for the Precondition Jobs
7.3. Metrics Reporting Triggered by Load Threshold
7.4. I/O Traces Reporting
Load Types
8.1. Noop
8.2. Create
8.2.1. Create New Items
8.2.2. Copy Mode
8.3. Read
8.3.1. Read With Disabled Validation
8.3.2. Read With Enabled Validation
8.3.3. Partial Read
8.3.3.1. Random Byte Ranges Read
8.3.3.1.1. Single Random Byte Range Read
8.3.3.1.2. Multiple Random Byte Ranges Read
8.3.3.2. Fixed Byte Ranges Read
8.3.3.2.1. Read First N bytes
8.3.3.2.2. Read Last N bytes
8.3.3.2.3. Read Bytes from N1 to N2
8.3.3.2.4. Read Multiple Fixed Ranges
8.4. Update
8.4.1. Update by Overwrite
8.4.2. Random Ranges Update
8.4.2.1. Single Random Range Update
8.4.2.2. Multiple Random Ranges Update
8.4.3. Fixed Ranges Update
8.4.3.1. Overwrite First N bytes
8.4.3.2. Overwrite Last N bytes
8.4.3.3. Overwrite Bytes from N1 to N2
8.4.3.4. Append
8.4.3.5. Multiple Fixed Ranges Update
8.5. Delete
Scenarios
9.1. Scenarios Syntax
9.2. Default Scenario
9.3. Custom Scenario File
9.4. Job Configuration in the Scenario
9.4.1. Override Default Configuration in the Scenario
9.4.2. Job Configuration Inheritance
9.4.3. Reusing The Items in the Scenario
9.4.4. Environment Values Substitution in the Scenario
9.5. Scenario Job Types
9.5.1. Shell Command Job
9.5.1.1. Blocking Shell Command Job
9.5.1.2. Non-blocking Shell Command Job
9.5.2. Load Job
9.5.3. Precondition Load Job
9.5.4. Parallel Job
9.5.5. Sequential Job
9.5.6. Loop Job
9.5.6.1. Loop by Count
9.5.6.2. Loop by Range
9.5.6.3. Loop by Sequence
9.5.6.4. Infinite Loop
9.5.7. Mixed Job
9.5.7.1. Separate Configuration in the Mixed Job
9.5.7.2. Weighted Job
9.5.8. Chain Job
9.5.8.1. Separate Configuration in the Chain Job
9.5.8.1.2. Delay Between Operations in the Chain Job
Storage Driver
10.1. Distributed Storage Drivers
10.1.1. Single Local Separate Storage Driver Service
10.1.2. Two Local Separate Storage Driver Services (at different ports)
10.1.3. Single Remote Storage Driver Service
10.1.4. Two Remote Storage Driver Services
10.1.5. Large Count of Remote Storage Driver Services
10.2. Preparing the Storage
10.2.1. Auth Token Precondition Hook
10.2.2. Destination Path Precondition Hook
10.3. Filesystem Storage Driver
10.4. Network Storage Driver
10.4.1. Node Balancing
10.4.2. SSL/TLS
10.4.3. Socket Timeout
10.4.4. I/O Buffer Size Adjustment for Optimal Performance
10.4.5. HTTP Storage Driver
10.4.5.2. Atmos
10.4.5.2.1. Authentication
10.4.5.2.2. Filesystem access
10.4.5.3. S3
10.4.5.3.1. Authentication
10.4.5.3.2. Filesystem access
10.4.5.3.3. Versioning
10.4.5.3.4. Multipart Upload
10.4.5.4. Swift
10.4.5.4.1. Authentication
10.4.5.4.2. Versioning
10.4.5.4.3. Create Dynamic Large Objects

1. Configuration

1.1. Configuration Syntax

How to:

TODO

Tests coverage:

com.emc.mongoose.tests.unit.ValidateConfigTest

1.2. CLI Arguments Aliasing

How to:

TODO

Tests coverage:

com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest

1.3. Configuration Pattern Values

How to:

Dynamic HTTP headers with generated values:

java -jar <MONGOOSE_DIR>/mongoose.jar --storage-net-http-headers-myOwnHeaderName=MyOwnHeaderValue\ %d[0-1000]\ %f{###.##}[-2--1]\ %D{yyyy-MM-dd'T'HH:mm:ssZ}[1970/01/01-2016/01/01]

Or "variable" files output path:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-output-path=/mnt/storage/%p\{16\;2\} --storage-type=fs ...

Tests coverage:

TODO

2. Items

2.1. Item Types

2.1.1. Data Items

The data items type is used by default.

2.1.1.1. Fixed Size Data Items

Fixed data item size is used by default.

2.1.1.1.1. Empty Data Items

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=0

Tests coverage:

TODO

2.1.1.1.2. Small Data Items (1B-100KB)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=1

Tests coverage:

com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadBucketListingTest com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest

2.1.1.1.3. Intermediate Size Data Items (100KB-10MB)

The data items size of 1MB is used by default.

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=100KB

Tests coverage:

com.emc.mongoose.tests.system.CreateNoLimitHighConcurrencyTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

2.1.1.1.4. Big Data Items (10MB-100MB)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=10MB

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.ReadBigDataItemsTest

2.1.1.1.5. Very Big Data Items (100MB-10GB)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=1GB

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsMetricsThresholdTest

2.1.1.1.6. Huge Data Items (10GB-1TB)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=1TB

Tests coverage:

TODO

2.1.1.1.7. Ultimate Data Items (>=1TB)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=1PB

Tests coverage:

TODO

2.1.1.2. Random Size Data Items

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=5MB-15MB

Tests coverage:

TODO

2.1.1.3. Biased Random Size Data Items

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-size=0-100MB,0.2

Tests coverage:

TODO

2.1.2. Path Items

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-type=path

Tests coverage:

TODO

2.1.3. Token Items

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-type=token

Tests coverage:

TODO

2.2. Items Input

Items input is a source of the items which should be used to perform the operations (crete/read/etc). The items input may be a file or a path which should be listed.

2.2.1. Items Input File

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-input-file=<PATH_TO_ITEMS_FILE> ...

Tests coverage:

com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

2.2.2. Items Path Listing Input

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-input-path=/bucket1 ...

Tests coverage:

com.emc.mongoose.tests.system.ReadBucketListingTest com.emc.mongoose.tests.unit.BufferingInputBaseTest

2.2.3. New Items Input

2.2.3.1. Random Item Ids

Random item ids are used by default. The collision probability is highly negligible (2^-63-1).

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CreateByTimeTest

2.2.3.2. Ascending Item Ids

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-naming-type=asc ...

Tests coverage:

TODO

2.2.3.3. Descending Item Ids

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-naming-type=desc ...

Tests coverage:

TODO

2.2.3.4. Items Id Prefix

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-naming-prefix=item_ ...

Tests coverage:

TODO

2.2.3.5. Items Id Radix

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-naming-radix=10 ...

Tests coverage:

TODO

2.2.3.6. Items Id Offset

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-naming-offset=12345 ...

Tests coverage:

TODO

2.2.3.7. Items Id Length

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-naming-length=13 ...

Tests coverage:

TODO

2.3. Items Output

2.3.1. Items Output Delay

The processed items info may be output with a specified delay. This may be useful to test a storage replication using the "chain" job (see the scenario job types for details). The configured delay is in seconds.

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-output-delay=60

Tests coverage:

TODO

2.3.2. Items Output File

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-output-file=items.csv

Tests coverage:

com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

2.3.3. Items Destination Path

2.3.3.1. Constant Items Destination Path

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-output-path=/bucketOrContainerOrDir

Tests coverage:

com.emc.mongoose.tests.system.ReadBucketListingTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

2.3.3.2. Pattern Items Destination Path

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-output-path=/mnt/storage/%p\{16\;2\} ...

Tests coverage:

TODO

3. Content

While creating/verifying/updating the data items Mongoose is able to use different content sources. By default it uses the memory buffer filled with random data. Also Mongoose is able to fill this content source buffer with a data from any external file.

3.1. Uniform Random Data Payload

The uniform random data payload is used by default.

Tests coverage:

TODO

3.2. Payload From the External File

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-data-content-file=<PATH_TO_CONTENT_FILE>

Tests coverage:

TODO

4. Concurrency

4.1. Default Concurrency Level (1)

The concurrency equal to 1 is used by default.

Tests coverage:

com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

4.2. Small Concurrency Level (2-10)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-concurrency=10

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest

4.3. Medium Concurrency Level (11-100)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-concurrency=100

Tests coverage:

com.emc.mongoose.tests.system.CreateByTimeTest

4.4. High Concurrency Level (101-1K)

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-concurrency=1000

Tests coverage:

com.emc.mongoose.tests.system.CreateNoLimitHighConcurrencyTest com.emc.mongoose.tests.system.ReadBucketListingTest

4.5. Very High Concurrency Level (1K-10K)

Note:

System's max open files limit may be required to increased to use high concurrency levels:
ulimit -n 10000

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-concurrency=10000

Tests coverage: manual testing only is applicable.

4.6. Huge Concurrency Level (10K-100K)

Note:

System's max open files limit may be required to increased to use high concurrency levels:
ulimit -n 100000

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-concurrency=100000

Tests coverage: manual testing only is applicable.

4.7. Ultimate Concurrency Level (100K-1M)

Note:

System's max open files limit may be required to increased to use high concurrency levels:
ulimit -n 1048576

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-concurrency=1000000

Tests coverage: manual testing only is applicable.

5. Circularity

"Circularity" forces the load job to recycle the I/O tasks executing them again and again. It may be useful to perform read/update/append the objects/files multiple times each.

Note:

The circularity feature is applicable to read and update load types only.

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-circular

Tests coverage:

com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest

6. Load Jobs

6.1. Load Jobs Naming

By default Mongoose generates the load job name containing the timestamp. The load job name is used as the output log files parent directory name. It may be useful to override the default load job name with a descriptive one.

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-job-name=myTest1

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

6.2. Load Jobs Limitation

6.2.1. Load Jobs Are Infinite by Default

A load job tries to execute eternally if it is possible.

Tests coverage:

com.emc.mongoose.tests.system.CreateNoLimitHighConcurrencyTest

6.2.2. Limit Load Job by Processed Item Count

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-limit-count=1000

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

6.2.3. Limit Load Job by Rate

It may be useful to limit the load job's rate by a max number of operations per second. The rate limit value may be a real number, for example 0.01 (op/s).

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-limit-rate=1234.5

Tests coverage:

com.emc.mongoose.tests.unit.RateThrottleTest

6.2.4. Limit Load Job by Processed Data Size

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-limit-size=100GB

Tests coverage:

com.emc.mongoose.tests.system.ReadBigDataItemsTest

6.2.5. Limit Load Job by Time

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-limit-time=15m

Tests coverage:

com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest

6.2.6. Limit Load Job by End of Items Input

Any load job configured with the valid items input should finish (at most) when all the items got from the input are processed (copied/read/updated/deleted). This is true only if load job is not configured to recycle the I/O tasks (circularity feature is disabled).

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-input-[file|path]=<INPUT_FILE_OR_PATH> ...

Tests coverage:

com.emc.mongoose.tests.system.ReadBigDataItemsTest

7. Metrics Reporting

7.1. Metrics Periodic Reporting

The default time interval between the metric outputs is 10s. This value may be changed.

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-metrics-period=1m

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest

7.2. Metrics Reporting is Suppressed for the Precondition Jobs

How to:

TODO

Tests coverage:

TODO

7.3. Metrics Reporting Triggered by Load Threshold

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --load-metrics-threshold=0.95

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest

7.4. I/O Traces Reporting

There's an ability to log the info about every I/O operation been executed versus a storage. This kind of info is called "I/O trace".

How to:

Edit the config/logging.json json configuration file, find the following section near the line # ~324:

				{
					"name": "ioTraceFile",
					"type": "loadJobFile",
					"fileName": "io.trace.csv",
					"PatternLayout": {
						"header": "StorageNode,ItemPath,IoTypeCode,StatusCode,ReqTimeStart[us],Duration[us],RespLatency[us],DataLatency[us],TransferSize\n",
						"pattern": "%m"
					},
					"Filters": {
						"Filter": [
							{
								"type": "MarkerFilter",
								"marker": "ioTrace",
								"onMatch": "NEUTRAL",
								"onMismatch": "DENY"
							},
							{
								"type": "ThresholdFilter",
								"level": "INFO",
								"onMatch": "ACCEPT",
								"onMismatch": "DENY"
							}
						]
					}
				}

Find out the "ThresholdFilter" sub section and change the level value on the next line from "INFO" to "DEBUG".

Tests coverage:

com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadBigDataItemsTest

8. Load Types

8.1. Noop

The "dry run" operation type. Does everything except actual storage I/O. May be useful to measure the Mongoose's internal performance.

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --noop

8.2. Create

Create load type is used by default. The behavior may differ on the other configuration parameters.

8.2.1. Create New Items

"Create" performs writing new items to a storage by default.

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

8.2.2. Copy Mode

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --item-input-[file|path]=<INPUT_FILE_OR_PATH> --item-output-path=/bucketOrDir

Tests coverage:

TODO

8.3. Read

8.3.1. Read With Disabled Validation

Read load jobs doesn't perform a content validation by default.

Tests coverage:

com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest

8.3.2. Read With Enabled Validation

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --read --item-data-verify ...

Tests coverage:

com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.ReadBucketListingTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

8.3.3. Partial Read

8.3.3.1. Random Byte Ranges Read

8.3.3.1.1. Single Random Byte Range Read

How to:

TODO

Tests coverage:

com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest

8.3.3.1.2. Multiple Random Byte Ranges Read

How to:

8.4.1. Update by Overwrite

How to:

9. Scenarios

9.1. Scenarios Syntax

There are a JSON schema file in the distribution: scenario/scenario-schema.json. An user may automatically validate the scenarios using this schema. This should help to write one's own custom scenario.

How to:

TODO

Tests coverage:

com.emc.mongoose.tests.unit.ValidateScenariosTest

9.2. Default Scenario

By default Mongoose tries to execute the so called default scenario file (located at <MONGOOSE_DIR>scenario/default.json). This scenario contains onyl a single load job.

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.CreateNoLimitHighConcurrencyTest com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.ReadBucketListingTest com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest

9.3. Custom Scenario File

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --scenario-file=myCustomScenario.json

Tests coverage:

com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

9.4. Job Configuration in the Scenario

9.4.1. Override Default Configuration in the Scenario

How to:

{
   "type" : "load",
   "config" : {
      // here are the configuration hierarchy
   }
}

Tests coverage:

com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest

9.4.2. Job Configuration Inheritance

How to:

{
   "type" : "sequential",
   "config" : {
      // the configuration specified here will be inherited by the container elements
   },
   "jobs" : [
      {
         "type" : "load",
         ...
      }
      ...
   ]
}

Tests coverage:

TODO

9.4.3. Reusing The Items in the Scenario

How to:

{
   "type" : "sequential",
   "jobs" : [
      {
         "type" : "precondition",
         "config" : {
            "item" : {
               "output" : {
                  "file" : "items.csv"
               }
            }
            ...
         }
      }, {
         "type" : "",
         "config" : {
            "item" : {
               "input" : {
                  "file" : "items.csv"
               }
            }
            ...
         }
      }
   ]
}

Tests coverage:

com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest

How to:

TODO

Tests coverage:

9.4.4. Environment Values Substitution in the Scenario

How to:

TODO

Tests coverage:

TODO

9.5. Scenario Job Types

9.5.1. Shell Command Job

9.5.1.1. Blocking Shell Command Job

How to:

Sleep between the jobs for example:

{
   "type" : "sequential",
   "config" : {
      // shared configuration values inherited by the children jobs
   },
   "jobs" : [
      {
         "type" : "load",
         "config" : {
            // specific configuration for the 1st load job
         }
      }, {
         "type" : "command",
         "value" : "sleep 5s"
      }, {
         "type" : "load",
         "config" : {
            // specific configuration for the 2nd load job
         }
      }
   ]
}

Tests coverage:

TODO

9.5.1.2. Non-blocking Shell Command Job

How to:

{
   "type" : "command",
   "value" : "find /",
   "blocking" : false
}

Tests coverage:

TODO

9.5.2. Load Job

How to:

TODO

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadBucketListingTest com.emc.mongoose.tests.system.ReadSmallDataItemsMetricsThresholdTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

9.5.3. Precondition Load Job

How to:

{
   "type" : "precondition",
   "config" : {
      // here are the configuration hierarchy
   }
}

Tests coverage:

com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest

9.5.4. Parallel Job

How to:

{
   "type" : "parallel",
   "jobs" : [
      {
         "type" : "",
         ...
      }, {
         "type" : "",
         ...
      }
      ...
   ]
}

Tests coverage:

TODO

9.5.5. Sequential Job

How to:

{
   "type" : "sequential",
   "jobs" : [
      {
         "type" : "",
         ...
      }, {
         "type" : "",
         ...
      }
      ...
   ]
}

Tests coverage:

com.emc.mongoose.tests.system.CircularAppendTest com.emc.mongoose.tests.system.CircularUpdateTest com.emc.mongoose.tests.system.ReadCircularUpdatedTest com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomRangesTest com.emc.mongoose.tests.system.ReadSingleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleFixedRangesTest com.emc.mongoose.tests.system.ReadUpdatedMultipleRandomRangesTest

9.5.6. Loop Job

9.5.6.1. Loop by Count

How to:

{
    "type" : "for",
    "value" : 10,
    "jobs" : [
        {
            "type" : "load"
        }
    ]
}

Tests coverage:

TODO

9.5.6.2. Loop by Range

How to:

{
    "type" : "for",
    "value" : "i",
    "in" : "2.71828182846-3.1415926,0.1",
    "jobs" : [
        {
            "type" : "command",
            "value" : "echo ${i}"
        }
    ]
}

Tests coverage:

TODO

9.5.6.3. Loop by Sequence

How to:

{
   "type" : "for",
   "value" : "concurrency",
   "in" : [
      1, 10, 100, 1000, 10000, 100000
   ],
   "config" : {
      "load" : {
         "concurrency" : "${concurrency}"
      }
   },
   "jobs" : [
      {
            "type" : "load"
      }
   ]
}

Tests coverage:

TODO

9.5.6.4. Infinite Loop

How to:

{
   "type" : "for",
   "jobs" : [
      {
            "type" : "load"
      }
   ]
}

9.5.8.1. Separate Configuration in the Chain Job

How to:

The delay in the example below is 1 minute. Minimum configurable delay is 1 second.

{
	"type" : "chain",
	"config" : [
		{
			"item" : {
				"output" : {
					"delay" : "1m",
					"path" : "/default"
				}
			},
			"load" : {
				"metrics" : {
					"trace" : {
						"itemInfo" : true,
						"reqTimeStart" : true,
						"duration" : true
					}
				}
			}
		},
		{
			"load" : {
				"type" : "read"
			}
		}
	]
}

Tests coverage:

TODO

9.5.8.1.2. Delay Between Operations in the Chain Job

How to:

{
	"type" : "chain",
	"config" : [
		{
			"item" : {
				"output" : {
					"delay" : "1m",
					"path" : "/default"
				}
			},
			"storage" : {
			    "net" : {
                    "node" : {
                        "addrs" : [
                            "10.123.45.67",
                            "10.123.45.68",
                            "10.123.45.69",
                            "10.123.45.70"
                        ]
                    }
                }
			}
		},
		{
			"load" : {
				"type" : "read"
			},
            "storage" : {
                "net" : {
                    "node" : {
                        "addrs" : [
                            "10.234.56.78",
                            "10.234.56.79",
                            "10.234.56.80",
                            "10.244.56.81"
                        ]
                    }
                }
            }
		}
	]
}

Tests coverage:

TODO

10. Storage Driver

Tests coverage:

com.emc.mongoose.tests.system.ReadSingleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleRandomFileRangesTest com.emc.mongoose.tests.system.ReadMultipleFixedFileRangesTest

10.4. Network Storage Driver

10.4.1. Node Balancing

How to:

TODO

Tests coverage:

TODO

10.4.2. SSL/TLS

How to:

java -jar <MONGOOSE_DIR>/mongoose.jar --storage-net-ssl --storage-net-node-port=9021 ...

Tests coverage:

TODO

10.4.3. Socket Timeout

How to:

TODO

Tests coverage:

TODO

10.4.4. I/O Buffer Size Adjustment for Optimal Performance

How to:

TODO

Tests coverage:

com.emc.mongoose.tests.system.CreateBigDataItemsTest com.emc.mongoose.tests.system.CreateByTimeTest com.emc.mongoose.tests.system.ReadBigDataItemsTest com.emc.mongoose.tests.system.ReadBucketListingTest

10.4.5. HTTP Storage Driver

10.4.5.2. Atmos

10.4.5.2.1. Authentication

How to:

TODO

Tests coverage:

TODO

10.4.5.2.2. Filesystem access

How to:

10.4.5.4.1. Authentication

How to:

TODO

Tests coverage:

TODO

10.4.5.4.2. Versioning

How to:

TODO

Tests coverage:

TODO

10.4.5.4.3. Create Dynamic Large Objects

How to:

TODO

Tests coverage:

TODO

About

Documentation

Overview
Deployment
User Guide
Troubleshooting
Reference
- API
  - Extensions
  - Monitoring
  - Input
    - CLI
    - Configuration
    - Scenarios
  - Output
    - General
    - Metrics
  - Components
  - Dependencies
- Design

v3.2 User Guide

Contents

1. Configuration

1.1. Configuration Syntax

1.2. CLI Arguments Aliasing

1.3. Configuration Pattern Values

2. Items

2.1. Item Types

2.1.1. Data Items

2.1.1.1. Fixed Size Data Items

2.1.1.1.1. Empty Data Items

2.1.1.1.2. Small Data Items (1B-100KB)

2.1.1.1.3. Intermediate Size Data Items (100KB-10MB)

2.1.1.1.4. Big Data Items (10MB-100MB)

2.1.1.1.5. Very Big Data Items (100MB-10GB)

2.1.1.1.6. Huge Data Items (10GB-1TB)

2.1.1.1.7. Ultimate Data Items (>=1TB)

2.1.1.2. Random Size Data Items

2.1.1.3. Biased Random Size Data Items

2.1.2. Path Items

2.1.3. Token Items

2.2. Items Input

2.2.1. Items Input File

2.2.2. Items Path Listing Input

2.2.3. New Items Input

2.2.3.1. Random Item Ids

2.2.3.2. Ascending Item Ids

2.2.3.3. Descending Item Ids

2.2.3.4. Items Id Prefix

2.2.3.5. Items Id Radix

2.2.3.6. Items Id Offset

2.2.3.7. Items Id Length

2.3. Items Output

2.3.1. Items Output Delay

2.3.2. Items Output File

2.3.3. Items Destination Path

2.3.3.1. Constant Items Destination Path

2.3.3.2. Pattern Items Destination Path

3. Content

3.1. Uniform Random Data Payload

3.2. Payload From the External File

4. Concurrency

4.1. Default Concurrency Level (1)

4.2. Small Concurrency Level (2-10)

4.3. Medium Concurrency Level (11-100)

4.4. High Concurrency Level (101-1K)

4.5. Very High Concurrency Level (1K-10K)

4.6. Huge Concurrency Level (10K-100K)

4.7. Ultimate Concurrency Level (100K-1M)

5. Circularity

6. Load Jobs

6.1. Load Jobs Naming

6.2. Load Jobs Limitation

6.2.1. Load Jobs Are Infinite by Default

6.2.2. Limit Load Job by Processed Item Count

6.2.3. Limit Load Job by Rate

6.2.4. Limit Load Job by Processed Data Size

6.2.5. Limit Load Job by Time

6.2.6. Limit Load Job by End of Items Input

7. Metrics Reporting

7.1. Metrics Periodic Reporting

7.2. Metrics Reporting is Suppressed for the Precondition Jobs

7.3. Metrics Reporting Triggered by Load Threshold

7.4. I/O Traces Reporting

8. Load Types

8.1. Noop

8.2. Create

8.2.1. Create New Items

8.2.2. Copy Mode

8.3. Read

8.3.1. Read With Disabled Validation

8.3.2. Read With Enabled Validation

8.3.3. Partial Read

8.3.3.1. Random Byte Ranges Read

8.3.3.1.1. Single Random Byte Range Read

8.3.3.1.2. Multiple Random Byte Ranges Read

8.3.3.2. Fixed Byte Ranges Read

8.3.3.2.1. Read First N bytes

8.3.3.2.2. Read Last N bytes

8.3.3.2.3. Read Bytes from N1 to N2