Vm benchmark tests (#299)

Vm microbenchmark consists of few tests:
 - compressions (deflate and snappy)
 - factorization 
 - matrix QR decompostion
 - matrix SVD decompostion
 - matrix product
 - sha3 hash computing
 - recurive fibonacci computing

Each test exports the `main` function and contains few compile-time parameters that can be adjusted using the corresponding environment variables.

Author: mikevoronov

bench/vm/tests/README.md

@@ -0,0 +1,108 @@
+## VM benchmark tests
+
+This microbenchmark consists of a few tests which can be used to measure performance of various WebAssembly virtual machines. It should be noted this benchmark is by definition is not comprehensive and might not reflect performance on real life workloads. 
+
+However, each test was crafted in a way that would prevent the virtual machine from optimizing them using the dead code elimination. Most tests also prevent intermediate results memoization (with `fibonacci_bigint` and `factorization_reikna` being exceptions) and allow to specify an RNG seed to obtain repeatable benchmark results.
+
+Each test exports the `main` function which is called by the benchmark runner. There are also few compile-time parameters in each test that can be adjusted using the corresponding environment variables. The benchmark has been successfully tested on Mac and should run smoothly on Linux as well.
+
+To run tests Rust Cargo package manager should be installed – the easiest way would be to follow these instructions:
+
+```shell
+# download and install rustup
+curl -sSf https://static.rust-lang.org/rustup.sh | sh
+
+# install the latest nightly toolchain
+~/.cargo/bin/rustup toolchain install nightly
+
+# make shure that Rust is up to date
+rustup update
+
+# install the Webassembly target for Rust
+rustup target add wasm32-unknown-unknown --toolchain nightly
+```
+Below we also assume that Fluence GitHub repo has been already cloned and we are in the `vm_bench/tests` directory:
+
+```shell
+git clone --recursive https://github.com/fluencelabs/fluence
+cd fluence/vm/vm_bench/tests
+```
+
+Random numbers in tests are generated with [IsaacRng](https://doc.rust-lang.org/1.0.0/rand/isaac/struct.IsaacRng.html). It should also be noted relative results of the QR and SVG decomposition tests can be somewhat different on different hardware because of the floating point non-determinism.
+
+### Compression
+
+This test compresses a sequence of `SEQUENCE_SIZE` bytes and has `ITERATIONS_COUNT` iterations of compressions. On each iteration a new sequence is generated by the seed that was computed based on the previous sequence. As the first seed the `SEED` parameter is used. Two compression algorithms ([deflate](https://docs.rs/deflate) and [snappy](https://docs.rs/snap)) are supported and can be chosen by specifying the `deflate_compression` flag.
+
+To run:
+
+```shell
+cd recursive_hash
+ITERATIONS_COUNT=1 SEED=1000000 SEQUENCE_SIZE=1024 cargo build --release --target wasm32-unknown-unknown [--feature "deflate_compression"]
+```
+
+### Factorization
+
+This test factorizes provided `FACTORIZED_NUMBER` using [reikna](https://docs.rs/reikna/0.10.0/reikna/) library.
+
+To run:
+
+```shell
+cd factorization_reikna
+FACTORIZED_NUMBER=2147483647 cargo build --release --target wasm32-unknown-unknown
+```
+
+### Recursive fibonacci computing
+
+This test recursively computes the Fibonacci number with the index `FIB_NUMBER`.
+
+To run:
+
+```shell
+cd fibonacci_bigint
+FIB_NUMBER=38 cargo build --release --target wasm32-unknown-unknown
+```
+
+### Matrix product
+
+This test computes a product of random generated matrices of size `MATRIX_SIZE`. There are `ITERATIONS_COUNT` iterations; on each iteration new matrices are generated using the seed that was computed using the previous product result. As the first seed the `SEED` parameter is used.
+
+To run:
+
+```shell
+cd matrix_product
+ITERATIONS_COUNT=1000000 SEED=1 MATRIX_SIZE=10 cargo build --release --target wasm32-unknown-unknown
+```
+
+### Matrix QR decomposition
+
+This test computes a QR decomposition of random generated matrices of size `MATRIX_SIZE`. There are `ITERATIONS_COUNT` iterations of decomposition; on each iteration a new matrix is generated using the seed that was computed using the previous decomposition result. As the first seed the `SEED` parameter is used.
+
+To run:
+
+```shell
+cd matrix_qr_decomposition
+ITERATIONS_COUNT=1000000 SEED=1 MATRIX_SIZE=10 cargo build --release --target wasm32-unknown-unknown
+```
+
+### Matrix SVD decomposition
+
+This test computes an SVD decomposition of random generated matrices of size `MATRIX_SIZE`. There are `ITERATIONS_COUNT` iterations of decomposition; on each iteration a new matrix is generated using the seed that was computed using the previous decomposition result. As the first seed the `SEED` parameter is used.
+
+To run:
+
+```shell
+cd matrix_svd_decomposition
+ITERATIONS_COUNT=1000000 SEED=1 MATRIX_SIZE=10 cargo build --release --target wasm32-unknown-unknown
+```
+
+### Recursive hash computing
+
+This test iteratively computes a hash chain `hash(hash( ... hash(x)))` of length `ITERATIONS_COUNT`, where `x` is the initial value specified by `INITIAL_VALUE`.
+
+To run:
+
+```shell
+cd recursive_hash
+ITERATIONS_COUNT=10000000 INITIAL_VALUE=0 cargo build --release --target wasm32-unknown-unknown
+```

bench/vm/tests/compression/Cargo.toml

@@ -0,0 +1,17 @@
+[package]
+name = "compression"
+version = "0.1.0"
+authors = ["Fluence Labs"]
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+snap = "0.2"
+deflate = "0.7.19"
+rand = "0.6.1"
+rand_isaac = "0.1.0"
+
+[features]
+default = []
+deflate_compression = []

bench/vm/tests/compression/rust-toolchain

@@ -0,0 +1 @@
+nightly

bench/vm/tests/compression/src/lib.rs

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2018 Fluence Labs Limited
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+mod settings;
+
+extern crate deflate;
+extern crate rand;
+extern crate rand_isaac;
+extern crate snap;
+
+use deflate::deflate_bytes;
+use rand::{Rng, SeedableRng};
+use rand_isaac::IsaacRng;
+use settings::{ITERATIONS_COUNT, SEED, SEQUENCE_SIZE};
+
+type Sequence = Vec<u8>;
+
+/// Generates pseudo-random byte sequence by given seed and given size.
+fn generate_sequence(seed: u64, size: usize) -> Sequence {
+    let mut rng: IsaacRng = SeedableRng::seed_from_u64(seed);
+    let mut result_sequence = Sequence::with_capacity(size);
+
+    for _ in 0..size {
+        result_sequence.push(rng.gen::<u8>());
+    }
+    result_sequence
+}
+
+/// Compresses provided sequence by deflate or snappy algorithm.
+fn compress_sequence(sequence: &Sequence) -> Sequence {
+    if cfg!(feature = "deflate_compression") {
+        return deflate_bytes(&sequence);
+    }
+
+    return snap::Encoder::new().compress_vec(&sequence).unwrap();
+}
+
+#[no_mangle]
+pub extern "C" fn main() -> u8 {
+    let seed: u64 = SEED.parse::<u64>().unwrap();
+    let iterations_count: u64 = ITERATIONS_COUNT.parse::<u64>().unwrap();
+    let sequence_size: usize = SEQUENCE_SIZE.parse::<usize>().unwrap();
+
+    let mut compressed_sequence = generate_sequence(seed, sequence_size);
+
+    for _ in 1..iterations_count {
+        let new_seed: usize = compressed_sequence.len()
+            + compressed_sequence.iter().fold(0u8, |x1, x2| x1 ^ x2) as usize;
+        compressed_sequence = generate_sequence(new_seed as u64, sequence_size);
+        compressed_sequence = compress_sequence(&compressed_sequence);
+    }
+
+    compressed_sequence.iter().fold(0u8, |x1, x2| x1 ^ x2)
+}

bench/vm/tests/compression/src/settings.rs

@@ -0,0 +1,23 @@
+/*
+ * Copyright 2018 Fluence Labs Limited
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/// This value is used as the first seed of pseudo-random generator for sequence generation.
+pub const SEED: &str = env!("SEED");
+
+/// Count of compression iterations.
+pub const ITERATIONS_COUNT: &str = env!("ITERATIONS_COUNT");
+
+/// Size of sequence that should be compressed on each iteration.
+pub const SEQUENCE_SIZE: &str = env!("SEQUENCE_SIZE");

bench/vm/tests/factorization_reikna/Cargo.toml

@@ -0,0 +1,10 @@
+[package]
+name = "factorization_reikna"
+version = "0.1.0"
+authors = ["Fluence Labs"]
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+reikna = "0.6.0"

bench/vm/tests/factorization_reikna/rust-toolchain

@@ -0,0 +1 @@
+nightly

bench/vm/tests/factorization_reikna/src/lib.rs

@@ -0,0 +1,29 @@
+/*
+ * Copyright 2018 Fluence Labs Limited
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+mod settings;
+extern crate reikna;
+
+use reikna::prime;
+use settings::FACTORIZED_NUMBER;
+
+#[no_mangle]
+pub extern "C" fn main() -> u64 {
+    let factorized_number: u64 = FACTORIZED_NUMBER.parse::<u64>().unwrap();
+    // reikna uses Atkin or Eratosthenes seive to factorize given number
+    let factors = prime::factorize(factorized_number);
+
+    factors[0]
+}

bench/vm/tests/factorization_reikna/src/settings.rs

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2018 Fluence Labs Limited
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/// A requested number that should be factorized by this test.
+pub const FACTORIZED_NUMBER: &str = env!("FACTORIZED_NUMBER");

bench/vm/tests/fibonacci_bigint/Cargo.toml

@@ -0,0 +1,11 @@
+[package]
+name = "fibonacci_bigint"
+version = "0.1.0"
+authors = ["Fluence Labs"]
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+num-bigint = "0.2"
+num-traits = "0.2"

bench/vm/tests/fibonacci_bigint/rust-toolchain

@@ -0,0 +1 @@
+nightly

bench/vm/tests/fibonacci_bigint/src/lib.rs

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2018 Fluence Labs Limited
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+mod settings;
+extern crate num_bigint;
+extern crate num_traits;
+
+use num_bigint::BigUint;
+use num_traits::One;
+use settings::FIB_NUMBER;
+use std::ops::Sub;
+
+/// Recursively computes a fibonacci number F_num for the given num.
+fn fib(num: &BigUint) -> BigUint {
+    if num.le(&BigUint::from(2u32)) {
+        return One::one();
+    }
+
+    fib(&num.sub(1u32)) + fib(&num.sub(2u32))
+}
+
+#[no_mangle]
+pub extern "C" fn main() -> u8 {
+    let fib_number: BigUint = BigUint::from(FIB_NUMBER.parse::<u64>().unwrap());
+
+    fib(&fib_number)
+        .to_bytes_le()
+        .iter()
+        .fold(0u8, |x1, x2| x1 ^ x2)
+}

bench/vm/tests/fibonacci_bigint/src/settings.rs

@@ -0,0 +1,17 @@
+/*
+ * Copyright 2018 Fluence Labs Limited
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+/// A requested fibonacci number that would be computed.
+pub const FIB_NUMBER: &str = env!("FIB_NUMBER");

bench/vm/tests/matrix_product/Cargo.toml

@@ -0,0 +1,12 @@
+[package]
+name = "matrix_product"
+version = "0.1.0"
+authors = ["Fluence Labs"]
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+nalgebra = { version = "0.16", features = [ "alloc" ] }
+rand = "0.6.1"
+rand_isaac = "0.1.0"

bench/vm/tests/matrix_product/rust-toolchain

@@ -0,0 +1 @@
+nightly