9-nodes cluster in VMware on Windows PC #285

andriytk · 2020-12-14T19:08:36Z

andriytk
Dec 14, 2020
Collaborator

Managed to start 9-nodes cluster (7 server nodes with the disks + 1 cmu + 1 client node) in VMware+Vagrant environment on my Windows PC desktop with 8+2 EC-configuration and run a smoke performance test:

$ m0vg env
M0_VM_SSU_NR=7
M0_VM_CMU_CPUS=8
M0_VM_CMU_MEM_MB=8192
M0_VM_SSU_CPUS=4
M0_VM_CLIENT_CPUS=4
M0_VM_CLIENT_MEM_MB=2048
M0_VM_CMU_ENABLE_NFS_SERVER=yes
M0_VM_CMU_ENABLE_PUBLIC_NET=yes
M0_VM_SHARE_TYPE=smb
$
$ m0vg up cmu ssu{1..7} client1
Bringing machine 'cmu' up with 'vmware_desktop' provider...
Bringing machine 'ssu1' up with 'vmware_desktop' provider...
Bringing machine 'ssu2' up with 'vmware_desktop' provider...
Bringing machine 'ssu3' up with 'vmware_desktop' provider...
Bringing machine 'ssu4' up with 'vmware_desktop' provider...
Bringing machine 'ssu5' up with 'vmware_desktop' provider...
Bringing machine 'ssu6' up with 'vmware_desktop' provider...
Bringing machine 'ssu7' up with 'vmware_desktop' provider...
Bringing machine 'client1' up with 'vmware_desktop' provider...
...
$ m0vg status
Current machine states:

cmu                       running (vmware_desktop)
ssu1                      running (vmware_desktop)
ssu2                      running (vmware_desktop)
ssu3                      running (vmware_desktop)
ssu4                      running (vmware_desktop)
ssu5                      running (vmware_desktop)
ssu6                      running (vmware_desktop)
ssu7                      running (vmware_desktop)
client1                   running (vmware_desktop)

This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.

$
$ ssh [email protected]
Last login: Mon Dec 14 18:16:11 2020 from 192.168.85.1
[vagrant@cmu ~]$
[vagrant@cmu ~]$ hctl bootstrap --mkfs ~/cmu-ssu7-cleint1-cdf.yaml
2020-12-14 18:45:51: Generating cluster configuration...Warning: Permanently added the ECDSA host key for IP address '192.168.1.64' to the list of known hosts.
 OK
2020-12-14 18:46:03: Starting Consul server on this node......... OK
2020-12-14 18:46:10: Importing configuration into the KV store... OK
2020-12-14 18:46:11: Starting Consul on other nodes..... OK
2020-12-14 18:46:14: Updating Consul configuraton from the KV store... OK
2020-12-14 18:46:17: Waiting for the RC Leader to get elected... OK
2020-12-14 18:46:17: Starting Motr (phase1, mkfs)... OK
2020-12-14 18:46:32: Starting Motr (phase1, m0d)... OK
2020-12-14 18:46:37: Starting Motr (phase2, mkfs)... OK
2020-12-14 18:47:01: Starting Motr (phase2, m0d)... OK
2020-12-14 18:47:12: Checking health of services... OK
[vagrant@cmu ~]$
[vagrant@cmu ~]$ hctl status
Data pool:
    # fid name
    0x6f00000000000001:0x11b 'default'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x199 'all': 'default' None None
Services:
    client1.local
    [started]  hax        0x7200000000000001:0x10f  192.168.85.141@tcp:12345:1:1
    [unknown]  m0_client  0x7200000000000001:0x112  192.168.85.141@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x115  192.168.85.141@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x118  192.168.85.141@tcp:12345:4:3
    ssu1.local
    [started]  hax        0x7200000000000001:0x16  192.168.85.133@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x19  192.168.85.133@tcp:12345:2:1
    [started]  ioservice  0x7200000000000001:0x1c  192.168.85.133@tcp:12345:2:2
    [unknown]  m0_client  0x7200000000000001:0x30  192.168.85.133@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x33  192.168.85.133@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x36  192.168.85.133@tcp:12345:4:3
    cmu.local  (RC)
    [started]  hax        0x7200000000000001:0x4   192.168.85.132@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x7   192.168.85.132@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xa   192.168.85.132@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xd   192.168.85.132@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x10  192.168.85.132@tcp:12345:4:3
    ssu2.local
    [started]  hax        0x7200000000000001:0x3c  192.168.85.134@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x3f  192.168.85.134@tcp:12345:2:1
    [started]  ioservice  0x7200000000000001:0x42  192.168.85.134@tcp:12345:2:2
    [unknown]  m0_client  0x7200000000000001:0x56  192.168.85.134@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x59  192.168.85.134@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x5c  192.168.85.134@tcp:12345:4:3
    ssu3.local
    [started]  hax        0x7200000000000001:0x62  192.168.85.135@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0x65  192.168.85.135@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0x79  192.168.85.135@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x7c  192.168.85.135@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x7f  192.168.85.135@tcp:12345:4:3
    ssu4.local
    [started]  hax        0x7200000000000001:0x85  192.168.85.136@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0x88  192.168.85.136@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0x9c  192.168.85.136@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x9f  192.168.85.136@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0xa2  192.168.85.136@tcp:12345:4:3
    ssu5.local
    [started]  hax        0x7200000000000001:0xa8  192.168.85.137@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0xab  192.168.85.137@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xbf  192.168.85.137@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xc2  192.168.85.137@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0xc5  192.168.85.137@tcp:12345:4:3
    ssu6.local
    [started]  hax        0x7200000000000001:0xcb  192.168.85.138@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0xce  192.168.85.138@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xe2  192.168.85.138@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xe5  192.168.85.138@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0xe8  192.168.85.138@tcp:12345:4:3
    ssu7.local
    [started]  hax        0x7200000000000001:0xee  192.168.85.139@tcp:12345:1:1
    [started]  ioservice  0x7200000000000001:0xf1  192.168.85.139@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0x105  192.168.85.139@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0x108  192.168.85.139@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x10b  192.168.85.139@tcp:12345:4:3
[vagrant@cmu ~]$
[vagrant@cmu ~]$
[vagrant@cmu ~]$
[vagrant@cmu ~]$ cd /data/motr/bindings/go/mcp/
[vagrant@cmu mcp]$ go build
[vagrant@cmu mcp]$ hctl status | head
Data pool:
    # fid name
    0x6f00000000000001:0x11b 'default'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x199 'all': 'default' None None
Services:
    client1.local
    [started]  hax        0x7200000000000001:0x10f  192.168.85.141@tcp:12345:1:1
    [unknown]  m0_client  0x7200000000000001:0x112  192.168.85.141@tcp:12345:4:1
[vagrant@cmu mcp]$ hctl status | grep -A5 `hostname`
    cmu.local  (RC)
    [started]  hax        0x7200000000000001:0x4   192.168.85.132@tcp:12345:1:1
    [started]  confd      0x7200000000000001:0x7   192.168.85.132@tcp:12345:2:1
    [unknown]  m0_client  0x7200000000000001:0xa   192.168.85.132@tcp:12345:4:1
    [unknown]  m0_client  0x7200000000000001:0xd   192.168.85.132@tcp:12345:4:2
    [unknown]  m0_client  0x7200000000000001:0x10  192.168.85.132@tcp:12345:4:3
[vagrant@cmu mcp]$ export LD_LIBRARY_PATH=../../../motr/.libs/
[vagrant@cmu mcp]$ pdsh -w cmu.local,ssu[1-7].local,client1.local sudo chmod g+rw /dev/m0lnet
[vagrant@cmu mcp]$ pdsh -w cmu.local,ssu[1-7].local,client1.local sudo chgrp vagrant /dev/m0lnet
[vagrant@cmu mcp]$ ./mcp -prof 0x7000000000000001:0x199 -hax 192.168.85.132@tcp:12345:1:1 -proc 0x7200000000000001:0xa -ep 192.168.85.132@tcp:12345:4:1 -osz $((128*1024)) -v -bsz 256 -threads 2 /dev/zero 12345:123456700
2020/12/14 18:49:40 mio.go:477: W: off=0 len=268435456 bs=134217728 gs=33554432 speed=77 (Mbytes/sec)
2020/12/14 18:49:42 mio.go:477: W: off=268435456 len=268435456 bs=134217728 gs=33554432 speed=184 (Mbytes/sec)
2020/12/14 18:49:43 mio.go:477: W: off=536870912 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:49:45 mio.go:477: W: off=805306368 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:49:46 mio.go:477: W: off=1073741824 len=268435456 bs=134217728 gs=33554432 speed=172 (Mbytes/sec)
2020/12/14 18:49:48 mio.go:477: W: off=1342177280 len=268435456 bs=134217728 gs=33554432 speed=176 (Mbytes/sec)
2020/12/14 18:49:49 mio.go:477: W: off=1610612736 len=268435456 bs=134217728 gs=33554432 speed=191 (Mbytes/sec)
2020/12/14 18:49:50 mio.go:477: W: off=1879048192 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:49:52 mio.go:477: W: off=2147483648 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:49:53 mio.go:477: W: off=2415919104 len=268435456 bs=134217728 gs=33554432 speed=186 (Mbytes/sec)
2020/12/14 18:49:55 mio.go:477: W: off=2684354560 len=268435456 bs=134217728 gs=33554432 speed=176 (Mbytes/sec)
2020/12/14 18:49:56 mio.go:477: W: off=2952790016 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:49:58 mio.go:477: W: off=3221225472 len=268435456 bs=134217728 gs=33554432 speed=165 (Mbytes/sec)
2020/12/14 18:49:59 mio.go:477: W: off=3489660928 len=268435456 bs=134217728 gs=33554432 speed=168 (Mbytes/sec)
2020/12/14 18:50:01 mio.go:477: W: off=3758096384 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:50:02 mio.go:477: W: off=4026531840 len=268435456 bs=134217728 gs=33554432 speed=191 (Mbytes/sec)
2020/12/14 18:50:04 mio.go:477: W: off=4294967296 len=268435456 bs=134217728 gs=33554432 speed=188 (Mbytes/sec)
2020/12/14 18:50:05 mio.go:477: W: off=4563402752 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:50:06 mio.go:477: W: off=4831838208 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:08 mio.go:477: W: off=5100273664 len=268435456 bs=134217728 gs=33554432 speed=191 (Mbytes/sec)
2020/12/14 18:50:09 mio.go:477: W: off=5368709120 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
2020/12/14 18:50:10 mio.go:477: W: off=5637144576 len=268435456 bs=134217728 gs=33554432 speed=189 (Mbytes/sec)
2020/12/14 18:50:12 mio.go:477: W: off=5905580032 len=268435456 bs=134217728 gs=33554432 speed=192 (Mbytes/sec)
2020/12/14 18:50:13 mio.go:477: W: off=6174015488 len=268435456 bs=134217728 gs=33554432 speed=188 (Mbytes/sec)
2020/12/14 18:50:14 mio.go:477: W: off=6442450944 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
2020/12/14 18:50:16 mio.go:477: W: off=6710886400 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:17 mio.go:477: W: off=6979321856 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:18 mio.go:477: W: off=7247757312 len=268435456 bs=134217728 gs=33554432 speed=192 (Mbytes/sec)
2020/12/14 18:50:20 mio.go:477: W: off=7516192768 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:21 mio.go:477: W: off=7784628224 len=268435456 bs=134217728 gs=33554432 speed=194 (Mbytes/sec)
2020/12/14 18:50:22 mio.go:477: W: off=8053063680 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
2020/12/14 18:50:24 mio.go:477: W: off=8321499136 len=268435456 bs=134217728 gs=33554432 speed=187 (Mbytes/sec)
2020/12/14 18:50:25 mio.go:477: W: off=8589934592 len=268435456 bs=134217728 gs=33554432 speed=185 (Mbytes/sec)
2020/12/14 18:50:27 mio.go:477: W: off=8858370048 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:50:28 mio.go:477: W: off=9126805504 len=268435456 bs=134217728 gs=33554432 speed=188 (Mbytes/sec)
2020/12/14 18:50:29 mio.go:477: W: off=9395240960 len=268435456 bs=134217728 gs=33554432 speed=183 (Mbytes/sec)
2020/12/14 18:50:31 mio.go:477: W: off=9663676416 len=268435456 bs=134217728 gs=33554432 speed=164 (Mbytes/sec)
2020/12/14 18:50:32 mio.go:477: W: off=9932111872 len=268435456 bs=134217728 gs=33554432 speed=179 (Mbytes/sec)
2020/12/14 18:50:34 mio.go:477: W: off=10200547328 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:35 mio.go:477: W: off=10468982784 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:50:36 mio.go:477: W: off=10737418240 len=268435456 bs=134217728 gs=33554432 speed=195 (Mbytes/sec)
2020/12/14 18:50:38 mio.go:477: W: off=11005853696 len=268435456 bs=134217728 gs=33554432 speed=194 (Mbytes/sec)
2020/12/14 18:50:39 mio.go:477: W: off=11274289152 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:40 mio.go:477: W: off=11542724608 len=268435456 bs=134217728 gs=33554432 speed=197 (Mbytes/sec)
2020/12/14 18:50:42 mio.go:477: W: off=11811160064 len=268435456 bs=134217728 gs=33554432 speed=199 (Mbytes/sec)
2020/12/14 18:50:43 mio.go:477: W: off=12079595520 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:45 mio.go:477: W: off=12348030976 len=268435456 bs=134217728 gs=33554432 speed=190 (Mbytes/sec)
2020/12/14 18:50:46 mio.go:477: W: off=12616466432 len=268435456 bs=134217728 gs=33554432 speed=196 (Mbytes/sec)
2020/12/14 18:50:47 mio.go:477: W: off=12884901888 len=268435456 bs=134217728 gs=33554432 speed=193 (Mbytes/sec)
^C
[vagrant@cmu mcp]$

Memory footprint and CPU load during the test:

Note: in case Motr is installed not from rpm, but compiled manually - make sure to configure it with --disable-expensive-checks to get some reasonable perf numbers.

Versions:

Motr - commit fb19a78
Hare - commit 571ae39

Cluster configuration file (CDF): cmu-ssu7-cleint1-cdf.yaml.txt.

huanghua78 · 2020-12-15T03:38:56Z

huanghua78
Dec 15, 2020

Fantastic, @andriytk .
How much memory is configured for each VM?
How many CPU cores are configured for each VM?
How much disk space for each internal dick?

2 replies

andriytk Dec 15, 2020
Collaborator Author

Thanks @huanghua78 for reminding me mentioning it, let me update it...

andriytk Dec 15, 2020
Collaborator Author

So, I've added the output of m0vg env command there in the beginning which shows that:

8GB RAM is configured for cmu and 2GBs (the default value) is configured for ssu and client1 nodes.
8 CPU cores is configured for cmu and 4 cores is configured for ssu and client1 nodes.
By default 6 2GB-disks are configured per each ssu node (6 * 2 * 7 == 84 GB in total).

I've configured more resources in cmu to speed up the building of Motr mainly. But in VMware environment (in contrast to VirtualBox) the RAM is shared among host and VMs - unused memory can be used by host or another VMs. So if the memory is not used by cmu - it won't affect the others. Also cmu can be used as a client node which calculates the parity and might need some computational resources for this.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

9-nodes cluster in VMware on Windows PC #285

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

9-nodes cluster in VMware on Windows PC #285

andriytk Dec 14, 2020 Collaborator

Replies: 1 comment · 2 replies

huanghua78 Dec 15, 2020

andriytk Dec 15, 2020 Collaborator Author

andriytk Dec 15, 2020 Collaborator Author

andriytk
Dec 14, 2020
Collaborator

Replies: 1 comment 2 replies

huanghua78
Dec 15, 2020

andriytk Dec 15, 2020
Collaborator Author

andriytk Dec 15, 2020
Collaborator Author