Add parser for CIFS client stats

[SuperQ]

The CIFS kernel module includes client stats in /proc/fs/cifs/Stats, but only if the kernel is configured with CONFIG_CIFS_STATS.

It includes per mountpoint info, so it should be pretty useful.

[SuperQ]

Here's a sample stats file: (Ubuntu 4.15.0-20-generic kernel)

Resources in use
CIFS Session: 2
Share (unique mount targets): 4
SMB Request/Response Buffer: 2 Pool size: 6
SMB Small Req/Resp Buffer: 2 Pool size: 30
Operations (MIDs): 0

0 session 0 share reconnects
Total vfs operations: 90 maximum at one time: 2

1) \\server\share1
SMBs: 20
Negotiates: 0 sent 0 failed
SessionSetups: 0 sent 0 failed
Logoffs: 0 sent 0 failed
TreeConnects: 0 sent 0 failed
TreeDisconnects: 0 sent 0 failed
Creates: 0 sent 2 failed
Closes: 0 sent 0 failed
Flushes: 0 sent 0 failed
Reads: 0 sent 0 failed
Writes: 0 sent 0 failed
Locks: 0 sent 0 failed
IOCTLs: 0 sent 0 failed
Cancels: 0 sent 0 failed
Echos: 0 sent 0 failed
QueryDirectories: 0 sent 0 failed
ChangeNotifies: 0 sent 0 failed
QueryInfos: 0 sent 0 failed
SetInfos: 0 sent 0 failed
OplockBreaks: 0 sent 0 failed
2) \\server\share2
SMBs: 285
Negotiates: 0 sent 0 failed
SessionSetups: 0 sent 0 failed
Logoffs: 0 sent 0 failed
TreeConnects: 0 sent 0 failed
TreeDisconnects: 0 sent 0 failed
Creates: 0 sent 2 failed
Closes: 0 sent 0 failed
Flushes: 0 sent 0 failed
Reads: 0 sent 0 failed
Writes: 0 sent 0 failed
Locks: 0 sent 0 failed
IOCTLs: 0 sent 0 failed
Cancels: 0 sent 0 failed
Echos: 0 sent 0 failed
QueryDirectories: 0 sent 6 failed
ChangeNotifies: 0 sent 0 failed
QueryInfos: 0 sent 0 failed
SetInfos: 0 sent 0 failed
OplockBreaks: 0 sent 0 failed

[shibumi]

This is what I get on CentOS7 (kernel: 3.10.0-862.3.3.el7.x86_64):

Resources in use
CIFS Session: 1
Share (unique mount targets): 1
SMB Request/Response Buffer: 1 Pool size: 5
SMB Small Req/Resp Buffer: 1 Pool size: 30
Operations (MIDs): 0

0 session 0 share reconnects
Total vfs operations: 21 maximum at one time: 2

1) \\server\\share
SMBs: 13 Oplocks breaks: 0
Reads:  0 Bytes: 0
Writes: 0 Bytes: 0
Flushes: 0
Locks: 0 HardLinks: 0 Symlinks: 0
Opens: 0 Closes: 0 Deletes: 0
Posix Opens: 0 Posix Mkdirs: 0
Mkdirs: 0 Rmdirs: 0
Renames: 0 T2 Renames 0
FindFirst: 2 FNext 0 FClose 0

@SuperQ which Distribution is this?

[shibumi]

@SuperQ any feedback to my first data model? https://github.com/shibumi/procfs/blob/cifs-support/cifs/cifs.go

I am unsure if I should add the key/value pairs of your output as well. I guess this would make sense, but what would I do with it when some kernels don't provide these pairs? Just initialize them as 0?

I am also thinking about the right parsing process. I thought about something similar like the NFS parser.. so reading it line by line, trying to get the key and regex the number out of it?

Problem here: The lines contain more multiple key/value pairs and I have blank lines and some keys consist of multiple words like for example: SMB Request/Response Buffer.

Another problem is that the lines are totally different. In your output we have sent/failed pairs.. I haven't.

So do we stick to the newer kernel? Or do we want to have downwards compatibility to older kernels?

[shibumi]

Ok now I am totally confused. I have tested this in a vagrant box with ubuntu 18.04 and kernel 4.15.0-22-generic. Here is the output what I get here:

Resources in use
CIFS Session: 1
Share (unique mount targets): 2
SMB Request/Response Buffer: 1 Pool size: 5
SMB Small Req/Resp Buffer: 1 Pool size: 30
Operations (MIDs): 0

0 session 0 share reconnects
Total vfs operations: 16 maximum at one time: 2

1) \\server\share
SMBs: 9 Oplocks breaks: 0
Reads:  0 Bytes: 0
Writes: 0 Bytes: 0
Flushes: 0
Locks: 0 HardLinks: 0 Symlinks: 0
Opens: 0 Closes: 0 Deletes: 0
Posix Opens: 0 Posix Mkdirs: 0
Mkdirs: 0 Rmdirs: 0
Renames: 0 T2 Renames 0
FindFirst: 1 FNext 0 FClose 0

As you can see is the output equal to the one from CentOS with kernel 3.15...

Why do you have a totally different output? Another kernel module? Or are you running in Server Mode with a Samba Server? Are you the Server or the Client? My both examples are for Clients.

[SuperQ]

Mine is a client example, but maybe it depends on what the CIFS server is. My CIFS server is Samba 4.4.16 on a QNAP NAS.

[SuperQ]

As for the struct, that's a good question. One option is we could create a struct of pointers, and return nil pointers for missing data. This makes using the functions more annoying, but solves the different versions.

[c-r-h]

Got yer note.

Regarding the per-tree-connect output (each \\server\share is a Tree Connect, which == a mount), all of the information being reported, in each format, is available to the client. It's not stuff that the client would have to query the server to retrieve.

Let me know what version of the CIFS tools are in use in each case. Eg.:
$ mount.cifs -V
mount.cifs version: 6.0

I'll ping the folks in charge and see what info they can give you regarding the formats you're getting back.

[shibumi]

@SuperQ possible that it's related to the used samba version. I have used the mount flag vers=1.0 for the CIFS mounts... I can try vers=2.0 tomorrow.. everything above is not supported by our storage cluster.

@c-r-h I will give you intel about the mount.cifs version tomorrow as well. Your help is welcome ;)

[SuperQ]

Thanks @c-r-h!

[c-r-h]

@shibumi: Right on track.

Steve says:

The short answer is pretty simple - CIFS mounts (RHEL defaults to CIFS/SMB1, ie "vers=1.0") vs. SMB3 mounts. They could probably make an easy parsing change by correlating the output with /proc/mounts (or equivalently /proc/fs/cifs/DebugData) - there is also a debug ioctl that can return some additional info on the mounts.

[c-r-h]

Oh... and it's not the Samba version, but the protocol version that makes a difference. The vers= parameter determines which protocol version should be used. (Basically, it sets the minimum version that the client will negotiate.)

[shibumi]

@c-r-h
On CentOS 7: mount.cifs version: 6.2
On Ubuntu 18.04: mount.cifs version: 6.8

Sadly I have no possibility to test the different versions.. seems like our storage cluster only supports version 1.0.

Btw this is what I get from /proc/fs/cifs/DebugData:

Display Internal CIFS Data Structures for Debugging
---------------------------------------------------
CIFS Version 2.09
Features: dfs lanman posix spnego xattr acl
Active VFS Requests: 0
Servers:
Number of credits: 50
1) Name: 139.174.2.4  Domain: workgroup Uses: 1 OS: Windows 5.0
	NOS: Windows 2000 LAN Manager	Capability: 0x8000d3fd
	SMB session status: 1	TCP status: 1
	Local Users To Server: 1 SecMode: 0x3 Req On Wire: 0
	Shares:
	1) \\server\share Mounts: 1 Type: NTFS DevInfo: 0x20 Attributes: 0x4004f
	PathComponentMax: 255 Status: 1 type: DISK 

	MIDs:

No SMB Version.. so seems like we need to parse /proc/mounts/ for the SMB version, except you know another location for getting this value.

[shibumi]

@c-r-h One question. Is a mix out of SMB1 and SMB3 possible? so something like this?:

Resources in use
CIFS Session: 1
Share (unique mount targets): 2
SMB Request/Response Buffer: 1 Pool size: 5
SMB Small Req/Resp Buffer: 1 Pool size: 30
Operations (MIDs): 0

0 session 0 share reconnects
Total vfs operations: 16 maximum at one time: 2

1) \\server1\share1
SMBs: 9 Oplocks breaks: 0
Reads:  0 Bytes: 0
Writes: 0 Bytes: 0
Flushes: 0
Locks: 0 HardLinks: 0 Symlinks: 0
Opens: 0 Closes: 0 Deletes: 0
Posix Opens: 0 Posix Mkdirs: 0
Mkdirs: 0 Rmdirs: 0
Renames: 0 T2 Renames 0
FindFirst: 1 FNext 0 FClose 0

2) \\server2\share2
SMBs: 20
Negotiates: 0 sent 0 failed
SessionSetups: 0 sent 0 failed
Logoffs: 0 sent 0 failed
TreeConnects: 0 sent 0 failed
TreeDisconnects: 0 sent 0 failed
Creates: 0 sent 2 failed
Closes: 0 sent 0 failed
Flushes: 0 sent 0 failed
Reads: 0 sent 0 failed
Writes: 0 sent 0 failed
Locks: 0 sent 0 failed
IOCTLs: 0 sent 0 failed
Cancels: 0 sent 0 failed
Echos: 0 sent 0 failed
QueryDirectories: 0 sent 0 failed
ChangeNotifies: 0 sent 0 failed
QueryInfos: 0 sent 0 failed
SetInfos: 0 sent 0 failed
OplockBreaks: 0 sent 0 failed

[SuperQ]

Yes, mix is valid.

The above examples seem to be vers=2.x.

I don't have a sample of vers=3.0 yet.

[SuperQ]

Ok, I got a sample for vers=3.0. It looks like there is no stats difference for 2.x or 3.x.

[c-r-h]

Quick notes:

• Version 2.09 of the CIFS kernel client is very old. I'm on an outdated Linux Mint system (based on Ubuntu 14.04.5; kernel 3.13) and I'm running CIFS v6.0.
• SMB1 and SMB2 are different protocols, but SMB3 is just SMB2 with added features (marketing upgrade).

The chart on this page may be useful: http://cifs.com/

Oh... forgot to say...
Ben's correct that you can have different protocols per connection. The protocol dialect is negotiated at the start of the connection. Note, however, that there can be multiple tree connects (mounts) per connection. All of those would be subject to the initial dialect negotiation.

Each protocol version (SMB1, SMB2, SMB3) has multiple successive dialects.

[shibumi]

Thanks for the feedback. I have nearly finished the implementation of support for SMB1. Next step will be SMB2 and SMB3 + Test Units. I hope that I can get this finished tomorrow.

[shibumi]

I have a working version now that supports SMB1 and SMB2/SMB3. Here is my current branch:
https://github.com/shibumi/procfs/tree/cifs-support-smb2/cifs

What is missing so far:

• Tests (!!)
• calling the parser in the fs package
• rebasing and squashing

Am I missing something?

Note: This is my first go code that goes beyond parsing some JSON.. so feedback is really appreciated

[SuperQ]

Looks good so far!

[shibumi]

In case you want to test it out:

Here is a main.go file with all structs and regexes: https://paste.xinu.at/1UJMr2/
And here are some test files that I have used: https://paste.xinu.at/m-RrpVpb/

[shibumi]

Current Status:

• fixed multiple bugs
• Wrote a testing file

Todo:

• Comments, comments, comments....
• More Testcases
• Maybe a cleanup of the parse_cifs.go file.. the current for loop is much spaghetti code

https://github.com/shibumi/procfs/tree/shibumi/cifs

The tests are still failing.. Somebody an idea?

>> checking code style
>> checking license header
>> vetting code
rm -v sysfs/fixtures/.unpacked
removed 'sysfs/fixtures/.unpacked'
./ttar -C sysfs/ -c -f sysfs/fixtures.ttar fixtures/
./ttar -C sysfs/ -x -f sysfs/fixtures.ttar
touch sysfs/fixtures/.unpacked
>> running all tests
ok  	github.com/prometheus/procfs	(cached)
ok  	github.com/prometheus/procfs/bcache	(cached)
--- FAIL: TestNewCifsRPCStats (0.01s)
    --- FAIL: TestNewCifsRPCStats/invalid_file (0.00s)
    	parse_cifs_test.go:113: expected an error, but none occured
FAIL
FAIL	github.com/prometheus/procfs/cifs	0.019s
?   	github.com/prometheus/procfs/internal/util	[no test files]
ok  	github.com/prometheus/procfs/nfs	(cached)
ok  	github.com/prometheus/procfs/sysfs	1.020s
ok  	github.com/prometheus/procfs/xfs	(cached)
make: *** [Makefile:44: test] Error 1

[shibumi]

Ok I have worked on the tests, added some comments and added the cifs part to the fs.go file. Anything else to do?

[shibumi]

Looks like the Samba team has dropped SMBv1:
https://github.com/samba-team/samba/blob/59cca4c5d699be80b4ed22b40d8914787415c507/WHATSNEW.txt

This leads to a question in my head. Do we want still want to support smbv1 for the parser?
If we would only support smbv2 and smbv3 this would make parsing a lot easier, because we would have no context switches anymore (smbv2 and smbv3 share the same statistics format)

[c-r-h]

SMB1 will still be available in Samba. We're just turning it off by default as Microsoft has previously done in Windows. All that needs to be done to re-enable it is to change a setting in the smb.conf file.

The same is true for the SMB1 protocol in the Linux SMB kernel client (aka. the CIFS kernel client). SMB1 will be disabled by default but can be re-enabled. The code will still be there.

I imagine that legacy installations will continue to use SMB1 for some time. Whether or not you want to support these installations is a separate question.

SMB2 and SMB3 share the same statistics format because they are the same protocol SMB3 is just SMB2 with some added features.

[shibumi]

Hi,
I've thought I am going to revive this thread, because I have finally time to work a bit on this issue.
As mentioned in the PR, I have developed a multiline regex for parsing the smb1 and smb2/smb3 groups. This multiline regex is ugly as hell and looks like this:

(?m)(?:(?:(?P<smbID>\d+)\) \\\\(?P<smbServer>[A-Za-z1-9-.]+)(?P<smbShare>.+)\nSMBs:\s+(?P<smbs>\d+) Oplocks breaks:\s+(?P<breaks>\d+)\nReads:\s+(?P<reads>\d+) Bytes:\s+(?P<readsBytes>\d+)\nWrites:\s+(?P<writes>\d+) Bytes:\s+(?P<writesBytes>\d+)\nFlushes:\s+(?P<flushes>\d+)\nLocks:\s+(?P<locks>\d+) HardLinks:\s+(?P<hardlinks>\d+) Symlinks:\s+(?P<symlinks>\d+)\nOpens:\s+(?P<opens>\d+) Closes:\s+(?P<closes>\d+) Deletes:\s+(?P<deletes>\d+)\nPosix Opens:\s+(?P<posixOpens>\d+) Posix Mkdirs:\s+(?P<posixMkdirs>\d+)\nMkdirs:\s+(?P<mkdirs>\d+) Rmdirs:\s+(?P<rmdirs>\d+)\nRenames:\s+(?P<renames>\d+) T2 Renames\s+(?P<t2Renames>\d+)\nFindFirst:\s+(?P<findFirst>\d+) FNext\s+(?P<fNext>\d+) FClose\s+(?P<fClose>\d+)|(?P<smb3ID>\d+)\) \\\\(?P<smb3Server>[A-Za-z1-9-.]+)(?P<smb3Share>.+)\nSMBs:\s+(?P<smb3s>\d+)\nNegotiates:\s+(?P<negotiatesSent>\d+) sent\s+(?P<negotiatesFailed>\d+) failed\nSessionSetups:\s+(?P<sessionSetupsSent>\d+) sent\s+(?P<sessionSetupsFailed>\d+) failed\nLogoffs:\s+(?P<logoffsSent>\d+) sent\s+(?P<logoffsFailed>\d+) failed\nTreeConnects:\s+(?P<treeConnectsSent>\d+) sent\s+(?P<treeConnectsFailed>\d+) failed\nTreeDisconnects:\s+(?P<treeDisconnectsSent>\d+) sent\s+(?P<treeDisconnectsFailed>\d+) failed\nCreates:\s+(?P<createsSent>\d+) sent\s+(?P<createsFailed>\d+) failed\nCloses:\s+(?P<closesSent>\d+) sent\s+(?P<closesFailed>\d+) failed\nFlushes:\s+(?P<flushesSent>\d+) sent\s+(?P<flushesFailed>\d+) failed\nReads:\s+(?P<readsSent>\d+) sent\s+(?P<readsFailed>\d+) failed\nWrites:\s+(?P<writesSent>\d+) sent\s+(?P<writesFailed>\d+) failed\nLocks:\s+(?P<locksSent>\d+) sent\s+(?P<locksFailed>\d+) failed\nIOCTLs:\s+(?P<ioctlsSent>\d+) sent\s+(?P<ioctlsFailed>\d+) failed\nCancels:\s+(?P<cancelsSent>\d+) sent\s+(?P<cancelsFailed>\d+) failed\nEchos:\s+(?P<echosSent>\d+) sent\s+(?P<echosFailed>\d+) failed\nQueryDirectories:\s+(?P<queryDirectoriesSent>\d+) sent\s+(?P<queryDirectoriesFailed>\d+) failed\nChangeNotifies:\s+(?P<changeNotifiesSent>\d+) sent\s+(?P<changeNotifiesFailed>\d+) failed\nQueryInfos:\s+(?P<queryInfosSent>\d+) sent\s+(?P<queryInfosFailed>\d+) failed\nSetInfos:\s+(?P<setInfosSent>\d+) sent\s+(?P<setInfosFailed>\d+) failed\nOplockBreaks:\s+(?P<opLockBreaksSent>\d+) sent\s+(?P<opLockBreaksFailed>\d+) failed)+)

Any comments on this before I start coding?

(Short Recap: We want to have SMB1 and SMB2/SMB3 statistics in procfs, but the kernel provides no serialization format. So the only way to parse this is via regex. Except you have a better idea...)

Go Playground link to the regex (code generated by regex101.com): https://play.golang.org/p/WDDG6gjJdca

[shibumi]

Hi,
I made a proof of concept exporter with the related multiline regex:

https://github.com/shibumi/cifs-exporter

You can have a look over it and decide if this is the functionality you would want to see in procfs. It's the best solution I came up with.