-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy patharchitecture.htm
198 lines (189 loc) · 9.55 KB
/
architecture.htm
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>cmail: Development & Architecture information</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<link rel="icon" href="static/cmail_logo.png" type="image/x-icon" />
<link rel="shortcut icon" href="static/cmail_logo.png" type="image/x-icon" />
<link rel="stylesheet" type="text/css" href="static/cmail.css" />
<meta name="description" content="cmail - Internet mail processing suite with SQLite backend" />
<meta name="keywords" content="cmail, smtp, email, pop, webmail, messaging, software architecture" />
<meta name="robots" content="index,follow" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
</head>
<body>
<div>
<a href="index.htm">Back</a> to the main page
</div>
<div>
<h1>Architecture & development information</h1>
This section discusses general architectural decisions, tools
and structures used in the development of cmail and strives to
be an informational guide for first-time contributors.
</div>
<div class="section">
<h2>System structure diagram</h2>
<p>
Click to view at full resolution.
</p>
<a href="static/generated/system.svg">
<img src="static/generated/system.svg" style="height:15em;" />
</a>
<p>
The system structure diagram describes how the various modules of
cmail interact with eachother as well as remote systems.
</p>
</div>
<div class="section">
<h2>Starting to contribute</h2>
Looking for a place to start? Here are some pointers:
<ul>
<li>
If you're interested in contributing to the core cmail code, the <a href="https://github.com/cmail-mta/cmail/blob/master/DEVELOPMENT.txt">development guide
in the repository</a> may be interesting to you, as it covers issues such as code style & conventions, licensing and tools used for development.
</li>
<li>
There are some issues marked <a href="https://github.com/cmail-mta/cmail/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22"><code>help wanted</code></a>
and <a href="https://github.com/cmail-mta/cmail/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22"><code>good first issue</code></a> on the issue tracking system
</li>
<li>
Find the documentation somewhat lacking? Consider extending and
updating the <a href="https://github.com/cmail-mta/homepage">homepage</a>
</li>
<li>
If you're interested in learning about the protocols involved in mail processing,
consider writing <a href="https://github.com/cmail-mta/cmail/tree/master/tests/expect">protocol level tests</a>
against the cmail implementations - edge cases are particularly welcome!
</li>
<li>
Grep your clone of the source repository for occurences of <strong>FIXME</strong> or <strong>TODO</strong>
and see if there's something simple to do
</li>
<li>
Created a great connector or helper program? Consider adding it to the
<a href="https://github.com/cmail-mta/contrib">contrib repository</a>
</li>
<li>
Opening well researched issues on Github (e.g. against the <a href="https://github.com/cmail-mta/homepage/issues">homepage</a>
or <a href="https://github.com/cmail-mta/cmail/issues">main</a> repositories) is also a great way to help
</li>
</div>
<div class="section">
<h2>Development tooling</h2>
cmail uses a variety of tools to help maintain code quality,
automate tasks and aid development.
<p>
At the most basic level, this means cmail is regularly built
and checked for compile-time warnings using a variety of compilers,
and operating systems, including
<ul>
<li>gcc</li>
<li>clang</li>
<li>tcc</li>
</ul>
</p>
<p>
Code quality is regularly being analyzed by
<ul>
<li>
Running development test builds within <a href="http://valgrind.org/">
<code>valgrind</code></a> to check for access violations and memory leaks
at run-time (invoked via <code>make run</code>)
</li>
<li>
Running <code><a href="http://cppcheck.sourceforge.net/">cppcheck</a></code>
to gather a quick list of warnings from the source (via <code>make cppcheck</code>)
</li>
<li>
Using clangs <code><a href="http://clang-analyzer.llvm.org/scan-build.html">scan-build</a></code>
static analyzer to generate warnings from the source (invoked via <code>scan-build make</code>)
</li>
<li>
Scanning builds with the free offer of the <a href="https://scan.coverity.com/">Coverity</a>
static analyzer solution (<a href="https://scan.coverity.com/projects/5438">Coverity Scan</a>)
</li>
<li>
Stress-testing selected interfaces with fuzzers (eg.
<a href="http://lcamtuf.coredump.cx/afl/">afl-fuzz</a>, further information about
how to invoke fuzzing are in the works)
</li>
</ul>
</p>
</div>
<div class="section">
<h2>Testing & Test Coverage</h2>
cmail features a growing suite of automated tests, written mostly in <a href="http://tcl.tk/">Tcl</a> using the
<a href="http://expect.sf.net/">Expect</a> toolkit. In order to run them, you'll need a complete installation of cmail
which is <strong>NOT</strong> in production use, as the test suite will heavily modify and overwrite any existing configuration
in order to exercise different parts of the code. Running the suite on a dedicated VM is probably the best way to proceed.
<p>
Coverage information can be extracted by running the test suite against a build with the <a href="https://gcc.gnu.org/onlinedocs/gcc/Gcov.html">gcov</a>
coverage analyzer enabled and thereafter analyzing the output with <a href="http://ltp.sourceforge.net/coverage/lcov.php">lcov</a>.
As these steps need to assume a lot about the environment they are run in, there is as of yet no completely automated way of running the
test suite.
</p>
<h3>Coverage reports</h3>
You can view generated coverage reports, which will be updated at irregular intervals.
<ul>
<li><a href="coverage-reports/cmail-smtpd/index.html">cmail-smtpd (commit 013e2e9c78e956ff6acc3e3aee6c189e4a839c3f)</a></li>
<li><a href="coverage-reports/cmail-popd/index.html">cmail-popd (commit 013e2e9c78e956ff6acc3e3aee6c189e4a839c3f)</a></li>
</ul>
<!--
<h3>Runnning the test suite</h3>
TODO
<h3>Generating coverage reports</h3>
TODO
//-->
</div>
<div class="section">
<h2>Generating Callgraphs</h2>
Call graphs are a visual representation of the architecture and are tremendously helpful in understanding
code and control flow in projects. This makes them highly useful to newcomers for gaining a high-level understanding
of how a project works, as well as to seasoned developers wanting to visualize how best to fit a particular feature
into the exisiting architecture.
<p>
The cmail main makefile contains a special target named <code>rtldumps</code> which
generates GCC rtldump files into the <code>rtldumps/</code> folder. These can be used to easily create call graphs
for all the cmail daemons with the help of <a href="https://github.com/cbdevnet/rtl2dot">rtl2dot</a> and
<a href="http://www.graphviz.org/">GraphViz</a>.
</p>
<p>
For example, to create a graph of the call hierarchy of cmail-smtpd, perform the following steps:
<ol>
<li>
In your cmail repository clone, run <code class="command">make rtldumps</code>.
This will create a folder named <code>rtldumps</code>, containing the raw data for the graph.
</li>
<li>
In the <code>rtldumps</code> folder, locate the file containing the smtpd data. It will
usually be named something like <code>smtpd.c.170r.expand</code>.
</li>
<li>
Create the graph data for feeding into GraphViz with
<code class="command">rtl2dot.py smtpd.c.170r.expand > smtpd.dot</code>. This will create <code>smtpd.dot</code>.
</li>
<li>
Create the final image with GraphViz by running <code class="command">dot -Tsvg smtpd.dot > smtpd.svg</code>.
This will create an SVG image of the call graph. Note that <code>dot</code> is also able to
create raster images in formats like PNG and JPG by supplying the appropriate -T<type> parameters.
</li>
</ol>
The latter steps can also be automated by pipelining the commands like so:
<code class="block">rtl2dot.py smtpd.c.170r.expand | dot -Tsvg > smtpd.svg</code>
</p>
<p>
The previous steps generate a full callgraph, containing all functions called via any one possible path from <code>main</code>.
This includes common functions such as <code>logprintf</code>, which are called from almost every function, eg. to print
messages to the log files, as well as library functions which are not directly part of the project.
<p>
</p>
In order to create a clearer visualisation of interesting code paths, it is necessary to exclude some functions from the graph.
<code>rtl2dot.py</code> supports exclusion of functions by their name with the <code>--ignore</code> argument,
as well as exclusion of library calls with the <code>--local</code> argument. A beautified callgraph can thus be
created by running <code class="block">rtl2dot.py smtpd.c.170r.expand --root core_loop --ignore "client_send|logprintf|common_*"
--local | dot -Tsvg > smtpd.svg</code> This also sets the root function of the graph to <code>core_loop</code>,
which is where the main processing functions are called in the cmail daemons.
</p>
</div>
</body>
</html>