-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathgetting_started.html
264 lines (238 loc) · 19.1 KB
/
getting_started.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Getting started — grid-control documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
<script src="_static/js/html5shiv.min.js"></script>
<![endif]-->
<script src="_static/jquery.js?v=5d32c60e"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="_static/documentation_options.js?v=5929fcd5"></script>
<script src="_static/doctools.js?v=9a2dae69"></script>
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="_static/js/theme.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="NAF specifics" href="NAF.html" />
<link rel="prev" title="Installation" href="installation.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="index.html" class="icon icon-home">
grid-control
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a><ul>
<li class="toctree-l2"><a class="reference internal" href="installation.html#requirements">Requirements</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html#installation-from-pypi-pip">Installation from PyPI (<code class="docutils literal notranslate"><span class="pre">pip</span></code>)</a></li>
<li class="toctree-l2"><a class="reference internal" href="installation.html#clone-from-git">Clone from git</a><ul>
<li class="toctree-l3"><a class="reference internal" href="installation.html#switch-to-a-different-branch">Switch to a different branch</a></li>
<li class="toctree-l3"><a class="reference internal" href="installation.html#pull-remote-changes">Pull remote changes</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Getting started</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#first-configuration-run-jobs-on-the-local-machine">First configuration (run jobs on the local machine)</a></li>
<li class="toctree-l2"><a class="reference internal" href="#further-examples">Further examples</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="NAF.html">NAF specifics</a><ul>
<li class="toctree-l2"><a class="reference internal" href="NAF.html#central-installation">Central installation</a></li>
<li class="toctree-l2"><a class="reference internal" href="NAF.html#backend">Backend</a></li>
<li class="toctree-l2"><a class="reference internal" href="NAF.html#submitting-to-centos7-nodes">Submitting to CentOS7 nodes</a></li>
<li class="toctree-l2"><a class="reference internal" href="NAF.html#custom-entries-in-condor-submit-file">Custom entries in condor submit file</a></li>
<li class="toctree-l2"><a class="reference internal" href="NAF.html#singularity-virtualization">Singularity (Virtualization)</a></li>
<li class="toctree-l2"><a class="reference internal" href="NAF.html#use-a-voms-proxy">Use a VOMS proxy</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="tips_tricks.html">Tips & tricks</a><ul>
<li class="toctree-l2"><a class="reference internal" href="tips_tricks.html#gnu-screen">GNU Screen</a></li>
<li class="toctree-l2"><a class="reference internal" href="tips_tricks.html#global-configuration">Global configuration</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="error_codes.html">Error codes</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">grid-control</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item active">Getting started</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/getting_started.rst.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section id="getting-started">
<h1>Getting started<a class="headerlink" href="#getting-started" title="Link to this heading"></a></h1>
<p>grid-control needs a configuration file written in a simple ini-style language. It can also be used with python configs or by directly calling using the python API, which is not covered here.</p>
<p>In the following, it is assumed that the executable is called <code class="docutils literal notranslate"><span class="pre">grid-control</span></code>. You might need to replace this with <code class="docutils literal notranslate"><span class="pre">go.py</span></code> or <code class="docutils literal notranslate"><span class="pre">gridcontrol</span></code> depending on your <a class="reference internal" href="installation.html"><span class="doc">installation</span></a>.</p>
<img alt="_images/gc_example.gif" src="_images/gc_example.gif" />
<section id="first-configuration-run-jobs-on-the-local-machine">
<h2>First configuration (run jobs on the local machine)<a class="headerlink" href="#first-configuration-run-jobs-on-the-local-machine" title="Link to this heading"></a></h2>
<p>This most basic example is not using a batch system, but running jobs on your local machine (<code class="docutils literal notranslate"><span class="pre">backend</span> <span class="pre">=</span> <span class="pre">host</span></code>). This can be useful for testing new configurations.</p>
<div class="literal-block-wrapper docutils container" id="id1">
<div class="code-block-caption"><span class="caption-text">host.conf</span><a class="headerlink" href="#id1" title="Link to this code"></a></div>
<div class="highlight-ini notranslate"><div class="highlight"><pre><span></span><span class="k">[global]</span>
<span class="na">task</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">UserTask</span><span class="w"> </span><span class="c1">; Job uses user written scripts</span>
<span class="na">backend</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">host</span><span class="w"> </span><span class="c1">; Run process(es) on the local machine</span>
<span class="k">[jobs]</span>
<span class="na">jobs</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">2</span><span class="w"> </span><span class="c1">; Submit script two times</span>
<span class="na">wall time</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">1:00</span><span class="w"> </span><span class="c1">; Jobs will take max 1h</span>
<span class="k">[UserTask]</span>
<span class="c1">; define what the jobs should do</span>
<span class="na">executable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">echo</span>
<span class="na">arguments</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">"I am job number @MY_JOBID@ and I ran on the machine $(hostname)"</span>
<span class="c1">; "echo" is a built-in command, so we don't have to send the executable</span>
<span class="na">send executable</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">False</span>
</pre></div>
</div>
</div>
<p>The configuration is divided into sections specified in brackets. The task specified in the <code class="docutils literal notranslate"><span class="pre">[global]</span></code> section is <code class="docutils literal notranslate"><span class="pre">UserTask</span></code>,
while the configuration of the specific user task is specified in the <code class="docutils literal notranslate"><span class="pre">[UserTask]</span></code> section.
Comments start with a <code class="docutils literal notranslate"><span class="pre">;</span></code>.</p>
<p>Now we create a task by calling <code class="docutils literal notranslate"><span class="pre">grid-control</span></code> and passing the configuration file as an argument. We use the <code class="docutils literal notranslate"><span class="pre">-s</span></code> command line flag to disable job sumbission for now. This is always a good idea to see how many jobs are generated etc. and check if there are any obvious errors.</p>
<div class="highlight-shell-session notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>grid-control<span class="w"> </span>-s<span class="w"> </span>host.conf
<span class="go">*****************************************************************</span>
<span class="go">* _ _ _ _ *</span>
<span class="go">* __ _ _ __(_) __| | ___ ___ _ __ | |_ _ __ ___ | | *</span>
<span class="go">* / _` | '__| |/ _` |____ / __/ _ \| '_ \| __| '__/ _ \| | *</span>
<span class="go">* | (_| | | | | (_| |____| (_| (_) | | | | |_| | | (_) | | *</span>
<span class="go">* \__, |_| |_|\__,_| \___\___/|_| |_|\__|_| \___/|_| *</span>
<span class="go">* |___/ *</span>
<span class="go">* The Swiss Army knife of job submission tools *</span>
<span class="go">* arXiv:1707.03198 *</span>
<span class="go">* *</span>
<span class="go">* https://github.com/grid-control/grid-control *</span>
<span class="go">*****************************************************************</span>
<span class="go">Revision: 1.9.89 (aa054081)</span>
<span class="go">Starting initialization of /home/jolange/playground/gc/work.host!</span>
<span class="go">Do you want to create the working directory /home/jolange/playground/gc/work.host? [YES/no]: yes</span>
<span class="go">Current task ID: GC5aeb5265a4fe</span>
<span class="go">Current workdir: /home/jolange/playground/gc/work.host</span>
<span class="go">Task started on: 2019-06-26</span>
<span class="go">Using batch system: HOST</span>
<span class="go">-----------------------------------------------------------------</span>
<span class="go">REPORT SUMMARY: host / GC5aeb5265a4fe</span>
<span class="go">---------------</span>
<span class="go">Total number of jobs: 2 Successful jobs: 0 0%</span>
<span class="go">Jobs being processed: 0 Failing jobs: 0 0%</span>
<span class="go">Detailed Status Information:</span>
<span class="go">Jobs INIT: 2 100% Jobs SUBMITTED: 0 0%</span>
<span class="go">Jobs DISABLED: 0 0% Jobs READY: 0 0%</span>
<span class="go">Jobs WAITING: 0 0% Jobs QUEUED: 0 0%</span>
<span class="go">Jobs ABORTED: 0 0% Jobs RUNNING: 0 0%</span>
<span class="go">Jobs CANCEL: 0 0% Jobs UNKNOWN: 0 0%</span>
<span class="go">Jobs CANCELLED: 0 0% Jobs DONE: 0 0%</span>
<span class="go">Jobs FAILED: 0 0% Jobs SUCCESS: 0 0%</span>
<span class="go">-----------------------------------------------------------------</span>
<span class="go">2019-06-26 14:14:46 - Time left for access token "AFSAccessToken": 23h 54min 31sec</span>
</pre></div>
</div>
<p>The task has been initialzed and a working directory <code class="file docutils literal notranslate"><span class="pre">work.host</span></code> has been created to capture the state of the current task.
Two jobs have been created (both in the INIT state), because we explicitly specified this.
Everything looks fine and we can actually submit the jobs: We invoke <code class="docutils literal notranslate"><span class="pre">grid-control</span></code> again, but without the <code class="docutils literal notranslate"><span class="pre">-s</span></code> flag to enable job submission. The current task state will be picked up from the working directory corresponding to the configuration file.</p>
<div class="highlight-shell-session notranslate"><div class="highlight"><pre><span></span><span class="gp">$ </span>grid-control<span class="w"> </span>-c<span class="w"> </span>host.conf
<span class="go">*****************************************************************</span>
<span class="go">* _ _ _ _ *</span>
<span class="go">* __ _ _ __(_) __| | ___ ___ _ __ | |_ _ __ ___ | | *</span>
<span class="go">* / _` | '__| |/ _` |____ / __/ _ \| '_ \| __| '__/ _ \| | *</span>
<span class="go">* | (_| | | | | (_| |____| (_| (_) | | | | |_| | | (_) | | *</span>
<span class="go">* \__, |_| |_|\__,_| \___\___/|_| |_|\__|_| \___/|_| *</span>
<span class="go">* |___/ *</span>
<span class="go">* The Swiss Army knife of job submission tools *</span>
<span class="go">* arXiv:1707.03198 *</span>
<span class="go">* *</span>
<span class="go">* https://github.com/grid-control/grid-control *</span>
<span class="go">*****************************************************************</span>
<span class="go">Revision: 1.9.89 (aa054081)</span>
<span class="go">Current task ID: GC5aeb5265a4fe</span>
<span class="go">Current workdir: /home/jolange/playground/gc/work.host</span>
<span class="go">Task started on: 2019-06-26</span>
<span class="go">Using batch system: HOST</span>
<span class="go">-----------------------------------------------------------------</span>
<span class="go">REPORT SUMMARY: host / GC5aeb5265a4fe</span>
<span class="go">---------------</span>
<span class="go">Total number of jobs: 2 Successful jobs: 0 0%</span>
<span class="go">Jobs being processed: 0 Failing jobs: 0 0%</span>
<span class="go">Detailed Status Information:</span>
<span class="go">Jobs INIT: 2 100% Jobs SUBMITTED: 0 0%</span>
<span class="go">Jobs DISABLED: 0 0% Jobs READY: 0 0%</span>
<span class="go">Jobs WAITING: 0 0% Jobs QUEUED: 0 0%</span>
<span class="go">Jobs ABORTED: 0 0% Jobs RUNNING: 0 0%</span>
<span class="go">Jobs CANCEL: 0 0% Jobs UNKNOWN: 0 0%</span>
<span class="go">Jobs CANCELLED: 0 0% Jobs DONE: 0 0%</span>
<span class="go">Jobs FAILED: 0 0% Jobs SUCCESS: 0 0%</span>
<span class="go">-----------------------------------------------------------------</span>
<span class="go">Running in continuous mode. Press ^C to exit.</span>
<span class="go">2019-06-26 14:15:14 - Time left for access token "AFSAccessToken": 23h 54min 03sec</span>
<span class="go">2019-06-26 14:15:14 - Job 0 state changed from INIT to SUBMITTED</span>
<span class="go">2019-06-26 14:15:14 - Job 1 state changed from INIT to SUBMITTED</span>
<span class="go">2019-06-26 14:15:20 - Job 0 state changed from SUBMITTED to DONE</span>
<span class="go">2019-06-26 14:15:20 - Job 1 state changed from SUBMITTED to DONE</span>
<span class="go">2019-06-26 14:15:25 - Job 0 state changed from DONE to SUCCESS (runtime 0h 00min 00sec)</span>
<span class="go">2019-06-26 14:15:25 - Job 1 state changed from DONE to SUCCESS (runtime 0h 00min 00sec)</span>
<span class="go">2019-06-26 14:15:30 - Task successfully completed. Quitting grid-control!</span>
<span class="go">2019-06-26 14:15:30 - Workdir was /home/jolange/playground/gc/work.host</span>
</pre></div>
</div>
<p>While grid-control is running, it checks the status of running jobs, retrieves and analyzes finished jobs and then submits jobs that are ready. We used the continuous mode (<code class="docutils literal notranslate"><span class="pre">-c</span></code>) to let grid-control repeat this until the task is finished. We can interrupt this by pressing <kbd class="kbd compound docutils literal notranslate"><kbd class="kbd docutils literal notranslate">Ctrl</kbd>-<kbd class="kbd docutils literal notranslate">C</kbd></kbd> and restart it by just calling <code class="docutils literal notranslate"><span class="pre">grid-control</span> <span class="pre">[-s]</span> <span class="pre"><conf></span></code> again.</p>
</section>
<section id="further-examples">
<h2>Further examples<a class="headerlink" href="#further-examples" title="Link to this heading"></a></h2>
<p>Further example configurations can be found here:
<a class="reference external" href="https://github.com/grid-control/grid-control/tree/master/docs/examples">https://github.com/grid-control/grid-control/tree/master/docs/examples</a></p>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="installation.html" class="btn btn-neutral float-left" title="Installation" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="NAF.html" class="btn btn-neutral float-right" title="NAF specifics" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2019, Johannes Lange.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(false);
});
</script>
</body>
</html>