Skip to content

Commit

Permalink
Documentation fix for innfeed backlog files
Browse files Browse the repository at this point in the history
What innfeed actually does was not accurate.
  • Loading branch information
Julien-Elie committed Jun 26, 2023
1 parent 6a8a48c commit e370638
Show file tree
Hide file tree
Showing 3 changed files with 39 additions and 32 deletions.
16 changes: 9 additions & 7 deletions doc/pod/innfeed.conf.pod
Original file line number Diff line number Diff line change
Expand Up @@ -240,10 +240,10 @@ before starting to write new entries to disk.

=item I<backlog-ckpt-period>

This key requires a positive integer value and defaults to C<30>.
It specifies how many seconds elapse between checkpoints of the input
backlog file. Too small a number will mean frequent disk accesses; too
large a number will mean after a crash, B<innfeed> will re-offer more
This key requires a positive integer value and defaults to C<30>. It
specifies how many seconds elapse between checkpoints and rewrites of the
input backlog file. Too small a number will mean frequent disk accesses;
too large a number will mean after a crash, B<innfeed> will re-offer more
already-processed articles than necessary.

=item I<backlog-newfile-period>
Expand All @@ -255,9 +255,11 @@ generated backlog files that are to be picked up and processed.
=item I<backlog-rotate-period>

This key requires a positive integer value and defaults to C<60>.
It specifies how many seconds elapse before B<innfeed> checks for a
manually created backlog file and moves the output backlog file to the
input backlog file.
It specifies how many seconds elapse before B<innfeed> moves an externally
generated backlog file to the input backlog file (if I<backlog-newfile-period>
seconds have elapsed) or in the absence of such a file, moves the output
backlog file to the input backlog file. No moves occur if the input backlog
file is not empty.

=item I<dns-retry>

Expand Down
22 changes: 14 additions & 8 deletions doc/pod/innfeed.pod
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,11 @@ B<innfeed> expects a couple of things to be able to run correctly: a
directory where it can store backlog files and a configuration file to
describe which peers it should handle.

The configuration file is described in innfeed.conf(5). The B<-c> option
can be used to specify a different file. For each peer (say, C<foo>),
B<innfeed> manages up to 4 files in the backlog directory:
The configuration file is described in innfeed.conf(5). The B<-c> option can
be used to specify a different file, and B<-b> to specify a different backlog
directory. The I<backlog-*> keys in the configuration file parameterize the
behaviour of backlogging. For each peer (say, C<foo>), B<innfeed> manages up
to 4 files in the backlog directory:

=over 2

Expand All @@ -92,14 +94,18 @@ for re-processing.
=item *

A F<foo.output> file where B<innfeed> is writing information on articles
that could not be processed (normally due to a slow or blocked peer).
that could not be processed (normally due to a slow or blocked peer). Every
I<backlog-rotate-period> seconds, B<innfeed> checks whether it is not empty,
and, if no F<foo> file exists and F<foo.input> is empty, will then rename
F<foo.output> to F<foo.input> and start reading from it.

=item *

A F<foo> file that is never created by B<innfeed>, but if B<innfeed>
notices it, it will rename it to F<foo.input> at the next opportunity and
will start reading from it. This lets you create a batch file and put it
in a place where B<innfeed> will find it.
A F<foo> file that is never created by B<innfeed>, but if B<innfeed> notices
it when checking every I<backlog-newfile-period> seconds, it will rename it to
F<foo.input> at the next opportunity (every I<backlog-rotate-period> seconds
if F<foo.input> is empty) and will start reading from it. This lets you
create a batch file and put it in a place where B<innfeed> will find it.

=back

Expand Down
33 changes: 16 additions & 17 deletions innfeed/tape.c
Original file line number Diff line number Diff line change
Expand Up @@ -261,14 +261,13 @@ tapeConfigLoadCbk(void *data)
}


/* Create a new Tape object. There are three potential files involved in
I/O. 'peerName' is what the admin may have dropped in by
hand. 'peerName.input' is the file that was being used as input the last
time things were run. 'peerName.output' is the file that was being used
as output. The file 'peerName' is appended to 'peerName.input' (or
renamed if 'peerName.input' doesn't exist). Then 'peerName.output' is
appeneded (or renamed) to 'peerName.input' */

/* Create a new Tape object. There are three potential files involved in I/O.
* 'peerName' is what the admin may have dropped in by hand.
* 'peerName.input' is the file that was being used as input the last
* time things were run.
* 'peerName.output' is the file that was being used as output.
* See prepareFiles() for how they are used.
*/
static bool inited = false;
Tape
newTape(const char *peerName, bool dontRotate)
Expand Down Expand Up @@ -921,20 +920,20 @@ checkpointTape(Tape tape)

/* Prepare the tape file(s) for input and output */

/* For a given Tape there are
* three possible files: PEER.input PEER and
* PEER.output. PEER.input and PEER.output are private to
* innfeed. PEER is where a sysadmin can drop a file that (s)he
* wants to inject into the process. If the first line of the input file
* contains only an integer (possibly surrounded by spaces), then this is
* taken to be the position to seek to before reading.
/* For a given Tape, there are three possible files:
* PEER.input, PEER, and PEER.output.
* PEER.input and PEER.output are private to innfeed.
* PEER is where a sysadmin can drop a file that (s)he wants to inject into
* the process. If the first line of the input file contains only an integer
* (possibly surrounded by spaces), then this is taken to be the position to
* seek to before reading.
*
* prepareFiles will process them in a manner much like the following shell
* prepareFiles() will process them in a manner much like the following shell
* commands:
*
* if [ ! -f PEER.input ]; then
* if [ -f PEER ]; then mv PEER PEER.input
* elif [ -f PEER.output ]; then mv PEER.output PEER; fi
* elif [ -f PEER.output ]; then mv PEER.output PEER.input; fi
* fi
*
* At this point PEER.input is opened for reading if it exists.
Expand Down

0 comments on commit e370638

Please sign in to comment.