Skip to content

Commit

Permalink
pullnews: Fix Bytes computation and rnews files
Browse files Browse the repository at this point in the history
The value of the Bytes header field was not right (CRLF was counted as
1 byte).

rnews batches were not using the expected LF line termination, and
generated header fields (Bytes, Xref) were not using the expected CRLF
line termination.

close #285
  • Loading branch information
Julien-Elie committed Dec 9, 2023
1 parent bb83b08 commit 541f672
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 12 deletions.
10 changes: 10 additions & 0 deletions doc/pod/news.pod
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,16 @@ is not. Thanks to Enrik Berkhan for the patch for B<nnrpd>.

=item *

Fixed the computation of the Bytes header field by B<pullnews> (in header-only
mode with the B<-B> flag).

=item *

Fixed the generation of B<rnews> batches by B<pullnews> (when using the B<-r>
flag) which had not the expected native LF line termination.

=item *

B<innd> no longer malfunctions nor throttles when the maximum number of file
descriptors supported by the system is reached. If needing to use more file
descriptors than the default system limit, a new C<LARGE_FD_SETSIZE> option
Expand Down
34 changes: 22 additions & 12 deletions frontends/pullnews.in
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@
# Full changelog can be found in the Git commit history of the
# INN project. Major changes are:
#
# December 2023:
# Julien Élie fixed the computation of the Bytes header field,
# the use of trailing LF in rnews batches (-r option), and
# trailing CRLF in generated header fields.
#
# February-March 2023:
# Julien Élie added TLS support for both downstream and upstream
# servers. Also made pullnews robust on socket timeout and lock
Expand Down Expand Up @@ -1008,6 +1013,7 @@ sub crossFeedGroup {
for (my $idx = 0; $idx < @{$article}; $idx++) {
$line_len = length($article->[$idx]);
$len += $line_len;
# Dot-stuffing already undone by Net::NNTP.
$tx_len += $line_len;
$info{server}->{$server}->{bytes} += $line_len;
$info{bytes} += $line_len;
Expand Down Expand Up @@ -1150,7 +1156,7 @@ sub crossFeedGroup {
my $xref_h
= "Xref: "
. $upstreamParams->{$server}->{name}
. " $group:$i\n";
. " $group:$i\r\n";
splice(@{$article}, $idx_blank_pre_body, 0, $xref_h);
$tx_len += length($xref_h);
$idx_blank_pre_body++;
Expand All @@ -1159,12 +1165,11 @@ sub crossFeedGroup {
if ($add_bytes_header) {
# Compute the number of bytes the same way the :bytes
# metadata item would do. The additional Bytes header
# field is not counted, as well as header fields removed by
# pullnews.
my $bytes_real_count = $tx_len + scalar(@{$article});
my $bytes_h = "Bytes: $bytes_real_count\n";
# field is not counted (as it does not form part of the
# article in a header-only feed), as well as header fields
# removed by pullnews.
my $bytes_h = "Bytes: $tx_len\r\n";
splice(@{$article}, $idx_blank_pre_body, 0, $bytes_h);
$tx_len += length($bytes_h);
$idx_blank_pre_body++;
}

Expand All @@ -1173,10 +1178,6 @@ sub crossFeedGroup {
and @{$article} > $idx_blank_pre_body + 1)
{
splice(@{$article}, $idx_blank_pre_body + 1);
$tx_len = 0;
for my $line (@{$article}) {
$tx_len += length($line);
}
}

if (not $skip_due_to_hdrs and ref $hashfeed) {
Expand Down Expand Up @@ -1207,8 +1208,17 @@ sub crossFeedGroup {
) . "\n";
}
} elsif ($rnews) {
printf RNEWS "#! rnews %d\n", $tx_len;
map { print RNEWS $_ } @{$article};
# Change all trailing CRLF to LF as rnews expects articles
# in native format (rnews converts them to wire format when
# processing them).
# Net::NNTP has already removed dot-stuffing.
my $artwithoutCR = "";
foreach my $line (@{$article}) {
$line =~ s/\r\n$/\n/;
$artwithoutCR .= $line;
}
printf RNEWS "#! rnews %d\n", length($artwithoutCR);
print RNEWS $artwithoutCR;
print LOG "+" unless $quiet;
$fed{$group}++;
$info{'rnews'}->{fed}++;
Expand Down

0 comments on commit 541f672

Please sign in to comment.