Permit increasing FD_SETSIZE at build time

Add a -DLARGE_FD_SETSIZE option to set the value of FD_SETSIZE at
build time.  The default system value is sometimes not enough for busy
news servers.

Thanks to Jesse Rehmer for the bug report.  Patch taken from Diablo,
and adapted for INN.

The long-term solution is of course to switch to libevent, as this
patch is a terrible hack.

see #273

doc/pod/inn.conf.pod

@@ -1645,6 +1645,12 @@ Solaris versions prior to 11.0.  See the L<Solaris documentation about file
 descriptors|https://support.oracle.com/knowledge/Sun%20Microsystems/1005979_1.html>
 for more details.
 
+Note for expert users building INN from sources: if you need using more than
+C<FD_SETSIZE> file descriptors (as defined in the F<sys/select.h> system
+header, usually defaulting to 1024), you can increase this value for instance
+to C<4096> by rebuilding INN with the C<-DLARGE_FD_SETSIZE=4096> option given
+to the compiler.
+
 =back
 
 =head2 Paths Names

doc/pod/install.pod

@@ -1725,6 +1725,11 @@ C<1024> (and may need to if you have a particularly large site), but that
 can cause RPC and some stdio applications to break.  It therefore probably
 isn't a good idea on a machine that isn't dedicated to INN.
 
+Note for expert users building INN from sources: if you need using more than
+C<FD_SETSIZE> file descriptors, you can increase this value for instance to
+C<4096> by building INN with the C<-DLARGE_FD_SETSIZE=4096> option given to
+the compiler.
+
 =head1 Starting and Stopping the System
 
 INN is started via the shell script B<rc.news>.  This must be run as the

doc/pod/news.pod

@@ -38,6 +38,13 @@ NoCeM issuers, and make sure the right PGP keys are present on your system.
 Fixed a hang when posting articles if COMPRESS DEFLATE is active but TLS
 is not.  Thanks to Enrik Berkhan for the patch for B<nnrpd>.
 
+=item *
+
+If needing to use more file descriptors than the default system limit, a new
+C<LARGE_FD_SETSIZE> option can be set at build time.  See the documentation
+for I<rlimitnofile> in F<inn.conf> for more information.  Thanks to Jesse
+Rehmer for the bug report.
+
 =back
 
 =head1 Changes in 2.7.1 (2023-04-16)

include/portable/sd-daemon.h

@@ -48,6 +48,18 @@ int sd_notify(int, const char *);
 int sd_notifyf(int, const char *, ...);
 #endif
 
+/*
+ * See portable/system.h for more information about the following code.
+ *
+ * systemd/sd-daemon.h includes linux/posix_types.h which unconditionally
+ * redefines __FD_SETSIZE to 1024 (even if already set), so we need redefining
+ * it again, if wanted.
+ */
+#if defined(HAVE_SD_NOTIFY) && defined(__linux__) && LARGE_FD_SETSIZE > 1024
+#    undef __FD_SETSIZE
+#    define __FD_SETSIZE LARGE_FD_SETSIZE
+#endif
+
 END_DECLS
 
 #endif /* !PORTABLE_SD_DAEMON_H */

include/portable/system.h

@@ -46,6 +46,39 @@
 /* BEGIN_DECL and __attribute__. */
 #include "portable/macros.h"
 
+/*
+ * Provide a way to increase FD_SETSIZE at build time for servers needing more
+ * than the usual default of 1024 file descriptors.
+ * It otherwise leads to corruption of the tracking of open file descriptors
+ * in programs like innd because fd_set is not large enough.
+ * Use for instance -DLARGE_FD_SETSIZE=4096 at build time to increase the
+ * limit to 4096.
+ *
+ * FD_SETSIZE cannot be increased on Linux, but __FD_SETSIZE can with
+ * glibc 2.2 and also probably later versions.  We do this by including
+ * bits/types.h which defines __FD_SETSIZE first (before any other include),
+ * then we redefine __FD_SETSIZE.  Of course, a user program may *never*
+ * include bits/whatever.h directly, so this is a dirty hack!
+ * We assume that on systems other than Linux, just defining FD_SETSIZE works.
+ * Idea taken from Diablo which also has running instances needing more than
+ * 1024 files descriptors.
+ * Naturally, the right long-term solution is to switch to libevent which
+ * handles select/poll/epoll/etc. and does not suffer from the hard-coded
+ * fd_set size for select(2) calls.
+ */
+#if LARGE_FD_SETSIZE > 1024
+#    if defined(__linux__)
+#        include <features.h>
+#        if (__GLIBC__ > 2) || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 2)
+#            include <bits/types.h>
+#            undef __FD_SETSIZE
+#            define __FD_SETSIZE LARGE_FD_SETSIZE
+#        endif
+#    else
+#        define FD_SETSIZE LARGE_FD_SETSIZE
+#    endif
+#endif
+
 /* A set of standard ANSI C headers.  We don't care about pre-ANSI systems. */
 #if HAVE_INTTYPES_H
 #    include <inttypes.h>

support/getrra-c-util

@@ -119,6 +119,23 @@ AC_DEFUN([_INN_LIB_SQLITE3_INTERNAL],\
                         ${TEMP}
                 fi
 
+                if [ "$3" = "sd-daemon.h" ]; then
+                    sed -i -e '53 i \
+/*\
+ * See portable/system.h for more information about the following code.\
+ *\
+ * systemd/sd-daemon.h includes linux/posix_types.h which unconditionally\
+ * redefines __FD_SETSIZE to 1024 (even if already set), so we need redefining\
+ * it again, if wanted.\
+ */\
+#if defined(HAVE_SD_NOTIFY) && defined(__linux__) && LARGE_FD_SETSIZE > 1024\
+#    undef __FD_SETSIZE\
+#    define __FD_SETSIZE LARGE_FD_SETSIZE\
+#endif\
+' \
+                        ${TEMP}
+                fi
+
                 if [ "$3" = "socket.h" ]; then
                     sed -i -e 's/  If we.*visibility//g' \
                         -e '190d' \
@@ -134,7 +151,41 @@ AC_DEFUN([_INN_LIB_SQLITE3_INTERNAL],\
 
                 if [ "$3" = "system.h" ]; then
                     # Add INN-specific stuff.
-                    sed -i -e "108 i \\
+                    sed -i -e '49 i \
+/*\
+ * Provide a way to increase FD_SETSIZE at build time for servers needing more\
+ * than the usual default of 1024 file descriptors.\
+ * It otherwise leads to corruption of the tracking of open file descriptors\
+ * in programs like innd because fd_set is not large enough.\
+ * Use for instance -DLARGE_FD_SETSIZE=4096 at build time to increase the\
+ * limit to 4096.\
+ *\
+ * FD_SETSIZE cannot be increased on Linux, but __FD_SETSIZE can with\
+ * glibc 2.2 and also probably later versions.  We do this by including\
+ * bits/types.h which defines __FD_SETSIZE first (before any other include),\
+ * then we redefine __FD_SETSIZE.  Of course, a user program may *never*\
+ * include bits/whatever.h directly, so this is a dirty hack!\
+ * We assume that on systems other than Linux, just defining FD_SETSIZE works.\
+ * Idea taken from Diablo which also has running instances needing more than\
+ * 1024 files descriptors.\
+ * Naturally, the right long-term solution is to switch to libevent which\
+ * handles select/poll/epoll/etc. and does not suffer from the hard-coded\
+ * fd_set size for select(2) calls.\
+ */\
+#if LARGE_FD_SETSIZE > 1024\
+#    if defined(__linux__)\
+#        include <features.h>\
+#        if (__GLIBC__ > 2) || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 2)\
+#            include <bits/types.h>\
+#            undef __FD_SETSIZE\
+#            define __FD_SETSIZE LARGE_FD_SETSIZE\
+#        endif\
+#    else\
+#        define FD_SETSIZE LARGE_FD_SETSIZE\
+#    endif\
+#endif\
+' \
+                        -e "108 i \\
 /*\\
  * This almost certainly isn't necessary, but it's not hurting anything.\\
  * gcc assumes that if SEEK_SET isn't defined none of the rest are either,\\