NEWSBATCH(8)

NAME
sendbatches, batchsplit - news batching to other sites
batcher, batchih, batchsm, batchra - news-batch preparation
comp, comp12, nocomp, compcun, compc7, compb - news-batch compression
c7encode, bencode - compressed-news-batch encoding
viauux, viauuxz, viauuxcun, viauuxl - news-batch transmission via uucp
viamail, viaemail, viapmail - news-batch transmission via mail
viainews, viarsh, vianowhere - news-batch transmission by misc. means

SYNOPSIS
/usr/lib/newsbin/batch/sendbatches [ -p ] [ site ... ]

.../batchsplit batchsize
.../batcher listfile
.../batchih listfile
.../batchsm listfile
.../batchra listfile
.../comp
.../comp12
.../nocomp
.../compcun
.../compc7
.../compb
.../c7encode
.../bencode [ file ]
.../viauux site
.../viauuxz site
.../viauuxcun site
.../viauuxl group
.../viamail site
.../viaemail site
.../viapmail site
.../viainews site
.../viarsh site
.../vianowhere site

DESCRIPTION
Sendbatches administers batched transmission of news to other sites. It should be run periodically, under userid news, by cron(8) or similar means. It prepares and sends batches of news, subject to restrictions on available space and length of outbound queues.

Normally, sendbatches does locking to ensure that only one sendbatches is running at a time. The -p option suppresses the locking and permits parallel sendbatches runs, although lower-level locking is done to ensure that only one is trying to prepare batches for any particular site at a given time. Parallel sendbatches runs impose very heavy system loads but may be useful to systems with extensive hardware parallelism and many outbound news feeds.

Each site that can have batches sent to it needs a site directory under /usr/spool/news/out.going. If sendbatches is invoked with specific sites given, it considers batching for those sites, only, in that order. By default, sendbatches consults the batchparms file (see below) to determine what to do: If there is a /default/ entry in batchparms, sendbatches will consider batching for all sites that have directories in /usr/spool/news/out.going, in oldest-first order by modification time of the directory. If there is no /default/ entry, sendbatches considers batching for those sites named in batchparms, in the order named.

To use the batcher, names of files to be sent to a specific site should be appended to a togo file in its site directory. The batcher expects the lines in togo to have two fields, a filename (as a full pathname, or relative to /usr/spool/news) of an article and its size in bytes. A missing size field is arbitrarily assumed to be a default average.

Sendbatches uses a number of auxiliary programs to do the real work. The search path it uses to find them includes, in order, the site directory for the site in question, /usr/lib/news/bin, and /usr/lib/newsbin/batch. This permits per-site and per-news-database overrides of the default behaviors. Sendbatches provides all these programs with environment variables NEWSSITE, containing the name of the site that batches are being prepared for, and NEWSSITEDIR, containing the full pathname of the site directory, in case these are useful in customization.

The names of most of the auxiliary programs, and some other parameters, are taken from the file /usr/lib/news/batchparms, an ASCII text file. Empty lines and lines starting with `#' are ignored. Other lines specify the behavior for sites, one line per site. A site line is six fields, separated by white space. The first field is the site name; a line whose site name is /default/ specifies what parameters should be used for sites not explicitly mentioned. (The presence or absence of such a line also influences the behavior of sendbatches when invoked without arguments; see above.) The second field is the size of batches to be prepared (before compression), in bytes. The third field is the maximum length of the output queue for transmission to that site. The fourth, fifth, and sixth fields are the names of the programs to be used to build, compress, and transmit (respectively) batches to that site.

For each site being considered for batches, sendbatches first determines whether there are in fact any articles to be batched. Assuming there are, sendbatches then finds the batchparms line for that site and invokes queuelen (see newsaux(8)) to find out the size of the outbound queue for the site. Sendbatches limits the number of batches prepared to the minimum of the limits implied by queue lengths and available space.

Sendbatches uses batchsplit as necessary to slice chunks out of the togo file, each chunk containing the togo lines for a batch limited to the specified size. Exception: a single article bigger than the specified size will still go out as one batch.

Each chunk is then processed through a batch preparer (typically batcher), which assembles the articles into a batch, a batch compressor (typically compcun), which performs compression and/or any other auxiliary processing needed, and a batch transmitter (typically viauux), which sends the batch on its way (e.g. enqueues it for transmission). The preparer is run with /usr/spool/news as the current directory, so non-absolute pathnames in the chunk are valid filenames; the others are run in the site directory for easier access to site-specific customization files.

Batch preparers in the standard distribution are:

    batcher
        normal batching

    batchih
        ihave-sending part of uucp ihave/sendme (not to be confused with NNTP)

    batchsm
        sendme-sending part of ihave/sendme

    batchra
        requested-article-sending part of ihave/sendme

Batchih, batchsm, and batchra have to map from the phony ``site name'' given in their batchparms line to the name of the site they should actually send to; they do this by stripping off the last `.' and everything that follows (usually `.ihave' or `.sendme' respectively, but on machines which limit the size of filenames these may have to be shortened).

Caution: batchih and batchsm do their work by constructing an article and feeding it to inews, which means that the batch size must be within what inews will accept (see inews(1)).

Batch compressors in the standard distribution are:

    comp
        ordinary compression

    comp12
        12-bit compression (for sites with limited decompression ability)

    compcun
        compression plus the silly B-news-compatible `#! cunbatch' header

    nocomp
        pass data straight through, no compression

    compc7
        compression plus 7-bit encoding using c7encode

    compb
        compression plus very conservative 6-bit encoding using bencode

Batch transmitters in the standard distribution are:

    viauux
        normal transmission via UUCP

    viauuxz
        like viauux except with -z option given to uux (for old UUCPs where don't-report-result-on-zero-status is not default)

    viauuxcun
        like viauux except it invokes cunbatch rather than rnews at the other end (for some very old news sites)

    viauuxl
        multicast transmission using the -l option of uux (not found on all systems) to send the same batch to all systems listed in the file `/usr/lib/news/sites.group'

    viamail
        mail the batch to site!rnews

    viaemail
        mail the batch to site!enews

    viapmail
        mail the batch to site!rnews, attempting to protect an unencoded batch against the vagaries of mailers

    viainews
        feed the batch back to inews, ignoring the site argument (normally useful only for ihave/sendme)

    viarsh
        use rsh to run rnews on the site via Ethernet, Internet, etc. (the directory containing rnews must be in the default PATH on site)

    vianowhere
        dummy transmitter for batchers like batchra that don't need a real one

C7encode encodes 8-bit data from standard input into a 7-bit form optimized for transmission by uucp `f' protocol. The encoding is complex and bizarre.

Bencode takes 8-bit data from the named source file (the default is standard input) and encodes it using only the ASCII characters ``A'' - ``Z'', ``a'' - ``z'', ``0'' - ``9'', ``+'', and ``-''. The ASCII characters blank, newline, and ``/'' also appear in the encoded file, but do not represent encoded bits. The encoded file is terminated with a byte count and cyclic redundancy check for detecting corrupted files. This ought to suffice to get data through almost any network.

Sendbatches logs some information about sites with backlogs in /usr/lib/news/batchlog (see also newsdaily(8)). This is intended to help detection and diagnosis of flow problems.

FILES
/usr/lib/news/LOCKbatch	overall lock for sendbatches
/usr/lib/news/LOCK	overall news lock (used by batchsplit)
/usr/lib/news/L.*	lock temporaries
/usr/spool/news/out.going/*	batch directories
/usr/spool/news/out.going/*/LOCK	per-site locks
/usr/spool/news/out.going/*/L.*	per-site lock temporaries
/usr/lib/news/batchparms	parameter file
/usr/lib/news/sites.*	viauuxl multicast lists
also see text

SEE ALSO
inews(1), compress(1), uux(1), relaynews(8), rnews(8)

DIAGNOSTICS
Complaints, if any, from sendbatches and its auxiliaries are mailed to `usenet'.

HISTORY
Written at University of Toronto as part of the C News project. A number of the ideas came from Chris Lewis, who was then with Motorola. Bencode written at University of Waterloo by Reg Quinton and Ken Lalonde.

BUGS
Sendbatches estimates available space without considering the effects of compression; this is usually too conservative.

Viapmail is obsolescent; viaemail with a batch encoded with bencode is vastly superior.

Queue-length limiting does not work for viauuxl because queuelen (see newsaux(8)) does not know about multicast groups. Also, viauuxl has not been tested well.

Viarsh does not incorporate a spooling subsystem, so a slow site stalls the entire batching system and a non-responding site loses news. It is not recommended for bulk transmission or where high reliability is essential.