  ULOGD 2.x - the Netfilter Userspace Logging Daemon
  Harald Welte <laforge@netfilter.org>, Eric Leblond <eric@inl.fr>
  Revision 2009/04/18

  This is the documentation for ulogd-2.x, the second generation
  Netfilter Userspace logging daemon. ulogd makes use of the Linux >=
  2.6.14 nfnetlink_log and nfnetlink_conntrack subsystems.
  ______________________________________________________________________

  Table of Contents


  1. DESIGN
     1.1 CONCEPT
     1.2 DETAILS

  2. INSTALLATION
     2.1 Linux kernel
     2.2 Userspace libraries
     2.3 ulogd
        2.3.1 Recompiling the source

  3. Configuration
     3.1 iptables NFLOG target
        3.1.1 Quick Setup
        3.1.2 NFLOG target reference
     3.2 ulogd
        3.2.1 ulogd configfile syntax reference
        3.2.2 ulogd commandline option reference

  4. Signals / Logrotate
  5. Available plugins
     5.1 Input plugins
        5.1.1 ulogd_inppkt_NFLOG.so
        5.1.2 ulogd_inpflow_NFCT.so
     5.2 Interpreter plugins
        5.2.1 ulogd_raw2packet_BASE.so
        5.2.2 ulogd_filter_PWSNIFF.so
        5.2.3 ulogd_filter_IFINDEX.so
        5.2.4 ulogd_LOCAL.so
        5.2.5 ulogd_filter_HWHDR.so
        5.2.6 ulogd_filter_IP2BIN.so
        5.2.7 ulogd_filter_IP2HBIN.so
        5.2.8 ulogd_filter_IP2STR.so
        5.2.9 ulogd_filter_PRINTFLOW.so
        5.2.10 ulogd_filter_PRINTPKT.so
        5.2.11 ulogd_filter_MARK.so
     5.3 Output plugins
        5.3.1 ulogd_output_OPRINT.so
        5.3.2 ulogd_output_LOGEMU.so
        5.3.3 ulogd_output_MYSQL.so
        5.3.4 ulogd_output_PGSQL.so
        5.3.5 ulogd_output_PCAP.so
        5.3.6 ulogd_output_SQLITE3.so
        5.3.7 ulogd_output_SYSLOG.so

  6. QUESTIONS / COMMENTS

  ______________________________________________________________________

  11..  DDEESSIIGGNN

  11..11..  CCOONNCCEEPPTT

  ulogd-2.x wants to provide a flexible, almost universal logging daemon
  for netfilter logging. This encompasses both packet-based logging
  (logging of policy violations) and flow-based logging, e.g. for
  accounting purpose.

  ulogd consists of a small core and a number of plugins. All the real
  power lies in the plugins, and in the user who configures the
  interactions between those plugins.


     IInnppuutt PPlluuggiinnss
        Input plugins acts data source. They get data from somewhere
        outside of ulogd, and convert it into a list of ulogd keys.

     FFiilltteerr PPlluuggiinnss
        Filter plugins interpret and/or filter data that was received
        from the Input Plugin. A good example is parsing a raw packet
        into IPv4 / TCP / ... header information.

     OOuuttppuutt PPlluuggiinnss
        Output plugins describe how and where to put the information
        gained by the Input Plugin and processed by one or more Filter
        Plugins.  The easiest way is to build a line per packet and
        fprint it to a file.  Some people might want to log into a SQL
        database or want an output conforming to the IETF IPFIX
        language.

  By means of the configuration file, the administrator can build any
  number of Plugin Stacks. A plugin stack is a series of plugins,
  starting with an Input plugin, none, one or multiple filter plugins,
  and one output plugin on top.

  11..22..  DDEETTAAIILLSS

  The major clue is providing a framework which is as flexible as
  possible.  Nobody knows what strange network protocols are out there
  :) But at the same time, logging of a packet filter is often very
  performance critical.  Like all ulogd releases since 0.3.x, the
  ulogd-2.x core doesn't do any dynamic allocations at runtime. Yes,
  obviously, at startup time the config file is parsed, and allocations
  are made. But after that, everything is pre-allocated. As an
  additional improvement over ulogd-1.x, there are also no hashtable
  lookups for key resolval. All input/output keys of plugins within
  every stack are resolved at config file parsing time, and directly
  interconnected by pointers.

  22..  IINNSSTTAALLLLAATTIIOONN

  22..11..  LLiinnuuxx kkeerrnneell

  To use the NFCT or NFLOG input plugin, you will need a 2.6.14 or later
  kernel.

  22..22..  UUsseerrssppaaccee lliibbrraarriieess

  If you plan to use NFCT and NFLOG input plugin, you will need to
  compile libnfnetlink, libnetfilter_conntrack and libnetfilter_log
  libraries which can be downloaded from
  <http://ftp.netfilter.org/pub/>.  A simple './configure; make; make
  install' will be enough to have library installed on your system.

  22..33..  uullooggdd

  22..33..11..  RReeccoommppiilliinngg tthhee ssoouurrccee

  Download the ulogd package from  <http://ftp.netfilter.org/pub/ulogd/>
  and untar it.

  If you want to build ulogd with MySQL support, type './configure
  --with-mysql'. You may also have to specify the path of the mysql
  libraries using '--with-mysql=path'. To build ulogd without MySQL
  support, just use './configure'.

  To compile and install the program, call 'make install'.

  33..  CCoonnffiigguurraattiioonn

  33..11..  iippttaabblleess NNFFLLOOGG ttaarrggeett

  33..11..11..  QQuuiicckk SSeettuupp

  Just add rules using the NFLOG target to your firewalling chain. A
  very basic example:


       iptables -A FORWARD -j NFLOG --nflog-group 32 --nflog-prefix foo

  To increase logging performance, try to use the


       --nflog-qthreshold N

  option (where 1 < N <= 50). The number you specify is the amount of
  packets batched together in one multipart netlink message. If you set
  this to 20, the kernel schedules ulogd only once every 20 packets. All
  20 packets are then processed by ulogd. This reduces the number of
  context switches between kernel and userspace.

  Of course you can combine the NFLOG target with the different
  netfilter match modules. For a more detailed description, have a look
  at the netfilter HOWTO's, available on the netfilter homepage.

  33..11..22..  NNFFLLOOGG ttaarrggeett rreeffeerreennccee


     ----nnfflloogg--ggrroouupp NN
        The number of the netlink multicast group to which NFLOG'ed
        packets are sent.  You will have to use the same group number in
        the NFLOG target and ulogd in order to make logging work.

     ----nnfflloogg--rraannggee NN
        Copyrange. This works like the 'snaplen' parameter of tcpdump.
        You can specify a number of bytes up to which the packet is
        copied. If you say '40', you will receive the first fourty bytes
        of every packet. Leave it to 0 to dump the whole packet.

     ----nnfflloogg--tthhrreesshhoolldd NN
        Queue threshold. If a packet is matched by the iptables rule,
        and already N packets are in the queue, the queue is flushed to
        userspace. You can use this to implement a policy like: Use a
        big queue in order to gain high performance, but still have
        certain packets logged immediately to userspace.

     ----nnfflloogg--pprreeffiixx SSTTRRIINNGG
        A string that is associated with every packet logged by this
        rule. You can use this option to later tell from which rule the
        packet was logged.

  33..22..  uullooggdd

  ulogd is what this is all about, so let's describe it's
  configuration...

  33..22..11..  uullooggdd ccoonnffiiggffiillee ssyynnttaaxx rreeffeerreennccee

  All configurable parameters of ulogd are in the configfile, typically
  located at '/etc/ulogd.conf'.

  The following configuration parameters are available:

     llooggffiillee
        The main logfile, where ulogd reports any errors, warnings and
        other unexpected conditions. Apart from a regular filename, the
        following special values can be used; ``syslog'' to log via the
        unix syslog(3) mechanism. ``stdout'' to log to stdout.

     lloogglleevveell
        This specifies, how verbose the logging to logfile is. Currently
        defined loglevels are: 1=debug information, 3=informational
        messages, 5=noticable exceptional conditions, 7=error
        conditions, 8=fatal errors, program abort.

     pplluuggiinn
        This option is followed by a filename of a ulogd plugin, which
        ulogd should load upon initialization. This option may appear
        more than once.

     ssttaacckk
        This option is followed by a list of plugin instances which will
        start with an input plugin, contains optionnal filtering plugin
        and finish by an output plugin. This option may appear more than
        once.

  33..22..22..  uullooggdd ccoommmmaannddlliinnee ooppttiioonn rreeffeerreennccee

  Apart from the configfile, there are a couple of commandline options
  to ulogd:

     --hh ----hheellpp
        Print a help message about the commandline options.

     --VV ----vveerrssiioonn
        Print version information about ulogd.

     --dd ----ddaaeemmoonn
        For off into daemon mode. Unless you are debugging, you will
        want to use this most of the time.

     --cc ----ccoonnffiiggffiillee
        Using this commandline option, an alternate config file can be
        used. This is important if multiple instances of ulogd are to be
        run on a single machine.

     --ii ----iinnffoo
        Display informations about the plugin whom filename is given as
        argument.

  44..  SSiiggnnaallss // LLooggrroottaattee

  ulogd understands two kinds of signals:

     SSIIGGHHUUPP
        Close and re-open all logfiles. This is mainly intended for
        logrotate scripts.  Also closes and re-opens database
        connections.

     SSIIGGUUSSRR11
        Reload configuration file. This is not fully implemented yet.

     SSIIGGUUSSRR22
        Dump the whole conntrack table and flush counters afterwards.
        Only Plugin ulogd_inpflow_NFCT.so uses this signal.

  55..  AAvvaaiillaabbllee pplluuggiinnss

  It is important to understand that ulogd without plugins does nothing.
  It will receive packets, and do nothing with them.

  There are two kinds of plugins, interpreter and output plugins.
  Interpreter plugins parse the packet, output plugins write the
  interpreted information to some logfile/database/...

  You can get information about plugins by running

  ______________________________________________________________________
  ulogd -i path/to/plugin/file.so
  ______________________________________________________________________

  55..11..  IInnppuutt pplluuggiinnss

  ulogd comes with the following input plugins:

  55..11..11..  uullooggdd__iinnppppkktt__NNFFLLOOGG..ssoo

  This interfaces the new nfnetlink_log interface. To compile, you need
  libnetfilter_log installed in your system.

     ggrroouupp
        The number of the netlink multicast group to which NFLOG'ed
        packets are sent.  You will have to use the same group number in
        the NFLOG target (--nflog-group) and in the input plugin.

     aaddddrreessssffaammiillyy
        You will need to specify the value of the protocol if you are
        not loging IPv4 packet.  addressfamily is 7 to bridged packet
        and 10 for IPv6 packet.

     nnuummeerriicc__llaabbeell
        You can use this label to store information relative to the
        logging. The administrator can define a convention which can be
        used later to differenciate packet. For example, it can store
        the severity of the logged event.

     nneettlliinnkk__ssoocckkeett__bbuuffffeerr__ssiizzee
        Specify the base socket buffer size. This start value will be
        increased if needed up to netlink_socket_buffer_maxsize.

     nneettlliinnkk__ssoocckkeett__bbuuffffeerr__mmaaxxssiizzee
        Specify the base socket buffer maximum size.

  55..11..22..  uullooggdd__iinnppffllooww__NNFFCCTT..ssoo

  This interfaces the nfnetlink_conntrack kernel subsystem, and provides
  flow-based logging. To compile, you need libnetfilter_conntrack
  installed on your system.

     ppoolllliinntteerrvvaall
        Change connection tracking dump interval.

     hhaasshh__eennaabbllee
        If set to 1 (default) a internal hash will be stored and only
        destroy event will reach the output plugin.  It set to 0, all
        events are reveived by the output plugin.

     hhaasshh__bbuucckkeettss
        Size of the internal hash bucket.

     hhaasshh__mmaaxx__eennttrriieess
        Maximum number of entries in the internal connection hash.

     eevveenntt__mmaasskk
        Select event received from kernel based on a mask. Event types
        are defined as follows:

        +o  Creation event: 0x00000001

        +o  Update event: 0x00000002

        +o  Destroy event: 0x00000004

     nneettlliinnkk__ssoocckkeett__bbuuffffeerr__ssiizzee
        Specify the base socket buffer size. This start value will be
        increased if needed up to netlink_socket_buffer_maxsize.

     nneettlliinnkk__ssoocckkeett__bbuuffffeerr__mmaaxxssiizzee
        Specify the base socket buffer maximum size.

  55..22..  IInntteerrpprreetteerr pplluuggiinnss

  ulogd comes with the following interpreter plugins:

  55..22..11..  uullooggdd__rraaww22ppaacckkeett__BBAASSEE..ssoo

  Basic interpreter plugin for nfmark, timestamp, mac address, ip
  header, tcp header, udp header, icmp header, ah/esp header... Most
  people will want to load this very important plugin.

  55..22..22..  uullooggdd__ffiilltteerr__PPWWSSNNIIFFFF..ssoo

  Example interpreter plugin to log plaintext passwords as used with FTP
  and POP3. Don't blame me for writing this plugin! The protocols are
  inherently insecure, and there are a lot of other tools for sniffing
  passwords... it's just an example.

  55..22..33..  uullooggdd__ffiilltteerr__IIFFIINNDDEEXX..ssoo

  Filter plugin that provides translation from the numerical ifindex
  (e.g. '1') to the network interface name (e.g. 'eth4').

  55..22..44..  uullooggdd__LLOOCCAALL..ssoo

  This is a 'virtual interpreter'. It doesn't really return any
  information on the packet itself, rather the local system time and
  hostname. Please note that the time is the time at the time of
  logging, not the packets receive time.

  55..22..55..  uullooggdd__ffiilltteerr__HHWWHHDDRR..ssoo

  This plugin convert hardware header to string. In the case of ethernet
  packet, it basically convert mac address to a string represetation.

  55..22..66..  uullooggdd__ffiilltteerr__IIPP22BBIINN..ssoo

  This plugin convert IP addresses to a binary form usable by databases
  like MySQL.

  55..22..77..  uullooggdd__ffiilltteerr__IIPP22HHBBIINN..ssoo

  This plugin convert IP addresses to a binary form in host order usable
  by databases like MySQL.

  55..22..88..  uullooggdd__ffiilltteerr__IIPP22SSTTRR..ssoo

  This plugin convert IP addresses to string.

  55..22..99..  uullooggdd__ffiilltteerr__PPRRIINNTTFFLLOOWW..ssoo

  Convert the keys relative to a flow in a string readable by human.

  55..22..1100..  uullooggdd__ffiilltteerr__PPRRIINNTTPPKKTT..ssoo

  Convert the keys relative to a packet in a string readable by human.
  This plugin has to be used to print packet in the format similar to
  the LOG target format.

  55..22..1111..  uullooggdd__ffiilltteerr__MMAARRKK..ssoo

  When this plugin is put in a stack, only messages were the mark
  (packet mark or connection mark) matches the given mark/mask will be
  logged.

     mmaarrkk
        Define the mark which will be used to check packet or flow.

     mmaasskk
        Define the mask which will be used to check packet or flow.

  55..33..  OOuuttppuutt pplluuggiinnss

  ulogd comes with the following output plugins:

  55..33..11..  uullooggdd__oouuttppuutt__OOPPRRIINNTT..ssoo

  A very simple output module, dumping all packets in the format


       ===>PACKET BOUNDARY
       key=value
       key=value
       ...
       ===>PACKET BOUNDARY
       ...

  to a file. The only useful application is debugging.

  The module defines the following configuration directives:

     dduummppffiillee
        The filename where it should log to. The default is
        /var/log/ulogd_oprint.log

  55..33..22..  uullooggdd__oouuttppuutt__LLOOGGEEMMUU..ssoo

  An output module which tries to emulate the old syslog-based LOG
  targed as far as possible. Logging is done to a seperate textfile
  instead of syslog, though.

  The module defines the following configuration directives:

     ffiillee
        The filename where it should log to. The default is
        /var/log/ulogd_syslogemu.log

     ssyynncc
        Set this to 1 if you want to have your logfile written
        synchronously. This may reduce performance, but makes your log-
        lines appear immediately. The default is 0

  55..33..33..  uullooggdd__oouuttppuutt__MMYYSSQQLL..ssoo

  An output plugin for logging into a mysql database. This is only
  compiled if you have the mysql libraries installed, and the configure
  script was able to detect them. (that is: --with-mysql was specified
  for ./configure)

  The plugin automagically runs a procedure with arguments taken from a
  the configurable table; It connects to mysql during the startup phase
  of ulogd and obtains a list of the columns in the table. Then it tries
  to resolve the column names against keys of interpreter plugins. This
  way you can easily select which information you want to log - just by
  the layout of the table.

  If, for example, your table contains a field called 'ip_saddr', ulogd
  will resolve this against the key 'ip.saddr' and put the ip address as
  32bit unsigned integer into the corresponding argument of table.

  The file 'doc/mysql-ulogd2.sql' contains a schema for both packet and
  flow logging.

  The module defines the following configuration directives:

     ttaabbllee
        Name of the table which ulogd will use to build arguments list.

     pprroocceedduurree
        Stored procedure that will be run with the argument specified in
        the table variable.  Behaviour of the procedure option can be
        twitted by using specific name.  If procedure name is:

        +o  "INSERT": A classic INSERT SQL query is done in the table
           pointed by the "table" variable.

        +o  start with "INSERT ": Configuration has to specify the start
           of the INSERT query that will be used. For example, a typical
           value is "INSERT INTO ulog2".

        +o  start with "CALL": the named stored procedure is executed
           with the "CALL" MySQL command.

        +o  Otherwise the named stored function is executed with the
           "SELECT" MySQL command.

     ddbb Name of the mysql database.

     hhoosstt
        Name of the mysql database host.

     ppoorrtt
        TCP port number of mysql database server.

     uusseerr
        Name of the mysql user.

     ppaassss
        Password for mysql.

     rreeccoonnnneecctt
        Number of reconnection attempt before declaring the output
        plugin as dead.

     ccoonnnneecctt__ttiimmeeoouutt
        Database connection timeout.

  55..33..44..  uullooggdd__oouuttppuutt__PPGGSSQQLL..ssoo

  An output plugin for logging into a postgresql database. This is only
  compiled if you have the pgsql libraries installed, and the configure
  script was able to detect them. (that is: --with-pgsql was specified
  for ./configure)

  The plugin automagically runs a procedure with arguments taken from a
  the configurable table; It connects to pgsql during the startup phase
  of ulogd and obtains a list of the columns in the table. Then it tries
  to resolve the column names against keys of interpreter plugins. This
  way you can easily build your own procedure and select it arguments
  just by modifying the layout of the table.

  If, for example, your table contains a field called 'ip_saddr', ulogd
  will resolve this against the key 'ip.saddr' and put the ip address as
  32bit unsigned integer into the table.

  The file 'doc/pgsql-ulogd2.sql' contains a schema for both packet and
  flow logging.

  The module defines the following configuration directives:

     ttaabbllee
        Name of the table which ulogd will use to build arguments list.

     pprroocceedduurree
        Stored procedure that will be run with the argument specified in
        the table variable.

     sscchheemmaa
        PGSQL schema to use.

     ddbb Name of the database.

     hhoosstt
        Name of the mysql database host.

     ppoorrtt
        TCP port number of database server.

     uusseerr
        Name of the sql user.

     ppaassss
        Password for sql user.

     rreeccoonnnneecctt
        Number of reconnection attempt before declaring the output
        plugin as dead.

     ccoonnnneecctt__ttiimmeeoouutt
        Database connection timeout.

  55..33..55..  uullooggdd__oouuttppuutt__PPCCAAPP..ssoo

  An output plugin that can be used to generate libpcap-style packet
  logfiles.  This can be useful for later analysing the packet log with
  tools like tcpdump or ethereal.

  The module defines the following configuration directives:

     ffiillee
        The filename where it should log to. The default is:
        /var/log/ulogd.pcap

     ssyynncc
        Set this to 1 if you want to have your pcap logfile written
        synchronously. This may reduce performance, but makes your
        packets appear immediately in the file on disk. The default is 0

  55..33..66..  uullooggdd__oouuttppuutt__SSQQLLIITTEE33..ssoo

  An output plugin for logging into a SQLITE v3 database. This is only
  compiled if you have the sqlite libraries installed, and the configure
  script was able to detect them. (that is: --with-sqlite3 was specified
  for ./configure)

  The plugin automagically inserts the data into the configured table;
  It opens the sqlite db during the startup phase of ulogd and obtains a
  list of the columns in the table. Then it tries to resolve the column
  names against keys of interpreter plugins. This way you can easily
  select which information you want to log - just by the layout of the
  table.

  If, for example, your table contains a field called 'ip_saddr', ulogd
  will resolve this against the key 'ip.saddr' and put the ip address as
  32bit unsigned integer into the table.

  You may want to have a look at the file 'doc/sqlite3.table' as an
  example table including fields to log all keys from ulogd_BASE.so.
  Just delete the fields you are not interested in, and create the
  table. This file contains two tables, one for packet-based logging and
  another for flow-based logging.

  To create the database file with the tables, you have to invoke the
  following command: sqlite3 ulogd.sqlite3db < sqlite3.table

  To check that we are logging stuff into it correctly: sqlite3
  ulogd.sqlite3db "SELECT * from ulog_ct"

  The module defines the following configuration directives:

     ttaabbllee
        Name of the table to which ulogd should log.

     ddbb Name of the database.

  55..33..77..  uullooggdd__oouuttppuutt__SSYYSSLLOOGG..ssoo

  An output plugin that really logs via syslogd. Lines will look exactly
  like printed with traditional LOG target.

  The module defines the following configuration directives:

     ffaacciilliittyy
        The syslog facility (LOG_DAEMON, LOG_KERN, LOG_LOCAL0 ..
        LOG_LOCAL7, LOG_USER)

     lleevveell
        The syslog level (LOG_EMERG, LOG_ALERT, LOG_CRIT, LOG_ERR,
        LOG_WARNING, LOG_NOTICE, LOG_INFO, LOG_DEBUG)

  66..  QQUUEESSTTIIOONNSS // CCOOMMMMEENNTTSS

  All comments / questions / ... are appreciated.

  Just drop a note to netfilter-devel@vger.kernel.org.


  The preferred method for reporting bugs is the netfilter bugzilla
  system, available at  <http://bugzilla.netfilter.org/>.



