Linux Configure Netconsole To Log Messages Over UDP Network

Linux Configure Netconsole To Log Messages Over UDP Network

Linux can be configured to log dmesg output to another system via network using syslog. It is done using kernel level networking stuff ia UDP port 514. There is module called netconsole which logs kernel printk messages over udp allowing debugging of problem where disk logging fails and serial consoles are impractical. Most modern distro has this netconsole as a built-in module. netconsole initializes immediately after NIC cards. There are two steps to configure netconsole:

Syslogd server - Let us assume 192.168.1.100 IP having FQDN - syslogd.nixcraft.in. Please note that the remote host can run either 'netcat -u -l -p <port>' or syslogd.
All other systems running netconsole module in kernel

Step # 1: Configure Centralized syslogd

Login to syslogd.nixcraft.in server. Open syslogd configuration file. Different UNIX / Linux variant have different configuration files

Red Hat / CentOS / Fedora Linux Configuration

If you are using Red Hat / CentOS / Fedora Linux open /etc/sysconfig/syslog file and set SYSLOGD_OPTIONS option for udp logging.

# vi /etc/sysconfig/syslog

Configure syslogd option as follows:
SYSLOGD_OPTIONS="-m 0 -r -x"

Save and close the file. Restart syslogd, enter:
# service syslog restart

Debian / Ubuntu Linux Configuration

If you are using Debian / Ubuntu Linux open file /etc/default/syslogd set SYSLOGD option for udp logging.
# vi /etc/default/syslogd

Configure syslogd option as follows:
SYSLOGD_OPTIONS="-r"

# /etc/init.d/sysklogd restart

FreeBSD configuration

If you are using FreeBSD open /etc/rc.conf and set syslogd_flags option option for udp logging. Please note that FreeBSD by default accepts network connections. Please refer to syslogd man page for more information.

Firewall configuration

You may need to open UDP port 514 to allow network login. Sample iptables rules to open UDP port 514:
MYNET="192.168.1.0/24"
SLSERVER="192.168.1.100"
iptables -A INPUT -p udp -s $MYNET --sport 1024:65535 -d $SLSERVER --dport 514 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -s $SLSERVER --sport 514 -d $MYNET --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Step # 2: Configure Linux Netconsole

You need to configure netconsole service. Once this service started, you are allowed a remote syslog daemon to record console output from local system. The local port number that the netconsole module will use 6666 (default). You need to set the IP address of the remote syslog server to send messages.

Open /etc/sysconfig/netconsole file under CentOS / RHEL / Fedora Linux, enter:
# vi /etc/sysconfig/netconsole

Set SYSLOGADDR to 192.168.1.100 (IP address of remote syslog server)
SYSLOGADDR=192.168.0.1

Save and close the file. Restart netconsole service, enter:
# /etc/init.d/netconsole restart

A note about Debian / Ubuntu Linux

Red Hat has netconsole init script. However, under Debian / Ubuntu Linux, you need to manually configure netconsole. Type the following command to start netconsole by loading kernel netconsole module, enter:
# modprobe netconsole 6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8

Where,

6666 - Local port
192.168.1.5 - Local system IP
eth0 - Local system interface
514 - Remote syslogd udp port
192.168.1.100 - Remote syslogd IP
00:19:D1:2A:BA:A8 - Remote syslogd Mac
You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text:
# echo 'options netconsole netconsole=6666@192.168.1.5/eth0,514@192.168.1.100/00:19:D1:2A:BA:A8 '> /etc/modprobe.d/netconsole

How do I verify netconsole is logging messages over UDP network?

Login to remote syslog udp server (i.e. 192.168.1.100 our sample syslogd system), enter:
# tail -f /var/log/messages

/var/log/messages is default log file under many distributions to log messages. Refer to /etc/syslog.conf for exact location of your file.

How do I use nc / netcat instead of messing with syslogd?

This is called one minute configuration. You can easily get output on 192.168.1.100 without using syslogd. All you have to do is run netcat (nc) command, on 192.168.1.100:
$ nc -l -p 30000 -u

Login to any other box, enter command:
# modprobe netconsole 6666@192.168.1.5/eth0,30000@192.168.1.100/00:19:D1:2A:BA:A8

Output should start to appear on 192.168.1.100 from 192.168.1.5 without configuring syslogd or anything else.