Trust me, there will be a time when the network goes down, or maybe you'll need to get into a server's BIOS to change a setting, or boot it in single-user mode. Without physical access, or an expensive remote management card, none of this is possible. However, with a little open source software and some cheap hardware, you can set up a remote console server for multiple machines. We'll take a look at two programs that can help: Minicom, for when networking is down and to view BIOS and boot screens, and conserver, for day-to-day console access.

You'll also need some hardware. First, scrape up an older machine to use as a console server. It need not be very powerful; just about anything that runs Linux will work. It probably will have two on-board serial ports. You can buy an additional four-port serial card for around $40. Six-foot null modem cables to connect servers to the console server should cost you about $3. If you throw in a modem, you will be able to get into the console server even if the local LAN is down.

Get started by enabling terminal logins to the servers. Once you've connected the null modem cables from the servers to your new console server, check the servers' /etc/inittab files and make sure you have a line like this on them all (but not on the console server):

S0:12345:respawn:/sbin/agetty -L 57600 ttyS0 vt102 

This tells init to spawn a console on the serial line ttyS0 (think of it like COM1 in Windows) at the speed of 57600bps. This example assumes you have agetty installed. If not, use your distribution's package manager to install it first, or if your distribution installs mgetty instead, feel free to use it.

To enable root logins on the servers, edit the file /etc/securetty and add the line:

ttyS0

Now for a quick test of serial connectivity. On the console server, issue the command:

cat </dev/ttyS0 

This listens on the serial port and prints anything it receives to standard output. On the server, issue the command:

echo console server test >/dev/ttyS0 

This echoes the phrase "console server test" to the serial port. If you see this phrase appear on the console server, you know serial communication is working, and you can do something useful with it.

To start, try Minicom, which is included with just about every distribution. Minicom is useful for viewing BIOS and startup messages. Run Minicom from the command line on the console server. By default, Minicom points to the serial line linked to /dev/modem. Change this by bringing up Minicom in setup mode with:

minicom -s 

This brings up a pretty self-explanatory GUI for altering Minicom's settings. We are concerned with Serial port setup. Change the serial device to the serial line corresponding to the server you want to connect to. For instance, imagine you have two servers hooked up to your console server -- servers A and B. Server A is attached via null modem cable to /dev/ttyS0 and server B is attached to /dev/ttyS1. If you want to view server B's console, change this setting to /dev/ttyS1. You can also go in and change the speed of the line, but the default of 9600bps will work for our purposes.

Now, from a command line, enter:

minicom -o 

This starts Minicom without any modem initialization strings. In a matter of seconds you should see a console login prompt for server you are attaching to. At this point you have a rudimentary console server; now let's take a little further.

One of the main benefits of a console server is the ability to get into the BIOS and view system initialization. To view BIOS output, the server's motherboard needs to support redirecting output to a serial port. This feature is standard on just about every server motherboard. If you are using a "white box PC" for a server, your mileage may vary. Viewing the system's startup messages is a sure thing either way. For GRUB's /boot/grub/menu.lst add:

console=ttyS0,9600 

and for LILO's /etc/lilo.conf add:

serial=0,9600n8 

to the options for the kernel you want to boot. This needs to be changed on each server.

At this point, you've got everything you need to log in to remote servers and change BIOS settings. However, Minicom can be cumbersome when it comes to viewing multiple consoles. It also lacks some of the useful features found in conserver.

To use conserver, each server you want to monitor needs to run the conserver server. The console server that you log into to check on your servers will run the conserver client. Installing both server and client is simple, and can be accomplished with the usual ./configure;make;make install. If you have the OpenSSL libraries installed, use./configure --with-openssl to enable encrypted communication. The servers will run /usr/local/sbin/conserver and the console server will attach to them with the client /usr/local/bin/console.

On your servers, create the directory /usr/local/etc/ if it does not exist, and copy to it the files conserver.cf and conserver.passwd from the conserver.cf subdirectory of the conserver download. The password file is pretty self-explanatory. Any username followed by *passwd* tells conserver to use the system password for that user. Just add in your own username and use that user's system password.

You also need to add the line:

console 782/tcp conserver # console server 

to the file /etc/services on all servers. Conserver uses TCP/IP for its communication between servers. This means conserver will not work if networking is down. In that case, you'd have to go back to Minicom. So why use conserver at all? The answer is logging, and the ability to create a master conserver that can connect to multiple servers. Conserver also allows multiple users to attach to the same console. This can be useful for training purposes. You can also move from console to console within the client.

Let's take a look at a sample conserver configuration. Here is a sample conserver.cf:

### set up global access
default full { rw *; }

### set the defaults for all the consoles
# these get applied before anything else
default * {
# The '&' character is substituted with the console name
logfile /var/consoles/&;
# timestamps every hour with activity and break logging
timestamp 1hab;
# include the 'full' default
include full;
# master server is localhost
master localhost;
}

console shell {
master localhost;
timestamp 5;
type exec;
exec "";
}

console ssh {
type exec;
exec ssh localhost;
}

### define a group of users
group sysadmin {
users youruser;
}

### list of clients we allow
access * {
allowed 192.168.0.0/16;
trusted 127.0.0.1;
}

Overall it is pretty simple. We set up some defaults for use by all consoles, define a couple of consoles (shell and ssh), define a list of users, and then define what networks have access to the server.

From your console server, you can now issue the command:

console -l youruser shell 

This should prompt for the password for the user youruser. Once authenticated, you will see this prompt:

[Enter `^Ec?' for help] 

Entering Ctrl-E c ? displays the conserver help menu. Entering Ctrl-E c a will attach you to the console. Entering Ctrl-E c ; allows you to type in the name of any other consoles in your conserver.cf file and attach to them. In this example, we can also attach to the ssh console.

The console client expects to connect to a server with the name of console, which means you need to add an alias for each server to /etc/hosts or in DNS or you will get an error.

The directory /var/consoles/ now contains a log file for each console (shell and ssh). All console output created while attached is now logged in those files. If you leave one of these consoles running, you can capture error messages on the console that you might otherwise never see.

The combination of Minicom and conserver gives system administrators a cost-effective way to manage servers. With Minicom up and running, you can change BIOS settings and drop into run-level one without traveling to the remote location. If you have conserver, you can disable a user's SSH access to a machine and force the use of conserver (with SSL encryption of course). This technique could be handy for logging those situations where no one seems to know who accidentally deleted an important file. It can also help out when you forget what you did to fix something a week after the fact.