ORC Owl Logo 2  

Owl River Company

 
  Your IP is: 54.224.13.210


   http://www.owlriver.com/tips/pxe-install/

PXE installation outline.


The thought has been for the last few years, that modern commercial computer environments should be manageable from cradle to grave, from a central systems administration locale.

Get those new chassis in through the loading dock, already set up with blank hardware from the manufacturer. Known hardware, qualified to meet business needs, with a reasonable price and ready spares available. Have low cost assistants plug them into the LAN; add the keyboard, mouse and monitor, and then AC power.

Meanwhile, back at a central systems admin point, the system designers, working from a tested and policy driven design, have populated a PXE server, a DHCP server, a DNS server, a TFTP server, and possibly a smart HTTP based, SQL back-ended management console and control interface. These servers hand out a basic operating system, sufficiently smart to pull the rest of an operating system and applications down across the LAN, and to install and configure the software, hands off.

If the IS department is really clever, each night, the central servers can send the 'WOL' -- wake on lan -- packet to a given unit, and wake it up again, perhaps to back up local content off the unit's hard drive back to the backup server, perhaps to update Windows virus scanner profiles.

... That's a win in the corporate world. TCO heaven, so to speak. But wait, there's more:

A couple of more interesting paths exist. With the various 'free' *nix variants available, doing automated network installs, or clean environment builds, or development testing, or maybe being a really nice display head for a central application 'terminal server' is also possible with the PXE booting option.



In this outline, (first written in mid 2001 and revised over time), we work through setting up various parts of the server environment needed to do this later case, in a context of automating installation of test software images. Some aspects are outside of our scope -- details on general DHCP, or DNS, or webserver setup. We focus on the parts special to PXE boots, and installs. For this simplified example, our local subnet is an RFC 1918 'non-routable' one: 10.16.33.0/24 with a single server at: 10.16.33.105 handling many functions. In actual production, we spread the load around with RPM, rsync, SSH and SCP, and a bit of shell scripting 'glue'.

0. One advantage of doing testing on a single server, we can open a window so we can watch what is happening on installation server host, through its logfiles. On the logserver, one can do this:
$ sudo tail -f /var/log/messages
[The clever *nix admins out there have already consolidated logging to a central log-host with the syslogd -r option. You wise guys need to send extensions to this document, and we'll roll them in.] We use this technique in our buildfarm, along with output from the logger command, to keep tabs on progress on our headless builders.

1. Set up a dhcp and pxe server on the installation server host
ftp://ftp.owlriver.com/pub/local/ORC/dhcp/
Red Hat's pxe package is was a straight packaging of the code from Intel, who led the WFM - Wired for Management - initatives back in 1998 and 1999; see the readme in the package, and is fine to use.

Later note: Red Hat also now (late 2003) offers a more current packing of the ISC dhcp-3.x, but at the time of the initial development and preparation of this outline, it did not yet have it . It might be used instead.

Later still (March 2007): When I went to set up a new server, as my old PXE installation server hardware, a HP Netserver, has aged well, and it has provided service for eight years. The drives are dying, though, and so I need to move on I see that Red Hat seems to have discontinued shipping the pxe packaging based on from RHEL 3, on. Time marches on.
[herrold@centos-4 ~]$ srcfind pxe-
/mnt/nfs/var/ftp/pub/mirror/ORC/pxe/pxe-0.1-36.src.rpm
/mnt/nfs/var/ftp/pub/mirror/centos/centos-2/SRPMS/pxe-0.1-23.src.rpm
/mnt/nfs/var/ftp/pub/mirror/redhat/rhl/9/SRPMS/pxe-0.1-36.src.rpm
/mnt/nfs/var/ftp/pub/mirror/redhat/rawhide/SRPMS/pxe-0.1-36.src.rpm
/mnt/nfs/var/ftp/pub/mirror/fedora/legacy/7.3/SRPMS/pxe-0.1-31.99.7.3.src.rpm
[herrold@centos-4 ~]$
Trial building our old packaging required the 'as86' assembler, and I had not added the explicit BuildRequire for dev86. I'll add that tweak in as well, and check for a later version upstream. (time passes) hmmm. It then turns out that the bcc now present has removed awawreness of the -1 option -- reasonable in light of the hardware one can now encounter. Okay -- I surrender to progress this once, and will use the tftp-server package .... (grumble, grumble)

2. Set up a tftp server on the installation server host
ftp://ftp.owlriver.com/pub/local/ORC/atftp/
[root@ftp pxelinux.cfg]# cat /etc/xinetd.d/tftp # default: off # description: The tftp server serves files using the trivial file transfer # protocol. The tftp protocol is often used to boot diskless # workstations, download configuration files to network-aware printers, # and to start the installation process for some operating systems. service tftp { disable = no socket_type = dgram protocol = udp wait = yes user = root server = /usr/sbin/in.tftpd server_args = --verbose=7 --no-blksize # server_args = --verbose=7 --no-tsize --no-blksize # cps = 500 2 } #

Note: The setup interaction with the xinetd, and atftp, have changed over time, and some debugging may be needed. The documentation on that is a xinetd function, and what is being passed through client atftp server, is a bit opaque, and takes some care in reading.

Red Hat also offers a packaging of the ISC tftp-server, but we find that the ISC version does not yield enough detail for debugging and development. However, it may be used as well, with some changes in the configuration. We have also observed that with the configurations as shipped, the tftp-server and xinetd are 'touchy' and tend to lock up under load with 'hung' closed sockets. Frankly, our approach is better. ;)

Later, again: As noted, we have cut over, but are concerned that we will lack sufficient debugging detail when things start to not work.

3. As tftp service runs from the xinetd, restart the xinetd to cause it to re-read its configuration.

$ sudo /sbin/service xinetd restart

4. Install syslinux for helper files
ftp://ftp.owlriver.com/pub/local/ORC/syslinux/
Red Hat also now offers a more current packaging of the HPA syslinux-2.x, but at the time of development and preparation of this outline, it did not. It may be used instead.

5. Add needed paths to the filesystem in support of a Red Hat install tftp environment. The "-p" option to mkdir speeds the process.:
$ sudo mkdir -p /tftpboot/rhl/pxe/pxelinux.cfg/

6. Get a 'tftp-able' kernel ('vmlinuz') and initial ram disk ('initrd') set in position. In our example, our install image is placeed at: /var/ftp/pub/install/ftpinstall/disc1/images/pxeboot/. See: ftp://ftp.owlriver.com/pub/local/ORC/k12ltsp/ for a script named install-from-iso.sh, which automates performing this process, at boot time, automatically and accurately.

cp /var/ftp/pub/install/ftpinstall/disc1/images/pxeboot/vmlinuz \ /tftpboot/rhl/pxe/ cp /var/ftp/pub/install/ftpinstall/disc1/images/pxeboot/initrd.img \ /tftpboot/rhl/pxe/

7. The kernel vmlinuz and the initial ram disk initrd are a matched set. If several installations are set up in a testing facility, it is customary to version the sets, and use links to select the active set, or switch configurations. The relevant files to consider are the /etc/dhcpd.conf, or the contents at /tftpboot/rhl/pxe/ .

[root@ftp pxe]# ls -l | awk {'print $1" "$9" "$10" "$11'} | grep -v ^[t] lrwxrwxrwx initrd.img -> initrd.img-orc-20 -rw-r--r-- initrd.img-orc-20 -rw-r--r-- initrd.img-rhl-73 -rw-r--r-- initrd.img-rhl-80 -rw-r--r-- initrd.img-rhl-8094 -rw-r--r-- pxelinux.0 lrwxrwxrwx pxelinux.bin -> pxelinux.0 drwxr-xr-x pxelinux.cfg lrwxrwxrwx vmlinuz -> vmlinuz-orc-20 -rw-r--r-- vmlinuz-orc-20 -rw-r--r-- vmlinuz-rhl-73 -rw-r--r-- vmlinuz-rhl-80 -rw-r--r-- vmlinuz-rhl-8094 [root@ftp pxe]#

8. Let the dhcp server know to hand out this kernel and initial ram disk. Edit /etc/dhcpd.conf content, to include a stanza thus:

# allow booting; allow bootp; option option-128 code 128 = string; option option-129 code 129 = text; # This is not used in PXE installs - it is for LTSP option root-path "10.16.33.105:/opt/ltsp/i386"; # next-server 10.16.33.105; filename "/rhl/pxe/pxelinux.cfg/" ; #

Note: We have left intact some comments from our LTSP development work. They are harmless, as they relate to information used when the installation initrd is not present.

9. And restart the dhcpd again to re-read the configuration file changes.

$ sudo /sbin/service dhcpd restart

10. Copy pxelinux.0 over to the boot area. Note that this matches the location ("/tftpboot/rhl/pxe/") we have told the DHCP server to hand out. Add the link to conform to the default name which syslinux looks for ("pxelinux.bin").

cd /tftpboot/rhl/pxe/ cp /usr/lib/syslinux/pxelinux.0 . ln -s pxelinux.0 pxelinux.bin

11. Then populate the 'default' action file in /tftpboot/rhl/pxe/pxelinux.cfg/

# cat /tftpboot/rhl/pxe/pxelinux.cfg/default default linux serial 0,38400n8 label linux kernel vmlinuz append ksdevice=eth0 load_ramdisk=1 prompt_ramdisk=0 \ ramdisk_size=16384 initrd=initrd.img network \ ks=http://10.16.33.105/pub/kickstart/ks.cfg
(Note: We use the conventional 'backslash continuation' convention here for typographical clarity of presentation. Please be aware that it is NOT presently supported in syslinux booting.)

Kickstart is somewhat limited, without the 'smarts' in some versions to use 'resolver' code to translate numeric IP's into hostnames and vice versa. As such, here and elsewhere in this outline, we hard-code numeric IP's.

12. But as we are PXE installing in a DHCP space 10.x.x.x, and under the syslinux PXE code, we will first find '0A' [0x0A is hex for decimal 10 -- our first IP octet], in advance of 'falling through' to default. Set it up separately:

root@ftp pxelinux.cfg]# cat 0A default linux serial 0,38400n8 label linux kernel vmlinuz append ksdevice=eth0 local_ramdisk=1 initrd=initrd.img \ ks=http://10.16.33.105/pub/kickstart/ks.cfg # # JAW's variant 030116 # # append ksdevice=eth0 local_ramdisk=1 initrd=initrd.img network # 1 root=/dev/ram0 rw init=/linuxrc initrd=initrd.lts # append root=/dev/ram0 rw init=/linuxrc initrd=initrd.lts

  Quick tip:
To compute the conventional 'dotted-quad' decimal IP to 'eight-hex-digits' filename, this works:
[herrold@oldnews herrold]$ perl -e 'printf ("%02X%02X%02X%02X\n",10,16,33,2);'
0A102102
[herrold@oldnews herrold]$
tip o' the hat to: Aurora FAQ - (local copy)
 

13. Time for a 'smoke-test'. Remember that sudo tail -f /var/log/messages back at step 0? We can see progress as the PXE client calls for the setup information under the atftp server.

$ sudo tail -f /var/log/messages

14. In the last line of '0A', we are, in this instance, pulling our 'ks.cfg' -- our kickstart config -- from the local webserver. We assume that an apache webserver is installed and that a custom directory '/pub/kickstart/' exists off the web root. This allows us to:

$ sudo tail -f /var/log/httpd/access_log
to see progress in calling for that setup file. A ks.cfg is set up, under another name in /root/ during a normal RHL install, or may be fashioned with 'redhat-config-kickstart'.

15. A sample ks.cfg is as follows:

[root@ftp kickstart]# cat ks.cfg # Kickstart file automatically generated by anaconda. # ... we have modified it for our base install install lang en_US langsupport --default en_US.UTF-8 en_US.UTF-8 keyboard us mouse generic3ps/2 --device psaux skipx # next lime should be all on one line -- split for the example # network --device eth0 --bootproto static --ip 192.168.0.3 --netmask \ # 255.255.255.0 --gateway 192.168.0.1 --nameserver 198.30.29.8 network --device eth0 --bootproto dhcp # url --url ftp://10.16.33.105/pub/install/ftpinstall/ # We have mangled out the password's md5 sum here for security rootpw --iscrypted $1$gIWn1/m4$sonV123456abcdefWtPjs. firewall --disabled authconfig --enableshadow --enablemd5 timezone America/New_York bootloader --location=mbr # zerombr yes clearpart --all # clearpart --linux # part /boot --fstype ext3 --size 150 part swap --size 250 part / --fstype ext3 --size 500 part /usr --fstype ext3 --size 1000 --grow # %packages @ Text-based Internet joe openssh-server # %post

16. The ks.cfg content has changed over time, and usually needs to be matched to the release being tested. It is moving to a XML'ized format which will simplify computer assisted generation and customization of such configurations. If several installations are set up in a testing facility, it is customary to version the sets, and use links to select the active set, or switch configurations. The relevant files to consider is the /var/www/html/pub/kickstart/ks.cfg in our example.

17. We'll take and place that file for our initial testing:
mkdir -p /var/www/html/pub/kickstart cp /root/anaconda-ks.cfg /var/www/html/pub/kickstart/ks.cfg

18. Some curiousities exist with some ks.cfg . First, at least until after RHL 8.0, the install binaries source location is not added to the file automatically -- we edit it into ks.cfg with a value like this:
url --url ftp://10.16.33.105/pub/install/ftpinstall
This 'url' here relates to the archive of files which are installed, rather than to the "ks.cfg" which is being delivered and managed througha HTTP web interface.

... and, second, that (properly and expectedly) one has to explicitly authorize it to wipe any existing partitions:
clearpart --linux --drives=hda

19. And with the sudo tail -f /var/log/messages console window visible, we should be able to watch the boot, and install process proceed. It shows us the dhcp lease engotation, and the tftp transfers occurring.

20. We can also the webserver access file with a separate sudo tail -f /var/log/httpd/access_log in a second console window. We should also be able to see the call for the ks.cfg, late in the boot process, and just before the install process proceeds.

21. The line respecting a serial console, and the practice of exporting an X display to a monitoring unit for installation screenshots and debugging are outside of the scope of this piece.

Things to watch for:


Clearly the tftp server, and the FTP server can get 'overwhelmed' by the load which a local network install can impose on the server side. This will show up as installations which randomly fail at varying points in an install, when an FTP server locks up. We have noticed this with the non-daemon vsftpd which accompanies RHL 8.0.

Also, the TFTP and XINETD combination in recent (RHL 7.2, 7.3, and 8.0) have had mid-life security patches, and again some combinations evidence the lockup issue.

The best advice to offer is to turn logging 'way up, and then watch several of the log files, keeping an eye open for an unexpected 'freeze'; if encountered, consider substituting another version or implementation of the troublesome component package. Also be suspicious of your FTP server, intervening physical layer hub and switch interference, and even non-PXE compliant older NIC's.

It is sometimes the case, that despite everything seeming to be perfect, an install will start and them mysteriously fail. In our experience, network card and switch speed and duplex re-negotiation may be in play. To rule out such cases, simplify the network to a very small network seqment of matched components, and review the TCPDUMP techniques pieces we have written, to view the desired packets down closer to the Physical Layer. As elsewhere, 'proving' each layer of the ISO stack from the bottom up is sometimes the only remaining effective way to diagnose and proceed in development.

Credits and ending notes:


This represents the result of much independent lab experimentation and following in the footsteps and using the Open Source software of others. H. Peter Arvin of Syslinux; members of the syslinux mailing list, the Red Hat beta testers mailing list, the kickstart mailing list, the ltsp mailing list, and the k12ltsp mailing list have each helped provoke the thoughts presented here. More locally, Jim Wildman, of the Central Ohio Linux User Group has blazed part of this trail, and served as a reviewer as well. My thanks to all.

Please report any errors, unclarity, or suggestions for improvement to: info+pxe@owlriver.com . Please confirm that you are viewing the most recent copy of this site, and mention the revision date below.

-- Russ Herrold


Copyright (C) 2003 R P Herrold
      herrold@owlriver.com  NIC: RPH5 (US)
   My words are not deathless prose,
      but they are mine.

       Owl River Company
   "The World is Open to Linux (tm)"
   ... Open Source LINUX solutions ...
      info@owlriver.com
         Columbus, OH
See also: Hands-off kickstart methods

Manufacturers:
Not recommended Intel - Intel Boot Agent (PXE, et al.)
Intel Proutil setup tool (local copy) [requires DOS (Windows) boot environment - Please let us know if you can get it working in Wine or FreeDOS, and we'll add how you did it.]
Pro 100 Intel Boot Agent update tool (local copy) (see also here)
Intel Proadmin setup tool (local copy)
Intel card identification guide. - Intel Pro-100+ and Pro 100 S
Intel PXE Development Kit - (local)

Not recommended 3com - 3com guide - (local PDF) - 3com 3C905C-TX-M
3com - 3com guide - (local PDF) - Managed PC Boot Agent
3com - 3com guide - (local PDF) - OS Deployment Guide
3com - 3com updates
[The 3com cards require special modification of the boot image by imggen -- ugghhh. (binary tarball) - (local binary tarball)]

Other voices:
Netboot Howto - (Local copy)
Alf Wachsmann - How to Install Red Hat Linux via PXE and Kickstart - (local .ps)
Yellow Dog - Linux for Power PC's -- (local) -- See also: Erik's Apple Network Server Page
K12LTSP outline
FNAL tip - (local)
FNAL syslinux.cfg - (local)
FNAL ks.cfg - (local)
CPQ Linux tip - Kickstart - http://www.cpqlinux.com/whitepaper-kickstart.html
CPQ Linux tip - PXE whitepaper - http://www.cpqlinux.com/whitepaper-pxe.html
CPQ Linux tip - PXE server - http://www.cpqlinux.com/pxeserver.html - (local)
ramdisk.txt - how-to on building one - (local)
Deploy and Maintain - Kickstart, PXE, yum and cfengine
Note: We take snapshots to local copies of remote sites to cache content, as sites have gone dark on us unexpectedly. PLEASE refer to the primary site link first; let us know of material updates of remote sites, and we'll refresh the local copy.

Other architectures:
YDL NetBoot - (local)

Non-RPM distributions:
Debian Fully Automatic Installaton - (Local PDF)

Other approaches:
Dell OpenManage - ( local )
Nilo the Network Interface Loader
kano PXE - Intel propietary pxe extension reimplementation

Non-Linux Systems:
Windows
Sysdesign MBR discussion (local copy)
Bootable USB keys (local copy)

rev 050212 RPH
http://www.owlriver.com/tips/pxe-install/

Up More Tips
       

Back to Top Page
[legal] [ no spam policy ] [ Copyright] © 2008 Owl River Company
All rights reserved.

Last modified: Sat, 03 Mar 2007 18:02:46 -0500
http://www.owlriver.com/tips/pxe-install/index.php