wiki:DevelopmentInstructions

Overview

The BCCD-NG environment is stored entirely in the Subversion database. Read-write access is provided over ssh, at svn+ssh://bccd-ng.cluster.earlham.edu/cluster/svnroot/bccd-ng/. Development branches are under svn+ssh://bccd-ng.cluster.earlham.edu/cluster/svnroot/bccd-ng/branches, with livecd-devel as the current branch. Under tags are tags for specific milestones, and trunk is the production branch. The sql directory is used to store the schema of the ancillary databases, which is currently just the testing database.

The BCCD-NG ISO is assembled in a single pass in the bin/build_livecd.pl script. It is composed of a series of tests from the Bccd.pm perl module, found in packages/usr/local/lib/site_perl/5.8.8/Bccd.pm. Tests take different numbers of arguments, so look at the signature of the specific test subroutine to figure that out. In the near future I'll put in documentation for all the tests as well. Keep in mind that every run_test call will have at least four arguments: the name of the test, the success return code the test should expect (which is left blank in nearly all cases for a safe default), the message the test should print out, and the first (and maybe only) argument for the test.

For historical reasons, the liberation script is at packages/liberate.pl. This probably should be moved to a more sensible location in the future. The script has the same layout as the build_livecd.pl script. Remember that the liberation should assume no network connectivity, so all files should be fetched in the ISO build. Files that are specific to the liberation can be stored in /root/liberate, which is reference by the $libfetch variable.

For more information on specific tests, see the POD documentation within Bccd.pm.

Build requirements

  • Perl modules
    • WWW::Mechanize
    • DBI
    • DBD::Pg
  • debootstrap
  • Subversion
  • cloop-utils
  • genisoimage

Running tests

Common variables

  • $project: Use wherever you would use the project name.
  • $websvn: Use wherever you need to point into the branch's subversion repository.
  • $builddir: The full, un-chroot'd path to the temporary build directory.
  • $software: Software directory within the build directory.
  • $projdir: The common directory for diskless node information.
  • $newhome: The home directory of the bccd user.
  • $checkin: The directory where nodes checkin to the master node.
  • $libfetch: The directory where files used only after liberation are placed.
  • $test_name: The name of the test. Only used if you're updating the test database.
  • $stage: The name of the current stage. Will be BUILD for the build.
  • $svnrev: The subversion revision that the script is building against.
  • $tmpdir: The temporary directory for the build.
  • $packages: List of packages to install with apt-get.

Common operations

The following are a list of operations that are common to the build_livecd.pl script. They are somewhat ordered, in that if you can get by with an earlier operation, you should.

Debian CLI

This should always be preferred to any of the other options. Use apt-get, debootstrap, debconf, etc. wherever possible, called from the system test in Bccd.pm. This might look like

$Bccd->run_test(
                "system",
                "",
                "Running apt-get install."
                "apt-get install $packages"
                );

Changing files

If you have to, change only part of the file using sed or perl regexen, Again, you can call these from the system tests in Bccd.pm. This might look like

$Bccd->run_test(
                "system",
                "",
                "Changing interface in networking."
                "sed -e 's/ifconfig eth0/ifconfig eth1/g' < $libdir/$projdir/etc/init.d/networking > $libdir/etc/init.d/networking"
                );

Entire file

You can store an entire file in subversion. Fetch it using the revfetch test, which takes the URL to the file, the subversion revison, and where you want to squirt the file in the build. This might look like

$Bccd->run_test(
              "revfetch",
              "",
              "Fetched checkin cron.",
              $svnrev,
              "$websvn/packages/etc/cron.d/node-checkin",
              "$builddir/etc/cron.d/node-checkin"
              );

Entire tree

Use this only if you are fetching a directory tree and not overwriting existing files. There are too many dependency problems involved with overwriting existing files in the build. chdir using the chdir test to the base directory of the destination directory tree. Then use the recrevfetch test, which takes a subversion revision, and a repository URL. This might look like

$Bccd->run_test(
              "chdir",
              "",
              "cd $builddir",
              "$builddir"
              );

$Bccd->run_test(
              "recrevfetch",
              "",
              "Fetched bin tools.",
              $svnrev,
              "$websvn/packages/bin/"
              );

Tar balls

The BCCD-NG development originally used tar balls to distribute changes. This is deprecated for a number of reasons, but where it's still working I haven't touched it. Do what I say and not what I do, and use the options above.

Testing changes

Changes should always be tested before committed to trunk. This can be done on any Debian system with a checkout of the subversion repository.

  1. As root, cd into branches/livecd-devel/bin/.
  2. export PERL5LIB=../packages/usr/local/lib/site_perl/5.8.8/
  3. script -c "perl build_livecd.pl --arch i386 --suite lenny --outiso /tmp/test.iso --svnrev some-rev " /tmp/build.script
  4. You can view the output of the test in /tmp/build.script.
  5. If everything is OK, cd into the trunk branch.
  6. Run svn merge --dry-run -r rev1:rev2 svn+ssh://bccd-ng.cluster.earlham.edu/cluster/svnroot/bccd-ng/branches/livecd-devel/ .. rev1 is the last commit into the trunk, which you can see with svn log. rev2 is the new revision that you just committed to livecd-devel. Make sure the files being updated are the files you want updated. Look for any files with a "C" by them. You will have to merge those files by hand in the next step.
  7. svn merge -m rev1:rev2 svn+ssh://hp.cluster.earlham.edu/cluster/svnroot/bccd-ng/branches/livecd-devel/ .. Merge any conflicted files by hand, then run svn resolved filename. After everything is OK, run svn commit in the trunk directory, and commit your changes. Make sure that build_livecd.pl still has its websvn variable pointing into trunk and not livecd-devel.

Adding packages

Via aptitude

aptitude is easy if there is an existing deb package in the tree on a0. Simply add a test for aptitude install package-name in build_livecd.pl. Note that this is aptitude and not apt-get, because I've had less problems with aptitude barfing on dependency problems.

Via svn

This option involves adding a "tree" to the subversion repository that contains the software build. Follow these steps to do it:

  1. From the live CD or liberation (liberation is recommended for disk space reasons), build and install the software into a subdirectory of /cluster/software.
  2. Make a symlink from software-xx.yy.zz to software, where xx.yy.zz is a version number.
  3. tar up software-xx.yy.zz and software, and move the tar ball to a host with a local subversion checkout.
  4. cd into the development branch, and then to trees/software/cluster/software.
  5. un-tar the tar ball.
  6. Run svn add software*.
  7. Commit the changes.
  8. Build a test ISO and test the new software.
  9. Merge the changes into the trunk.

Local builds

Using a local package mirror

By default, the build process uses the package mirror located at http://debmirror.cluster.earlham.edu/apt/mirror/bccd-ng/debian/. The script will allow the use of a local debmirror. Supply the hostname to the local repository using the --debmirror option. Don't supply the full URL; the rest of the URL will be constructed later in the script.

Using a local Subversion mirror

The build script fetches files out of Subversion using WebDAV at http://bccd-ng.cluster.earlham/svn/bccd-ng/branch-name, where branch-name would be trunch or branches/livecd-devel. You can supply an alternative WebDAV URL with --websvn.

Editing the splash screen

See the isolinux instructions for changing a boot screen: http://www.sweb.cz/Frantisek.Rysanek/splash/isolinux-splash-HOWTO.html

Kernel updates

Perform all these steps in a liberated BCCD-NG system:

  1. Build the kernel from source tar ball. Use a pre-existing config file if possible. Note that you have to use gcc 4.2 to build 2.6.22.9.
  2. Run make modules_install.
  3. Copy bzImage to the vmlinuz file in /boot.
  4. Run mkinitramfs -o /boot/initramfs-<kernel-version> <kernel-version>.
  5. Extract the old /var/lib/tftpboot/initramfs-<kernel-version> and the new initramfs. gzip -d -c initramfs|cpio -i
  6. Copy init and scripts/nfs from the old initramfs to the new initramfs.
  7. Make a new tftpboot initramfs. find .|cpio -o -H newc|gzip -c
  8. Uncompress trees/KNOPPIX/boot/isolinux/minirt.gz, and mount it over a loop back device as an ext2 filesystem.
  9. cd into the image, and go into the modules directory.
  10. Copy in all the modules already in the modules directory from the modules from the new kernel.
  11. Unmount the image, compress it, and replace the one in Subversion.

Rebuilds

You can make rebuilds faster by bypassing Subversion commits and just rebuilding the cloop and KNOPPIX ISO images. To do this, do one full build with --nocleanup at the end, which will preserve the build directory in /tmp. Take a note of the directory that the script uses. Make your changes in the temporary directory (like $TMPDIR/build-bccd, or $TMPDIR/KNOPPIX). Then run genisoimage -R -U -V BCCD-NG -publisher ccg -hide-rr-moved -cache-inodes -no-bak -pad $TMPDIR/build-bccd/ | create_compressed_fs - 65536 > $TMPDIR/KNOPPIX/KNOPPIX/KNOPPIX && genisoimage -pad -l -r -J -v -V BCCD-NG -no-emul-boot -boot-load-size 4 -boot-info-table -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat -hide-rr-moved -o $OUTISO $TMPDIR/KNOPPIX

Environment management

As mentioned in the user manual, the BCCD uses Modules to manage a user's environment (variables, aliases, etc.). This provides a Tcl-based framework to manage a user's environment as software packages are added and deleted. Every package installed in /cluster/software should get a module directory in /usr/local/etc/modules, with every version of the package as a separate file in the directory. Every modules file is basically a Tcl script invoked by the main modules command. For example, lam-7.1.4 has this file in /usr/local/etc/modules/lam-7.1.4:

#%Module1.0#####################################################################

prereq          modules
conflict    mpich2
conflict        openmpi
set             MOD_LAM              $env(BCCD_SW)/lam
set      MOD_LAM_VERSION      7.1.4
set      MOD_LAM_DIR          $MOD_LAM-$MOD_LAM_VERSION
module-whatis "LAM"

setenv      LAM_RSH             "ssh -q"
setenv      LAMHOME             $MOD_LAM_DIR

prepend-path    PATH                $MOD_LAM_DIR/bin/
prepend-path    LD_LIBRARY_PATH     $MOD_LAM_DIR/lib/
prepend-path    MANPATH             $MOD_LAM_DIR/man/
prepend-path    -d " " LDFLAGS             -L$MOD_LAM_DIR/lib/
prepend-path    -d " " CPPFLAGS            -I$MOD_LAM_DIR/include/
  • The first line is a magic number to identify this as a modules file.
  • prereq modules will cause an error to be printed if the main BCCD modules file (called modules) isn't loaded first.
  • The two conflict lines make sure other MPI implementations aren't loaded at the same time this module is loaded, since the binary names conflict.
  • The next three lines set internal Tcl variables that are scoped only within this file. These are not environment variables.
  • module-whatis "LAM" is just a free-form identification for what this module provides.
  • The two setenv lines set user environment variables to whatever values are provided in the third field.
  • The last five lines use the prepend-path function to ensure that LAM's files are found properly in a user's search paths. The -d " " changes the delimiter from the default of :, since LDFLAGS and CPPFLAGS are both space-delimited.

These module files should be placed in trees/usr/local/etc/modules/software-name/version, which is part of the MODULEPATH search path. module load will load the lexigraphically last software version by default, but you can override this behavior by symlinking the version you want to be default to default.

USB automounting

The BCCD uses udev to detect when USB storage devices are plugged in, and mounts them automatically. The udev setup is in source:trunk/packages/etc/udev/rules.d/99-automount_usb.rules, and the script that the rule file calls is in source:trunk/trees/bin/bccd-usbwall.

Last modified 12 years ago Last modified on Jan 24, 2009 3:41:17 PM