FreeBSD on ZFS root, using 'gptzfsboot'

This is old; I did this when I got new NAS hardware and wanted to use the then new 'gptzfsboot' code which allows for ZFS root without messing about with UFS partitions on each mirror member disk ( to work around FreeBSD's bootloader not supporting ZFS ).
One of the prime motivators was the idea of being able to upgrade the disks in my NAS when they filled up without having to mess about. I wanted to be able to buy a new pair of bigger disks, add them in, create a 4-way mirror, let it sync, remove the old disks, leaving a bigger mirror and importantly, a bootable archivable pair of disks.
At the rate disk size gets bigger, this has proved to be a good strategy.
I bought an HP Microserver for this; FreeBSD works very well on it - it has 4 drive bays (needed for this strategy), and with offers at the time was 100GBP not including upgrading the memory to its maximum, or HDDs. Still though, very cheap, so I bought 2 for chassis redundancy.

Anyway, I wouldn't have bothered blogging it, it's well covered online; but I had some nice tweaks on the source HOWTO I based this install on, and more importantly the disk upgrade process ( which will be the next post... ) had a few sub-optimal moments, which I thought were worth sharing. Do please email me if I'm wrong!

This install was based on: http://www.b0rken.org/freebsd/zfs.html

Here is basically the same process again, but tweaked, as I wanted 'geom' mirrored swap, and some NAS-centric additions.

Prerequisites:

  • 1 HP Microserver ( a great choice of cheap hardware for a FreeBSD NAS )
  • 1 FreeBSD 8.1 ( at time of install, hopefully same for {8,9}.x ) RELEASE memstick image, written to a USB flash drive

Process:

  • Boot from the USB stick
  • At the loader menu, press 6 to escape to a loader prompt
  • Load required modules, then continue booting:



Type '?' for a list of commands, 'help' for more detailed help.
OK load ahci
/boot/kernel/ahci.ko size 0xfd88 at 0x126b000
OK load zfs
/boot/kernel/zfs.ko size 0x19eb18 at 0x127b000
loading required module 'opensolaris'
/boot/kernel/opensolaris.ko size 0x3868 at 0x141a000
OK boot

  • In sysinstall, set your keyboard language and then choose Fixit from the main menu, specifying the appropriate device to run the environment from (DVD or USB)
  • GPT partition first disk, ada0, and install bootcode:


# gpart create -s gpt ada0
# gpart add -s 128K -t freebsd-boot ada0
ada0p1 added
# gpart add -s 4G -t freebsd-swap ada0
ada0p2 added
# gpart add -t freebsd-zfs ada0
ada0p3 added
# gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ada0
ada0 has bootcode

  • Repeat the previous steps for second disk, ada1.
  • Show GPT layout so far:


# gpart show
=>        34  5860533101  ada0  GPT  (2.7T)
          34         256     1  freebsd-boot  (128K)
         290     8388608     2  freebsd-swap  (4.0G)
     8388898  5852144237     3  freebsd-zfs  (2.7T)

=>        34  5860533101  ada1  GPT  (2.7T)
          34         256     1  freebsd-boot  (128K)
         290     8388608     2  freebsd-swap  (4.0G)
     8388898  5852144237     3  freebsd-zfs  (2.7T)

  • Set the single partition defined in the protective MBR as active. Some systems have a BIOS that checks for the existence of an active MBR partition on the boot drive and fails to boot if it isn't found. Intel motherboards seem to do this:


# fdisk -a -1 /dev/ada0 # fdisk -a -1 /dev/ada1

  • Create /boot/zfs directory for zpool.cache file:


# mkdir /boot/zfs

  • Create mirrored zpool from freebsd-zfs partitions on each disk:


# zpool create pool0 mirror ada0p3 ada1p3
# zpool status -v pool0
  pool: pool0
 state: ONLINE
 scrub: non requested
config:

	NAME        STATE     READ WRITE CKSUM
	pool0       ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    ada0p3  ONLINE       0     0     0
	    ada1p3  ONLINE       0     0     0

errors: No known data errors

  • Create ZFS hierarchy, mounted under /mnt:


# zfs set mountpoint=none pool0
# zfs create pool0/ROOT
# zfs create -o mountpount=/mnt pool0/ROOT/freebsd
# zfs create -o mountpoint=/home pool0/HOME

  • Note! HOME is separate to ROOT, and the boot zfs is not a top level zfs, this is the opensolaris way, and allows for separate boot environments...
  • These next commands are optional, but recommended, as you may wish to specify different options, such as compression, on these zfs data sets:


# zfs create pool0/ROOT/freebsd/var
# zfs create pool0/ROOT/freebsd/usr
# zfs create pool0/ROOT/freebsd/usr/local
# zfs create pool0/ROOT/freebsd/usr/ports
# zfs create pool0/ROOT/freebsd/usr/ports/distfiles
# zfs create pool0/ROOT/freebsd/usr/src

  • Extract FreeBSD distribution to ZFS filesystems:


# export DESTDIR=/mnt
# cd /dist/8.1-RELEASE
# for dir in base catpages dict doc info lib32 manpages; do (cd $dir; ./install.sh); done
# cd kernels
# ./install.sh GENERIC
# cd ../src
# ./install.sh all

  • Copy GENERIC kernel to expected location:


# cd /mnt/boot
# cp -Rpv GENERIC/* kernel/

  • Create the geom swap mirror:


# kldload /mnt2/boot/kernel/geom_mirror.ko
# kldstat
# gmirror label -b prefer swap /dev/ada[01]p2

  • Edit loader.conf and rc.conf to load modules and start ZFS on boot:


# cat >> /mnt/boot/loader.conf <<EOloader
ahci_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:pool0/ROOT/freebsd"
geom_mirror_load="YES"
EOloader

# echo 'zfs_enable="YES"' >> /mnt/etc/rc.conf

  • Set up fstab to mount swap and tmpfs* on boot:


# cat >> /mnt/etc/fstab <<EOfstab
tmpfs                   /tmp    tmpfs   rw,mode=01777   0       0
/dev/mirror/swap        none    swap    sw              0       0
EOfstab

(* Using tmpfs with zfs is apparently not a good idea - see ZFSKnownProblems on the FreeBSD wiki)

  • Copy zpool.cache to /mnt/boot:


# cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache

  • Export LD_LIBRARY_PATH:


# export LD_LIBRARY_PATH=/dist/lib

  • Unmount all ZFS filesystems and then change their mountpoints:


# zfs umount -a
# zfs set mountpoint=legacy pool0/ROOT/freebsd
# zfs set mountpoint=/usr pool0/ROOT/freebsd/usr
# zfs set mountpoint=/var pool0/ROOT/freebsd/var
# zfs set mountpoint=/home pool0/home

  • Set bootfs property on pool:


# zpool set bootfs=pool0/ROOT/freebsd pool0

  • Exit Fixit environment and reboot

The system should now boot into the new ZFS-based FreeBSD installation. At this stage, it is very minimally configured. The following still needs to be done:

    • Set root password:


# passwd

    • Configure hostname and networking:
      ( check the ethernet device driver when configuring networking, HP Microserver has 'bge' )


# cat >> /etc/rc.conf <<EOrcconf
hostname="nas.local"
ifconfig_bge0="DHCP"
EOrcconf

    • Configure sshd:


# echo 'sshd_enable="YES"' >> /etc/rc.conf
# /etc/rc.d/sshd start

    • Configure timezone:
      ( this is for UTC, you may want 'Europe/London' or somesuch )


# cp /usr/share/zoneinfo/Etc/UTC /etc/localtime
# ntpdate -b pool.ntp.org
# cat >> /etc/rc.conf <<EOrcconf
ntpd_enable="YES"
ntpd_sync_on_start="YES"
EOrcconf
# /etc/rc.d/ntpd start

  • Given this host will be a NAS device, disk checking and snapshotting is recommended, and you will almost certainly want 'samba', and probably something like 'mt-daapd'... I also use my NAS host as a remote logger host...

    ( note! the pkg_add commands at time of install failed '-r', so explicitly grab URLs, also, some dependencies were not followed, so there is a good chance versions have changed, and you will need to update the commands... or if you're lucky, maybe 'pkg_add -r' will work for you properly! )

    • Install 'smartd':


# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/Latest/smartmontools.tbz
# cp /usr/local/etc/smartd.conf{.sample,}
# echo 'smartd_enable="YES"' >> /etc/rc.conf
# echo daily_status_smart_devices="`ls /dev/ada? 2>/dev/null`" >> /etc/periodic.conf
# echo 'daily_status_zfs_enable="YES"' >> /etc/periodic.conf
# /usr/local/etc/rc.d/smartd start

    • Install ports, and ports utils:


# portsnap fetch
# portsnap extract
# portsnap update
# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/All/db41-4.1.25_4.tbz
# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/Latest/portupgrade.tbz

    • Install 'freebsd-snapshot':
      ( a great way of automating ZFS snapshots - it does UFS snapshotting too, but we do not care! )


# portinstall freebsd-snapshot
# cat >> /etc/rc.conf <<EOrcconf
snapshot_enable="YES"
snapshot_schedule="/,/usr,/usr/local:2:1:0 /var:0:2:4 /home:2:6:8@8,12,16,20"
EOrcconf

    • Install 'beadm':
      ( this will allow you to manipulate boot environments the opensolaris way, this almost entirely de-pains the freebsd update process )


# portinstall beadm

  • From here, we will be using '/home/media' as the primary media store for 'samba' and 'mt-daapd'
    ( users needing local write access to 'outgoing' need to be in 'media' group )
  • Also, set up as a remote logger host.

    • Create a 'media' user and more importantly home dir, set appropriate permissions:


# pw user add -n media -m -s /usr/sbin/nologin
# mkdir -p /home/media/{outgo,incom}ing
# chmod 2775 /home/media/outgoing
# chmod 0777 /home/media/incoming
# chgrp nobody /home/media/incoming

    • Install 'samba':
      ( you will almost certainly need to edit 'smb.conf', but this should work for the setup described here )


# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/All/libexecinfo-1.1_3.tbz
# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/All/popt-1.16.tbz
# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/All/libgcrypt-1.4.6.tbz
# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/Latest/samba35.tbz
# cat >> /etc/rc.conf <<EOrcconf
smbd_enable="YES"
nmbd_enable="YES"
winbindd_enable="YES"
EOrcconf
# cp /usr/local/etc/smb.conf{.sample,}
# patch -p0 <<EOpatch
--- /usr/local/etc/smb.conf.sample      2011-02-01 22:32:01.079977859 +0000
+++ /usr/local/etc/smb.conf     2012-11-24 22:05:12.770126208 +0000
@@ -293,10 +293,10 @@

 # This is a DRAFT sample configuration for the ACLs on the ZFS partition.
 #
-;    nt acl support = yes
-;    inherit acls = no
-;    map acl inherit = yes
-;
+    nt acl support = yes
+    inherit acls = no
+    map acl inherit = yes
+
 ;[zpool]
 ;    path = /tank/zpool
 ;    unix extensions = no
@@ -304,3 +304,22 @@
 ;    nfs4:mode = special
 ;    nfs4:acedup = merge
 ;    nfs4:chown = yes
+
+[media]
+    comment = media - ro
+    path = /home/media/outgoing
+    guest ok = Yes
+    browseable = yes
+
+[incoming]
+    comment = incoming - rw
+    path = /home/media/incoming
+    browseable = yes
+    force directory mode = 0777
+    force create mode = 0777
+    force group = nobody
+    force user = nobody
+    public = yes
+    writeable = yes
+    read only = no
+
EOpatch
# /usr/local/etc/rc.d/samba start

    • Install 'mt-daapd':
      ( after tests, this seems the best/simplest DAAP server, modern devices seem to like DAAP )


# cp /usr/local/etc/mt-daapd.conf{.sample,}
# patch -p0 <<EOpatch
--- /usr/local/etc/mt-daapd.conf.sample 2011-01-06 00:27:06.000000000 +0000
+++ /usr/local/etc/mt-daapd.conf        2011-02-06 18:53:44.296566530 +0000
@@ -42,7 +42,8 @@
 # Location of the mp3 files to share
 #

-mp3_dir                /usr/local/share/mt-daapd
+#mp3_dir               /usr/local/share/mt-daapd
+mp3_dir                /home/media

 #
 # servername (required)
EOpatch
# echo 'mt_daapd_enable="YES"' >> /etc/rc.conf
# /usr/local/etc/rc.d/mt-daapd start

    • Install 'syslog-ng':
      ( this is much easier to setup as a remote logger, just point clients at it, logs will appear in '/var/log/remote' )


# pkg_add ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8-stable/Latest/syslog-ng.tbz
# cat >> /etc/rc.conf <<EOrcconf
syslogd_enable="NO"
syslog_ng_enable="YES"
EOrcconf
# mkdir /var/log/remote
# patch -p0 <<EOpatch
--- /usr/local/etc/syslog-ng.conf.sample        2011-01-24 22:50:50.000000000 +0000
+++ /usr/local/etc/syslog-ng.conf       2012-02-14 21:54:16.692669003 +0000
@@ -8,7 +8,12 @@
 #
 # options
 #
-options { long_hostnames(off); flush_lines(0); };
+#options { long_hostnames(off); flush_lines(0); };
+options { long_hostnames(off);
+          flush_lines(0);
+          check_hostname(yes);
+          keep_hostname(yes);
+          chain_hostnames(no); };

 #
 # sources
@@ -38,6 +43,10 @@
 destination console { file("/dev/console"); };
 destination allusers { usertty("*"); };
 #destination loghost { udp("loghost" port(514)); };
+destination logpile {
+           file("/var/log/remote/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY"
+           owner(root) group(wheel) perm(0600)
+           create_dirs(yes) dir_perm(0700)); };

 #
 # log facility filters
@@ -181,3 +190,7 @@
 # *.*                                                  /var/log/ppp.log
 #
 log { source(src); filter(f_ppp); destination(ppp); };
+
+# remote host logs
+log { source(src); destination(logpile); };
+
EOpatch
# /etc/rc.d/syslogd stop
# /usr/local/etc/rc.d/syslog-ng start

next post, upgrading to bigger disks...

Posted by doug on Saturday, November 24, 2012