The Linux Daily

Merging or Creating Read-Only Filesystems with Unionfs

Recently, I was asked to make an embedded system running Linux completely read-only. To accomplish this, I utilized the unionfs utility. The standard blurb from the man page will shed some light on it:

Unionfs is not a command, but rather a file system. This manual page describes additional mount options that Unionfs supports.
Unionfs is a stackable unification file system, which can appear to merge the contents of several directories (branches), while keeping their physical content separate. Unionfs is useful for unified source tree management, merged contents of split CD-ROM, merged separate software package directories, data grids, and more. Unionfs allows any mix of read-only and read-write branches, as well as insertion and deletion of branches anywhere in the fan-out. To maintain unix semantics, Unionfs handles elimination of duplicates, partial-error conditions, and more.

In a nutshell, Unionfs allows you to layer directories into a single, mounted directory. Once the module has been compiled and inserted, it’s very easy to use. For example, to mount a read-only filesystem, you’ll want to replace any lines in your linuxrc file (if you don’t know what this is, you’ll want to read the article from logicsupply) that mount your filesystem (at /mnt/root, for example) with something like this:

insmod /unionfs.ko
mkdir /mnt/root.ro /mnt/root.rw
mount -o ro /mnt/root.ro
mount -t tmpfs root.rw root.rw
mount -t unionfs -o dirs=/mnt/root.rw=rw:/mnt/root.ro=ro unionfs /mnt/root

In the above example, we insert the unionfs.ko module located in the / directory. We then create the directories for which we will merge. These directories are where we’ve mounted our root filesystem and the tmpfs filesystem. Next, we mount our root filesystem as readonly to the root.ro mount point and then mount a tmpfs filesystem to root.rw which will become our writable layer. Finally, we merge the two together and voila! A read-only filesystem. The next thing to do is to chroot to /mnt/root and you’re done. Keep in mind that any changes made are being written to the tmpfs layer of unionfs and will not be persistent.

Take a look at these great articles on other example usage:
http://www.logicsupply.com/blog/2009/01/27/how-to-build-a-read-only-linux-system/
http://www.linuxjournal.com/article/7714

The most difficult part is getting your kernel to support it, but even this isn’t too difficult thanks to good patches on the main project website here:
http://www.filesystems.org/project-unionfs.html

We’ll run through an example. Say that your embedded system is using the 2.6.24 kernel and you want to use Unionfs. The first things you’ll need to do is find the 2.6.24 patch on the Unionfs main project website, download it, extract it, and then apply it using a command like the following from within the kernel source directory (the same directory with the Makefile):

patch -p1 < unionfs-2.5.4_for_2.6.24.7.diff

When patched, you’ll then need to use make menuconfig to select the Unionfs module. Save the .config changes when prompted to do so and use the make command to compile the kernel and modules. Install the kernel Image and the module into your embedded system and then try to insmod the driver. It should just work. Then, you can use the commands outlined above.

And that’s that! Be sure to check out the links provided. It would be too difficult to include everything there is to know about Unionfs in a single post. I hope I was able to provide you with at least a start in using it. Good luck and have fun!

Convert Filesystem from JFS to EXT3 and Keep Data Intact

A common and simple procedure in Linux is to change from one filesystem to another. There are many guides out there that will assist you in doing this. The method shown below is a simple procedure that makes a copy of your files and data, converts the filesystem, and then copies all files and data back again (note: this wouldn’t work on a system that has the kernel, initrd, and filesystem all rolled into one like desktop Linux). Here are some steps that were taken to successfully switch a partition formatted in JFS to EXT3. Keep in mind that this was done on an embedded Linux device that boots from an SD card and then loads the filesystem (distribution) located on the fourth partition (hence the /dev/sdc4 dev node). Verify your device node using fdisk -l and preview the steps and determine if they will work for you before plunging in…

mount -t jfs /dev/sdc4 /mnt/p4
cd /mnt/p4
tar cjvf /myfiles/distro/my_distro_files.tar.bz2 .
rm -rf *
cd ~
umount /dev/sdc4
mkfs -t ext3 /dev/sdc4
mount -t ext3 /dev/sdc4 /mnt/p4
cd /mnt/p4
tar xvf /myfiles/distro/my_distro_files.tar.bz2

Distribute Program Compilation Over Several Machines on a Network

I just learned about a couple little jewels of applications today which help you compile your large programs faster by distributing them over several different computers on the network. The tools are called distcc and icecream. Here’s a blurb from their man pages:

distcc
distcc distributes compilation of C code across several machines on a network. distcc should always generate the same results as a local compile, is simple to install and use, and is often much faster than a local compile.

icecream
Icecream is created by SUSE and is based on ideas and code by distcc. Like distcc it takes compile jobs from your (KDE) build and distributes it to remote machines allowing a parallel build on several machines you’ve got. But unlike distcc Icecream uses a central server that schedules the compile jobs to the fastest free server and is as this dynamic. This advantage pays off mostly for shared computers, if you’re the only user on x machines, you have full control over them anyway.

More information can be found here:
http://linuxreviews.org/man/distcc/
http://code.google.com/p/distcc/
http://en.opensuse.org/Icecream

Silence the rt73 Driver Message of MimeAssocReqAction()

If you utilize the rt2501usb chipset (rt73 driver) and are connect to your wireless LAN using WPA2 encryption, then you may have noticed a flurry of messages showing up in your syslog or dmesg output, filling up you logs:

[ 1321.830000] MlmeAssocReqAction(): WPA2/WPA2PSK fill the ReqVarIEs with CipherTmp!
[ 1323.980000] MlmeAssocReqAction(): WPA2/WPA2PSK fill the ReqVarIEs with CipherTmp!
[ 1328.170000] MlmeAssocReqAction(): WPA2/WPA2PSK fill the ReqVarIEs with CipherTmp!
[ 1330.480000] MlmeAssocReqAction(): WPA2/WPA2PSK fill the ReqVarIEs with CipherTmp!
[ 1332.990000] MlmeAssocReqAction(): WPA2/WPA2PSK fill the ReqVarIEs with CipherTmp!
[ 1337.060000] MlmeAssocReqAction(): WPA2/WPA2PSK fill the ReqVarIEs with CipherTmp!

It seems to me that these are simply status/debugging messages and don’t serve much of a purpose other than fillings up the logs with information that’s not all that valuable to an end-user. The connection still works, so it doesn’t seem out of line to simply comment out the offending lines in the source code. If you don’t have the source code already, then you’ll need to goto RaLink and download them. There are readme files that will help you compile from scratch. When you have the source code, simply use grep to find the messages:

grep -rn "ReqVarIEs with CipherTmp"

You should find that the lines occur in the assoc.c file at line 348 and line 402. Comment these lines out and recompile. After this, your logs will be scotch-free of all those messages.

Example Usage of ‘jobs’ Command

This is one of those commands that hit me as “Whoa! How did I miss that?” — the jobs command. It simply lists the jobs currently running on the system. Unlike ps, it only lists what’s running in the foreground and the background. So, say you stop a process with ctrl+z and you want to later kill it, you can use the jobs command to quickly find it and then kill it off. For example:

# dd if=/dev/zero of=/dev/null
^Z
[1]+  Stopped                 dd if=/dev/zero of=/dev/null
# jobs
[1]+  Stopped                 dd if=/dev/zero of=/dev/null
# kill %1
# jobs
[1]+  Terminated              dd if=/dev/zero of=/dev/null
# jobs
#

You can easily kill off more than one process as well by simply including the %number after kill like:

# jobs
[1]   Stopped                 dd if=/dev/file1 of=/dev/null
[2]-  Stopped                 dd if=/dev/file2 of=/dev/null
[3]   Running                 dd if=/dev/file3 of=/dev/null &
[4]+  Stopped                 dd if=/dev/file4 of=/dev/null
# kill %3 %2
# jobs
[1]-  Stopped                 dd if=/dev/file1 of=/dev/null
[2]   Terminated              dd if=/dev/file2 of=/dev/null
[3]   Terminated              dd if=/dev/file3 of=/dev/null
[4]+  Stopped                 dd if=/dev/file4 of=/dev/null
#

As always, remember to read the man pages for more information.

A Quick CPU Intensive Script for Testing Purposes with md5sum

If you’re needing to create some sort of CPU stress test script on the fly with minimal resources on a Linux box, a simple md5sum loop comes in handy. Even with ‘watered-down’ versions of a Linux filesystem (ie. busybox) generally have at least the capabilities of running a loop and have the md5sum tool compiled and ready to use. A quick rundown on how to set one up is outlined below.

First, create a relatively large file (as large as you can make one without being overkill):

dd if=/dev/urandom of=testfile count=20 bs=1024k

Second, create a script (I named it stress_cpu, but you can call it what you want) with the following contents:

#!/bin/sh

i=0

while [ 1 ]
do
   md5sum testfile
   i=`expr $i + 1`
   echo "Iteration: $i"
done

Save the file and then give it executable permissions:

chmod +x stress_cpu

Finally, run it:

./stress_cpu

You may notice when running this script that the initial md5sum test takes about 10 seconds longer than any subsequent tests. This is due to the way md5sum works and how things are written to cache for faster access. If you don’t want anything to be cached for good measure, and you have /proc mounted with the /proc/sys/vm/drop_caches file, then you can use the command sync; echo 1 > /proc/sys/vm/drop_caches immediately following the md5sum command in the script. See the proc man page for more information.

Compare Output of Command and Match to CSV List or File

The following script provides a method for scanning the output of a command, matching it in another list, and performing an action. The example script, install.sh, shown is using the lsusb command to scan USB devices inserted against the known-devices.csv file and perform an action. Here’s how it looks in motion:

./install.sh

* This script will automatically determine the wireless
* adapter and install the appropriate drivers.

Detected: Asus WL-167g
Running Script in 3rd Field of CSV... OKAY!

Scripting language is pretty much self-documenting so I won’t bother explaining too much. If you have questions or suggestions, please let me know. The following script is flexible and can be used in many different situations so I encourage you to customize it to fit your needs. Enjoy!

install.sh: The main script that is ran to try and match the output of the lsusb command with another files list and act accordingly.

#!/bin/sh

# Create tmp files for verification along the way
touch tmp
touch tmp2

echo
echo "* This script will automatically determine the wireless"
echo "* adapter and install the appropriate drivers."
echo

# Run lsusb and grab only the prod/vend id field
lsusb | cut -d' ' -f 6 | while read line
do
   # Use grep to search for $line of lsusb output in known-devices.csv file and save to tmp
   grep "$line" known-devices.csv > tmp
   if [ $? = 0 ]; then
      echo -n "Detected: "

      # Use awk to print only field 2 of the tmp file (also field 2 of known-devices.csv)
      awk 'BEGIN{FS=",";}
      {
         {print $2}
      }' < tmp

      # Prepare and execute the third field of the known-devices.csv file
      echo -n "Running Script in 3rd Field of CSV... "
      cut -d',' -f 3 tmp > tmp2
      chmod +x tmp2
      ./tmp2
   fi
done

# Check the tmp2 file and see if it is empty.  If so, display error.
if [ `ls -l tmp2 | awk '{print $5}'` -eq 0 ]
then
  echo "No known devices found."
fi

known-devices.csv: The comma separated file in which the install.sh uses to check against. In this particular example, the prod/vend usb ID is the first field, the friendly name is in the second field, and the installation command is in the third field.

USB ID   | USB NAME     | DRIVER INSTALL COMMAND(S)                            |
--------------------------------------------------------------------------------
0ace:1215, IOGear GWU523, lib/zd1211/zd1211rw_install.sh
0b05:1723, Asus WL-167g , lib/rt73/rt73_install.sh

rt73_install.sh: This is the script that is called by the known-devices.csv file above. If there was a simple one-line command, you could specify it in the third field of the known-devices.csv file.

#!/bin/sh

   insmod rt73.ko > /dev/null 2> /dev/null
   if [ $? -eq 0 ] ; then
      echo "OKAY!"
    (
        # These are optional permanent install commands that will be ran upon
        # successful completion of previous command.
        mkdir -p lib/modules/`uname -r`/kernel/drivers/net/wireless/rt73/
        cp rt73.ko lib/modules/`uname -r`/kernel/drivers/net/wireless/rt73/rt73.ko
     )  > /dev/null 2> /dev/null
   else
      echo "FAILED!"
      echo -n "Error: Driver already installed, wasn't found, or failed to insert."
      echo "Check 'lsmod' and 'dmesg' for debugging information."
      exit
   fi

Allow Ctrl+C Escape or Interrupts in Shell Script Using Traps

Just ran across this today while looking at a script one of my colleges wrote. You can use bash traps to monitor for user input like Ctrl+C. Here’s an example taken from the Linux Dev Center (use the link to read more about it):

#!/bin/sh
# zpg - UNCOMPRESS FILE, DISPLAY WITH pg
# Usage: zpg file
stat=1  # DEFAULT EXIT STATUS; RESET TO 0 BEFORE NORMAL EXIT
temp=/tmp/zpg$$
trap 'rm -f $temp; exit $stat' 0
trap 'echo "`basename $0`: Ouch! Quitting early." 1>&2' 1 2 15

case $# in
1) gzcat "$1" >$temp
   pg $temp
   stat=0
   ;;
*) echo "Usage: `basename $0` filename" 1>&2 ;;
esac

Grabbing or Displaying Direct Frame Buffer Image

Without going into too much detail, here’s a very simple way of grabbing what’s being displayed on the screen in an embedded Linux system:

cat /dev/fb0 > image.raw

You can then display the image back to the screen by using:

cat image.raw > /dev/fb0

If you’re worried about size, you can compress the raw image file and then unpack it when you need to display an image (like a splashscreen):

gzip image.raw
gunzip image.raw.gz > /dev/fb

Convert Human Directory Name to Unix Directory Name

The following script will take a human directory name like “/My Favorites/Music/Obscure Band’s Hits/01 – Wish I Knew.mp3″ from standard input or file to a directory name that unix commands will recognize like “/My Favorites/Music/Obscure Band’s Hits/01 – Wish I Knew.mp3″. This can be useful in a scripting. Ultimately, this isn’t very practical since all you really need to do is use quotes around the human directory name, but there might be a situation that presents itself when you need this, so here it is:

echo "directory name" | sed 's/ /\ /g' | sed s/"'"/'\'"'"/g | sed s/")"/'\'")"/g | sed s/"("/'\'"("/g | sed s/"&"/'\'"&"/g | sed s/","/'\'","/g

You can see that it will replace any character that requires a break including spaces, quotes, apostrophes, ampersands, commas, etc…

Example usage:

localhost:~ username$ echo "/My Favorites/Music/Obscure Band's Hits/01 - Wish I Knew.mp3" | sed 's/ /\ /g' | sed s/"'"/'\'"'"/g | sed s/")"/'\'")"/g | sed s/"("/'\'"("/g | sed s/"&"/'\'"&"/g | sed s/","/'\'","/g
/My Favorites/Music/Obscure Band's Hits/01 - Wish I Knew.mp3
localhost:~ username$