The Linux Daily

Attn Subscribers: Google Reader Being Shutdown

Google announced March 13th that it’s going through some more spring cleaning which includes Google Reader (see http://googleblog.blogspot.com/2013/03/a-second-spring-of-cleaning.html and http://googlereader.blogspot.com/2013/03/powering-down-google-reader.html). I’d like to remind my subscribers, of which 90% or so are using Reader or iGoogle, may need to resubscribe or transfer their subscription to iGoogle or another feed reader. You should be able to click the RSS Feed link and choose an RSS reader. Thanks for your continued support!

Tutorial: Installing Django on Shared Hosting Service Such as Bluehost.com

This guide will walk you through how to install and run Django on a shared host such as Bluehost. Because I use Bluehost, I was able to verify the steps you see below, but you might need to modify some of the steps to work with your specific host. The whole thing should take less than 10 minutes. If I’ve done my job right, you should be able to copy/paste multiple lines to run all commands in a block as if it were a script. If you feel more comfortable, you can run each command line by line. Let’s get to it…

[Read more →]

Handy Terminal Keyboard Shortcuts

Put these into your “Terminal Guru” belt and be more productive!

Cursor Movement Control
Ctrl-a: Move cursor to the start of a line
Ctrl-e: Move cursor to the end of a line
Ctrl-Left/Right: Navigate word by word (may not work in all terminals)

Modify Text
Ctrl-w: Delete the whole word to the left of the cursor
Ctrl-k: Erase to end of line
Ctrl-u: Erase to beginning of line

Scrolling/Buffer Control
Shift-PageUp/PageDown: Scroll through current buffer
Ctrl-s: Pause terminal output (program will keep running)
Ctrl-q: Release terminal output (after being paused)
Ctrl-l: Clears the screen. Use this instead of the clear command.

History
Ctrl-r: Search the history (enter to run the command once found)

Bonus Tip: Use ‘!!’ command to run last command and ‘!com’ to run the last command starting with ‘com’.

Process Control
Ctrl-d: Exit
Ctrl-c: Kill the current process
Ctrl-z: Put the current process in the background (fg will restore it)

Are there any keyboard shortcuts that you can’t live without? Tell us about them in the comments below.

msmtp – a (fairly) simple mail submission program

As an oldtime Unix guy, I’ve always been used to having the BSD mail utility to hand, and a suitably configured mail system, so that I can script jobs to run and email the results back to me. I use mail as a sort of glorified syslog facility. With smaller single board Linux computers we don’t always want to install a full mail setup – resources often tend to be limited. A few years back I discovered msmtp http://msmtp.sourceforge.net/.

This utility is an smtp client that submits a file in standard mail format to a mail server. It can submit plain text email or use TLS/SSL etc. I use a couple of script wrappers to emulate, sort of, sendmail and the sending part of the BSD mail utility.

Of course to use msmtp you need a mail server to which you can submit email for delivery. My home server is my mail server, but you could use your ISP’s smtp server. Another problem is that msmtp just fails if it can’t connect to the mail server – it’s up to you to handle that and do something else with that precious message you can’t mail just now! My sample scripts do not deal with that situation.

This very simple script I call sendmail, and it will need customising for your setup…

#!/bin/sh
#

# set these for your setup...
MailServer=mail.server
Domain=MachineName
From="me\@my.domain.com"

exec msmtp --host=$MailServer --domain=$Domain --from=$From $*

This is my simple script to emulate the simple parts of the send functionality in the BSD mail utility. It has many shortcomings, but it has served me well…

#!/bin/sh
#
# A sort of shell replacment for the send functionality of
# the standard "mail" utility.
#
# mail [-s subject] recipient(s)
#

u=`id -un`
d="my.email.domain"
r=""
s=""
v=""

while [ $# -gt 0 ]; do
 p="$1"
 shift
 case "$p" in
 -s)
        s="$1"
        if [ $# -gt 0 ]; then shift ; fi
        ;;
 -v)
        v="-d"
        ;;
 -*)
        echo 1>&2 "Option \"$p\" not recognised."
        exit 1
        ;;
 *)
        r="$r , $p"
        ;;
 esac
done
if [ "$r" = "" ]; then
 echo 1>&2 "No Recipients."
 exit 1
fi
if [ "$s" = "" ]; then
 printf "Subject: "
 read s
fi

s="Subject: $s \n"

(echo -e "From: ${u}@$d \nTo: $r \n$s \n\n"
 cat ) | sendmail -t $v

So if you have a job to run on the platform, then this will email the output to you…

my_job | mail -s “my_job output” [email protected]

msmtp can be loaded from the package systems of most distributions, but I have had occasion to cross-compile the package for installing on a system without package management. I had only limited libraries on my cross compile system, and found that after downloading and extracting the sourcecode from sourceforge, I had to cross compile without some of the advanced features. I used

./configure –build=arm –disable-ssl –disable-gsasl –disable-nls

before doing the make to build the binaries. The resultant binary just submitted plain text email, but that was ok for my use – YMMV. The resultant binary, suitably stripped,is pretty lean.

I recently revisited using msmtp to pre-test a change to my ISP’s new smtp server, before committing the change to my mailserver’s sendmail setup. It can also be useful for testing security settings etc on mail submissions systems.

Jim

Kermit lives!

As a veteran of the IT industry I’ve seen software and OSes come and go, but there are some pieces of software that I seem to have used on a lot of platforms for a lot of years. One of the oldest of these is the file transfer software Kermit, from Columbia University – http://www.columbia.edu/kermit/

I’ve used kermit in the early 80′s, on various CP/M systems, the very first IBM PCs and on the BBC Micro, and have been using it since on various early Unices, and other OSes that are now only memories in old fogy minds like mine. I even used hacked versions of kermit for building an email system between various computers joined by rs232 links, and then gatewayed out to the big wide world in the late 1980′s.

I used it extensively in the 90′s for automating and controlling transfers and connections over dial-up modems. It is still my console of choice when I need to hook up microcontrollers and single board computers with rs232. It’s configurability, features and programmability make it second to none.

However it has always erked me that it’s licensing prevented it being available in the standard repositories of the major Linux distributions. I have got used to downloading the source tar ball and compiling my own executables whenever I’ve needed kermit, and cursed silently that yum or apt-get would not simply do the job for me.

So imagine my joy when I belatedly discovered that Columbia University have cancelled the Kermit project and allowed it to be re-licensed and development continue at http://www.kermitproject.org/. The new license is a Revised 3-Clause BSD License which will at last allow Kermit to join the Free Software Family as a full member – and about time too!

If you are not familiar with Kermit, and need to go beyond where minicom etc can take you, then do check it out. It might appear a bit old school but it is very, very powerfull.

Jim

Resuming stalled or held CUPS print jobs…

I have a network printer, and it’s surprising how often I print a job and then find the printer is not switched on, and the print job gets put in HOLD state. Using “lpstat -l” one can view the print job status, and with the “lp” command can resume the print job. But I’m idle, so I created this very simple script to do it all in one command – I call it “lprel”.

#!/bin/sh
#
# script to resume the top job on the default printer queue

j=`lpstat | head -1 | cut -d' ' -f1`
if [ "$j" = "" ]; then
  echo 1>&2 "No jobs in queue."
  exit 1
fi
echo "releasing \"$j\"..."
lp -i "$j" -H resume
lpstat -l

cheers
Jim

Linux Motherboard upgrade

I’ve always been pretty amazed at how easy it is to change hardware under a Linux system and for it mostly to just work – especially with most modern distros. Unlike those that are stuck with Windows systems, it is possible to do a motherboard upgrade on a linux system with the minimum of fuss, and without doing a system install!

Indeed, mostly it’s just a matter of remembering to make the system forget about the MAC address of the on-board ethernet. Otherwise after the system is booted, the system thinks that the eth0 is missing and instead configures the hardware as eth1.

Maybe networkmanager, or whatever, copes, but I like to have command-line control over my network setup, and on my Debian based systems it’s eth0 that is configured properly in /etc/network/interfaces.

So before I shut down my old motherboard system for the last time, I delete the eth0 line in the file /etc/udev/rules.d/70-persistent-net.rules so that when the system reboots with new hardware, it allocates eth0 to the new onboard ethernet interface.

However, I recently recycled an Intel D945GSEJT motherboard as a replacement for a VIA EPIA MoBo in my home server. The Intel board had been my guts of my desktop before I upgraded to a dual core ATOM based board – I like my hardware to use less Watts. My server runs Ubuntu server 8.04 LTS, and, despite the forced amnesia of the VIA’s ethernet MAC address, I still didn’t have an ethernet connection after the Motherboard transplant. A quick google discovered a load of issues with Realtek r8169 driver in older kernels – and Ubuntu Server 8.04 LTS is hardly state of the art.

Not having time to do a real fix, I reached for an old PCI Ethernet card I sometimes used when I needed a machine with 2 ethernet ports. Remembering to hack /etc/udev/rules.d/70-persistent-net.rules, disabling the onboard ethernet device in the BIOS and rebooting immediately got me a working system.

I’ll leave it like it is until after I get up the courage to do the upgrade to Ubuntu server 10.04LTS. Then I’ll rescue the PCI ethernet card and add it to my box of useful bits and pieces that might just get me out of a fix. That box has all sorts of cables and adapters, as well as an old PCI graphics card, a PCI SATA adapter, a PCI IDE adapter and the PCI Ethernet card. Anyone got recommendations for other useful bits and pieces to add to the box?

cheers
Jim Jackson

Self Documenting Scripts

As a “good” programmer I like to put comments at the top of my scripts to say what the script does, and how it is used. I also like the script to output a useful help message when a user gets options/arguments wrong, or when they use the option ‘-h’. I found it was a pain keeping the 2 in step, and developed this simple scheme to only have one place for this info.

I’ll use bash here but I’m sure people can adapt to other scripting languages.

#!/bin/sh
#
## 
## Usage: helpdemo 
##
## This demos a self documenting scheme for scripts.
##

prog="$0"
me=`basename "$prog"`

dohelp () {
  grep '^##' "$prog" | sed -e 's/^##//' -e "s/_PROG_/$me/" 1>&2
}

echo "Program name is: $me"
echo "Program file is: $prog"
echo

dohelp
exit

Prefix any lines you want to be output as “help” by ‘##’ at the beginning of the line. All such lines are printed out to stderr by the dohelp function. ‘sed’ in this function also strips off the leading ‘##’ from the lines and substitutes the filename of the invoked script for ‘_PROG_’, so that if you change the name of the script, it still magically refers to the new name.

It’s a simple scheme, and can obviously be extended, e.g. one could change the dohelp function thus…

dohelp () {
  pfx="$1"
  if [ "$pfx" = "" ]; then pfx='##' ; fi
  grep "^$pfx" "$prog" | sed -e "s/^$pfx//" -e "s/_PROG_/$me/" 1>&2
}

dohelp can now be called to select out lines with a different prefix, but its default behaviour when given no prefix, is as before.

This has formed part of my standard shell script template for many years. Hope others find it useful.

Jim Jackson

Roll your own ‘lsof’ – sort of

Ever read the ‘lsof’ utility man page? It’s a shame, because the idea of the utility is wonderful – list open files and who is using them.

I had need to see what files were in use by imapd on my server. “lsof | grep imapd” gave me far more than I needed, along with long lines of info I didn’t really need all of, and all the libraries in use, etc, etc. That’s when I thought, RTFM[1] … so I did, and eventually gave up. Maybe I was having a bad day, but the manual seemed impenetrable, so I turned to
rolling my own.

I’d explored around the /proc file system a bit before, and was sure everything I need was there, and it turned out a lot easier than I’d thought. So here goes…

Under /proc, every process has a directory, with an entry called “cmdline” which gives access to that process’s command line arguments. There is a sub-directory called “fd” that contains links to each file the process has open. With this info I knocked together this little script, I called it ‘pof’:

#!/bin/sh
#
cd /proc
while [ "$1" != "" ]; do
  for n in [1-9]* ; do
    if [ "$n" != "self" -a -f $n/cmdline ] then
      if grep "$1" $n/cmdline > /dev/null ; then
        cmd=`tr '\0' ' ' < $n/cmdline`
        printf "%-8d %s\n" $n "$cmd"
        for m in $n/fd/* ; do
          printf " %-14s %s %s\n" `stat --printf="%N" $m`
        done
      fi
    fi
  done
  shift
done

It is called with an string or regular expression that you want to search each command line for. If it is found, the open file links are displayed for that process. So now ‘pof imapd’ gives me what I want.

The ‘cmdline’ file contains each argument of the command line as a null (‘\0′) terminated string, so I need to use ‘tr’ to convert the nulls to spaces before printing out.

On my system the imapd daemon is launched from inetd, so it’s first 3 files, stdin/stdout/stderr, are network sockets – so I need to dig a bit deeper and find out how to get the remote IP address of the socket.

Anyway I’m sure someone will tell me how easy ‘lsof’ is to use to get any info you want, but delving into /proc is fun!

Jim Jackson

[1] //en.wikipedia.org/wiki/RTFM

Script to Watch for Changes to Disk and Run Script Immediately

By utilizing inotifywait from the inotify-tools package, we can monitor changes to disk and immediately run a script or command on that file. No need for a cron job here…

To install inotify-tools, we simply use apt-get (or yum, depending on your distro):

sudo apt-get install inotify-tools

Here is one example script which utilizes inotifywait:

#!/bin/sh

watch_dir="/media/md0/"
inotifywait -mr -e create -e delete -e modify --timefmt '%Y-%m-%dT%H%M%S' --format '%w;%f;%e;%T' "$watch_dir" | while IFS=';' read -r DIR FILE EVENT TIME; do

  echo "${EVENT} ${TIME}: ${DIR}${FILE} >> /home/user/watched

  case "$EVENT" in
    "CREATE"|"MODIFY")
      echo "Run create/modify script on file: ${DIR}${FILE}" >> /home/user/watched
      ;;
    "DELETE")
      echo "Run delete script on file: ${DIR}${FILE}" >> /home/user/watched
      ;;
  esac
done

Give your script a name, executable rights, and run it in the background. For example:
chmod +x myscript.sh && ./myscript.sh &

Obviously, you’ll need to modify the script to fit your needs, but this should give you a jump start on your project.

Note: There’s an infinite loop bug in the script above. Remove the ‘modify’ options and this will clear things up. However, if you do that, then you’re really only looking for file creation or deletion. You could add more conditionals to the script above to prevent the loop. If anybody has a suggestion, I’d be happy to update the script.

Take this a step further by following my guide on Automatically Start a Script at Linux Bootup