The Linux Daily

Python Script to Grab All CSS for Given URL(s)

While at work, I needed a script that would grab all CSS elements that a webpage was using, both internal and external, given a URL and concatenate these elements into a single file. I came up with the following Python script. There’s a very minimal amount of pre-setup work in order to run it though. First, you must have python installed. I’m going to assume that 1) you do, and 2) you know how to use it. The next step is to download and install the beautifulsoup package. This is how I accomplished this on my Ubuntu box (could vary depending on your distribution):

sudo apt-get install curl
curl -O http://python-distribute.org/distribute_setup.py
sudo python distribute_setup.py
sudo easy_install beautifulsoup

After that, you should be good to go. Copy / paste the following into a python script (I called mine fetch_css.py) and then run it with python fetch_css.py.

# -*- coding: utf-8 -*-
import urllib2
from urlparse import urlparse
from BeautifulSoup import BeautifulSoup

def fetch_css( url ):

    try:
       response = urllib2.urlopen(url)
       html_data = response.read()
       response.close()

       soup = BeautifulSoup(''.join(html_data))

       # Find all external style sheet references
       ext_styles = soup.findAll('link', rel="stylesheet")

       # Find all internal styles
       int_styles = soup.findAll('style', type="text/css")

       # TODO: Find styles defined inline?
       # Might not be useful... which <p style> is which?

       # Loop through all the found int styles, extract style text, store in text
       # first, check to see if there are any results within int_styles.
       int_css_data = ''
       int_found = 1
       if len(int_styles) != 0:
          for i in int_styles:
              print "Found an internal stylesheet"
              int_css_data += i.find(text=True)
       else:
           int_found = 0
           print "No internal stylesheets found"

       # Loop through all the found ext stylesheet, extract the relative URL,
       # append the base URL, and fetch all content in that URL
       # first, check to see if there are any results within ext_styles.
       ext_css_data = ''
       ext_found = 1
       if len(ext_styles) != 0:
          for i in ext_styles:
              # Check to see if the href to css style is absolute or relative
              o = urlparse(i['href'])
              if o.scheme == "":
                 css_url = url + '/' + i['href']  # added "/" just in case
                 print "Found external stylesheet: " + css_url
              else:
                 css_url = i['href']
                 print "Found external stylesheet: " + css_url

              response = urllib2.urlopen(css_url)
              ext_css_data += response.read()
              response.close()
       else:
           ext_found = 0
           print "No external stylesheets found"

       # Combine all internal and external styles into one stylesheet (must convert
       # string to unicode and ignore errors!
       # FIXME: Having problems picking up JP characters:
       #    html[lang="ja-JP"] select{font-family:"Hiragino Kaku Gothic Pro", "ããè´ Pro W3"
       # I already tried ext_css_data.encode('utf-8'), but this didn't work
       all_css_data = int_css_data + unicode(ext_css_data, errors='ignore')

       return all_css_data, int_found, ext_found
    except:
        return "",0,0

################################################################################
# Specify URL(s) here
################################################################################
urls = {
    'jaresfencing': "http://jaresfencing.com",
    'derekhildreth': "http://derekhildreth.com",
    'thelinuxdaily': "http://thelinuxdaily.com",
    'myurl1': "http://myurl1.com"
}

for k, v in urls.items():
   print "nFetching: " + v
   print "--------------------------------------------------------------------------------"
   out, int_found, ext_found = fetch_css(v)
   if ext_found == 1 or int_found == 1:
      filename = k + '_css.out'
      f = open( filename, 'w')
      f.write(out)
      print "Styles successfully written to: " + filename + "n"
      f.close()
   elif out == "":
      print "Error: URL not found!"
   else:
      print "No styles found for " + v + "n"

## OPTIONAL CSS PARSING STEP ##
# MUST INSTALL CSSUTILS with 'sudo easy_install cssutils'
#include cssutils
#sheet = cssutils.parseString(all_css_data)
#f2 = open('temp2', 'w')
#f2.write(sheet.cssText)
#f2.close()

Seems to work for me! I could see potential for many improvements, but it’s pretty robust as it is. Enjoy.

Script to Batch Random Rename Files

I used this script the other day when I wanted to randomize a group of photos from within my current working directory in the terminal (so my digital photo frame displayed randomly instead of sequentially). You could certainly spend some extra time making this more robust, but it suited my needs. You’ll probably want to modify it a bit to suit yours.

Edit: Thanks to the commenter “thewanderer”, I revisited this script to solve the issue of duplicate $RANDOM values. Now, there’s a recursive function added so files won’t be overwritten!

#!/bin/sh

if [ $# -lt 1 ]; then
   echo "Example Usage: $0 /dev/sdb"
   exit 1
fi

DIR=$1

rename(){
   rand=$RANDOM
   if [ -f "${rand}.JPG" ]; then
      rename "$i"
   else
      mv "$i" "${rand}.JPG"
   fi
}

echo "This will rename all files randomly in $DIR"
echo -e "Continue?  ( y/n ) : c"
read answer
if [ "$answer" = "n" ] || [ "$answer" = "N" ]
then
   echo "Exiting..."
   exit 1
else
(
   cd $DIR
   for i in *.JPG; do rename "$i"; done
   cd -
) >/dev/null 2>&1 </dev/null

   echo "Files have been renamed with a random number."
fi

Article on Using Python to Send Email

I came across this one while I was trying to figure out a way to script sending an email. I found it to be a very useful and helpful guide and wanted to share, so here it is: http://docs.python.org/library/email-examples.html.

PS. I thought about hitting you with an April Fools joke, but I’ll spare you. Have a good one today!

Testing Input String for Numeric, Alpha-Numeric, Alpha, or Other POSIX

If you’re trying to scrub your input of a script, a good way to do it is using POSIX along with the tr -d command. This is demonstrated in the shell script below.

#!/bin/sh

# Any of the following can be used, only a few are
# demonstrated:
#[:alnum:]  # Alphanumeric characters
#[:alpha:]  # Alphabetic characters
#[:lower:]  # Lowercase letters
#[:upper:]  # Uppercase letters
#[:digit:]  # Decimal digits
#[:xdigit:] # Hexadecimal digits
#[:punct:]  # Punctuation
#[:blank:]  # Tabs and spaces
#[:space:]  # Whitespace characters
#[:cntrl:]  # Control characters
#[:print:]  # All printable characters
#[:graph:]  # All printable characters except for space
#[a-zA-Z0-9] # Same as [:alnum:]. POSIX can be used.

if [ $# -lt 1 ]; then
   echo "Usage: $0 <text>"
   exit
fi

text=$1

# Check that input is only numeric
if [ -z `echo $text | tr -d "[:digit:]"` ]; then
   echo "Input contains only numeric characters"
else
   echo "Error: input contains non-numeric characters"
fi

# Check that input is only alpha
if [ -z `echo $text | tr -d "[:alpha:]"` ]; then
   echo "Input contains alpha characters"
else
   echo "Error: input contains non-alpha characters"
fi

# Check that input is only alphanumeric
if [ -z `echo $text | tr -d "[:alnum:]"` ]; then
   echo "Input contains alphanumeric characters"
else
   echo "Error: input contains non-alphanumeric characters"
fi

Let’s give it a test run, shall we?

user@localhost$ ./input.sh text
Error: input contains non-numeric characters
Input contains alpha characters
Input contains alphanumeric characters
user@localhost$ ./input.sh text123
Error: input contains non-numeric characters
Error: input contains non-alpha characters
Input contains alphanumeric characters
user@localhost$ ./input.sh 123
Input contains only numeric characters
Error: input contains non-alpha characters
Input contains alphanumeric characters
user@localhost$

Set stderr or stdout Redirection for Entire Script

Here’s a quick to to enable standard error (stderr) redirection for an entire script. Just put this command on the top of your script:

exec 2>> stderr.log

Here’s a quick example script to demonstrate:

#!/bin/sh

exec 2>> stderr.log

echo "Hello There! This stdout statement has been redirected to stderr." 1>&2

The output of cat stderr.log:

Hello There! This stdout statement has been redirected to stderr.

The same thing applies for standard out (stdout), except you use “1″ instead of “2″:

exec 1>> stderr.log

Here’s a quick example script to demonstrate:

#!/bin/sh

exec 2>> stderr.log

echo "Hello There! This is a stdout statement."

The output of cat stderr.log:

Hello There! This is a stdout statement.

A Nice Color Scheme Tester Script for Terminal Shell

This is a shell script that outputs your current unix color scheme. This is handy for when you’re trying to come up with your own theme. It comes from http://www.frexx.de/xterm-256-notes/. I’m taking no original credit for this script. I simply want to share it with the readers since I found it very clean and useful. You simply copy the code to a script, use ‘chmod +x colortheme.sh’ as usual, and then run it with ./colorscheme.sh. I’m going to duplicate the script here for redundancy (I want to make sure it’s always there for myself too!):

#!/bin/bash
#
# Description:
#
#    Prints a color table of 8bg * 8fg * 2 states (regular/bold)
#
# Copyright:
#
#    (C) 2009 Wolfgang Frisch <xororand@unfoog.de>
#
# License:
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation, either version 3 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>.

echo
echo Table for 16-color terminal escape sequences.
echo Replace ESC with \033 in bash.
echo
echo "Background | Foreground colors"
echo "---------------------------------------------------------------------"
for((bg=40;bg<=47;bg++)); do
	for((bold=0;bold<=1;bold++)) do
		echo -en "\033[0m"" ESC[${bg}m   | "
		for((fg=30;fg<=37;fg++)); do
			if [ $bold == "0" ]; then
				echo -en "\033[${bg}m\033[${fg}m [${fg}m  "
			else
				echo -en "\033[${bg}m\033[1;${fg}m [1;${fg}m"
			fi
		done
		echo -e "\033[0m"
	done
	echo "--------------------------------------------------------------------- "
done

echo
echo

Debug Scripts with ‘set -x’

A valuable tool when developing and debugging scripts is set. In particular, the -x or -xtrace options of set will “print a trace of simple commands and their arguments after they are expanded and before they are executed”. Here’s an example of using set -x:

Script

$ cat setx.sh
#!/bin/sh

echo "Hello There!"
echo "Here's the date..."
set -x
date
echo "Also, a calendar..."
cal
echo "Bye!"
$

Execution

$ ./setx.sh
Hello There!
Here's the date...
+ date
Sun Aug  8 21:33:03 MST 2010
+ echo 'Also, a calendar...'
Also, a calendar...
+ cal
    August 2010
Su Mo Tu We Th Fr Sa
 1  2  3  4  5  6  7
 8  9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31

+ echo 'Bye!'
Bye!
$

Be sure to check out the man page for even more useful options of set.

How To: Add a “Press Any Key to Continue” Message to a Script

Adding a “Press any key to continue” message to a script is actually quite easy because it’s already built into the read command. Here’s a couple examples of how to use it:

#!/bin/sh

echo "A First Method"
read -s -n 1 -p "Press any key to continue..."

# insert echo here for cleaner output
echo

echo "A Second Method"
echo "Press any key to continue..."
read -s -n 1 any_key
echo "Now exiting"
exit 0

Note: “any_key” was simply a made up name. It can be anything. As always, be sure to take a look at the man page for more information!

Run Shell Script Subroutines in Terminal with Source Command

If you run a set of commands frequently, you might want to think about creating a sourced shell script with subroutines. For example, maybe you like to see the current date, a calendar, and a quick fortune (with the fortune package installed). Instead of typing cal and then date and then /usr/games/fortune -s manually each time, simply include it in a shell script that contains a subroutine that will do it all for you. All you need to do is create the script and subroutines and then source it like this:

. /my.subr

Notice the space between the period [.] and the script name. I usually use the .subr extension on my sourced scripts to tell them apart, but you can use whatever filename you want (it doesn’t even have to have an extension).

Then, simply call your subroutine within your shell script. Continuing with the example scenario described above, here is the shell script called my.subr that was sourced…

startmyday(){
   cal
   date
   echo
   /usr/games/fortune -s
}

anotherfunction(){
   echo "put whatever you want in this function call"
   echo "this is just a filler."
}

yetanotherfunction(){
   echo "put whatever you want in this function call"
   echo "this is yet another filler."
}

This is showing the sourced shell script subroutines in action…

# startday
     June 2010
Su Mo Tu We Th Fr Sa
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30         

Fri Jun 11 04:17:56 UTC 2010

Do not drink coffee in early A.M.  It will keep you awake until noon.
# anotherfunction
put whatever you want in this function call
this is just a filler.
# yetanotherfunction
put whatever you want in this function call
this is yet another filler.
#

Script to Wait for Wireless AP Association During System Startup

The following is a script that could prove to be useful in an embedded Linux environment utilizing a wireless adapter. In order to connect to a wireless network, the wireless adapter needs to associate with an AP which can take some time occasionally. If you’ve edited the /etc/network/interfaces file to automatically obtain an IP address via DHCP, and it doesn’t seem to be getting an IP address during system startup, then this script might be able to help. This script is intended to run on system startup and wait for the access point association. If one is not found, it will eventually timeout.

#!/bin/bash
#
# CONNECT TO WIRELESS NETWORK
# Author: Errol E. Burrow II <eburrow@gmail.com>
#
# call this script from /etc/rc.local
# this script uses fping which can be installed
# with sudo apt-get install fping
#  

# globals
MYIP="192.168.0.121"
WAPNAME="SSVR3"
GATEWAY="192.168.0.141"
PINGTESTIP="10.10.10.1"
RETRYCOUNT=10

# turn on extra regex features
shopt -s extglob

until [ $RETRYCOUNT -lt 1 ]; do
	ifconfig wlan0 $MYIP
	iwconfig wlan0 essid $WAPNAME
	# get the access point mac address for wlan0
	ap=$(iwconfig wlan0 | sed "s/Access/~+&/" | tr "~" "n" | grep "+" | cut -c16- | tr -d " ")
	echo "access point mac: [$ap]"
	# check if the ap actually has an expected value
	if [[ $ap =~ [0-9a-f]{2}[:-] ]]; then
		echo "success: connected to wifi access point!"
		route add default gw $GATEWAY
		echo "added default gateway $GATEWAY"

		echo "testing ping to $PINGTESTIP"
		if fping -a -r1 $PINGTESTIP > /dev/null
			then
				echo "success: $PINGTESTIP is alive!"
				break
			else
				echo "warning: $PINGTESTIP cannot be reached!"
		fi
	else
		echo "warning: not connected to wifi access point"
		# try again
	fi
	echo "retrying connection: $RETRYCOUNT"
	let RETRYCOUNT-=1
	# sleep a bit
	sleep 1
done