Recently in Free/Open Source Software (F/OSS) Category

My hardware:

• Canon Rebel XTi (uses Compact Flash (CF) memory)
• Inland USB multi-format SD/CF card reader ($12 at Micro Center)
• Dell PowerEdge 1800 running Fedora Core 11

My goal is to start out with files like this on the Compact Flash card::

[CF Card]/dcim/102canon/img_9999.CR2
[CF Card]/dcim/102canon/img_9999.JPG
[CF Card]/dcim/103canon/img_0000.cr2
[CF Card]/dcim/103canon/img_0000.jpg

run the script, and end up with the same files, transferred to the server and renamed like this:

[server photo repository]/2009/200902/20090215/img_1029999.cr2
[server photo repository]/2009/200902/20090215/img_1029999.jpg
[server photo repository]/2009/200902/20090215/img_1030000.cr2
[server photo repository]/2009/200902/20090215/img_1030000.jpg

Removal of files from CF card is performed manually by the Format function on the camera, once I'm sure all the files have survived the trip from CF to hard drive.

The (re)naming convention assures unique file names for photos coming from my camera until I get to 10,000,000 photos. At my current rate of taking pictures, that would be several thousand years. It also allows me to easily manage my photos by year, month, or day (and eventually decade) as I choose.

I've written a script to automatically copy and rename my files from the CF card to an appropriate directory on the server when it is run.

#!/bin/ksh
startdir=`pwd`
cfsourcedir="/mnt/usb"

# make sure to use your device name here, check output of 'dmesg' \
# on your server with card reader connected.
cfdeviceid="/dev/sdc1"
canondir="${cfsourcedir}/dcim"
datepath="/photos/raw/`/bin/date +%Y/%Y%m/%Y%m%d`"

sudo umount $cfsourcedir

sudo mount $cfdeviceid $cfsourcedir
result=$?;

if [ $result -eq 0 ];then

   if [ -d $datepath ]; then
      echo "Directory $datepath exists"
   else
      mkdir $datepath
   fi
   #echo "canondir: $canondir"
   #echo "datepath: $datepath"

   rsync -az $canondir/* $datepath --stats --progress | \
   tee -a /tmp/loadfromcf.out
   cd $datepath
   for i in `find -type d |sed 's/^.\///g' |grep -v ^\.$`
   do
      cd $i
      directorynumber=`echo $i | sed 's/CANON//g' |sed 's/canon//g'`
      for j in `ls -1 *`
      do 
         k=`echo $j |sed "s/img_/img_$directorynumber/g" |\
   sed 's/IMG_/IMG_${directorynumber}/g' |sed 's/JPG/jpg/g' |sed 's/CR2/cr2/g'`
         mv $j ../$k
         chown speed:speed ../$k
         chmod 0544 ../$k
      done
      cd ..
      rmdir $i 
   done
fi

sudo umount /mnt/usb

if [[ $result = 0 ]];then
   grep -i Number /tmp/loadfromcf.out | tail -2
   pwd
fi

I've also contemplated modifying this script so it would automatically (cron) check for a CF card in the reader, then automatically start the copying process. If I make this modification, I'll post it here as well.

I'll be adding code/comments as I improve this script.

I've been working with variations of Unix for a long time now and thought I'd jot down some of my favorite tips and tricks. They are mostly OS/distribution/shell/language independent (unless I indicate otherwise...)

1. Get rid of blank lines in a file
   grep . inputfile > outputfile
   This matches (and thus prints) only lines that contain some text, not blank (empty) lines.


2. comm
   Many people never cross paths with the comm command, but it is very useful. I works similarly to diff, but outputs the contents of two compared files into three columns. The first column is content only in the first file, second column is content only in the second file, and third column is content that is in both files (matches between the two files.) While this may not seem useful at first, you can select which columns to output, so if you only want to know what is in both file1 and file 2 (column 2) you'd suppress columns 1 and 2, by running:
   comm -12 file1 file2
   Don't forget that your input files must be sorted.


3. paste
   Systems administrators frequently use the cut command to parse files, but many people I run into have never used the paste command. The paste command will concatenate two files line by line (as opposed to file by file, like cat.)


4. less instead of more
   This is not available on all Unix-based OSes, but the less command works very similar to more, but will let you move through a file forwards and backwards more easily. Want to jump to the end of the file, type Shift-G Depending on the version of less you are running, it will provide context highlighting when you search for a pattern.


5. Jump to vi from more
   While paging through a file in more, press "v" to jump to editting the file in vi at the current position in the file.


6. Jump to a line number when editing a file with vi
   vi +linenumber filename
   will open up the file with the cursor automatically moved down to the specified line. This is useful when you get an error that indicates "syntax error on line 2047." You can jump straight to the problem without fumbling around.


7. Invisible characters become visible
   Sometimes you'll end up with carriage returns on each line in a file originally created on a DOS/Windows system, or filenames with spaces, tab, or other control characters in them, but you can't see them typically.
   The cat command provides three useful options -v, -e, and -t that will let you understand these invisible characters
   -v (displays non-printing characters)
-e (prints a "$" at the end of each line to indicate a NL character)
-t (prints "^I" for each Tab in the file)
cat -vet filename |more


8. Remove DOS ^M from ends of lines
   The "^M" characters are visible when editing in vi but here are two approaches to remove the characters.
sed 's/<Ctrl-V><Ctrl-M>//g' -i filename
   or in vi: <Esc>:%s/<Ctrl-V><Ctrl-M>//g

If you ever do end up with me interviewing you, I'll likely work one or two of these into the discussion to explore your level of knowledge about Unix. If you need more information, remember: man pages are your friend.