Recently in Linux Category

So you fell for Sandisk's marketing line about "supporting Linux." The full truth is they are Linux-tolerant and their device behaves well with Linux. Expect that you'll be doing a lot of the leg work on your own. I've pulled together resources I've found and created to put all the pieces in one place so you can at least start where I've left off.

Preparing your Fuze
First, update your firmware to the latest version and be sure you set your Fuze to connect in MSC mode.
From Fuze menu: Settings -> System Settings -> USB Mode -> MSC
(everytime you upgrade your Fuze you'll have to re-set the USB Mode.)

Re-labeling your Fuze
The VFAT label of your Fuze controls the name of the mountpoint when your Fuze is automounted at. It will be mounted in /media/VFAT_LABELNAME/ (/media/SANSA FUZE/ by default.) You can re-label the VFAT filesystem of your Fuze to eliminate the space character (an inconvenience), give your Fuze a more meaningful name, or give your Fuze a differentiating name if you use multiple Fuzes.

1. You need to have the 'mtools' package installed and configure a .mtoolsrc file in your home directory.
   sudo apt-get install mtools

2. run 'mount' and note the /dev/ name of your mounted Fuze (default will look something like "/dev/sdc on /media/SANSA FUZE type vfat") in this case "/dev/sdc" is the value to note, your device name may vary.

3. Create a .mtoolsrc file in your home directory:
   gedit .mtoolsrc

   with the following contents:
   drive f: file="/dev/sdc"
   mtools_skip_check=1
   (please substitute your device name from step 2 above for "/dev/sdc" above when you type it in...)

4. Then run the 'mlabel' command as root to re-label your Fuze:
   sudo mlabel f:

   You'll be prompted to create a new label, choose one without spaces, perhaps choose a unique label for if you use multiple Fuzes.
   

Content Management
I've configured scripts to replicate all (MSC) files from the Fuze (and microSD card on-board the Fuze) to my home directory on my primary computer. This creates a complete backup of all content on the Fuze (and microSD) and allows me to manage files either on the Fuze or to the local directory in my home directory. I can add songs into the directory on the computer when the Fuze is absent (left at work, etc.) and re-sync the files onto the Fuze when it is back at home...

This is done using a set of simple scripts calling the Unix rsync command to mirror two sets of files. It is very efficient and only transfers files that have changed, so even a large dataset transfers quickly if you've only changed a couple files. I

Example rsync script:
script goes here in the near future

Music
Not a lot to note here, the Fuze supports a wide range of music files from MP3, AAC, FLAC, OGG and seems to be fairly tolerant (I haven't seen any "Unsupported Media Format" errors for music files.)

Podcasts
Choose your favorite Podcatcher and point it at your /

FM Radio

Voice Recording

Slot Radio
I haven't sprung for a Slot Radio card yet (and I don't want to displace my microSDHC card) but I'm wondering what sort of DRM has been implemented on them. The marketing materials say you get 1,000 songs, but they play in a random (or at least randomized) order, you can skip forward but not backwards. Apparently you can add more media to the Slot Radio card if it has free space remaining.

Images
The Fuze will downsize your JPGs if you, but there's not much point in putting large .JPG files on your Fuze just to eat up storage. I prefer resizing the images appropriately before putting them on the Fuze. video4fuze is the easiest option for properly resizing images for display on the Fuze. (Note this is the same tool as the "Video" section below.)

Video
The Fuze is very strict about video. Deviating from the standard will result in "Unsupported Media Format".

The easiest general solution for Video currently is video4fuze which provides you a easy GUI interface to video transcoding (using mencoder). You can combine this tool with the rsync technique mentioned above to transcode videos offline, then sync onto your Fuze later.

Miscellaneous Useful Specs
From the owner's manual:
Screen resolution: 220x176

Notes:
3:2 aspect ratio = 219x146 or 171x114
4:3 aspect ratio = 220x165 or 176x132
16:9 aspect ratio = 208x117 or 176x99

Shortfalls:
There doesn't seem to be anyway to sort or arrange files using subdirectories (especially photos and videos.) You can create subdirectories, but the Fuze flattens the filespace out and sorts them all alphabetically.

I'm considering some different approaches to organizing my MP3s to make them easier to browse through. Sansa has a lot of room for improvement in this area.

We recently got two Crayola-branded 2.1 megapixel kids cameras. There are also 1mp and 5mp variants on the market with Crayola branding. These are very fundamental cameras, slightly kid-ruggedized with very few options. You have two choices for resolution (1600x1200 or 1280x1024), two choices for image quality, and a video option. The camera does not take very good low-light photos and tends to blow out the picture with the flash. The flash is fully automatic with no settings or ability to switch it on or off as needed.

As these came from the factory, I had significant problems with duplicate directory names /foo/bar/DCIM/JLCAM1/) that could not be properly accessed. Despite the fact the you could browse photos on the camera itself, no files showed up when connected via USB to Ubuntu (the filesystem mounted fine, but no files showed up below that directory.) I was able to use photorec to successfully 'recover' the files that were not visible to the system successfully from the camera.

Here's some output with some basic system commands:
# ls -l /media
drwx------ 3 speed speed 4096 1969-12-31 18:00 RIAN2GBUSD
drwx------ 3 speed speed 16384 1969-12-31 18:00 RIANCAM

# find /media/RIAN2GBUSD -exec ls -li {} \;
227491 drwx------ 2 speed speed 4096 2010-01-01 12:01 DCIM
227494 drwx------ 0 speed speed 0 2011-01-03 12:59 100JLCAM
227494 drwx------ 0 speed speed 0 2011-01-03 12:59 100JLCAM
(note duplicate directories with identical inodes)

# find /media/RIANCAM -exec ls -li {} \;
227496 drwx------ 2 speed speed 2048 2010-01-01 12:01 DCIM
227499 drwx------ 0 speed speed 0 2011-03-07 22:09 100JLCAM
227499 drwx------ 0 speed speed 0 2011-03-07 22:09 100JLCAM

# ls -la /media/RIANCAM/DCIM/100JLCAM/
total 0
# ls -la /media/RIAN2GBUSD/DCIM/100JLCAM/
total 0
(note apparently empty directories)

Despite these obvious indications to the contrary (remember, I can see the photos on the camera itself) I was able to extract all 251 photos successfully. The biggest catch is photorec takes quite a while (30 minutes) to process a little over 2GB of storage (much of it unused space.)

I re-formatted the filesystem on the camera and re-created, labeled, and formatted a new FAT16 (<32M) partition. Now pictures appear to show up as normal in the Nautilus File Browser when the camera is connected via USB.

Crayola Camera:
Serial Number: JL2007BV2000
Firmware Version: 1.0
lsusb output:
ID 0979:0371 Jeilin Technology Corp., Ltd

Storage capacity: 26 MB (25,935,872 bytes) Internal
Partitioning: Master Boot Record
Partition Type: FAT16 (<32M) (0x04)
Photo size: 1600x1200 24bit JPEG (~128KB filesize)
Focal length: 28.8mm (35 mm equivalent: 194.3 mm) (as reported by exiftool)
Aperture: f/2.8

Unless you've got the correct packages installed, you'll likely get an error with some missing dependencies. You can find the missing packages at:

libicu38
libboost-filesystem1.34.1
libboost-regex1.34.1
libboost-thread1.34.1
libboost-iostreams1.34.1
libboost-signals1.34.1
libboost-date-time1.34.1

Click the above links, choose your architecture, choose a mirror, then download and open with GDebi.

Then install the Amazon MP3 downloader as normal.

When performing a find and execing a grep on the results (scenario: searching for a string in all files in a directory and all subdirectories recursively) on Solaris, it has always frustrated me that the output shows matches but doesn't identify the filename that the match occurred in.

$ find . -type f -exec grep string {} \;

horse
horsefeathers

I've worked out a couple solutions that were cumbersome and kludgy when I recently came across a simple, direct solution to the problem.

In Solaris, when you grep for a pattern in one file, the grep command doesn't output the name of the file a match is found in. There is no option to force the display of filenames (like -H in Linux) so you are left to engineer a solution yourself. The best solution I've worked out involves providing a second filename (/dev/null) to the exec'd grep command, forcing it to print out the filename a match occurred in, when a match occurs.

$ find . -type f -exec grep string {} /dev/null \;

./filename1:horse
./filename2:horsefeathers

Problem solved and the solution is cross-platform.

My hardware:

• Canon Rebel XTi (uses Compact Flash (CF) memory)
• Inland USB multi-format SD/CF card reader ($12 at Micro Center)
• Dell PowerEdge 1800 running Fedora Core 11

My goal is to start out with files like this on the Compact Flash card::

[CF Card]/dcim/102canon/img_9999.CR2
[CF Card]/dcim/102canon/img_9999.JPG
[CF Card]/dcim/103canon/img_0000.cr2
[CF Card]/dcim/103canon/img_0000.jpg

run the script, and end up with the same files, transferred to the server and renamed like this:

[server photo repository]/2009/200902/20090215/img_1029999.cr2
[server photo repository]/2009/200902/20090215/img_1029999.jpg
[server photo repository]/2009/200902/20090215/img_1030000.cr2
[server photo repository]/2009/200902/20090215/img_1030000.jpg

Removal of files from CF card is performed manually by the Format function on the camera, once I'm sure all the files have survived the trip from CF to hard drive.

The (re)naming convention assures unique file names for photos coming from my camera until I get to 10,000,000 photos. At my current rate of taking pictures, that would be several thousand years. It also allows me to easily manage my photos by year, month, or day (and eventually decade) as I choose.

I've written a script to automatically copy and rename my files from the CF card to an appropriate directory on the server when it is run.

#!/bin/ksh
startdir=`pwd`
cfsourcedir="/mnt/usb"

# make sure to use your device name here, check output of 'dmesg' \
# on your server with card reader connected.
cfdeviceid="/dev/sdc1"
canondir="${cfsourcedir}/dcim"
datepath="/photos/raw/`/bin/date +%Y/%Y%m/%Y%m%d`"

sudo umount $cfsourcedir

sudo mount $cfdeviceid $cfsourcedir
result=$?;

if [ $result -eq 0 ];then

   if [ -d $datepath ]; then
      echo "Directory $datepath exists"
   else
      mkdir $datepath
   fi
   #echo "canondir: $canondir"
   #echo "datepath: $datepath"

   rsync -az $canondir/* $datepath --stats --progress | \
   tee -a /tmp/loadfromcf.out
   cd $datepath
   for i in `find -type d |sed 's/^.\///g' |grep -v ^\.$`
   do
      cd $i
      directorynumber=`echo $i | sed 's/CANON//g' |sed 's/canon//g'`
      for j in `ls -1 *`
      do 
         k=`echo $j |sed "s/img_/img_$directorynumber/g" |\
   sed 's/IMG_/IMG_${directorynumber}/g' |sed 's/JPG/jpg/g' |sed 's/CR2/cr2/g'`
         mv $j ../$k
         chown speed:speed ../$k
         chmod 0544 ../$k
      done
      cd ..
      rmdir $i 
   done
fi

sudo umount /mnt/usb

if [[ $result = 0 ]];then
   grep -i Number /tmp/loadfromcf.out | tail -2
   pwd
fi

I've also contemplated modifying this script so it would automatically (cron) check for a CF card in the reader, then automatically start the copying process. If I make this modification, I'll post it here as well.

I'll be adding code/comments as I improve this script.

I've been working with variations of Unix for a long time now and thought I'd jot down some of my favorite tips and tricks. They are mostly OS/distribution/shell/language independent (unless I indicate otherwise...)

1. Get rid of blank lines in a file
   grep . inputfile > outputfile
   This matches (and thus prints) only lines that contain some text, not blank (empty) lines.


2. comm
   Many people never cross paths with the comm command, but it is very useful. I works similarly to diff, but outputs the contents of two compared files into three columns. The first column is content only in the first file, second column is content only in the second file, and third column is content that is in both files (matches between the two files.) While this may not seem useful at first, you can select which columns to output, so if you only want to know what is in both file1 and file 2 (column 2) you'd suppress columns 1 and 2, by running:
   comm -12 file1 file2
   Don't forget that your input files must be sorted.


3. paste
   Systems administrators frequently use the cut command to parse files, but many people I run into have never used the paste command. The paste command will concatenate two files line by line (as opposed to file by file, like cat.)


4. less instead of more
   This is not available on all Unix-based OSes, but the less command works very similar to more, but will let you move through a file forwards and backwards more easily. Want to jump to the end of the file, type Shift-G Depending on the version of less you are running, it will provide context highlighting when you search for a pattern.


5. Jump to vi from more
   While paging through a file in more, press "v" to jump to editting the file in vi at the current position in the file.


6. Jump to a line number when editing a file with vi
   vi +linenumber filename
   will open up the file with the cursor automatically moved down to the specified line. This is useful when you get an error that indicates "syntax error on line 2047." You can jump straight to the problem without fumbling around.


7. Invisible characters become visible
   Sometimes you'll end up with carriage returns on each line in a file originally created on a DOS/Windows system, or filenames with spaces, tab, or other control characters in them, but you can't see them typically.
   The cat command provides three useful options -v, -e, and -t that will let you understand these invisible characters
   -v (displays non-printing characters)
-e (prints a "$" at the end of each line to indicate a NL character)
-t (prints "^I" for each Tab in the file)
cat -vet filename |more


8. Remove DOS ^M from ends of lines
   The "^M" characters are visible when editing in vi but here are two approaches to remove the characters.
sed 's/<Ctrl-V><Ctrl-M>//g' -i filename
   or in vi: <Esc>:%s/<Ctrl-V><Ctrl-M>//g

If you ever do end up with me interviewing you, I'll likely work one or two of these into the discussion to explore your level of knowledge about Unix. If you need more information, remember: man pages are your friend.

A while back I contacted the top five or so vendors of home weather stations to see if they offered any product that would simply measure various atmospheric variables and allow the home user to poll the station for data via some sort of open API. My search came up empty. Most of the time I felt like I had just asked them if they made pants with four legs.

I know there is a segment of consumers want a weather appliance that sits on their counter and tells them the weather. There is another segment that wants to be able to work with and track the data (for which some vendors provide their own software, usually Windows only.) Still another segments literally wants to just have the information available through an API that I can interact with. That last segment is me; that's what I'm looking for.

If you have a weather station that will:

• provide basic weather information
  • temperature
  • humidity
  • dew point
  • wind speed
  • wind direction
  • barometric pressure
  • precipitation
• allow me to poll the weather station
  • when I want
  • for what values I want
  • from on open API from whatever computer OS I want

then I would like to talk to you about purchasing one of your weather stations.

You can reach me at speed-weat@transmit.net.

I am currently using a Canon Rebel XTi DSLR camera that records onto Compact Flash media. I'm primarily interested in filing, categorizing and annotating my image files. Post-processing (raw conversion, adjustments, etc.) is usually minimal unless I really messed up my settings or am working commercially.

1. Transfer the images, as taken, to fileserver in a working directory. This is done via USB-connected Compact Flash reader attached to my fileserver (I'm considering adding a built-in card reader if my motherboard provides USB connections internally, need to investigate.) The files are written to a dedicated RAID 1 share on my Fedora Core 6 (currently) Linux fileserver.
2. Set the Compact Flash card aside, just in case...
3. From Unix shell, check that the file ownership and permissions are set correctly on the copied files. File permissions are all changed to 0544 for my purposes, so the files are read-only, but my automated system can access the files.
4. If necessary, renumber (rename) files. Canon stores 10,000 photos per folder, then automatically increments to the next folder. I prefer to append the folder number to the image number to form individual file names like so:
   DCIM/CANON101/IMG_2345.JPG becomes IMG_1012345.JPG. The little shell script I use to rename my files looks like this:

   for i in `ls -1`
         do
         j=`echo $i |sed `s/IMG_/IMG_101/g'`
         mv $i $j
         chmod 0544 $j
      done
   

   I currently keep all my photos in one directory (may rethink this later) so it is important for me to keep filenames unique.
5. Record the filenames of the new files in the work directory, move them to the production directory.
6. Process the photos through raw conversion using Bibble Pro
7. Process the new files (recorded in the previous step) through the script that loads the image records, vital statistics and meta data (EXIF) into the database:
   • sample the image and record the average color in HEX
   • record default values (photographer, photographer e-mail, photographer URL, file directory, filename, url
   • extract EXIF data with jhead and load that data
   • create appropriate .html files which call scripts via SSI and verify new .html file permissions
   • categorize photos where I can by date/time and other extracted information
8. Create test lightbox view of thumbnails
9. Correct any images that need their orientation (rotation) adjusted. Usually this is a rotate operation -90 degrees to counteract the camera being used vertically instead of horizontally.Now done in Bibble Pro, so not an issue.
10. Rebuild thumbnail files for new images. Other than thumbnails, all images are resized on-demand, real-time, and fed as a BLOB from my server to your browser. Neat tricks with Perl, Imagemagick, and Apache.
11. Rebuild category files to account for new files assigned to categories.
12. Verify that everything works correctly and looks sane/sober/kosher.
13. Format Compact Flash media.

At this point, I've got all the information about the image into the database and appropriate web pages have been built to index and display the photos on my online photo galleries. Now I can sleep restfully. Additional notation, categorization, sorting, filing, etc. is an on-going task (with 20,500+ photos and growing, it may never be finished.) The photo files and databases are backed up on a RAID 1 partition locally (to protect against hardware failure) and are also mirrored remotely (using rsync) so I'll always have a copy off-site (in case of local catastrophe (weather, fire, etc.))

I'm in a constant development process with this project, right now. The to-do list is long. Here are some details of the current implementation.

All photos have appropriate copyright notation and all rights are reserved. All photos are available for licensing. All photos are available in their original, unmodified resolution and format. Please contact me for specific information about licensing.

Anyone else out there tired of messing with junk video players that do everything but play the video you want to watch? They all work 90% of the time and leave you hanging the rest.

I've found the Golden Video Player: VLC Media Player

This thing has a library of useful features and supports Windows, WinCE, BeOS, and every version of Unix (Mac OS/X, Linux, Solaris, BSDs) you could ever want. It plays almost every format on one player, across multiple platforms, reliably. Wow! Did I mention it's free and the source code is available! As if this wasn't enough, it can also be used as a video streaming server as well.

The install on Windows platforms is absolutely painless. My install on Fedora Core 4 was a bit more challenging, but I finally got all the pieces together. I highly recommend using apt-get and making sure you've got the "dag" repository in your /etc/apt/sources.list file, then, as root, issue the command:
apt-get install videolan-client
(Note: not 'vlc'.) There will be a bunch of dependent packages that will tag along for the install automatically.

This is what Free Software is all about. This project sits on the shoulders of many other quality free and open source software efforts. No contribution is required, but if you'd like to contribute your time, materials, skills, or anything else the VideoLAN team, I'm sure they would appreciate it.

I've had a Sony Mavica MVC-CD300 digital camera for several years now. I love the camera in general. I've taken just over 10,000 photos with the camera in the last four and a half years (many are available for viewing online.) It's like a good old friend at this point.

It's a 3.3 mega pixel camera, Carl Zeiss lens with 3x optical zoom, writes to 8cm mini-CD/RW discs, a nice all-around camera.

Recently my old friend pulled a new trick on me.

Sony chose to name the photo files on the CD in the form DSC#####.JPG on the discs. My very first picture was DSC00001.JPG and my 9999th picture was DSC09999.JPG. I expected the picture after that would be DSC10000.JPG (since there are 5 digits available.) No, it rolled over and started at DSC00001.JPG again. Since I use the original filenames of my pictures when filing them, this causes a bit of a problem now as I'm potentially trying to store multiple files with the same filenames into my photo repository. Not a very nice implementation, Sony. You've got five digits, why not use them?

As part of my work flow, I load my photos into a 'working' directory and change their owner and permissions (this is in Linux) so they are owned by me and are 'a-w' (actually 0544) on their permissions. This makes it so the file will usually not be overwritten, or at least I'll see a warning message if I try to overwrite or delete the file. I've since added another step to my work flow, I run the following short bash/ksh script on all files in my 'working' directory now that I've passed the 10,000 mark:

for i in `ls -1 DSC0*.JPG`
do
   j=`echo $i | sed 's/SC0/SC1/g'`
   mv -i $i $j
done

It will rename any 'SC0' pattern to 'SC1', effectively renaming all my newer DSC0####.JPG files to DSC1####.JPG, as Sony should have done in the first place. I have listed the Unix 'mv' command with the '-i' interactive switch so you'll have to confirm each rename. Once you are comfortable with the script (try it on ONE file first) you can remove the '-i' script and it will rename all DSC0*.JPG files in the current directory. I'll still have to update my script once every 10,000 photos, but at four and a half years per update, I think I can manage.

Do all Sony Mavica cameras do this? Is this a function of the naming standard (DSC* seems to be vendor-independent)?

I'll post more on my workflow in a separate entry.

[Update: 7/27/2005]
The standard Sony uses for file naming is known as "Camera File System - CP-3461" (.PDF) (or DCF for short) The standards body that published this document is JEITA, the same folks publishing the EXIF standard - CP-3451(and -1) (.PDF). They are quite active in the realm of digital camera standards. Many big Japanese electronics manufacturers are members of JEITA.

DSC=Digital Still Camera
DCIM=Digital Camera Images

So, Sony decided to follow JEITA CP-3461 section 5.2.1 which splits the 8 characters available into 4 "Free" (vendor-assignable) characters and 4 "File numbering" characters. To paraphrase the standard, files are numbered 0001 to 9999, 0000 shall not be used. This effectively limits the consumer to 9,999 unique filenames (per directory.) The standard does have some wording in it about multiple directories, but that assumes the files will only exist in their original directory and will never cross paths with duplicately named files elsewhere. Sony doesn't seem to have implemented these variable directory names in their first- and second-generation Mavica CD cameras (CD1000, CD200, CD300.) So, the blame is off Sony and on JEITA.

JEITA: When facing a similar problem to the one computer network technicians face when addressing networks, why would you tie up half your address space (4 out of 8 character positions) with free text? Why not a minimally small, manufacturer-unique identification code (alpha-numeric), maybe one or two characters? That would at least allow a larger "address space" for the file numbering. Another approach would have been using an alpha-numeric code for the "File numbering" positions, greatly increasing the usable number of pictures before duplication of filenames.

If 5 numeric character places are used for "File numbering" the limit pushes to 99,999, 6 places yields 999,999 unique filenames, a much more reasonable figure. If alpha-numeric characters are used for file numbering (0-9, then A-Z, in order) even the 4-place scheme works better, yielding 1.67 million unique file names before repeating.

Do the Japanese engineers only think consumers will take less than 10,000 pictures with their cameras? Do they know something you don't? Mine (a Sony Mavica CD300 review, review, review) made it to 10,500 and is going strong.
[/Update]