Recently in Linux Category

When performing a find and execing a grep on the results (scenario: searching for a string in all files in a directory and all subdirectories recursively) on Solaris, it has always frustrated me that the output shows matches but doesn't identify the filename that the match occurred in.

$ find . -type f -exec grep string {} \;

horse
horsefeathers

I've worked out a couple solutions that were cumbersome and kludgy when I recently came across a simple, direct solution to the problem.

In Solaris, when you grep for a pattern in one file, the grep command doesn't output the name of the file a match is found in. There is no option to force the display of filenames (like -H in Linux) so you are left to engineer a solution yourself. The best solution I've worked out involves providing a second filename (/dev/null) to the exec'd grep command, forcing it to print out the filename a match occurred in, when a match occurs.

$ find . -type f -exec grep string {} /dev/null \;

./filename1:horse
./filename2:horsefeathers

Problem solved and the solution is cross-platform.

My hardware:

• Canon Rebel XTi (uses Compact Flash (CF) memory)
• Inland USB multi-format SD/CF card reader ($12 at Micro Center)
• Dell PowerEdge 1800 running Fedora Core 10

My goal is to start out with files like this on the Compact Flash card::

[CF Card]/dcim/102canon/img_9999.CR2
[CF Card]/dcim/102canon/img_9999.JPG
[CF Card]/dcim/103canon/img_0000.cr2
[CF Card]/dcim/103canon/img_0000.jpg

run the script, and end up with the same files, transferred to the server and renamed like this:

[server photo repository]/2009/200902/20090215/img_1029999.cr2
[server photo repository]/2009/200902/20090215/img_1029999.jpg
[server photo repository]/2009/200902/20090215/img_1030000.cr2
[server photo repository]/2009/200902/20090215/img_1030000.jpg

Removal of files from CF card is performed manually by the Format function on the camera, once I'm sure all the files have survived the trip from CF to hard drive.

The (re)naming convention assures unique file names for photos coming from my camera until I get to 10,000,000 photos. At my current rate of taking pictures, that would be several thousand years. It also allows me to easily manage my photos by year, month, or day (and eventually decade) as I choose.

I've written a script to automatically copy and rename my files from the CF card to an appropriate directory on the server when it is run.

#!/bin/ksh
startdir=`pwd`
cfsourcedir="/mnt/usb"

# make sure to use your device name here, check output of 'dmesg' \
# on your server with card reader connected.
cfdeviceid="/dev/sdc1"
canondir="${cfsourcedir}/dcim"
datepath="/photos/raw/`/bin/date +%Y/%Y%m/%Y%m%d`"

sudo umount $cfsourcedir

sudo mount $cfdeviceid $cfsourcedir
result=$?;

if [ $result -eq 0 ];then

   if [ -d $datepath ]; then
      echo "Directory $datepath exists"
   else
      mkdir $datepath
   fi
   #echo "canondir: $canondir"
   #echo "datepath: $datepath"

   rsync -az $canondir/* $datepath --stats --progress | \
   tee -a /tmp/loadfromcf.out
   cd $datepath
   for i in `find -type d |sed 's/^.\///g' |grep -v ^\.$`
   do
      cd $i
      directorynumber=`echo $i | sed 's/CANON//g' |sed 's/canon//g'`
      for j in `ls -1 *`
      do 
         k=`echo $j |sed "s/img_/img_$directorynumber/g" |\
   sed 's/IMG_/IMG_${directorynumber}/g' |sed 's/JPG/jpg/g' |sed 's/CR2/cr2/g'`
         mv $j ../$k
         chown speed:speed ../$k
         chmod 0544 ../$k
      done
      cd ..
      rmdir $i 
   done
fi

sudo umount /mnt/usb

if [[ $result = 0 ]];then
   grep -i Number /tmp/loadfromcf.out | tail -2
   pwd
fi

I've also contemplated modifying this script so it would automatically (cron) check for a CF card in the reader, then automatically start the copying process. If I make this modification, I'll post it here as well.

I'll be adding code/comments as I improve this script.

I've been working with variations of Unix for a long time now and thought I'd jot down some of my favorite tips and tricks. They are mostly OS/distribution/shell/language independent (unless I indicate otherwise...)

1. Get rid of blank lines in a file
   grep . inputfile > outputfile
   This matches (and thus prints) only lines that contain some text, not blank (empty) lines.


2. comm
   Many people never cross paths with the comm command, but it is very useful. I works similarly to diff, but outputs the contents of two compared files into three columns. The first column is content only in the first file, second column is content only in the second file, and third column is content that is in both files (matches between the two files.) While this may not seem useful at first, you can select which columns to output, so if you only want to know what is in both file1 and file 2 (column 2) you'd suppress columns 1 and 2, by running:
   comm -12 file1 file2
   Don't forget that your input files must be sorted.


3. paste
   Systems administrators frequently use the cut command to parse files, but many people I run into have never used the paste command. The paste command will concatenate two files line by line (as opposed to file by file, like cat.)


4. less instead of more
   This is not available on all Unix-based OSes, but the less command works very similar to more, but will let you move through a file forwards and backwards more easily. Want to jump to the end of the file, type Shift-G Depending on the version of less you are running, it will provide context highlighting when you search for a pattern.


5. Jump to vi from more
   While paging through a file in more, press "v" to jump to editting the file in vi at the current position in the file.


6. Jump to a line number when editing a file with vi
   vi +linenumber filename
   will open up the file with the cursor automatically moved down to the specified line. This is useful when you get an error that indicates "syntax error on line 2047." You can jump straight to the problem without fumbling around.


7. Invisible characters become visible
   Sometimes you'll end up with carriage returns on each line in a file originally created on a DOS/Windows system, or filenames with spaces, tab, or other control characters in them, but you can't see them typically.
   The cat command provides three useful options -v, -e, and -t that will let you understand these invisible characters
   -v (displays non-printing characters)
-e (prints a "$" at the end of each line to indicate a NL character)
-t (prints "^I" for each Tab in the file)
cat -vet filename |more


8. Remove DOS ^M from ends of lines
   The "^M" characters are visible when editing in vi but here are two approaches to remove the characters.
sed 's/<Ctrl-V><Ctrl-M>//g' -i filename
   or in vi: <Esc>:%s/<Ctrl-V><Ctrl-M>//g

If you ever do end up with me interviewing you, I'll likely work one or two of these into the discussion to explore your level of knowledge about Unix. If you need more information, remember: man pages are your friend.

A while back I contacted the top five or so vendors of home weather stations to see if they offered any product that would simply measure various atmospheric variables and allow the home user to poll the station for data via some sort of open API. My search came up empty. Most of the time I felt like I had just asked them if they made pants with four legs.

I know there is a segment of consumers want a weather appliance that sits on their counter and tells them the weather. There is another segment that wants to be able to work with and track the data (for which some vendors provide their own software, usually Windows only.) Still another segments literally wants to just have the information available through an API that I can interact with. That last segment is me; that's what I'm looking for.

If you have a weather station that will:

• provide basic weather information
  • temperature
  • humidity
  • dew point
  • wind speed
  • wind direction
  • barometric pressure
  • precipitation
• allow me to poll the weather station
  • when I want
  • for what values I want
  • from on open API from whatever computer OS I want

then I would like to talk to you about purchasing one of your weather stations.

You can reach me at speed-weat@transmit.net.

I am currently using a Canon Rebel XTi DSLR camera that records onto Compact Flash media. I'm primarily interested in filing, categorizing and annotating my image files. Post-processing (raw conversion, adjustments, etc.) is usually minimal unless I really messed up my settings or am working commercially.

1. Transfer the images, as taken, to fileserver in a working directory. This is done via USB-connected Compact Flash reader attached to my fileserver (I'm considering adding a built-in card reader if my motherboard provides USB connections internally, need to investigate.) The files are written to a dedicated RAID 1 share on my Fedora Core 6 (currently) Linux fileserver.
2. Set the Compact Flash card aside, just in case...
3. From Unix shell, check that the file ownership and permissions are set correctly on the copied files. File permissions are all changed to 0544 for my purposes, so the files are read-only, but my automated system can access the files.
4. If necessary, renumber (rename) files. Canon stores 10,000 photos per folder, then automatically increments to the next folder. I prefer to append the folder number to the image number to form individual file names like so:
   DCIM/CANON101/IMG_2345.JPG becomes IMG_1012345.JPG. The little shell script I use to rename my files looks like this:

   for i in `ls -1`
         do
         j=`echo $i |sed `s/IMG_/IMG_101/g'`
         mv $i $j
         chmod 0544 $j
      done
   

   I currently keep all my photos in one directory (may rethink this later) so it is important for me to keep filenames unique.
5. Record the filenames of the new files in the work directory, move them to the production directory.
6. Process the photos through raw conversion using Bibble Pro
7. Process the new files (recorded in the previous step) through the script that loads the image records, vital statistics and meta data (EXIF) into the database:
   • sample the image and record the average color in HEX
   • record default values (photographer, photographer e-mail, photographer URL, file directory, filename, url
   • extract EXIF data with jhead and load that data
   • create appropriate .html files which call scripts via SSI and verify new .html file permissions
   • categorize photos where I can by date/time and other extracted information
8. Create test lightbox view of thumbnails
9. Correct any images that need their orientation (rotation) adjusted. Usually this is a rotate operation -90 degrees to counteract the camera being used vertically instead of horizontally.Now done in Bibble Pro, so not an issue.
10. Rebuild thumbnail files for new images. Other than thumbnails, all images are resized on-demand, real-time, and fed as a BLOB from my server to your browser. Neat tricks with Perl, Imagemagick, and Apache.
11. Rebuild category files to account for new files assigned to categories.
12. Verify that everything works correctly and looks sane/sober/kosher.
Format Compact Flash media.

At this point, I've got all the information about the image into the database and appropriate web pages have been built to index and display the photos on my online photo galleries. Now I can sleep restfully. Additional notation, categorization, sorting, filing, etc. is an on-going task (with 20,500+ photos and growing, it may never be finished.) The photo files and databases are backed up on a RAID 1 partition locally (to protect against hardware failure) and are also mirrored remotely (using rsync) so I'll always have a copy off-site (in case of local catastrophe (weather, fire, etc.))

I'm in a constant development process with this project, right now. The to-do list is long. Here are some details of the current implementation.

All photos have appropriate copyright notation and all rights are reserved. All photos are available for licensing. All photos are available in their original, unmodified resolution and format. Please contact me for specific information about licensing.

Anyone else out there tired of messing with junk video players that do everything but play the video you want to watch? They all work 90% of the time and leave you hanging the rest.

I've found the Golden Video Player: VLC Media Player

This thing has a library of useful features and supports Windows, WinCE, BeOS, and every version of Unix (Mac OS/X, Linux, Solaris, BSDs) you could ever want. It plays almost every format on one player, across multiple platforms, reliably. Wow! Did I mention it's free and the source code is available! As if this wasn't enough, it can also be used as a video streaming server as well.

The install on Windows platforms is absolutely painless. My install on Fedora Core 4 was a bit more challenging, but I finally got all the pieces together. I highly recommend using apt-get and making sure you've got the "dag" repository in your /etc/apt/sources.list file, then, as root, issue the command:
apt-get install videolan-client
(Note: not 'vlc'.) There will be a bunch of dependent packages that will tag along for the install automatically.

This is what Free Software is all about. This project sits on the shoulders of many other quality free and open source software efforts. No contribution is required, but if you'd like to contribute your time, materials, skills, or anything else the VideoLAN team, I'm sure they would appreciate it.

After reading about the Mattel Juiceboxes available at discount pricing story on Slashdot a couple months back, I bit on the Overstock.Com deal and bought a set of 6 new, but discontinued Mattel JuiceBox toy multimedia players for the low, low price of around $25. These originally retailed for $70-80+ each (small wonder they failed) so the chance to buy a Linux (uClinux) device to experiment with for a few pennies on the dollar of the original price was too good to pass. I'd read they could be used to play mpg video, jpg and mp3 audio files, but found out too late that a MP3 Adapter Kit would be required to do this. By the time I went to go track down the adapters, Mattel had sold out and only a few are available at premium prices on-line. I intended to use 3 of these as small photo frames and 3 to experiment with.

I'm stuck with 6 JuiceBoxen that won't do anything but play their demo and perform some simple system diags because I missed the rush on MP3 Adapter Kits for these multimedia players. I'm hoping someone bought too many MP3 Adapter Kits and would be willing to trade or sell them for a reasonable price. I'd like to trade 3 NIB Juiceboxes for 3 MP3 Adapter Kits straight up if someone needs Juiceboxes but has too many MP3 Adapter Kits. If you want to sell you MP3 Adapter Kits, let me know your pricing. Drop me a line if you've got these MP3 Adapter Kits you want to trade or sell.

I've had a Sony Mavica MVC-CD300 digital camera for several years now. I love the camera in general. I've taken just over 10,000 photos with the camera in the last four and a half years (many are available for viewing online.) It's like a good old friend at this point.

It's a 3.3 mega pixel camera, Carl Zeiss lens with 3x optical zoom, writes to 8cm mini-CD/RW discs, a nice all-around camera.

Recently my old friend pulled a new trick on me.

Sony chose to name the photo files on the CD in the form DSC#####.JPG on the discs. My very first picture was DSC00001.JPG and my 9999th picture was DSC09999.JPG. I expected the picture after that would be DSC10000.JPG (since there are 5 digits available.) No, it rolled over and started at DSC00001.JPG again. Since I use the original filenames of my pictures when filing them, this causes a bit of a problem now as I'm potentially trying to store multiple files with the same filenames into my photo repository. Not a very nice implementation, Sony. You've got five digits, why not use them?

As part of my work flow, I load my photos into a 'working' directory and change their owner and permissions (this is in Linux) so they are owned by me and are 'a-w' (actually 0544) on their permissions. This makes it so the file will usually not be overwritten, or at least I'll see a warning message if I try to overwrite or delete the file. I've since added another step to my work flow, I run the following short bash/ksh script on all files in my 'working' directory now that I've passed the 10,000 mark:

for i in `ls -1 DSC0*.JPG`
do
   j=`echo $i | sed 's/SC0/SC1/g'`
   mv -i $i $j
done

It will rename any 'SC0' pattern to 'SC1', effectively renaming all my newer DSC0####.JPG files to DSC1####.JPG, as Sony should have done in the first place. I have listed the Unix 'mv' command with the '-i' interactive switch so you'll have to confirm each rename. Once you are comfortable with the script (try it on ONE file first) you can remove the '-i' script and it will rename all DSC0*.JPG files in the current directory. I'll still have to update my script once every 10,000 photos, but at four and a half years per update, I think I can manage.

Do all Sony Mavica cameras do this? Is this a function of the naming standard (DSC* seems to be vendor-independent)?

I'll post more on my workflow in a separate entry.

From the default configuration, I make the following changes to my squid.conf file

http_port 0.0.0.0:3128
cache_dir null /tmp
(for non-caching proxy)
or
cache_dir ufs /squiddir 5300 16 256
(for caching proxy, use a separate disk spindle where possible, separate controller channel if available.)
acl our_networks src (My class C)/24 127.0.0.1
http_access allow our_networks
cache_mgr root
cache_effective_user squid
cache_effective_group squid

Start the squid service (as root: 'service squid start' on Fedora) and you should have a functional proxy (caching or non-caching, as you chose in your configuration file.

Be sure your firewall is blocking port 3128 from the public Internet, or you will be in for a big surprise.

Point your favorite browser at (your proxy's IP address) port 3128 (unless you chose a different port in your configuration) and you'll be ready to browse through the proxy.

Rather than opening up port 3128 to the public Internet, I would strongly suggest configuring an ssh tunnel from wherever you are back to your proxy.

For SSH.com Secure Shell, the settings are:
Listen Port: 3128
Destination Host: 127.0.0.1
(assuming that the box you are ssh'ing to is your proxy, otherwise specify the IP address of the proxy in as seen from the box you are SSH'ing to (usually the private, local network IP address) here.)
Destination Port: 3128
Allow Local Connections Only: Yes
Type: tcp

For PuTTY the settings are:
Source Port: 3128
Destination: 127.0.0.1:3128
(Select "Local" radio button)

Again, 127.0.0.1 assumes the box you are creating an SSH session to is the same box running the proxy. If they are two separate boxes, specify the IP address you would use to connect between box1 and box2 (probably your NAT'd local, private IP address.)

You can choose to use a hostname here, but be sure your ssh target box knows your proxy box by this DNS name for it to work. The DNS reference is between you ssh target and proxy, not necessarily the same reference as returned by a public DNS query on the Internet.

Connect an SSH session with the correct tunneling settings, point your browser's proxy settings to 127.0.0.1:3128 and you should be browsing through your proxy. Opera will show you this on the status line "Sending request to www.slashdot.org (your.proxy.ip.here)". Firefox doesn't seem to show this by default.

All you browser developers, I'm still waiting for a nice standard "Proxy On/Off button" to be built into a browser, default view, default installation. Opera had a nice check box feature that can be added to a tool bar. Works great, but I lost the URL to where to get it.