Recently in Free/Open Source Software (F/OSS) Category

I've been working with variations of Unix for a long time now and thought I'd jot down some of my favorite tips and tricks. They are mostly OS/distribution/shell/language independent (unless I indicate otherwise...)

1. Get rid of blank lines in a file
   grep . inputfile > outputfile
   This matches (and thus prints) only lines that contain some text, not blank (empty) lines.


2. comm
   Many people never cross paths with the comm command, but it is very useful. I works similarly to diff, but outputs the contents of two compared files into three columns. The first column is content only in the first file, second column is content only in the second file, and third column is content that is in both files (matches between the two files.) While this may not seem useful at first, you can select which columns to output, so if you only want to know what is in both file1 and file 2 (column 2) you'd suppress columns 1 and 2, by running:
   comm -12 file1 file2
   Don't forget that your input files must be sorted.


3. paste
   Systems administrators frequently use the cut> command to parse files, but many people I run into have never used the paste command. The paste command will concatenate two files line by line (as opposed to file by file, like cat.)


4. less instead of more
   This is not available on all Unix-based OSes, but the less command works very similar to more, but will let you move through a file forwards and backwards more easily. Want to jump to the end of the file, type Shift-G Depending on the version of less you are running, it will provide context highlighting when you search for a pattern.


5. Jump to vi from more
   While paging through a file in more, press "v" to jump to editting the file in vi at the current position in the file.


6. Jump to a line number when editing a file with vi
   vi +linenumber filename
   will open up the file with the cursor automatically moved down to the specified line. This is useful when you get an error that indicates "syntax error on line 2047." You can jump straight to the problem without fumbling around.


7. Invisible characters become visible
   Sometimes you'll end up with carriage returns on each line in a file originally created on a DOS/Windows system, or filenames with spaces, tab, or other control characters in them, but you can't see them typically.
   The cat command provides three useful options -v, -e, and -t that will let you understand these invisible characters
   -v (displays non-printing characters)
-e (prints a "$" at the end of each line to indicate a NL character)
-t (prints "^I" for each Tab in the file)
cat -vet filename |more


8. Remove DOS ^M from ends of lines
   The "^M" characters are visible when editing in vi but here are two approaches to remove the characters.
sed 's/<Ctrl-V><Ctrl-M>//g' -i filename
   or in vi: <Esc>:%s/<Ctrl-V><Ctrl-M>//g

If you ever do end up with me interviewing you, I'll likely work one or two of these into the discussion to explore your level of knowledge about Unix. If you need more information, remember: man pages are your friend.