I've been working with variations of Unix for a long time now and thought I'd jot down some of my favorite tips and tricks. They are mostly OS/distribution/shell/language independent (unless I indicate otherwise...)
- Get rid of blank lines in a file
grep . inputfile > outputfile
This matches (and thus prints) only lines that contain some text, not blank (empty) lines. - comm
Many people never cross paths with the comm command, but it is very useful. I works similarly to diff, but outputs the contents of two compared files into three columns. The first column is content only in the first file, second column is content only in the second file, and third column is content that is in both files (matches between the two files.) While this may not seem useful at first, you can select which columns to output, so if you only want to know what is in both file1 and file 2 (column 2) you'd suppress columns 1 and 2, by running:
comm -12 file1 file2
Don't forget that your input files must be sorted. - paste
Systems administrators frequently use thecutcommand to parse files, but many people I run into have never used thepastecommand. Thepastecommand will concatenate two files line by line (as opposed to file by file, likecat.) - less instead of more
This is not available on all Unix-based OSes, but thelesscommand works very similar to more, but will let you move through a file forwards and backwards more easily. Want to jump to the end of the file, typeShift-GDepending on the version oflessyou are running, it will provide context highlighting when you search for a pattern. - Jump to vi from more
While paging through a file in more, press "v" to jump to editting the file in vi at the current position in the file. - Jump to a line number when editing a file with vi
vi +linenumber filename
will open up the file with the cursor automatically moved down to the specified line. This is useful when you get an error that indicates "syntax error on line 2047." You can jump straight to the problem without fumbling around. - Invisible characters become visible
Sometimes you'll end up with carriage returns on each line in a file originally created on a DOS/Windows system, or filenames with spaces, tab, or other control characters in them, but you can't see them typically.
Thecatcommand provides three useful options -v, -e, and -t that will let you understand these invisible characters
-v (displays non-printing characters)
-e (prints a "$" at the end of each line to indicate a NL character)
-t (prints "^I" for each Tab in the file)
cat -vet filename |more - Remove DOS ^M from ends of lines
The "^M" characters are visible when editing in vi but here are two approaches to remove the characters.
sed 's/<Ctrl-V><Ctrl-M>//g' -i filename
or in vi:<Esc>:%s/<Ctrl-V><Ctrl-M>//g
If you ever do end up with me interviewing you, I'll likely work one or two of these into the discussion to explore your level of knowledge about Unix. If you need more information, remember: man pages are your friend.
Small comments and additions, referenced by the number int he blog item above:
5. Pressing v to open vi also works in less.
6. If the emphasis is doing it all from the commandline, then as stated above is best, however, you can do this with one less keystroke (including the extra ENTER required). :-) 'vi filename' and then ":2047" sends you right to the line.
8. From within vi, you can also do ":set ff=unix" and then save it. (For completeness, to format _to_ dos, just do ":set ff=dos" and then save it.) I agree though that using the %s/// function is a good, useful thing to know about.
Just another FYI about #8. The program unix2dos and dos2unix will quickly add or subtract the ^M character from file.
You are right those programs (unix2dos,dos2unix) will clean up the ^M characters, but they aren't always available. vi, grep, sed, awk are universal across Unix varieties and Linux distributions. As with everything Unix there are always a variety of ways to get to the same results.
I use dtox to remove the M from the end of lines on the old SCO box I work on. You can put them back with dtod.
you can even leave the //g off the end of point 8, just
:s/
does the trick
another nice vi trick is to delete any lines with a particular pattern on, which you can also delete blank lines with:
:g//d
e.g. :g/^$/d deletes blank lines
:g/^#/d deletes any commented lines in a shell/perl script
Thanks--I didn't know some of your tips.
To remove ^M from files, this also works across all or most unix systems:
tr -d '\r'
There is no file name argument. Input comes from stdin and output goes to stdout, so you typically use redirection with it.
in this case less is more, literally <<nudge> <<nudge>> ha ha
btw, you can also use awk to remove the annoying part of windows line breaks or really old mac line breaks - "\r" -- it's obviously more complicated but it can be done.
awk -v RS="\r" -v ORS="" {print} output
@Phil Garcia
thanks for the tr -d '\r'. i have been in love with tr for a while but didn't know about -d. :)
Pretty cool post. I just stumbled upon your blog and wanted to say
that I have really liked reading your blog posts. Anyway
I’ll be subscribing to your blog and I hope you post again soon!