twenty-five sed tricks

terminal_iconSed is a stream editor. A stream editor is used to perform basic text transformations on an input stream While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the input(s), and is consequently more efficient. But it is sed’s ability to filter text in a pipeline which particularly distinguishes it from other types of editors.


1) Stream YouTube URL directly to mplayer.

mplayer -fs $(echo “$(curl -s $youtube_url | sed -n “/watch_fullscreen/s;.*\(video_id.\+\)&title.*;\1;p”)”)

This is the result of a several week venture without X. I found myself totally happy without X (and by extension without flash) and was able to do just about anything but watch YouTube videos… so this a the solution I came up with for that. I am sure this can be done better but this does indeed work… and tends to work far better than YouTube’s ghetto proprietary flash player 😉

Replace $youtube_url with any youtube URL you want and this will scrape the site for the _real_ URL to the full quality .FLV file on Youtube’s server and will then will hand that over to mplayer (or vlc or whatever you want) to be streamed.

In some browsers you can replace $youtube_url with just a % or put this in a shell script so all YouTube URLs can be handed directly off to your media player of choice for true streaming without the need for Flash or a downloader like clive. (I do however fully recommend clive if you wish to archive videos instead of streaming them)

If any interest is shown I would be more than happy to provide similar commands for other sites. Most streaming flash players use similar logic to YouTube.

Also, if you want to download videos, just add the -dumpstream option to mplayer.

2) Google Translate

translate(){ wget -qO- “$1&langpair=$2|${3:-en}” | sed ‘s/.*”translatedText”:”\([^"]*\)”.*}/\1\n/’; }


translate <phrase> <source-language> <output-language>Example:

translate hello en esSee this for a list of language codes:

3) Print all the lines between 10 and 20 of a file

sed -n ’10,20p’ <filename>

Similarly, if you want to print from 10 to the end of line you can use: sed -n ’10,$p’ filename
This is especially useful if you are dealing with a large file. Sometimes you just want to extract a sample without opening the entire file.

4) Graphical tree of sub-directories

ls -R | grep “:$” | sed -e ‘s/:$//’ -e ‘s/[^-][^\/]*\//–/g’ -e ‘s/^/   /’ -e ‘s/-/|/’

Prints a graphical directory tree from your current directory

5) Check your unread Gmail from the command line

curl -u username –silent “” | perl -ne ‘print “\t” if /<name>/; print “$2\n” if /<(title|name)>(.*)<\/\1>/;’

Checks the Gmail ATOM feed for your account, parses it and outputs a list of unread messages.

6) To print a specific line from a file

sed -n 5p <file>

You can get one specific line during any procedure. Very interesting to be used when you know what line you want.

7) Find geographical location of an ip address

lynx -dump$1|grep address|egrep ‘city|state|country’|awk ‘{print $3,$4,$5,$6,$7,$8}’|sed ‘s\ip address flag \\’|sed ‘s\My\\’

I save this to bin/iptrace and run “iptrace ipaddress” to get the Country, City and State of an ip address using the service.

I add the following to my script to get a tinyurl of the map as well:

URL=`lynx -dump$1|grep details|awk ‘{print $2}’`

lynx -dump$URL|grep tinyurl|grep “19. http”|awk ‘{print $2}’

8) Convert PDF to JPG

for file in `ls *.pdf`; do convert -verbose -colorspace RGB -resize 800 -interlace none -density 300 -quality 80 $file `echo $file | sed ‘s/\.pdf$/\.jpg/’`; done

(relies on ‘imagemagick’)

This command will convert all .pdf files in a directory into a 800px (wide or height, whichever is smaller) image (with the aspect ratio kept) .jpg.

If the file is named ‘example1.pdf’ it will be named ‘example1.jpg’ when it is complete.

This is a VERY worthwhile command! People pay hundreds of dollars for this in the Windows world.

My .jpg files average between 150kB to 300kB, but your’s may differ.

9) Remove a line in a text file. Useful to fix “ssh host key change” warnings

sed -i 8d ~/.ssh/known_hosts

10) Recursive search and replace old with new string, inside files

grep -rl oldstring . |xargs sed -i -e ‘s/oldstring/newstring/’

recursively traverse the directory structure from . down, look for string “oldstring” in all files, and replace it with “newstring”, wherever found


grep -rl oldstring . |xargs perl -pi~ -e 's/oldstring/newstring'

11) Efficiently print a line deep in a huge log file

sed ’1000000!d;q’ < massive-log-file.log

Sed stops parsing at the match and so is much more effecient than piping head into tail or similar. Grab a line range using

sed '999995,1000005!d' < my_massive_file

12) View all date formats, Quick Reference Help Alias

alias dateh=’date –help|sed “/^ *%a/,/^ *%Z/!d;y/_/!/;s/^ *%\([:a-z]\+\) \+/\1_/gI;s/%/#/g;s/^\([a-y]\|[z:]\+\)_/%%\1_%\1_/I”|while read L;do date “+${L}”|sed y/!#/%%/;done|column -ts_’

If you have used bash for any scripting, you’ve used the date command alot. It’s perfect for using as a way to create filename’s dynamically within aliases,functions, and commands like below.. This is actually an update to my first alias, since a few commenters (below) had good observations on what was wrong with my first command.

# creating a date-based ssh-key for

ssh-keygen -f ~/.ssh/`date +git-$USER@$HOSTNAME-%m-%d-%g` -C '' # /home/gpl/.ssh/ create a tar+gzip backup of the current directory

tar -czf $(date +$HOME/.backups/%m-%d-%g-%R-`sed -u 's/\//#/g' <<< $PWD`.tgz) . # tar -czf /home/gpl/.backups/04-22-10-01:13-#home#gpl#.rr#src.tgz .I personally find myself having to reference

date --help quite a bit as a result. So this nice alias saves me a lot of time. This is one bdash mofo. Works in sh and bash (posix), but will likely need to be changed for other shells due to the parameter substitution going on.. Just extend the sed command, I prefer sed to pretty much everything anyways.. but it’s always preferable to put in the extra effort to go for as much builtin use as you can. Otherwise it’s not a top one-liner, it’s a lazyboy recliner.

Here’s the old version:

alias dateh='date --help|sed "/^ *%%/,/^ *%Z/!d;s/ \+/ /g"|while read l;do date "+ %${l/% */}_${l/% */}_${l#* }";done|column -s_ -t'This trick from  [ bash_profile ]

13) Generate a Random MAC address

MAC=`(date; cat /proc/interrupts) | md5sum | sed -r ‘s/^(.{10}).*$/\1/; s/([0-9a-f]{2})/\1:/g; s/:$//;’`

Original author unknown (I believe off of a wifi hacking forum).

Used in conjuction with ifconfig and cron.. can be handy (especially spoofing AP’s)

14) Change prompt to MS-DOS one (joke)

export PS1=”C:\$( pwd | sed ‘s:/:\\\\\\:g’ )\\> “

15) Delete the specified line

sed -i 8d ~/.ssh/known_hosts

16) geoip information

curl -s “$(curl -s” | sed ‘/^IP:/!d;s/<[^>][^>]*>//g’

This script gives a single line as shown in the sample output.

17) Stream YouTube URL directly to mplayer

id=”dMH0bHeiRNg”;mplayer -fs$id\&t=$(curl -s$id | sed -n ‘s/.*, “t”: “\([^"]*\)”, .*/\1/p’)

The original doesn’t work for me – but this does. I’m guessing that Youtube updated the video page so the original doesn’t work.

18) Another Curl your IP command

curl -s | sed ‘s/[a-zA-Z<>/ :]//g’

Just another curl command to get your public facing IP

19) Print just line 4 from a textfile

sed -n ’4{p;q}’

Prints the 4th line and then quits. (Credit goes to flatcap in comments:

20) Working random fact generator

wget -O – 2>/dev/null | grep \<strong\> | sed “s;^.*<i>\(.*\)</i>.*$;\1;”

Though without infinite time and knowledge of how the site will be designed in the future this may stop working, it still will serve as a simple straight forward starting point.

This uses the observation that the only item marked as strong on the page is the single logical line that includes the italicized fact.

If future revisions of the page show failure, or intermittent failure, one may simply alter the above to read.

wget -O - 2>/dev/null | tee lastfact | grep \<strong\> | sed "s;^.*<i>\(.*\)</i>.*$;\1;"

The file lastfact, can then be examined whenever the command fails.

21) Given process ID print its environment variables

sed ‘s/\o0/\n/g’ /proc/INSERT_PID_HERE/environ

22) Pronounce an English word using

pronounce(){ wget -qO- $(wget -qO- “$@” | grep ‘soundUrl’ | head -n 1 | sed ‘s|.*soundUrl=\([^&]*\)&.*|\1|’ | sed ‘s/%3A/:/g;s/%2F/\//g’) | mpg123 -; }

23) find and replace tabs for spaces within files recursively

find ./ -type f -exec sed -i ‘s/\t/  /g’ {} \;

24) count IPv4 connections per IP

netstat -anp |grep ‘tcp\|udp’ | awk ‘{print $5}’ | sed s/::ffff:// | cut -d: -f1 | sort | uniq -c | sort -n

usefull in case of abuser/DoS attacks.

25) display an embeded help message from bash script header

[ "$1" == "–help" ] && { sed -n -e ‘/^# Usage:/,/^$/ s/^# \?//p’ < $0; exit; }

With this one liner you can easily output a standard help message using the following convention:

Usage: is the start marker

Stop at the last #

The unwritten rules of Tech Support

1. Have your tried rebooting the damn computer, yet? //Hint: do this before you waste my time

2. If I ask you a question, then yes, it is IMPORTANT!

3. I will tell you if your computer was SUPPOSED to do that.

4. When I ask you to do something, then that task, right then and there, is your purpose in life.

5. Don’t tell me you have a white computer, you racist bastard.

6. Pretending that you know more than I means you’re an idiot.

7. Lying is a sin, Lying to me is cyber-damnation.

8. If you did something “computery, techy, etc”, then this problem is probably your fault.

9. If I’m doing something that looks complicated, it probably is and a thanks from you is probably in order.

10. Of course members of that sex get faster, better tech support – I do have a libido.

11. By all means, act geeky, nerdy, ect in order to relate to me – I in turn will get to laugh about it to with tech friends later.

12. Don’t ask me to explain every little thing to you, but if you do then I get to be as sarcastic as I want.

13. This is my job, let me do it.

14. Any movies, music, or pictures on your hard drive belong to you – any copies on my hard drive belong to me.

15. ANY CP I find on your computer will “magically” flag the FBI (you sick fuck). If you don’t know what CP is, don’t worry about it.

16. Don’t, I REPEAT, DON’T CLICK ON THE POP UPS. If you do, I get to rip you a new one.

17. Non disclosure means I can’t discuss, distribute, or declassify – doesn’t mean I can’t have.

18. Details, details, details – always important.

19. Hacker = Good. Cracker, Script Kiddie, etc = BAD. Hollywood got it wrong, now shut up.

20. Microsoft products suck. Apple products are only slightly behind.

21. The “stuff” I install is useful and better. You should use it.

22. The computer jokes you make are usually lame. See,, or STFU.

23. The smiles are mandatory by the company, the thoughts of killing you in a horrible, horrible way are free from me.

24. Linux is not a secret code. Ubuntu is not a Pokemon.

25. Video games are not a waste of time – they keep me literally from dominating all of you. If you don’t understand this, then just let me play the damn game.