How to get a random line from a file in bash.

Posted: May 2, 2011 | Author: Hilary Mason | Filed under: blog | Tags: bash, code, one-liner, remember, tips | 24 Comments »

I work with a lot of data, and while I’d like to pretend it’s all in upside-down quasi-indexed b-tree rocket ships or some other advanced database, the truth is that much of it is in text files. I often find myself wanting to see a random line from one of these files, just to get a sense of what the data looks like.

I thought there must be an easy bash way to do this, but I couldn’t find it (‘shuf’ isn’t installed on my server), so I turned to twitter, and now I’m pleased to present more methods for finding a random line than you ever expected!

sort -R | head -n 1

If you can use this, do so! If it isn’t available, consider one of the following commands:

@andrewgilmartin suggests using awk:

awk 'BEGIN { srand() } rand() >= 0.5 { print; exit }'

@devinteske offered one of the easiest to solutions to read:

tail -$((RANDOM/(32767/`wc -l</etc/group|tr -d ' '`))) /etc/group|head -1

@terrycojones piped up with this gem:

split -l 1 < file; cat `for i in x*; do echo $RANDOM $i; done | sort -n | cut -f2 -d' ' | head -n 1`; rm x*

@FirefighterBlu3 does sed++:

file=/etc/passwd; lc="$(($RANDOM % $(wc -l $file|awk '{print $1}')))"; sed -n "${lc}p" $file

@burleyarch collects the whole set:

f=YOUR_FILE; n=$(expr $RANDOM \* `cat $f | wc -l` \/ 32768 + 1); head -n $n $f | tail -1

All of the options using $RANDOM should be used with the understanding that the max possible value is 32767, so it will only be random on files that have fewer than 32,767 lines.

@xn with an excellent use of cut:

awk 'BEGIN { OFS="\t"; srand() } { print rand(), $0 }' | sort -n | cut -f2- | head -1

@paulrbrown with a badass example of od:

echo `cat /dev/urandom | od -N4 -An -i `' % '`wc -l < file` | bc | sed 's/-//g' | xargs -I % head -n % file | tail -n 1

And finally, from @alexlines, who actually developed his solution into a blog post:

dd if=file skip=$(expr $(date +%N) \% $(stat -c "%s" file)) ibs=1 count=200 2>/dev/null|sed -n '2{p;q;}'

And, of course, @ceonyc brought some comic relief:

@hmason Good bash one-liner? Take my code, please.

---

SELECT * FROM wp_posts WHERE post_title LIKE '%posttitlekeyword%';

UPDATE wp_comments SET comment_post_id=2 WHERE comment_post_id=1;