bash: get http response codes for a list of URLs

I had a file with a list of URLs, and I wanted to grab the HTTP response codes for each of them. I’m sure this quick bash script isn’t the best way to do it, but it works, and I’ll probably want to do this again someday, so here it is!


while read line    
    echo $(curl --write-out %{http_code} --silent --output /dev/null $line)
done <$1 

How to get a random line from a file in bash.

I work with a lot of data, and while I’d like to pretend it’s all in upside-down quasi-indexed b-tree rocket ships or some other advanced database, the truth is that much of it is in text files. I often find myself wanting to see a random line from one of these files, just to get a sense of what the data looks like.

I thought there must be an easy bash way to do this, but I couldn’t find it (‘shuf’ isn’t installed on my server), so I turned to twitter, and now I’m pleased to present more methods for finding a random line than you ever expected!

sort -R | head -n 1

If you can use this, do so! If it isn’t available, consider one of the following commands:

@andrewgilmartin suggests using awk:

awk 'BEGIN { srand() } rand() >= 0.5 { print; exit }'

@devinteske offered one of the easiest to solutions to read:

tail -$((RANDOM/(32767/`wc -l</etc/group|tr -d ' '`))) /etc/group|head -1

@terrycojones piped up with this gem:

split -l 1 < file; cat `for i in x*; do echo $RANDOM $i; done | sort -n | cut -f2 -d' ' | head -n 1`; rm x*

@FirefighterBlu3 does sed++:

file=/etc/passwd; lc="$(($RANDOM % $(wc -l $file|awk '{print $1}')))"; sed -n "${lc}p" $file

@burleyarch collects the whole set:

f=YOUR_FILE; n=$(expr $RANDOM \* `cat $f | wc -l` \/ 32768 + 1); head -n $n $f | tail -1

All of the options using $RANDOM should be used with the understanding that the max possible value is 32767, so it will only be random on files that have fewer than 32,767 lines.

@xn with an excellent use of cut:

awk 'BEGIN { OFS="\t"; srand() } { print rand(), $0 }' | sort -n | cut -f2- | head -1

@paulrbrown with a badass example of od:

echo `cat /dev/urandom | od -N4 -An -i `' % '`wc -l < file` | bc | sed 's/-//g' | xargs -I % head -n % file | tail -n 1

And finally, from @alexlines, who actually developed his solution into a blog post:

dd if=file skip=$(expr $(date +%N) \% $(stat -c "%s" file)) ibs=1 count=200 2>/dev/null|sed -n '2{p;q;}'

And, of course, @ceonyc brought some comic relief:

@hmason Good bash one-liner? Take my code, please.