←
→

bash: get http response codes for a list of URLs

Posted: August 24, 2011 | Author: | Filed under: blog | Tags: , , , | 14 Comments »

I had a file with a list of URLs, and I wanted to grab the HTTP response codes for each of them. I’m sure this quick bash script isn’t the best way to do it, but it works, and I’ll probably want to do this again someday, so here it is!

#!/bin/bash

while read line    
do    
    echo $(curl --write-out %{http_code} --silent --output /dev/null $line)
done <$1
  • http://xn.pinkhamster.net/ Christian G. Warden

    I used something similar to test whether redirects were set up properly for a bunch of URLs.
    curl’s –location and –write-out %{url_effective} were helpful:https://gist.github.com/44637c15808766872745/80def66b2cdd1fa829c66457ccb7f52cfc6347f4

  • http://calliopesounds.blogspot.com/ Andrew Gilmartin

    Would adding –head, which would, under normal circumstances, make the request faster, also give the same results you needed?

    • http://www.hilarymason.com Hilary Mason

      Yes, nice catch! Thanks!

  • http://twitter.com/brocknoland brocknoland

    Nice!! It solves the problem at hand, which is the important item.

    I typically use xargs when I want to do something on a list. It’s kind of like the functional programming map function.

    #!/bin/bash
    xargs -I {} curl -I –write-out %{http_code} –silent “{}” –output /dev/null < $1

  • http://twitter.com/_vkris Vivek Krishna

    If you want to run it in parallel, 
         cat urls.txt | xargs -n 1 -P NoOfThreads -I {} curl –write-out %{http_code} –silent –output /dev/null {}

    • http://kenegozi.com/blog Ken Egozi

      Liked, even though it would consume a thread per parallelism level, which is wasteful as most of what the thread does is wait for network IO.

      with large inputs it would make sense to use a scripting language that allow for easy async IO in non-ugly syntax. Too late to come up with one otoh :)

      • http://twitter.com/teepark teepark

        actually it’ll run “NoOfThreads” *processes*, not threads.

        but yeah, if you really need serious parallelism then you can probably be bothered to write more than one line.

      • http://mendicantbug.com Jason A

        *cough* nodejs *cough*

        • http://www.hilarymason.com Hilary Mason

          Jason, can you elaborate on that? What’s the advantage of doing this with node.js?

  • E A

    Thanks for this. I’ve been looking for how to do this for a long time.

    Some comments:

    1) #!/usr/bin/env bash is more portable than “#!/bin/bash”

    2) using $() is slower than just using && echo

    3) This isn’t bash specific – it will work on any shell

    4) Using –head would be useful, but may cause different results to GET results.

    • Guest

      or even using #!/bin/sh

  • http://www.facebook.com/profile.php?id=42410031 Giti Shorish

    This is cool!

  • Mcgowan

    anything that “< $1" could just as well be preceded with "cat ${*:–}", the canonical idiom for "all the named files OR the standard input".  by brian kernighan and rob pike's 1984 "the unix programming environment"

  • http://twitter.com/Mandragola69 Mandragola

    4 lines .. poetry ;)

←
→