bash: get http response codes for a list of URLs

I had a file with a list of URLs, and I wanted to grab the HTTP response codes for each of them. I’m sure this quick bash script isn’t the best way to do it, but it works, and I’ll probably want to do this again someday, so here it is!


while read line    
    echo $(curl --write-out %{http_code} --silent --output /dev/null $line)
done <$1 

14 Comments on “bash: get http response codes for a list of URLs”

  1. I used something similar to test whether redirects were set up properly for a bunch of URLs.
    curl’s –location and –write-out %{url_effective} were helpful:

  2. Would adding –head, which would, under normal circumstances, make the request faster, also give the same results you needed?

  3. brocknoland says:

    Nice!! It solves the problem at hand, which is the important item.

    I typically use xargs when I want to do something on a list. It’s kind of like the functional programming map function.

    xargs -I {} curl -I –write-out %{http_code} –silent “{}” –output /dev/null < $1

  4. If you want to run it in parallel, 
         cat urls.txt | xargs -n 1 -P NoOfThreads -I {} curl –write-out %{http_code} –silent –output /dev/null {}

    • Ken Egozi says:

      Liked, even though it would consume a thread per parallelism level, which is wasteful as most of what the thread does is wait for network IO.

      with large inputs it would make sense to use a scripting language that allow for easy async IO in non-ugly syntax. Too late to come up with one otoh :)

  5. E A says:

    Thanks for this. I’ve been looking for how to do this for a long time.

    Some comments:

    1) #!/usr/bin/env bash is more portable than “#!/bin/bash”

    2) using $() is slower than just using && echo

    3) This isn’t bash specific – it will work on any shell

    4) Using –head would be useful, but may cause different results to GET results.

  6. Mcgowan says:

    anything that “< $1" could just as well be preceded with "cat ${*:–}", the canonical idiom for "all the named files OR the standard input".  by brian kernighan and rob pike's 1984 "the unix programming environment"