Linux and Curl: How to use Bash to Read a File Line by Line and Execute Curl command to get HTTP Response Code

Linux Curl command is very amazing. It’s very simple command which is use to send or get data from and to any server. Server would be any server like end point URL, ftp endpoint, etc.

In this tutorial we will go over how to read file line by line and then perform curl operation to get HTTP response code for each HTTP URL.

Flow would look like this:

1. Create file crunchify.txt
2. Add 5 URLs including http:// as part of URL
3. Create curl command and read crunchify.txt file
4. Perform curl operation to get HTTP response code
5. Print HTTP Response code

Let’s get started:

Step-1

Go to Mac Terminal if you are running this script on Macbook or open bash shell on Linux/Unix terminal

Step-2

• Go to ~/<username>/Downloads/
• Create file crunchify.txt
• Put below URLs into file and save file using command vi crunchify.txt to create file and :wq to save file.

https://crunchify.com
http://google.com
https://www.facebook.com
http://x.com
https://crunchify.com:8080

Step-3

Execute below command to get result.

for URL in `cat crunchify.txt`; do echo $URL; curl -m 10 -s -I $1 "$URL" | grep HTTP/1.1 |  awk {'print $2'}; done

• -m: Maximum time in seconds that you allow the whole operation to take. This is useful for preventing your batch jobs from hanging for hours due to any network issue
• -s: show error message
• -I: Show document info only
• awk: matches the pattern and prints result

Result:

bash-3.2$ for URL in `cat crunchify.txt`; do echo $URL; curl -m 10 -s -I $1 "$URL" | grep HTTP/1.1 |  awk {'print $2'}; done
https://crunchify.com
200
http://google.com
301
https://www.facebook.com
200
http://x.com
405
https://crunchify.com:8080

If you want to see detailed result and response just try adding -v into curl and you will see detailed verbose result in command prompt. Here is an updated command.

for URL in `cat crunchify.txt`; do echo $URL; curl -v -m 10 -s -I $1 "$URL" | grep HTTP/1.1 |  awk {'print $2'}; done

And that’s it, above command will print HTTP response code for each URL.

Above command and tutorial works well if you have any of below questions too:

• Use bash to read a file and then execute a command
• Run a command on each line in a file
• bash – Execute a command once per line of piped input?
• HowTo : Read a file Line By Line
• Shell Script to execute a command on every line of a file