Basic cURL Commands Every Web Developer Should Know

Whether you‘re a software developer, sysadmin, or tech enthusiast, chances are you‘ve come across cURL at some point. Curious what this ubiquitous tool is all about? You‘ve come to the right place!.

By the end of this article, you will have a strong foundation of this powerful command-line tool to start curling like pro !

Brief History Of cURL

cURL came into existenance into 1997, created by Daniel Stenberg to facilitate the transferring data over various network protocols. Back then it was named HTTP Get focused solely on HTTP request to fetch data.

Daniel was motivated by the need for command line that could makes downloading data without complex code. By 1998, the project was renamed to cURL to reflect its boarder protocol support.

Fub Fact : The name cURL is not an acronym. The c for client while URL indicated it does internet transfers.

What is cURL Used For

cURL is used for transferring data to or from server and some of most used cases are :

1. Transferring data with various protocols like HTTP, FTP, LDAP, MQTT, POP3, SMTP, Telnet etc.

2. Making API calls to web APIs — cURL lets you to easily test REST or SOAP APIs.

3. Automation : It can be used to script repitative tasks like data transfers.

4. web scraping : extract data from website through scripts using cURL.

5. Downloading Files on remote sever via SFTP, FTP etc.

6. uploading data and Files to web severs and APIs.

Installing cURL

to check cURL is install on your system or not . use curl —version

if not install from package manager based on your system:

• Linux : install from : apt, yam , zyapper

• Window : Download binaries from cURL website or use a package manager like Chocolatey

• MacOS : brew install curl

Essential cURL Commands

cURL is very easy to use . its basic syntax is curl [options] [URL]

1 . Basic Webpage Retrieval | GET Request

curl https://example.com

This command fetch HTML source code and prints it to terminal.

2 . Redirects

website often redirect user to different URLS. To follow redirects use -L or —location option.

curl -L https://bit.ly/shortened-url

to limit the redirect hops to prevent infinite loops use :

curl -L —max-redirs 5 https://bit.ly/shortened-url

Use case : Testing shortened URLs, following API redirects, or checking redirect chains in your applications.

3 . Saving Output to a file

Instead of displaying output to a terminal , you can save into a file using -O or -o options.

# Save with original filename
curl -O https://example.com/data.json

# Save with custom filename
curl -o my-data.json https://api.example.com/export

# Save headers and body separately
curl -D headers.txt -o response.json https://api.example.com/data

Use case : Downloading Files, saving API responses for analysis, or archiving test results.

4 . Understanding HTTP Headers

HTTP headers provide valuable information about server’s response. you can view these headers using -i or -I option.

# Include headers with response
curl -i https://api.example.com/endpoint

# Headers only (useful for checking status codes) -I or --head
curl -I https://api.example.com/endpoint

Use case : Checking response codes, debugging caching issues, or examining API headers.

5 . Setting User-Agent

The User-Agent header identifies the client making the request . some website may server different content based on the User-Agent. you can customize the header using -A or —user-agent option:

6 . Working With Cookies

Cookies are the small piece of data that website store on the user’s computer to remember information about them.

# Save cookies to file
curl -c cookies.txt https://example.com/login

# Use saved cookies
curl -b cookies.txt https://example.com/dashboard

7 . Handling POST Request

POST request are commonly used to submit data to the server such as form data or JSON payloads. you can send POST request using -d or —data option

curl -d "name=John&age=30" https://example.com/submit

# send JSON Data
curl -X POST \
      -H "Content-Type:application/json" \
       -d '{"name":"John Doe", "email":"john@example.com"}'
        https://example.com/submit

# referencing JSON File
curl -X POST \
      -H "Content-Type:application/json" \
       -d @user-data.json \
        https://example.com/submit

Use case :Creating resources, submitting forms, or testing POST endpoints with structured data.

8 . Authentication

Many API require authentication . cURL provides several options for handling authentication including basic authentication and Beare token

# Basic Authentication
curl -u username:password https://example.com/protected
#Bearer Token
curl -H "Authorization:Bearer TOKEN" \
       https://example.com/api/protected

9 . Downloading Files

cURL is also useful for downloading files from command line. we have already seen -O option to save file with some name .

curl -O https://example.com/large_file.zip

# To resume Interupted downloads
curl -C - -O https://example.com/large_file.zip

10 . Uploading Files

cURL can also upload files to server.

# Standard form data
curl -X POST -d "username=john&password=secret" https://example.com/login

# File upload
curl -F "file=@document.pdf" -F "description=Important doc" https://example.com/upload

11 . Working With Proxy

Route requests through proxies for testing or security:

curl --proxy http://proxy.company.com:8080 https://api.example.com

12 . Timeouts

Network operation can sometime take long time or even hang indefinitely. To prevent this you can set timeout using —connect-timeout or —max-time

• —connect-timeout specifies the maximum time in seconds to wait for a connection to be established

• —max-time specifies the maximum time in seconds that the entire application is allowed to take.

# 10-second connection timeout, 30-second total timeout
curl --connect-timeout 10 --max-time 30 https://slow-api.example.com

13 . Verbose mode

When requests fail or behave unexpectedly, verbose mode shows exactly what’s happening:

curl -v https://example.com

This displays:

• Request headers sent

• Response headers received

• SSL/TLS handshake details

• Connection information

Use cases : Debugging failed requests, understanding API behavior, or troubleshooting SSL issues.

Advanced cURL Technque

1 . Using jq for JSON parsing

first install jq

sudo apt update
sudo apt install jq

then use to display json data like:

curl https://api.github.com/users/dipeshxhy | jq
# to get specific fields
curl https://api.github.com/users/hiteshchoudhary | jq ".public_repos"

2 . SSL Certificate Handling

For development environments with self-signed certificates:

# Skip certificate verification (development only)
curl -k https://localhost:8443/api

# Use custom CA certificate
curl --cacert custom-ca.pem https://secure-api.example.com

Best Practice for Using cURL

To maximize the effectiveness of cURL and avoid common pitfalls, follow these best practices:

• Error Handling: Always check the exit code of cURL to ensure that the command executed successfully. A non-zero exit code indicates an error.

• Security: Be careful when using cURL with sensitive data, such as passwords or API keys. Avoid storing these credentials directly in scripts. Consider using environment variables or configuration files.

• Rate Limiting: Respect API rate limits to avoid being blocked. Implement delays in your scripts to prevent excessive requests.

• Documentation: Consult the cURL documentation for detailed information about options and usage. The man curl command provides access to the manual page on Linux systems.

• Testing: Thoroughly test your cURL commands and scripts to ensure that they work as expected. Use a testing environment before deploying to production.

Summary

cURL is a powerful and versatile command-line tool for transferring data with URLs. It supports a wide range of protocols including HTTP, HTTPS, FTP, and more. cURL is widely used by developers and system administrators for testing APIs, automating file downloads and uploads, debugging network issues, and scripting complex data transfers. Its flexibility and extensive options make it an essential tool for web development, DevOps, and cloud workflows.

Mastering these commands will allow you to not only test and debug your own applications, but also to effectively integrate with external services and automate crucial tasks in your development workflow.