Whether you‘re a software developer, sysadmin, or tech enthusiast, chances are you‘ve come across cURL at some point. Curious what this ubiquitous tool is all about? You‘ve come to the right place!.
By the end of this article, you will have a strong foundation of this powerful command-line tool to start curling like pro !
Brief History Of cURL
cURL came into existenance into 1997, created by Daniel Stenberg to facilitate the transferring data over various network protocols. Back then it was named HTTP Get focused solely on HTTP request to fetch data.
Daniel was motivated by the need for command line that could makes downloading data without complex code. By 1998, the project was renamed to cURL to reflect its boarder protocol support.
Fub Fact : The name cURL is not an acronym. The c for client while URL indicated it does internet transfers.
What is cURL Used For
cURL is used for transferring data to or from server and some of most used cases are :
Transferring data with various protocols like HTTP, FTP, LDAP, MQTT, POP3, SMTP, Telnet etc.
Making API calls to web APIs — cURL lets you to easily test REST or SOAP APIs.
Automation : It can be used to script repitative tasks like data transfers.
web scraping : extract data from website through scripts using cURL.
Downloading Files on remote sever via SFTP, FTP etc.
uploading data and Files to web severs and APIs.
Installing cURL
to check cURL is install on your system or not . use curl —version
if not install from package manager based on your system:
Linux : install from : apt, yam , zyapper
Window : Download binaries from cURL website or use a package manager like Chocolatey
MacOS : brew install curl
Essential cURL Commands
cURL is very easy to use . its basic syntax is curl [options] [URL]
1 . Basic Webpage Retrieval | GET Request
curl https://example.com
This command fetch HTML source code and prints it to terminal.
2 . Redirects
website often redirect user to different URLS. To follow redirects use -L or —location option.
curl -L https://bit.ly/shortened-url
to limit the redirect hops to prevent infinite loops use :
curl -L —max-redirs 5 https://bit.ly/shortened-url
Use case : Testing shortened URLs, following API redirects, or checking redirect chains in your applications.
3 . Saving Output to a file
Instead of displaying output to a terminal , you can save into a file using -O or -o options.
# Save with original filename
curl -O https://example.com/data.json
# Save with custom filename
curl -o my-data.json https://api.example.com/export
# Save headers and body separately
curl -D headers.txt -o response.json https://api.example.com/data
Use case : Downloading Files, saving API responses for analysis, or archiving test results.
4 . Understanding HTTP Headers
HTTP headers provide valuable information about server’s response. you can view these headers using -i or -I option.
# Include headers with response
curl -i https://api.example.com/endpoint
# Headers only (useful for checking status codes) -I or --head
curl -I https://api.example.com/endpoint
Use case : Checking response codes, debugging caching issues, or examining API headers.
5 . Setting User-Agent
The User-Agent header identifies the client making the request . some website may server different content based on the User-Agent. you can customize the header using -A or —user-agent option:
6 . Working With Cookies
Cookies are the small piece of data that website store on the user’s computer to remember information about them.
# Save cookies to file
curl -c cookies.txt https://example.com/login
# Use saved cookies
curl -b cookies.txt https://example.com/dashboard
7 . Handling POST Request
POST request are commonly used to submit data to the server such as form data or JSON payloads. you can send POST request using -d or —data option
curl -d "name=John&age=30" https://example.com/submit
# send JSON Data
curl -X POST \
-H "Content-Type:application/json" \
-d
https://example.com/submit
# referencing JSON File
curl -X POST \
-H "Content-Type:application/json" \
-d @user-data.json \
https://example.com/submit
Use case :Creating resources, submitting forms, or testing POST endpoints with structured data.
8 . Authentication
Many API require authentication . cURL provides several options for handling authentication including basic authentication and Beare token
# Basic Authentication
curl -u username:password https://example.com/protected
#Bearer Token
curl -H "Authorization:Bearer TOKEN" \
https://example.com/api/protected
9 . Downloading Files
cURL is also useful for downloading files from command line. we have already seen -O option to save file with some name .
curl -O https://example.com/large_file.zip
# To resume Interupted downloads
curl -C - -O https://example.com/large_file.zip
10 . Uploading Files
cURL can also upload files to server.
# Standard form data
curl -X POST -d "username=john&password=secret" https://example.com/login
# File upload
curl -F "file=@document.pdf" -F "description=Important doc" https://example.com/upload
11 . Working With Proxy
Route requests through proxies for testing or security:
curl --proxy http://proxy.company.com:8080 https://api.example.com
12 . Timeouts
Network operation can sometime take long time or even hang indefinitely. To prevent this you can set timeout using —connect-timeout or —max-time
# 10-second connection timeout, 30-second total timeout
curl --connect-timeout 10 --max-time 30 https://slow-api.example.com
13 . Verbose mode
When requests fail or behave unexpectedly, verbose mode shows exactly what’s happening:
curl -v https://example.com
This displays:
Use cases : Debugging failed requests, understanding API behavior, or troubleshooting SSL issues.
Advanced cURL Technque
1 . Using jq for JSON parsing
first install jq
sudo apt update
sudo apt install jq
then use to display json data like:
curl https://api.github.com/users/dipeshxhy | jq
# to get specific fields
curl https://api.github.com/users/hiteshchoudhary | jq ".public_repos"
2 . SSL Certificate Handling
For development environments with self-signed certificates:
# Skip certificate verification (development only)
curl -k https://localhost:8443/api
# Use custom CA certificate
curl --cacert custom-ca.pem https://secure-api.example.com
Best Practice for Using cURL
To maximize the effectiveness of cURL and avoid common pitfalls, follow these best practices:
Error Handling: Always check the exit code of cURL to ensure that the command executed successfully. A non-zero exit code indicates an error.
Security: Be careful when using cURL with sensitive data, such as passwords or API keys. Avoid storing these credentials directly in scripts. Consider using environment variables or configuration files.
Rate Limiting: Respect API rate limits to avoid being blocked. Implement delays in your scripts to prevent excessive requests.
Documentation: Consult the cURL documentation for detailed information about options and usage. The man curl command provides access to the manual page on Linux systems.
Testing: Thoroughly test your cURL commands and scripts to ensure that they work as expected. Use a testing environment before deploying to production.
Summary
cURL is a powerful and versatile command-line tool for transferring data with URLs. It supports a wide range of protocols including HTTP, HTTPS, FTP, and more. cURL is widely used by developers and system administrators for testing APIs, automating file downloads and uploads, debugging network issues, and scripting complex data transfers. Its flexibility and extensive options make it an essential tool for web development, DevOps, and cloud workflows.
Mastering these commands will allow you to not only test and debug your own applications, but also to effectively integrate with external services and automate crucial tasks in your development workflow.
