HTTP agents

November 29, 2023
December 11, 2014

Intercept, debug & mock HTTP with HTTP Toolkit

Mimicking Cross Origin Request

curl -H "Origin: http://example.com" --verbose \
  https://www.googleapis.com/discovery/v1/apis?fields=

# CORS preflight, should include "Access-Control-Allow-Methods"
curl -H "Origin: http://example.com" \
  -H "Access-Control-Request-Method: POST" \
  -H "Access-Control-Request-Headers: X-Requested-With" \
  -X OPTIONS --verbose \
  https://www.googleapis.com/discovery/v1/apis?fields=
# CORS preflight
http --print -Hh OPTIONS \
    https://www.googleapis.com/discovery/v1/apis\?fields\= \
    Origin:http://example.com \
    Access-Control-Request-Method:POST \
    Access-Control-Request-Headers:X-Requested-With

cURL

#153: 17 Years of curl With Daniel Stenberg - The Changelog
Everything curl - GitBook
Chapter 3: cURL | Conquering the Command Line | Softcover.io

curl
    -X '{HTTP_METHOD}'
    -H '${KEY}: ${VALUE}' # header
    -B 'NAME1=VALUE1; NAME2=VALUE2' # cookie or cookie file
    -d '${BODY}' # body text (or file with '@')
    -L # follow redirects
    ${URL}
curl --data "param1=value1&param2=value2" ${URL}
curl --form "fileupload=@my-file.txt" ${URL}
curl -F "NAME1=VALUE1" -F "NAME2=VALUE2" ${URL}
curl -X POST -H "Content-Type: application/json" -d '{"type": "json"}' ${URL}

curl - How To Use
curl - Manual
curl - Tutorial

15 Practical Linux cURL Command Examples (cURL Download Examples)
POST Form Data with cURL

Embed curl - Embeddable curl commands for your web site.
Parsing curl Commands with shlex - YouTube
Hurl.it - Make HTTP requests

curl vs libcurl
curl vs Wget

curlify

Converts browser/Node request to curl command line.

Node curlify
Chrome DevTools
Firefox DevTools

NickCarneiro/curlconverter
Convert curl command syntax to python requests code

wget

The Ultimate Wget Download Guide With 15 Awesome Examples
rockdaboot/wget2: The successor of GNU Wget

wget - man page - ManKier
Linux Commands – Parallel Downloading with wget | Baeldung on Linux

wget -p -k -nd -q -E -r -R js,txt,css -nc %pg%

# no upwards
get all images, etc. needed to display HTML page
recursively
make links in downloaded HTML point to local files
don't create directories
quiet
save HTML documents with `.html` extension
comma-separated list of rejected extensions: js,txt,css
skip downloads that would download to existing files

# clone website (form Just Enough Linux)
wget -e robots=off -r -np -nc -k -c <URL>
# -e robots=off: ignore we are a robot
# -r,--recursive: recursive retrieving
# -np,--no-parent: do not ascend to the parent directory when `-r`
# -nc,--no-clobber: don't redownload
# -k,--convert-links: convert links suitable for local viewing
# -c,--continue: continue getting a partially-downloaded file.

# my wget_all.sh
wget -r -l inf -nH -np -k -c -U "${SCRIPT_NAME}" ${URL}
# -m,--mirror: mirroring, equivalent to `-r -N -l inf --no-remove-listing`
# -N: ignore timestamp, conflicts with -nc
# -l: recursion depth
# -nH,--no-host-directories: do not generate directory with host name
# -U,--user-agent: set user agent
# download all links in file, to specific folder
wget -P./download -i list.txt

xh

ducaale/xh: Friendly and fast tool for sending HTTP requests
HTTPie feature parity checklist · Issue #4 · ducaale/xh · GitHub

It's a re-implementation of httpie:

# wget replacement
xh --download URL

# automatically uses POST
xh URL foo=bar number:=7 query==value
# headers
xh URL header:value remove-header: header-without-value;

xh --session session.json URL

httpie

HTTP client for CLI, more sensible than cURL
HTTPie – command-line HTTP client for the API era
HTTPie demo
jkbrzt/httpie

httpie cheatsheet
httpie: A CLI http client that will make you smile (by @radekpazdera)
HTTPie - HTTP for Humans – Mitesh Shah
httpie: A CLI http client that will make you smile (by @radekpazdera)

eliangcs/http-prompt: HTTPie + prompt_toolkit = an interactive command-line HTTP client featuring autocomplete and syntax highlighting

# create session to store cookies
http --session dpms85 POST http://10.6.64.85/sso token=${DPMS_TOKEN}
# test CORS
http POST http://api.server/login \
  Origin:http://example.com \
  Access-Control-Request-Method:POST \
  Access-Control-Request-Headers:X-Requested-With <<EOF
{
  "user": "user",
  "password": "password"
}
EOF
# use with Here Doc
TOKEN=$(http -b POST https://api.server/auth username=admin password=password)
http -b POST https://api.server/tokens "Authorization: Bearer ${TOKEN}" <<EOF
{
    "foo": "bar"
    "tokens": [
        "82b8d60bb4434b0083a9",
        "20458cb886154d618095",
        "c0b8191918a64e60967a",
        "3ce30cd4f5e340d09585",
        "871142606b564ce2b1de"
    ]
}
EOF

aria2

aria2
aria2/aria2: aria2 is a lightweight multi-protocol & multi-source, cross platform download utility operated in command-line. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink.

aria2c -x 4 -k 1M [url]  # 4 connections

HTTrack

HTTrack Website Copier - Free Software Offline Browser (GNU GPL)

httrack -c8 [url]

GUI App

I recommend REST Client on VSCode or scripts on HTTPie/xh/curl
that way you own your collections

Hurl - Run and Test HTTP Requests similar to REST Client, but more powerful

Download HTTPie for Desktop – HTTPie

Graphing COVID time series data using Chart.js and Postman | by Joyce Lin | Better Practices | Medium
How to visualize data in Postman - YouTube

Inspector web GUI from Swagger

REST Client

REST Client - Visual Studio Marketplace

Show request headers as well as response · Issue #99 · Huachao/vscode-restclient
Set previewOption to exchange (default full) to view requests

I Abandoned Postman for This NEW VS Code Extension | Build, Test & Sell APIs 🤑 - YouTube
Bye bye Postman ! Let's share your REST API calls in team, easily ! - DEV Community 👩‍💻👨‍💻
REST Client for your early Rest API-based project using Visual Studio Code - DEV Community
VS Code’s REST Client Plugin is All You Need to Make API Calls | by Paige Niedringhaus | Bits and Pieces
[VSCode 插件推荐] REST Client: 也许是比 Postman 更好的选择 - 知乎

In VS Code REST CLIENT plugin - how do you store variables from data retrieved from a request? - Stack Overflow

Thunder Client

POSTMan in VS Code

Thunder Client - Lightweight Rest API Client Extension for VS Code import OpenAPI, OAuth2 client
Thunder Client - Visual Studio Marketplace

RapidAPI Client

RapidAPI Client - Visual Studio Marketplace HTTP requests in plain text, generate types

Desktop app available on MacOS

Hoppscotch

Hoppscotch - Open source API development ecosystem open source port of Postman, originally Postwoman
hoppscotch/hoppscotch: 👽 Open source API development ecosystem https://hoppscotch.io
I created Postwoman 👽 - An online, open source API request builder - DEV Community 👩‍💻👨‍💻

Postman - A powerful HTTP client to test web services
REST Client | Postman API Platform [Free Download] also supports GraphQL
Postman Docs
Postman NOT recommended as it is cloud-required (2023-10 with the deprecation of Scratchpad)

Newman v3 – Postman Blog
postmanlabs/newman: Newman is a command-line collection runner for Postman

Review: Postman Client Makes RESTful API Exploration a Breeze

Insomnia REST Client open source, also supports GraphQL, NOT recommended as it is cloud-required (2023-10)
Use Insomnia for API Requests

Node.js

zeke/npm-collection-http-clients: A review of HTTP clients for Node.js and browsers

fetch API WHATWG standard
Fetch - from simple to scalable implementation - DEV Community
How to monitor the progress of a Javascript fetch - request and cancel it on demand. - DEV Community
posva/mande: 600 bytes convenient and modern wrapper around fetch with smart extensible defaults

HTTP Requests Compared: Why Axios Is Better Than Node-Fetch (Automatic Transformations, More…
Axios vs Fetch — Which To Use in 2019 | by Malcolm Laing | Frontend Digest | Medium

Axios
axios/axios: Promise based HTTP client for the browser and node.js
How To Use Axios in an Optimized and Scalable Way With React - DEV Community
The Modern Way to Use Promise- Based HTTP Requests: axios-hooks
Handling Errors With Axios
Axios Tutorials - Mastering JS

Isomorphic (through bundler):
sindresorhus/ky: 🌳 Tiny & elegant JavaScript HTTP client based on the browser Fetch API reduce boilerplate
visionmedia/superagent
lukeed/httpie: A Node.js HTTP client as easy as pie! 🥧

Server:
bitinn/node-fetch
tomas/needle
hapijs/wreck previously nipple
sindresorhus/got
`

Go

API Clients for Humans | Gopher Academy Blog

Python

psf/requests: A simple, yet elegant HTTP library. no async support
Requests: HTTP for Humans™ — Requests documentation
Python’s Requests Library (Guide) – Real Python

Python’s Requests Library (Guide) – Real Python
Advanced usage of Python requests - timeouts, retries, hooks

HTTPX also httpx[cli]
asynchronous client library that supports HTTP/1.1 and HTTP/2
encode/httpx: A next generation HTTP client for Python. 🦋
HTTPX Tutorial - A next-generation HTTP client for Python - YouTube
Asynchronous HTTP Requests in Python with HTTPX and asyncio

kennethreitz/requests3: Requests 3.0, for Humans and Machines, alike. 🤖
encode/httpcore

Rust

hyper.rs | hyper
reqwest - Rust
hyperium/h2: HTTP 2.0 client & server implementation for Rust.

PHP

Guzzle, PHP HTTP client — Guzzle Documentation Caching and Batching

HAR

HAR 1.2 Spec | Software is hard
HTTP Archive Viewer
HAR Resources | A community curated list of resources, tools, projects and applications that support HTTP Archive (HAR)
HAR Adopters | Software is hard

Mashape/harplayer: Replay HAR logs
kkovacs/har-replay: A small, basic tool to replay requests from a HTTP Archive (HAR) file, for load testing
pilsna/replay-har: A command line tool for replaying HTTP archive files. Python
formalin14/WWW-HarWalk: Replay HTTP requests from HAR ( HTTP Archive ) file Perl

Stuk/server-replay: Replay server responses from a HAR file
Netflix/pollyjs: Record, Replay, and Stub HTTP Interactions.

YSlow - Official Open Source Project Website get YSlow score with HAR
pcapperf get PageSpeed score with PCAP/HAR, PCAP -> HAR
shaunakv1/node-chrome-har-replay: A node.js script that takes chorme HAR network log file, replays it and generates performance benchmark

Others

fstab/h2c: http2client
hazbo/httpu: The terminal-first http client

antoinechalifour/memento: Memento is a development-only tool that caches HTTP calls once they have been executed.

Load Testing

Load Testing: An Unorthodox Guide

Locust - A modern load testing framework Python
alteryx/locust-grasshopper: a load testing tool extended from locust
Introducing Grasshopper - An Open Source Python Library for Load Testing

fcsonline/drill: Drill is an HTTP load testing application written in Rust
codesenberg/bombardier: Fast cross-platform HTTP benchmarking tool written in Go
h2load - HTTP/2 benchmarking tool - HOW-TO — nghttp2 documentation
mcollina/autocannon: fast HTTP/1.1 benchmarking tool written in Node.js
hatoo/oha: Ohayou(おはよう), HTTP load generator, inspired by rakyll/hey with tui animation.

k6.io - Performance testing for developers, like unit-testing, for performance
grafana/k6: A modern load testing tool, using Go and JavaScript - https://k6.io

SmokePing - About SmokePing deluxe latency measurement tool
Smokeping on Nginx

ab - Apache HTTP server benchmarking tool - Apache HTTP Server
Use ab -k (keepalive) to avoid testing connections (kernel responsibility)
Simultaneously benchmark many URLs with ApacheBench and GNU parallel · Simon Holywell

alexfernandez/loadtest: Runs a load test on the selected URL. Easy to extend minimally for your own ends. Node.js
wg/wrk: Modern HTTP benchmarking tool 😴inactive
giltene/wrk2: A constant throughput, correct latency recording variant of wrk 😴inactive
rakyll/hey: HTTP load generator, ApacheBench (ab) replacement 😴inactive

Gatling

Gatling - Professional Load Testing Tool
gatling/gatling: Modern Load Testing as Code
Gatling - Reference
52-technologies-in-2016/10-gatling
Load Testing APIs and Websites with Gatling: It’s Never Too Late to Get Started
Gatling vs JMeter - What to Use for Performance Testing

Stresstests with Gatling by Niko Köbler - YouTube
Stéphane Landelle - Load Testing Done Right with Gatling - YouTube

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class ParameterizedSimulation extends Simulation {
  val url = System.getenv("GATLING_URL")
  val requests = Integer.parseInt(System.getenv("GATLING_REQUESTS"))
  val users = Integer.parseInt(System.getenv("GATLING_USERS"))
  val reqs_per_user = requests / users
  val rampTime = Integer.parseInt(System.getenv("GATLING_RAMP_TIME"))
  val scn = scenario("My scenario").repeat(reqs_per_user) {
    exec(
      http("Dinosaur")
        .get(url)
        .check(status.in(Seq(200,304)))
    )
  }

  setUp(scn.inject(rampUsers(users) over (rampTime seconds)))
}
./gatling.sh -s <class> -on <result-basename> -rd <description>

OpenReplay

OpenReplay | Open-Source Session Replay
openreplay/openreplay: OpenReplay is developer-friendly, open-source session replay.

Dumping HTTP request/response

httpbin(1): HTTP Client Testing Service

Hookbin - Capture and Inspect HTTP Requests
Hookbin - Capture and Inspect HTTP Requests | CSS-Tricks

Mockbin by Mashape
Mockable: Quickly create REST and SOAP mocks

RequestBin — A modern request bin to collect, inspect and debug HTTP requests and webhooks - Pipedream
Webhook.site - Test, process and transform emails and HTTP requests

httpbin.org
mccutchen/go-httpbin: A reasonably complete and well-tested golang port of httpbin, with zero dependencies outside the go stdlib.

httpbin.js

my code 😃

leesei/httpbin.js: HTTPbin-like server implemented in Node.js

# https://github.com/jakubroztocil/httpie
brew install httpie

# https://www.npmjs.com/package/httpbin.js
npm install -g httpbin.js
npm install -g bunyan

Term1 (server)

httpbin.js | bunyan

Term2 (http agent)

# check request header and body
http -v --json POST http://localhost:35000/prq/xyz?a=b\&c=d key=value foo=bar
http -v --form POST http://localhost:35000/prq/xyz?a=b\&c=d key=value foo=bar