Web Scraping

References

Formdata & payload

POST /some-path HTTP/1.1
Content-Type: application/x-www-form-urlencoded

# output
foo=bar&name=John

POST /some-path HTTP/1.1
Content-Type: application/json

# output
{ "foo" : "bar", "name" : "John" }

Tools

• curlconverter - convert curl commands to Python, JavaScript, PHP, R, Go, Rust, Dart, JSON, Ansible, Elixir.
• ratelimiter - Simple Python module providing rate limiting.

Resources

• Avoiding bot detection: How to scrape the web without getting blocked?
• HTTP Status Codes
• HTTP Status Dogs - Hypertext Transfer Protocol Response status codes. And dogs.
• HTTP Cats - Cats for every HyperText Transfer Protocol response status code.