I suspect most techies reading this will roll their eyes, but for me it was a chance to scratch the surface of a really powerful tool. Putting this here so I can find it again when I inevitably run into a similar issue.
I was asked to quickly set up a script that would hit some API or other and retrieve information using Python. It’s the most vanilla task you could imagine. And I miserably failed at it.
I just couldn’t figure it out. The API documentation was okay-ish, they provided examples, but only using
cURL. I figured that couldn’t be so problematic, surely ‘translating’ a bunch of
cURL commands to Python’s
requests couldn’t be so hard. Hah.
cURL commands in Python just wouldn’t give me the same responses. The
cURL commands would return the right information, while the Python requests inevitably ended up with HTTP
4XX requests. There had to be a difference, but how I could I see what?
After staring myself blind for a solid hour or more on the documentation, and running command after command with the same output, I stumbled across this StackOverflow answer. The answer proposes to set up a netcat listener and see what comes in using
nc -kl 8123. After pointing all my requests to
http://localhost:8123/, I could see what they looked like.
Turns out that the issue was related to the headers. The API documentation’s
cURL examples set the
Content-Type header to
application/x-www-form-urlencoded but promptly passed the content body as a JSON string (i.e. not URL encoded). Instead, I had (apparently like an idiot), set the correct header, but had
requests URL-encode my body, causing the (misconfigured?) API to reject the request. After adjusting the body-content, the whole thing went off without a hitch.
Seems like a fairly basic issue, how did I mess that up? Probably because I barely know what
cURL does, aside that it’s everyone go-to tool for anything. I’m chalking this one up to lack of experience. Thanks `netcat!