I suspect most techies reading this will roll their eyes, but for me it was a chance to scratch the surface of a really powerful tool. Putting this here so I can find it again when I inevitably run into a similar issue.
I was asked to quickly set up a script that would hit some API or other and retrieve information using Python. It’s the most vanilla task you could imagine. And I miserably failed at it.
I just couldn’t figure it out. The API documentation was okay-ish, they provided examples, but only using cURL. I figured that couldn’t be so problematic, surely ‘translating’ a bunch of cURL commands to Python’s requests couldn’t be so hard. Hah.
Reproducing the cURL commands in Python just wouldn’t give me the same responses. The cURL commands would return the right information, while the Python requests inevitably ended up with HTTP 4XX requests. There had to be a difference, but how I could I see what?
After staring myself blind for a solid hour or more on the documentation, and running command after command with the same output, I stumbled across this StackOverflow answer. The answer proposes to set up a netcat listener and see what comes in using nc -kl 8123. After pointing all my requests to http://localhost:8123/, I could see what they looked like.
Turns out that the issue was related to the headers. The API documentation’s cURL examples set the Content-Type header to application/x-www-form-urlencoded but promptly passed the content body as a JSON string (i.e. not URL encoded). Instead, I had (apparently like an idiot), set the correct header, but had requests URL-encode my body, causing the (misconfigured?) API to reject the request. After adjusting the body-content, the whole thing went off without a hitch.
Seems like a fairly basic issue, how did I mess that up? Probably because I barely know what cURL does, aside that it’s everyone go-to tool for anything. I’m chalking this one up to lack of experience. Thanks `netcat!