Cope with JSONDecodeError in requests.get().json() in Python 2 and 3

16 November 2016   6 comments   Python

Mind that age!

This blog post is 4 years old! Most likely, its content is outdated. Especially if it's technical.

UPDATE June 2020

Thanks to commentors, I've been made aware that requests will use simplejson if it's installed to handle the deserialization of the JSON. So if you have simplejson in your requirements.txt or pyproject.toml you have to change to this:

import requests
import simplejson

response = requests.get(url)
try:
    print(response.json())
except simplejson.errors.JSONDecodeError:
    print("N'est pas JSON")

Or, make your app more solid by doing the same trick that requests does:

import requests
try:
    from simplejson.errors import JSONDecodeError
except ImportError:
    from json.decoder import JSONDecodeError

response = requests.get(url)
try:
    print(response.json())
except JSONDecodeError:
    print("N'est pas JSON")

Now, back to the old blog post...


Suppose you don't know with a hundred percent certainty that an API will respond in with a JSON payload you need to protect yourself.

This is how you do it in Python 3:

import json
import requests

response = requests.get(url)
try:
    print(response.json())
except json.decoder.JSONDecodeError:
    print("N'est pas JSON")

This is how you do it in Python 2:

import requests

response = requests.get(url)
try:
    print response.json()
except ValueError:
    print "N'est pas JSON"

Here's how you make the code work across both:

import json
import requests

try:
    from json.decoder import JSONDecodeError
except ImportError:
    JSONDecodeError = ValueError

response = requests.get(url)
try:
    print(response.json())
except JSONDecodeError:
    print("N'est pas JSON")

Comments

Jack

My immediate thought was why not do something like this, but then I thought for the sake of repetition and clarity your solution makes more sense.

import json
import requests

response = requests.get(url)
try:
    print(response.json())
except (json.decoder.JSONDecodeError, ValueError):
    print("N'est pas JSON")

Florian

JSONDecodeError is a subclass of ValueError, so I'd expect a simple `except ValueError:` to work fine on both versions.

Peter Bengtsson

Cool! I didn't know that. But it makes perfect sense.
Granted there are benefits to using specific exceptions to avoid "over swallowing" errors if you cover multiple lines.

dmtucker

simplejson.scanner.JSONDecodeError in py2

Jack M

The exception is wrong in this guide. requests uses simplejson not json, so the exception will not be caught.

.local/lib/python3.6/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Peter Bengtsson

It depends on if you have `simplejson` installed or not. If you look at `site-packages/requests/compat.py` you'll see that it does this:

try:
    import simplejson as json
except ImportError:
    import json

Basically, if `simplejson` is installed in the environment, requests uses that instead.

You need to be aware of that when you are aware of `response.json()` potentially failing.

I think I need to write an update about that.

Your email will never ever be published

Related posts