Unexpected token in JSON for special characters

Question

Why am I getting SyntaxError: Unexpected token < in JSON at position 0 when trying to send a post request that contains characters in body like – or ’ ?

This is how I'm sending the request (I've added the hard coded body just as an example):

import request from 'request';

export default {
  postScriptRequest(body) {
    return new Promise((resolve, reject) => {
      const options = {
        'method': 'post',
        'body': JSON.stringify({
          "text": "Sending – instead of - and ’ instead of ' returns Unexpected token in JSON"
        }),
        'headers': {
          'Content-Length': JSON.stringify(body).length
        },
        'url': 'my-url.com'
      }
      request(options, (error, response, body) => {
        if(response.statusCode == 200) {
          resolve(body);
        }
        else {
          reject(response.statusCode);
        }
      });
    });
  }
}

Something tells me your JSON.stringify(body).length is incorrect. The length of a string isn't the length of the content, right? Wouldn't a japanese character, say, こ be length "1" but take up more? `"こ".charCodeAt(0).toString("16").length/2` (2 bytes ) or `"–".charCodeAt(0).toString("16").length/2` (2 bytes ) — Cody Ｇ, Jan 09 '19 at 14:22
If that's the case, then how can I calculate the content length of a body that contains both ASCII and non-ASCII characters? Also, if the content length is incorrect then shouldn't I get `request size did not match content length` instead of the current error message? — Valip, Jan 09 '19 at 14:27
Maybe you need `"Content-type": "application/json; charset=UTF-8"` ? — Cody Ｇ, Jan 09 '19 at 14:31
So the error is happening on the server when it tries to do JSON.parse somewhere, right? Where is that code? (Have you checked what string the server is actually receiving / trying to parse? ) — Cody Ｇ, Jan 09 '19 at 14:36
It works calculating the content length using Elliot's solution, but the `Content-type` header needs to be also specified. You should post your 1st and 3rd comments as a solution. — Valip, Jan 09 '19 at 14:48

score 2 · Answer 1 · answered Jan 09 '19 at 14:34

One simple way to test if the issue is length is to use a buffer instead. That is:

let buffer = new Buffer(JSON.stringify(...));

// use buffer.length instead of string length
// pass buffer.toString() as the body

(The buffer length will correctly have a length of the total bytes of the message, as opposed to the number of characters.)

Cody Ｇ · Accepted Answer · 2019-01-09T15:14:10.623

You may need to specify (as a header)

"Content-type": "application/json; charset=UTF-8"

and calculate the correct content-length, because a javscript string length is not necessarily the byte-length required by content-length:

You could calculate the content-length based on something like:

"1こ3".split('').reduce((a,c,i)=>a+c.charCodeAt(0).toString("16").length/2,0)

But I'm not sure how this byte-length calculation compares to other solutions.

Unexpected token in JSON for special characters

2 Answers2