How to handle weirdly combined websocket messages?

  • A+

I'm connecting to an external websocket api using the node ws library (node 10.8.0 on Ubuntu 16.04). I've got a listener which simply parses the json and passes it to the callback:'message', (rawdata) => {     let data = null;     try {         data = JSON.parse(rawdata);     } catch (e) {         console.log('Failed parsing the following string as json: ' + rawdata);         return;     }     mycallback(data); }); 

I now receive errors in which the rawData looks as follows (I formatted and removed irrelevant contents):

�~A {     "id": 1,     etc.. }�~� {     "id": 2,     etc.. 

I then wondered; what are these characters? Seeing the structure I initially thought that the first weird sign must be an opening bracket of an array ([) and the second one a comma (,) so that it creates an array of objects.

I then investigated the problem further by writing the rawdata to a file whenever it encounters a JSON parsing error. In an hour or so it has saved about 1500 of these error files, meaning this happens a lot. I cated a couple of these files in the terminal, of which I uploaded an example below:

How to handle weirdly combined websocket messages?

A few things are interesting here:

  1. The files always start with one of these weird signs.
  2. The files appear to exist out of multiple messages which should have been received separately. The weird signs separate those individual messages.
  3. The files always end with an unfinished json object.
  4. The files are of varying lengths. They are not always the same size and are thus not cut off on a specific length.

I'm not very experience with websockets, but could it be that my websocket somehow receives a stream of messages that it concatenates together, with these weird signs as separators, and then randomly cuts off the last message? Maybe because I'm getting a constant very fast stream of messages?

Or could it be because of an error (or functionality) server side in that it combines those individual messages?

Does anybody know what's going on here? All tips are welcome!


@bendataclear suggested to interpret it as utf8. So I did, and I pasted a screenshot of the results below. The first print is as it is, and the second one interpreted as utf8. To me this doesn't look like anything. I could of course convert to utf8, and then split by those characters. Although the last message is always cut off, this would at least make some of the messages readble. Other ideas still welcome though.

How to handle weirdly combined websocket messages?


My assumption is that you're working only with English/ASCII characters and something probably messed the stream. (NOTE:I am assuming), there are no special characters, if it's so, then I will suggest you pass the entire json string into this function:

function cleanString(input) {     var output = "";     for (var i=0; i<input.length; i++) {         if (input.charCodeAt(i) <= 127) {             output += input.charAt(i);         }     }     console.log(output); }  //example cleanString("�~�")

You can make reference to How to remove invalid UTF-8 characters from a JavaScript string?


:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: