vbscript ADO stream charset output

Question

My webhook is receiving POST request (aplication/JSON) from a 360Dialog (whatsapp) API with escaped Unicode characters like this: \u05db\u05e0\u05e1\u05d2\u05db\u05d9. It should be Hebrew letters.

I'm trying to decode that using JavaScript runat server but seems like it is not changing. I found a potential solution in this question's solution but it still saves the un-escaped Unicode into the database.

<script language="javascript" runat="server">
URL = {
    decode : function(s){return decodeURIComponent(s.replace(/\+/g,  " "))}
}
</script>

<%
  rs("smstext")=URL.decode(body2)
%>

The POST request is coming from 360dialog (a Whatsapp API) and hitting my webhook.

the request sends an application/json POST with information of incoming Whatsapp messages.

It seems the POST itself already has the Hebrew in it as \u05e0\u05e1\u05d9\u05d5\u05df i guess i need to figure out how to set the charset for that?

also, this unanswered question seems like my same problem.

I am trying to convert a request.BinaryRead into utf-8. the output in the database is this: \u05e0\u05e1\u05d9\u05d5\u05df instead of נסיון

I am probably misunderstanding something as the output is not what I expected.

my code is:

<%@LANGUAGE="VBSCRIPT" CODEPAGE="65001"%>
<!--#include virtual="/include/aspjson.asp" -->
<%
If Request.TotalBytes > 0 Then
    Dim lngBytesCount
        lngBytesCount = Request.TotalBytes
    body = BytesToStr(Request.BinaryRead(lngBytesCount))

Set db = CreateObject("ADODB.Connection")
    db.Open "DSN=xxx"
set rs = Server.CreateObject("ADODB.Recordset")
rs.open "SELECT * FROM log_sms", db, 3, 3
  rs.addnew
    rs("smstext")=body
  rs.update
rs.close
End if
Function BytesToStr(bytes)
    Dim Stream
    Set Stream = Server.CreateObject("Adodb.Stream")
        Stream.Type = 1 'adTypeBinary
        Stream.Open
        Stream.Write bytes
        Stream.Position = 0
        Stream.Type = 2 'adTypeText
        Stream.Charset = "utf-8"
        BytesToStr = Stream.ReadText
        Stream.Close
    Set Stream = Nothing
End Function
%>

If I replace rs("smstext")=body with rs("smstext")="נסיון", the value in the database is saved correctly.

Can you explain what it is you are trying to do? Having only part of the puzzle makes it hard to advise how best to approach the problem. Looks like you are uploading a json file? — user692942, Aug 31 '22 at 20:38
@user692942 the POST request is coming from 360dialog (a whatsapp api) and hitting my webhook. when i create a post request to that same webhook with https://reqbin.com/ the encoding is correct for the hebrew. i guess it has to do with the 360dialog encoding? have i got any way out of it? — Noam Smadja, Aug 31 '22 at 20:38
You’ll need to decode the escaped characters if that is what your webhook is receiving from 360dialog. You’ll need some kind of JSON decoder to decode the encoded JSON string. [This might help](https://stackoverflow.com/questions/46263831/classic-asp-convert-latin-characters-to-unicode-escape-strings#comment107500765_46269905). — user692942, Aug 31 '22 at 20:56
As VBScript doesn't have binary streams VBS can't do anything with the data. All you can do is save it to disk using ADODB. Then read from disk as a string. — Lundt, Sep 01 '22 at 03:02
*VBS can't do anything with the data* means that the data can't be assigned to a variable in VBScript because VBScript doesn't have byte arrays (unlike VBA that does have byte arrays). — Lundt, Sep 01 '22 at 05:54
After some more "homework" i learnt that the POST request itself is sending escaped unicode instead of utf-8 hebrew letters. when creating a request myself with hebrew charachters the decoding works correctly. so, i am now trying to figure out how should i proceed with converting said escaped characters into actual readable hebrew letters. i have edited my question and added the code i was trying to use (taken from a solution to a different question on SO) — Noam Smadja, Sep 01 '22 at 09:43
@Lundt we are talking about Classic ASP (with VBScript as the scripting language) you absolutely can access the raw binary using `Request.BinaryRead()`. — user692942, Sep 01 '22 at 09:44
@NoamSmadja that library that I mentioned in my other comment does the unescaping of the characters for you. Just to say your approach using `BytesToStr()` is sound in this instance, you just need to take that body and unescape the escape characters using a JSON parser like [JSON object class 3.8.1 By RCDMK](https://github.com/rcdmk/aspJSON). — user692942, Sep 01 '22 at 09:45
@NoamSmadja yes, my link was to the comment, not the answer. — user692942, Sep 01 '22 at 09:50
thank you! Looking into it. This whole encoding decoding is like chineese to me :D I will get it eventually — Noam Smadja, Sep 01 '22 at 09:51
Conclusion: i was using the library from [this other aspjson library](http://aspjson.com). I now replaced it with [rcdmk/aspJSON](https://github.com/rcdmk/aspJSON) and it decodes the escaped characters correctly :) thank you for your patiance and help! — Noam Smadja, Sep 01 '22 at 10:23

score 1 · Accepted Answer · answered Sep 01 '22 at 10:35

1

The approach is sound the problem is because the text is escaped in the JSON body you will need to unescape those characters before saving the content to a database.

Would recommend using this particular JSON Parser as it will automatically handle unescaping the characters for you.

Useful Links

Classic ASP Convert Latin Characters to Unicode Escape Strings *(Contains useful information about escaping, which can help with the unescaping aspect).

answered Sep 01 '22 at 10:35

user692942

16,398
7
76
175

1

Yes, this was the way to go! for reference i added my full code as an answer [here](https://stackoverflow.com/a/73567962/177146) – Noam Smadja Sep 01 '22 at 10:39
i am curious: could 360dialog change their encoding in a way that would not send escaped unicode for hebrew? I've used json in the past but never had such issue with escaped characters. or is it due to the conversion from binary? just trying to grasp the bigger picture – Noam Smadja Sep 01 '22 at 10:47
It's a conscientious decision they have made to escape the characters. Personally, if the JSON is UTF-8 why would you not just encode them as literal characters? Escaping is pointless in my opinion. The only reason I can think of for doing this is to guarantee the text is passed as ASCII, which if using UTF-8 is pointless anyway. – user692942 Sep 01 '22 at 11:10

score 1 · Answer 2 · answered Sep 01 '22 at 10:35

The solution (thanks to @user692942 comments):

I replaced the aspJSON library I was using with rcdmk/aspJSON. It takes care of decoding escaped characters already and simplified the process.

Since the POST request to my webhook page is in application/JSON a JSON library is required anyways.

to my little understanding, i must use BinaryRead to fetch the data from such a request. And thus have to convert from byte to str.

the working code is as follows:

<%@LANGUAGE="VBSCRIPT" CODEPAGE="65001" LCID="1037"%>
<!--#include virtual="/wa/aspjson.asp" -->
<%
If Request.TotalBytes > 0 Then
    Dim lngBytesCount
        lngBytesCount = Request.TotalBytes
    jsonbyte = Request.BinaryRead(lngBytesCount)
    jsonStr = BytesToStr(jsonbyte)

Set JSON = New JSONobject
Set jsn = JSON.Parse(jsonStr)

      Set contact = jsn.value("contacts")(0)
      set profile = contact.value("profile")
          profname = profile.value("name")
          wa_id = contact.value("wa_id")

      Set message = jsn.value("messages")(0)
          from = message.value("from")
          id = message.value("id")
          timestamp = message.value("timestamp")
          mtype = message.value("type")
      set text = message.value("text")
          body = text.value("body")
End If

Set db = CreateObject("ADODB.Connection")
    db.Open "DSN=xxx"
set rs = Server.CreateObject("ADODB.Recordset")
rs.open "SELECT * FROM log_sms", db, 3, 3
  rs.addnew
    rs("sent")=DateAdd("s", timestamp, DateSerial(1970,1,1))
    rs("from")=from
    rs("to")="whatsapp"
    rs("smstext")=body
    rs("result")="received"
    rs("msgid")=id
    rs("snr")="r"
    rs("type")=mtype
  rs.update
rs.close

Function BytesToStr(bytes)
    Dim Stream
    Set Stream = Server.CreateObject("Adodb.Stream")
        Stream.Type = 1 'adTypeBinary
        Stream.Open
        Stream.Write bytes
        Stream.Position = 0
        Stream.Type = 2 'adTypeText
        Stream.Charset = "utf-8"
        BytesToStr = Stream.ReadText
        Stream.Close
    Set Stream = Nothing
End Function

No problem, glad that approach worked for you. – user692942 Sep 02 '22 at 00:25 — user692942, Sep 02 '22 at 00:25

vbscript ADO stream charset output

2 Answers2

Useful Links