convertation from utf-8 to utf-16

[Anonymous]

Dear colleagues!

    I am getting string data from ODBC in UTF-8 encoding. Can anybody explain how to convert this data to UTF-16 using Power BI functions?

Something like it's possible in .NET:

Private Function Utf8ToUtf16(utf8String As String) As String

    ' Get UTF-8 bytes by reading each byte with ANSI encoding
    Dim utf8Bytes As Byte() = Encoding.Default.GetBytes(utf8String)

    ' Convert UTF-8 bytes to UTF-16 bytes
    Dim utf16Bytes As Byte() = Encoding.Convert(Encoding.UTF8, Encoding.Unicode, utf8Bytes)

    ' Return UTF-16 bytes as UTF-16 string
    Utf8ToUtf16 = Encoding.Unicode.GetString(utf16Bytes)
End Function

[v-juanli-msft]

Hi @Anonymous 

You could use M function

TextEncoding.Utf8 or TextEncoding.Utf16

Best Regards
Maggie

Community Support Team _ Maggie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

Hi v-juanli-msft,

    Thank you for information. I have tried to use this functions like this:

#"Added Custom" = Table.AddColumn(Source, "CUSTCODEUTF16", each Text.FromBinary(Text.ToBinary([CUSTCODE], TextEncoding.Utf8), TextEncoding.Utf16))

similar I did it in .Net

But unfortunately get wrong result:

11\Р РћР—РќР˜Р¦Рђ

was converted to: ㄱ큜슠킠토킛钀ꃐ鳑ꃐ飂ꃐꛂꃐ鋑

It's look that I am  doing something wrong.

Can you send some example - how to use this functions for conversation?

[v-juanli-msft]

Hi @Anonymous 

As searched , the difference between utf-8 and utf-16 is how many bytes it require to represent a character in memory. UTF-8 uses minimum one byte, while UTF-16 uses minimum 2 bytes.

When you import data into Power BI, Power BI will cast the data to the corresponding data type.

Data type in Power Query are listed as below:

For example, Text type - A Unicode character data string. Can be strings, numbers, or dates represented in a text format. Maximum string length is 268,435,456 Unicode characters (256 mega characters) or 536,870,912 bytes.

So you don't worry about the filed memory.

Best Regards
Maggie

Community Support Team _ Maggie Li
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous · Accepted solution]

Dear colleagues, thank you for help!

    It was my missunderstunding with encoding number. When I has changed query to this:

 #"Added Custom" = Table.AddColumn(Source, "CUSTCODEUTF16", each Text.FromBinary(Text.ToBinary([CUSTCODE], 1251), TextEncoding.Utf8)

everything start working correctly.

[soldier_rong]

Hi @Anonymous ,

Hope you are well.

When I try to use below formula, then the result is "error"

Table.AddColumn("Query1", "CUSTCODEUTF16", each Text.FromBinary(Text.ToBinary([Company Name], 1251), TextEncoding.Utf8)