DataFormat Error: We reached the end of the buffer

[AnooshaKeen]

Hi,

I am using Hive as data source. I am connecting to it using Impala connector. In the data, I have a text column which has data in json format. When I parse that text data to json, it gives me the error "DataFormat Error: We reached the end of the buffer"".

This happens when the text length is greater than or equal to 32767. For texts <32767 it works fine. 

Note: I have validated the json formats. They are correct. Moreover, I have the latest version of PowerBI (September 2021) and I have tried it both on 42GB and 8GB machine. Memory doesn't make any difference in this case.

I have attached screenshot of error and the steps I did for transformation.

[AnooshaKeen]

Hi @v-yiruan-msft ! Thank you for your response.

1- I have installed the latest version of PowerBI.

2- I tried to get rid of error using Table.Buffer and the Unicode thing. Also, I reduced the columns. But nothing worked.

[v-yiruan-msft]

Hi @AnooshaKeen ,

I found these threads below that have similar problems to yours, please check the solutions inside. Hope they can help you solve the problem too.

1. DataFormat.Error: We reached the end of the buffer.

---

 It worked when i tried reinstalling the latest Power BI update.

---

It seems that you have already installed the latest version of Power BI Desktop... Is your Power BI Desktop 32-bit or 64-bit? If it is 32-bit, please try to update to 64-bit and check whether it can work.

2. Help with We Reach the end of the buffer message

---

a. use table.buffer

b. The issue went away when I changed the "file origin" property to "unicode" instead of "None". It seems Json connector picks "None" file origin by default which doesn't work for all Json documents.

---

3. We reached end of buffer Error

I figured this was probably a memory issue, so I removed columns I determined were unnecessary and was able to get the API URL to load. Loading so many custom fields through the API really seems to slow it down, but I got the ones I needed.

---

Best Regards