Spark+Hdinsight+1 mill rows=ODBC error

[afuyo]

Hi,

I am trying to load data from Spark. The csv data file is about 50Mb, and results in a table of one million rows. 

Not a very large data set and I have no problem to load it directly as csv. However I wanted to create PoC on how Power BI works together with Spark on Azure platform.

It won't work and the application generates some   ODBC errors(see below). Then I fire up Tableau and it works like a charm. I even test Tableau with much larger data sets and it still works fine. 

I've decreased the number of rows in the table to 250K and then Power BI works again and the data gets loaded as expected.

Is there any limit on how much data can be loaded into Power BI desktop from Spark? Am I missing any configuration? Any help would be greatly appreciated. 

Best Regards

Artur 

Sep 05 14:46:11.594 ERROR 21192 Connection::SQLGetInfoW: [Microsoft][ODBC] (11180) SQLGetInfo property not found: 1750
Sep 05 14:46:40.318 ERROR 18192 Connection::SQLGetInfoW: [Microsoft][ODBC] (11180) SQLGetInfo property not found: 1750
Sep 05 14:47:01.520 ERROR 18192 QueryExecutor::Execute: [Microsoft][Hardy] (35) Error from server: error code: '0' error message: 'ExecuteStatement finished with operation state: ERROR_STATE'.
Sep 05 14:47:01.520 ERROR 18192 StatementState::DoExecDirect: [Microsoft][Hardy] (35) Error from server: error code: '0' error message: 'ExecuteStatement finished with operation state: ERROR_STATE'.
Sep 05 14:47:01.520 ERROR 18192 StatementStateAllocated::SQLExecDirectW: [Microsoft][Hardy] (35) Error from server: error code: '0' error message: 'ExecuteStatement finished with operation state: ERROR_STATE'.
Sep 05 14:47:01.520 ERROR 18192 Statement::SQLExecDirectW: [Microsoft][Hardy] (35) Error from server: error code: '0' error message: 'ExecuteStatement finished with operation state: ERROR_STATE'.

[ankitpatira]

@afuyo File size limit for power bi desktop file is 1GB. I assume you're using Azure HDInsight Spark connector from power bi desktop which is still in beta and that is why i guess you're experiencing data size limit issue. You can get data from Spark on HDInsight straight into powerbi via powerbi service. Check out this post here. Maybe there is no limit via power bi service.

[arify]

Hi @afuyo, what version of Power BI are you using? We've fixed a similar issue in the latest versions.

[afuyo]

@ankitpatira   Thank you for your input. Yes, indeed powerbi service seems to handle much larger data sets. I've tested with 12 GB and it worked fine with decent response times. However the limitations of power bi service, like not beeing able to create data model makes it too simplistic for our use. Thanks again for you answer and the link you've provided. 

@arify  Version: 2.38.4491.282 64-bit (August, 2016). I think it's thelatest version.  I realize that Spark connector is still in beta. It will be exciting to see what path the development of the power bi program will take. 

[arify]

That's odd, can you please reproduce this error while Tracing is enabled (Options > Diagnostics > Enable Tracing), and send us a frown email with those trace files (Microsoft.Mashup***.log) attached?

(To send a frown, click the smiley at the top of the Power BI Desktop window)

Thanks

[afuyo]

@arify I've sent the frown now.