cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
josef78
Super User
Super User

Performance and memory leak issue - PBIRS Jan 2022

Hello,

 

After upgrading our production environment to the Power BI Report Server version of January 2022 (1.13.8054.40631), we have significant performance issues.

 

At real user load (including a slight evening load), we have a significant slowdown in report queries (approximately by 50%), and much higher CPU usage. This happens a few hours after the last restart of services, and worsens over time.

 

The problem is only on front-end servers (we have scale-out deployment). And the problem is probably somethere in the RSPowerBI.exe process, which higlly utilizes the CPU and also consumes a large amount of RAM (more than the main msmdsrv.exe process). There is nothing unusual in RSPowerBI * .log.

 

Does anyone else have these problems?

 

Is there a solution?

1 ACCEPTED SOLUTION
Petebro
Microsoft
Microsoft

I wanted to reply to let you know the May release is now live with a fix for the issue reported on this thread. 

 

https://community.powerbi.com/t5/Report-Server/Power-BI-Report-Server-May-2022-has-been-Released/m-p...

View solution in original post

51 REPLIES 51
Steffe
Helper I
Helper I

Any news on this issue? Seems like there are many organization facing the same issue.

Are Microsoft working on it?

 

Hello, I raised issue ticket too. I have the answer that MS has identified an bug. Now I'm waiting for a bugfix or a new build. I hope for a new build or that it will be resolved in a may release.

Thanks @josef78 

This sounds promesing 🙂

Andrino
Frequent Visitor

It looks like Microsoft doesn't do anything about this issue. Author of this topic posted ticket to Microsoft Support. I'll try to post ticket too if there will be no bug fixes at nearest time.

file
Frequent Visitor

To bad, hope they fix this in the May-release then.

file
Frequent Visitor

We are also experiencing high memory usage after upgrading to January 2022, even after the latest update as well. The memory increases during the day until the service crashes and then it repeats. We even changed the settings for ModelCleanupCycleMinutes and ModelExpirationMinutes but it does not solve the problem.

We have a mix of live/import reports and SSRS-reports.

Hello,

my current workarround is periodically restart RSPowerBI process (not whole PowerBIReportServer service, because it is time consuming and also need reload all models in to memory).

I'm killing RSPowerBI process every hour (with scheduled powershell script Stop-Process -Name RSPowerBI -Force), PowerBI service immediately start new clean process without any outage.

Andrino
Frequent Visitor

If there is an active Power BI report data refresh process then with killing RSPowerBI.exe this process will also be killed and stucked in SubscriptionHistory table of PBIRS database. To prevent this I've wrapped this command into the script with checking of existence of active PBI data refresh processes:

$dataSource = "YourSqlServerDBEngineInstanceWherePBIRSDatabaseIsLocated";
$database = "master";
$sql = "
    SELECT [SH].[SubscriptionHistoryID]
    FROM [PBIRS].[dbo].[SubscriptionHistory] [SH] WITH(NOLOCK)
    WHERE [SH].[Status] = 1
";
$auth = "Integrated Security=SSPI;"
$connectionString = "Provider=sqloledb; " + "Data Source=$dataSource; " + "Initial Catalog=$database; " + "$auth; ";
$connection = New-Object System.Data.OleDb.OleDbConnection $connectionString;
$command = New-Object System.Data.OleDb.OleDbCommand $sql, $connection;
$connection.Open();
$adapter = New-Object System.Data.OleDb.OleDbDataAdapter $command;
$dataset = New-Object System.Data.DataSet;
[void] $adapter.Fill($dataSet);
$connection.Close();
$rowsCount=($dataset.Tables | Select-Object -Expand Rows).Count;
if ($rowsCount -eq 0) {
    Stop-Process -Name RSPowerBI -Force;
};

Yes, thank you, it is great improvement.

 

If is Power BI RS deployed on single server (or server is shared for frontend and background task), it need this check.

If is Power BI RS in scale-out deployment, and there are dedicated servers for frontend and dedicated servers for background processes, this issue is related only to frontend PBIRS, where is need to killing RSPowerBI process (in this case is this Subscription check not necessary).

Andrino
Frequent Visitor

I'm also experiencing this problem. The key point of this behavior is leaping high CPU usage. RAM and disk I/O performance are not relevant to this problem (we've tried to increase memory limits of DB Engine and rsconfig and move database to fast nvme disk with no effects). When PBIRS is recently restarted, the problem is not observed. But after some time we can see high CPU usage by rspowerbi.exe process with leaping peaks of CPU usage (up to 100%) and at the same time noticable degradation of portal's and reports response.

Valley_Girl
Helper I
Helper I

@josef78 I'm experiencing the same problem, have you escalated it to Microsoft? I've had to reboot the server 3 times in a week as the server has crashed 😔 


@Valley_Girl wrote:

 I'm experiencing the same problem, have you escalated it to Microsoft? I've had to reboot the server 3 times in a week as the server has crashed 😔 


You should really escalate to Microsoft also. You should not experience crashes. If it is the same issue the more reports then if there are multiple reports from different customers it will increase the priority of this issue. And if it's not the same issue you don't want to wait for this issue to be fixed only to find it does not help your situation.

Microsoft are archiving my ticket because they can't find anything convincing in the logs are are suggesting repairing the OS and the PBI Report Server.... At the moment we have a 4am restart in place on the server to stop it crashing which is limiting how often the issue occurs. Please log tickets with Microsoft if you haven't already, otherwise they won't know there's a bug.

@d_gosbell I've raised it with Microsoft this morning 🙂

@Valley_Girl Thank you for ticket. I will rise ticket probably tomorrow (i'm busy, sorry 😞 ), if it increase priority. 

@d_gosbell please, if possible and if it increase priority, I will be grateful if you also create a ticket. Thank you so much

josef78
Super User
Super User

Additionally information, check and repro steps:

I installed clean Power BI Report Server, with default configuration, with September 2021 version. There I saved two small PBI reports (aprox. 50MB) with aprox ten visuals on page. From client computer I opened these reports and setup 5sec refresh interval (using Edge extension) to simulate constant load. On server I setup PerfMon with Procesor utilization, Handles and Private bytes (allocated memory) for Process RSPowerBI. There is result 30minutes of run after restart service:

Sep21Sep21

After this test I upgraded this server to latest January 2022 (1.13.8086.22725) version, restarted, and run same test (exactly same reports and same simulated load). And there is result:

Jan22Jan22


The main cause of problem is that this process (RSPowerBI) does not manage OS handles properly. In the Sep21 version, this process uses an almost constant number of handles (under 1500), over whole period. After the upgrade to Jan22, the number of event handles in this process grows rapidly (over 60000 during half hour), the growth is proportional to the number of dax queries passed thru, and probably due to the number of leaked handles there is excessive memory allocation (possibly in combination with query/result size), and both are to lead to higher CPU utilization.

josef78
Super User
Super User

There is new build of Jan22 version avaliable (1.13.8086.22725), but is still does not solve this problem.

v-henryk-mstf
Community Support
Community Support

Hi @josef78 ,

 

Please follow this document to troubleshoot the problem if it helps.

Monitor report performance in Power BI - Power BI | Microsoft Docs


Looking forward to your feedback.


Best Regards,
Henry


If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Hi @v-henryk-mstf ,

it is not related to Power BI Desktop application. This bug is related Power BI Report Server, server side processing in RSPowerBI.exe process (which provide interface between client and integrated tabular storage (msmdsrv.exe), or external SSAS service).

v-henryk-mstf
Community Support
Community Support

Hi @josef78 ,

 

According to your description, for this kind of problem, it may be a problem related to iFrame in IE, try to switch to another browser, such as Edge or Chrome.

 

Refer to the following similar solutions.

Embedded PowerBI performance and memory leak - Page 2 - Microsoft Power BI Community


If the problem is still not resolved, please point it out. Looking forward to your reply.


Best Regards,
Henry


If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

No, it is not client-side related issue, is not related to your link. 

 

It is problem on Power BI Report Server side, exactly in RSPowerBI.exe process, where is critical memory leak bug.

Helpful resources

Announcements
September Update

Check it Out!

Click here to learn more about the September 2022 updates!

Power BI Dev Camp Session 26

Check it Out!

Mark your calendars and join us on Thursday, September 29 at 11a PDT for a great session with Ted Pattison!