Issue: Power BI RS May 2023 release - Child Process [Portal] Died

[josef78]

Hello all,

After upgrade to Power BI RS May 2023 (15.0.1112.48) on production enviroment and load, I have numerous issues with RSPortal.exe process.

Power BI portal sometimes drops some requests and is unavailable for a while in the day, and it gets worse over time (until I restart the entire service). 

In RSHostingService log I found many (every few minutes under load) warning like this:

Child Process [Portal] Died. Check the log for further information

 Sometimes followed with message:

Immediate restart of process Portal

Or after some delay (when service is unavaliable) followed with message like:

Throttled restart of process by 670 761,67ms: Portal

In RSPortal log I not found nothing interesting, except sometimes repeating message:

Received a request with an unmatched or no authentication scheme.

I not found any of these messages in log before May 2023 version. I think this can be a critical problem on large environments.

Please anyone, do you have any similar problems?

Thanks for the help.

[v-alipat · Accepted solution]

Hi all! 

@josef78 , @YusufCakar 

Thank you for reporting the problem. 

We are looking into it now, and here is a possible workaround for your situation: 

Navigate to RSPortal.exe.config file (Typically located at "C:\Program Files\Microsoft Power BI Report Server\PBIRS\Portal") 

Find there an element <system.diagnostics>, and remove it 
(Remove this piece: )

  <system.diagnostics>

    <trace autoflush="true" />

    <sources>

      <source name="System.Net.HttpListener">

        <listeners>

          <add name="System.Net.HttpListener" />

        </listeners>

      </source>

    </sources>

    <sharedListeners>

      <add name="System.Net.HttpListener" type="Microsoft.BIServer.HostingEnvironment.HttpListenerLogger, Microsoft.BIServer.HostingEnvironment" />

    </sharedListeners>

    <switches>

      <add name="System.Net.HttpListener" value="Verbose" />

    </switches>

  </system.diagnostics>

Could you please try this out, and let us know if this actually helps? 

Looking forward to hear back from you, while we continue our investigation. 

Thanks,

Alexandra 

[v-alipat]

Hi all! 

Letting you know that the fix for this issue was released and the new version can be downloaded here: 

Microsoft Power BI Report Server - May 2023

The problem should not happen in the new version, if you still experience the sympthom after upgrade, please let us know. 

Thanks,

Alexandra 

[Medina]

Hi v-alipat,

While applying the new version of Microsoft Power BI Report Server (the link you shared), we got the following message 'Presenoker' unwanted software was prevented was detected by Microsoft Defender for Endpoint on [servername]

Note: I replaced the actual server name with [servername]

[NTex]

Hi Alexandra

Actually we recently upgraded our test environments with this version 15.0.1112.79 and we start having issues.

Uploading a singe file (~100MB) that usually would take 20 secs, increased to 8 mins alone (this is using Reporting Services Powershell Module, it's an automation we use to integrate our Pipelines from AzureDevOps).

Also, uploading the file manually through the browser acts normal = 20 secs.

Then we found this thread and removing that piece with systems diagnostics, solved it.

So I think the bug might not be squashed yet ?

Thanks

[v-alipat]

Hi @NTex , 

We are investigating your case, but we are unable to replicate the problem. 

Could you please share some more details? 

For example: 

1) Which command are you using for upload? 

Is it Write-RsRestCatalogItem, or other? 

2) Are you uploading PBIX or RDL report? 

3) Are you sure that removing that piece with systems diagnostics solved the problem, could it be anything else? If you add the piece back, does it bring back the problem? 

Thanks! 

Alexandra 

[NTex]

Hi Alexandra

Here it goes:

1) Which command are you using for upload? 

Is it Write-RsRestCatalogItem, or other? 

Yes, we are using the Reporting Service Tools module (GitHub - microsoft/ReportingServicesTools: Reporting Services Powershell Tools) this module is used inside a deploy script, that we ran through a Power Shell task on Azure DevOps to upload files to our SRS/PBI. We have environments with 0.6.5, 0.7.5 and 0.8.0 and the 3 versions suffered from this.

Command we are using in our PS:

Write-RsRestCatalogItem -WebSession $session -Path $PBIXFile -RsFolder $folderPath -Description $Description -Overwrite

2) Are you uploading PBIX or RDL report? 

This actually was happening to all the files in folder, XLSX, PBIX and RDL.

3) Are you sure that removing that piece with systems diagnostics solved the problem, could it be anything else? If you add the piece back, does it bring back the problem? 

Yes, we are sure. We didn't had any issue with previous version. The reason we found this thread was due to fact, we were looking for performance issues related to this specific version that was released recently.

This was the only fix for us, a folder with 55 files with ~800MB's, used to take ~3 mins, went up to ~40/50 mins, so was very noticeable for us.

We spent few hours even debugging / breakpoint the module to see if there was anything wrong related to our specific case.

4) Now if there a specific extension that we could use on Azure DevOps to upload PowerBI, since all are Microsoft products feel free to recommend something.

Thanks for your support

[v-alipat]

Hi again, 

We've tried to reproduce your issue but still were not able to. We suggest you to contact official Microsoft support for further assistance. 

https://support.fabric.microsoft.com/en-us/support/ 

Thanks,

Alexandra 

[NTex]

Hi Alexandra

Environments are fine with that workaround for us, we will wait for the next version to be released and will test again, if the issue is gone.

Thanks for your feedback and tests.

[v-alipat]

Hi @NTex , thank you for letting us know and sharing your repro steps, we will investigate! 

[mpsrshl]

Hey, just out of curiosity, how big is your instance of PBIRS? I'm about to run the server upgrade for this weekend. 

We have a dual core 4 logical cpu, 64 RAM.

Never had speed or clogging issues. I hope it goes smooth we need this upgrade to do embedding reports 

[josef78]

Hi,

in total (multi server scale-out PBIRS) we have 16 physical core hi-perf xeon and 1TB RAM. And currently planning upgrade to twice as much.

[v-alipat · Accepted solution]

Hi all! 

@josef78 , @YusufCakar 

Thank you for reporting the problem. 

We are looking into it now, and here is a possible workaround for your situation: 

Navigate to RSPortal.exe.config file (Typically located at "C:\Program Files\Microsoft Power BI Report Server\PBIRS\Portal") 

Find there an element <system.diagnostics>, and remove it 
(Remove this piece: )

  <system.diagnostics>

    <trace autoflush="true" />

    <sources>

      <source name="System.Net.HttpListener">

        <listeners>

          <add name="System.Net.HttpListener" />

        </listeners>

      </source>

    </sources>

    <sharedListeners>

      <add name="System.Net.HttpListener" type="Microsoft.BIServer.HostingEnvironment.HttpListenerLogger, Microsoft.BIServer.HostingEnvironment" />

    </sharedListeners>

    <switches>

      <add name="System.Net.HttpListener" value="Verbose" />

    </switches>

  </system.diagnostics>

Could you please try this out, and let us know if this actually helps? 

Looking forward to hear back from you, while we continue our investigation. 

Thanks,

Alexandra 

[Gorblimey]

This workaround seems to work for us too! Thank you very much indeed.

Before applying the workaround the issue was reproducible by updating Reports that use the Power Query "sharepoint.table" command to read from SharePoint 2019 libraries (on-premise).

[YusufCakar]

Hello,
Thank you for your quick response.
But downgrade was the only salvation for us. Everything is okay right now. I think there is a problem in May 2023.
I hope the problem will be fixed soon.
Kind regards.

[v-alipat]

Hello again, 

Sure, work is in progress, we will find the root cause and fix it. 

That new config part is our "primary suspect" at the moment, that's why we are looking for a confirmation that disabling it solves the problem for you too 🙂 

Do I get it right that you didn't have a chance to try the workaround before downgrading? 

This info would be very helpful for us. 

Thanks,

Alexandra  

[jmillette007]

Hello,

After recent upgrade to Power BI Report Server (Version1.17.8621.39162(May 2023)) to fix a different issue, our calls to the Report Server's REST endpoints fail, with the following messages in the RSPortal log files:

2023-09-07 14:20:30.0183|WARN|52|Received a request with an unmatched or no authentication scheme. 401

The server is hosting the published reports just fine, but we need these REST services to work as it did in the previous version of the Report Server. It is September :-), will a version which fixes this be available shortly?

BTW, we tried removing the <system.diagnostic> section from the config file and that didn't help.

Thank you,

Jean

[d_gosbell]

---

@jmillette007 wrote:

BTW, we tried removing the <system.diagnostic> section from the config file and that didn't help.

---

@jmillette007 this sounds like a different issue to the one in this thread. I would suggest creating a ticket with Microsoft support.

[YusufCakar]

Hello again, 

-Do I get it right that you didn't have a chance to try the workaround before downgrading? 

-Yes. You got it right. Sorry for not being able to test it. 😢

Kind regards.

[josef78]

Thank you for your help and quick response.

I used this workaround. At first look, it is maybe better. But in this place, it's evening out of office hours, and I'm not able to check it at the real production load now.
I'll check more tomorrow and come back.

Thanks, 

Josef

[josef78]

Hello,

After half a day of Friday's load, this workaround seems to work great. There are no outages related to the RSProtal process dying.

I will continue to monitor and check again on Monday evening after a full standard day (Friday is usually less utilised).

Thank you so much.

[josef78]

Hello @v-alipat ,

I can confirm, that this workaround is working great. There no mentioned errors and no outages.

Please what are next steps? New fixed version will be released?

Thank you much,

Josef