Analysis Services stops frequently

[mohammadjavaher]

Hi.

I receive these lines frequently in msmdsrv.log (error is in the last line and type, category and event ID are always the same):

(8/13/2023 1:35:02 PM) Message: Resource tracking started. (Type: 1, Category: 289, Event ID: 0x41210036)
(8/13/2023 1:35:02 PM) Message: The Query thread pool now has 24 minimum threads, 24 maximum threads, and a concurrency of 24.  Its thread pool affinity mask is 0x0000000000000fff. (Type: 1, Category: 289, Event ID: 0x4121000A)
(8/13/2023 1:35:02 PM) Message: The Command thread pool now has 1 minimum threads, 24 maximum threads, and a concurrency of 24.  Its thread pool affinity mask is 0x0000000000000fff. (Type: 1, Category: 289, Event ID: 0x4121000A)
(8/13/2023 1:35:02 PM) Message: The Parsing Short thread pool now has 4 minimum threads, 24 maximum threads, and a concurrency of 24.  Its thread pool affinity mask is 0x0000000000000fff. (Type: 1, Category: 289, Event ID: 0x4121000A)
(8/13/2023 1:35:02 PM) Message: The Parsing Long thread pool now has 4 minimum threads, 24 maximum threads, and a concurrency of 24.  Its thread pool affinity mask is 0x0000000000000fff. (Type: 1, Category: 289, Event ID: 0x4121000A)
(8/13/2023 1:35:02 PM) Message: The Processing thread pool now has 24 minimum threads, 64 maximum threads, and a concurrency of 24.  Its thread pool affinity mask is 0x0000000000000fff. (Type: 1, Category: 289, Event ID: 0x4121000A)
(8/13/2023 1:35:02 PM) Message: The IOProcessing thread subpool with affinity 0x0000000000000fff now has 1 minimum threads, 120 maximum threads, and a concurrency of 24. (Type: 1, Category: 289, Event ID: 0x4121000B)
(8/13/2023 1:35:02 PM) Message: DLL 'xe.dll' loaded with version '2022.160.455.0'. (Type: 1, Category: 289, Event ID: 0x41210038)
(8/13/2023 1:35:02 PM) Message: DLL 'msmgdsrv.dll' loaded with version '2022.160.77.22'. (Type: 1, Category: 289, Event ID: 0x41210038)
(8/13/2023 1:35:03 PM) Message: Regeneration of the server Gen2 cryptokey file skipped as server gen2 cryptokey present in Data directory. (Type: 1, Category: 289, Event ID: 0x4121005B)
(8/13/2023 1:35:03 PM) Message: Server config file will be updated due to Gen2ServerKeyToGenerate=0, config prop Gen2ServerKey enforced=1, -f input parameter enforced=0. (Type: 1, Category: 289, Event ID: 0x4121005C)
(8/13/2023 1:35:03 PM) Message: The Vertipaq thread pool now has 2 minimum threads, 12 maximum threads, and a concurrency of -1.  Its thread pool affinity mask is 0x0000000000000fff. (Type: 1, Category: 289, Event ID: 0x4121000A)
(8/13/2023 1:35:03 PM) Message: Scheduler will use runnable queue of type = SHORT_BIAS_WITH_FAST_CANCELLATION (reason: fallback to default settings). (Type: 1, Category: 289, Event ID: 0x41210032)
(8/13/2023 1:35:03 PM) Message: Scheduler uses Short Bias policy. Nodes = 1, Groups = 1, CPUs = 6, Cores = 12. Bias settings: nReservedFastCompute = 75 and nDecayIntervalMs = 60000. (Type: 1, Category: 289, Event ID: 0x41210034)
(8/13/2023 1:35:03 PM) Message: Scheduler uses Short Bias policy. Supported Configuration = 1, Default Vertipaq Leaf Queue = 1, Stealing Enabled = 1, Multicore Environment = 1, Enough Reservation for Fast and Slow Workload = 1. (Type: 1, Category: 289, Event ID: 0x41210035)
(8/13/2023 1:35:03 PM) Message: Transaction recovery started. (Type: 1, Category: 289, Event ID: 0x41210029)
(8/13/2023 1:35:03 PM) Message: DLL 'Microsoft.AnalysisServices.MInterop' loaded with version '16.0.77.22'. (Type: 1, Category: 289, Event ID: 0x41210038)
(8/13/2023 1:35:03 PM) Message: Transaction recovery successful (Type: 1, Category: 289, Event ID: 0x4121002A)
(8/13/2023 1:35:04 PM) Message: The flight recorder snapshot definition file 'C:\Program Files\Microsoft Power BI Report Server\PBIRS\ASEngine\FlightRecorderSnapshotDef.xml' was not found, skipping initialization of snapshot activity. (Type: 1, Category: 289, Event ID: 0x41210021)
(8/13/2023 1:35:05 PM) Message: An error occurred at critical stage of transaction; we intercepted it, but have to terminate the process. (Type: 3, Category: 289, Event ID: 0xC1210058)

It's apparent that there is something wrong with AS. No report can be opened. No report can be refreshed either. I receive "Failed to connect to AS" when refreshing reports. Tried running PBI service as different accounts, disabling AV (KES) and repairing RS installation (May 2023, 1.17.8552.27722) with no luck. This happened all of a sudden without any change in configurations. No process is listening on 5132 according to netstats. I have circumvented this situation by disabling CriticalErrorHandling, but I believe that this is not the right way. I'm looking for the root cause as I have two high-load RS instances (one for each of the two companies I work for) and any unresolved issues can make severe problems in the future.

Thanks.

------------------------------

EDIT: My second server (server for the second company I work for) just started to give the same error from August, 31st morning. PLEASE everyone if you have any idea, let me know. The bad news is that although I have modified msmdsrv.ini on both servers, AS engine stops working at a frequency of "2-3 times a day" to "every 1-2 days". I don't know what causes a critical error in AS engine.Thanks!

[mohammadjavaher]

So, I managed to overcome this situation as follows:

1. Wrote a view to show latest refresh status of each report.
2. Wrote a stored procedure which force-refreshes each failed refresh plan using AddEvent SP already present in RS instance, and then sends an email alert using already-configure database mail profile.
3. Wrote a Windows service using VB.Net which checks report refresh failures in cofigurable periods. If any failure from AS engine is detected, it restarts Power BI RS service, waits for a configurable couple of seconds so RS reaches full functionality, and then calls that SP on step 2 to force refresh failed reports.

This way, I'm at least sure server will come back online whenever AS engine fails. I have a report which is refreshed every 5 minutes and this guarantees that AS engine failure is detected shortly.

Hope this helps someone else.

Regards.

[d_gosbell]

You should raise a ticket with Microsoft support. This is not expected behaviour.

[mohammadjavaher]

Hi.

Thanks for your reply. Unfortunately, I cannot receive MS support because I'm from Iran. I should receive support from third parties.

Regards