This has been a longstanding, intermittent problem that has become a critical issue as of late. Many times, simply performing a manual dataset refresh takes hours for the same dataset that typically takes seconds. The source type (e.g. SQL, D365, etc.) does not seem to matter because it appears to occur regardless.
The most critical aspect of this issue is the user's inability to manually stop a running refresh. Instead, the user is forced to wait until it finishes, which many times takes hours.
Refresh in progress
A refresh is in progress. Please wait for the current refresh to finish before clicking Refresh Now. Please try again later or contact support. If you contact support, please provide these details.