normally the first refresh takes more time tham others when done on your computer, cause when you open the power bi desktop the model its calculating the queries in the background, process thats needed to refresh, so if you let the file open for a while before refreshing you will see it will refresh quick on the first time, the time its takes to do the query on the background depends on the complexity of the query, the data volumen and your pc specs and internet if its using online resources
Did I answer your question? Mark my post as a solution! / Did it help? Give some Kudos!