cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
Responsive Resident
Responsive Resident

Storing and using information from a dynamic data source using PBI desktop.

Hello,

 

This is something that I've worked in for a couple of months, and have been successfully using it for 3 months, and now would like to share with the community. If you want to know where this comes from, please visit this post.

 

 

Now I'm going to explain the whole process, as detailed as possible.

 

1) Connect Power BI to a data source.  

 

For this example, I’m going to connect to a web page that publishes information regarding IBEX 35

1.png2.png3.png

 

After that, we will create a column: Refreshment date = now ()

 

*For users that use regional settings different to English ones, I recommend to disable time intelligence, under Options, Data Load.

 

2) Prepare the system to use the script.

 

I use a Virtual Machine for this, but it can be done from any pc that meets the requirements.

 

First, you will need to install Power Shell’s Package Manager. https://www.microsoft.com/en-us/download/details.aspx?id=51451
*
Edited: It seems this is not necessary in windows 10.

 

Second, run PowerShell as an admin, and follow these steps. (from https://stackoverflow.com/questions/29828756/the-term-install-module-is-not-recognized-as-the-name-o...)4.png

 

Third, download and install SQL_AS_AMO.msi and SQL_AS_ADOMD.msi from https://www.microsoft.com/download/details.aspx?id=52676

Now, you are ready to edit the script.

 

3) Edit the script.

 

You can download it from my onedriveInstructions are placed inside the script.

 

4) Use the task scheduler.

 

Since we don’t want to run the script manually every day (weekends included), let´s program that task!

Download this file first, then open the task scheduler.

Then, select where you want to store the task, and import it.5.png

 

 

I have it set to 10 PM every day, you can change it. Then go to actions, and update the file path to the PowerShell script.

After that, just to check everything is working fine, I’d run the task, just in case…

 

5) Add the stored information to the data model.

 

Now we can go back to power bi. At this point, I prefer to make a copy of the pbix file, with a different name, so that the one that is being used for the script is as simple as it can be.

First, get data from a folder, should be the folder with your table(s) name, inside the “daily” folder created on your desktop.

6.png

Then, click on Edit, Combine & Edit.

7.png

 

 

We check that everything is in order…8.png

 

 

Now we have two tables, one with the information from the dynamic feed, and one from the info that we have stored. Now we need to rename the (2) table, and the columns.9.png

 

 

I recommend doing it from the Query Editor.

10.png

 

 

6) Appending Queries

 

Now is time to put all the info together, and for that we will “Append Queries as new”.

From the query editor, Append queries as new.11.png

12.png

 

Now we have everything in one table, I recommend changing the appends name, and hiding the other two tables in the report view.13.png

 

 

 

After that, we will create a column: Data's Date = IF (ISBLANK (FinalData [Refreshment date]); now (); FinalData [Refreshment date])

That way, we will be able to know when that data was stored.

14.png

 

 

 

In my case, since I need to use the “live” data and the historical one, I use another column to separate it:

Historical Data = "Data from " & DATEDIFF (FinalData [Data's Date (Text)]; NOW (); DAY) & " Day(s) ago"15.png


In case you need to establish relationships between tables, by using this script to store historical data, you will no longer have unique ids to stablish relationships in some cases, what I do to solve that issue, is to add the date to the ids.

 

xxxID = table1[_xxxid_value] & FORMAT(Table1[Data's Date];"MM/DD/YYYY")

If you are using power bi desktop, you don’t need to do anything else, but if you want to publish it to pbi online, you will need to set up a data gateway.

 

 

I hope you’ve enjoyed this long guide, in case you have any doubts, questions, or would like to add something, please feel free to send me an email to salva.gm@outlook.es

 

I would also like to give my special thanks to Jorge Diaz, for his guidance and support whilst developing this, without him, this would have probably ended as a “happy idea”, and to Imke, from thebiccountant.com, for pointing us the right direction.

 

 

 

 

 

 

 

 

 

 

 

 

 

42 REPLIES 42
Helper II
Helper II

Hi Salavador, 

 

I'm stuck with the PowerShell script. It executes up to this line: 

 

$model.RequestRefresh("Full")

 

And then it shows me this error: 

 

Trying to refresh Power BI now

You cannot call a method on a null-valued expression.

Script finished with errors at 09-09-2020_07_14_25

Closing the connection

You cannot call a method on a null-valued expression.

 

Can you point me to the right direction? 

I am getting the exact same error...... Anybody a solution to this?

Helper I
Helper I

Hello @Salvador 

 

I'm getting stuck in running PBI_Exporter with PowerShell.  When I run the PBI_Exporter, below notification would always shows up.

Does anyone here have the same question?

 

Capture.PNG

 

Best,

Syrus

Responsive Resident
Responsive Resident

Hello @Syruswan,

 

Could you send me the powershell file saved as a .txt to check it please?

 

My email is salva.gm@outlook.es

 

Regards,

Salva.

Resolver I
Resolver I

I can't seem to download the Task Scheduler.xml from your OneDrive. I press the button to download and nothing happens.

Hello @ninos-shiba ,

 

I've just tried it, and downloaded without any issues. If you still cant download it, please send me an email to salva.gm@outlook.es and I'll email you the file.

 

Regards,

Salva.

Hello @Salvador ,

 

Thanks for your solution, I am half way through to setup this, but not able to access the links provided by you.

gives me below error.

 

 

This site can’t be reached

The webpage at <<URL>>might be temporarily down or it may have moved permanently to a new web address.

ERR_TUNNEL_CONNECTION_FAILED

Hello @anirudhgupta87,

 

I'll refresh the links as soon as I can, thanks for letting me know.

 

Regards,

Salva.

Thanks a lot @Salvador , thanks heaps !!

Frequent Visitor

Hi Salvador...this is a great idea but the data i have is over 10000 rows and the file gets updated everyday with new info and the old file gets deleted..I'm thinking the size of the file would be a bit impracticle if i keep a record of all the data..all i need is the summary of various status for the previous day as a count that then goes into my final table is it possible to achieve this using this technique?

Hello,

 

If you create the a table in your model with the summarized info that you need, you could use this method. You can also use dax studio to export info to csv, but havent found a way to automatize it.

Frequent Visitor

Hello,

 

could you please be so kind to re-publish the script? it is no longer available

 

thanks a lot

You find the code on message 14 (page 2) of this thread.

Imke Feldmann (The BIccountant)

If you liked my solution, please give it a thumbs up. And if I did answer your question, please mark this post as a solution. Thanks!

How to integrate M-code into your solution -- How to get your questions answered quickly -- How to provide sample data -- Check out more PBI- learning resources here -- Performance Tipps for M-queries

I'm having trouble with the script/ I get the error

'Log_message' is not recognised as the name of a cmdlet, function etc. but only in line 254 and 304 of the script. It is used multiple times above these lines. Could anybody help?


@lpedejong wrote:

I'm having trouble with the script/ I get the error

'Log_message' is not recognised as the name of a cmdlet, function etc. but only in line 254 and 304 of the script. It is used multiple times above these lines. Could anybody help?


Fixed this isue... now when opening power bi i get a syntax error

afbeelding.png

Responsive Resident
Responsive Resident

Hi,

Could you check the script to check if the .pbix that has to be used matches the name of the pbix that should be used?

Advocate II
Advocate II

Hey everyone!
I've contacted Salvador via email and he has sent me an updated link for the scripts on steps 3 and 4: here's the link to the folder

Post Prodigy
Post Prodigy

Sorry, I am having trouble following your steps as it's quite not clear in some points. 

 

Can I do the same for data from salesforce?

 

How did you get the combine and edit option?

Responsive Resident
Responsive Resident

Hello Koluman,

 

You can do the same with data from any source, since what you are doing, is capturing data that is already contained in your power bi desktop version.

 

The combine and edit appears if you select folder as your data source, if you select, for example, an excel, you will only have edit or load.

 

Regards,

Salvador.

Anonymous
Not applicable

@Salvador   how does the 'HistoricalData' table gets created?.. it was not mentioned anywhere in the ps script or steps u created?

Helpful resources

Announcements
secondImage

Experience what’s next for Power BI

Join us for an in-depth look at the new Power BI features and capabilities at the free Microsoft Business Applications Launch Event.

secondImage

Power BI Women

Join our monthly meetings and learning sessions.

secondImage

Congratulations!

We are excited to announce the Power BI Super Users!

secondImage

The largest Power BI virtual conference

100+ sessions, 100+ speakers, Product managers, MVPs, and experts. All about Power BI. Attend online or watch the recordings.

Top Solution Authors
Top Kudoed Authors