Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
higgy7
Frequent Visitor

Create a 2nd Dataflow based on a filtered subset of another Dataflow

I have a dataflow which queries a sql server database to extract 12 months worth of data. I use the native query option to run this so it fully folds. 

higgy7_0-1656515944398.png

 

The data is 1 minute interval and an incremental refresh runs several times a day to update.

 

As this is a large set of data and takes a long time to load into Power BI Desktop i wanted to create a subset containing 2 months of data. I would like this subset to query the dataflow created above and not the source sql server database.

 

Within the same workspace i created a dataflow from a linked entry and selected the above dataflow. I then selected reference and added a filter to the reference table. I also turned enable load off on the linked table as i don't want the user to see this.

 

higgy7_1-1656516316962.png

 

A few questions,

1.Does this new dataflow query against the orginal instead of the source sql server database?

2. Do i need to schedule refreshes on the 2nd dataflow or will the data update when the 1st dataflow refreshes?

3. Is this the best practice for creating a subset and the most efficent in terms of performance?

 

I thought this should be a very simple process as there is minmal transformations but the Dataflow refreshes tend to fail regulalry with no helpful error messages.

 

Any help or feedback greatly appreciated.

 

2 REPLIES 2
higgy7
Frequent Visitor

Thanks for the response. This doesn't seem to be the case in my current process:

Latest data in 2nd dataflow

higgy7_0-1656577882492.png

Latest data in 1st dataflow:

higgy7_2-1656578415828.png

 

I was wondering if another part of my dataflow cause complications. I have several dataflows as inputs say dataflow a, b and c with separate refresh schedules, but each one relates to a different subset in dataflow d:

 

higgy7_1-1656578285948.png

Should each filtered table automatically update when its related dataflow updates? If so this is not currently happening.

otravers
Community Champion
Community Champion

1. The new dataflow queries against the original dataflow (i.e. ADLS Storage Gen2 underneath), not the source SQL db.

 

2. If the linked entities are in the same workspace then refreshing dataflow #1 will trigger a refresh of its dependents:

https://docs.microsoft.com/en-us/power-query/dataflows/linked-entities#refresh-logic-of-linked-entit...

------------------------------------------------
1. How to get your question answered quickly - good questions get good answers!
2. Learning how to fish > being spoon-fed without active thinking.
3. Please accept as a solution posts that resolve your questions.
------------------------------------------------
BI Blog: Datamarts | RLS/OLS | Dev Tools | Languages | Aggregations | XMLA/APIs | Field Parameters | Custom Visuals

Helpful resources

Announcements
LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.

PBI_APRIL_CAROUSEL1

Power BI Monthly Update - April 2024

Check out the April 2024 Power BI update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors
Top Kudoed Authors