Anyone figured out a nifty way to pass parameters to a dataflow?

[Syndicate_Admin]

I love Power Platform dataflows but am finding the lack of a formal way to pass parameters to them rather frustrating. I was wondering if the community has come up a nifty workaround for implementing a parameter-like functionality. Anyone have some brilliant ideas?

One of my use cases is I want to trigger a dataflow on a file when a person pushes a button in an app, but the dataflow needs to then interact with that SPECIFIC file in a folder and no others. I've got a Power Automate flow set up to refresh the dataflow.

[Syndicate_Admin]

i ended up making a 'Dataflow Job Queue' table in Dataverse.

when the Dataflow runs, it will get the first row off of the top of that job queue, reads the parameters from it, and executes.

there are some cloud flows which orchestrate the queue.

seems to work okay.

[Syndicate_Admin]

Thanks, great idea. 👏🏻 I implemented the same. I'll monitor whether this keeps working, so far so good. 🤞🏻

[Syndicate_Admin]

good job!

mines been running for about a month, so far so good 🙂

[Syndicate_Admin]

@dee2005 Did you have any luck with this? I have a very similar scenario. Reading the above I think @Syndicate_Admin was confusing PowerApps Dataflows (previously data integrations) with Power Automate Flows. As this has been posted under the PowerQuery heading I presume you were referring to Power Apps Dataflows which are similar but not identical to PowerQueries?

I wish MS would take more care when naming and re-naming these things. 

For the sake of clarity (but at the risk of confusing things further) I think the following are similar either in terms of functionality or name but each is distinct in some way:

PowerQuery - used within Excel to automate data processing.

PowerApps DataFlow - similar in operation to Power Query but Cloud-based and standalone.

Power BI DataFlow - very similar to the PowerApps version above but distinct. It has some different functionality.

Power Automate - very different to each of the above. The previous three are more about data manipulation whereas the Power Automate Flows deal with other tasks as well.

[mahoneypat]

What is done with the data after that? This sounds like an atypical scenario, and I am wondering if a different approach might simplify things.

Two initial thoughts

1. if these are Excel files formatted as tables, why not just keep to Power Automate and use List Rows in a table

2. Why not just pull the data from all 5 files and provide a slicer on filename the users can use

Pat

[Syndicate_Admin]

Thanks for the reply, @Syndicate_Admin. I picked this use case as just one example of cases when dataflows need parameters, but there are others. It's a common practice when invoking Stored Procedures, custom functions, and even using Azure Data Factory pipelines.

In the specific scenario above, the files in question may be everything from CSVs to Excel files, and the number of records in each file varies dramatically based on the user uploading them (100 one day, 500k the next). Theoretically, there could be 1+ million records.

These files need to then be processed using the full power of Power Platform Dataflows as the data is then used in Power Apps as well as in Azure Data Factory processes. I had considered Power Automate, but Power Automate's 100k record limit in Apply to Each loops + its concurrency limits made me think Dataflows.

[mahoneypat]

Not totally clear on your use case, but you could pass the file name and/or path from PowerApps to your Flow, add a step to write that value somewhere (e.g., SharePoint list item, Excel table), and then use a second query that extracts those values and returns text (fixed query that returns dynamic text) that is used in your main query.

Pat

[Syndicate_Admin]

@Syndicate_Admin I'd like to try your suggestion, I am wondering how do I get that text configured in my dataflow main query?

"...text (fixed query that returns dynamic text) that is used in your main query"

[Syndicate_Admin]

Thanks for the reply, @Syndicate_Admin . To extrapolate on my use case a bit, I’m imagining cases where, say, 5 users click the button to start a dataflow within a second of each other. Without having the ability to pass parameters DIRECTLY to the dataflow, there’s no reliable way of which I can think to have Dataflows pick up the specific item that is to serve as a parameter. Let’s say each person’s click corresponds to a file. If there are 5 files in a folder, how do we tell the dataflow which file to use so that all 5 user flow runs don’t change the same file?

Hope the use case is making sense.