Entities do not appear in Dataflow when creating Linked Entities

[Doodle]

Hi,

I created a data query in PBI Desktop. The query combines multiple files and so during that process, PBI created the helper files (sample file, parameter, transform sample file, and transform file). The data was retrieved in PBI Desktop without error.

I then created a dataflow by copying all the query items (sample file, parameter, table, etc.) and creating them as entities in the new dataflow. I created the folder structure exactly as it was in PBI desktop. Again, the data was retrieved in dataflow without error. 

The image below shows two dataflows that were created. Each has two entities (tables).... 1) the transform sample file and 2) the output table from combining all the source files.  In the FY19 dataflow i enabled load for both tables while in the FY20 dataflow I unchecked "enable load" for the 1) transform sample file only.

Issue:

I then attempted to create a dataflow of linked entities and this is where i have the issue.  The output tables mentioned earlier do not appear in the dataflow entities selection panel. I double checked to ensure that both are enabled to load and still they do not show up in the list. The 1) transform sample file in FY20 correctly does not appear because i unchecked enable load.

Please advise.

[richard-powerbi]

As mentioned here:

https://community.powerbi.com/t5/Service/Dataflow-should-be-considered-as-preview/m-p/857007#M81702

Export and import Json can fix.

[Doodle]

Work around doesnt resolve the issue anymore.

To address the original issue of this thread I used the work around of changing date field formats to text format based on the recent discussions on this and other threads. It worked for me yesterday. Now today, I am trying to create a new dataflow using linked entities and am encountering a new but related issue.

The new issue is as follows:

1. I created a dataflow that combines data from many excel files. After the files are combined (i.e. in the final table, not the transform sample table), there are some steps to add the date.

2. As mentioned, the date field was changed to text format and the table now appears in the list of entities. (yesterday's issue)

3. Now Today's issue is that although the table appears, there is no data in the table when trying to link to this dataflow/entity. I have tried both in PBI Service (creating a new linked entity) as shown in the screen capture, and I have tried in PBI Desktop by "getting data" from the dataflow--both show no data.

4.  Surprisingly, when i go to edit the dataflow entity directly, there is preview data shown.  Also, when linking to the entitie, the transform sample file does show data.

To summarize, the workaround is enabling the table to show up, but now new issue is that there is no data in the table.

[Doodle]

Disregard the issue I mentioned above.

During my various testings to resolve the date field format issue, I turned off the "Enhanced Datflows Compute Engine (preview)" in the premium admin portal. Once I switched it back on and reset the Premium server, the date field work around does work as others have mentioned and the data appears as it should. 

[richard-powerbi]

How to turn Enhanced Dataflows Compute Engine off without premium? Can't find.

[rargyle]

MS Product Team are aware of the issue with Dataflow entities not appearing in Desktop browser. They are planning on implementing a fix by the end of the week. 

See thread:

https://community.powerbi.com/t5/Service/Just-started-this-week-DATAFLOW-table-does-not-show-up-in-P...

I have had other issues with dataflows this week, and their tech team suggested I wait until next Monday and check again then. If there are issues remaining next week then submit a support ticket.

[jeffshieldsdev]

My issue is entities not appearing in Desktop or Service...Microsoft Support said the bug has been identified and the fix would take 4 weeks.

Trying to see if I can get them to find out more information from the product team about what's causing the issue--so maybe I can find a workaround instead of waiting a month to share my dataflows.

[jcullum]

@jeffshieldsdev the attached thread as supplied by @rargyle contains all the information required to workaround the issue.  What you have described is consistent with everyone's issue (well not Doodle's new issue).  Any entity containing a Date field is now being stored as "Unclassified".  This results in new and existing Entity not appearing in the 

Workaround:

Edit your dataflow, change all your Date fields to a data type of DateTime or Text, then your entities will appear in both the Service and Deskop.  Change the data type field in the Desktop to back "Date" as required for relationships and visuals.

[jeffshieldsdev]

Right now I'm getting a 500 error trying to save dataflows--so I can't test now.

I don't have any Dates but a lot of DateTimes...I'll change them all to Text and see what happens.  Thanks for the suggestion.

[jcullum]

@jeffshieldsdev That sounds new.  DateTime fields were working fine.  I am not sure the change to text is going to fix your problem.  I have been using a mixture of DateTime and Text.

In the Edit Dataflow web UI, expand the Entities (not to be confused with Editing the Entities) and make sure none of the fields are appearing as "Unclassified".  Any that are you will need to fix by editing the entities.

As an aside.

I get a 500 error if I use the Web UI for the  sharepoint.files connector.  For some reason the UI doesn't include the [ApiVersion = 15] parameter.  I have to use the advanced editor and manually add that into the connector.  I have asked my internal team to raise that issue with MS also.

Good luck with it all.

[venal]

I heard that it is already escalated to the Product Group(PG) Team for their assistance and next update would be this weekend.

Please refer the support page for further updates from PG Team.

https://powerbi.microsoft.com/en-us/support/

Data Flow Current Issues:-

Power BI Customers who are using Dataflows may see scheduled refreshes not processed. As a workaround, customers can perform a manual refresh. Engineers are working on a fix and will provide an ETA soon

Customers creating a new Dataflow from the Power BI service will see the default setting "Allow combining data from multiple sources" set to true instead of false. As a workaround, customers can uncheck the setting. Engineers are working on a fix and expect it to be deployed by end-of-day 11/22/2019.

One more Issue

Data Flow Entities are missing while loading it from Power BI Desktop.

If this post helps, then please consider Accept it as the solution to help the other members find it more
If this post was helpful may I ask you to mark it as solution and click on thumb symbol?

[jcullum]

@venal This is not the the same issue based on what you described.  That workaround of manually refreshing will not work for issues related to Date columns with being set as a data type of Unclassified.  

Please read the full thread, we do not want to be conflating all the separate issues that people are experiencing within the Dataflows component of the service.  The Unclassified issue needs to be resolved as a matter of urgency.

[venal]

@jcullum 

Please monitor the same on this week, if the issue is occurs please submit a ticket to invetigate.

[jcullum]

I am having a similar issue, I could not create a new linked entity from a new dataflows Entity (which I created today).  I also lost an existing entity link.   It was a date table, however when i tried to load the dataflow containing a group of linked entities it throws an error complaining the original entity wasn't valid.

The best I can tell is Entity with a Date in it, will not longer appear for selection when linking.  The workaround suggested didn't help me.  I have 2 workspaces, one containing fact/dimension dataflows and anything will a date is problematic. I can see Dates are being stored as "Unclassified" for the data type.  To expand a little:

Workspace 1:  has 2 dataflows for building the entities.  Any field that contains a date field are now unclassified.

Workspace 2:  has 2 dataflows that link the entities from Workspace one.  I cannot see any Entities from Workspace 1 that contain a Date field.

When i try to link from workspace 1 to workspace 1 (just a test i wouldn't actually do that in the real world), I cannot see any Entities that contain a Date Field.  The new entity i created  cannot be linked and doesn't appear either and it contains no dates.

Last week I was having a problem with Date fields where being stored as DateTime (despite selecting "Date" as the type).  I am not sure if they have attempted to fix that, but this is a pretty big regression.  Essentially dataflows are now unusable. 

It is pretty bad if it is going to take 5 days.  I cannot raise a support ticket as I don't have the privileges to do it (my company has strict controls).  I would be interested if anyone else has these symptoms.

[Doodle]

Just want to add that this issue is only occuring on dataflows that were created this week.

We have old dataflows with linked entities (built over a week ago) within the same workspace that DO have the entities appear correctly.

[Anonymous]

@Doodle Make sure you do an edit the existing ones which work. If you make any change, the tables start to crash. Or may be save a copy of JSON from the dataflow if you need to do any changes.

[Doodle]

I understand you mean "do not edit". Thanks for that!

[Anonymous]

Ohh sorry... correction: "Make sure you do an not edit the existing ones which work. If you make any change, the tables start to crash. Or may be save a copy of JSON from the dataflow if you need to do any changes."

Sorry for the confusion. Thank you @Doodle 

[jeffshieldsdev]

Similar issue for me: empty dataflows on expand in the navigator, previewing data within the source dataflow works just fine.

Enhanced Dataflows Compute Engine is off.  My DateTime columns are listed as DateTime.

I submitted a support ticket and showed them the error...unfortunately I'm on Premium so they had to transfer my ticket to premium support so now I'm waiting to hear back from them.

[jcullum]

I am having a similar issue, I could not create a new linked entity from a new dataflows Entity (which I created today).  I also lost an existing entity link.   It was a date table, however when i tried to load the dataflow containing a group of linked entities it throws an error complaining the original entity wasn't valid.

The best I can tell is Entity with a Date in it, will not longer appear for selection when linking.  The workaround suggested didn't help me.  I have 2 workspaces, one containing fact/dimension dataflows and anything will a date is problematic. I can see Dates are being stored as "Unclassified" for the data type.  To expand a little:

Workspace 1:  has 2 dataflows for building the entities.  Any field that contains a date field are now unclassified.

Workspace 2:  has 2 dataflows that link the entities from Workspace one.  I cannot see any Entities from Workspace 1 that contain a Date field.

When i try to link from workspace 1 to workspace 1 (just a test i wouldn't actually do that in the real world), I cannot see any Entities that contain a Date Field.  The new entity i created  cannot be linked and doesn't appear either and it contains no dates.

Last week I was having a problem with Date fields where being stored as DateTime (despite selecting "Date" as the type).  I am not sure if they have attempted to fix that, but this is a pretty big regression.  Essentially dataflows are now unusable. 

It is pretty bad if it is going to take 5 days.  I cannot raise a support ticket as I don't have the privileges to do it (my company has strict controls).  I would be interested if anyone else has these symptoms.

[Alex_Rajkov]

I can confirm the mentioned issues.

entities containing dates aren't shown anymore / therefore cannot be linked.

this is for dataflows I created / edited & saved this week.

big issue! We need a fix asap!!