CSV file not uploading new rows automatically

[Mughees]

Hi.

I have a CSV file that I use in POWER BI. It is updated on weekly basis and I generally replace the old one with the new one. When I uploaded it initially, it had 3,227 rows. A week later when i replaced old CSV with the new one, it had 3777 rows. However, POWER BI is not uploading the new rows. I have tried to find its solutions at multiple sites but couldn t figure it out. 

I even tried to find solution on advanced editor but it only has option to expand the columns. 

Can someone please guide me how it can be fixed.

[Mughees]

I tried to load the same file which I am using currently and it was loaded with the right number of rows: However, when I pasted the advanced editor query code of the previous file in it, tHe number of rows for the new data file also got reduced to 3236 as the old one. Despite the fact that previously it loaded "8379" actual rows. Below is the code:

let
Source = Csv.Document(File.Contents("E:\data\automation\contact sample.CSV"),[Delimiter=",", Columns=88, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"DENTER", type text}, {"COUNTRY", type text}, {"YRONSET", Int64.Type}, {"PROVINCE", type text}, {"PCODE", type text}, {"DISTRICT", type text}, {"DCODE", Int64.Type}, {"GCODE", Int64.Type}, {"TEHSIL", type text}, {"TCODE", Int64.Type}, {"UC", type text}, {"UCODE", Int64.Type}, {"MP", type text}, {"MPTYPE", type text}, {"OTMPOP", type text}, {"SERIALD", Int64.Type}, {"EPID", type text}, {"URGENT", type text}, {"NAME", type text}, {"ADDRESS", type text}, {"SEX", type text}, {"DOB", type text}, {"AGE", Int64.Type}, {"FLANG", type text}, {"OLANG", type text}, {"DONSET", type text}, {"MTHONSET", type text}, {"WKONSET", type text}, {"DNOT", type text}, {"DOI", type text}, {"NOTBY", type text}, {"TYPESITE", type text}, {"DOSESR", type text}, {"EPICARD", type text}, {"DOSESN", type text}, {"DOSES", Int64.Type}, {"VDOSES", type text}, {"DOSESL6", type text}, {"RMDOSES", type text}, {"ORMDOSES", type text}, {"DLOPV", type text}, {"IPVDOSE", type text}, {"DSTOOL1", type text}, {"DSTOOL2", type text}, {"DSTSENT1", type text}, {"DSTSENT2", type text}, {"ADEQ", type text}, {"FEVER", type text}, {"ASYM", type text}, {"PROGRESS", type text}, {"PDIAG", type text}, {"CONTACT", type text}, {"ELIGCONT", type text}, {"INADAFP", type text}, {"URGHARD", type text}, {"URGINSEC", type text}, {"SPECCONC", type text}, {"OTHREAS", type text}, {"SOTHREAS", type text}, {"WILDCONT", type text}, {"VDPVCONT", type text}, {"DCRES", type text}, {"DIRES", type text}, {"LABNO", Int64.Type}, {"DSTLAB1", type text}, {"DSTLAB2", type text}, {"STCOND1", Int64.Type}, {"STCOND2", type text}, {"L20B", Int64.Type}, {"P11", Int64.Type}, {"P12", type text}, {"P1", Int64.Type}, {"P21", Int64.Type}, {"P22", type text}, {"P2", Int64.Type}, {"P31", Int64.Type}, {"P32", type text}, {"P3", Int64.Type}, {"ENTERO1", Int64.Type}, {"ENTERO2", type text}, {"ENTERO", Int64.Type}, {"DFUP", type text}, {"FUP", type text}, {"ERC", type text}, {"CLASS", type text}, {"DIAG", type text}, {"OTHER", type text}, {"", type text}}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Changed Type", "EPID", Splitter.SplitTextByDelimiter("-", QuoteStyle.None), {"EPID.1", "EPID.2"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"EPID.1", type text}, {"EPID.2", type text}}),
#"Renamed Columns" = Table.RenameColumns(#"Changed Type1",{{"EPID.1", "EPID"}, {"EPID.2", "contact_#"}}),
#"Changed Type2" = Table.TransformColumnTypes(#"Renamed Columns",{{"DSTOOL1", type date}}),
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Changed Type2", {"DSTOOL1"}),
#"Changed Type3" = Table.TransformColumnTypes(#"Removed Errors",{{"DSTOOL1", type text}})
in
#"Changed Type3"

Can someone please guide me where is the issue here? Because the source file old and new are same and both have 8379 rows

[Anonymous]

Looking at your Query M sample, it doesn't have any row number limits.

Istead of trying to refresh all tables, try to refresh just the csv data, go to the table view, on the fiels pannel (left side) go for the csv table and in the more options menu (the tree dots) click in incremental refresh, then it should get just the new rows, 3228 to 3777.

[Mughees]

Hi Diego,

It didn t work actually. I tried it couple of times. I was trying to figure out if I can add some range in my advance query for rows, but couldn t get around it. Any other solution in mind

[themistoklis]

Hello @Mughees ,

When you say it is not uploading the enw rows what do you mean.

Also you have other transformations which may exclude specific rows. Have you checked that.

To make sure that all rows are loaded to PowerBI you should always check the first step in Power Query

[Mughees]

I have a data set online that is updated and I download it in csv file format. So everyday, that file is updated and few new rows are added in it but number of columns remain the same. I just export that file in a specific folder and replace it with the old one. This file is linked to POWER BI. So if new rows are being added, they are not being loaded in POWER BI. I cannot figure out how can I check on power query this issue. 

[Anonymous]

Looking at this reply I think I might know what's going around.

The Power BI works with data models, not the file itself, that way even if your file is online, in a onedrive or sharepoint folder, when you upload it again, the power BI datamodel keep the original data, from the file you used to create the dashboard, so, you have two options. Manually open the pbix file and click in "Refresh", or go to the workspace the file is publishd and set a routine for automatic refresh your data, this video might help you in that routine:

https://youtu.be/ClmZ-8p6x9g?t=281