cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
DanBurgon
New Member

Data Load Times with Teradata

Hi,

 

We are storing our reporting datasets in Teradata (in the cloud) and then loading them into Power BI, but am finding the load speed is very slow; this isn't too much of a problem for our more aggregated datasets but it is becoming a problem for granular datasets that we would like to work with. As an example, we recently tried to load a transaction (ish) level table of ~150m rows: the load ran for 12 hours, getting to about 120m rows (about 3k rows per second), and then our VPN dropped out. We're finding this speeds up when we load to the service, but not enough for it not to be an issue.

 

As a comparison, we have been experienting with Amazon Redshift as a data store and the load rates for that on similar data sets is of the order or 100k rows per second. However, Redshift isn't going to be a viable choice for us at the moment, so we're sticking with Teradata. 

 

My question then is, what is the issue with Teradata? Is it that Power BI just doesn't work well with it, or should we be looking at something else like the network?

 

Any pointers would be greatly appreciated!

2 REPLIES 2
DanBurgon
New Member

Thanks for your response @lbendlin 

 

  • In the particular example above we didn't try and load it to the service for that reason 🙂
  • Incremental for some examples of what we do would work, but for others we need to refresh the the whole dataset as historical records can change.
  • I haven't heard of partition bootstrapping - do you have a link to more information?
  • Regardless of what we can do with incremental, partion boot strapping etc. should Teradata be that slow? Should we be looking at gateways, network, a new data platform etc? Or do we need to just live with it?

 

 

lbendlin
Super User
Super User

Interesting. Your load should have conked out at 300 minutes - that is supposedly a hard limit in the Power BI service.

 

Have you considered using  incremental refresh and partition bootstrapping?

Helpful resources

Announcements
Power Platform Conf 2022 768x460.jpg

Join us for Microsoft Power Platform Conference

The first Microsoft-sponsored Power Platform Conference is coming in September. 100+ speakers, 150+ sessions, and what's new and next for Power Platform.

Power BI Dev Camp Session 23 768x460.jpg

Check it Out!

Mark your calendars and join us on Thursday, June 30 at 11a PDT for a great session with Ted Pattison!

June 20 episode 7 with aka link 768x460.jpg

The Power BI Community Show

Join us on June 20 at 11 am PDT when Kim Manis shares the latest on Azure Synapse analytics, the Microsoft Intelligent Data Platform, and notable Power BI Updates from Microsoft Build 2022.

Top Solution Authors