Problem using "d3-array".rollup (and it spread argument) in a Power BI Custom Visual

[Schizzomarino]

Hello to you all,

I am hoping someone will have a solution for a problem I have encountered using "d3-array" in a custom visual for Power BI.
I have been using d3 v5 as it's got really nice data munching functions that keep my code clean and give good performance; but some of these are now giving me trouble: the rollup function from "d3-array" in particular. Here a description of my situation:

After importing d3 and rollup from d3-array (as rollup), and installing the typescript definitions, when I try to use rollup with two keys, i.e.:
Array.from(rollup(data, ([d]) => d.value, d => +d.date, d => d.name))
I get the following error for the above line: "Expected 3 arguments, but got 4"
If I remove the second or first key the above works, but then I don't get the Map I am looking for and would have liked to keep the above code as it is just so clean...

A small part of the data for clarity:

[{"date":36526,"name":"Coca-Cola","category":"Beverages","value":72537},{"date": 36526,"name": "Microsoft","category": "Technology", "value":70196},...]

Any typescript advice on how to get this to work?

Notes:
Rollup takes arguments in this form: (values, reduce, ...keys); so that keys can be multiple.

This works perfectly fine in a standard web page with the same d3 and d3-array libraries.

Thanks in advance for any advice on the above,

Daniel

[Schizzomarino]

Hi Daniel @dm-p ,

thanks for the reply. The reasons I don't do the grouping with dataViewMapping:

Reuse: The pbiviz libraries change so much with every new version, I prefer using d3's latest v5 version so I can reuse it in future projets while updating to the newest and best pbiviz library.
I would like to know how to remove limitations that d3V5 encounters in Power BI.
Many objects are needed as to animate the visual: This is only one of the many data munching processes I need to do, including ranking the data and creating before and after snapshots for animation. I fear using dataViewMapping would be a lot more complicated than 4 lines of code in d3.
Finally, I will be using this also on standalone webpage visualisations (to embed directly with data within webpages, with no Power BI) and would much rather use the same processes within the custom visualisation.

The output I am looking for here is an array of arrays, with the deepest arrays made up of a map (the original data:

[{"date":36526,"name":"Coca-Cola","category":"Beverages","value":72537},{"date": 36526,"name": "Microsoft","category": "Technology", "value":70196},...]

Should become, this (consider the date formatting here is of no consequence):

Thanks in advance for any guidance you may be able to provide

Dan

[v-evelk]

Hi Dan,

"I would like to know how to remove limitations that d3V5 encounters in Power BI."

If you use the 3rd version of tools there cannot be any limitations for external libraries usage.

If you built your porject using the 2nd version of tools, there is an instruction how to migrate.

Kind Regards,

Evgenii Elkin,
Software Engineer
Microsoft Power BI Custom Visuals
pbicvsupport@microsoft.com

[Schizzomarino]

Thanks Evgenii @v-evelk ,

I am using pbiviz 3.1.5 and have imported the rollup function from d3-array as such:

• import { rollup } from "d3-array"

(I have also installed the types for d3 and it)

• npm install --save @types/d3
• npm install --save @types/d3-array
  

And when only one key is used the rollup function works, it looks to me the problem resides in recognising and using the spread operator that is part of the arguments for rollup.

I am thinking it might be a typescript limitation rather than Power BI, but was hoping for a workaround that would allow me to use the data functions within d3 (rather than recoding them in typescript myself)

best,

Daniel

[jppp]

Hi @Schizzomarino ,

It looks to me that it is an issue with the current typedefinitions (@types/d3-array) as the current only accepts one key, see: https://github.com/DefinitelyTyped/DefinitelyTyped/blob/cbcfdd4b12b863a82f5f1903c47fe22fa3b6975c/typ... is already an issue reported, but with not much happening: https://github.com/DefinitelyTyped/DefinitelyTyped/issues/32162

This kind of errors are usually related to outdated typedefinitions used by TypeScript and you can easily update the current typedefiniton and add the missing '...'

-JP

[Schizzomarino]

Thanks @jppp ,

I had not noticed this... Just goes to show how I still need to learn about definitions in typescript

So looking at the type definition I found this

/* 

@param a The array to group. 

@param reduce The reduce function.  

@param key The key function.

*/

export function rollup<TObject, TKey, TReduce>(a: Iterable<TObject>, reduce: (value: TObject[]) => TReduce, key: (value: TObject) => TKey) : Map<TKey, TReduce>;

Unfortunately I can't figure where to insert the spread operator here, would someone know how to change it, so the above line from the type definitions file provides the third argument as a spread one (key/TReduce)?

Thanks again JP

Daniel

[jppp]

Hi @Schizzomarino ,

You can adjust your local version in the node_modules/@types/d3-array folder.

And I think you only need to add the spread '...' before the key parameter:

export function rollup<TObject, TKey, TReduce>(a: Iterable<TObject>, reduce: (value: TObject[]) => TReduce, ...key: (value: TObject) => TKey) : Map<TKey, TReduce>;

-JP

[Schizzomarino]

Hi @jppp ,

I had tried that amongst a whole pletora of other combinations; but Visual Studio will not compile this... I get the following tooltip:

I have tried so many variations but can't find one that works; and I found this discussion on this:
https://stackoverflow.com/questions/51737041/mapping-generic-spreaded-argument-types-in-typescript

For now I can't quite wrap my head around the technical topic in the above thread... I finsd it rather complicated, probably why it has taken two years, so far, to not have it fixed  :-0

Would anyone know how to get this to work? It would be great after two years to find this solution and update the types mapping for d3-array in the git repository

Thanks again @jppp for pointing me in the right direction, but before I set yours as the correct answer I want to see if anyone has code that will work to define spread arguments when exporting types definitions in Typescript

[jppp · Accepted solution]

Hi @Schizzomarino ,

Sorry, but I took a shot before trying it myself. But if I change the key attribute to ...key: any[] I can get your code working.

I do realise that the typing are a bit off now, but looks like TypeScript doesn't work correctly with this type of parameter binding.

But the most important thing: it compiles!

-JP

[Schizzomarino]

Thanks so much @jppp 
That worked!

If "any[]" (and "any" for that matter) will enable loose type setting but (seem to) work flawlessly, what is stopping everyone from creating such loose typescript definitions?

[dm-p]

Hi @Schizzomarino,

I can help with this, however it's not clear what your expected results are from your post. Can you perhaps share what you would expect your data to look like once processed, or explain how you expect the aggregations to work? I assume you're trying to sum the value and group by name and then by date?

And just to check-off: if this data is coming from the data model to begin with, is there a reason you're not setting up the capabilities and dataViewMapping to do the grouping for you? This way, Power BI will do the computation before the data even hits your visual code and you won't need to do the computation client-side. There will obviously be reasons to do this client-side (for instance, if you're consuming external data sources other than the data model), but just wanted to confirm whether you had considered this as it's a big difference between straight-porting some d3 code into the framework, vs. leveraging the capabilities to get your data in the right shape to begin with.

Either way, let me know what your expected outcome is for the data and I'll do my best to help.

Regards,

Daniel

[Schizzomarino]

Hi Daniel, @dm-p , 

thanks for the reply. The reasons I don't do the grouping with dataViewMapping:

• Reuse: The pbiviz libraries change so much with every new version, I prefer using d3's latest v5 version so I can reuse it in future projets while updating to the newest and best pbiviz library.
• I would like to know how to remove limitations that d3V5 encounters in Power BI.
• Many objects are needed as to animate the visual: This is only one of the many data munching processes I need to do, including ranking the data and creating before and after snapshots for animation. I fear using dataViewMapping would be a lot more complicated than 4 lines of code in d3.
• Finally, I will be using this also on standalone webpage visualisations (to embed directly with data within webpages, with no Power BI) and would much rather use the same processes within the custom visualisation.

The output I am looking for here is an array of arrays, with the deepest arrays made up of a map (the next step would be that the pairs are iterated over using d3.pairs functionality, e.g.  for ([[ka, a], [kb, b]] of d3.pairs(datevalues)){...} ), so the original data, this:

[{"date":36526,"name":"Coca-Cola","category":"Beverages","value":72537},{"date": 36526,"name": "Microsoft","category": "Technology", "value":70196},...]

Should become, this:

Thanks in advance for any guidance you may be able to provide

Dan