mssparkutils.notebook.run not running notebooks in isolated session context

[shietpas_cbts]

When calling multiple notebooks using the mssparkutils, they do not run in thread-safe manner and will share session context veriables. This is true if you call two notebooks sequentially using mssparkutils.notebook.run sequentally in a notebook or if you call in parallel using mssparkutils.notebook.runMultiple(). In either case, values set in session using 

spark.conf.set will be visible by other other notebooks. And if running in parallel, the notebooks may interfere with each other if using the same variable names.

Temporary Views are also shared across their sessions, which could also create issues.

DBUtils runs notebooks in isoluation and I believe mssparkutils should also run in isolation.

[v-gchenna-msft]

Hi @shietpas_cbts ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

[v-gchenna-msft · Accepted solution]

Hi @shietpas_cbts ,

Apologies for the delay in response, we have an update from the internal team -

spark.conf.set will be share in child notebooks, but local variable in the notebook will not be shared by other notebooks.

---

Hope this is helpful. Please let me know incase of further queries.

[v-gchenna-msft]

Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.

[v-yetao1-msft]

Hi  @shietpas_cbts 

The Issues forum is for posts dealing with Power BI related errors. I have moved your post to the corresponding forum. Thank you for your cooperation.

Best Regards,
Community Support Team _ Ailsa Tao