VS Code Synapse integration not working

[paatrickd]

Hi,

I want to open a notebook from Fabric in Visual Studio Code.

I followed the installation guide and managed to activate the Synapse Kernal and open a notebook in VS Code, however it does not recognize spark. See the code below, spark is of Nonetype.

What should I do?

[v-nikhilan-msft · Accepted solution]

Hi @paatrickd 
Thanks for using Fabric Community.
Can you please follow the below troubleshooting steps and provide the details:

1) Right click the ipython notebook.

 

2) Open the python log file: ../../logs/${NOTEBOOK_ARTIFACT_ID}/PySparkLighter.log and check the session failure reason.

Please provide the error details so that I can help you better.

Thank you.

[v-nikhilan-msft · Accepted solution]

Hi @paatrickd 
Thanks for the update.
Maybe you have installed conda, but forgot to set the environment variable. Please check that.
After creating the conda environment, it will install the spark-lighter-lib whl package from blob. The whl package will be automatically upgraded when the version in the conda environment does not match the minimum required version of the extension. If there is any issue with the upgration, you could run the below code :

conda activate ${CONDA_ENV_NAME}
pip uninstall spark-lighter-lib
pip install https://path/to/spark-lighter-lib.whl

Also try these steps.

1. Open the VS Code command palette and select/type 'Python: Create Terminal'
2. In the terminal, run the following command: conda install pyspark
3. Restart VS Code and try again.

Hope this helps. Please let me know if you have any further questions.

[akanane]

Hello, i am experiencing the same issue.

Error code: 

17:18:25,788 root ERROR Failed to initialize Spark Lighter variables. [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

I installed the required software and setup the JAVA_HOME and path.

Thank you

[v-nikhilan-msft]

Hi @akanane ,
Can you please check the syntax of JAVA_HOME environment variable?

Please change it to %JAVA_HOME%.

Hope this helps. Please let me know if you have any further questions. Glad to help.

[akanane]

Hello @v-nikhilan-msft, thanks for point it out the typo.. I fixed i but still cannot use spark, this is the error code now:
09:42:41,388 root INFO current_directory c:\Users\akanane\AppData\Local\miniconda3\772d3ce8-1d2e-4d53-b848-97351ede78a3\SynapseNotebook\4627fe3a-a5dc-4ff7-b3f5-3067b88b080f\BuildFactSales
09:42:41,388 root INFO workspace_path c:\Users\akanane\AppData\Local\miniconda3\772d3ce8-1d2e-4d53-b848-97351ede78a3
09:42:41,388 root INFO log_path c:\Users\akanane\AppData\Local\miniconda3\772d3ce8-1d2e-4d53-b848-97351ede78a3\logs\4627fe3a-a5dc-4ff7-b3f5-3067b88b080f
09:42:41,388 root INFO Using synapse remote kernel ...
09:42:41,388 root INFO Should attach session in dev mode False
09:42:41,468 root INFO Starting session b60ba0ac-6d16-4341-9ae9-f96f55ae67ab...
09:42:43,629 root INFO https://pbipwus13-westus.pbidedicated.windows.net/webapi/capacities/07652219-C048-4E90-B99A-868EC7DC...
09:42:45,71 root INFO <session_management.SessionStatus object at 0x0000019A9A936800>
09:42:45,79 root ERROR Failed to initialize Spark Lighter variables. Create session failed {'code': 'InternalError', 'subCode': 0, 'message': 'An internal error occurred.', 'timeStamp': '2023-11-02T08:42:44.9891628Z', 'httpStatusCode': 500, 'hresult': -2147467259, 'details': [{'code': 'RootActivityId', 'message': 'a038d8a0-e2e4-4375-a8ed-3dfb9866d56f'}, {'code': 'Param1', 'message': 'Response status code does not indicate success: 430 ().'}]}
09:42:45,80 root INFO Registering Spark Lighter magics for IPython...
09:42:45,80 root INFO Registered Spark Lighter magics for IPython.

[v-nikhilan-msft]

Hi @akanane 
You are facing this error because all the available capacity is currently being used. Cancel some currently running Notebooks or Spark Job Definition jobs, increase your available capacity and then try to rerun the notebook.

If the issue still persists please let me know.
Thanks.

[paatrickd]

Hi,

Thank you for your response.

However, I do not have any logs. When following your troubleshooting steps the directory does not contain any log folder.

[v-nikhilan-msft · Accepted solution]

Hi @paatrickd ,
Go to the upper folder to find a folder called logs.

[v-nikhilan-msft · Accepted solution]

Hi @paatrickd 
Thanks for the update.
Maybe you have installed conda, but forgot to set the environment variable. Please check that.
After creating the conda environment, it will install the spark-lighter-lib whl package from blob. The whl package will be automatically upgraded when the version in the conda environment does not match the minimum required version of the extension. If there is any issue with the upgration, you could run the below code :

conda activate ${CONDA_ENV_NAME}
pip uninstall spark-lighter-lib
pip install https://path/to/spark-lighter-lib.whl

Also try these steps.

1. Open the VS Code command palette and select/type 'Python: Create Terminal'
2. In the terminal, run the following command: conda install pyspark
3. Restart VS Code and try again.

Hope this helps. Please let me know if you have any further questions.

[l-man]

I have been able to follow the recommendations up until this point. When I try to activate the environment, I get the following error:
CondaError: Run 'conda init' before 'conda activate'

Running conda init does nothing, the last console log is "No action taken".
I also tried uninstalling spark-lighter-lib but it was not installed.

I insalled spark-lighter-lib using the pip command suggested by BiJoe below, this did not fix the problem.

[ramonsuarez]

I'm on an Ubuntu machine and spark-lighter-lib is not installed. I cannot find it with Pip, Conda, and APT. Searching for it in Google only directs to two tickets here. Where can I find the wheel or how can I install the package? 

Thanks

[BiJoe]

This answer solved my problem not being able to install runtime Apache Spark runtime 1.2, which is installed automatically after runtime 1.1. The Conda environment had no pyspark library due to a wheel download error. In PowerShell I activated the 1.2 runtime Conda environment, then 

pip install https://tridentvscodeextension.blob.core.windows.net/spark-lighter-lib/spark34/spark_lighter_lib-34.0.0.3-py3-none-any.whl

took approximately 10 minutes.  After the download completed, Synapse Pyspark was installed successfully.

[GalRabin]

Hello,

I'm reaching out for assistance regarding an issue I've encountered, which is similar to the one described in the VS Code extension overview - Microsoft Fabric | Microsoft Learn. Despite following all the steps mentioned in the tutorial, I'm facing a problem. I tried to locate the logs that should have been created in the parent directory of the notebook, as per the instructions, but there were no logs to be found.

Would you be able to help me resolve this issue?

Thank you.

[v-nikhilan-msft · Accepted solution]

Hi @paatrickd 
Thanks for using Fabric Community.
Can you please follow the below troubleshooting steps and provide the details:

1) Right click the ipython notebook.

 

2) Open the python log file: ../../logs/${NOTEBOOK_ARTIFACT_ID}/PySparkLighter.log and check the session failure reason.

Please provide the error details so that I can help you better.

Thank you.

[v-nikhilan-msft]

Hi @paatrickd ,
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.