I'm trying to get my VSCode -> Fabric integration working. I've been through this thread many times, and it was instructive and helpful.
-- I've installed the prereqs listed in the docs, have JAVA_HOME setup, conda and java in the path
-- After installing the VSCode synapse extension, on first use it creates the two conda environments (fabric-synapse-runtime-1-1 and fabric-synapse-runtime-1-2), and on reviewing those initialization logs I see no errors
-- I can browse my workspaces and open notebooks
The problem is whenI try to use a spark command, there is a long pause (session starting in Fabric?), following by an error:
- Example 1-
%%sparksql
SELECT * FROM TableName
--
UsageError: Cell magic `%%sql` not found.
- Example 2-
df = spark.sql("SELECT * FROM LakehouseName.TableName LIMIT 1000")
--
AttributeError: 'NoneType' object has no attribute 'sql'
I've pulled the logs from the local Fabric notebook folder--There are messages in the logs, but I don't know whether these are "normal" or not, since I've not yet used a system where this feature works. Logs are below.
I suspect the issue is in the PySparkLighter.log, but I can't connect the error with any configuration step I may have missed.
Failed to initialize Spark Lighter variables. An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. <-- See below for context
Any ideas?
******************* SparkLighter.log ******************
[WARN ] 2024-01-26 16:52:01.302 [main] Shell: Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
[INFO ] 2024-01-26 16:52:01.963 [Thread-2] SparkContext: Running Spark version 3.4.1
[WARN ] 2024-01-26 16:52:02.013 [Thread-2] NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[INFO ] 2024-01-26 16:52:02.083 [Thread-2] ResourceUtils: ==============================================================
[INFO ] 2024-01-26 16:52:02.084 [Thread-2] ResourceUtils: No custom resources configured for spark.driver.
[INFO ] 2024-01-26 16:52:02.084 [Thread-2] ResourceUtils: ==============================================================
[INFO ] 2024-01-26 16:52:02.085 [Thread-2] SparkContext: Submitted application: Data Cloud: Spark Lighter Connection 2024-01-26 16:51:58.578904
[INFO ] 2024-01-26 16:52:02.099 [Thread-2] ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
[INFO ] 2024-01-26 16:52:02.104 [Thread-2] ResourceProfile: Limiting resource is cpu
[INFO ] 2024-01-26 16:52:02.105 [Thread-2] ResourceProfileManager: Added ResourceProfile id: 0
[INFO ] 2024-01-26 16:52:02.141 [Thread-2] SecurityManager: Changing view acls to: RobertKerr
[INFO ] 2024-01-26 16:52:02.143 [Thread-2] SecurityManager: Changing modify acls to: RobertKerr
[INFO ] 2024-01-26 16:52:02.143 [Thread-2] SecurityManager: Changing view acls groups to:
[INFO ] 2024-01-26 16:52:02.143 [Thread-2] SecurityManager: Changing modify acls groups to:
[INFO ] 2024-01-26 16:52:02.144 [Thread-2] SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: RobertKerr; groups with view permissions: EMPTY; users with modify permissions: RobertKerr; groups with modify permissions: EMPTY
[INFO ] 2024-01-26 16:52:02.357 [Thread-2] Utils: Successfully started service 'sparkDriver' on port 49959.
[INFO ] 2024-01-26 16:52:02.381 [Thread-2] SparkEnv: Registering MapOutputTracker
************* PySparkLighter.log ******************
16:51:36,461 root INFO current_directory c:\Users\RobertKerr\source\CrabShack\4fe867d4-8b9c-451f-94cc-b0b950a52f55\SynapseNotebook\c2564846-6518-4589-8682-db06c92f76f8\Load and Enrich Reviews
16:51:36,461 root INFO workspace_path c:\Users\RobertKerr\source\CrabShack\4fe867d4-8b9c-451f-94cc-b0b950a52f55
16:51:36,462 root INFO log_path c:\Users\RobertKerr\source\CrabShack\4fe867d4-8b9c-451f-94cc-b0b950a52f55\logs\c2564846-6518-4589-8682-db06c92f76f8
16:51:36,462 root INFO Using synapse remote kernel ...
16:51:36,462 root INFO Should attach session in dev mode False
16:51:37,633 root INFO Starting session 0feb87f2-4d8a-4de0-ad12-af0ed67c5a1d...
16:51:38,664 root INFO Getting refresh token...
16:51:41,985 root INFO <session_management.SessionStatus object at 0x0000026C4DCEE830>
16:51:44,201 root INFO Trident session states: starting
16:51:48,750 root INFO Trident session states: busy
16:51:53,240 root INFO Trident session states: busy
16:51:57,654 root INFO Trident session states: idle
16:51:57,656 root INFO Starting session 0feb87f2-4d8a-4de0-ad12-af0ed67c5a1d finished...
16:51:57,656 root INFO Attaching spark session 0feb87f2-4d8a-4de0-ad12-af0ed67c5a1d for Spark Lighter ...
16:51:58,578 root INFO log4j.properties variable LOG_FILE_PATH c:\Users\RobertKerr\source\CrabShack\4fe867d4-8b9c-451f-94cc-b0b950a52f55\logs\c2564846-6518-4589-8682-db06c92f76f8\SparkLighter.log
16:52:02,399 root ERROR Failed to initialize Spark Lighter variables. An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.ExceptionInInitializerError
at org.apache.spark.unsafe.array.ByteArrayMethods.<clinit>(ByteArrayMethods.java:52)
at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes$lzycompute(MemoryManager.scala:306)
at org.apache.spark.memory.MemoryManager.defaultPageSizeBytes(MemoryManager.scala:296)
at org.apache.spark.memory.MemoryManager.$anonfun$pageSizeBytes$1(MemoryManager.scala:315)
at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.memory.MemoryManager.<init>(MemoryManager.scala:315)
at org.apache.spark.memory.UnifiedMemoryManager.<init>(UnifiedMemoryManager.scala:58)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:215)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:328)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:199)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:287)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:505)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:75)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:53)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalStateException: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:113)
... 27 more
Caused by: java.lang.NoSuchMethodException: java.nio.DirectByteBuffer.<init>(long,int)
at java.base/java.lang.Class.getConstructor0(Class.java:3761)
at java.base/java.lang.Class.getDeclaredConstructor(Class.java:2930)
at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:71)
... 27 more
16:52:02,400 root INFO Registering Spark Lighter magics for IPython...
16:52:02,400 root INFO Registered Spark Lighter magics for IPython.
