Hi @v-gchenna-msft and @JMCDOWELL
I found the issue I was having.
In Fabric the top level folders of a workspace are condired "managed one lake folders" .
In the structure of a workspace these folders "Files" and "Tables" /MyLakehouse.lakehouse/Files and /MyLakehouse.lakehouse/Tables
At this level one cannot create or delete folders using the msspartutils only from the "second" level inside those folders.
For example: the code below will fail
import pandas as pd
data = {'name':['John']}
df = pd.DataFrame(data)
path = 'ManagedLevel'
try:
if not mssparkutils.fs.exists(path):
print(f'path does not exit, creating path: {path}')
mssparkutils.fs.mkdirs(path)
else:
print(f'path: {path} already exists')
print('trying to save pandas df in a just created folder')
df.to_csv(f'/lakehouse/default/{path}/able_to_save.csv')
print('save csv successfully')
except Exception as e:
print(e)
Because I am trying to create a filder at the Managed Level or same level as Files and Tables.
The error is a bad request code 400 (not authorized code).
Operation failed: "Bad Request", 400, HEAD,
However if I try and create a folder below the managed level then the code works fine to create the folder but fails to save the pandas dataframe.
import pandas as pd
data = {'name':['John']}
df = pd.DataFrame(data)
path = 'Files/second_level'
try:
if not mssparkutils.fs.exists(path):
print(f'path does not exit, creating path: {path}')
mssparkutils.fs.mkdirs(path)
else:
print(f'path: {path} already exists')
print('trying to save pandas df in a just created folder')
df.to_csv(f'/lakehouse/default/{path}/able_to_save.csv')
print('save csv successfully')
except Exception as e:
print(e)
path does not exit, creating path: Files/second_level
trying to save pandas df in a just created folder
[Errno 2] No such file or directory: '/lakehouse/default/Files/second_level/able_to_save.csv'
I think because the condition that You can perform CRUD (Create, Read, Update and Delete) operations on any folder or file created within these managed folders, and perform read-only operations on workspace and item folders.
Finally at a folder within the second_level it is fine and it all works.
import pandas as pd
data = {'name':['John']}
df = pd.DataFrame(data)
path = 'Files/second_level/third_level'
try:
if not mssparkutils.fs.exists(path):
print(f'path does not exit, creating path: {path}')
mssparkutils.fs.mkdirs(path)
else:
print(f'path: {path} already exists')
print('trying to save pandas df in a just created folder')
df.to_csv(f'/lakehouse/default/{path}/able_to_save.csv')
print('save csv successfully')
except Exception as e:
print(e)
path does not exit, creating path: Files/second_level/third_level
trying to save pandas df in a just created folder
save csv successfully
