Filter CSV data within the loading process

[Anonymous]

Hi

I have to load and combine txt files that each have multiple CSV formatted blocks. Something like:

# Some header info1
# Some header info2
#@startFirstBlock
A;B;C;D
E;F;G;H
I;L;M;N
#@endFirstBlock

#@startSecondBlock
1;2;3
4;5;6
#@endSecondBlock

As I need only the " header" rows and the ones between "#@startFirstBlock" and "#@endFirstBlock", is there a way to only load these rows skipping the rest during the loading process?

My question/problem is essentially performance driven: I have 100s of files with 1000s of rows in each "block" and each file can contain 10 and more of these blocks. To load all is taking verrrrrry long and once loaded I have to delete the rows from the not needed blocks (that also takes time). Is there a way to just load what I need from the start?

Thanks for helping.

[ImkeF · Accepted solution]

Hi @Anonymous ,
no, that's not possible. These kind of files have to be ingested fully before anything can be extracted from them.
But it could be that you are making expensive transformations on them. Check out my performance tips for traps to avoid: Speed/Performance aspects – The BIccountant