Xcalar allows you to unleash the power of python based tools that will allow you to write your own code for importing parquet data. As a simple example, you can use pyarrow to import data. It would be good to know how your data is organized. If you can send us how you are currently working on this problem.
- Do you have parquet files?
- Do you organize your directories into parquet datasets?
You can develop an Import UDF in python using pyarrow, which works on a single parquet file. Xcalar will then help you parallelize your processing massively, if you had, for instance, a hundred thousand such parquet files. You only need to write code that parses a single parquet file.
Please look at our discourse articles to see how to write UDFs. You can just search for UDF.
If you need more information about parquet, please let me know and I will be happy to help.