With the addition of Jupyter Notebook and the upgrade to Python 3.6, debugging a UDF is much easier than as described previously. The short answer is to debug your UDF in Jupyter Notebook. Xcalar's built-in Jupyter Notebook uses the same Python environment as your cluster node 0.
Engineering has provided tools to facilitate editing Map and Import UDFs in Jupyter:
- Selecting to import a Custom Format data source file will allow you to click the WRITE PARSER USING JUPYTER button and work in Jupyter with the actual source file.
- In Jupyter, you can add the Create Map UDF code snippet, which will allow you to select columns of data to modify with your map function.
Both facilitate saving your UDF in Xcalar Design's UDF editor, and applying to the data you used to create your UDF. Debugging in Jupyter will allow you to resolve Python coding errors without resorting to accessing log files.
The known cases where your Python would work in Jupyter, but not directly in Xcalar Design are either Xcalar Administration concerns or would require your Xcalar Administrator to consider cluster configuration to empower the work:
- Python on other nodes do not have the same Python packages as node 0 of your cluster.
- The user running the Xcalar software has different privileges on different nodes.
- Node 0 is otherwise configured differently from other nodes in the cluster.
Xcalar explicitly supports multithreaded Python applications. If coding multi-threaded Python functions, your cluster commonly will benefit from system tuning away from the default server configurations. Currently, that will likely entail collaborating with your Admin and potentially Xcalar professional services.
In the circumstance where your UDF is working, but you need visibility into edge conditions, the best practice is to use the Python native logging facility.
Log entries are stored in the xpu.out files stored on each node.
Your Xcalar Admin can access current logs via the Logs panel at the bottom of the Monitor:Administration page . Admins can browse historical logs from the location described by your cluster's XcalarLogCompletePath setting, also accessible to Xcalar Admins on the Monitor:Administration page.
Happy UDF writing!