Release Date: July 31, 2020
In this release, Xcalar Design, which was the previous IDE, has been broken down into several sub products and has been enabled with new capabilities. We now have 3 exciting offerings: Xcalar Notebook, Xcalar Load, and Xcalar Shell.
New IDE - Xcalar Notebook
Xcalar Notebook as the new integrated developer environment (IDE) has been delivered to users. Users can create data applications using the Xcalar Programming Model in Notebook which aids in increasing user productivity.
5 major components
The Notebook screen is split into 5 key regions:
- Resources panel: Users can view and manage Tables, SQL Statements, Modules, Table Functions and Scalar Functions.
- SQL panel: Users can write and execute ad-hoc SQL statement, modify the SQL tab along with the SQL statements.
- Scalar function panel: Users can write their own python functions and invoke them either as part of their SQL statements or as part of a map operator in the App Builder.
- App Builder: Users can view their SQL execution plans as well as building functions inside Modules using SQL, python and built-in operators.
- Table panel: Users can view their table results in an excel spreadsheet style and with a simple click, they can carry out basic table operations.
SQL-first developing model
Users can query tables which are listed in the Tables section of the Resources panel. After hitting the green arrow in the SQL editor panel, users can visualize the SQL execution planand resultant table.
If users want to extend the SQL statement with Xcalar built-in operators, they can convert the SQL execution plan or SQL graph to an editable module and continue working on building their algorithm in the App Builder.
Build and save data applications with module(s)
Note that SQL Graph is View only. To create a data application, users can convert the SQL Graph into a module first and start building their data flow. After it’s done, users can save their algorithm as an application and schedule it on the on-demand or always-on cluster.
Xcalar Load has been redesigned and re-architected with significant improvements to how users load data into Xcalar. It integrates with AWS Kinesis and allows users to create tables from source data via connectors in a high performance manner.
Xcalar Shell has been packaged with Xcalar 2.3. Users can now manage system configurations, SDLC, monitor cluster performance and their data applications in a programmatic fashion.
- Xcalar Shell offerings
Xcalar Streaming - Kafka (Kafka version 2.2.2)
Xcalar Streaming has been integrated in Xcalar 2.3. Users can work with their Kafka data seamlessly by invoking the Kafka driver in Xcalar Shell. The Kafka driver listens on a Kafka broker, polls for incoming messages, and automatically invokes user-configurable handlers in a Data Application to perform a variety of tasks.
- Configure and run the Kafka application
- Manage the Kafka application
More Features and Enhancements in Xcalar 2.3
SQL Function in the previous version has now been renamed to Table Function. This is a new and improved version of the SQL Function. Users can now select a SQL statement to create a Table Function that can then be invoked in other SQL snippets.
The Table Function created with SQL can also be extended with Xcalar’s built-in operators and Python code.
Xcalar System Apps
StatsMart and LogMart are created as Xcalar System Apps to enhance the troubleshooting capabilities.
StatsMart collects jobStats, systemStats, clusterConfigStats and puts them into tables, which are listed in the StatsMart ER diagram. Users can upload these tables by running the Xcalar Shell command. Once the stats tables have been populated in Xcalar, users are able to query tables or use pre-cooked data applications and sql snippets for basic analysis.
LogMart analyzes 14 log files across the system and parses them into different tables. Users can trigger LogMart by running the Xcalar Shell command. Once the log tables have been populated in Xcalar, users are able to query tables for troubleshooting.
Optimized Application Enhancements - published table supported
Within this release, users can publish a table in their scheduled applications for BI tools access. So far we have 3 output offerings in Xcalar: publish a table, function output, and export a table into the destination folder.
Inline Help System Enhancements
To continuously improve user experience, we have added the latest inline help throughout the UI to get users onboard efficiently.
SQL enhancements - Create table supported
Xcalar now supports CREATE TABLE AS statements, which will publish a table after execution.
New Technical Documentations
- Xcalar Cloud Deployment Guide for AWS
- Enabling access to Xcalar from Tableau
Xcalar 2.3 is available on AWS as a Virtual Private Cloud (VPC) deployment.