Release Date: April 30, 2019
Xcalar Design Redesign
Your user experience is important to us, therefore for this release, we concentrated on improvements to the >Xcalar Design user interface (UI) for increasing your analytics productivity. From importing your data to exporting the final algorithm, Xcalar Design has been revamped and redesigned. This includes a richer and more accessible work area for authoring your SQL data queries or dataflows for data manipulation, and improved query and algorithm performance monitoring and testing features. This was achieved by creating two workspace modes within Xcalar Design:
- For users who prefer to work with SQL code, such as database administrators, the SQL Mode introduces more options and features and functions that enable you to quickly import and create tables from your source data, create and apply SQL queries for analysis, create tables from result sets for more analysis, and export your results outside of Xcalar. The SQL Mode also includes new and improved features that enable you to display and analyze your query statements and their performance and includes extensions that expand the Xcalar SQL syntax.
- The Advanced Mode augments the SQL Mode with features and functions for monitoring and troubleshooting your queries. You can also create and export dataflows in this mode.
Features and Enhancements in Xcalar 2.0
The following sections describe the new features and enhancements for Xcalar release 2.0:
This section describes the new SQL workspace enhancements and new features. In this release, we have revamped the SQL working environment with feature adds and improvements in:
- The following new and improved features were applied to the SQL code editor section:
- Importing SQL code.
- Exporting SQL code.
- New SQL syntax enhancement features enable you to include:
- SQL functions in your SQL code.
- User-defined functions (UDFs) in your SQL code.
- Xcalar defined functions in your SQL code.
Monitoring and tracking during query execution.
Table, table schema, result set, and optimized dataflow preview display from the SQL Workspace:
- Quickly preview tables and their schema.
Table preview details, which include the creation time, row count, table size, and the table state that also enables table activation or deactivation.
- Verify columns and rows from the result set display.
- Create a table from the result set.
- Export a result set outside of Xcalar.
Analyzing and troubleshooting result sets from the SQL Workspace:
Advanced mode enhancements
In this release, we have redesigned how you create and troubleshoot a dataflow with a new improved working environment, including feature adds and improvements in:
Jupyter Notebook improvements:
In this release, the Xcalar Design Jupyter Notebook libraries include the Jupyter Notebook SQL magic predefined functions. These enable you to include simple SQL commands in your Python code that are executable in Xcalar Design.
- Each phase of the dataflow graph is configurable.
- Troubleshoot as you create your dataflow algorithm.
- Create SQL functions from your dataflows that can be inserted into SQL queries.
Storing, accessing, and managing files:
Manage your Python user-defined function (UDF) files easily and efficiently with the new File Manager feature. By default, File Manager enables you to open, copy, delete, download, duplicate, search, and share files. UDFs are easier to locate, access, and apply on your data source, such as during importing or as part of your SQL query. Also, in this release, we have made UDFs shareable between users when stored in File Manager’s shared directory.
Auto memory management improvements:
Previously, users were responsible for memory management.
In this release, memory is automatically managed and conserved for you by Xcalar. This is particularly useful when you are creating your dataflows and do not have the time to worry about memory management.
For example, inactive results sets are now automatically dropped by Xcalar after a certain period of inactivity or when memory consumption is too high. NOTE: Xcalar understands that you may have result sets that you are currently not working with, but are required later. For these, Xcalar gives you the option to either regenerate them when they are required or prevent them from being dropped with the lock feature.
Tutorial workbooks that enable you to learn as you use Xcalar Design.
FNF (field not found) is now treated the same as the SQL NULL term. The SQL NULL term represents a missing value, where a NULL value in a field of a table represents a field that contains no value or a blank field.
Improved Parameter feature:
In this release, you can now create and save parameters from the new Parameter feature in the Advanced Mode Workspace.
The PARAMETER button is located in the Advanced Mode menu bar.
Improved Aggregate feature:
In this release, you can now create and save aggregates from the new Aggregate feature in the Advanced Mode Workspace.
The AGGREGATES button is located in the Advanced Mode menu bar.
Additional features and improvements
The following sections describe other new features and enhancements for Xcalar release 2.0:
- Faster Workbook activation
Previously, when activating a workbook all sessions, such as dataflow operations, were automatically re-executed, taking time and resources.
In this release, when activating a workbook:
- Sessions are not replayed, instead, workbooks will contain un-executed dataflows that if required must be re-run.
- Since workbooks no longer replay sessions, the preview feature has been discontinued.
- User-defined functions (UDFs) are now an element of the workbook and as such must be stored in a directory of the new File Manager feature.
Previously, you had to create a batch dataflow (optimized dataflow) before you could share or use your algorithm on other data.
In this release:
- You can create Advanced Mode dataflows with or without optimization.
- Download and share your optimized or non-optimized dataflows directly from the Advanced Mode Workspace.
NOTE: SQL dataflows are always optimized.
Enhanced exporting data flexibility
The following export drivers are now supported, providing you more control over the format of your exported data:
- fast_csv, which splits your results into smaller files during export.
- legacy_udf, which exports your results using UDFs.
- multiple_csv, which includes an option to include a column name header row and splits your results into smaller files during export.
- single_csv, which exports the results as one file.
Xcalar now supports the following new and improved operators:
- The Link in and Link out Nodes enable you to create and test phases of your dataflow. Smaller dataflows are created and linked together using these Nodes. The Link out Node (last Node of a dataflow) saves the result set of a dataflow as a reference point that is used by the Link in Node (starting Node of a dataflow).
- The Link out Optimized and Export Nodes, optimizes your dataflow so that it can be applied on other tables.
- The PublishIMD Node enables you to create a table that is accessible by Xcalar SQL and third-party visualization tools, such as Tableau.
- The Custom Node enables you to save a dataflow as a Node that can be used in another dataflow.
Xcalar now supports the following new function:
New Data Types
Xcalar now supports the following new data types:
Import Data Source enhancements
This section describes the enhancements and new features in the Import Data Source page.
Xcalar tables must now be associated with a schema that describes the table by its column names and their data type formats. You can choose to manually configure the schema, with or without help from Xcalar Design, or have Xcalar create the schema for you.
- The new schema section includes the Xcalar Design Schema Wizard, which helps you create your schema.
- The schema also supports identifying records with either a primary or composite key.
With the Advanced Mode, you can create either a table or a dataset from your data source, which is used as the source Input Node for your new dataflow.
For more information on how to use the new features and enhancements in Xcalar Design, see the Xcalar Online Help documentation.