Release Date: September 17, 2018
Temporal Data Management Redesign
In this release, we have made significant improvements to how you manage and work with Xcalar Published tables.
The Xcalar Published table supports the tracking of continuous data changes, enabling you to revert to any data state at any given time from the IMD screen and view the results in a Xcalar table.
NOTE: IMD stands for insert, modify (update), delete, which are commonly used data manipulation operations for making changes to data.
We redesigned the IMD screen to improve the way you can manage multiple Published tables for viewing and analyzing your temporal data, added features that you can apply to help save memory, and applied filter functions for specifying columns based on a condition or a column name, as shown in the screenshot below:
SQL Creation, Management, and Verification Improvements
In this release, we have improved how you create, manage, and verify your SQL queries statements in the SQL code editor.
We redesigned the SQL code editor to help you preserve and manage SQL queries for reuse. We enhanced the code editor by providing autocomplete suggestions and keyboard shortcuts and provided a new panel for you to view and verify the status and results your applied SQL queries. The new SQL code editor enhancements are shown below:
Features and Enhancements in Xcalar 1.4.1
The following sections describe the new features and enhancements for Xcalar release 1.4.1:
Published Table Enhancements
This section describes the enhancements and new features for working with Published tables.
The following new and improved features were applied to the IMD screen:
Added an Inactive Tables section that lets you control which of your Published tables are active or inactive.
Previously by default, all Published tables were active. We listened to your suggestions and replaced the Hidden Tables section with an Inactive Tables section. This enables you to control which Published tables are active and conserves memory and processing time.
Added a new View columns page that lists the column names and data types in the Published table.
Added new icons for managing your Published tables:
Check box icon – The Check all check box enables you to quickly select or deselect all your Published or Inactive tables and the Table check box enables you to run actions on selected Published or Inactive tables.
Drag icon – Enables you to manually organize your tables.
Deactivate and Activate icons – Change the Published table’s state from active to temporary.
Refresh icon – Refreshes the IMD screen.
Added new memory features:
Coalesce icon – This function combines all delta table changes in the Published table, which saves memory and improves refresh times.
Load More Updates tooltip - Xcalar Design limits the display of delta update time-bars to 128. When the number of delta updates exceed this limit, the Load More Updates tooltip displays next to the last updated time-bar. You can display more delta update time-bars by clicking on the Load More Updates tooltip.
Renamed the Refresh Tables element to Generate Table and added filter functions that enable you to specify columns in the generated Xcalar table based on either a condition or a name.
This section describes the enhancements and new features for working with SQL.
The following new and improved features were applied to the the SQL code editor section:
Added automatic saves for your SQL query entries.
You can enter one or multiple SQL query statements in the SQL code editor, which remain until you remove them, even after you log out of Xcalar Design.
Added multiple SQL query statement execution.
You can run multiple SQL queries simultaneously with one EXECUTE SQL click.
NOTE: For a multiple SQL query statement execution, you must end each SQL query statement with a semicolon.
Added autocomplete suggestion and insertion to increase the speed of entering your SQL queries.
Added the code snippet creation and management feature that enables you to preserve one or multiple SQL queries for reuse. Includes a Default Snippet that enables you to quickly apply SQL queries on your SQL Table alias.
SQL keyboard shortcuts are provided to simplify and improve the entering of your SQL queries.
A SQL Query History panel was added, which displays in detail the SQL query statements that were applied to your data. The panel contains the following entry fields:
- Status – Lists the final processing execution state of the query statement as Canceled, Compiling, Done, Failed, Interrupted, or Running.
- Query – Displays the query that was applied to your data.
- Start Time – Displays the time that the query was applied.
- Duration – How long the query took to complete.
- View Results – Displays the results of the query and includes links to the resultant Xcalar table or error message.
Syntax - Xcalar SQL now supports the CUBE, GROUPING SET, GROUPING, and GROUPING_ID operators for the GROUP BY clause.
Operations Support Enhancements
Xcalar now supports the following new and improved operations:
Group By operation:
- You can choose to use only distinct values when determining the aggregate values during a Group By operation.
- You can use the Group By operation to create a table containing only aggregate values.
- The new explodeString Map function splits a string into multiple strings based on a delimiter and creates rows containing each of the resultant strings, duplicating the other fields.
* The Union option in the column drop down has been renamed to Set Operation.
Import Data Source Enhancements
This section describes the enhancements and new features in the Import Data Source page.
New File Format Types and Changes to Existing File Types
Select and restructure JSON data fields using JMESPaths
Previously, Xcalar Design ingested JSON data as-is.
In this release, you can now select and modify specific fields from a JSON data source using the JMESPath syntax. For more information see http://jmespath.org/.
For projects where a broader use of fields within a dataset would be necessary, the best practice is to import an entire dataset.
Responsive XML XPaths and Key Selection
Previously, Xcalar Design users would enter a list of individual XPath syntax strings to determine which XML fields to import.
In this release, you can now add new XPaths and secondary keys from within an XPath. You can, also, refresh the import preview with a click of a button.
Additional Parser for Parquet Dataset
Previously, Xcalar Design used the Apache PyArrow open source Python module to parse an individual parquet file.
In this release, you can choose whether to parse a parquet file using either the Apache PyArrow library or Apache Parquet Tools. PyArrow is the current choice for full parquet dataset parsing.
This section describes the enhancements and new features that improve platform security.
Improved Xcalar Design user identification access and authentication
User identification access and authentication now use server-side session tokens to secure customer activity for all Xcalar interfaces––Xcalar Design, Jupyter Notebook, and the Xcalar API. This technique enables user audit trail tracking. A session token times out after 30 minutes of inactivity. To continue work after a time out, a user can create a new token by logging in again.
IMPORTANT: The session tokens are enforced for XD using cookies. Users must enable cookies in their Xcalar Design supported browser to access Xcalar Design.
For more information on all how to use the new features and enhancements in Xcalar Design, see the Xcalar Online Help documentation.