Xcalar works agnostically of any scheduling system. It does not matter if it is Jenkins. A few others come to mind.
- Jenkins
- Bitbucket
- CircleCI
- MS Team Foundation Server
- TeamCity
- Bamboo
- Chef
- Phabricator
- Drone.io
- Octopus Deploy
- CruiseControl
- AWS CodePipeline
Please check out our Training Manual for Operationalization.
If you login to Xcalar Design, and create a Batch Dataflow schedule it will create a crontab entry that looks like this:
[xcalar@node0 ~]$ crontab -l
* * * * * (export XLRDIR=/opt/xcalar ; python /opt/xcalar/scripts/scheduleRetinas/executeFromCron/executeScheduledRetina.py userreview_XcalarScheduledRetina) 2>&1 /usr/bin/logger -t ScheduledRetina_userreview
You can copy this command into a shell script and this script can then be executed from any job such as Jenkins.
#!/bin/bash
export XLRDIR=/opt/xcalar ; python /opt/xcalar/scripts/scheduleRetinas/executeFromCron/executeScheduledRetina.py userreview_XcalarScheduledRetina) 2>&1 /usr/bin/logger -t ScheduledRetina_userreview
In the above example, replace "userreview" with the name of your batch. This replacement will be done in two places in the above script.
When you execute this script, it will return you status that looks like this:
Aug 16 14:11:01 feynman ScheduledRetina_userreview: 27348
Aug 16 14:11:26 feynman ScheduledRetina_userreview: {'status': 0, 'parameters': [{u'parameterName': u'N', u'parameterValue': '2'}], 'exportLoc': 'Default', 'randomNum': 29.545802865494224, 'startTime': 1502917742000, 'endTime': 1502917886000}
When you see the status code as 0, this means there are no errors with your batch.
Sometimes you will see errors in the logs:
{'status': 439, 'parameters': [{u'parameterName': u'N', u'parameterValue': '338'}], 'exportLoc': 'Default', 'randomNum': 27.711649413438504, 'startTime': 1499799320000, 'endTime': 1499799534000}"}
Timestamp: The first timestamp “Aug 16 14:14:01” marks the start time of the run. Notice that each run begins at the start of each minute, based on the minutely schedule which has been set. +The second timestamp “Aug 16 14:14:25” marks the end time of the run. This indicates that the run took 24 seconds to complete. The resolution is only a second. For millisec resolution, look below.
Hostname: The string “feynman” is the name of the host on which this run was invoked.
Batchname: The string “ScheduledRetina_userreview” contains the batch name, which is “userreview”. This tells you which batch dataflow is running.
Process Id: The number 27348 is the process Id of the usrnode.
Status: The status code is 0, which indicates that this run was executed successfully. Sometimes you see an error code, as shown above.
Parameter Name: The name is ‘N’ and this refers to the monotonically increasing integer that you setup during the parameterization phase for the export file.
Parameter Value: The value is 5, which refers to the value of N. In this case you know this is the 5th run in the batch schedule.
Export Location: This is the export target, and is the Default target.
Random Number: The randomNum is a unique hash code for each batch run. You can use this to uniquely identify a batch run.
Start Time: The startTime is the number of milliseconds since epoch and marks the start of the batch run.
End Time: The endTime is the number of milliseconds since epoch and marks the end of the batch run.
