export - Events data export

Base classes for defining events export jobs and process chains.

class pycstbox.export.EventsExportJob(jobname, jobid, parms, logger=None)

Base abstract class for defining an export job.

The export job is the unit of work responsible for extracting the data, transforming them according to the target system specification and sending the result to it.

It is also responsible for handing the two steps retry mechanism in case of process failure :

  • retry on the moment of the failure, based on the provided configuration
  • put aside the job for later, so that it can be re-attempted on next data export time slot

To get details on the sequencing of the operation, have a look at the run() method documentation.

Parameters:
  • jobname (str) – job name (should be unique across the application)
  • jobid (str) – the job id. Must be unique and allow sorting a job list and reflect their chronology
  • parms – the execution parameters
  • logger – (optional) message logger
Raises ValueError:
 

if mandatory parameters are not all non empty
static make_jobid(ts=datetime.datetime(2015, 6, 19, 17, 7, 36, 658131))

Return a job id based on a given timestamp, formatted as YYMMDDHHMMSS.

If not specified, the current UTC date and time is used.

Can be overridden if an other kind of id is needed.

run(max_try=1, retry_delay=10)

Job main line process.

This method handles automatic retries in case of data transmission error. If max_try is provided and set to a value greater than 1, the transfer will be retried up to the corresponding total count. A delay of retry_delay seconds will be waited before successive attempts.

There is normally no need for overriding this method, apart if a specific logic is needed.

Parameters:
  • max_try (int) – total number of execution attempts (> 0)
  • retry_delay (int) – the delay (in seconds) to be waited before restarting the process in case of multiple attempts. Don’t choose something too short to avoid being banned as a DOS attack.
Returns:

0 if successful execution, any other value for reporting an error. Codes ERR_xxx are pre-defined to report an error in one of the steps. In case of error, the property last_error can be used to store it for later access.
Raises ValueError:
 

if invalid parameter(s)
prepare()

Initialization of the job optional callback.

Called at the very beginning of the job, before anything else being done. Can be overridden by sub-classes if they need to do something special at this step.

Raises ExportError:
 in case of error
export_events()

Events export mandatory callback.

Export the events in whatever form is suited for the process. Called just after the “prepare” step.

Returns:the number of exported events
Raises ExportError:
 in case of error
send_data()

Export data sending mandatory callback.

Responsible for transmitting the exported data to the target system. Called if export_events() method returns a not null event count.

Raises ExportError:
 in case of error
cleanup(error=None)

Final cleaning optional callback.

Invoked at the very end of the process, whatever is the result of previous steps.

Parameters:error – error encountered in processing if any
class pycstbox.export.Backlog(name, root=None)

A jobs backlog, implemented as a container and backed by a file based storage.

A backlog manages information related to jobs which need to be re-run later. This implementation uses a separate directory for each named backlog, the directory name being the same as the backlog one. These directories are created under a root which path is provided when creating the instance.

The default implementation uses pickle for persisting the data, but one can switch to any other format by overriding the methods _store() and _load().

Parameters:
  • name (str) – the name of the backlog. It must be a valid filename on the target OS
  • root (str) – path of the root directory under which the backlog are stored. Each backlog uses a sub-directory If not specified, uses a sub-directory of DEFAULT_LOCATION_ROOT and which name is the backlog one. The directory (and its parents) is created if not yet existing
Raises ValueError:
 

if path does not comply with above constraints
clear()

Clears all backlog items.

delete()

Deletes the backlog directory.

The backlog can no more be used afterwards, and any operation will raise an exception.

Raises IOError:if not empty.