Log Module

Module that handles the cluster log:

• Download from master and slaves
• Extract app data
• Extract worker data

log.download(log_folder, instances, master_dns, output_folder, config)

Download the logs from the master and the worker nodes

Parameters:

• log_folder – the log folder of the application
• instances – the instances of the cluster
• master_dns – the dns of the master instances
• output_folder – the output folder where to save the logs

Returns:

the output folder

log.download_master(i, output_folder, log_folder, config)

Download log from master instance

Parameters:

• i – master instance
• output_folder – output folder where save the log
• log_folder – log folder on the master instance

Returns:

output_folder and the app_id: the application id

log.download_slave(i, output_folder, app_id, config)

Download log from slave instance: * The worker log that includes the controller output * The cpu monitoring log

Parameters:

• i – the slave instance
• output_folder – the output folder where to save log
• app_id – the application

Returns:

output_folder: the output folder

log.load_app_data(app_log_path)

Function that parse the application data like stage ids, start, deadline, end, tasktimestamps from the app_log

Parameters: app_log_path – The log of the application with log level INFO Returns: app_info dictionary

log.load_worker_data(worker_log, cpu_log, config)

Load the controller data from the worker_log and combine with the cpu_real data from cpu_log

Parameters:

• worker_log – the path of the log of the worker
• cpu_log – the path of the cpu monitoring tool log of the worker
• config – the configuration dictionary

Returns:

worker_dict the dictionary of the worker’s data