plaso.multi_process package

Submodules

plaso.multi_process.analysis_engine module

The task-based multi-process processing analysis engine.

class plaso.multi_process.analysis_engine.AnalysisMultiProcessEngine(worker_memory_limit=None, worker_timeout=None)

Bases: TaskMultiProcessEngine

Task-based multi-process analysis engine.

This class contains functionality to: * monitor and manage analysis tasks; * merge results returned by analysis worker processes.

AnalyzeEvents(session, storage_writer, data_location, analysis_plugins, processing_configuration, event_filter=None, event_filter_expression=None, status_update_callback=None, storage_file_path=None)

Analyzes events in a Plaso storage.

Parameters

• session (Session) – session in which the events are analyzed.

• storage_writer (StorageWriter) – storage writer.

• data_location (str) – path to the location that data files should be loaded from.

• analysis_plugins (dict[str, AnalysisPlugin]) – analysis plugins that should be run and their names.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• event_filter (Optional[EventObjectFilter]) – event filter.

• event_filter_expression (Optional[str]) – event filter expression.

• status_update_callback (Optional[function]) – callback function for status updates.

• storage_file_path (Optional[str]) – path to the session storage file.

Returns

processing status.

Return type

ProcessingStatus

Raises

• KeyboardInterrupt – if a keyboard interrupt was raised.

• ValueError – if analysis plugins are missing.

plaso.multi_process.analysis_process module

The multi-process analysis worker process.

class plaso.multi_process.analysis_process.AnalysisProcess(event_queue, analysis_plugin, processing_configuration, user_accounts, data_location=None, event_filter_expression=None, **kwargs)

Bases: MultiProcessTaskProcess

Multi-processing analysis worker process.

SignalAbort()

Signals the process to abort.

plaso.multi_process.base_process module

Base class for a process used in multi-processing.

class plaso.multi_process.base_process.MultiProcessBaseProcess(processing_configuration, enable_sigsegv_handler=False, **kwargs)

Bases: Process

Interface for multi-processing process.

rpc_port

port number of the process status RPC server.

Type

int

abstract SignalAbort()

Signals the process to abort.

property name

process name.

Type

str

run()

Runs the process.

plaso.multi_process.engine module

The multi-process processing engine.

class plaso.multi_process.engine.MultiProcessEngine

Bases: BaseEngine

Multi-process engine base.

This class contains functionality to: * monitor and manage worker processes; * retrieve a process status information via RPC; * manage the status update thread.

plaso.multi_process.extraction_engine module

The task-based multi-process processing extraction engine.

class plaso.multi_process.extraction_engine.ExtractionMultiProcessEngine(maximum_number_of_tasks=None, number_of_worker_processes=0, status_update_callback=None, worker_memory_limit=None, worker_timeout=None)

Bases: TaskMultiProcessEngine

Task-based multi-process extraction engine.

This class contains functionality to: * monitor and manage extraction tasks; * merge results returned by extraction worker processes.

ProcessSourceMulti(storage_writer, session_identifier, processing_configuration, system_configurations, file_system_path_specs, enable_sigsegv_handler=False, storage_file_path=None)

Processes file systems within a source.

Parameters

• storage_writer (StorageWriter) – storage writer for a session storage.

• session_identifier (str) – the identifier of the session the tasks are part of.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• system_configurations (list[SystemConfigurationArtifact]) – system configurations.

• file_system_path_specs (list[dfvfs.PathSpec]) – path specifications of the source file systems to process.

• enable_sigsegv_handler (Optional[bool]) – True if the SIGSEGV handler should be enabled.

• storage_file_path (Optional[str]) – path to the session storage file.

Returns

processing status.

Return type

ProcessingStatus

Raises

BadConfigOption – if an invalid collection filter was specified or if the preferred time zone is invalid.

plaso.multi_process.extraction_process module

The multi-process extraction worker process.

class plaso.multi_process.extraction_process.ExtractionWorkerProcess(task_queue, processing_configuration, system_configurations, registry_find_specs, **kwargs)

Bases: MultiProcessTaskProcess

Multi-processing extraction worker process.

SignalAbort()

Signals the process to abort.

plaso.multi_process.logger module

The multi-processing sub module logger.

plaso.multi_process.merge_helpers module

Classes to assist in merging attribute containers of tasks.

class plaso.multi_process.merge_helpers.AnalysisTaskMergeHelper(task_storage_reader, task_identifier)

Bases: BaseTaskMergeHelper

Assists in merging attribute containers of an analysis task.

class plaso.multi_process.merge_helpers.BaseTaskMergeHelper(task_storage_reader, task_identifier)

Bases: object

Interface of heler for merging task related attribute containers.

task_identifier

identifier of the task that is merged.

Type

str

Close()

Closes the task storage reader.

GetAttributeContainer()

Retrieves an attribute container to merge.

Returns

attribute container or None if not available.

Return type

AttributeContainer

GetAttributeContainerIdentifier(lookup_key)

Retrieves an attribute container.

Parameters

lookup_key (str) – lookup key that identifies the attribute container.

Returns

attribute container identifier that maps

to the lookup key or None if not available.

Return type

AttributeContainerIdentifier

SetAttributeContainerIdentifier(lookup_key, identifier)

Sets an attribute container.

Parameters

• lookup_key (str) – lookup key that identifies the attribute container.

• identifier (AttributeContainerIdentifier) – attribute container identifier.

class plaso.multi_process.merge_helpers.ExtractionTaskMergeHelper(task_storage_reader, task_identifier)

Bases: BaseTaskMergeHelper

Assists in merging attribute containers of an extraction task.

plaso.multi_process.output_engine module

The output and formatting multi-processing engine.

class plaso.multi_process.output_engine.OutputAndFormattingMultiProcessEngine

Bases: MultiProcessEngine

Output and formatting multi-processing engine.

ExportEvents(storage_reader, output_module, processing_configuration, deduplicate_events=True, event_filter=None, status_update_callback=None, time_slice=None, use_time_slicer=False)

Exports events using an output module.

Parameters

• storage_reader (StorageReader) – storage reader.

• output_module (OutputModule) – output module.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• deduplicate_events (Optional[bool]) – True if events should be deduplicated.

• event_filter (Optional[EventObjectFilter]) – event filter.

• status_update_callback (Optional[function]) – callback function for status updates.

• time_slice (Optional[TimeSlice]) – slice of time to output.

• use_time_slicer (Optional[bool]) – True if the ‘time slicer’ should be used. The ‘time slicer’ will provide a context of events around an event of interest.

Raises

BadConfigOption – if the message formatters file or directory cannot be read.

class plaso.multi_process.output_engine.PsortEventHeap

Bases: object

Psort event heap.

PopEvent()

Pops an event from the heap.

Returns

containing:

str: identifier of the event MACB group or None if the event cannot

be grouped.

str: identifier of the event content. EventObject: event. EventData: event data. EventDataStream: event data stream.

Return type

tuple

PopEvents()

Pops events from the heap.

Yields

tuple –

containing:

str: identifier of the event MACB group or None if the event cannot

be grouped.

str: identifier of the event content. EventObject: event. EventData: event data. EventDataStream: event data stream.

PushEvent(event, event_data, event_data_stream)

Pushes an event onto the heap.

Parameters

• event (EventObject) – event.

• event_data (EventData) – event data.

• event_data_stream (EventDataStream) – event data stream.

property number_of_events

number of events on the heap.

Type

int

plaso.multi_process.plaso_queue module

Queue management implementation for Plaso.

This file contains an implementation of a queue used by plaso for queue management.

The queue has been abstracted in order to provide support for different implementations of the queueing mechanism, to support multi processing and scalability.

class plaso.multi_process.plaso_queue.Queue

Bases: object

Class that implements the queue interface.

abstract Close(abort=False)

Closes the queue.

Parameters

abort (Optional[bool]) – whether the Close is the result of an abort condition. If True, queue contents may be lost.

abstract IsEmpty()

Determines if the queue is empty.

abstract Open()

Opens the queue, ready to enqueue or dequeue items.

abstract PopItem()

Pops an item off the queue.

Raises

QueueEmpty – when the queue is empty.

abstract PushItem(item, block=True)

Pushes an item onto the queue.

Parameters

• item (object) – item to add.

• block (bool) – whether to block if the queue is full.

Raises

QueueFull – if the queue is full, and the item could not be added.

class plaso.multi_process.plaso_queue.QueueAbort

Bases: object

Class that implements a queue abort.

plaso.multi_process.plaso_xmlrpc module

XML RPC server and client.

class plaso.multi_process.plaso_xmlrpc.ThreadedXMLRPCServer(callback)

Bases: RPCServer

Threaded XML RPC server.

Start(hostname, port)

Starts the process status RPC server.

Parameters

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns

True if the RPC server was successfully started.

Return type

bool

Stop()

Stops the process status RPC server.

class plaso.multi_process.plaso_xmlrpc.XMLProcessStatusRPCClient

Bases: XMLRPCClient

XML process status RPC client.

class plaso.multi_process.plaso_xmlrpc.XMLProcessStatusRPCServer(callback)

Bases: ThreadedXMLRPCServer

XML process status threaded RPC server.

class plaso.multi_process.plaso_xmlrpc.XMLRPCClient

Bases: RPCClient

XML RPC client.

CallFunction()

Calls the function via RPC.

Close()

Closes the RPC communication channel to the server.

Open(hostname, port)

Opens a RPC communication channel to the server.

Parameters

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns

True if the communication channel was established.

Return type

bool

plaso.multi_process.rpc module

The RPC client and server interface.

class plaso.multi_process.rpc.RPCClient

Bases: object

RPC client interface.

abstract CallFunction()

Calls the function via RPC.

abstract Close()

Closes the RPC communication channel to the server.

abstract Open(hostname, port)

Opens a RPC communication channel to the server.

Parameters

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns

True if the communication channel was established.

Return type

bool

class plaso.multi_process.rpc.RPCServer(callback)

Bases: object

RPC server interface.

abstract Start(hostname, port)

Starts the RPC server.

Parameters

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns

True if the RPC server was successfully started.

Return type

bool

abstract Stop()

Stops the RPC server.

plaso.multi_process.task_engine module

The task-based multi-process processing engine.

class plaso.multi_process.task_engine.TaskMultiProcessEngine

Bases: MultiProcessEngine

Task-based multi-process engine base.

This class contains functionality to: * manage task storage used to store task results.

plaso.multi_process.task_manager module

The task manager.

class plaso.multi_process.task_manager.TaskManager

Bases: object

Manages tasks and tracks their completion and status.

A task being tracked by the manager must be in exactly one of the following states:

• abandoned: a task assumed to be abandoned because a tasks that has been

  queued or was processing exceeds the maximum inactive time.

• merging: a task that is being merged by the engine.

• pending_merge: the task has been processed and is ready to be merged with

  the session storage.

• processed: a worker has completed processing the task, but it is not ready

  to be merged into the session storage.

• processing: a worker is processing the task.

• queued: the task is waiting for a worker to start processing it. It is also

  possible that a worker has already completed the task, but no status update was collected from the worker while it processed the task.

Once the engine reports that a task is completely merged, it is removed from the task manager.

Tasks are considered “pending” when there is more work that needs to be done to complete these tasks. Pending applies to tasks that are: * not abandoned; * abandoned, but need to be retried.

Abandoned tasks without corresponding retry tasks are considered “failed” when the foreman is done processing.

CheckTaskToMerge(task)

Checks if the task should be merged.

Parameters

task (Task) – task.

Returns

True if the task should be merged.

Return type

bool

Raises

KeyError – if the task was not queued, processing or abandoned.

CompleteTask(task)

Completes a task.

The task is complete and can be removed from the task manager.

Parameters

task (Task) – task.

Raises

KeyError – if the task was not merging.

CreateRetryTask()

Creates a task that to retry a previously abandoned task.

Returns

a task that was abandoned but should be retried or None if there are

no abandoned tasks that should be retried.

Return type

Task

CreateTask(session_identifier, storage_format='sqlite')

Creates a task.

Parameters

• session_identifier (str) – the identifier of the session the task is part of.

• storage_format (Optional[str]) – the storage format that the task should be stored in.

Returns

task attribute container.

Return type

Task

GetFailedTasks()

Retrieves all failed tasks.

Failed tasks are tasks that were abandoned and have no retry task once the foreman is done processing.

Returns

tasks.

Return type

list[Task]

GetProcessedTaskByIdentifier(task_identifier)

Retrieves a task that has been processed.

Parameters

task_identifier (str) – unique identifier of the task.

Returns

a task that has been processed.

Return type

Task

Raises

KeyError – if the task was not processing, queued or abandoned.

GetStatusInformation()

Retrieves status information about the tasks.

Returns

tasks status information.

Return type

TasksStatus

GetTaskPendingMerge(current_task)

Retrieves the first task that is pending merge or has a higher priority.

This function will check if there is a task with a higher merge priority than the current_task being merged. If so, that task with the higher priority is returned.

Parameters

current_task (Task) – current task being merged or None if no such task.

Returns

the next task to merge or None if there is no task pending merge or

with a higher priority.

Return type

Task

HasPendingTasks()

Determines if there are tasks running or in need of retrying.

Returns

True if there are tasks that are active, ready to be merged or

need to be retried.

Return type

bool

RemoveTask(task)

Removes an abandoned task.

Parameters

task (Task) – task.

Raises

KeyError – if the task was not abandoned or the task was abandoned and was not retried.

SampleTaskStatus(task, status)

Takes a sample of the status of the task for profiling.

Parameters

• task (Task) – a task.

• status (str) – status.

StartProfiling(configuration, identifier)

Starts profiling.

Parameters

• configuration (ProfilingConfiguration) – profiling configuration.

• identifier (str) – identifier of the profiling session used to create the sample filename.

StopProfiling()

Stops profiling.

UpdateTaskAsPendingMerge(task)

Updates the task manager to reflect that the task is ready to be merged.

Parameters

task (Task) – task.

Raises

KeyError – if the task was not queued, processing or abandoned, or the task was abandoned and has a retry task.

UpdateTaskAsProcessingByIdentifier(task_identifier)

Updates the task manager to reflect the task is processing.

Parameters

task_identifier (str) – unique identifier of the task.

Raises

KeyError – if the task is not known to the task manager.

plaso.multi_process.task_process module

Base class for a process that handles tasks used in multi-processing.

class plaso.multi_process.task_process.MultiProcessTaskProcess(processing_configuration, enable_sigsegv_handler=False, **kwargs)

Bases: MultiProcessBaseProcess

Interface for multi-processing process that handles tasks.

plaso.multi_process.zeromq_queue module

ZeroMQ implementations of the Plaso queue interface.

class plaso.multi_process.zeromq_queue.ZeroMQBufferedQueue(buffer_timeout_seconds=2, buffer_max_size=10000, delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for buffered Plaso queues.

Buffered queues use a regular Python queue to store items that are pushed or popped from the queue without blocking on underlying ZeroMQ operations.

This class should not be instantiated directly, a subclass should be instantiated instead.

Close(abort=False)

Closes the queue.

Parameters

abort (Optional[bool]) – whether the Close is the result of an abort condition. If True, queue contents may be lost.

Raises

• QueueAlreadyClosed – if the queue is not started, or has already been closed.

• RuntimeError – if closed or terminate event is missing.

Empty()

Removes all items from the internal buffer.

class plaso.multi_process.zeromq_queue.ZeroMQBufferedReplyBindQueue(buffer_timeout_seconds=2, buffer_max_size=10000, delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQBufferedReplyQueue

A Plaso queue backed by a ZeroMQ REP socket that binds to a port.

This queue may only be used to pop items, not to push.

SOCKET_CONNECTION_TYPE = 1

class plaso.multi_process.zeromq_queue.ZeroMQBufferedReplyQueue(buffer_timeout_seconds=2, buffer_max_size=10000, delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQBufferedQueue

Parent class for buffered Plaso queues backed by ZeroMQ REP sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to push items, not to pop.

PopItem()

Pops an item of the queue.

Provided for compatibility with the API, but doesn’t actually work.

Raises

WrongQueueType – As Pop is not supported by this queue.

PushItem(item, block=True)

Push an item on to the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Parameters

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises

• QueueAlreadyClosed – if the queue is closed.

• QueueFull – if the internal buffer was full and it was not possible to push the item to the buffer within the timeout.

• RuntimeError – if closed event is missing.

class plaso.multi_process.zeromq_queue.ZeroMQPullConnectQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQPullQueue

A Plaso queue backed by a ZeroMQ PULL socket that connects to a port.

This queue may only be used to pop items, not to push.

SOCKET_CONNECTION_TYPE = 2

class plaso.multi_process.zeromq_queue.ZeroMQPullQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for Plaso queues backed by ZeroMQ PULL sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to pop items, not to push.

PopItem()

Pops an item off the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Returns

item from the queue.

Return type

object

Raises

• KeyboardInterrupt – if the process is sent a KeyboardInterrupt while popping an item.

• QueueEmpty – if the queue is empty, and no item could be popped within the queue timeout.

• RuntimeError – if closed or terminate event is missing.

• zmq.error.ZMQError – if a ZeroMQ error occurs.

PushItem(item, block=True)

Pushes an item on to the queue.

Provided for compatibility with the API, but doesn’t actually work.

Parameters

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises

WrongQueueType – As Push is not supported this queue.

class plaso.multi_process.zeromq_queue.ZeroMQPushBindQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQPushQueue

A Plaso queue backed by a ZeroMQ PUSH socket that binds to a port.

This queue may only be used to push items, not to pop.

SOCKET_CONNECTION_TYPE = 1

class plaso.multi_process.zeromq_queue.ZeroMQPushQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for Plaso queues backed by ZeroMQ PUSH sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to push items, not to pop.

PopItem()

Pops an item of the queue.

Provided for compatibility with the API, but doesn’t actually work.

Raises

WrongQueueType – As Pull is not supported this queue.

PushItem(item, block=True)

Push an item on to the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Parameters

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises

• KeyboardInterrupt – if the process is sent a KeyboardInterrupt while pushing an item.

• QueueFull – if it was not possible to push the item to the queue within the timeout.

• RuntimeError – if terminate event is missing.

• zmq.error.ZMQError – if a ZeroMQ specific error occurs.

class plaso.multi_process.zeromq_queue.ZeroMQQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: Queue

Interface for a ZeroMQ backed queue.

name

name to identify the queue.

Type

str

port

TCP port that the queue is connected or bound to. If the queue is not yet bound or connected to a port, this value will be None.

Type

int

timeout_seconds

number of seconds that calls to PopItem and PushItem may block for, before returning queue.QueueEmpty.

Type

int

Close(abort=False)

Closes the queue.

Parameters

abort (Optional[bool]) – whether the Close is the result of an abort condition. If True, queue contents may be lost.

Raises

• QueueAlreadyClosed – if the queue is not started, or has already been closed.

• RuntimeError – if closed or terminate event is missing.

IsBound()

Checks if the queue is bound to a port.

IsConnected()

Checks if the queue is connected to a port.

IsEmpty()

Checks if the queue is empty.

ZeroMQ queues don’t have a concept of “empty” - there could always be messages on the queue that a producer or consumer is unaware of. Thus, the queue is never empty, so we return False. Note that it is possible that a queue is unable to pop an item from a queue within a timeout, which will cause PopItem to raise a QueueEmpty exception, but this is a different condition.

Returns

False, to indicate the the queue isn’t empty.

Return type

bool

Open()

Opens this queue, causing the creation of a ZeroMQ socket.

Raises

QueueAlreadyStarted – if the queue is already started, and a socket already exists.

abstract PopItem()

Pops an item off the queue.

Returns

item from the queue.

Return type

object

Raises

QueueEmpty – if the queue is empty, and no item could be popped within the queue timeout.

abstract PushItem(item, block=True)

Pushes an item on to the queue.

Parameters

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises

QueueAlreadyClosed – if the queue is closed.

SOCKET_CONNECTION_BIND = 1

SOCKET_CONNECTION_CONNECT = 2

SOCKET_CONNECTION_TYPE = None

class plaso.multi_process.zeromq_queue.ZeroMQRequestConnectQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQRequestQueue

A Plaso queue backed by a ZeroMQ REQ socket that connects to a port.

This queue may only be used to pop items, not to push.

SOCKET_CONNECTION_TYPE = 2

class plaso.multi_process.zeromq_queue.ZeroMQRequestQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for Plaso queues backed by ZeroMQ REQ sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to pop items, not to push.

PopItem()

Pops an item off the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Returns

item from the queue.

Return type

object

Raises

• KeyboardInterrupt – if the process is sent a KeyboardInterrupt while popping an item.

• QueueEmpty – if the queue is empty, and no item could be popped within the queue timeout.

• RuntimeError – if terminate event is missing.

• zmq.error.ZMQError – if an error occurs in ZeroMQ.

PushItem(item, block=True)

Pushes an item on to the queue.

Provided for compatibility with the API, but doesn’t actually work.

Parameters

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises

WrongQueueType – As Push is not supported this queue.

Module contents