plaso.multi_process package

Submodules

plaso.multi_process.analysis_engine module

The task-based multi-process processing analysis engine.

class plaso.multi_process.analysis_engine.AnalysisMultiProcessEngine(worker_memory_limit=None, worker_timeout=None)

Bases: TaskMultiProcessEngine

Task-based multi-process analysis engine.

This class contains functionality to: * monitor and manage analysis tasks; * merge results returned by analysis worker processes.

AnalyzeEvents(session, storage_writer, data_location, analysis_plugins, processing_configuration, event_filter=None, event_filter_expression=None, status_update_callback=None, storage_file_path=None)

Analyzes events in a Plaso storage.

Parameters:

• session (Session) – session in which the events are analyzed.

• storage_writer (StorageWriter) – storage writer.

• data_location (str) – path to the location that data files should be loaded from.

• analysis_plugins (dict[str, AnalysisPlugin]) – analysis plugins that should be run and their names.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• event_filter (Optional[EventObjectFilter]) – event filter.

• event_filter_expression (Optional[str]) – event filter expression.

• status_update_callback (Optional[function]) – callback function for status updates.

• storage_file_path (Optional[str]) – path to the session storage file.

Returns:

processing status.

Return type:

ProcessingStatus

Raises:

• KeyboardInterrupt – if a keyboard interrupt was raised.

• ValueError – if analysis plugins are missing.

__init__(worker_memory_limit=None, worker_timeout=None)

Initializes a task-based multi-process analysis engine.

Parameters:

• worker_memory_limit (Optional[int]) – maximum amount of memory a worker is allowed to consume, where None represents the default memory limit and 0 represents no limit.

• worker_timeout (Optional[float]) – number of minutes before a worker process that is not providing status updates is considered inactive, where None or 0.0 represents the default timeout.

plaso.multi_process.analysis_process module

The multi-process analysis worker process.

class plaso.multi_process.analysis_process.AnalysisProcess(event_queue, analysis_plugin, processing_configuration, user_accounts, data_location=None, event_filter_expression=None, **kwargs)

Bases: MultiProcessTaskProcess

Multi-processing analysis worker process.

SignalAbort()

Signals the process to abort.

__init__(event_queue, analysis_plugin, processing_configuration, user_accounts, data_location=None, event_filter_expression=None, **kwargs)

Initializes an analysis worker process.

Non-specified keyword arguments (kwargs) are directly passed to multiprocessing.Process.

Parameters:

• event_queue (plaso_queue.Queue) – event queue.

• analysis_plugin (AnalysisPlugin) – plugin running in the process.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• user_accounts (list[UserAccountArtifact]) – user accounts.

• data_location (Optional[str]) – path to the location that data files should be loaded from.

• event_filter_expression (Optional[str]) – event filter expression.

plaso.multi_process.base_process module

Base class for a process used in multi-processing.

class plaso.multi_process.base_process.MultiProcessBaseProcess(processing_configuration, enable_sigsegv_handler=False, **kwargs)

Bases: Process

Interface for multi-processing process.

rpc_port

port number of the process status RPC server.

Type:

int

abstractmethod SignalAbort()

Signals the process to abort.

__init__(processing_configuration, enable_sigsegv_handler=False, **kwargs)

Initializes a process.

Parameters:

• processing_configuration (ProcessingConfiguration) – processing configuration.

• enable_sigsegv_handler (Optional[bool]) – True if the SIGSEGV handler should be enabled.

• kwargs (dict[str,object]) – keyword arguments to pass to multiprocessing.Process.

property name

process name.

Type:

str

run()

Runs the process.

plaso.multi_process.engine module

The multi-process processing engine.

class plaso.multi_process.engine.MultiProcessEngine

Bases: BaseEngine

Multi-process engine base.

This class contains functionality to: * monitor and manage worker processes; * retrieve a process status information via RPC; * manage the status update thread.

__init__()

Initializes a multi-process engine.

plaso.multi_process.extraction_engine module

The task-based multi-process processing extraction engine.

class plaso.multi_process.extraction_engine.ExtractionMultiProcessEngine(maximum_number_of_tasks=None, number_of_worker_processes=0, status_update_callback=None, worker_memory_limit=None, worker_timeout=None)

Bases: TaskMultiProcessEngine

Task-based multi-process extraction engine.

This class contains functionality to: * monitor and manage extraction tasks; * merge results returned by extraction worker processes.

ProcessSourceMulti(storage_writer, session_identifier, processing_configuration, system_configurations, file_system_path_specs, enable_sigsegv_handler=False, storage_file_path=None)

Processes file systems within a source.

Parameters:

• storage_writer (StorageWriter) – storage writer for a session storage.

• session_identifier (str) – the identifier of the session the tasks are part of.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• system_configurations (list[SystemConfigurationArtifact]) – system configurations.

• file_system_path_specs (list[dfvfs.PathSpec]) – path specifications of the source file systems to process.

• enable_sigsegv_handler (Optional[bool]) – True if the SIGSEGV handler should be enabled.

• storage_file_path (Optional[str]) – path to the session storage file.

Returns:

processing status.

Return type:

ProcessingStatus

Raises:

BadConfigOption – if an invalid collection filter was specified or if the preferred time zone is invalid.

__init__(maximum_number_of_tasks=None, number_of_worker_processes=0, status_update_callback=None, worker_memory_limit=None, worker_timeout=None)

Initializes an engine.

Parameters:

• maximum_number_of_tasks (Optional[int]) – maximum number of concurrent tasks, where 0 represents no limit.

• number_of_worker_processes (Optional[int]) – number of worker processes.

• status_update_callback (Optional[function]) – callback function for status updates.

• worker_memory_limit (Optional[int]) – maximum amount of memory a worker is allowed to consume, where None represents the default memory limit and 0 represents no limit.

• worker_timeout (Optional[float]) – number of minutes before a worker process that is not providing status updates is considered inactive, where None or 0.0 represents the default timeout.

plaso.multi_process.extraction_process module

The multi-process extraction worker process.

class plaso.multi_process.extraction_process.ExtractionWorkerProcess(task_queue, processing_configuration, system_configurations, windows_event_log_providers, registry_find_specs, **kwargs)

Bases: MultiProcessTaskProcess

Multi-processing extraction worker process.

SignalAbort()

Signals the process to abort.

__init__(task_queue, processing_configuration, system_configurations, windows_event_log_providers, registry_find_specs, **kwargs)

Initializes an extraction worker process.

Non-specified keyword arguments (kwargs) are directly passed to multiprocessing.Process.

Parameters:

• task_queue (PlasoQueue) – task queue.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• system_configurations (list[SystemConfigurationArtifact]) – system configurations.

• windows_event_log_providers (list[WindowsEventLogProviderArtifact]) – Windows EventLog providers.

• registry_find_specs (list[dfwinreg.FindSpec]) – Windows Registry find specifications.

• kwargs – keyword arguments to pass to multiprocessing.Process.

plaso.multi_process.logger module

The multi-processing sub module logger.

plaso.multi_process.merge_helpers module

Classes to assist in merging attribute containers of tasks.

class plaso.multi_process.merge_helpers.AnalysisTaskMergeHelper(task_storage_reader, task_identifier)

Bases: BaseTaskMergeHelper

Assists in merging attribute containers of an analysis task.

class plaso.multi_process.merge_helpers.BaseTaskMergeHelper(task_storage_reader, task_identifier)

Bases: object

Interface of heler for merging task related attribute containers.

task_identifier

identifier of the task that is merged.

Type:

str

Close()

Closes the task storage reader.

GetAttributeContainer()

Retrieves an attribute container to merge.

Returns:

attribute container or None if not available.

Return type:

AttributeContainer

GetAttributeContainerIdentifier(lookup_key)

Retrieves an attribute container.

Parameters:

lookup_key (str) – lookup key that identifies the attribute container.

Returns:

attribute container identifier that maps

to the lookup key or None if not available.

Return type:

AttributeContainerIdentifier

SetAttributeContainerIdentifier(lookup_key, identifier)

Sets an attribute container.

Parameters:

• lookup_key (str) – lookup key that identifies the attribute container.

• identifier (AttributeContainerIdentifier) – attribute container identifier.

__init__(task_storage_reader, task_identifier)

Initialize a helper for merging task related attribute containers.

Parameters:

• task_storage_reader (StorageReader) – task storage reader.

• task_identifier (str) – identifier of the task that is merged.

class plaso.multi_process.merge_helpers.ExtractionTaskMergeHelper(task_storage_reader, task_identifier)

Bases: BaseTaskMergeHelper

Assists in merging attribute containers of an extraction task.

plaso.multi_process.output_engine module

The output and formatting multi-processing engine.

class plaso.multi_process.output_engine.OutputAndFormattingMultiProcessEngine

Bases: MultiProcessEngine

Output and formatting multi-processing engine.

ExportEvents(storage_reader, output_module, processing_configuration, deduplicate_events=True, event_filter=None, status_update_callback=None, time_slice=None, use_time_slicer=False)

Exports events using an output module.

Parameters:

• storage_reader (StorageReader) – storage reader.

• output_module (OutputModule) – output module.

• processing_configuration (ProcessingConfiguration) – processing configuration.

• deduplicate_events (Optional[bool]) – True if events should be deduplicated.

• event_filter (Optional[EventObjectFilter]) – event filter.

• status_update_callback (Optional[function]) – callback function for status updates.

• time_slice (Optional[TimeSlice]) – slice of time to output.

• use_time_slicer (Optional[bool]) – True if the ‘time slicer’ should be used. The ‘time slicer’ will provide a context of events around an event of interest.

Raises:

BadConfigOption – if the message formatters file or directory cannot be read.

__init__()

Initializes an output and formatting multi-processing engine.

class plaso.multi_process.output_engine.PsortEventHeap

Bases: object

Psort event heap.

PopEvent()

Pops an event from the heap.

Returns:

containing:

str: identifier of the event MACB group or None if the event cannot

be grouped.

str: identifier of the event content. EventObject: event. EventData: event data. EventDataStream: event data stream.

Return type:

tuple

PopEvents()

Pops events from the heap.

Yields:

tuple –

containing:

str: identifier of the event MACB group or None if the event cannot

be grouped.

str: identifier of the event content. EventObject: event. EventData: event data. EventDataStream: event data stream.

PushEvent(event, event_data, event_data_stream)

Pushes an event onto the heap.

Parameters:

• event (EventObject) – event.

• event_data (EventData) – event data.

• event_data_stream (EventDataStream) – event data stream.

__init__()

Initializes a psort events heap.

property number_of_events

number of events on the heap.

Type:

int

plaso.multi_process.plaso_queue module

Queue management implementation for Plaso.

This file contains an implementation of a queue used by plaso for queue management.

The queue has been abstracted in order to provide support for different implementations of the queueing mechanism, to support multi processing and scalability.

class plaso.multi_process.plaso_queue.Queue

Bases: object

Class that implements the queue interface.

abstractmethod Close(abort=False)

Closes the queue.

Parameters:

abort (Optional[bool]) – whether the Close is the result of an abort condition. If True, queue contents may be lost.

abstractmethod IsEmpty()

Determines if the queue is empty.

abstractmethod Open()

Opens the queue, ready to enqueue or dequeue items.

abstractmethod PopItem()

Pops an item off the queue.

Raises:

QueueEmpty – when the queue is empty.

abstractmethod PushItem(item, block=True)

Pushes an item onto the queue.

Parameters:

• item (object) – item to add.

• block (bool) – whether to block if the queue is full.

Raises:

QueueFull – if the queue is full, and the item could not be added.

class plaso.multi_process.plaso_queue.QueueAbort

Bases: object

Class that implements a queue abort.

plaso.multi_process.plaso_xmlrpc module

XML RPC server and client.

class plaso.multi_process.plaso_xmlrpc.ThreadedXMLRPCServer(callback)

Bases: RPCServer

Threaded XML RPC server.

Start(hostname, port)

Starts the process status RPC server.

Parameters:

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns:

True if the RPC server was successfully started.

Return type:

bool

Stop()

Stops the process status RPC server.

__init__(callback)

Initialize a threaded RPC server.

Parameters:

callback (function) – callback function to invoke on get status RPC request.

class plaso.multi_process.plaso_xmlrpc.XMLProcessStatusRPCClient

Bases: XMLRPCClient

XML process status RPC client.

class plaso.multi_process.plaso_xmlrpc.XMLProcessStatusRPCServer(callback)

Bases: ThreadedXMLRPCServer

XML process status threaded RPC server.

class plaso.multi_process.plaso_xmlrpc.XMLRPCClient

Bases: RPCClient

XML RPC client.

CallFunction()

Calls the function via RPC.

Close()

Closes the RPC communication channel to the server.

Open(hostname, port)

Opens a RPC communication channel to the server.

Parameters:

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns:

True if the communication channel was established.

Return type:

bool

__init__()

Initializes a RPC client.

plaso.multi_process.rpc module

The RPC client and server interface.

class plaso.multi_process.rpc.RPCClient

Bases: object

RPC client interface.

abstractmethod CallFunction()

Calls the function via RPC.

abstractmethod Close()

Closes the RPC communication channel to the server.

abstractmethod Open(hostname, port)

Opens a RPC communication channel to the server.

Parameters:

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns:

True if the communication channel was established.

Return type:

bool

class plaso.multi_process.rpc.RPCServer(callback)

Bases: object

RPC server interface.

abstractmethod Start(hostname, port)

Starts the RPC server.

Parameters:

• hostname (str) – hostname or IP address to connect to for requests.

• port (int) – port to connect to for requests.

Returns:

True if the RPC server was successfully started.

Return type:

bool

abstractmethod Stop()

Stops the RPC server.

__init__(callback)

Initializes the RPC server object.

Parameters:

callback (function) – callback to invoke on get status RPC request.

plaso.multi_process.task_engine module

The task-based multi-process processing engine.

class plaso.multi_process.task_engine.TaskMultiProcessEngine

Bases: MultiProcessEngine

Task-based multi-process engine base.

This class contains functionality to: * manage task storage used to store task results.

__init__()

Initializes a task-based multi-process engine.

plaso.multi_process.task_manager module

The task manager.

class plaso.multi_process.task_manager.TaskManager

Bases: object

Manages tasks and tracks their completion and status.

A task being tracked by the manager must be in exactly one of the following states:

• abandoned: a task assumed to be abandoned because a tasks that has been

  queued or was processing exceeds the maximum inactive time.

• merging: a task that is being merged by the engine.

• pending_merge: the task has been processed and is ready to be merged with

  the session storage.

• processed: a worker has completed processing the task, but it is not ready

  to be merged into the session storage.

• processing: a worker is processing the task.

• queued: the task is waiting for a worker to start processing it. It is also

  possible that a worker has already completed the task, but no status update was collected from the worker while it processed the task.

Once the engine reports that a task is completely merged, it is removed from the task manager.

Tasks are considered “pending” when there is more work that needs to be done to complete these tasks. Pending applies to tasks that are: * not abandoned; * abandoned, but need to be retried.

Abandoned tasks without corresponding retry tasks are considered “failed” when the foreman is done processing.

CheckTaskToMerge(task)

Checks if the task should be merged.

Parameters:

task (Task) – task.

Returns:

True if the task should be merged.

Return type:

bool

Raises:

KeyError – if the task was not queued, processing or abandoned.

CompleteTask(task)

Completes a task.

The task is complete and can be removed from the task manager.

Parameters:

task (Task) – task.

Raises:

KeyError – if the task was not merging.

CreateRetryTask()

Creates a task that to retry a previously abandoned task.

Returns:

a task that was abandoned but should be retried or None if there are

no abandoned tasks that should be retried.

Return type:

Task

CreateTask(session_identifier, storage_format='sqlite')

Creates a task.

Parameters:

• session_identifier (str) – the identifier of the session the task is part of.

• storage_format (Optional[str]) – the storage format that the task should be stored in.

Returns:

task attribute container.

Return type:

Task

GetFailedTasks()

Retrieves all failed tasks.

Failed tasks are tasks that were abandoned and have no retry task once the foreman is done processing.

Returns:

tasks.

Return type:

list[Task]

GetProcessedTaskByIdentifier(task_identifier)

Retrieves a task that has been processed.

Parameters:

task_identifier (str) – unique identifier of the task.

Returns:

a task that has been processed.

Return type:

Task

Raises:

KeyError – if the task was not processing, queued or abandoned.

GetStatusInformation()

Retrieves status information about the tasks.

Returns:

tasks status information.

Return type:

TasksStatus

GetTaskPendingMerge(current_task)

Retrieves the first task that is pending merge or has a higher priority.

This function will check if there is a task with a higher merge priority than the current_task being merged. If so, that task with the higher priority is returned.

Parameters:

current_task (Task) – current task being merged or None if no such task.

Returns:

the next task to merge or None if there is no task pending merge or

with a higher priority.

Return type:

Task

HasPendingTasks()

Determines if there are tasks running or in need of retrying.

Returns:

True if there are tasks that are active, ready to be merged or

need to be retried.

Return type:

bool

RemoveTask(task)

Removes an abandoned task.

Parameters:

task (Task) – task.

Raises:

KeyError – if the task was not abandoned or the task was abandoned and was not retried.

SampleTaskStatus(task, status)

Takes a sample of the status of the task for profiling.

Parameters:

• task (Task) – a task.

• status (str) – status.

StartProfiling(configuration, identifier)

Starts profiling.

Parameters:

• configuration (ProfilingConfiguration) – profiling configuration.

• identifier (str) – identifier of the profiling session used to create the sample filename.

StopProfiling()

Stops profiling.

UpdateTaskAsPendingMerge(task)

Updates the task manager to reflect that the task is ready to be merged.

Parameters:

task (Task) – task.

Raises:

KeyError – if the task was not queued, processing or abandoned, or the task was abandoned and has a retry task.

UpdateTaskAsProcessingByIdentifier(task_identifier)

Updates the task manager to reflect the task is processing.

Parameters:

task_identifier (str) – unique identifier of the task.

Raises:

KeyError – if the task is not known to the task manager.

__init__()

Initializes a task manager.

plaso.multi_process.task_process module

Base class for a process that handles tasks used in multi-processing.

class plaso.multi_process.task_process.MultiProcessTaskProcess(processing_configuration, enable_sigsegv_handler=False, **kwargs)

Bases: MultiProcessBaseProcess

Interface for multi-processing process that handles tasks.

__init__(processing_configuration, enable_sigsegv_handler=False, **kwargs)

Initializes a process.

Parameters:

• processing_configuration (ProcessingConfiguration) – processing configuration.

• enable_sigsegv_handler (Optional[bool]) – True if the SIGSEGV handler should be enabled.

• kwargs (dict[str,object]) – keyword arguments to pass to multiprocessing.Process.

plaso.multi_process.zeromq_queue module

ZeroMQ implementations of the Plaso queue interface.

class plaso.multi_process.zeromq_queue.ZeroMQBufferedQueue(buffer_timeout_seconds=2, buffer_max_size=10000, delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for buffered Plaso queues.

Buffered queues use a regular Python queue to store items that are pushed or popped from the queue without blocking on underlying ZeroMQ operations.

This class should not be instantiated directly, a subclass should be instantiated instead.

Close(abort=False)

Closes the queue.

Parameters:

abort (Optional[bool]) – whether the Close is the result of an abort condition. If True, queue contents may be lost.

Raises:

• QueueAlreadyClosed – if the queue is not started, or has already been closed.

• RuntimeError – if closed or terminate event is missing.

Empty()

Removes all items from the internal buffer.

__init__(buffer_timeout_seconds=2, buffer_max_size=10000, delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Initializes a buffered, ZeroMQ backed queue.

Parameters:

• buffer_max_size (Optional[int]) – maximum number of items to store in the buffer, before or after they are sent/received via ZeroMQ.

• buffer_timeout_seconds (Optional[int]) – number of seconds to wait when doing a put or get to/from the internal buffer.

• delay_open (Optional[bool]) – whether a ZeroMQ socket should be created the first time the queue is pushed to or popped from, rather than at queue object initialization. This is useful if a queue needs to be passed to a child process from a parent process.

• linger_seconds (Optional[int]) – number of seconds that the underlying ZeroMQ socket can remain open after the queue object has been closed, to allow queued items to be transferred to other ZeroMQ sockets.

• maximum_items (Optional[int]) – maximum number of items to queue on the ZeroMQ socket. ZeroMQ refers to this value as “high water mark” or “hwm”. Note that this limit only applies at one “end” of the queue. The default of 1000 is the ZeroMQ default value.

• name (Optional[str]) – name to identify the queue.

• port (Optional[int]) – The TCP port to use for the queue. None indicates that the queue should choose a random port to bind to.

• timeout_seconds (Optional[int]) – number of seconds that calls to PopItem and PushItem may block for, before returning queue.QueueEmpty.

class plaso.multi_process.zeromq_queue.ZeroMQBufferedReplyBindQueue(buffer_timeout_seconds=2, buffer_max_size=10000, delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQBufferedReplyQueue

A Plaso queue backed by a ZeroMQ REP socket that binds to a port.

This queue may only be used to pop items, not to push.

SOCKET_CONNECTION_TYPE = 1

class plaso.multi_process.zeromq_queue.ZeroMQBufferedReplyQueue(buffer_timeout_seconds=2, buffer_max_size=10000, delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQBufferedQueue

Parent class for buffered Plaso queues backed by ZeroMQ REP sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to push items, not to pop.

PopItem()

Pops an item of the queue.

Provided for compatibility with the API, but doesn’t actually work.

Raises:

WrongQueueType – As Pop is not supported by this queue.

PushItem(item, block=True)

Push an item on to the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Parameters:

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises:

• QueueAlreadyClosed – if the queue is closed.

• QueueFull – if the internal buffer was full and it was not possible to push the item to the buffer within the timeout.

• RuntimeError – if closed event is missing.

class plaso.multi_process.zeromq_queue.ZeroMQPullConnectQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQPullQueue

A Plaso queue backed by a ZeroMQ PULL socket that connects to a port.

This queue may only be used to pop items, not to push.

SOCKET_CONNECTION_TYPE = 2

class plaso.multi_process.zeromq_queue.ZeroMQPullQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for Plaso queues backed by ZeroMQ PULL sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to pop items, not to push.

PopItem()

Pops an item off the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Returns:

item from the queue.

Return type:

object

Raises:

• KeyboardInterrupt – if the process is sent a KeyboardInterrupt while popping an item.

• QueueEmpty – if the queue is empty, and no item could be popped within the queue timeout.

• RuntimeError – if closed or terminate event is missing.

• zmq.error.ZMQError – if a ZeroMQ error occurs.

PushItem(item, block=True)

Pushes an item on to the queue.

Provided for compatibility with the API, but doesn’t actually work.

Parameters:

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises:

WrongQueueType – As Push is not supported this queue.

class plaso.multi_process.zeromq_queue.ZeroMQPushBindQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQPushQueue

A Plaso queue backed by a ZeroMQ PUSH socket that binds to a port.

This queue may only be used to push items, not to pop.

SOCKET_CONNECTION_TYPE = 1

class plaso.multi_process.zeromq_queue.ZeroMQPushQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for Plaso queues backed by ZeroMQ PUSH sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to push items, not to pop.

PopItem()

Pops an item of the queue.

Provided for compatibility with the API, but doesn’t actually work.

Raises:

WrongQueueType – As Pull is not supported this queue.

PushItem(item, block=True)

Push an item on to the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Parameters:

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises:

• KeyboardInterrupt – if the process is sent a KeyboardInterrupt while pushing an item.

• QueueFull – if it was not possible to push the item to the queue within the timeout.

• RuntimeError – if terminate event is missing.

• zmq.error.ZMQError – if a ZeroMQ specific error occurs.

class plaso.multi_process.zeromq_queue.ZeroMQQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: Queue

Interface for a ZeroMQ backed queue.

name

name to identify the queue.

Type:

str

port

TCP port that the queue is connected or bound to. If the queue is not yet bound or connected to a port, this value will be None.

Type:

int

timeout_seconds

number of seconds that calls to PopItem and PushItem may block for, before returning queue.QueueEmpty.

Type:

int

Close(abort=False)

Closes the queue.

Parameters:

abort (Optional[bool]) – whether the Close is the result of an abort condition. If True, queue contents may be lost.

Raises:

• QueueAlreadyClosed – if the queue is not started, or has already been closed.

• RuntimeError – if closed or terminate event is missing.

IsBound()

Checks if the queue is bound to a port.

IsConnected()

Checks if the queue is connected to a port.

IsEmpty()

Checks if the queue is empty.

ZeroMQ queues don’t have a concept of “empty” - there could always be messages on the queue that a producer or consumer is unaware of. Thus, the queue is never empty, so we return False. Note that it is possible that a queue is unable to pop an item from a queue within a timeout, which will cause PopItem to raise a QueueEmpty exception, but this is a different condition.

Returns:

False, to indicate the the queue isn’t empty.

Return type:

bool

Open()

Opens this queue, causing the creation of a ZeroMQ socket.

Raises:

QueueAlreadyStarted – if the queue is already started, and a socket already exists.

abstractmethod PopItem()

Pops an item off the queue.

Returns:

item from the queue.

Return type:

object

Raises:

QueueEmpty – if the queue is empty, and no item could be popped within the queue timeout.

abstractmethod PushItem(item, block=True)

Pushes an item on to the queue.

Parameters:

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises:

QueueAlreadyClosed – if the queue is closed.

SOCKET_CONNECTION_BIND = 1

SOCKET_CONNECTION_CONNECT = 2

SOCKET_CONNECTION_TYPE = None

__init__(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Initializes a ZeroMQ backed queue.

Parameters:

• delay_open (Optional[bool]) – whether a ZeroMQ socket should be created the first time the queue is pushed to or popped from, rather than at queue object initialization. This is useful if a queue needs to be passed to a child process from a parent process.

• linger_seconds (Optional[int]) – number of seconds that the underlying ZeroMQ socket can remain open after the queue has been closed, to allow queued items to be transferred to other ZeroMQ sockets.

• maximum_items (Optional[int]) – maximum number of items to queue on the ZeroMQ socket. ZeroMQ refers to this value as “high water mark” or “hwm”. Note that this limit only applies at one “end” of the queue. The default of 1000 is the ZeroMQ default value.

• name (Optional[str]) – Optional name to identify the queue.

• port (Optional[int]) – The TCP port to use for the queue. The default is None, which indicates that the queue should choose a random port to bind to.

• timeout_seconds (Optional[int]) – number of seconds that calls to PopItem and PushItem may block for, before returning queue.QueueEmpty.

Raises:

ValueError – if the queue is configured to connect to an endpoint, but no port is specified.

class plaso.multi_process.zeromq_queue.ZeroMQRequestConnectQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQRequestQueue

A Plaso queue backed by a ZeroMQ REQ socket that connects to a port.

This queue may only be used to pop items, not to push.

SOCKET_CONNECTION_TYPE = 2

class plaso.multi_process.zeromq_queue.ZeroMQRequestQueue(delay_open=True, linger_seconds=10, maximum_items=1000, name='Unnamed', port=None, timeout_seconds=5)

Bases: ZeroMQQueue

Parent class for Plaso queues backed by ZeroMQ REQ sockets.

This class should not be instantiated directly, a subclass should be instantiated instead.

Instances of this class or subclasses may only be used to pop items, not to push.

PopItem()

Pops an item off the queue.

If no ZeroMQ socket has been created, one will be created the first time this method is called.

Returns:

item from the queue.

Return type:

object

Raises:

• KeyboardInterrupt – if the process is sent a KeyboardInterrupt while popping an item.

• QueueEmpty – if the queue is empty, and no item could be popped within the queue timeout.

• RuntimeError – if terminate event is missing.

• zmq.error.ZMQError – if an error occurs in ZeroMQ.

PushItem(item, block=True)

Pushes an item on to the queue.

Provided for compatibility with the API, but doesn’t actually work.

Parameters:

• item (object) – item to push on the queue.

• block (Optional[bool]) – whether the push should be performed in blocking or non-blocking mode.

Raises:

WrongQueueType – As Push is not supported this queue.

Module contents