tcex.tcex_job module

TcEx Framework Job Module

class tcex.tcex_job.TcExJob(tcex)[source]

Bases: object

Job processing functionality.

Warning

This module will be deprecated in version 0.9.0.

Supports batch indicator adds and allows structured group adds.

_chunk_indicators()[source]

Split indicator list into smaller more manageable numbers.

_group_add(resource_type, resource_name, owner, data=None)[source]

Add a group to ThreatConnect.

Parameters:
  • resource_type (string) – String with resource [group] type.
  • resource_name (string) – The name of the group.
  • owner (string) – The name of the TC owner for the group add.
  • data (Optional [dictionary]) – Any optional group parameters, tags, or attributes.
Returns:

The ID of the created resource.

Return type:

(integer)

_group_delete_by_id(resource_type, resource_id)[source]

Delete a group from ThreatConnect.

Parameters:
  • resource_type (string) – String with resource [group] type.
  • resource_id (string) – The ID of the group.
_group_delete_by_name(owner, resource_type, resource_name)[source]

Delete group(s) from ThreatConnect by Name.

Parameters:
  • resource_type (string) – String with resource [group] type.
  • resource_name (string) – The name of the group.
_indicators_replace_stub(owner)[source]

Process Indicator data replacing associatedGroup stub values with the Group ID.

Format of a stub ${<group name>:<group type>}.

Parameters:
  • indicators (list) – Batch Indicator Data.
  • owner (string) – The owner name for the indicator to be written.
_process_association(owner)[source]

Process groups and write to TC API.

Parameters:owner (string) – The owner name for the indicator to be written.
_process_file_occurrences(owner)[source]

Process file occurrences and write to TC API.

Parameters:owner (string) – The owner name for the indicator to be written.
_process_group_association(owner)[source]

Process group associations and write to ThreatConnect API.

Parameters:owner (string) – The owner name for the associations to be written.
_process_groups(owner, group_action)[source]

Process groups and write to ThreatConnect API.

Parameters:owner (string) – The owner name for the groups to be written.
_process_indicators(owner, batch)[source]

Process batch indicators and write to ThreatConnect API.

Parameters:
  • owner (string) – The owner name for the indicator to be written.
  • batch (boolean) – Use the batch API to create indicators.
_process_indicators_batch(owner)[source]

Process batch indicators and write to ThreatConnect API.

Note

Failed attributes and/or tags will not cause a batch import to fail.

Parameters:owner (string) – The owner name for the indicator to be written.
_process_indicators_v2(owner)[source]

Process batch indicators and write to ThreatConnect API using /v2/indicators.

Parameters:owner (string) – The owner name for the indicator to be written.
association(associations)[source]

Add association data to TcEx job.

This method will add association data to the association list.

Warning

There is no validation of the data passed to this method.

Example Data (required fields are highlighted)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
[{
  'association_value': '1.1.1.1',
  'association_type': 'Address',
  'resource_value': 'adversary-002',
  'resource_type': 'Adversary'
},{
  'association_value': 'adversary-001',
  'association_type': 'Adversary',
  'resource_value': '1.1.1.1',
  'resource_type': 'Address'
},{
  'association_value': 'adversary-001',
  'association_type': 'Adversary',
  'resource_value': 'threat-001',
  'resource_type': 'Threat',
}]

Example Data for Creating Indicator-to-Indicator Associations (required fields are highlighted)

1
2
3
4
5
6
7
{
  "association_value": "ASN1234",
  "association_type": tcex.safe_rt("ASN"),
  "resource_value": "1.2.3.4",
  "resource_type": "Address",
  "custom_association_name": "ASN to Address"
}
Parameters:associations (dict | list) – Dictionary or List containing association data
batch_action(action)[source]

Set the default batch action for argument parser.

Parameters:action (string) – Set batch job action.
batch_chunk(chunk_size)[source]

Set batch chunk_size for argument parser.

Parameters:chunk_size (integer) – Set batch job chunk size.
batch_halt_on_error(halt_on_error)[source]

Set batch halt on error boolean for argument parser.

Parameters:halt_on_error (boolean) – Boolean value for halt on error.
batch_indicator_success()[source]

Check completion for all batch jobs associated with this instance.

Iterate over self._indicator_batch_ids set in _process_indicators() and return the status of all job.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "status": "Success",
  "data": {
    "batchStatus": {
      "id": 392,
      "status": "Completed",
      "errorCount": 1,
      "successCount": 4,
      "unprocessCount": 0
    }
  }
}
Returns:The status results from the Batch jobs.
Return type:(dictionary)
batch_poll_interval(interval)[source]

Set batch polling interval for argument parser.

Parameters:interval (integer) – Seconds between polling.
batch_poll_interval_max(interval_max)[source]

Set batch polling interval max for argument parser.

Parameters:interval_max (integer) – Max seconds before timeout on batch.
batch_status(batch_id)[source]

Check the status of a batch job.

This method will get the status of a batch job. Any errors returned from the batch status will be automatically logged.

Critical errors are defined in the __init__ method in the self._batch_failures dictionary. If any of the critical errors are found the execution of the App will halt with a exit code of 1. All other errors will set the status code to 3, but will not cause the execution to halt.

Parameters:batch_id (int) – Id of the batch job.
Returns:A dictionary with status and error boolean.
Return type:(dict)
batch_write_type(write_type)[source]

Set batch attributes write type for argument parser.

Parameters:write_type (string) – Type of Append or Replace.
file_occurrence(fo)[source]

Add a file occurrence to job.

Parameters:fo (dictionary) – The file occurrence data.

Warning

There is no validation of the data passed to this method.

Example Data (required fields are highlighted)

1
2
3
4
5
6
{
    "date" : "2014-11-03T00:00:00-05:00",
    "fileName" : "win999301.dll",
    "hash": "BE7DE2F0CF48294400C714C9E28ECD01",
    "path" : "C:\Windows\System"
}

Note

The hash in the example above is not posted in the body, but extracted to use in the URI.

group(group)[source]

Add group data to TcEx job.

This method will accept a list or dictionary of formatted Group data and submit it to the API when the process() method is called.

Warning

There is no validation of the data passed to this method. Any duplicate group name will be skipped.

Example Data (required fields are highlighted)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
{
  'attribute': [
    {
      'type': 'Description',
      'value': 'Test Description'
    }
  ],
  'eventDate': '2015-03-7T00:00:00Z'
  'name': 'adversary-001',
  'tag': [{
    'name': 'Pop Star'
  }],
  'type': 'Adversary'
}
Parameters:group (dict | list) – Dictionary or List containing group data.
group_association(associations)[source]

Add group association data to TcEx job.

This method will add group association data to the group association list.

Warning

There is no validation of the data passed to this method.

Example Data (required fields are highlighted)

1
2
3
4
5
6
{
  'group_name': 'adversary-001',
  'group_type': 'Adversary',
  'indicator': '1.1.1.1',
  'indicator_type': 'Address'
}
Parameters:associations (dict | list) – Dictionary or List containing group association data.
group_association_len

The current length of the group association list.

Returns:The length of the group association list.
Return type:(integer)
group_cache(owner, resource_type)[source]

Cache group data from the ThreatConnect Platform.

The method will cache ThreatConnect group data by owner and type.

Cache Structure

Owner -> Group Type -> Group Name:Group Id
Parameters:
  • owner (string) – The name of the ThreatConnect owner
  • resource_type (string) – The resource type name
Returns:

Dictionary of group resources

Return type:

(dictionary)

group_cache_add(name, owner, resource_type, resource_id)[source]

Add a group to the group cache.

Parameters:
  • name (string) – The name of the Group.
  • owner (string) – The TC Owner where the resource should be found.
  • resource_type (string) – The resource type name.
Returns:

The ID for the provided group name and owner.

Return type:

(integer)

group_cache_remove(name, owner, resource_type)[source]

Remove a group from the group cache.

Parameters:
  • name (string) – The name of the Group.
  • owner (string) – The TC Owner where the resource should be found.
  • resource_type (string) – The resource type name.
Returns:

The ID for the provided group name and owner.

Return type:

(integer)

group_cache_type(group_id, owner)[source]

Get the group type for the provided group id.

Cache Structure

Owner -> Group Id:Group Type
Parameters:
  • group_id (string) – The group id to lookup.
  • owner (string) – The TC Owner where the resource should be found.
  • resource_type (string) – The resource type name.
Returns:

The ID for the provided group name and owner.

Return type:

(integer)

group_id(name, owner, resource_type)[source]

Get the group id from the group cache.

Parameters:
  • name (string) – The name of the Group.
  • owner (string) – The TC Owner where the resource should be found.
  • resource_type (string) – The resource type name.
Returns:

The ID for the provided group name and owner.

Return type:

(integer)

group_len

The current length of the group list.

Returns:The length of the group list.
Return type:(integer)
group_results

Result dictionary of failed, saved, not_saved, and submitted groups.

Returns:Dictionary of group names for each status.
Return type:(dictionary)
indicator(indicator)[source]

Add indicator data to TcEx job.

This method will add indicator data to this TcEx job to be submitted via batch import.

Warning

There is no validation of the data passed to this method. Any duplicate indicator will be skipped.

Example Data (required fields are highlighted)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
  'associatedGroup': [
    '1',
    '8'
  ],
  'attribute': [
    {
      'type': 'Attribute Name',
      'value': 'Description'
    }
  ],
  'confidence': 5,
  'rating': 3.2,
  'summary': '1.1.1.1',
  'tag': [
    {
      'name': 'APT',
    },
    {
      'name': 'CrimeWare',
    }
  ],
  'type': 'Address'
}
Parameters:indicator (dict | list) – Dictionary or List containing indicator data.
indicator_data

Return the current indicator list.

Returns:The indicator list.
Return type:(list)
indicator_len

The current length of the indicator list.

Returns:The length of the indicator list
Return type:(integer)
indicator_results

Result dictionary of failed, saved, not_saved, and submitted indicators.

Returns:Dictionary of indicator values for each status.
Return type:(dictionary)
process(owner, indicator_batch=True, group_action='skip')[source]

Process all groups, indicator data, and associations.

Process each of the supported data types for this job, in the following order (left to right):

groups > indicators > file occurrences > group associations > associations
Parameters:
  • owner (string) – The owner name for the data to be written.
  • indicator_batch (bool) – If true use the Batch Api otherwise use /v2 REST API.
  • group_action (string) – The action to use on group create (duplicate, replace, skip).
unprocessed_indicators

Return indicators (unprocessed).

Returns:The list of unprocessed indicator.
Return type:(list)