Batch API
Overview
The Batch API allows you to create and modify threat intelligence data in bulk. Currently, there are two versions of the Batch API: V1 and V2. The V1 Batch API supports Indicators only, whereas the V2 Batch API supports Indicators and Groups. Using the Batch API will improve performance when importing large amounts of data.
After you create a batch job, you can upload a JSON file containing the objects to be created or modified to the Batch API and trigger the batch job that will process the data in the file. The contents of the uploaded file must contain valid JSON and mimic the data structure of a JSON bulk report file.
Prerequisites
To use the Batch API, the following prerequisites must be met:
Batch API must be enabled on your ThreatConnect instance
Document storage must be enabled on your ThreatConnect instance
To create batch jobs in and upload data to an Organization, your API user account must have an Organization role of Standard User or higher
To create batch jobs in and upload data to a Community or Source, your API user account must have a Community role of Editor or higher
V2 Batch API
The following sections describe how to use the V2 Batch API, including how to create a batch job and upload a file to the job for processing. With the V2 Batch API, you can perform the following actions:
Import Indicators and Groups
Create Group-to-Group, Indicator-to-Group, Group-to-Indicator, and, Indicator-to-Indicator associations
Create and update Attributes, Tags, and Security Labels for Indicators and Groups
Specify how to manage file hash merges and collisions for File Indicators
Attention
The V2 Batch API does not support cross-owner associations.
Note
To create Indicator-to-Indicator associations with the V2 Batch API, your instance must be running ThreatConnect 7.8.2 or newer.
Create a Batch Job
Send a request in the following format to create a batch job that will use the V2 Batch API.
Request
POST /v2/batch
Content-type: application/json; charset=utf-8
{
"version": "V2",
"owner": "Demo Organization",
"haltOnError": true,
"action": "Create",
"attributeWriteType": "Append",
"tagWriteType": "Append",
"securityLabelWriteType": "Append",
"fileMergeMode": "Merge",
"hashCollisionMode": "FavorIncoming"
}
version: <String> The version of the Batch API to use. Set this field’s value toV2to use the V2 Batch API.owner: <String> REQUIRED The name of the Organization, Community, or Source in which the batch job will create or update data.haltOnError: <Boolean> REQUIRED Specifies how the batch process should proceed if it encounters an error. Accepted values include the following:true: The batch process will stop processing the entire batch job the first time it encounters an error.
false: The batch process will attempt to continue processing the batch job after encountering an error.
action: <String> REQUIRED The action that the batch job will perform on incoming data. Accepted values include the following:Create: The batch job will create new and update existing data in the specified owner.
Delete: The batch job will delete existing data in the specified owner that matches any of the incoming data.
attributeWriteType: <String> REQUIRED Specifies how the batch job will handle incoming Attributes. Accepted values include the following:Append: The batch job will append the incoming Attributes to an Indicator or Group. Note that duplicate Attributes may be added, as redundancy checking is not performed.
Replace: The batch job will replace all Attributes added to an Indicator or Group with the incoming Attributes.
Singleton: The batch job will replace existing Attributes added to an Indicator or Group only if the incoming data include Attributes with the same Attribute Type(s) as the existing Attributes. Otherwise, existing Attributes added to an Indicator or Group will remain unchanged.
Static: The batch job will ignore incoming Attributes and not update existing Attributes added to an Indicator or Group.
tagWriteType: <String> Specifies how the batch job will handle incoming Tags. Accepted values include the following:Append: The batch job will append the incoming Tags to an Indicator or Group.
Replace (default): The batch job will replace all Tags added to an Indicator or Group with the incoming Tags.
securityLabelWriteType: <String> Specifies how the batch job will handle incoming Security Labels. Accepted values include the following:Append: The batch job will append the incoming Security Labels to an Indicator or Group.
Replace (default): The batch job will replace all Security Labels added to an Indicator or Group with the incoming Security Labels.
fileMergeMode: <String> Specifies whether the batch job will perform a merge operation when the file hashes in an incoming File Indicator match two or more existing File Indicators in the specified owner (e.g., an incoming File Indicator contains an MD5 and SHA1 that each match a separate File Indicator in the specified owner). Accepted values include the following:Distribute: The batch job will not perform a merge operation; instead, it will add the metadata (e.g., Threat and Confidence Ratings, Tags, Attributes) for the incoming File Indicator to all matching File Indicators (up to three possible) in the specified owner.
Merge (default): The batch job will perform a merge operation and combine two or more existing File Indicators with separate file hashes into a single File Indicator with all file hashes. During this process, the File Indicator with the most recent
lastModifieddate will be used as the “primary” copy and retain its Threat and Confidence Ratings (at least, until their values are overwritten by the values for the incoming File Indicator). Attributes, Security Labels, and Tags associated with all merged File Indicators will be applied to the “primary” copy as well. Because Attributes are appended to the “primary” copy, duplicate Attributes may be present if each File Indicator that was merged had similar Attributes.
hashCollisionMode: <String> Specifies how the batch job will handle file hash collisions that occur when importing or merging File Indicators (e.g., an incoming File Indicator contains an MD5 and SHA1, while an existing File Indicator contains the same MD5 but a different SHA1). Accepted values include the following:FavorExisting: The batch job will favor hashes in the existing File Indicator and ignore conflicting hashes in the incoming File Indicator.
FavorIncoming (default): The batch job will favor hashes in the incoming File Indicator and overwrite conflicting hashes in the existing File Indicator.
IgnoreExisting: If a conflict exists between two or more existing File Indicators, the batch job will delete the existing File Indicators that caused conflict and then import the incoming File Indicator. If a conflict exists between one or more existing File Indicators and an incoming File Indicator, the batch job will ignore the existing Indicators. It will also remove their file hashes from the incoming Indicator, resulting in a new File Indicator being created without any conflicting file hashes. If all available file hashes are removed from the incoming Indicator, no new Indicator will be created.
IgnoreIncoming: The batch job will not import the incoming File Indicator causing the conflict, and existing File Indicators will not be updated.
Split: The batch job will not perform a merge; instead, it will apply the metadata (e.g., Threat and Confidence Ratings, Tags, Attributes) for the incoming File Indicator to all matching File Indicators in the specified owner.
Note
If haltOnError is set to true and an error occurs during the batch process, the status for the batch job will be set to Completed and the value for errorCount will be greater than zero. Also, the value for unprocessedCount will be greater than zero unless the uploaded file did not contain valid JSON.
Response (Success)
HTTP/1.1 201 Created
{
"status": "Success",
"data": {
"batchId": 2446
}
}
Response (Insufficient Privileges)
HTTP/1.1 401 Unauthorized
Unable to perform the requested operation due to the following error(s): You do not have permission to create Indicators; Groups; Attributes; Tags; Security Labels;
Response (Incorrect Settings)
HTTP/1.1 503 Service Unavailable
Batch job api is not available. Ensure that batchJobEnabled is true and document storage is enabled and configured;
Attention
The parameters defined in the body of a POST request to create a batch job must be accurate. When troubleshooting issues with the Batch API, make sure the parameter names in the request body are correct and that an accepted value is provided for each parameter.
Upload an Input File to a Batch Job
The V2 Batch API expects to ingest a JSON file containing one or more lists of objects, where Indicator and Group objects are contained within the indicator and group arrays. If your instance is running ThreatConnect 7.8.2 or newer, you may also define associations to create within the association array.
Note
You can also define associations within the indicator and group arrays, regardless of which ThreatConnect version your instance is running. For more information, see the “Indicator Fields” and “Group Fields” sections.
The following is the batch input file schema for the V2 Batch API. For a list of additional fields you can use when defining Indicator and Group objects, see the “Indicator Fields” and “Group Fields” section, respectively.
indicator: <Array of Objects> The Indicators to create or update.summary: <String> REQUIRED The Indicator’s summary. Note that you may substitutesummarywith the field that contains the Indicator’s value for the specified Indicator type (e.g., you can useipinstead ofsummaryto define an Address Indicator’s IPv4 or IPv6 address value).type: <String> REQUIRED The Indicator’s type.
group: <Array of Objects> The Groups to create or update.name: <String> REQUIRED The Group’s name.type: <String> REQUIRED The Group’s type.xid: <String> REQUIRED The Group’s XID.
association: <Array of Objects> The associations to create. Supported association types include Indicator to Group, Group to Indicator, Group to Group, and Indicator to Indicator. (Requires ThreatConnect 7.8.2 or newer.)ref_1: <String> The summary of the Indicator or XID of the Group to use as the first object in the association. The Indicator or Group must be present in the batch input file or exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction withid_1. If both are provided,id_1will be favored overref_1.)id_1: <Integer> The ID of the Indicator or Group to use as the first object in the association. The Indicator or Group must exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction withref_1. If both are provided,id_1will be favored overref_1.)type_1: <String> The type of Indicator or Group defined forref_1orid_1. This field is required for Indicators and optional for Groups.ref_2: <String> The summary of the Indicator or XID of the Group to use as the second object in the association. The Indicator or Group must be present in the input file or exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction withid_2. If both are provided,id_2will be favored overref_2.)id_2: <Integer> The ID of the Indicator or Group to use as the second object in the association. The Indicator or Group must exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction withref_2. If both are provided,id_2will be favored overref_2.)type_2: <String> The type of Indicator or Group defined forref_2orid_2. This field is required for Indicators and optional for Groups.associationType: <String> For Indicator-to-Indicator associations only, the name of the association type (e.g., URL Host). This field is required if making an Indicator-to-Indicator association.
Attention
If using the association field, it must be placed after the indicator and group fields in a batch input file.
Indicator Fields
Field |
Description |
Accepted Data Type |
Required? |
|---|---|---|---|
active |
Specifies whether the Indicator’s status is active (true) or inactive (false)
Default value: true
|
Boolean |
Optional |
activeLocked |
Specifies whether to prevent CAL™ from updating the Indicator’s status
Default value: false
|
Boolean |
Optional |
address |
A valid email address (for Email Address Indicators only) |
String |
Required for Email Address Indicators if |
AS Number |
An autonomous system (AS) number that uniquely identifies each network on the Internet (for ASN Indicators only) |
String |
Required for ASN Indicators if |
associatedGroups |
An array of Groups associated to the Indicator |
Array of Objects |
Optional |
associatedGroups.groupXid |
The associated Group’s XID |
String |
Required if |
associatedIndicators |
An array of Indicators associated to the Indicator |
Array of Objects |
Optional |
associatedIndicators.indicatorType |
The associated Indicator’s type |
String |
Required if |
associatedIndicators.summary |
The associated Indicator’s summary |
String |
Required if |
attribute |
An array of Attributes added to the Indicator |
Array of Objects |
Optional |
attribute.displayed |
Specifies whether to display the Attribute as a default Attribute on the Details screen for the Indicator to which it is added (applies to Description and Source Attributes only)
Default value: false
|
Boolean |
Optional |
attribute.pinned |
Specifies whether to display the Attribute as a pinned Attribute on the Details screen for the Indicator to which it is added
Default value: false
|
Boolean |
Optional |
attribute.securityLabel |
An array of Security Labels applied to the Attribute |
Array of Objects |
Optional |
attribute.securityLabel.color |
The hex color code (e.g., FF0000) for the Security Label’s color |
String |
Required if |
attribute.securityLabel.description |
The Security Label’s description |
String |
Required if |
attribute.securityLabel.name |
The Security Label’s name |
String |
Required if |
attribute.source |
The Attribute’s source |
String |
Optional |
attribute.type |
The Attribute’s Type |
String |
Required if |
attribute.value |
The Attribute’s value |
String |
Required if |
Block |
A block of network IP addresses (for CIDR Indicators only) |
String |
Required for CIDR Indicators if |
confidence |
The Indicator’s Confidence Rating
Minimum value: 0
Maximum value: 100
|
Integer |
Optional |
description |
The value of the Indicator’s default Description Attribute |
String |
Optional |
externalDateAdded |
The date and time when the Indicator was created externally |
DateTime |
Optional |
externalDateExpires |
The date and time when the Indicator expires externally |
DateTime |
Optional |
externalLastModified |
The date and time when the Indicator was last modified externally |
DateTime |
Optional |
fileOccurrence |
The Indicator’s File Occurrences (for File Indicators only) |
Object |
Optional |
fileOccurrence.date |
The date and time of the File Occurrence |
DateTime |
Optional [1] |
fileOccurrence.fileName |
The name of the file corresponding to the File Occurrence |
String |
Optional [1] |
fileOccurrence.path |
The run path for the file corresponding to the File Occurrence |
String |
Optional [1] |
firstSeen |
The date and time when the Indicator was first seen |
DateTime |
Optional |
Hashtag |
A hashtag term used in social media (for Hashtag Indicators only) |
String |
Required for Hashtag Indicators if |
hostName |
A valid hostname or domain (for Host Indicators only) |
String |
Required for Host Indicators if |
ip |
A valid IPv4 or IPv6 address (for Address Indicators only) |
String |
Required for Address Indicators if |
Key Name |
A node in a hierarchical database (i.e., key) that contains data critical for the operation of Windows and the applications and services that run on Windows (for Registry Key Indicators only) |
String |
Required for Registry Indicators if |
lastSeen |
The date and time when the Indicator was last seen |
DateTime |
Optional |
md5 |
The File Indicator’s MD5 file hash |
String |
If |
Mutex |
A synchronization primitive used to identify malware files and related malware families (for Mutex Indicators only) |
String |
Required for Mutex Indicators if |
privateFlag |
Specifies whether to mark the Indicator as private (requires private Indicators being enabled on for your ThreatConnect instance)
Default value: false
|
Boolean |
Optional |
rating |
The Indicator’s Threat Rating
Minimum value: 0
Maximum value: 5
|
Integer |
Optional |
securityLabel |
An array of Security Labels applied to the Indicator |
Array of Objects |
Optional |
securityLabel.color |
The hex color code (e.g., FF0000) for the Security Label’s color |
String |
Required if |
securityLabel.description |
The Security Label’s description |
String |
Required if |
securityLabel.name |
The Security Label’s name |
String |
Required if |
sha1 |
The File Indicator’s SHA1 file hash |
String |
If |
sha256 |
The File Indicator’s SHA256 file hash |
String |
If |
size |
The File Indicator’s file size |
Integer |
Optional |
Subject |
The subject line of an email (for Email Subject Indicators only) |
String |
Required for Email Subject Indicators if |
summary |
The Indicator’s summary |
String |
Required if the field that contains the Indicator’s value for the specified Indicator type (e.g., |
tag |
An array of Tags applied to the Indicator |
Array of Objects |
Optional |
tag.name |
The Tag’s name |
String |
Required if |
text |
A valid URL, including protocol (for URL Indicators only) |
String |
Required for URL Indicators if |
type |
The Indicator’s type |
String |
Required |
User Agent String |
A characteristic identification string that a software agent uses when operating in a network protocol (for User Agent Indicators only) |
String |
Required for User Agent Indicators if |
Value Name |
A registry value associated with the specified registry key (for Registry Key Indicators only) |
String |
Required for Registry Key Indicators if |
Value Type |
A registry key value type (for Registry Key Indicators only)
Acceptable values:
* REG_NONE
* REG_BINARY
* REG_DWORD
* REG_DWORD_LITTLE_ENDIAN
* REG_DWORD_BIG_ENDIAN
* REG_EXPAND_SZ
* REG_LINK
* REG_MULTI_SZ
* REG_QWORD
* REG_QWORD_LITTLE_ENDIAN
* REG_SZ
|
String |
Required for Registry Key Indicators if |
Group Fields
Field |
Description |
Accepted Data Type |
Required? |
|---|---|---|---|
associatedGroupXid |
An array of XIDs for Groups associated to the Group |
String Array |
Optional |
associatedIndicators |
An array of Indicators associated to the Group |
Array of Objects |
Optional |
associatedIndicators.indicatorType |
The associated Indicator’s type |
String |
Required if |
associatedIndicators.summary |
The associated Indicator’s summary |
String |
Required if |
attribute |
An array of Attributes added to the Group |
Array of Objects |
Optional |
attribute.displayed |
Specifies whether to display the Attribute as a default Attribute on the Details screen for the Group to which it is added (applies to Description and Source Attributes only)
Default value: false
|
Boolean |
Optional |
attribute.pinned |
Specifies whether to display the Attribute as a pinned Attribute on the Details screen for the Group to which it is added
Default value: false
|
Boolean |
Optional |
attribute.securityLabel |
An array of Security Labels applied to the Attribute |
Array of Objects |
Optional |
attribute.securityLabel.color |
The hex color code (e.g., FF0000) for the Security Label’s color |
String |
Required if |
attribute.securityLabel.description |
The Security Label’s description |
String |
Required if |
attribute.securityLabel.name |
The Security Label’s name |
String |
Required if |
attribute.source |
The Attribute’s source |
String |
Optional |
attribute.type |
The Attribute’s Type |
String |
Required if |
attribute.value |
The Attribute’s value |
String |
Required if |
body |
The body of the email (for Email Groups only) |
String |
Required for Email Groups |
eventDate |
The date and time when the security event or incident took place (for Event and Incident Groups only) |
DateTime |
Optional |
externalDateAdded |
The date and time when the Group was created externally |
DateTime |
Optional |
externalDateExpires |
The date and time when the Group expires externally |
DateTime |
Optional |
externalLastModified |
The date and time when the Group was last modified externally |
DateTime |
Optional |
fileName |
The name of the Group’s file (for Document, Report, and Signature Groups only) |
String |
Optional |
fileText |
The contents of the signature file (for Signature Groups only) |
String |
Required for Signature Groups |
fileType |
The signature file’s type (for Signature Groups only)
Acceptable values:
* Bro
* ClamAV
* CybOX
* Iris Search Hash
* KQL
* OpenIOC
* Regex
* SPL
* Sigma
* Snort
* Suricata
* TQL Query
* YARA
|
String |
Required for Signature Groups only |
firstSeen |
The date and time when the Group was first seen |
DateTime |
Optional |
from |
The sender’s email address for the email (for Email Groups only) |
String |
Optional |
header |
The header of the email (for Email Groups only) |
String |
Required for Email Groups |
insights |
An AI-generated synopsis of the Group (for Document and Report Groups only) |
String |
Optional |
lastSeen |
The date and time when the Group was last seen |
DateTime |
Optional |
malware |
Specifies whether the file uploaded to the Group is a malware file (for Document Groups only)
Default value: false
|
Boolean |
Optional |
name |
The Group’s name |
String |
Required |
password |
The password to access a malware file uploaded to the Group (for Document Groups only) |
String |
Required only if |
securityLabel |
An array of Security Labels applied to the Group |
Array of Objects |
Optional |
securityLabel.color |
The hex color code (e.g., FF0000) for the Security Label’s color |
String |
Required if |
securityLabel.description |
The Security Label’s description |
String |
Required if |
securityLabel.name |
The Security Label’s name |
String |
Required if |
status |
The status of the security event or incident (for Event and Incident Groups only)
Acceptable values (Event):
* Needs Review
* False Positive
* No Further Action
* Escalated
Acceptable values (Incident):
* New
* Open
* Stalled
* Containment Achieved
* Restoration Achieved
* Incident Reported
* Closed
* Rejected
* Deleted
|
String |
Optional |
subject |
The subject line of the email (for Email Groups only) |
String |
Required for Email Groups |
tag |
An array of Tags applied to the Group |
Array of Objects |
Optional |
tag.name |
The Tag’s name |
String |
Required if |
to |
The recipient’s email address (for Email Groups only) |
String |
Optional |
type |
The Group’s type |
String |
Required |
xid |
The Group’s XID |
String |
Required |
Hint
Exporting indicators via the JSON Bulk Reports endpoint will create a file in the proper format for the Batch API.
Attention
The maximum number of Indicators that can be created in one batch job is 25,000. If creating more than 25,000 Indicators, use multiple batch jobs.
Example Input Files
Indicator-to-Group Association
The following input file demonstrates how to associate Indicators to Groups using the associatedGroups field within the indicator array. When the Batch API processes the file, it will do the following:
Create a Host Indicator (badguyz.com)
Create an Incident Group (Ransomware Attack at Company ABC)
Associate badguyz.com to Ransomware Attack at Company ABC
{
"indicator": [
{
"summary": "badguyz.com",
"type": "Host",
"associatedGroups": [
{
"groupXid": "00000000-0000-0000-0000-000000000000:0001"
}
],
"attribute": [
{
"type": "Description",
"value": "This host was involved in a ransomware attack that targeted employees at Company ABC.",
"securityLabel": [
{
"name": "TLP:AMBER"
}
]
}
],
"confidence": 60,
"rating": 3,
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
],
"group": [
{
"name": "Ransomware Attack at Company ABC",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0001",
"attribute": [
{
"type": "Description",
"value": "A ransomware attack that targeted employees at Company ABC.",
"displayed": true
}
],
"eventDate": "2024-08-04T00:00:00Z",
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
]
}
The following input file demonstrates how to associate Indicators to Groups using the association array. When the Batch API processes the file, it will do the following:
Create a Host Indicator (badguyz.com)
Create an Incident Group (Ransomware Attack at Company ABC)
Associate badguyz.com to Ransomware Attack at Company ABC
Associate badguyz.com to an existing Group (the Group whose ID is 12345) in the owner in which the batch job is processing data
{
"indicator": [
{
"summary": "badguyz.com",
"type": "Host",
"attribute": [
{
"type": "Description",
"value": "This host was involved in a ransomware attack that targeted employees at Company ABC.",
"securityLabel": [
{
"name": "TLP:AMBER"
}
]
}
],
"confidence": 60,
"rating": 3,
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
],
"group": [
{
"name": "Ransomware Attack at Company ABC",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0001",
"attribute": [
{
"type": "Description",
"value": "A ransomware attack that targeted employees at Company ABC.",
"displayed": true
}
],
"eventDate": "2024-08-04T00:00:00Z",
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
],
"association": [
{
"ref_1": "badguyz.com",
"type_1": "Host",
"ref_2": "00000000-0000-0000-0000-000000000000:0001"
},
{
"ref_1": "badguyz.com",
"type_1": "Host",
"id_2": 12345
}
]
}
Group-to-Indicator Association
Note
When creating Group-to-Indicator associations, including the Indicators in the input file improves the efficiency of the batch job. Otherwise, a lookup will need to be made for each Indicator not included in the input file.
The following input file demonstrates how to associate Groups to newly created Indicators using the associatedIndicators field within the group array. When the Batch API processes the file, it will do the following:
Create a URL Indicator (http://www.badguyz.com)
Create an Incident Group (Ransomware Attack at Company XYZ)
Associate Ransomware Attack at Company XYZ to http://www.badguyz.com
{
"indicator": [
{
"summary": "http://www.badguyz.com",
"type": "URL",
"attribute": [
{
"type": "Description",
"value": "This URL was involved in a ransomware attack that targeted employees at Company XYZ.",
"securityLabel": [
{
"name": "TLP:AMBER"
}
]
}
],
"confidence": 60,
"rating": 3,
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
],
"group": [
{
"name": "Ransomware Attack at Company XYZ",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0002",
"associatedIndicators": [
{
"summary": "http://www.badguyz.com",
"indicatorType": "URL"
}
],
"attribute": [
{
"type": "Description",
"value": "A ransomware attack that targeted employees at Company XYZ.",
"displayed": true
}
],
"eventDate": "2024-08-04T00:00:00Z",
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
]
}
The following input file demonstrates how to associate Groups to existing Indicators using the associatedIndicators field within the group array. When the Batch API processes the file, it will do the following:
Create an Incident Group (Ransomware Attack at Company XYZ)
Associate Ransomware Attack at Company XYZ to an existing Address Indicator (71.6.135.131) in the owner in which the batch job is processing data
Because the Address Indicator already exists in ThreatConnect, it does not need to be defined in the indicator array in the input file.
{
"group": [
{
"name": "Ransomware Attack at Company XYZ",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0002",
"associatedIndicators": [
{
"summary": "71.6.135.131",
"indicatorType": "Address"
}
],
"attribute": [
{
"type": "Description",
"value": "A ransomware attack that targeted employees at Company XYZ.",
"displayed": true
}
],
"eventDate": "2024-08-04T00:00:00Z",
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
]
}
The following input file demonstrates how to associate Groups to Indicators using the association array. When the Batch API processes the file, it will do the following:
Create a URL Indicator (http://www.badguyz.com)
Create an Incident Group (Ransomware Attack at Company XYZ)
Associate Ransomware Attack at Company XYZ to http://www.badguyz.com
Associate Ransomware Attack at Company XYZ to an existing Address Indicator (the Address Indicator whose ID is 54321) in the owner in which the batch job is processing data
{
"indicator": [
{
"summary": "http://www.badguyz.com",
"type": "URL",
"attribute": [
{
"type": "Description",
"value": "This URL was involved in a ransomware attack that targeted employees at Company XYZ.",
"securityLabel": [
{
"name": "TLP:AMBER"
}
]
}
],
"confidence": 60,
"rating": 3,
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
],
"group": [
{
"name": "Ransomware Attack at Company XYZ",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0002",
"attribute": [
{
"type": "Description",
"value": "A ransomware attack that targeted employees at Company XYZ.",
"displayed": true
}
],
"eventDate": "2024-08-04T00:00:00Z",
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
],
"association": [
{
"ref_1": "00000000-0000-0000-0000-000000000000:0002",
"ref_2": "http://www.badguyz.com",
"type_2": "URL"
},
{
"ref_1": "00000000-0000-0000-0000-000000000000:0002",
"id_2": 54321,
"type_2": "Address"
}
]
}
Group-to-Group Association
The following input file demonstrates how to associate Groups to other Groups using the associatedGroupXid field within the group array. When the Batch API processes the file, it will do the following:
Create two Incident Groups (Compromised User Accounts at Company ABC and Leaked Credentials at Company ABC)
Associate Compromised User Accounts at Company ABC to Leaked Credentials at Company ABC
{
"group": [
{
"name": "Compromised User Accounts at Company ABC",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0003",
"associatedGroupXid": [
"00000000-0000-0000-0000-000000000000:0004"
],
"attribute": [
{
"type": "Description",
"displayed": true,
"value": "An incident involving compromised user accounts at Company ABC."
}
],
"eventDate": "2023-11-01T00:00:00Z",
"tag": [
{
"name": "Phishing Attack"
}
]
},
{
"name": "Leaked Credentials at Company ABC",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0004",
"attribute": [
{
"type": "Description",
"displayed": true,
"value": "An incident involving leaked credentials at Company ABC."
}
],
"eventDate": "2023-11-01T00:00:00Z"
}
]
}
The following input file demonstrates how to associate Groups to other Groups using the association array. When the Batch API processes the file, it will do the following:
Create two Incident Groups (Compromised User Accounts at Company ABC and Leaked Credentials at Company ABC)
Associate Compromised User Accounts at Company ABC to Leaked Credentials at Company ABC
Associate Compromised User Accounts at Company ABC to an existing Group (the Group whose ID is 12345) in the owner in which the batch job is processing data
Create an association between two existing Groups in the owner in which the batch job is processing data
{
"group": [
{
"name": "Compromised User Accounts at Company ABC",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0003",
"attribute": [
{
"type": "Description",
"displayed": true,
"value": "An incident involving compromised user accounts at Company ABC."
}
],
"eventDate": "2023-11-01T00:00:00Z",
"tag": [
{
"name": "Phishing Attack"
}
]
},
{
"name": "Leaked Credentials at Company ABC",
"type": "Incident",
"xid": "00000000-0000-0000-0000-000000000000:0004",
"attribute": [
{
"type": "Description",
"displayed": true,
"value": "An incident involving leaked credentials at Company ABC."
}
],
"eventDate": "2023-11-01T00:00:00Z"
}
],
"association": [
{
"ref_1": "00000000-0000-0000-0000-000000000000:0003",
"ref_2": "00000000-0000-0000-0000-000000000000:0004"
},
{
"ref_1": "00000000-0000-0000-0000-000000000000:0003",
"id_2": 12345
},
{
"id_1": 100,
"id_2": 200
}
]
}
Indicator-to-Indicator Association
The following input file demonstrates how to associate Indicators to other Indicators using the association array. This is the only method you can use to create Indicator-to-Indicator associations with the Batch API.
When the Batch API processes the file, it will do the following:
Create a Host Indicator (verybadguyz.com)
Create a URL Indicator (http://www.verybadguyz.com)
Associate verybadguyz.com to http://www.verybadguyz.com using the URL Host association type
Associate verybadguyz.com to an existing Address Indicator (the Address Indicator whose ID is 54321) in the owner in which the batch job is processing data using the Host to Indicators association type
Create an association between two existing Indicators (an Address and an ASN) in the owner in which the batch job is processing data using the Address to Indicators association type
{
"indicator": [
{
"summary": "verybadguyz.com",
"type": "Host",
"attribute": [
{
"type": "Description",
"value": "A host used by the Very Bad Guyz hacker group.",
"securityLabel": [
{
"name": "TLP:AMBER"
}
]
}
],
"confidence": 60,
"rating": 3,
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
},
{
"summary": "http://www.verybadguyz.com",
"type": "URL",
"attribute": [
{
"type": "Description",
"value": "A URL used by the Very Bad Guyz hacker group.",
"securityLabel": [
{
"name": "TLP:AMBER"
}
]
}
],
"confidence": 60,
"rating": 3,
"securityLabel": [
{
"name": "TLP:AMBER"
}
],
"tag": [
{
"name": "Ransomware"
}
]
}
],
"association": [
{
"ref_1": "verybadguyz.com",
"type_1": "Host",
"ref_2": "http://www.verybadguyz.com",
"type_2": "URL",
"associationType": "URL Host"
},
{
"ref_1": "verybadguyz.com",
"type_1": "Host",
"id_2": 54321,
"type_2": "Address",
"associationType": "Host to Indicators"
},
{
"id_1": 54321,
"type_1": "Address",
"id_2": 654321,
"type_2": "ASN",
"associationType": "Address to Indicators"
}
]
}
File Indicator Considerations
File Indicators may have one or more of the following hashes: MD5, SHA1, and SHA256. When using the Batch API, you can provide values for these hashes using either of the following methods:
Define hashes in the Indicator’s summary
Define hashes individually
Define Hashes in the Indicator’s Summary
Use this method to define all hash values in the Indicator’s summary field as a concatenated string using colon delimiters (i.e., md5 : sha1 : sha256).
{
"indicator": [
{
"summary": "905ad8176a569a36421bf54c04ba7f95 : a52b6986d68cdfac53aa740566cbeade4452124e : 25bdabd23e349f5e5ea7890795b06d15d842bde1d43135c361e755f748ca05d0",
"type": "File"
}
]
}
Define Hashes Individually
Use this method to define each hash value using the individual md5, sha1, and sha256 fields for the Indicator. Note that the presence of one or more of these fields will result in the Indicator’s summary field being ignored during the import.
{
"indicator": [
{
"md5": "905ad8176a569a36421bf54c04ba7f95",
"sha1": "a52b6986d68cdfac53aa740566cbeade4452124e",
"sha256": "25bdabd23e349f5e5ea7890795b06d15d842bde1d43135c361e755f748ca05d0",
"type": "File"
}
]
}
Note
Occasionally, imported File Indicators may contain one or more hashes that existing File Indicators in the same owner also contain. Specifically, either an incoming or existing File Indicator will have additional hash type(s) that the other Indicator does not (e.g., the incoming Indicator has an MD5 and SHA1, while the existing Indicator has only the MD5, or vice versa). In this situation, the resulting File Indicator will end up with a “superset” of file hashes by either retaining the existing hash(es) or adding the new hash(es). However, certain situations may arise that require special processing when incoming file hash(es) cause conflicts with existing data (e.g., the incoming File Indicator contains an MD5 and SHA1, while the existing Indicator contains the same MD5 but a different SHA1). Use the fileMergeMode and hashCollisionMode fields to handle such situations.
Example Request
Send a request in the following format to upload an input file to the Batch API for a batch job (the batch job whose ID is 12345 in this example).
Request (HTTP)
POST /v2/batch/12345
Content-Type: application/octet-stream
<batch input file>
Request (cURL)
curl --location 'https://companyabc.threatconnect.com/api/v2/batch/12345' \
--header 'Timestamp: $UNIX_EPOCH_TIMESTAMP' \
--header 'Authorization: TC $ACCESS_ID:$SIGNATURE' \
--header 'Content-Type: application/octet-stream' \
--data '@/Users/jsmith/Desktop/batchInputFile.json'
Response (Success)
HTTP/1.1 202 Accepted
{
status: "Queued"
}
Response (Overlarge Input File)
HTTP/1.1 400 Bad Request
{
status: "Invalid",
description: "File size greater than allowable limit of 2000000"
}
Check the Status of a Batch Job
Send a request in the following format to check the status of a file upload for a batch job (the batch job whose ID is 12345 in this example). Possible statuses include the following:
Created
Queued
Running
Completed
Request
GET /v2/batch/12345
Response (Batch Job Still Running)
HTTP/1.1 200 OK
{
status: "Running"
}
Response (Batch Job Completed)
HTTP/1.1 200 OK
{
"status": "Success",
"data": {
"batchStatus": {
"id": 12345,
"status": "Completed",
"errorCount": 342,
"successCount": 405432,
"unprocessCount": 0
}
}
}
Query Parameters
The /v2/batch/{batchId} endpoint supports the following query parameter:
includeAdditional: <Boolean> Specifies whether to include counts of successful and unsuccessful Indicator creations, Group creations, and associations in the response. (Default value: false)
Retrieve Error Messages For a Batch Job
Send a request in the following format to retrieve error messages for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.
Request
GET /v2/batch/12345/errors
Response (Batch Job Still Running)
HTTP/1.1 400 Bad Request
{
status: "Invalid",
description: "Batch still in Running state"
}
Response (Batch Job Completed)
HTTP/1.1 200 OK
Content-Type: application/octet-stream ; boundary=
Content-Length:
Content-Encoding: gzip
Note
Responses for batch jobs that ended in partial failures will include an error file that includes Tag, Attribute, or Indicator errors (fail on first).
Retrieve Error Details For a Batch Job
Send a request in the following format to retrieve the details of errors for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.
Request
GET /v2/batch/12345/results
Response
HTTP/1.1 200 OK
[
{
"code": "0x1003",
"severity": "Error",
"errorReason": "com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Not a JSON Object: \"name\"",
"errorMessage": "Encountered an unexpected Exception while processing batch job. Last known JSON path: '$.group[1]': Last processed group[1] '00000000-0000-0000-0000-000000000000:0001'."
}
]
Query Parameters
The /v2/batch/{batchId}/results endpoint supports the following query parameters:
code: <String Array> The error code by which to filter results. Only one error code may be specified at a time, and the specified value must begin with the standard hexadecimal notation of 0x.contains: <String> The text included in theerrorReasonorerrorMessagefields by which to filter results.severity: <String Array> The severity by which to filter results. (Accepted values: err, error, info, warn, warning)
V1 Batch API
The following subsections describe how to create a batch job and upload a file to the job for processing with the V1 Batch API. With the V1 Batch API, you can perform the following actions:
Import Indicators
Create and update Attributes and Tags for Indicators
Create a Batch Job
Send a request in the following format to create a batch job that will use the V1 Batch API.
Request
POST /v2/batch
Content-type: application/json; charset=utf-8
{
"version": "V2",
"owner": "Demo Organization",
"haltOnError": true,
"action": "Create",
"attributeWriteType": "Append"
}
version: <String> The version of the Batch API to use. Set this field’s value toV1to use the V1 Batch API.owner: <String> REQUIRED The name of the Organization, Community, or Source in which the batch job will create or update data.haltOnError: <Boolean> REQUIRED Specifies how the batch process should proceed if it encounters an error. Accepted values include the following:true: The batch process will stop processing the entire batch job the first time it encounters an error.
false: The batch process will attempt to continue processing the batch job after encountering an error.
action: <String> REQUIRED The action that the batch job will perform on incoming data. Accepted values include the following:Create: The batch job will create new and update existing data in the specified owner.
Delete: The batch job will delete existing data in the specified owner that matches any of the incoming data.
attributeWriteType: <String> REQUIRED Specifies how the batch job will handle incoming Attributes. Accepted values include the following:Replace: The batch job will remove all Attributes from existing Indicators and Groups before adding incoming Attributes.
Static: The batch job will ignore incoming Attributes and not update existing Attributes added to existing Indicators and Groups.
Note
If haltOnError is set to true and an error occurs during the batch process, the status for the batch job will be set to Completed and the value for errorCount will be greater than zero. Also, the value for unprocessedCount will be greater than zero unless the uploaded file did not contain valid JSON.
Response (Success)
HTTP/1.1 201 Created
{
"status": "Success",
"data": {
"batchId": 2446
}
}
Response (Insufficient Privileges)
HTTP/1.1 401 Unauthorized
Unable to perform the requested operation due to the following error(s): You do not have permission to create Indicators; Groups; Attributes; Tags; Security Labels;
Response (Incorrect Settings)
HTTP/1.1 503 Service Unavailable
Batch job api is not available. Ensure that batchJobEnabled is true and document storage is enabled and configured;
Attention
When creating a batch job, the parameters defined in the body of the POST request must be accurate. When troubleshooting issues with the Batch API, make sure the parameter names in the request body are correct and that an accepted value is provided for each parameter.
Upload an Input File to a Batch Job
The Batch API expects to ingest a JSON file containing one or more lists of dictionaries. As shown in the following example, the V1 Batch API expects a single array of Indicator objects. At a minimum, you must include the following fields for each object in each array:
summary: <String> REQUIRED The Indicator’s summary.type: <String> REQUIRED The Indicator’s type.
See the following tables for a list of additional fields that you may also include for Indicator objects in the array.
Indicator Fields
Field |
Description |
Accepted Data Type |
Required? |
|---|---|---|---|
attribute |
An array of Attributes added to the Indicator |
Array of Objects |
Optional |
attribute.type |
The Attribute’s Type |
String |
Required if |
attribute.value |
The Attribute’s value |
String |
Required if |
confidence |
The Indicator’s Confidence Rating
Minimum value: 0
Maximum value: 100
|
Integer |
Optional |
description |
The value of the Indicator’s default Description Attribute |
String |
Optional |
rating |
The Indicator’s Threat Rating
Minimum value: 0
Maximum value: 5
|
Integer |
Optional |
source |
The value of the Indicator’s default Source Attribute |
String |
Optional |
summary |
The Indicator’s summary |
String |
Required |
tag |
An array of Tags applied to the Indicator |
Array of Objects |
Optional |
tag.name |
The Tag’s name |
String |
Required if |
type |
The Indicator’s type |
String |
Required |
Hint
Exporting indicators via the JSON Bulk Reports endpoint will create a file in the proper format for the Batch API.
Attention
The maximum number of Indicators that can be created in one batch job is 25,000. If creating more than 25,000 Indicators, use multiple batch jobs.
Example Input File
When this input file is uploaded to the Batch API, the API will create two Indicators (a Host and an Address). It will also add an Attribute and Tag to the Host Indicator.
[
{
"rating": 3,
"confidence": 60,
"description": "A malicious domain",
"summary": "super-malicious.ru",
"type": "Host",
"attribute": [
{
"type": "Additional Analysis and Context",
"value": "This malicious domain has been used in ransomware attacks."
}
],
"tag": [
{
"name": "Malicious Host"
}
]
},
{
"summary": "96.38.88.212",
"type": "Address"
}
]
Example Request
Send a request in the following format to upload an input file to the Batch API for a batch job (the batch job whose ID is 12345 in this example).
Request (HTTP)
POST /v2/batch/12345
Content-Type: application/octet-stream
<batch input file>
Request (cURL)
curl --location 'https://companyabc.threatconnect.com/api/v2/batch/12345' \
--header 'Timestamp: $UNIX_EPOCH_TIMESTAMP' \
--header 'Authorization: TC $ACCESS_ID:$SIGNATURE' \
--header 'Content-Type: application/octet-stream' \
--data '@/Users/jsmith/Desktop/batchInputFile.json'
Response (Success)
HTTP/1.1 202 Accepted
{
status: "Queued"
}
Response (Overlarge Input File)
HTTP/1.1 400 Bad Request
{
status: "Invalid",
description: "File size greater than allowable limit of 2000000"
}
Check the Status of a Batch Job
Send a request in the following format to check the status of a file upload for a batch job (the batch job whose ID is 12345 in this example). Possible statuses include the following:
Created
Queued
Running
Completed
Request
GET /v2/batch/12345
Response (Batch Job Still Running)
HTTP/1.1 200 OK
{
status: "Running"
}
Response (Batch Job Completed)
HTTP/1.1 200 OK
{
"status": "Success",
"data": {
"batchStatus": {
"id": 12345,
"status": "Completed",
"errorCount": 342,
"successCount": 405432,
"unprocessCount": 0
}
}
}
Retrieve Error Messages For a Batch Job
Send a request in the following format to retrieve error messages for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.
Request
GET /v2/batch/12345/errors
Response (Batch Job Still Running)
HTTP/1.1 400 Bad Request
{
status: "Invalid",
description: "Batch still in Running state"
}
Response (Batch Job Completed)
HTTP/1.1 200 OK
Content-Type: application/octet-stream ; boundary=
Content-Length:
Content-Encoding: gzip
Note
Responses for batch jobs that ended in partial failures will include an error file that includes Tag, Attribute, or Indicator errors (fail on first).
Retrieve Error Details For a Batch Job
Send a request in the following format to retrieve the details of errors for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.
Request
GET /v2/batch/12345/results
Response
HTTP/1.1 200 OK
[
{
"code": "0x1003",
"severity": "Error",
"errorReason": "com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Not a JSON Object: \"name\"",
"errorMessage": "Encountered an unexpected Exception while processing batch job. Last known JSON path: '$.group[1]': Last processed group[1] '00000000-0000-0000-0000-000000000000:0001'."
}
]
Query Parameters
The /v2/batch/{batchId}/results endpoint supports the following query parameters:
code: <String Array> The error code by which to filter results. Only one error code may be specified at a time, and the specified value must begin with the standard hexadecimal notation of 0x.contains: <String> The text included in theerrorReasonorerrorMessagefields by which to filter results.severity: <String Array> The severity by which to filter results. (Accepted values: err, error, info, warn, warning)