Batch API

Overview

The Batch API allows you to create and modify threat intelligence data in bulk. Currently, there are two versions of the Batch API: V1 and V2. The V1 Batch API supports Indicators only, whereas the V2 Batch API supports Indicators and Groups. Using the Batch API will improve performance when importing large amounts of data.

After you create a batch job, you can upload a JSON file containing the objects to be created or modified to the Batch API and trigger the batch job that will process the data in the file. The contents of the uploaded file must contain valid JSON and mimic the data structure of a JSON bulk report file.

Prerequisites

To use the Batch API, the following prerequisites must be met:

• Batch API must be enabled on your ThreatConnect instance

• Document storage must be enabled on your ThreatConnect instance

• To create batch jobs in and upload data to an Organization, your API user account must have an Organization role of Standard User or higher

• To create batch jobs in and upload data to a Community or Source, your API user account must have a Community role of Editor or higher

V2 Batch API

The following sections describe how to use the V2 Batch API, including how to create a batch job and upload a file to the job for processing. With the V2 Batch API, you can perform the following actions:

• Import Indicators and Groups

• Create Group-to-Group, Indicator-to-Group, Group-to-Indicator, and, Indicator-to-Indicator associations

• Create and update Attributes, Tags, and Security Labels for Indicators and Groups

• Specify how to manage file hash merges and collisions for File Indicators

Attention

The V2 Batch API does not support cross-owner associations.

Note

To create Indicator-to-Indicator associations with the V2 Batch API, your instance must be running ThreatConnect 7.8.2 or newer.

Create a Batch Job

Send a request in the following format to create a batch job that will use the V2 Batch API.

Request

POST /v2/batch
Content-type: application/json; charset=utf-8


{
    "version": "V2",
    "owner": "Demo Organization",
    "haltOnError": true,
    "action": "Create",
    "attributeWriteType": "Append",
    "tagWriteType": "Append",
    "securityLabelWriteType": "Append",
    "fileMergeMode": "Merge",
    "hashCollisionMode": "FavorIncoming"
}

• version: <String> The version of the Batch API to use. Set this field’s value to V2 to use the V2 Batch API.

• owner: <String> REQUIRED The name of the Organization, Community, or Source in which the batch job will create or update data.

• haltOnError: <Boolean> REQUIRED Specifies how the batch process should proceed if it encounters an error. Accepted values include the following:

  • true: The batch process will stop processing the entire batch job the first time it encounters an error.

  • false: The batch process will attempt to continue processing the batch job after encountering an error.

• action: <String> REQUIRED The action that the batch job will perform on incoming data. Accepted values include the following:

  • Create: The batch job will create new and update existing data in the specified owner.

  • Delete: The batch job will delete existing data in the specified owner that matches any of the incoming data.

• attributeWriteType: <String> REQUIRED Specifies how the batch job will handle incoming Attributes. Accepted values include the following:

  • Append: The batch job will append the incoming Attributes to an Indicator or Group. Note that duplicate Attributes may be added, as redundancy checking is not performed.

  • Replace: The batch job will replace all Attributes added to an Indicator or Group with the incoming Attributes.

  • Singleton: The batch job will replace existing Attributes added to an Indicator or Group only if the incoming data include Attributes with the same Attribute Type(s) as the existing Attributes. Otherwise, existing Attributes added to an Indicator or Group will remain unchanged.

  • Static: The batch job will ignore incoming Attributes and not update existing Attributes added to an Indicator or Group.

• tagWriteType: <String> Specifies how the batch job will handle incoming Tags. Accepted values include the following:

  • Append: The batch job will append the incoming Tags to an Indicator or Group.

  • Replace (default): The batch job will replace all Tags added to an Indicator or Group with the incoming Tags.

• securityLabelWriteType: <String> Specifies how the batch job will handle incoming Security Labels. Accepted values include the following:

  • Append: The batch job will append the incoming Security Labels to an Indicator or Group.

  • Replace (default): The batch job will replace all Security Labels added to an Indicator or Group with the incoming Security Labels.

• fileMergeMode: <String> Specifies whether the batch job will perform a merge operation when the file hashes in an incoming File Indicator match two or more existing File Indicators in the specified owner (e.g., an incoming File Indicator contains an MD5 and SHA1 that each match a separate File Indicator in the specified owner). Accepted values include the following:

  • Distribute: The batch job will not perform a merge operation; instead, it will add the metadata (e.g., Threat and Confidence Ratings, Tags, Attributes) for the incoming File Indicator to all matching File Indicators (up to three possible) in the specified owner.

  • Merge (default): The batch job will perform a merge operation and combine two or more existing File Indicators with separate file hashes into a single File Indicator with all file hashes. During this process, the File Indicator with the most recent lastModified date will be used as the “primary” copy and retain its Threat and Confidence Ratings (at least, until their values are overwritten by the values for the incoming File Indicator). Attributes, Security Labels, and Tags associated with all merged File Indicators will be applied to the “primary” copy as well. Because Attributes are appended to the “primary” copy, duplicate Attributes may be present if each File Indicator that was merged had similar Attributes.

• hashCollisionMode: <String> Specifies how the batch job will handle file hash collisions that occur when importing or merging File Indicators (e.g., an incoming File Indicator contains an MD5 and SHA1, while an existing File Indicator contains the same MD5 but a different SHA1). Accepted values include the following:

  • FavorExisting: The batch job will favor hashes in the existing File Indicator and ignore conflicting hashes in the incoming File Indicator.

  • FavorIncoming (default): The batch job will favor hashes in the incoming File Indicator and overwrite conflicting hashes in the existing File Indicator.

  • IgnoreExisting: If a conflict exists between two or more existing File Indicators, the batch job will delete the existing File Indicators that caused conflict and then import the incoming File Indicator. If a conflict exists between one or more existing File Indicators and an incoming File Indicator, the batch job will ignore the existing Indicators. It will also remove their file hashes from the incoming Indicator, resulting in a new File Indicator being created without any conflicting file hashes. If all available file hashes are removed from the incoming Indicator, no new Indicator will be created.

  • IgnoreIncoming: The batch job will not import the incoming File Indicator causing the conflict, and existing File Indicators will not be updated.

  • Split: The batch job will not perform a merge; instead, it will apply the metadata (e.g., Threat and Confidence Ratings, Tags, Attributes) for the incoming File Indicator to all matching File Indicators in the specified owner.

Note

If haltOnError is set to true and an error occurs during the batch process, the status for the batch job will be set to Completed and the value for errorCount will be greater than zero. Also, the value for unprocessedCount will be greater than zero unless the uploaded file did not contain valid JSON.

Response (Success)

HTTP/1.1 201 Created
{
    "status": "Success",
    "data": {
        "batchId": 2446
    }
}

Response (Insufficient Privileges)

HTTP/1.1 401 Unauthorized
Unable to perform the requested operation due to the following error(s): You do not have permission to create Indicators; Groups; Attributes; Tags; Security Labels;

Response (Incorrect Settings)

HTTP/1.1 503 Service Unavailable
Batch job api is not available.  Ensure that batchJobEnabled is true and document storage is enabled and configured;

Attention

The parameters defined in the body of a POST request to create a batch job must be accurate. When troubleshooting issues with the Batch API, make sure the parameter names in the request body are correct and that an accepted value is provided for each parameter.

Upload an Input File to a Batch Job

The V2 Batch API expects to ingest a JSON file containing one or more lists of objects, where Indicator and Group objects are contained within the indicator and group arrays. If your instance is running ThreatConnect 7.8.2 or newer, you may also define associations to create within the association array.

Note

You can also define associations within the indicator and group arrays, regardless of which ThreatConnect version your instance is running. For more information, see the “Indicator Fields” and “Group Fields” sections.

The following is the batch input file schema for the V2 Batch API. For a list of additional fields you can use when defining Indicator and Group objects, see the “Indicator Fields” and “Group Fields” section, respectively.

• indicator: <Array of Objects> The Indicators to create or update.

  • summary: <String> REQUIRED The Indicator’s summary. For File Indicators, you may substitute summary with the field that contains one of the Indicator’s file hashes (e.g., you can use md5 instead of summary to define a File Indicator’s MD5 hash value). See the “File Indicator Considerations” section for more information.

  • type: <String> REQUIRED The Indicator’s type.

• group: <Array of Objects> The Groups to create or update.

  • name: <String> REQUIRED The Group’s name.

  • type: <String> REQUIRED The Group’s type.

  • xid: <String> REQUIRED The Group’s XID.

• association: <Array of Objects> The associations to create. Supported association types include Indicator to Group, Group to Indicator, Group to Group, and Indicator to Indicator. (Requires ThreatConnect 7.8.2 or newer.)

  • ref_1: <String> The summary of the Indicator or XID of the Group to use as the first object in the association. The Indicator or Group must be present in the batch input file or exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction with id_1. If both are provided, id_1 will be favored over ref_1.)

  • id_1: <Integer> The ID of the Indicator or Group to use as the first object in the association. The Indicator or Group must exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction with ref_1. If both are provided, id_1 will be favored over ref_1.)

  • type_1: <String> The type of Indicator or Group defined for ref_1 or id_1. This field is required for Indicators and optional for Groups.

  • ref_2: <String> The summary of the Indicator or XID of the Group to use as the second object in the association. The Indicator or Group must be present in the input file or exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction with id_2. If both are provided, id_2 will be favored over ref_2.)

  • id_2: <Integer> The ID of the Indicator or Group to use as the second object in the association. The Indicator or Group must exist in the owner in which the batch job is processing data. (This field can be used interchangeably or in conjunction with ref_2. If both are provided, id_2 will be favored over ref_2.)

  • type_2: <String> The type of Indicator or Group defined for ref_2 or id_2. This field is required for Indicators and optional for Groups.

  • associationType: <String> For Indicator-to-Indicator associations only, the name of the association type (e.g., URL Host). This field is required if making an Indicator-to-Indicator association.

Attention

If using the association field, it must be placed after the indicator and group fields in a batch input file.

Indicator Fields

Field	Description	Accepted Data Type	Required?
active	Specifies whether the Indicator’s status is active (true) or inactive (false) Default value: true	Boolean	Optional
activeLocked	Specifies whether to prevent CAL™ from updating the Indicator’s status Default value: false	Boolean	Optional
associatedGroups	An array of Groups associated to the Indicator	Array of Objects	Optional
associatedGroups.groupXid	The associated Group’s XID	String	Required if associatedGroups is used
associatedIndicators	An array of Indicators associated to the Indicator	Array of Objects	Optional
associatedIndicators.indicatorType	The associated Indicator’s type	String	Required if associatedIndicators is used
associatedIndicators.summary	The associated Indicator’s summary	String	Required if associatedIndicators is used
attribute	An array of Attributes added to the Indicator	Array of Objects	Optional
attribute.displayed	Specifies whether to display the Attribute as a default Attribute on the Details screen for the Indicator to which it is added (applies to Description and Source Attributes only) Default value: false	Boolean	Optional
attribute.pinned	Specifies whether to display the Attribute as a pinned Attribute on the Details screen for the Indicator to which it is added Default value: false	Boolean	Optional
attribute.securityLabel	An array of Security Labels applied to the Attribute	Array of Objects	Optional
attribute.securityLabel.color	The hex color code (e.g., FF0000) for the Security Label’s color	String	Required if securityLabel is used within attribute and the Security Label does not exist
attribute.securityLabel.description	The Security Label’s description	String	Required if securityLabel is used within attribute and the Security Label does not exist
attribute.securityLabel.name	The Security Label’s name	String	Required if securityLabel is used within attribute
attribute.source	The Attribute’s source	String	Optional
attribute.type	The Attribute’s Type	String	Required if attribute is used
attribute.value	The Attribute’s value	String	Required if attribute is used
confidence	The Indicator’s Confidence Rating Minimum value: 0 Maximum value: 100	Integer	Optional
description	The value of the Indicator’s default Description Attribute	String	Optional
externalDateAdded	The date and time when the Indicator was created externally	DateTime	Optional
externalDateExpires	The date and time when the Indicator expires externally	DateTime	Optional
externalLastModified	The date and time when the Indicator was last modified externally	DateTime	Optional
firstSeen	The date and time when the Indicator was first seen	DateTime	Optional
lastSeen	The date and time when the Indicator was last seen	DateTime	Optional
md5 [1]	The File Indicator’s MD5 file hash	String	Optional
privateFlag	Specifies whether to mark the Indicator as private (requires private Indicators being enabled on for your ThreatConnect instance) Default value: false	Boolean	Optional
rating	The Indicator’s Threat Rating Minimum value: 0 Maximum value: 5	Integer	Optional
securityLabel	An array of Security Labels applied to the Indicator	Array of Objects	Optional
securityLabel.color	The hex color code (e.g., FF0000) for the Security Label’s color	String	Required if securityLabel is used and the Security Label does not exist
securityLabel.description	The Security Label’s description	String	Required if securityLabel is used and the Security Label does not exist
securityLabel.name	The Security Label’s name	String	Required if securityLabel is used
sha1 [1]	The File Indicator’s SHA1 file hash	String	Optional
sha256 [1]	The File Indicator’s SHA256 file hash	String	Optional
size	The File Indicator’s file size	Integer	Optional
summary [1]	The Indicator’s summary	String	Required
tag	An array of Tags applied to the Indicator	Array of Objects	Optional
tag.name	The Tag’s name	String	Required if tag is used
type	The Indicator’s type	String	Required

[1] (1,2,3,4)

For File Indicators, you can define their file hashes as a concatenated string using colon delimiters in the summary field or as separate file hashes in the md5, sha1, and sha256 fields. See the “File Indicator Considerations” section for more information.

Group Fields

Field	Description	Accepted Data Type	Required?
associatedGroupXid	An array of XIDs for Groups associated to the Group	String Array	Optional
associatedIndicators	An array of Indicators associated to the Group	Array of Objects	Optional
associatedIndicators.indicatorType	The associated Indicator’s type	String	Required if associatedIndicators is used
associatedIndicators.summary	The associated Indicator’s summary	String	Required if associatedIndicators is used
attribute	An array of Attributes added to the Group	Array of Objects	Optional
attribute.displayed	Specifies whether to display the Attribute as a default Attribute on the Details screen for the Group to which it is added (applies to Description and Source Attributes only) Default value: false	Boolean	Optional
attribute.pinned	Specifies whether to display the Attribute as a pinned Attribute on the Details screen for the Group to which it is added Default value: false	Boolean	Optional
attribute.securityLabel	An array of Security Labels applied to the Attribute	Array of Objects	Optional
attribute.securityLabel.color	The hex color code (e.g., FF0000) for the Security Label’s color	String	Required if securityLabel is used within attribute and the Security Label does not exist
attribute.securityLabel.description	The Security Label’s description	String	Required if securityLabel is used within attribute and the Security Label does not exist
attribute.securityLabel.name	The Security Label’s name	String	Required if securityLabel is used within attribute
attribute.source	The Attribute’s source	String	Optional
attribute.type	The Attribute’s Type	String	Required if attribute is used
attribute.value	The Attribute’s value	String	Required if attribute is used
body	The body of the email (for Email Groups only)	String	Required for Email Groups
eventDate	The date and time when the security event or incident took place (for Event and Incident Groups only)	DateTime	Optional
externalDateAdded	The date and time when the Group was created externally	DateTime	Optional
externalDateExpires	The date and time when the Group expires externally	DateTime	Optional
externalLastModified	The date and time when the Group was last modified externally	DateTime	Optional
fileName	The name of the Group’s file (for Document, Report, and Signature Groups only)	String	Required for Document, Report, and Signature Groups
fileText	The contents of the signature file (for Signature Groups only)	String	Required for Signature Groups
fileType	The signature file’s type (for Signature Groups only) Acceptable values: * Bro * ClamAV * CybOX * Iris Search Hash * KQL * OpenIOC * Regex * SPL * Sigma * Snort * Suricata * TQL Query * YARA	String	Required for Signature Groups only
firstSeen	The date and time when the Group was first seen	DateTime	Optional
from	The sender’s email address for the email (for Email Groups only)	String	Optional
header	The header of the email (for Email Groups only)	String	Required for Email Groups
insights	An AI-generated synopsis of the Group (for Document and Report Groups only)	String	Optional
lastSeen	The date and time when the Group was last seen	DateTime	Optional
malware	Specifies whether the file uploaded to the Group is a malware file (for Document Groups only) Default value: false	Boolean	Optional
name	The Group’s name	String	Required
password	The password to access a malware file uploaded to the Group (for Document Groups only)	String	Required only if malware is set to true for a Document Group
securityLabel	An array of Security Labels applied to the Group	Array of Objects	Optional
securityLabel.color	The hex color code (e.g., FF0000) for the Security Label’s color	String	Required if securityLabel is used and the Security Label does not exist
securityLabel.description	The Security Label’s description	String	Required if securityLabel is used and the Security Label does not exist
securityLabel.name	The Security Label’s name	String	Required if securityLabel is used
status	The status of the security event or incident (for Event and Incident Groups only) Acceptable values (Event): * Needs Review * False Positive * No Further Action * Escalated Acceptable values (Incident): * New * Open * Stalled * Containment Achieved * Restoration Achieved * Incident Reported * Closed * Rejected * Deleted	String	Optional
subject	The subject line of the email (for Email Groups only)	String	Required for Email Groups
tag	An array of Tags applied to the Group	Array of Objects	Optional
tag.name	The Tag’s name	String	Required if tag is used
to	The recipient’s email address (for Email Groups only)	String	Optional
type	The Group’s type	String	Required
xid	The Group’s XID	String	Required

Hint

Exporting indicators via the JSON Bulk Reports endpoint will create a file in the proper format for the Batch API.

Attention

The maximum number of Indicators that can be created in one batch job is 25,000. If creating more than 25,000 Indicators, use multiple batch jobs.

Example Input Files

Indicator-to-Group Association

The following input file demonstrates how to associate Indicators to Groups using the associatedGroups field within the indicator array. When the Batch API processes the file, it will do the following:

• Create a Host Indicator (badguyz.com)

• Create an Incident Group (Ransomware Attack at Company ABC)

• Associate badguyz.com to Ransomware Attack at Company ABC

{
    "indicator": [
        {
            "summary": "badguyz.com",
            "type": "Host",
            "associatedGroups": [
                {
                    "groupXid": "00000000-0000-0000-0000-000000000000:0001"
                }
            ],
            "attribute": [
                {
                    "type": "Description",
                    "value": "This host was involved in a ransomware attack that targeted employees at Company ABC.",
                    "securityLabel": [
                        {
                            "name": "TLP:AMBER"
                        }
                    ]
                }
            ],
            "confidence": 60,
            "rating": 3,
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ],
    "group": [
        {
            "name": "Ransomware Attack at Company ABC",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0001",
            "attribute": [
                {
                    "type": "Description",
                    "value": "A ransomware attack that targeted employees at Company ABC.",
                    "displayed": true
                }
            ],
            "eventDate": "2024-08-04T00:00:00Z",
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ]
}

The following input file demonstrates how to associate Indicators to Groups using the association array. When the Batch API processes the file, it will do the following:

• Create a Host Indicator (badguyz.com)

• Create an Incident Group (Ransomware Attack at Company ABC)

• Associate badguyz.com to Ransomware Attack at Company ABC

• Associate badguyz.com to an existing Group (the Group whose ID is 12345) in the owner in which the batch job is processing data

{
    "indicator": [
        {
            "summary": "badguyz.com",
            "type": "Host",
            "attribute": [
                {
                    "type": "Description",
                    "value": "This host was involved in a ransomware attack that targeted employees at Company ABC.",
                    "securityLabel": [
                        {
                            "name": "TLP:AMBER"
                        }
                    ]
                }
            ],
            "confidence": 60,
            "rating": 3,
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ],
    "group": [
        {
            "name": "Ransomware Attack at Company ABC",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0001",
            "attribute": [
                {
                    "type": "Description",
                    "value": "A ransomware attack that targeted employees at Company ABC.",
                    "displayed": true
                }
            ],
            "eventDate": "2024-08-04T00:00:00Z",
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ],
    "association": [
        {
            "ref_1": "badguyz.com",
            "type_1": "Host",
            "ref_2": "00000000-0000-0000-0000-000000000000:0001"
        },
        {
            "ref_1": "badguyz.com",
            "type_1": "Host",
            "id_2": 12345
        }
    ]
}

Group-to-Indicator Association

Note

When creating Group-to-Indicator associations, including the Indicators in the input file improves the efficiency of the batch job. Otherwise, a lookup will need to be made for each Indicator not included in the input file.

The following input file demonstrates how to associate Groups to newly created Indicators using the associatedIndicators field within the group array. When the Batch API processes the file, it will do the following:

• Create a URL Indicator (http://www.badguyz.com)

• Create an Incident Group (Ransomware Attack at Company XYZ)

• Associate Ransomware Attack at Company XYZ to http://www.badguyz.com

{
    "indicator": [
        {
            "summary": "http://www.badguyz.com",
            "type": "URL",
            "attribute": [
                {
                    "type": "Description",
                    "value": "This URL was involved in a ransomware attack that targeted employees at Company XYZ.",
                    "securityLabel": [
                        {
                            "name": "TLP:AMBER"
                        }
                    ]
                }
            ],
            "confidence": 60,
            "rating": 3,
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ],
    "group": [
        {
            "name": "Ransomware Attack at Company XYZ",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0002",
            "associatedIndicators": [
                {
                    "summary": "http://www.badguyz.com",
                    "indicatorType": "URL"
                }
            ],
            "attribute": [
                {
                    "type": "Description",
                    "value": "A ransomware attack that targeted employees at Company XYZ.",
                    "displayed": true
                }
            ],
            "eventDate": "2024-08-04T00:00:00Z",
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ]
}

The following input file demonstrates how to associate Groups to existing Indicators using the associatedIndicators field within the group array. When the Batch API processes the file, it will do the following:

• Create an Incident Group (Ransomware Attack at Company XYZ)

• Associate Ransomware Attack at Company XYZ to an existing Address Indicator (71.6.135.131) in the owner in which the batch job is processing data

Because the Address Indicator already exists in ThreatConnect, it does not need to be defined in the indicator array in the input file.

{
    "group": [
        {
            "name": "Ransomware Attack at Company XYZ",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0002",
            "associatedIndicators": [
                {
                    "summary": "71.6.135.131",
                    "indicatorType": "Address"
                }
            ],
            "attribute": [
                {
                    "type": "Description",
                    "value": "A ransomware attack that targeted employees at Company XYZ.",
                    "displayed": true
                }
            ],
            "eventDate": "2024-08-04T00:00:00Z",
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ]
}

The following input file demonstrates how to associate Groups to Indicators using the association array. When the Batch API processes the file, it will do the following:

• Create a URL Indicator (http://www.badguyz.com)

• Create an Incident Group (Ransomware Attack at Company XYZ)

• Associate Ransomware Attack at Company XYZ to http://www.badguyz.com

• Associate Ransomware Attack at Company XYZ to an existing Address Indicator (the Address Indicator whose ID is 54321) in the owner in which the batch job is processing data

{
    "indicator": [
        {
            "summary": "http://www.badguyz.com",
            "type": "URL",
            "attribute": [
                {
                    "type": "Description",
                    "value": "This URL was involved in a ransomware attack that targeted employees at Company XYZ.",
                    "securityLabel": [
                        {
                            "name": "TLP:AMBER"
                        }
                    ]
                }
            ],
            "confidence": 60,
            "rating": 3,
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ],
    "group": [
        {
            "name": "Ransomware Attack at Company XYZ",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0002",
            "attribute": [
                {
                    "type": "Description",
                    "value": "A ransomware attack that targeted employees at Company XYZ.",
                    "displayed": true
                }
            ],
            "eventDate": "2024-08-04T00:00:00Z",
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ],
    "association": [
        {
            "ref_1": "00000000-0000-0000-0000-000000000000:0002",
            "ref_2": "http://www.badguyz.com",
            "type_2": "URL"
        },
        {
            "ref_1": "00000000-0000-0000-0000-000000000000:0002",
            "id_2": 54321,
            "type_2": "Address"
        }
    ]
}

Group-to-Group Association

The following input file demonstrates how to associate Groups to other Groups using the associatedGroupXid field within the group array. When the Batch API processes the file, it will do the following:

• Create two Incident Groups (Compromised User Accounts at Company ABC and Leaked Credentials at Company ABC)

• Associate Compromised User Accounts at Company ABC to Leaked Credentials at Company ABC

{
    "group": [
        {
            "name": "Compromised User Accounts at Company ABC",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0003",
            "associatedGroupXid": [
                "00000000-0000-0000-0000-000000000000:0004"
            ],
            "attribute": [
                {
                    "type": "Description",
                    "displayed": true,
                    "value": "An incident involving compromised user accounts at Company ABC."
                }
            ],
            "eventDate": "2023-11-01T00:00:00Z",
            "tag": [
                {
                    "name": "Phishing Attack"
                }
            ]
        },
        {
            "name": "Leaked Credentials at Company ABC",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0004",
            "attribute": [
                {
                    "type": "Description",
                    "displayed": true,
                    "value": "An incident involving leaked credentials at Company ABC."
                }
            ],
            "eventDate": "2023-11-01T00:00:00Z"
        }
    ]
}

The following input file demonstrates how to associate Groups to other Groups using the association array. When the Batch API processes the file, it will do the following:

• Create two Incident Groups (Compromised User Accounts at Company ABC and Leaked Credentials at Company ABC)

• Associate Compromised User Accounts at Company ABC to Leaked Credentials at Company ABC

• Associate Compromised User Accounts at Company ABC to an existing Group (the Group whose ID is 12345) in the owner in which the batch job is processing data

• Create an association between two existing Groups in the owner in which the batch job is processing data

{
    "group": [
        {
            "name": "Compromised User Accounts at Company ABC",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0003",
            "attribute": [
                {
                    "type": "Description",
                    "displayed": true,
                    "value": "An incident involving compromised user accounts at Company ABC."
                }
            ],
            "eventDate": "2023-11-01T00:00:00Z",
            "tag": [
                {
                    "name": "Phishing Attack"
                }
            ]
        },
        {
            "name": "Leaked Credentials at Company ABC",
            "type": "Incident",
            "xid": "00000000-0000-0000-0000-000000000000:0004",
            "attribute": [
                {
                    "type": "Description",
                    "displayed": true,
                    "value": "An incident involving leaked credentials at Company ABC."
                }
            ],
            "eventDate": "2023-11-01T00:00:00Z"
        }
    ],
    "association": [
        {
            "ref_1": "00000000-0000-0000-0000-000000000000:0003",
            "ref_2": "00000000-0000-0000-0000-000000000000:0004"
        },
        {
            "ref_1": "00000000-0000-0000-0000-000000000000:0003",
            "id_2": 12345
        },
        {
            "id_1": 100,
            "id_2": 200
        }
    ]
}

Indicator-to-Indicator Association

The following input file demonstrates how to associate Indicators to other Indicators using the association array. This is the only method you can use to create Indicator-to-Indicator associations with the Batch API.

When the Batch API processes the file, it will do the following:

• Create a Host Indicator (verybadguyz.com)

• Create a URL Indicator (http://www.verybadguyz.com)

• Associate verybadguyz.com to http://www.verybadguyz.com using the URL Host association type

• Associate verybadguyz.com to an existing Address Indicator (the Address Indicator whose ID is 54321) in the owner in which the batch job is processing data using the Host to Indicators association type

• Create an association between two existing Indicators (an Address and an ASN) in the owner in which the batch job is processing data using the Address to Indicators association type

{
    "indicator": [
        {
            "summary": "verybadguyz.com",
            "type": "Host",
            "attribute": [
                {
                    "type": "Description",
                    "value": "A host used by the Very Bad Guyz hacker group.",
                    "securityLabel": [
                        {
                            "name": "TLP:AMBER"
                        }
                    ]
                }
            ],
            "confidence": 60,
            "rating": 3,
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        },
        {
            "summary": "http://www.verybadguyz.com",
            "type": "URL",
            "attribute": [
                {
                    "type": "Description",
                    "value": "A URL used by the Very Bad Guyz hacker group.",
                    "securityLabel": [
                        {
                            "name": "TLP:AMBER"
                        }
                    ]
                }
            ],
            "confidence": 60,
            "rating": 3,
            "securityLabel": [
                {
                    "name": "TLP:AMBER"
                }
            ],
            "tag": [
                {
                    "name": "Ransomware"
                }
            ]
        }
    ],
    "association": [
        {
            "ref_1": "verybadguyz.com",
            "type_1": "Host",
            "ref_2": "http://www.verybadguyz.com",
            "type_2": "URL",
            "associationType": "URL Host"
        },
        {
            "ref_1": "verybadguyz.com",
            "type_1": "Host",
            "id_2": 54321,
            "type_2": "Address",
            "associationType": "Host to Indicators"
        },
        {
            "id_1": 54321,
            "type_1": "Address",
            "id_2": 654321,
            "type_2": "ASN",
            "associationType": "Address to Indicators"
        }
    ]
}

File Indicator Considerations

File Indicators may have one or more of the following hashes: MD5, SHA1, and SHA256. When using the Batch API, you can provide values for these hashes using either of the following methods.

Define Hashes in the Indicator’s Summary

Use this method to define all hash values in the Indicator’s summary field as a concatenated string using colon delimiters (i.e., md5 : sha1 : sha256).

{
    "indicator": [
        {
            "summary": "905ad8176a569a36421bf54c04ba7f95 : a52b6986d68cdfac53aa740566cbeade4452124e : 25bdabd23e349f5e5ea7890795b06d15d842bde1d43135c361e755f748ca05d0",
            "type": "File"
        }
    ]
}

Define Hashes Individually

Use this method to define each hash value using the individual md5, sha1, and sha256 fields for the Indicator. Note that the presence of one or more of these fields will result in the Indicator’s summary field being ignored during the import.

{
    "indicator": [
        {
            "md5": "905ad8176a569a36421bf54c04ba7f95",
            "sha1": "a52b6986d68cdfac53aa740566cbeade4452124e",
            "sha256": "25bdabd23e349f5e5ea7890795b06d15d842bde1d43135c361e755f748ca05d0",
            "type": "File"
        }
    ]
}

Note

Occasionally, imported File Indicators may contain one or more hashes that existing File Indicators in the same owner also contain. Specifically, either an incoming or existing File Indicator will have additional hash type(s) that the other Indicator does not (e.g., the incoming Indicator has an MD5 and SHA1, while the existing Indicator has only the MD5, or vice versa). In this situation, the resulting File Indicator will end up with a “superset” of file hashes by either retaining the existing hash(es) or adding the new hash(es). However, certain situations may arise that require special processing when incoming file hash(es) cause conflicts with existing data (e.g., the incoming File Indicator contains an MD5 and SHA1, while the existing Indicator contains the same MD5 but a different SHA1). Use the fileMergeMode and hashCollisionMode fields to handle such situations.

Example Request

Send a request in the following format to upload an input file to the Batch API for a batch job (the batch job whose ID is 12345 in this example).

Request (HTTP)

POST /v2/batch/12345
Content-Type: application/octet-stream


<batch input file>

Request (cURL)

curl --location 'https://companyabc.threatconnect.com/api/v2/batch/12345' \
--header 'Timestamp: $UNIX_EPOCH_TIMESTAMP' \
--header 'Authorization: TC $ACCESS_ID:$SIGNATURE' \
--header 'Content-Type: application/octet-stream' \
--data '@/Users/jsmith/Desktop/batchInputFile.json'

Response (Success)

HTTP/1.1 202 Accepted
{
    status: "Queued"
}

Response (Overlarge Input File)

HTTP/1.1 400 Bad Request
{
    status: "Invalid",
    description: "File size greater than allowable limit of 2000000"
}

Check the Status of a Batch Job

Send a request in the following format to check the status of a file upload for a batch job (the batch job whose ID is 12345 in this example). Possible statuses include the following:

• Created

• Queued

• Running

• Completed

Request

GET /v2/batch/12345

Response (Batch Job Still Running)

HTTP/1.1 200 OK
{
    status: "Running"
}

Response (Batch Job Completed)

HTTP/1.1 200 OK
{
    "status": "Success",
    "data": {
        "batchStatus": {
            "id": 12345,
            "status": "Completed",
            "errorCount": 342,
            "successCount": 405432,
            "unprocessCount": 0
        }
    }
}

Query Parameters

The /v2/batch/{batchId} endpoint supports the following query parameter:

• includeAdditional: <Boolean> Specifies whether to include counts of successful and unsuccessful Indicator creations, Group creations, and associations in the response. (Default value: false)

Retrieve Error Messages For a Batch Job

Send a request in the following format to retrieve error messages for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.

Request

GET /v2/batch/12345/errors

Response (Batch Job Still Running)

HTTP/1.1 400 Bad Request
{
    status: "Invalid",
    description: "Batch still in Running state"
}

Response (Batch Job Completed)

HTTP/1.1 200 OK
Content-Type: application/octet-stream ; boundary=
Content-Length:
Content-Encoding: gzip

Note

Responses for batch jobs that ended in partial failures will include an error file that includes Tag, Attribute, or Indicator errors (fail on first).

Retrieve Error Details For a Batch Job

Send a request in the following format to retrieve the details of errors for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.

Request

GET /v2/batch/12345/results

Response

HTTP/1.1 200 OK
[
    {
        "code": "0x1003",
        "severity": "Error",
        "errorReason": "com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Not a JSON Object: \"name\"",
        "errorMessage": "Encountered an unexpected Exception while processing batch job. Last known JSON path: '$.group[1]': Last processed group[1] '00000000-0000-0000-0000-000000000000:0001'."
    }
]

Query Parameters

The /v2/batch/{batchId}/results endpoint supports the following query parameters:

• code: <String Array> The error code by which to filter results. Only one error code may be specified at a time, and the specified value must begin with the standard hexadecimal notation of 0x.

• contains: <String> The text included in the errorReason or errorMessage fields by which to filter results.

• severity: <String Array> The severity by which to filter results. (Accepted values: err, error, info, warn, warning)

V1 Batch API

The following subsections describe how to create a batch job and upload a file to the job for processing with the V1 Batch API. With the V1 Batch API, you can perform the following actions:

• Import Indicators

• Create and update Attributes and Tags for Indicators

Create a Batch Job

Send a request in the following format to create a batch job that will use the V1 Batch API.

Request

POST /v2/batch
Content-type: application/json; charset=utf-8


{
    "version": "V2",
    "owner": "Demo Organization",
    "haltOnError": true,
    "action": "Create",
    "attributeWriteType": "Append"
}

• version: <String> The version of the Batch API to use. Set this field’s value to V1 to use the V1 Batch API.

• owner: <String> REQUIRED The name of the Organization, Community, or Source in which the batch job will create or update data.

• haltOnError: <Boolean> REQUIRED Specifies how the batch process should proceed if it encounters an error. Accepted values include the following:

  • true: The batch process will stop processing the entire batch job the first time it encounters an error.

  • false: The batch process will attempt to continue processing the batch job after encountering an error.

• action: <String> REQUIRED The action that the batch job will perform on incoming data. Accepted values include the following:

  • Create: The batch job will create new and update existing data in the specified owner.

  • Delete: The batch job will delete existing data in the specified owner that matches any of the incoming data.

• attributeWriteType: <String> REQUIRED Specifies how the batch job will handle incoming Attributes. Accepted values include the following:

  • Replace: The batch job will remove all Attributes from existing Indicators and Groups before adding incoming Attributes.

  • Static: The batch job will ignore incoming Attributes and not update existing Attributes added to existing Indicators and Groups.

Note

If haltOnError is set to true and an error occurs during the batch process, the status for the batch job will be set to Completed and the value for errorCount will be greater than zero. Also, the value for unprocessedCount will be greater than zero unless the uploaded file did not contain valid JSON.

Response (Success)

HTTP/1.1 201 Created
{
    "status": "Success",
    "data": {
        "batchId": 2446
    }
}

Response (Insufficient Privileges)

HTTP/1.1 401 Unauthorized
Unable to perform the requested operation due to the following error(s): You do not have permission to create Indicators; Groups; Attributes; Tags; Security Labels;

Response (Incorrect Settings)

HTTP/1.1 503 Service Unavailable
Batch job api is not available.  Ensure that batchJobEnabled is true and document storage is enabled and configured;

Attention

When creating a batch job, the parameters defined in the body of the POST request must be accurate. When troubleshooting issues with the Batch API, make sure the parameter names in the request body are correct and that an accepted value is provided for each parameter.

Upload an Input File to a Batch Job

The Batch API expects to ingest a JSON file containing one or more lists of dictionaries. As shown in the following example, the V1 Batch API expects a single array of Indicator objects. At a minimum, you must include the following fields for each object in each array:

• summary: <String> REQUIRED The Indicator’s summary.

• type: <String> REQUIRED The Indicator’s type.

See the following tables for a list of additional fields that you may also include for Indicator objects in the array.

Indicator Fields

Field	Description	Accepted Data Type	Required?
attribute	An array of Attributes added to the Indicator	Array of Objects	Optional
attribute.type	The Attribute’s Type	String	Required if attribute is used
attribute.value	The Attribute’s value	String	Required if attribute is used
confidence	The Indicator’s Confidence Rating Minimum value: 0 Maximum value: 100	Integer	Optional
description	The value of the Indicator’s default Description Attribute	String	Optional
rating	The Indicator’s Threat Rating Minimum value: 0 Maximum value: 5	Integer	Optional
source	The value of the Indicator’s default Source Attribute	String	Optional
summary	The Indicator’s summary	String	Required
tag	An array of Tags applied to the Indicator	Array of Objects	Optional
tag.name	The Tag’s name	String	Required if tag is used
type	The Indicator’s type	String	Required

Hint

Exporting indicators via the JSON Bulk Reports endpoint will create a file in the proper format for the Batch API.

Attention

The maximum number of Indicators that can be created in one batch job is 25,000. If creating more than 25,000 Indicators, use multiple batch jobs.

Example Input File

When this input file is uploaded to the Batch API, the API will create two Indicators (a Host and an Address). It will also add an Attribute and Tag to the Host Indicator.

[
    {
        "rating": 3,
        "confidence": 60,
        "description": "A malicious domain",
        "summary": "super-malicious.ru",
        "type": "Host",
        "attribute": [
            {
                "type": "Additional Analysis and Context",
                "value": "This malicious domain has been used in ransomware attacks."
            }
        ],
        "tag": [
            {
                "name": "Malicious Host"
            }
        ]
    },
    {
        "summary": "96.38.88.212",
        "type": "Address"
    }
]

Example Request

Send a request in the following format to upload an input file to the Batch API for a batch job (the batch job whose ID is 12345 in this example).

Request (HTTP)

POST /v2/batch/12345
Content-Type: application/octet-stream


<batch input file>

Request (cURL)

curl --location 'https://companyabc.threatconnect.com/api/v2/batch/12345' \
--header 'Timestamp: $UNIX_EPOCH_TIMESTAMP' \
--header 'Authorization: TC $ACCESS_ID:$SIGNATURE' \
--header 'Content-Type: application/octet-stream' \
--data '@/Users/jsmith/Desktop/batchInputFile.json'

Response (Success)

HTTP/1.1 202 Accepted
{
    status: "Queued"
}

Response (Overlarge Input File)

HTTP/1.1 400 Bad Request
{
    status: "Invalid",
    description: "File size greater than allowable limit of 2000000"
}

Check the Status of a Batch Job

Send a request in the following format to check the status of a file upload for a batch job (the batch job whose ID is 12345 in this example). Possible statuses include the following:

• Created

• Queued

• Running

• Completed

Request

GET /v2/batch/12345

Response (Batch Job Still Running)

HTTP/1.1 200 OK
{
    status: "Running"
}

Response (Batch Job Completed)

HTTP/1.1 200 OK
{
    "status": "Success",
    "data": {
        "batchStatus": {
            "id": 12345,
            "status": "Completed",
            "errorCount": 342,
            "successCount": 405432,
            "unprocessCount": 0
        }
    }
}

Retrieve Error Messages For a Batch Job

Send a request in the following format to retrieve error messages for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.

Request

GET /v2/batch/12345/errors

Response (Batch Job Still Running)

HTTP/1.1 400 Bad Request
{
    status: "Invalid",
    description: "Batch still in Running state"
}

Response (Batch Job Completed)

HTTP/1.1 200 OK
Content-Type: application/octet-stream ; boundary=
Content-Length:
Content-Encoding: gzip

Note

Responses for batch jobs that ended in partial failures will include an error file that includes Tag, Attribute, or Indicator errors (fail on first).

Retrieve Error Details For a Batch Job

Send a request in the following format to retrieve the details of errors for a batch job with an errorCount greater than zero. If there are no errors for the specified batch job, a 404 error will be returned.

Request

GET /v2/batch/12345/results

Response

HTTP/1.1 200 OK
[
    {
        "code": "0x1003",
        "severity": "Error",
        "errorReason": "com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Not a JSON Object: \"name\"",
        "errorMessage": "Encountered an unexpected Exception while processing batch job. Last known JSON path: '$.group[1]': Last processed group[1] '00000000-0000-0000-0000-000000000000:0001'."
    }
]

Query Parameters

The /v2/batch/{batchId}/results endpoint supports the following query parameters:

• code: <String Array> The error code by which to filter results. Only one error code may be specified at a time, and the specified value must begin with the standard hexadecimal notation of 0x.

• contains: <String> The text included in the errorReason or errorMessage fields by which to filter results.

• severity: <String Array> The severity by which to filter results. (Accepted values: err, error, info, warn, warning)