TAVM - A Threat And Vulnerability Mining framework

The threat and vulnerability mining framework implements a modular approach to collect cyber threat intelligence data from publicly available information sources. It is designed to be extensible and easy to use.

Features

The main features of the framework are:

• Modular design
• Easy to add custom input and transformation adapters
• Transformation of unstructured data into STIX 2.1 bundles
• Automatic extraction of vulnerable Git repositories

Basic Structure

The framework uses a central management program written in Golang. This management program handles the configuration and setup of input and transformer adapters. It handles the routing of all CTI data through the different processing pipeline stages. The management program also handles the storage of the gathered raw data in a MongoDB database.

Input Adapters

Input adapters are responsible for fetching publicly available CTI data from different sources. The gathered data is then passed to the TAVM core application for further processing.

Each adapter is started as a separate process and communicates with the core application via REST API. The documentation of the REST API can be viewed as Swagger JSON document which can be found here.

Transformer Adapters

Transformer adapters are responsible for transforming the gathered raw data into STIX 2.1 bundles. The bundles are then published to a TAXII server by the TAVM core application.

Each adapter is started as a separate process and communicates with the core application via REST API. The documentation of the REST API can be viewed as Swagger JSON document which can be found here. Transformer adapters need to provide a REST Server which handles requests from the core application.

Output Adapters

Output adapters are responsible for publishing transformed CTI records to external systems. Currently, there are two output adapters available:

• TAXII Publisher
• Vulnerable Git Repository Exporter

TAXII Publisher

The TAXII publisher can be used to publish STIX 2.1 bundles to a TAXII server. A sample TAXII server implementation can be found here.

Vulnerable Git Repository Exporter

The vulnerable Git repository exporter stores all Git repositories which contain vulnerable software components in a MongoDB database. The data can then be exported using the provided REST API or frontend.

Building and using TAVM

See Building Pre-requisites for a list of pre-requisites.

A quick start guide can be found here.