Cyphon Architecture Overview¶
Cyphon ingests, sifts, reshapes, enhances, and stores data. It can also generate alerts, retrieve data related to the context of an alert, and allow users to take actions on alerts. Below is a brief overview of the Cyphon packages that provide these functionalities.
Cyphon can gather data from emails, logs, and external APIs, such as Twitter. In some cases, Cyphon must listen for incoming data, while in other cases, it must actively request new data. Cyphon’s Aggregator and Responder packages coordinate requests to external APIs. Cyphon uses Django Mailbox to ingest emails, and it works with Logstash and RabbitMQ to gather logs.
As Cyphon ingests data, it examines the data to determine how it should be processed. Based on the data’s characteristics, it can be sent to a specific parser and storage location. This preliminary sorting is performed by
Sieves within Cyphon’s Sifter package. The Sifter package contains separate subpackages for sifting through incoming emails, logs, and API data.
Once data has been sorted, it can be transformed into a custom data model using the Bottler package, which defines
Containers for holding data. Data is “bottled” with the help of a
Condenser, which converts the data into the model defined by the
Condensers are controlled by
Mungers, which process data that has passed through a
After data has been “bottled”, it can be analyzed using
Inspections in the Inspections package and
Procedures in the Lab package. An
Inspection tests data against a set of rules and returns a predefined value if the data is a match. A
Procedure is used to perform more complicated analyses, such as language detection. The results of
Procedures can be attached to data using Labels.
The Distilleries package coordinates several aspects of data processing. A
Distillery specifies the
Container for a set of documents, the
Collection where they are stored, and the Categories associated with the documents. A
Distillery can also enhance data before storing it by applying Labels, and it can construct teasers for those documents using Tastes.
The Warehouses package allows Cyphon to save and retrieve data. Cyphon supports both Elasticsearch and MongoDB storage engines. A
Warehouse can represent an Elasticsearch index or a MongoDB database, and a
Collection within a
Warehouse can represent an Elasticsearch document type or a MongoDB collection. The Engines package provides interfaces for these backends.
Cyphon can generate alerts based on either the characteristics of the data itself or the rate at which the data is being saved. The Watchdogs package analyzes the data being saved and generates alerts for data that meet certain conditions. The Monitors package monitors data flow and generates alerts if data is not being saved at an expected rate. The Alerts package manages records of these alerts.
The Tags package is used to automagically tag
Tags can link to
Articles, which can provide detailed information to help diagnose and remediate
DataTaggers can create new
Tags from specific data fields from
The Contexts package allows users to quickly investigate incidents by locating data that may pertain to an
Context uses the data from an
Alert to find related documents in a specific
Contexts can perform searches based on relative time frames and/or specific data fields, using values from an alert-related event.
Responding to Alerts¶
The Responder package allows users to take action on Alerts. An
Action takes an
Alert and sends a request to a REST API to perform a predefined operation. The response from the API is saved as a
Dispatch also saves a
Stamp of the API call, which shows the
AppUser who made the API call, the start and end times of the call, and the
Passport used to authenticate the call.
Cyphon manages users with the AppUsers package. An
AppUser is a modified version of Django’s default
AppUser permissions can be managed either on a per-user basis or by creating user
Groups with specific permissions.
Groups can also be associated with Alarms, which can be used to filter Alerts.