Throughout March, April and May 2019 we improved Machine Learning backend infrastructures and processes to accelerate discovery of new Indicators of Compromise (IoC). Higher number of discovered IoCs has two benefits: 1. higher percentage of Cognitive incidents will fall into Confirmed Threat category, 2. more IoCs used for (re)training of Machine Learned predictors means extended ability to get net-new detections. The improvemens include:
- Optimized RAndom Forests (OraF) - we implemented and open-sourced super-fast Random Forest training code (see http://github.com/cisco/oraf)
- Confirmed Threat IoC automated discovery - we implemented automation of new IoC discovery for existing Confirm Threats. The technique reduced manual confirmation load by half without compromising the precision of Confirmed Threat incidents (see IoC statistics below)
- Talos-integration - we built a Machine Learning pipeline and process for improved exchange of intelligence between Talos and Cognitive Intelligence (see Confirmed Threat IDs CTALxxxx in Coverage of Confirmed Threats Extended blog)
- Threat confirmation pipelines - we optimized infrastructures to take maximum effect of each verdict provided by analyst. IoC candidates come through Active Learning (see next item). Every manually confirmed or disproved IoC candidate gets automatically included in fresh Machine Learning ground truth, prediction models get re-trained, re-evaluated and queued for push-to-prod; existing Incident pool gets re-evaluated and Incidents related to the newly confirmed IoC move from Detected to Confirmed.
- Active Learning - we built clever prioritization of threat analytics tasks for human analysts - the Machine Learning system identifies the most promising potential new IoCs for confirmation by human, among millions of suspicious - previously unknown - findings each week. Note that we build on top of A.I. Loop, first introduced in Closing one learning loop blog. Note that we also improved robustness of Machine Learning evaluation pipelines (see our NeurIPS paper: Bad practices in evaluation methodology relevant to class-imbalanced problems)
The cumulative effect of these updates can be seen in Coverage of Confirmed Threats Extended blog, see net-new Confirmed Threats and increase in IoC numbers in updated Confirmed Threats.
