The Problem It's not uncommon for machine data systems to send and receive duplicate or repeated events. This could be due to a variety of reasons, for example; Misconfiguration (a.k.a layer-8 problems) on the source, intermediary or aggregation systems may cause duplicate data to be sent out. Bugs or software defects may cause a data source to occasionally … Continue reading Streaming Data Deduplication with Cribl
Trimming Unnecessary Fields from Logs
The author of a log has very different motivations from the consumer of that same log. For the author, they must conceive of all the use cases this data may be useful for and include information which will be relevant both now and in the future. The author must ask a number of important questions … Continue reading Trimming Unnecessary Fields from Logs
Helping Threat Hunters While Staying Compliant: Categorizing and Scoring AWS CloudTrail Events in Real-Time
AWS CloudTrail is a service that "enables governance, compliance, operational auditing, and risk auditing of your AWS account." It continuously monitors accounts and it is one of the most valuable and probably the best data source for security analysis in AWS. Security and Operations teams love Cloudtrail because of the added visibility into user and resource … Continue reading Helping Threat Hunters While Staying Compliant: Categorizing and Scoring AWS CloudTrail Events in Real-Time
Cribl LogStream 1.5: Now Supporting Splunk Universal Forwarder and Syslog
Normally I prefer to write a wittier headline for a release announcement, especially for a release this important. Three things are holding me back, however: a) I really want you to know we support the Universal Forwarder and Syslog, b) length of headline, c) it's April Fucking 1st, and it's possible this is so cool … Continue reading Cribl LogStream 1.5: Now Supporting Splunk Universal Forwarder and Syslog
Context is King: Turning Ugly Logs into Rich Structured Events
Logs themselves often do not contain the necessary information in themselves to point an investigator in the right direction. Let's say I'm troubleshooting a performance issue with my application. I may want to dig through all kinds of data sources, like proxy logs, web access logs or custom instrumentation. But, in these data sources, proxy … Continue reading Context is King: Turning Ugly Logs into Rich Structured Events
Cribl LogStream 1.4: Like a Log Shaver
Part of what makes Cribl unique is our focus on the particulars of working with gritty old logs. Logs present challenges not addressed by most data processing systems: working easily with overly verbose data and formats which can be weirdly structured, nested, and hard to parse. Not only are logs noisy by throwing lots of … Continue reading Cribl LogStream 1.4: Like a Log Shaver
Using Cribl to Analyze DNS Logs in Real-Time – PART 2
In a previous post we showed how to use detect data exfiltration with Cribl in real-time. The analysis focused on checking DNS labels from DNS logs for presence of base64 encoded data. In this post we will look at several other techniques that can help security engineers add dimensions to the data to help improve the fidelity and accuracy … Continue reading Using Cribl to Analyze DNS Logs in Real-Time – PART 2
