Have your cake and eat it too ... a post about how a fast feedback loop enables startups (or agile teams) to deliver the best value to customers. Some background We started working on Cribl LogStream almost exactly 1 year ago (early July 2018). In this time frame, we've managed to GA (Oct 2018) and … Continue reading Going native
Engineering Deep Dive: Streaming Aggregations Part 2 – Memory Optimization
With Aggregations in Cribl LogStream 1.6, you can send your logs directly to Cribl and shape, extract, filter, drop, and now, aggregate! This new, powerful capability allows users to easily transform logs into metrics while running real-time, tumbling window aggregate functions on them. If you missed Part 1 on aggregation time bucketing, feel free to … Continue reading Engineering Deep Dive: Streaming Aggregations Part 2 – Memory Optimization
Engineering Deep Dive: Streaming Aggregations Part 1 – Time Bucketing
With Aggregations in Cribl LogStream 1.6, you can send your logs directly to Cribl and shape, extract, filter, drop, and now, aggregate! This new, powerful capability allows users to easily transform logs into metrics while running real-time, tumbling window aggregate functions on them. In this post I would like to share various engineering problems/challenges we … Continue reading Engineering Deep Dive: Streaming Aggregations Part 1 – Time Bucketing
Cribl LogStream 1.6: Logs to Metrics (Prometheus, Statsd, Graphite, Splunk Metrics)
Very early on in my career, I worked for AT&T Wireless, before it became Cingular and then AT&T Wireless again. As a young engineer, I remember running across various technical situations I couldn't explain. Why can't we connect this network to that one if there is a legitimate business need? Why must we always allocate … Continue reading Cribl LogStream 1.6: Logs to Metrics (Prometheus, Statsd, Graphite, Splunk Metrics)
Getting Smart and Practical With Dynamic Sampling
In the past we've written multiple posts about how Cribl helps you maintain visibility in high-volume/low-value scenarios without having to egregiously scale your analytics infrastructure. This problem usually stems from the fact that machine data emitted by your infrastructure is not all created equal. Some events are way less valuable than others but yet they consume … Continue reading Getting Smart and Practical With Dynamic Sampling
Streaming Data Deduplication with Cribl
The Problem It's not uncommon for machine data systems to send and receive duplicate or repeated events. This could be due to a variety of reasons, for example; Misconfiguration (a.k.a layer-8 problems) on the source, intermediary or aggregation systems may cause duplicate data to be sent out. Bugs or software defects may cause a data source to occasionally … Continue reading Streaming Data Deduplication with Cribl
Trimming Unnecessary Fields from Logs
The author of a log has very different motivations from the consumer of that same log. For the author, they must conceive of all the use cases this data may be useful for and include information which will be relevant both now and in the future. The author must ask a number of important questions … Continue reading Trimming Unnecessary Fields from Logs