Can Kollective Predict the future?

Yes! Well… sort of. Kollective uses machine learning to make informed predictions about the future based on past behavior—so in a way, that’s predicting the future!

Currently, we focus on forecasting the "Bytes Received" metric in our Observability platform, with plans to expand to more metrics in the future. We perform these forecasts at four different levels:

  1. Across the entire tenant

  2. For one location with all applications

  3. For one application across all locations

  4. For one specific location and one specific application

For example, if you’re curious about how many bytes your entire tenant is expected to receive in four days, that’s where the tenant-level machine-learning model comes in. If you want to know the forecasted bytes for a specific location on Monday, we use the "one location with all applications" model. To forecast how many bytes a specific application will use on Monday across your entire tenant, we apply the "one application with all locations" model. Finally, if you want to know how many bytes an application is expected to use at a specific location on Monday, that’s handled by the "one location and one application" model.

Each prediction comes with a prediction (or uncertainty) interval, indicating how likely it is that your data will fall within the predicted range.

Using Prophet to Forecast on Time Series Data

So, how do we predict the future? We use Prophet, an open-source forecasting tool developed by Meta. Prophet is versatile and flexible, with many features that make it ideal for our needs, but we particularly want to highlight its seasonality feature.

Since our data reflects network traffic primarily during working hours, we observe significant variations between weekday and weekend traffic, as well as between daytime and nighttime traffic. Additionally, some locations or tenants may have recurring events on specific days. To account for these patterns, we integrate daily and weekly seasonality into our forecasting models. This enables the model to accurately capture reduced traffic on weekends and outside of regular work hours.

Criteria for a Model

A machine learning model is only as good as the data it’s trained on. That’s why it's crucial to have enough data to identify underlying patterns accurately. To ensure reliable predictions, we require at least 28 days (4 weeks) of hourly data, though our models can utilize up to 90 days of data.

Before data enters the model, we perform thorough data cleaning. For instance, most companies don't have events running 24/7, so there are often hours with null values. We fill in these gaps with zeros to ensure each hour in the time series has a numerical value.

Making Predictions and Detecting Anomalies

Using past data, Prophet identifies daily and weekly patterns, as well as longer-term trends, to predict hourly bytes received for the upcoming week. We assume a 99.5% upper bound to our prediction (or uncertainty) intervals as our threshold for anomalies. This approach helps us detect anomalies by flagging data points that fall outside the expected range.

Here’s an example of what that looks like. The grey curve is the data, the blue curve is the model and prediction (last 7 days in the visualization), and the dotted curve is the upper bound of the prediction interval. Any data that falls above the dotted line would be considered an anomaly.

Video data can be particularly noisy, but by using a large prediction interval, we reduce the chances of false alarms due to random fluctuations, ensuring that notifications are triggered only for genuinely anomalous events.

Conclusion

While Kollective may not predict the future in the literal sense, our use of machine learning provides powerful insights into what might lie ahead based on historical data. By focusing on the "Bytes Received" metric and employing forecasting models at various levels—from tenant-wide to specific locations and applications—we offer nuanced predictions for network traffic tailored to your needs. In this way, Kollective helps you stay ahead of potential issues by offering actionable insights for the future based on past behaviors and patterns.

Discover How Kollective Observability Can Benefit Your Business

Schedule a demo of Kollective's Video Assurance Platform today to experience firsthand how you can improve the experience of every user in your organization.

Tamra Heberling