Why Commercial Threat Intelligence Beats OSINT

Picture this: you’re in need of a new suit. You see that your favorite store is giving away designer suits for free and the pictures look fantastic. But digging into the details reveals that they’re only available in sizes too large for you and with an awkward cut.

Do you get the free suit, knowing you will have to spend time and money trying to make it work for you? Or do you simply purchase another suit you actually like instead?

Free stuff is great! But only when it’s the right fit.

The same can be said for open source intelligence (OSINT). While the lure of “free” might be strong, the time and resources spent trying to make it work for your organization can cost more than simply buying a commercial service that works from the get-go.

Open source intelligence: How comprehensive is the data?

OSINT feeds are free, but what does that mean about the comprehensiveness of the data? If a feed is run by a company, they may not dedicate many resources to it. Paying employees and infrastructure to give a product away is noble, but it won’t be a priority within the organization. Information may be out of date, or incomplete. The same goes for a hobbyist making his/her feed available for free.

Feeds that are community-based may or may not be more comprehensive, however, they are still prone to erroneous data. Questions to ask: who is submitted the threat, is it a complete entry, is it a real concern, and is this data actually being properly vetted and monitored?

For both of these cases, the amount of threat research being done may be very limited, as opposed to a commercial threat intelligence vendor where it is a crucial business activity.

Are threats validated?

Is that threat in your feed still an active threat?

Let’s say that you find an open source repository of “bad” IP addresses. “Perfect!” you think, as you add the exceptions to the firewall, making sure access is prohibited. Every month or so, you grab the newly updated list and add the new IPs.

However, who’s checking whether these IPs are still threats? The repository owners and contributors may not be, or at least not frequently. Addresses are reassigned all the time, especially after they have been identified by providers as threats. This means you may be blocking perfectly safe IPs inadvertently and/or growing a huge ACL in your firewall that is unnecessary.

Commercial threat intelligence vendors focus on ensuring threats are validated. At Malware Patrol, for example, we verify each indicator at least daily, many of them every 4-6 hours, and our feeds are updated hourly with any new data.

How far back does your OSINT trace?

We can learn a lot from the past as we face both the present and the future. With feeds that have some history, we may even find a very old piece of obscure malware on a machine somewhere still in our network. Just because malware and other types of threats are always evolving and going out of favor, doesn’t mean old ones can’t affect you – in the right circumstances. Historical data is also very important for threat research and threat hunting activities.

Some OSINT feeds have only been active for a couple of years (and may have very limited data early on), whereas others don’t even timestamp their data so you have no idea the history of a threat. Look for a threat intelligence vendor with rich historical data.

No format consistency

While each OSINT source has their own format for their feeds, there is no consistency across different providers. This means, for example, if you had three different phishing feed sources, that you would need to first collate the data and sort it into a common format, remove duplicate data (which may not be the same across fields) as well as erroneous entries. Some sources will not provide fields that others do (such as a first encountered timestamp) which may be highly useful to your security team. Another issue is that platforms that ingest threat intelligence require the data to be in a specific format – which OSINT feeds generally are not organized into.

Re-organizing this data takes someone’s time and energy. While merging the data to a common format that can be easily consumed by your systems may not be too complicated, cleaning the data is. Trying to track down the details of missing fields is even more difficult. Even with clever automation, there is often still a need for manual cleaning.

Commercial threat intelligence vendors present each threat stream in a standard format, or formats, that can plug directly into different security platforms and systems.

Is the OSINT relevant to your organization?

As threats become increasingly more intelligent, we need to be aware of the ones that are targeting our industry, infrastructure, employees, and/or assets. OSINT feeds are generalized, not specific in their cases, which means that you can be either a) consuming a whole lot of irrelevant data or b) spending time cleaning the data to remove irrelevant threats.

Sophisticated commercial intelligence vendors like Malware Patrol offer customization of threat feeds specific to your business needs.

At Malware Patrol, we’ve been gathering, researching, validating, and cleaning threat intelligence from a wide range of sources since 2005. Please get in touch for a consultation and free evaluation of our services.