Jun 08 2016

Palantir’s Struggles and Big Data’s Evolution in Information Security

The media has documented some recent struggles by Palantir in retaining customers and staff. Included in this coverage is an assertion in a BuzzFeed article that:

“Toward the end of last year, Palantir killed its effort to create an elite cybersecurity group, after executives decided it wasn’t gaining enough traction… In April 2015, employees were informed that American Express had dumped Palantir after 18 months of cybersecurity work… At its peak last year, two of the sources said, the cybersecurity team included more than 10 forward deployed engineers; the majority are now gone.”

PalantirIt’s not just BuzzFeed reporting on this story. Fortune, TechCrunch and others have uncovered unflattering information about the company’s status beyond the cybersecurity realm. However, as someone that works with big data analytics applied to information security, I have more than a passing interest in this topic. Palantir was among the first to suggest analytics could make a difference for cyber defense and its success helping to uncover Ghostnet in 2009 was an early proof point.

So what does Palantir’s apparent pull-back in cybersecurity say about the prospects for analytics in protecting our information assets? Probably not much. Data suggests the issues more likely stem from Palantir’s technology and business model than the market in general.

A $2 Billion Information Security Analytics Market

Global Market Insights released a report in March suggesting that the security analytics technology segment exceeded $2 billion in spending in 2015 and is expected to grow 26% annually through 2023 when it will reach $8 billion. Given the size of the market, we have to conclude that Palantir “wasn’t gaining enough traction” for reasons other than market demand. So let’s consider some other factors.

The Challenge of Legacy Technology

Outdated Legacy TechnologyPalantir’s tools were built in an earlier era and as a result they were developed on a proprietary code base. This is a similar situation faced by many SIEM vendors that today are attempting to add analytics capabilities. The proprietary foundation inherently adds complexity to the solutions. That complexity would make it harder for organizations like Amex to bring management of the Palantir system in-house and to integrate new data sources.

Proprietary systems are also notoriously slow and/or infrastructure intensive when facing the scalability requirements of analyzing Netlfow, pcap and other large data sources. These data categories are increasingly important given that attackers are becoming more proficient at avoiding detection by traditional information security tools but cannot effectively hide their presence in the data transport layer.

A High Cost Business Model

There is another complication that results from proprietary software base. High cost. You can’t just hire Palantir programmers off the street. To run and manage Palantir, you typically need to both license the software and engage in service contracts with the company.

Enterprises know they need to analyze data more closely to buttress their cyber defenses. But we shouldn’t be surprised they are reluctant to maintain open-ended service contracts in addition to high software costs. Some companies have suggested that Palantir’s pricing can be as high at $1 million per month.

A Lower Cost Alternative

If you were developing Palantir from scratch today, you would build something much more like IKANOW. It is built on a core of open source big data analytics tools with proven scalability and is available for about 5-10% of the cost. And, you can find resources that already know how to program the ELK stack and related technologies. You won’t be locked into high cost service contracts or lack insight into the systems being run because many professionals are already familiar with these mainstream open source technologies.

The scalability required to rapidly analyze data related to information security was not entirely understood eight years ago when Palantir was founded. Nor was the need for flexibility and rapid customization fully appreciated. As the company attempts to apply legacy technology to overwhelming data volumes, it is not surprising that Palantir’s cost structure and rigidity are out-of-step with the expectations of many enterprise customers.

If you can afford it and Palantir will still entertain supporting your use case, it might be a good tool for forensic analysis. However, tools like IKANOW are designed to be faster, more affordable and provide user-driven customization options that are the hallmark of open architecture solutions. It also goes beyond forensics to automate breach detection and risk scoring. All of these features are available for a fraction of the cost and put the user in control. IKANOW just published a white paper  that analyzes the pervasive sense by analysts and managers that they lack control over their information security environment, driven in part because they are overly dependent on proprietary vendor solutions. Palantir is an example of this problem and IKANOW is an antidote.

If you would like to learn more about IKANOW’s Information Security Analytics, please click below to download our most recent white paper below or request a demo.

Request a Demo Download White Paper

Share Post
Chris Morgan

Cofounder and Chief Technology Officer. Chris Morgan is responsible for technology innovation and delivering high-quality security analytics solutions to clients. He has more than 15 years of experience in research and development, software engineering, software development and product management. Morgan studied management at the Wharton School of Business of the University of Pennsylvania and economics and computer science at Virginia Polytechnic and State University.