Estimated reading time: 2 minutes, 55 seconds

What Happens to Big Data Projects   Featured

What Happens to Big Data Projects    Octavian Dan

Big data is fast gaining momentum, and so are the big data projects. Companies are increasing in size and ambition. However, the rising number of big data projects does not mean that they all succeed. Gartner estimates that the number of big data projects that fail is about 60 percent in 2016. In 2017, Gartner revised the number of big data projects that fail to be about 85 percent, which was even higher than the rate stated before. Nothing has changed since then. Even in 2021, the rate of failure still ranges at around 80 percent. Here are some of the reasons why big data projects fail.

  1. Poor integration

Siloed data is a leading technological problem that causes big data failures. Since data is stored in multiple sources, integrating it into one and using it to get insights that a company needs is a big challenge. This is even bigger problem if legacy systems are involved. It costs a lot of money and often does not result in the desired outcome. According to Alan Morrison of PwC, siloes create data lakes that are just data swamps. Organizations can only access a small percentage of data with little relationships that are inadequate to find patterns and get enough knowledge. Without a graph layer that interprets all instances of data mapped underneath, you have a data lake that is a data swamp.

  1. Not defining goals

Like any other project, big data projects require a proper definition of goals and objectives. Sadly, most people who undertake big data projects do not set goals that they need to achieve. Most of them think they can simply connect the structured and unstructured data and get the insight they need. As a project manager, you need to define the problem and develop the goals you want to attain. Having a clear definition of the problem and defining it in time helps achieve the desired goals accurately. However, many big data project leaders lack vision. This ends up confusing the company on big data projects and its desired objectives.

  1. Shortage of skills

There has been a widespread shortage of talent in the data science industry over the past few years.  A 2018 report by LinkedIn reported a shortage of more than 150,000 individuals with data science skills. These are people such as data engineers, mathematicians, data analysts, and others. Since the field is in its initial stages, it is often hard to get people with the required skills. This slows production and ends up stalling the well-intentioned big data initiatives. Additionally, many enterprises cannot run several projects simultaneously without the right skills because they lack enough personnel.

  1. Lack of transparency

Lack of transparency in big data projects can result in a disconnect between technical and business teams. For instance, while the data science teams usually focus on the accuracy of models that is often simple to measure, business teams, on the other hand, are concerned mainly with metrics like business insights, profits/financial benefits, and interoperability of the final model produced. The lack of clarity and proper alignment between the teams leads to the failure of big data projects as the different teams try to measure different metrics. This is made worse by the traditional data science initiatives that use blackbox models that lack accountability and are hard to interpret, making it difficult to scale.

The above reasons for the failure of big data projects indicate the need for proper plans when implementing big data projects. The problems can be sorted by planning ahead, working together, and setting realistic goals.

Read 1214 times
Rate this item
(0 votes)
Scott Koegler

Scott Koegler is Executive Editor for Big Data & Analytics Tech Brief

Visit other PMG Sites:

click me
PMG360 is committed to protecting the privacy of the personal data we collect from our subscribers/agents/customers/exhibitors and sponsors. On May 25th, the European's GDPR policy will be enforced. Nothing is changing about your current settings or how your information is processed, however, we have made a few changes. We have updated our Privacy Policy and Cookie Policy to make it easier for you to understand what information we collect, how and why we collect it.