- Poor integration
Siloed data is a leading technological problem that causes big data failures. Since data is stored in multiple sources, integrating it into one and using it to get insights that a company needs is a big challenge. This is even bigger problem if legacy systems are involved. It costs a lot of money and often does not result in the desired outcome. According to Alan Morrison of PwC, siloes create data lakes that are just data swamps. Organizations can only access a small percentage of data with little relationships that are inadequate to find patterns and get enough knowledge. Without a graph layer that interprets all instances of data mapped underneath, you have a data lake that is a data swamp.
- Not defining goals
Like any other project, big data projects require a proper definition of goals and objectives. Sadly, most people who undertake big data projects do not set goals that they need to achieve. Most of them think they can simply connect the structured and unstructured data and get the insight they need. As a project manager, you need to define the problem and develop the goals you want to attain. Having a clear definition of the problem and defining it in time helps achieve the desired goals accurately. However, many big data project leaders lack vision. This ends up confusing the company on big data projects and its desired objectives.
- Shortage of skills
There has been a widespread shortage of talent in the data science industry over the past few years. A 2018 report by LinkedIn reported a shortage of more than 150,000 individuals with data science skills. These are people such as data engineers, mathematicians, data analysts, and others. Since the field is in its initial stages, it is often hard to get people with the required skills. This slows production and ends up stalling the well-intentioned big data initiatives. Additionally, many enterprises cannot run several projects simultaneously without the right skills because they lack enough personnel.
- Lack of transparency
Lack of transparency in big data projects can result in a disconnect between technical and business teams. For instance, while the data science teams usually focus on the accuracy of models that is often simple to measure, business teams, on the other hand, are concerned mainly with metrics like business insights, profits/financial benefits, and interoperability of the final model produced. The lack of clarity and proper alignment between the teams leads to the failure of big data projects as the different teams try to measure different metrics. This is made worse by the traditional data science initiatives that use blackbox models that lack accountability and are hard to interpret, making it difficult to scale.
The above reasons for the failure of big data projects indicate the need for proper plans when implementing big data projects. The problems can be sorted by planning ahead, working together, and setting realistic goals.
