Many organisations today are guilty of data hoarding, simply storing everything just in case it might be needed one day.
This dark data, as it has become known, is often made up of day-to-day information that is captured and stored for indeterminate periods of time, without policy or any tools to mine its value, says Bryan Balfe, enterprise account manager at CommVault.
Not only does this information offer no business benefit, it also opens up organisations to risk. In addition, with the current rate of data growth, it presents a significant challenge and associated cost with regards to storage.
Organisations need to implement effective data management policies, backed up by tools that automate this process, in order to optimise storage costs, minimise risk and unlock the potential wealth of value that resides in dark data.
Organisational data is stored for a variety of reasons, including legal and compliance purposes. However, a study conducted by IDG Research Services indicates that just 28% of organisational data is stored because it has a clear business value. The remainder of this data – the dark data – is simply hoarded for no evident reason at all.
An indiscriminate policy of data retention such as this creates a type of ‘Aladdin’s Cave’ – organisations are aware that there is treasure buried deep within, but remain unable to find anything specific or unlock the value of their information.
One of the most prevalent reasons for this is that IT has traditionally been disconnected from the users and the business, and has had no way of differentiating between important business data and other information which is of little worth. In an increasingly regulated marketplace, this is becoming more problematic.
For example, the Protection of Personal Information (PoPI) Act requires organisations to be able to present information about customers on request, and also mandates that they only store information as long as it is relevant and required. If a business has no idea what data they are storing or why, they will be unable to comply with these requirements.
In addition, in a litigation situation it is up to an organisation to prove that they do or do not have the relevant information. In a situation where data is hoarded, e-discovery becomes incredibly time consuming and expensive.
This issue is only compounded by the well-known data explosion phenomenon. According to IDC, stored data is expected to increase at a compound annual growth of 53% between 2011 and 2016, and big data is expected to see a significant growth of around 30% year on year.
With so much data constantly being created, a non-policy of data hoarding will result in storage costs spiralling out of control. In addition, without a clear idea of what data is stored and where it is located, backup and archiving becomes increasingly cost-ineffective, time consuming and inefficient.
Aside from the challenges of dark data, the other side of the coin is that this information could potentially hold significant value for an organisation if they are only able to tap into this resource. If businesses are able to extract and analyse the day-to-day information locked into their data, they will be empowered to make predictive decisions and optimise processes in a way that adds significant competitive edge.
However, in order to achieve this desired future state, enterprises must take control of their dark data. Doing so requires organisations to modernise their data management processes and solutions across the enterprise.
One of the first aspects that must be addressed is the data management policy – what are the rules for storing, managing, moving and removing data throughout its lifecycle. How long does data need to be retained? What are the organisation’s policies on the defensible deletion of information that is no longer required? Where is information stored? Are there multiple copies that need to be managed? How does an organisation create an effective audit trail that proves they are deleting the correct information and that it is removed from all areas?
Policies on how information is stored, where it is stored, why it is retained and how it flows through your organisation are an essential first step in unlocking dark data. A defined policy is critical in enforcing how data is managed and assessed. Once this is in place, implementing the tools to automate data management policies and eliminate the risks associated with doing this manually is the next step.
The old adage ‘what you don’t know can’t hurt you’ has never been more false. In today’s world, what an organisation is unaware of in terms of their own data could have serious repercussions, from regulatory non-compliance and additional expenses to loss of competitive advantage, lack of agility and an inability to make sound, fact-based decisions.
Legacy data management processes and tools are unable to keep pace with today’s demanding and dynamic environment, and as data volumes continue to increase, this problem grows accordingly.
A new approach is essential for meeting the data managements and storage challenges of today and the future – an enterprise-wide strategy that includes processes and tools to enable organisations to take control and shed light on their dark data. This in turn helps to reduce risk, improve operational efficiencies, cut costs and derive more value from data for on-going competitive advantage.
