Quantum of Storage

Third year Formats and Workflows students have been joined over the last couple of weeks by Liam and Mark from Quantum to learn about modern storage solutions for large unstructured data.

It’s a fast moving field as demand continues to rapidly increase for the storage and retrieval of ever-larger quantities of data. The base requirements remain the same – that data should be stored with high availability, integrity and security – but the quantities of data and the scale of systems have been growing in recent years. This is largely due to newer infrastructures and technologies that have matured to the point that they can facilitate the rapid exchange of information across disparate locations. Now the expectation is that anything can be available anywhere and organisations are seeking systems to realise the advantages of automated workflows and remote collaborative working.

Liam was particularly keen to highlight the commercial context and ran through some common systems and architectures currently being deployed. We discussed how unstructured data isn’t just a problem for broadcasters and provides similar challenges to hospitals, meteorologists and surveillance operators to name a few. We looked at how data needs to be treated differently at different stages of its life-cycle and learned about a variety of the technologies that can be used to meet the requirements in a cost-effective way. Liam discussed Cloud Services and underlined some of the benefits and limitations. It was interesting to note that in the twelve months since we last met the chaps from Quantum, cloud services have gone from being a technology to watch, to a tool that is viable for deployment as part of a professional broadcast network. We looked at how such systems can be integrated effectively as part of a tiered storage solution.

Mark delved into more technical detail on some of the relevant technologies and directed the students to some of the bottlenecks that need to be designed against or acknowledged. The sector spends a lot of time discussing transfer protocols and how to optimise throughput and latency whilst maintaining data integrity all within constrained networks; Mark showed us that with modern network infrastructures, the issues can actually be nearer to the disc platters, where buffers and read-aheads have to be optimised to get data on and off the storage media fast enough for the network. We also spent some time looking at Artificial Intelligence and Cognitive Engines and how they can help us manage and access data to ultimately increase its value. The students were particularly intrigued by sentiment analysis, where complex data points such as speech and recognition and facial analysis can be combined to extract richer information from content.

Many thanks to Mark and Liam for taking time out of their schedule to share some of their insights with us. They were able to provide some valuable and up-to-date context and both have a truly entertaining ability to narrate such a complex field.