Datanami’s article “On the Origin of Business Insight in a Data-Rich World” discusses how dotData algorithms can excel at finding the meaningful business insights hidden in data.
During an architectural shift from on-prem Hadoop to cloud-based systems, questions abound about business insight. Big data was made affordable by the rise of Hadoop and other NoSQL technologies. The new technologies allowed thousands of companies to leverage the Hadoop clusters to a significant effect. Companies found that the new tech wasn’t as easy as expected and hired data engineers to make their big data projects work. Hadoop was a breakthrough technology, but it was too big of a hurdle for the average company to overcome due to its technical complexity. The data lake phenomenon continues on AWS, Azure, and Google Cloud, where Hadoop-style engines can be used for processing data without the hassle of managing hardware. A man who survived the dot-com implosion and survived Hadoop’s crash says we failed to learn some important Hadoop lessons.
To achieve better business outcomes, modernize the applications that can benefit from machine learning. Relational databases used a schema-on-write approach, but NoSQL and Hadoop systems used a schema-on-read approach. They were untyped and unconstrained, so developers could dump garbage in there. Big data makes data governance problems worse, which means that data governance is no longer critical. Machine learning algorithms can identify patterns and anomalies in big data, but they don’t provide business insight without context around them.
A PhD-level data scientist from dotData said his company’s AutoML-based algorithms can automatically generate business insights from raw data. Neural networks are used for big data analysis. However, companies using the AutoML version usually hand-code the models for production systems. Read the full article at Datanami.
