Paths for starting a big data service business

Sergey Nivens - Fotolia

Big data integration tool targets Hadoop skills gap

Information Builders has released a free developer version of its big data integration platform, which could potentially help channel companies take on Hadoop projects.

Channel partners can get a free taste of a big data integration tool from Information Builders that could help get them over the Hadoop talent gap.

The software company last week debuted a developer edition of iWay Big Data Integrator, which it is offering at no cost for six months. The tool could boost the productivity of channel partners working on big data projects and could help other partners launch businesses in the growing market, industry executives said. Big Data Integrator, an evolution of the iWay Hadoop Data Manager, initially launched in March 2017 on the Microsoft Azure Marketplace Cloud.

According to Information Builders, Big Data Integrator simplifies Hadoop, providing a graphical interface for Hadoop and Apache Spark data ingestion and transformation that reduces the need for coding. Hadoop, a software framework for storing and processing large data sets, provides a foundation for developing big data applications. Apache Spark is a data processing engine that supports in-memory computing and can run on Hadoop.

Jake Freivald, vice president of product marketing at Information Builders, based in New York, said harnessing the power of Hadoop is difficult and involves complicated programming, adding the task of managing Hadoop environments is somewhat arduous, as well.

Potential channel impact

A stumbling block for organizations -- including channel players such as VARs -- seeking to create big data applications via Hadoop has been the need to acquire specialized programming talent, Freivald noted. A VAR selling big data software, for example, might need to hire people to customize the application for individual customers.

"But the people they hire are very technical and don't necessarily understand the business requirements for customization," Freivald said. Such companies find themselves "spending more on highly paid developers who are relatively narrow in what they can do."

Big Data Integrator, however, targets more mainstream developers who understand data and business requirements, but may not be experts in Hadoop or related technologies, such as Sqoop and Flume. Information Builders' big data integration tool incorporates Sqoop for data replication, import and export, and Flume for gathering and ingesting unstructured data. Big Data Integrator also uses Information Builders' iWay Service Manager for ingesting non-Hadoop data from sources such as internet-of-things sensors.

Freivald said Big Data Integrator's graphical interface displays available data sources and reduces coding and debugging time. He said developers "can do what they need to do, instead of spending a lot of time on technical details."

Big data integration: Tool addresses skills shortage

David Menninger, an analyst at Ventana Research in Bend, Ore., who focuses on data, information and analytics technologies, suggested Information Builders' integration platform addresses the shortfall in Hadoop expertise.

"Our research shows that there is a real lack of Hadoop skills in the market still," he said, noting that one in six organizations reports having sufficient Hadoop skills.

It is really one tool in an arsenal that is much broader than the big data ingestion process.
David Menningeranalyst, Ventana Research

Menninger said Big Data Integrator can improve the efficiency of consulting firms focusing on Hadoop and big data. In addition, it can also help those service providers transfer big data projects to their customers. He said enterprise customers usually wish to assume responsibility for a project once the consultant has completed the job.

"They typically don't want to bring on a consulting firm forever," Menninger said. "These tools make it a whole lot easier to hand [a project] off."

That's because the customer organization can more readily maintain a big data application's code on its own, since Information Builders' big data integration tool doesn't require Sqoop expertise or other Hadoop-related talent, he said.

Freivald, meanwhile, said the data integration platform can also help channel partners move a big data application from one environment to another. For example, a VAR that built an application that uses Cloudera's Hadoop distribution may want to replicate that app for a customer that uses the MapR Technologies Inc. Hadoop distribution. "The Big Data Integrator-based processes would remain essentially the same," Freivald said.

In addition, Big Data Integrator can potentially lower the Hadoop skill barrier that prevents consulting firms and other channel companies from entering the big data sector, according to Menninger.

"This is a way for them to get involved in the market and participate," he said.

Channel companies and other organizations may consider Big Data Integrator a useful tool for simplifying data ingestion in Hadoop environments. But Menninger said an exclusive focus on an integration platform misses the bigger picture, citing Information Builders' portfolio of data management and analytics products.

"It is really one tool in an arsenal that is much broader than the big data ingestion process," he said of Big Data Integrator. "Hopefully, [customers] recognize more issues than just ingesting the data. If all you are doing is ingesting the data, you are probably setting yourself up for failure."

Next Steps

Learn how a university used commercial tools to stock a Hadoop data lake

Read about data governance products for Hadoop environments

Gain insight into the task of managing a data lake

Dig Deeper on Emerging Technology Markets