confluentthe company that built a streaming service on top of the open source Apache Kafka project, was always about helping companies capture data streams. First it was on prem, then later in the cloud. Moving the streaming service to the cloud has eliminated much of the complexity associated with managing the underlying infrastructure.
Today, at the current conference, the company is introducing a new tool called Stream Designer to make it easier to build a streaming data pipeline in a visual workflow. Users can simply plug in a set of data components to build a custom data stream, and Confluent takes care of the background coding for them, essentially moving the abstraction from infrastructure to design.
Company co-founder and CEO Jay Kreps says that while it’s still aimed at developers, its goal is to put stream design within the reach of more people than Apache Kafka experts.
“We still serve all kinds of developers, but this makes it something you can just click on and build a data pipeline that connects things, transforms data, and moves things from place to place. And it inherits and runs on the same underlying streaming infrastructure. That means all data streams are reusable, they’re all real-time, and they’re all horizontally scalable,” Kreps told londonbusinessblog.com.
He says that although the program writes the underlying code for the developer, it is still accessible to developers who need to work with it. “One of the nice things about how we’ve done this is that even though you don’t have to write any code, all the code is there. You can see exactly what he’s doing under the hood. You can see exactly the kind of transformations in SQL that we would do on this data, so it works with the existing developer toolchains as needed,” he said.
Example of Confluent Stream Designer Image Credits: confluent
For Kreps, this is all part of the evolution of Kafka, a project originally conceived at LinkedIn to move massive amounts of data. In the beginning it was a very technical venture, but over time the company has tried to make it more and more accessible to a larger number of people within an organization.
In addition to Stream Designer, the company is also announcing Stream Governance, which ensures that data is used correctly and that only authorized users can see data.
“For many organizations, it’s almost like there are two competing tensions: one is to unlock the data and use it to be effective and serve customers better and more efficiently. And the other is to lock it up and be safe and don’t let anything bad happen. And unless you have tools that help with both dimensions, you’re kind of stuck,” he said. Stream Governance helps users ensure they use the data flowing through these data streams in a secure and compliant manner.
While they were at it, Confluent announced a new Confluent for Startups program designed to let startups use the platform with free credits and access to expertise to help them get started with data streaming technologies.
Stream Designer and Stream Governance are available today as part of the Confluent cloud service subscription package, and there will be no additional charges for the new capabilities, he said.