Scramjet: Shaping the Future of Data Pipelines
With the spread of IoT and edge devices, data collection and processing to provide better services have become omnipresent today. But while a lot of the data collection can be done on edge, a lot of the heavy-lifting e.g. for training a deep neural network, is done in the cloud, i.e. on a cluster of a data center. But how do you get your data there?
To avoid all the struggle in dealing with APIs and lots of processing scripts, the startup Scramjet develops a framework of the same name that handles data pipelines end-to-end – from data collection to processing. Founded by a dream team of five, Wojciech Sacha, Natalia Czapracka, Michał Czapracki, Barbara Koźbiał, and Rafał Stokowski, it raised a pre-seed round in December 2021 by Sunfish Partners, SpeedUp Venture Capital Group, and various angel investors.
Learn more about the future of data pipelines from our interview with the CTO Michał Czapracki:
Why Did You Start Scramjet?
Some companies and even non-core branches of big organizations are simply too small to set up an entire data infrastructure for a new project – it would simply take too much time and effort. Instead, they do simple scripting, and so did I when working as a project manager and head of video technology previously at the media company Agora, responsible for the VoD service.
Studios had provided us with content, and we wanted to send them back usage reports, which involved parsing a lot of log files, as I needed the data in a spreadsheet. So I started developing what has become the Scramjet framework out of all the scripts I had written. I put it up on GitHub, and the downloads started coming in – thousands of downloads! Seemed like many people were facing similar challenges around data acquisition and transformation.
Around the time that it reached a hundred thousand downloads, I decided to build a startup around this open-source project. I reached out to five of my previous colleagues, and from there, it happened pretty organically: we didn’t think too much about the download numbers and more about the problem our framework solved. How could we allow people to deploy data processing scripts within a few seconds in a serverless manner? From this came the idea of the Scramjet Cloud Platform.
How Do Data Pipelines Work?
Currently, when building a data pipeline, you need to patch many different parts together, such as APIs and data processing scripts. We make this simple with our end-to-end data transformation platform – and since it’s end-to-end, we also include parts outside the core pipeline, like data acquisition, e.g. gathering data from sensors. AWS can process your data only if it’s already uploaded to the cloud – we get it there, and you can still use AWS.
But you can also use our Scramjet Cloud Platform for data processing, even on a Raspberry Pi, using our self-hosted Transform Hub. This is a game-changer for local processing, edge, and IoT applications. Usually, you don’t have the resources to do a lot of advanced machine learning on edge. But you can identify which data is confusing the neural network you have deployed, transform it into the right format, and send it back to the cloud. There, your neural network can be updated and then pushed back to the edge.
How Did You Evaluate Your Startup Idea?
Evaluating your startup is a never-ending story – you need to do it constantly. I built Scramjet because I wanted to make my own developer’s life easier and I knew friends and colleagues from the industry who could benefit from a platform like this. Of course, there are other solutions, but we’re the only one providing truly end-to-end services, and we aim to make our platform the simplest on the market.
We also applied for the Fast Track program by the European Innovation Council, where we had to fill in 80+ pages of forms and create a really extensive and solid market survey – and in the end, we were one of the few startups that won this EU grant.
We started reaching a lot of companies by outbound sales and getting inbounds from marketing – by now, we got over 50 meetings with potential clients. We are constantly refining our story, so we get a decent success rate out of these meetings and capture people’s interest. You always need to find the right person to talk to – every no brings you closer to a yes.
On a final note, this is a very exciting point in time for Europe: Through initiatives like GAIA-X and companies like Scaleway or Scramjet, we’re revolutionizing the cloud in Europe, based on European values, creating a massive cloud market and an ecosystem for the next generation of hyperscaler startups. We just need to focus and get to work!