I am constantly surprised at how much our organization continues to learn every year. If you had asked me four years ago what our product vision was and how we planned to achieve it, I would have probably shown you a very different technical approach compared to what I would show you today.
The approach we initially took with Pulse development focused on using open source technologies, which allowed us to construct our product capability at a low cost; however, this approach required experience and overhead in running big data analytics frameworks and knowledge as to how weave the technology that drives Pulse into it. Looking forward to today, we’ve completely flipped that idea on its head. Now, we are focusing on building a robust product offering that is enabled by technologies we don’t have to maintain.
Both designs (i.e., technology-focused vs. product-focused) would have achieved the same goal—to engage hard-to-reach populations and analyze large open data streams from novel areas—however, the tools and technologies we use to arrive at that goal have drastically varied over time.
Three to four years ago, we were introduced to some big data contracts, which allowed us to see what it would be like to work with data that could take up many terabytes worth of space. The following summarizes the kinds of applications we considered integrating into Pulse at that time:
- Apache Hadoop – A fundamental application for large scale batch data analysis
- Apache Hbase – A structured database built upon Hadoop
- Apache Storm – A stream processing framework for real-time data processing
- Apache Kafka – A message bus framework that allows high-volume producer/consumer asynchronous applications
- Apache Spark – An in-memory batch analytics framework
- Redis – An in-memory caching database
- Elasticsearch – An unstructured search and analytics engine for querying our data
These types of technologies, and many other Apache projects, are still critical to the success of Pulse and its supporting objectives. While our platform has become leaner based on the needs of our customers, the patterns learned working with many of the above technologies continue to shape our engineering view.
For example, we learned that running some of these projects at large scales requires a full-time team to maintain them. We found ourselves spending time monitoring, fixing, and updating the projects we use instead of innovating, which is essential in today’s accelerated tech environment. While that burden continues to wear on legacy systems, it has encouraged us to integrate technologies and products that allow us to focus on product rather than overhead.
Since that shift, a number of emerging technologies have reshaped our landscape internally.
- Docker – Allows us to place our micro-services inside a shippable container; product deployment has never been easier
- Lambda – Functions as a service, so you don’t have to worry about servers, just how your function executes
- Container Orchestration – Technologies like Kubernetes, Mesos, and Docker Swarm allow you to run your containers without thinking about the underlying hardware they sit on
These technologies (and many more) and how we can use them to exceed the needs of our customers are what excite me about the future of Pulse. The myriad of products offered “as a service” that we can integrate continues to expand, which is great news. This allows us to focus on making Pulse better, rather than on care and maintenance of the services within.
Determining where our focus should be has been one of the hardest lessons to learn, and it is always a juggling act when utilizing new technologies and frameworks. Do we run the open-source project on our own and spend engineering resources building and maintaining it? Or do we outsource the technology stack at a (perhaps higher) cost and focus on making our product better? Sometimes, those outsourced dollars actually save money and allow the team to focus on product instead of maintenance.
As we continue to plan for the next technology jump in our Pulse Platform, our eyes remain fixed on making both our customers happier and our internal workflows easier.
– Madison B. (CTO at IST Research)
Have questions or want to discuss more? Contact us at firstname.lastname@example.org