Talk: Solving Mysterious Issues using Common Debugging Principles

Since we started our first successful execution of “hello world” during our first programming assignment to maintain or roll out complex product and infrastructure changes, debugging has become an integral partof an engineer’s journey. Lack of sufficient logging, mysterious behaviours of systems, the inability to add a live-debugger, and the complexity of components involved in a single change can all make debugging a frustrating, challenging, time-consuming, and at times a discouraging and non-rewarding process. However, knowing some common debugging principles can help you identify root-causes in a more structured, timely, and playful way.

In this presentation, I will illustrate some common debugging principles using various examples, such as from mysterious issues we faced using Spark on large scale data engineering problems at Yelp. I will share how this journey can help design test set-ups for faster debugging iterations. I will also demonstrate how these mysterious situations and debugging principles can help propel the learning process for you and your teams, as these principles can be extrapolated across most complex industry-wide systems.

Speaker:
Manpreet Singh is a software engineer at Yelp whohas led multiple streaming and batch infrastructure projects. His team isresponsible for building and maintaining Yelp’s machine learning infrastructure and data mining tools. The components owned are powered by Spark, Kubernetes, and JupyterHub, an interactive notebook search and archiving tool. These components collectively handle the building of hundreds of ML models and the publishing and processing of billions of messages per day. In his time at Yelp, he has spearheaded the efforts on multiple Kafka cluster upgrades, such as re-architecturing Kafka clusters to enable on-demand Spark cluster for ML and data mining applications, upgrading Spark to the latest versions, maintaining Spark infrastructure platform, and replacing the Spark job scheduler from Mesos to native Kubernetes. He has also mentored many engineers.