WHAT IT TAKES TO BECOME A SUCCESSFUL AI PRODUCT MANAGER

As Director of Product Management, AI & ML at Etsy, Jyothi Nookula is in charge of all things AI at the US e-commerce platform. Holding 12 relevant patents, the Indian technologist has observed the dawn of the AI era from inside and now wants to share her knowledge and experience through her ELVTR course on Product Management for AI/ML.

In the following paragraphs, Nookula shares her thoughts on what it takes to become a successful product manager in this fledgling niche, the art of utilizing responsible algorithms, and what made generative AI the ultimate tech trend in 2023.

You are Director of Product Management, AI & ML at Etsy, so how do you implement AI/ML there?

I lead the team that builds the AI capabilities for Etsy. We build tools across all the ML development lifecycle, including features, model training, model deployment,  orchestration, evaluation, aka the whole inventory.

We handle the entire end-to-end development workflow and we provide support for ML needs at Etsy.

What skills does an AI/ML product manager need to have?

A core skill is of course an understanding of machine learning and artificial intelligence (AI). Then you have the regular product management skills, like the ability to prioritise and generate ideas.

In addition, you need technical skills: the ability to understand and translate customer needs into technical requirements. If a customer wants a problem to be solved, what does that mean from a technical perspective? Being able to constantly switch between business and technical conversations is essential.

Regular product managers don't have to do that. You don't have to talk to Java developers and talk to them in Java language to provide specifications. An AI product manager should be able to translate the business conversations into technical requirements such as what performance and model metrics will determine success and vice versa, and convert technical conversations they have with engineers into business level conversations.

Do you need an understanding of how the algorithms work?

That’s part of the technical understanding. So you need to understand AI/ML, how it works, the algorithms that are part of it, their limitations, and when to apply what algorithm, because there are so many algorithms today. Each use case requires a different algorithm. And then you need to understand how you evaluate algorithms and know what works and whether it meets your criteria.

As I said, the first thing is understanding how customer needs translate into technical requirements, and translating these into business conversations. The second is an understanding of ΑΙ/ΜL models: their capabilities, which model applies to what and how do you evaluate them. Third, you need to have an understanding of system design. How does the overall workflow work? 

Let's say you want to build a recommendation tool, like a news feed. That requires breaking the process down into smaller building blocks. There should be something to refresh, which means that you need to be able to provide recommendations to create a ranking. Then we want to show people content based on their friends, but we also want to introduce them to new stuff so that they can find something interesting, not just updates from their friends. 

You have to think the system design through, have an overall idea and break that down into smaller pieces at the system level. Finally, how do you build it in a responsible way? You need to understand the workflow your model needs, the limitations, and how it needs to be in production.

Does all that require mathematical skills?

As product managers, we need to understand the concepts, but we don't need to build the models ourselves. You don't need to understand the mathematical concepts behind how those algorithms work. But you need to know at the hypothetical or theoretical level how the algorithm works, what its inputs and outputs are and the different parameters it leverages.

Throughout our course, we talk about algorithms from a theoretical point of view, but not necessarily mathematical implementation. You don't implement the mathematical function unless you are a PhD scientist or you are looking for a particular optimization. There are off the shelf libraries for all these capabilities.

Our students are introduced to the major type of AI algorithms in a way that helps them understand how it functions, without having to know the mathematical reasons why it functions that way.

We also have demos where I show them how these models are used in an application: how you build a model, using the available libraries, and how it's applied to a particular use case, e.g. how you build an email spam classifier or an image classification model. Most of these things are not mathematical, we are just taking existing libraries and building by using code.

What’s ‘responsible AI’ and why is it important?

You want to make sure that the AI you deploy will act in the way you expect it to act. If your engineers have built a model for data sets that involve mainly male candidates, but you're deploying it in a production environment for all sexes, then the model won’t perform well against female candidates.

It's important to make sure that models are not biased. Privacy concerns have to be taken into account and the model should also be robust against attacks. There should be some transparency and control built into the model to give users some control, so that they can decide if they want to have this AI model or not.

As AI is applied in many industries, it becomes imperative for product managers who develop these products to make sure they are built in a way that is safe for users.

Does the data ever lie?

We explain during the course the different types of biases in data and what tools are available to detect these biases and remove them, or at least mitigate the problem. Not everything can be solved, and there is no totally unbiased system, but you can mitigate the bias.

You need to understand what your production distribution looks like and what’s the data distribution you have for model training. For example, your production environment might involve all users, but your data could include only male users.

We have case studies where interview attendees are mainly male and therefore the data is skewed to male. However, the production environment involves all users, both male and female. Female users might get penalized because the model wasn’t trained on data that includes females.

Understanding the data distribution, what's your production environment, the data usage, the data you have for training and whether you have a representative data sample is important.

There are techniques like upsampling where you take more samples from females during training to compensate for that bias. Or you can generate synthetic data that compensates for biased data, or launch a campaign to get extra data points, for example by partnering with other organizations to get more data points for female candidates.

How do you measure the impact of AI solutions?

There are different metrics to measure. Common techniques are precision and recall, but we will also dive into measurement techniques for classification and regression systems. For each type of model, there are different metrics.

We work through different case studies across different industries to understand whether regression or recall is the best approach. For example, in healthcare we want to make sure that our diagnosis is more precise, so that we have fewer false positives.

In an industry like advertising, we may want to prioritise recall, because precision while important is not paramount. Being precise is less of a problem. In healthcare, not being precise has ramifications. So being able to know which metrics to use in each industry is very important.

Are there cases where it’s not worth using AI in product management?

Students will learn when to use AI and when not to, depending on the complexity of the problem and the necessary computation, because AI is not cheap. It's like doing a cost-benefit analysis to understand whether costs outweigh benefits.

We also assess the complexity of the problem, whether a problem is complex enough for AI/ML. You can measure complexity by asking whether a rules-based solution could solve a problem. We have a four step framework to evaluate if a problem can be solved with AI.

For example, you need to know if you have sufficient data to build a model. Normally, you need millions of data points. If you don't have sufficient data, you may still think you need AI, but AI can't be applied. You can be creative and look for a rules-based solution while building systems to log enough data and build AI in the future. In some use cases, you have data, but you can't use it because of regulation.

Any AI/ML applications used for product management that have blown your mind?

AI/ML is like a table stake today. You use it on Google Maps or Alexa when you use Echoes, on your new phone, on Google. All that involves AI. It's not a new technology that only a few new products use. It's pervasive, all around us today. 

For each of these products there are product managers. For example, several product managers work on how to improve the experience of Google Maps. It's not something new that the world is waking up to. AI  has been around us for quite some time. It is just that these industries are now catching up to it. 

The whole generative AI space has grown in the last few months, especially with ChatGPT. It’s a product that pushed the boundaries of what we thought was not achievable. It has shown users how amazing AI can be and how far technology has come in terms of these tools having whole conversations and understanding context for users.

That has led many companies to include AI strategies in their portfolio. Several engines are being developed, tailored to particular use cases. For example, customer support with the rise of assistance bots.

What do you think ChatGPT offered that wasn’t previously available?

What drove the success of ChatGPT versus what we had before is the level of understanding of context. Before ChatGPT, context was often lost with NLP models. Especially with natural language processing, understanding context and the intent of the user was not always easy for machine learning algorithms. Chat GPT was the first product that could understand context.

And it could also remember, it has a longer memory than previous algorithms. Even today, if you ask Alexa a question and then you ask another question, it doesn't remember what you asked before. ChatGPT does remember and builds on that, just like two people conversing.

It’s also tailoring the message based on the context of the user. Users can ask for something specific, for example analysis in a pros and cons format. Tailoring content to the intent and the context of the user is something that didn't exist before.