You are viewing content from a past/completed QCon

Track: ML in Action

Location: Cyril Magnin III

Day of week: Tuesday

Applied Machine Learning track demonstrating how to train, score, and handle security and fraud use cases.

Track Host: Soups Ranjan

Financial Crime Risk @RevolutApp

Soups Ranjan heads Financial Crime Risk at Revolut, the fastest growing challenger bank in Europe. He leads the team in charge of preventing financial crime on Revolut’s platform using data science and machine learning. Soups has 14 years of experience applying machine learning to domains ranging from network security to advertising and cryptocurrencies. Prior to Revolut, Soups was the director of data science and risk at Coinbase, one of the largest cryptocurrency exchanges in the world. At Coinbase, Soups built many engineering teams from the ground up including data, risk and identity. Soups is the co-founder of, a roundtable forum for risk professionals in San Francisco and Seattle to share ideas on stopping financial crime. Soups holds a PhD in ECE focused on network security from Rice University. Soups currently lives in Berkeley with his family.

10:40am - 10:50am

When Do You Use ML vs. a Rules Based System?

When you have a hammer, everything looks like a nail. In this talk, I will provide examples of applications where machine learning makes sense and when it doesn't. I will motivate the discussion by providing examples from real-world applications in the risk domain (anti-fraud, cyber security, account takeover detection).

Soups Ranjan, Financial Crime Risk @RevolutApp

11:00am - 11:50am

Counterfactual Evaluation of Machine Learning Models

Stripe processes billions of dollars in payments a year and uses machine learning to detect and stop fraudulent transactions. Like models used for ad and search ranking, Stripe's models don't just score—they dictate actions that directly change outcomes. High-scoring transactions are blocked before they can ever get refunded or disputed by the card holder. Deploying an initial model that successfully blocks a substantial amount of fraud is a great first step, but since your model is altering outcomes, subsequent parts of the modeling process become more difficult:

  • How do you evaluate the model? You can't observe the eventual outcomes of the transactions you block (would they have been refunded or disputed?) or the ads you didn't show (would they have been clicked?) In general, how do you quantify the difference between the world with the model and the world without it?
  • How do you train new models? If your current model is blocking a lot of transactions, you have substantially fewer samples of fraud for your new training set. Furthermore, if your current model detects and blocks some types of fraud more than others, any new model you train will be biased towards detecting that residual fraud. Ideally, new models would be trained on the "unconditional" distribution that exists in the absence of the original model.

In this talk, I'll describe how injecting a small amount of randomness in the production scoring environment allows you to answer these questions. We'll see how to obtain estimates of precision and recall (standard measures of model performance) from production data and how to approximate the distribution of samples that would exist in a world without the original model so that new models can be trained soundly.

Michael Manapat, Head of Conversion Products @Stripe

12:50pm - 1:00pm

JupyterLab: The Next Generation Jupyter Web Interface

Project Jupyter provides building blocks for interactive and exploratory computing, which make science and data science reproducible across over 40 programming languages (Python, Julia, R, etc.). Central to the project is the Jupyter Notebook, a web-based interactive computing platform that allows users to author “computational narratives” that combine live code, equations, narrative text, visualizations, interactive dashboards, and other media. We will give an overview of JupyterLab, the next generation of the Jupyter Notebook.

JupyterLab goes beyond the classic Jupyter Notebook by providing a flexible and extensible web application with a set of reusable components. Users can arrange multiple notebooks, text editors, terminals, output areas, and custom components using tabs and collapsible sidebars. These components are carefully designed to enable the user to use them together or separately (for example, a user can send code from a file to a console with a keystroke, or can pop out an output from a notebook to work with it alone).

JupyterLab is based on a flexible application plugin system provided by PhosphorJS that makes it easy to customize existing components or extend it with new components. For example, users can install or write third-party plugins to view custom file formats, such as GeoJSON, interact with external services, such as Dask or Apache Spark, or display their data in effective and useful ways, such as interactive maps, tables, or plots.

Jason Grout, Scientific Software Developer @Bloomberg & JupyterLab / Sage Core Contributor

1:10pm - 2:00pm

Measuring Business Impact of Machine Learning System

I will provide an overview of how to do metric driven ML system development, primarily to answer following questions:

  • How do you bootstrap machine learning system for new objectives?
  • How do you tie up the ML system performances with business goals?
  • Do precision & recall always work as primary metrics?
  • How do you measure effectiveness of entire ML system in the context of product success?

These questions will be answered in the context of fraud detection case study.

Jevin Bhorania, Cash Data Science Lead @Square

2:25pm - 2:35pm

Machine Learning: Predicting Demand in Fashion

Apparel/fashion retailers often have to buy inventory more than a quarter in advance and so have to make bets on the total demand that they expect to see in the relevant season. Also, the set of products offered by the brands change every year, and even the historical demand for previous season’s products is known only partially as each product is carried in only a subset of the stores.

In this talk, we will show how we (at Celect) use the historical data (point of sales transaction, inventory, product attributes, product images, product descriptions) to build a SaaS solution that helps buyers and merchants predict the future demand of products for the upcoming season. The short talk will cover the real life problem statement, high level ML frameworks, and how the product is used by buyers and merchants.

Ritesh Madan, VP Engineering @celect

4:00pm - 4:10pm

Optimizing Fraud Model Thresholds @Airbnb

Like all online businesses, Airbnb faces fraudsters who attempt to use stolen credit cards. In this talk I’ll walk through how we leverage machine learning, experimentation, and analytics to identify and block fraudsters while minimizing impact on the overwhelming majority of good users.

First, I’ll introduce how we use machine-learning models to trigger frictions targeted at blocking fraudsters. Then, I’ll outline how we choose the model’s threshold by minimizing a loss function, and dive into each term in the loss function: the costs of false positives, false negatives, and true positives. Finally, I’ll walk through a numerical example comparing the optimization of blocking transactions versus applying a friction.

Dave Press, Data Science Manager @Airbnb

4:20pm - 5:10pm

Machine-Learning for Trust & Safety at Airbnb

In this talk, I will review some of the Trust & Safety challenges faced by Airbnb and other peer-to-peer marketplaces. Getting a deep understanding of the user’s identity is the foundation of trust for such marketplaces, where transactions are born online, but transition to offline and often intimate interactions. We shall cover the three crucial stages of establishing trustworthiness of a user:

(1) “verification” of the user’s identity;
(2) “screening” the past of the user;
(3) “predicting” the future risk in the behavior of this user.

We shall focus on the machine-learning challenges in each of these stages, and some of the solutions that have proven successful at Airbnb and Trooly.

Anish Das Sarma, Engineer Manager @Airbnb

2019 Tracks

  • Predictive Data Pipelines & Architectures

    Case Study focused look at end to end predictive pipelines from places like Salesforce, Uber, Linkedin, & Netflix

  • Sequential Data: Natural Language, Time Series, and Sound

    Techniques, practices, and approaches around time series and sequential data. Expect topics including image recognition, NLP/NLU, preprocess, & crunching of related algorithms.

  • ML in Action

    Applied track demonstrating how to train, score, and handle common machine learning use cases, including heavy concentration in the space of security and fraud