Here we explore a new feature from Pandora which allows artists to connect with fans via Artist Affinity Targeting. This article explores how the new feature benefits both artist and listener, how it was tested and perfected.
_____________________________
Guest post by Katrina Glaeser data scientist at Pandora
The Creator Services team was finally ready to pilot our Artist Audio Messaging feature, where artists could connect to fans in their own voice. Back in 2014 this was something completely new for any digital streaming service, we needed to be sure the impact to our listeners was positive and personal.
Artist Audio Messages can optionally include a link to an artist’s website or ticket purchasing site — something that was never possible to do with track introductions on AM/FM radio. Would listeners hear a “Check out my new album” message from an artist and confuse it with an advertisement? After a long period of working with partner artists and testing pilot messages, we ran some successful experiments to understand the effects before we happily released Artist Audio Messaging into the wild. Since then we’ve continued working to help artists at any career stage reach their audiences.
Our initial concern about the listener reaction meant that we were very cautious with how we targeted artist messages to our listeners. We wanted to be sure the listener knew why they were getting the message, so we targeted messages mainly to station owners. That means, in order to hear a message you needed to have started a Pandora station with the artist and you needed to be online listening to it. This turns out to be a great strategy for bands that already have a strong fan base and plenty of name recognition, but we were missing something important — all of the other artists that make up Pandora.
Consider an emerging artist who has only released a one track – even if that track is popping on one of our new music stations, this artist is unlikely to have a strong base of station owners. Imagine that our emerging artist with this new single is also just about to release an album, and then announce a tour, how should they reach their fans? Listeners don’t always express their fandom by adding an artist station, or even through thumbing, so we needed something better.
We love to personalize things here at Pandora; in this case personalization means thinking about the artist as well as the listener. We needed to include data about the artist’s life stage so here are some the questions we answered about each artist.
There are listeners like us, who listen all day, who have tons of thumbs up, and a tuner full of artist stations, but there are also many listeners who use the other kinds of stations we offer like the wildly popular Today’s Hits station, or some of our mood, activity and genre themed stations. There are also other listeners who rarely thumb songs up or down. We wanted to be able to help artists reach those kinds of fans as well, so we needed to incorporate information about the listener that put their behavior in context.
Then of course Pandora has an amazing amount of data about how our listeners interact with specific artists so we ask questions like:
This feature set can tell you a lot about the relationship between a listener and an artist on Pandora, but if you are a data scientist, the really important question to ask is “How to do you train and test a model?” For the training set, we used data from a set of audio messages that we boldly released with very little restrictions. Anyone who heard a song by one of these artists in the test period was eligible to hear the message before or after the track — not just station owners. We had enough of this data to represent the full distribution of listeners across Pandora and artists across different career stages.
To train and test the model, we used information about whether the listener clicked on or skipped the message as positive or negative signals respectively, similar to the way we think about thumbs up and down for songs. We already had some ways to target the messages, namely station ownership, so for the model to be adopted it had to do a better job of predicting click-throughs versus skips compared to a model based on the features that informed our initial targeting.
Even with a good variety of artists in our test set, we had, at the time, close to 200,000 artists spinning on the radio. Was this model, trained on a small set of artists going to perform well when applied to all other artists? While the initial instinct in evaluating the model is probably to test how the prediction did on a random holdout set of impressions, instead we evaluated the model on a set of randomly selected holdout artists. The model performed well on artists that weren’t represented in the training set, this meant we could comfortably generalize these results to all artists.
Once the model selection and parameter tuning process was complete, the model was quick to implement in production thanks to our in-house machine learning framework Savage ML. We computed the predicted probability that an individual would click, as opposed to a skip, an artist’s message and called that an Artist Affinity Score (patent pending).
This score has been successful in our initial testing, both at increasing the reach for small artists and maintaining the relevance and click through performance that we would expect. For example, it allowed artist Ledisi to reach an extra 7% of her audience compared to our broadest targeting option. With this additional 7% reach, she got a 12% boost on clicks-through for on one of her tour announcement messages. Overall, 21% of the listeners in high Artist Affinity segments would not have been reachable with our previous targeting methods.
This score has turned out to be, not only a great targeting mechanism for audio messages, but the right way to abstract a concept of fandom. The score has proved to be useful across many teams and projects and is currently being used to power Artist Audio Messages, entertainment industry advertiser segments, off platform marketing campaigns, new music recommendations and other projects still in development. One of the most rewarding parts of our work is getting to help new teams use this model to jump start a new recommendation or targeting project.
They say that necessity is the mother of invention, but in this case it was a combination of love for our listeners and our desire to connect artists of all sizes to fans that made this work — that and our great internal (soon to be open source) tools. Seriously, when it comes out, go check out SavageML.