Recognizing commercials using the Alphonso API

Liberty Global organized the Hack & Play Appathon in Ziggo dome on September 15th and 16th. More than 20 teams of hackers, designers and programmers were invited to create an app or a game for the Liberty Global product Horizon set-top box. Team Trifork joined with Dennis de Goede (Design & Frontend), Tony Abidi (Devops) and myself (Front & Backend).

Alphonso added another challenge to the appathon: Create the best integration with the Alphonso platform. Integration challenge? Sounds like a Trifork challenge to me.

Alphonso

Alphonso turned out to be a sound-analyzing solution that matches sounds to pre-indexed sounds. In this case they index live TV shows and commercials, so we could listen to the TV and get extra information about the broadcast and commercial. Problem one: to listen we need a microphone. Problem two: the Alphonso SDK was available for Android, iOS and in low level C. Not something you would use in a JavaScript environment provided by the Liberty Global team…

Horizon

The environment we got to work on was basically a Raspberry Pi 2 with a nightly build of a Webkit browser on it. Fast enough to stream the HD streams provided by the Horizon.tv site, but nothing fancy. Also provided was a JavaScript framework (Metrological Application Framework or MAF) that could be used to show apps in the Horizon Box environment. MAF contains a Rooms API which is basically an API around a Websocket connection to one of the Metrological servers.

The Concept

Who enjoys commercials on television? Hardly anyone we spoke to, including ourselves. On the other hand, who dislikes funny kitten movies? Again. Hardly anyone, right! So we created our concept: KitMercials.
We would use Alphonso to listen for commercials and trigger our app to display funny YouTube videos during the commercial. The live TV feed would still be displayed in the top corner and extra information about the commercial could be displayed as we received it from Alphonso.

The frontend would be responsible for the creation of the Room, the websocket connection. Then, we would create an Android app that must discover the room and connects to it. The app then listens to the sounds from the microphone and sends it to Alphonso for analysis. When Alphonso recognizes a commercial (which it does very good I must say!) the app gets the extra data like brand, description and optionally logo, website info etc. The app sends this data to the frontend which can trigger the KitMercial interface and starts playing the YouTube videos selected.

The result

Within 24 hours after the start we were live. Live stream on one of the Dutch commercial channels and automatically the kittens were on screen during commercials. And the crowd loved it.
With the excellent support from John Gee and Naren from Alphonso the first price was ours to get and Alphonso agreed: the first price in the integration category went to Team Trifork!