How to Measure Buzz? OTT Data Sets and Media Audiences in India
Ishita Tiwary / Concordia University

Film Companion piece on most streamed shows released in India, done in partnership with Ormax

Recently, the media criticism platform Film Companion partnered with Ormax on a regular feature which listed the five most viewed Hindi TV shows and movies on Indian streaming for the week.[1] The feature notes that “streaming platforms are notorious for their lack of transparency allowing them to quote numbers that are tough to verify”.[2]  They thus partnered with Ormax media, a consulting firm that has been keeping a database since 2018-19, amassed a large dataset, and hence was a more reliable metric to measure “success”.[3] These metrics included: Viewership Patterns, Marketing Buzz, and Content Strength. Crucially, the buzz score was defined by relying on audience recall of the show.[4]

How does one measure buzz? How does one measure success? In the past four years of my research on OTT platforms in India, my respondents have often been stumped by this question. The most common answer I got was that they measured success through social media buzz — how many people are tweeting about it or sharing it on Instagram and Facebook? Ien Ang notes that “studying media audiences is not interesting or meaningful in its own right, but becomes so only when it points towards a broader critical understanding of the peculiarities of contemporary culture.”[5] Here, the peculiarities are the ambiguous parameters of buzz and of measuring success on OTT. In the first post, I had written broadly about what Ormax does and how it intervenes in the space between data, audiences, and various kinds of media in India.[6] In this post, I specifically focus on streaming platforms and Ormax’s methodology in measuring buzz and success.

Keerat Grewal, the head of Ormax’s streaming division, outlined their data collection process in an interview to me. While the discussion was vast and detailed and touched upon various aspects of media audience research vis-à-vis streaming in India, for the purpose of brevity, I will contain myself to the collection of data.

Ormax Media has built their own dataset, has conducted their own consumer research, and has built tools for testing content. As Grewal stated during our conversation,

how do you talk about how well an OTT show has done? In fact there’s been no data on that as well. Platforms have their own data but today Netflix top performing show may be around say 5 million or 6 million viewership but how do they know what’s the top performing show on a Disney Plus Hotstar doing?… So from 2020 we started publishing the OTT viewership estimates. [7]

Grewal, 2023

To measure this, Ormax conducts weekly surveys amongst OTT audiences, “so every year we do the OTT sizing study. Based on that you identify your age, your gender, your market’s representation.”[8] Next, they identify which participant has watched what content. One of the problems they have identified is that although one knows the size of a platform subscriber base, the difficulty is to ascertain not the number of subscriber accounts but the number of people watching the content.

Studies on internet research acknowledge the diverse metrics available to measure audiences. However, this diversity coupled with the rapidly changing landscape of the internet make it difficult to ascertain what is true and for how long. Moreover, in a complex online sphere it also becomes difficult to measure biases and interferences.[9] As Turnbull notes, none of these methods, however, are entirely without problems since they usually involve some form of interpretation and/or self-reporting.[10)] Therefore, in a rapidly changing online sphere, where OTT platforms are not transparent about their data, how do third parties like Ormax design and conduct their research?

The participant selection criteria is defined in terms of age, gender, and viewership. For OTT audiences, Grewal points out, it is crucial to note their subscription to platforms and what content they have been watching in order for Ormax to define them.[11] This infrastructure of data collection is established by Ormax having their own call center in Surat, Gujrat which employs around 50 people. For OTT platforms, the participants are found online through their Ormax Online Panels (OOP) that recruit participants. They also have agents in the field who reach out and vet the participants as well.  As Grewal states,

because it’s our own system, we’ve got algorithms running in the back end which can access and see any kind of discrepancies in filling behavior. Maybe they’ve taken too long or too less time to fill in. You can junk clean much easily. And then there are still some projects which are done face to face on field. So because you don’t find these audiences very easily either. Telephony like kids, you can’t access them without parents. So all the work that we do for kids is done face to face. So sometimes when we do studies where you actually go to the house of a kid with the parent over there, you’ve got a pen and paper questionnaire and you fill it face to face.  Even rural India is done face to face. So yeah, these are the various methodologies that we approach. [12]

Grewal, 2023

On their website, Ormax states that they currently measure viewership for Hindi and English language content and international content.[13] The cumulative sample size is 5200+ and viewership is measured by an estimate of a number of unique people in India who watched a show (at least one full episode) or a film (at least 30 mins).

Data and explainer that is available on Ormax website that explicates their research on media audiences in India

There has been some critique of Ormax’s methodology.[14] The sample size is not robust for an audience study in India, which should be 100,000 to be representative. Issues of representation also came up in my conversation with Kishwar. At this point, it is important to note that reliable internet connectivity is not given in a deeply unequal and diverse country like India. The “digital divide” separates more affluent communities from those who are less privileged. In our interview, Grewal emphasized as Ormax’s approach is Pan Indian and the attempt to go to all states, although some states such as Jammu and Kashmir and Ladakh are excluded as they are difficult to access.[15] The urban-rural divide also figures here. Rural populations, claims Grewal, are difficult to track daily as some may not have mobile phones or “they’re not very articulate or literate. So if you call them up, you can’t really speak to them.”[16]  Through my interview with Grewal and research, I was able to find out about Ormax’s infrastructure and methodology for data collections. I have still not been able to get clarity on the type of questions posed (structured/unstructured) and the details of how the participants are recruited, which strata of society they belong to, and what is their demographic breakdown. However, there is enough information to argue that Ormax’s sample size is not robust, is not representative and is biased in favor of urban, presumably middle to upper class Indians (as they have access to reliable internet connectivity and mobile phones).

Audience report from Ormax Media’s website

It is also very important to note that Ormax  Media now dominates industry decisions based on data whether it comes to content, casting, marketing budgets etc. For example, in an interview which is a decade old, their CEO, Shailesh Kapoor made the claim that the firm tested a show three times over a period of six months.[17 ] The testing demonstrated issues with the casting of the female lead and narrative treatment but the basic idea tested well amongst audiences. The channel trusted the data and changed the female lead and the director. The show was successful. Having now transitioned to a media consulting firm, Ormax claims that 70 percent of their meetings with CEOs, CFOs etc have a boardroom impact, stating, “we have liberated ourselves of the distinction between research and consulting, or between various types of researches for that matter. All that matters is the impact on the client’s business.”[18] The major influence of Ormax on the industry  is at odds with the narrow scope of its methodology. A dataset that has not been made public and is perhaps as secretive as other OTT platforms, although they do have explainers on their website, a report available for download, and welcome queries through email. While the role of Ormax is novel in the Indian landscape in the way media producers are finally acknowledging the importance of research to making decisions, it is also important to critique this mode of research in order to facilitate the creation of content that is more representative of a diverse country like India.

---

Image Credits:

1. Film Companion (author’s screen grab)
2. Ormax Media OTT viewership estimates (author’s screen grab)
3. Ormax Media Insights’ Audience Report (author’s screen grab)

References:

1. Team FC, “Ormax Media 2023 Report: Farzi, Citadel, and Big Boss Come on Top.” Film Companion, last modified January 16, 2024. https://www.filmcompanion.in/features/indepth-stories/streaming/ormax-media-2023-report-farzi-citadel-and-big-boss-come-out-on-top [↩]
2. ibid [↩]
3. ibid [↩]
4. ibid [↩]
5. Ang, Ien. Living Room Wars: Rethinking Media Audiences for Post Modern World (New York: Routledge, 1996), 4 [↩]
6. Tiwary, Ishita. “The Rise of OTT Analysis in the Indian OTT Market: The Case of Ormax Media.” Flow Journal 30, no. 3 (2023), accessed April 7, 2024, https://www.flowjournal.org/2023/12/the-rise-of-data-analysis-in-indian-ott-market/. [↩]
7. Keerat Gerwal (Head of Ormax streaming division) in discussion with the author, May 2023 [↩]
8. ibid [↩]
9. Mytton, Graham, Peter Diem and Piet Hein Van Dam, Media Audience Research: A Guide for Professionals (Los Angeles, CA: Sage Publications, 2016 [↩]
10. Turnbull, Sue. Media Audiences: Is Anybody Watching? (Bloomsbury Publishing, 2020 [↩]
11. Keerat Gerwal (Head of Ormax streaming division) in discussion with the author, May 2023 [↩]
12. ibid [↩]
13. “Ormax’s OTT viewership estimates: An explainer.” Ormax, January 23, 2024. https://www.ormaxmedia.com/insights/stories/ormaxs-ott-viewership-estimates-an-explainer.html [↩]
14. Gupta, Soumya. “How Ormax made it”, Medium, March 31, 2023. https://daily.thesignal.co/p/ormax-koffee-with-karan-samantha-prabhu [↩]
15. Keerat Gerwal (Head of Ormax streaming division) in discussion with the author, May 2023 [↩]
16. ibid [↩]
17. “The Past, Present & Future of Ormax Media”, MXM Media , June 17, 2017. https://www.mxmindia.com/interviews/the-past-present-future-of-ormax-media/ [↩]
18. ibid [↩]