A Metanalysis of Film Program Rankings in the U.S. – Spreadsheet of University Film Programs Rankings
“Obviously there is no scientific formula to these prizes; it’s not like the movie that earns the most money at the box office or winds up on the most critics’ top ten lists is declared the official winner. The Oscars are based on the votes given by the members of the Academy of Motion Picture Arts and Sciences. As such, they are at the mercy of the whims, moods, and questionable taste of human beings.”
This the description of Matt Singer’s analysis of the Oscar award-winners in ScreenCrush. However, it could just as easily be used to describe the rankings for the film programs at universities.
“Obviously there is no scientific formula to these rankings; it’s not like the university with the best film program graduate starting salaries is declared the top program. The best film programs are based on the votes the people who publish the rankings decide to count. As such, they are at the mercy of the whims, moods, and questionable taste of human beings.”
In my current work for InGenius Prep (IGP), I helps high school students apply to and get accepted to some of the top universities in the nation. As part of work there, I am asked to review sample high school student profiles and identify the top 10 universities to where they should consider applying.
I was reviewing a student’s profile today who wants to go to one of the best film schools in the U.S. I wondered what the college rankings thought about the best film programs in the nation. In addition, I have a son studying film so I to spend a few hours this afternoon reviewing top film program rankings, evaluating their “methodologies” and attempting to create a composite ranking.
Of the five most prominent film program rankings, I thought the best methodology was provided by NICHE. They include several mentions of specific statistical analyses. Unfortunately, they did not provide an overview of the factors used. An overview of their ranking methodology is below.
The second-best methodology description, in my opinion, was done by The Wrap. Although their explanation was not as thorough as Niche’s, The Wrap described their process and provided the names of two people who reviewed the collected data and made the final ranking determination. These names are rarely provided.
Since both NICHE and The Wrap indicated not only a quantitative approach to their rankings, but also provided some description of their approaches, I decided to doubled their value when integrating them with the other rankings.
I ranked Metaflix’s film program rankings 3rd because although they provide little information about their process, they at least indicate to us that they have a thorough process. They mention using a proprietary algorithm used to order the colleges. When I hear words like “proprietary algorithm,” I question the lack of transparency and use of complex vocabulary to obscure what might be a simple process. It reminds me of famous “secret sauces” that often turn out to be large amounts of sugar.
Of the five rankings, the only one which I found fully transparent in their methodology was the U.S. News rankings. However, they only ranked programs with at least seven nominations, and they made it clear their rankings were based completely on the perception of the 29% of the fine arts academic program leaders who responded to their survey. Only nine film programs received at least seven nominations, so I gave each of these nine schools a bonus. Although I am thankful that U.S. News shared that their data was collected in 2020, this further cements my placement as 4th best.
Last, and least, would be The Hollywood Reporter’s film program rankings. They provide no criteria for their methodology and yet when you do an internet search for film program rankings, they have figured out how to make it so that their likely unscientific results are typically found first.
Stated Methodology for 5 Most Prominent Film Program Rankings
#1. Niche Criteria – 2023 (no date for when data collected)
- Factors represent balance between statistical rigor and practical relevance
- Factors built from Niche user data and student-submitted surveys based on aggregated data/responses across each school. Have a higher degree of confidence in the aggregated score for colleges with more responses, so a Bayesian method was applied to reflect this confidence.
- Factors built from factual information were inspected for bad data including outliers or inaccurate values. If necessary, this data could be adjusted or excluded.
- Produced a standardized score (z-score) for each factor at each college. This score evaluated distance from the average using standard deviations and allows each college’s score to be compared against others in a statistical manner.
- Assigned weights for each factor to ensure that no one factor could have a dramatic positive or negative impact on a particular school’s final score and that each school’s final score was a representation of the school’s performance.
- Weights determined by a) analyzing how different weights impacted the distribution of ranked schools; b) Niche student user preferences and c) industry research
- Overall score calculated for each college by applying the assigned weights to each college’s individual factor scores.
- Colleges missing data for 50% or more of the factors (by weight) and colleges that had at least 50% of the factors (by weight) but lacked one or more of the required factors were not included.
- Colleges that had all required factors and 500 or > full-time undergrad. students were eligible.
#2. The Wrap Criteria – October 31, 2022
- Reached out to every school listed and collected data points on everything from class size to student body diversity to scholarships to networking opportunities, along with updates since last year’s rankings on new facilities, new faculty and new programs.
- Reached out to a number of former and current film school deans and other experts for their unvarnished, off-the-record input on which schools they thought were performing above or below expectations.
- Sifted through that information, added a few other criteria — like the alumni each school generates — and crunched the numbers
- Rankings by: Executive Editor, Awards Steve Pond and Deputy Magazine Editor Steve Root
#3. Metaflix Criteria – 2023 – (no date for when data collected)
- Review data including consulting with variety of educators, industry professionals, & experts
- The data is then categorized and run through a proprietary algorithm.
#4. U.S. News Best New Media Criteria – 2020 (probably done every few years)
- Based on results of a peer assessment survey conducted by U.S. News
- U.S. News surveyed deans and other top academics at 227 MFA programs in art and design.
- Survey recipients – 1/school – were asked to rate the academic quality of fine arts programs on a scale of 1 (marginal) to 5 (outstanding).
- Scores for each school were totaled and divided by number of respondents who rated that school to produce each program’s average peer assessment score. Response rate was 29%.
- Programs are numerically ranked in descending order based on the number of nominations they received, as long as they received 7 or more nominations.
#5. Hollywood Reporter Criteria – August 5, 2022
- None listed – I would assume it is based on the perception of the person/people doing the ranking
Deep Thoughts on Higher Ed 