I, like many others, struggle with sticking to my goals. I was interested in analyzing data relevant to the topic and thought the crowdfunding platform Kickstarter might be an interesting place to look, as I was aware that not every funded Kickstarter delivered a product.
I focused on video games that were successfully funded. I used a large dataset containing information about Kickstarter projects,[1] from which I randomly selected[2] fully-funded video game Kickstarters from 2014 and 2022. Then, I manually collected other information about these projects. (Here are links to the datasets: 2014, 2022.)
In the process of analyzing the data, I realized the previous estimate of how many Kickstarter projects don't deliver rewards (~9%) was severely flawed. Imagine that you're trying to figure out how many Kickstarters don't ever end up giving backers rewards. It's 2015, and some of the projects you're asking backers about were created in 2015. Are you sensing the problem yet? The estimate from this study considered a project a failure when more than half of the backers they surveyed about it responded in either of the following ways: that they were no longer expecting their rewards, or that they had received rewards that were not the promised ones. Of course, a lot of backers fell into another category: expecting to receive their rewards.
So, while that research had found a failure rate of ~12% for fully-funded video games, I found a substantially higher rate. Of the 100 games that I looked at with funding deadlines in 2014, I could confirm that 68 of them had been released. There were 4 projects where I was unclear about whether they had produced anything, and the other 28 games seemed to have never come to fruition. Only one of these 28 seemed to not have been abandoned.
Obviously, the method used by Mollick (2015) would underestimate the number of failed projects because some of the people who were expecting to receive rewards from projects would never receive them. Despite the huge methodological problem, which the author addressed and excused unconvincingly, media outlets covered the topic with a problematic lack of suspicion (i.e. this).
I made a video on the topic of this discrepancy, but let's turn our attention now to my original idea: what can Kickstarter teach us about following through?
Why do Kickstarters tend to result in released video games?
While an ~1 in 4 chance of incompletion might sound terrible to a backer, it sounds like a lower rate of failure than I'd expect of people trying to complete projects. I promise that I'm not a judgey person, but if someone tells me they're making a video game, or I see someone online talking about their game in progress, I probably wouldn't expect the game to ever get finished. (There are definite exceptions though, like for people who have already made many video games!)
So, why do Kickstarter games get released at probably a higher rate than all the other games people want to make? I've come up with a list of factors that I think could be involved.
These factors might help Kickstarted games get made
Social pressure - After you're funded, backers can pester you about progress, sometimes tracking you down to other platforms. Your reputation could also be tarnished if you do not make progress. Some backers may also threaten lawsuits, but it seems like lawsuits around Kickstarters are rare
Self-selection of creators - People who are willing to make a Kickstarter are willing to put in a certain level of effort that others might not. They also might be more likely to have a working prototype
Money to use - After you're funded, you can use the money you got to give you more time to work on the game or hire help
Kickstarter's incentives - [???] I'm not clear on what Kickstarter does to ensure that backers get their rewards. It seems very minimal. Once in a while they do restrict creators, and you'll see the phrase "Kickstarter has restricted this creator" on their creator or project page, and that creator will not be able to start another Kickstarter
These factors may hinder Kickstarted games' completion
Intentional scammers - You might expect that some people making Kickstarters are pretending to want to make a game to get backers' money and will never make the game
Backer selection for complicated/ difficult-to-make games - Maybe games selected by Kickstarter backers tend to be more complicated, making it harder for those games to get made than games not on Kickstarter
Social pressure - You might expect that people trying to satisfy customers would be more likely to promise a game that is unachievable at their current skill level. It could also be possible that, with more pressure, creators would procrastinate more than those who are not making a Kickstarter
Are there any clues in the data?
Let's take a look at what my 2014 data might tell us about these factors. Before sampling, I knew that I'd be interested in looking at how projects of various funding categories differed, so I wanted to have an even number of projects from various levels of funding. I also wanted to make sure, though, that my sample would still be representative of the larger population, which would not be true if I selected the same amount of projects from the top 10% of funding amount as the lowest 10% of funding. Instead, I divided the population of potential projects I could analyze from that year into funding groups that each had the same number of projects.[3]
I thought social pressure was likely at play. It seems like social pressure would increase with funding amount. And, if social pressure causes games to get made, you'd think funding amount and completion would be correlated.
However, in my sample, I did not find a correlation between completion and funding amount. There may be a correlation that I would be able to see with a larger sample size.
Similarly, we may also expect that social pressure and funding amount would be related to the quality of the game, which would probably be reflected in how many people liked it. For 47 of the games, I was able to collect steam rating data. There was no obvious relationship here, but there's also not that much data.
It seems likely that the more funded games are more complicated. For instance, funding amount and time from funding deadline to release were correlated for the 65 projects whose release dates I could find (p = 0.015, Spearman correlation test). I think this could be because projects receiving more funding were more complicated or harder to make. (You can see this correlation visually when you log the X axis. If someone could explain to why that's the case, that'd be much appreciated!)
Each point represents one project. Each point's x coordinate represents how much funding it received (i.e. how much USD was pledged.) Each point's y coordinate represents the number of months between when the project stopped collecting funding to when the game was released. This is the same as the above, but the x axis is logged.
What do you think?
Thanks to my friends for motivating me to complete this project, among others.
I found this on Kaggle, which provides a preview of the dataset. This site used to have the full dataset on it for free, which I think might have been unintentional, as it seems like you are supposed to leave a tip before being able to access their dataset. I think it might be a continuation of this older dataset, which also has data about hundreds of thousands of Kickstarters. I am pretty sure that this dataset that I used for random sampling includes almost all Kickstarters.
A few things: I wanted to make sure that I had projects of various funding levels. Using the ntile() dplyr function in R, I separated all the possible projects to pick from into 5 funding categories with the same amount of projects. I performed this separately for the 2014 and 2022 datasets. I then randomly selected 20 projects from each funding level. Although all the projects I looked at had "Video Games" as their category name, some of the projects that were selected did not end up being video games. When this would happen, I would get another video game from the same funding category to replace the project. For the 2014 dataset, I selected Kickstarters that had funding deadlines in 2014. I didn't realize that that column existed when I was doing the 2022 ones, so that dataset has projects that started collected funding in 2022. I unfortunately based my sampling off of a column that did not have data for all projects. However, I don't think this would've made a big difference, as only 8 of the 425 successfully funded video game Kickstarters with deadlines in 2014 did not have an entry in this column. None of the 2022 successfully funded games had missing data in this column.
Again, I used the ntile() dplyr function in R, separating all the possible projects to pick from into 5 funding categories with the same amount of projects. I performed this separately for the 2014 and 2022 datasets. I then randomly selected 20 projects from each funding level.
I, like many others, struggle with sticking to my goals. I was interested in analyzing data relevant to the topic and thought the crowdfunding platform Kickstarter might be an interesting place to look, as I was aware that not every funded Kickstarter delivered a product.
I focused on video games that were successfully funded. I used a large dataset containing information about Kickstarter projects,[1] from which I randomly selected[2] fully-funded video game Kickstarters from 2014 and 2022. Then, I manually collected other information about these projects. (Here are links to the datasets: 2014, 2022.)
In the process of analyzing the data, I realized the previous estimate of how many Kickstarter projects don't deliver rewards (~9%) was severely flawed. Imagine that you're trying to figure out how many Kickstarters don't ever end up giving backers rewards. It's 2015, and some of the projects you're asking backers about were created in 2015. Are you sensing the problem yet?
The estimate from this study considered a project a failure when more than half of the backers they surveyed about it responded in either of the following ways: that they were no longer expecting their rewards, or that they had received rewards that were not the promised ones. Of course, a lot of backers fell into another category: expecting to receive their rewards.
So, while that research had found a failure rate of ~12% for fully-funded video games, I found a substantially higher rate. Of the 100 games that I looked at with funding deadlines in 2014, I could confirm that 68 of them had been released. There were 4 projects where I was unclear about whether they had produced anything, and the other 28 games seemed to have never come to fruition. Only one of these 28 seemed to not have been abandoned.
Obviously, the method used by Mollick (2015) would underestimate the number of failed projects because some of the people who were expecting to receive rewards from projects would never receive them. Despite the huge methodological problem, which the author addressed and excused unconvincingly, media outlets covered the topic with a problematic lack of suspicion (i.e. this).
I made a video on the topic of this discrepancy, but let's turn our attention now to my original idea: what can Kickstarter teach us about following through?
Why do Kickstarters tend to result in released video games?
While an ~1 in 4 chance of incompletion might sound terrible to a backer, it sounds like a lower rate of failure than I'd expect of people trying to complete projects. I promise that I'm not a judgey person, but if someone tells me they're making a video game, or I see someone online talking about their game in progress, I probably wouldn't expect the game to ever get finished. (There are definite exceptions though, like for people who have already made many video games!)
So, why do Kickstarter games get released at probably a higher rate than all the other games people want to make? I've come up with a list of factors that I think could be involved.
These factors might help Kickstarted games get made
These factors may hinder Kickstarted games' completion
Are there any clues in the data?
Let's take a look at what my 2014 data might tell us about these factors. Before sampling, I knew that I'd be interested in looking at how projects of various funding categories differed, so I wanted to have an even number of projects from various levels of funding. I also wanted to make sure, though, that my sample would still be representative of the larger population, which would not be true if I selected the same amount of projects from the top 10% of funding amount as the lowest 10% of funding. Instead, I divided the population of potential projects I could analyze from that year into funding groups that each had the same number of projects.[3]
I thought social pressure was likely at play. It seems like social pressure would increase with funding amount. And, if social pressure causes games to get made, you'd think funding amount and completion would be correlated.
However, in my sample, I did not find a correlation between completion and funding amount. There may be a correlation that I would be able to see with a larger sample size.
Similarly, we may also expect that social pressure and funding amount would be related to the quality of the game, which would probably be reflected in how many people liked it. For 47 of the games, I was able to collect steam rating data. There was no obvious relationship here, but there's also not that much data.
It seems likely that the more funded games are more complicated. For instance, funding amount and time from funding deadline to release were correlated for the 65 projects whose release dates I could find (p = 0.015, Spearman correlation test). I think this could be because projects receiving more funding were more complicated or harder to make. (You can see this correlation visually when you log the X axis. If someone could explain to why that's the case, that'd be much appreciated!)
What do you think?
Thanks to my friends for motivating me to complete this project, among others.
I found this on Kaggle, which provides a preview of the dataset. This site used to have the full dataset on it for free, which I think might have been unintentional, as it seems like you are supposed to leave a tip before being able to access their dataset. I think it might be a continuation of this older dataset, which also has data about hundreds of thousands of Kickstarters.
I am pretty sure that this dataset that I used for random sampling includes almost all Kickstarters.
A few things:
I wanted to make sure that I had projects of various funding levels. Using the ntile() dplyr function in R, I separated all the possible projects to pick from into 5 funding categories with the same amount of projects. I performed this separately for the 2014 and 2022 datasets. I then randomly selected 20 projects from each funding level.
Although all the projects I looked at had "Video Games" as their category name, some of the projects that were selected did not end up being video games. When this would happen, I would get another video game from the same funding category to replace the project.
For the 2014 dataset, I selected Kickstarters that had funding deadlines in 2014. I didn't realize that that column existed when I was doing the 2022 ones, so that dataset has projects that started collected funding in 2022.
I unfortunately based my sampling off of a column that did not have data for all projects. However, I don't think this would've made a big difference, as only 8 of the 425 successfully funded video game Kickstarters with deadlines in 2014 did not have an entry in this column. None of the 2022 successfully funded games had missing data in this column.
Again, I used the ntile() dplyr function in R, separating all the possible projects to pick from into 5 funding categories with the same amount of projects. I performed this separately for the 2014 and 2022 datasets. I then randomly selected 20 projects from each funding level.