> The trends reflect the increasingly intense tastes of the highest spending, most engaged consumers. https://logicmag.io/play/my-stepdad's-huge-data-set/ > > While a lot of people (most likely you and everyone you know) are consumers of internet porn (i.e., they watch it but don’t pay for it), a tiny fraction of those people are customers. Customers pay for porn, typically by clicking an ad on a tube site, going to a specific content site (often owned by MindGeek), and entering their credit card information. > > > > This “consumer” vs. “customer” division is key to understanding the use of data to perpetuate categories that seem peculiar to many people both inside and outside the industry. “We started partitioning this idea of consumers and customers a few years ago,” Adam Grayson, CFO of the legacy studio Evil Angel, told AVN. “It used to be a perfect one-to-one in our business, right? If somebody consumed your stuff, they paid for it. But now it’s probably 10,000 to one, or something.” > > > > There’s an analogy to be made with US politics: political analysts refer to “what the people want,” when in fact a fraction of “the people” are registered voters, and of those, only a percentage show up and vote. Candidates often try to cater to that subset of “likely voters”— regardless of what the majority of the people want. In porn, it’s similar. You have the people (the consumers), the registered voters (the customers), and the actual people who vote (the customers who result in a conversion—a specific payment for a website subscription, a movie, or a scene). Porn companies, when trying to figure out what people want, focus on the customers who convert. It’s their tastes that set the tone for professionally produced content and the industry as a whole. > > > > By 2018, we are now over a decade into the tube era. That means that most LA-area studios are getting their marching orders from out-of-town business people armed with up-to-the-minute customer data. Porn performers tend to roll their eyes at some of these orders, but they don’t have much choice. I have been on sets where performers crack up at some of the messages that are coming “from above,” particularly concerning a repetitive obsession with scenes of “family roleplay” (incest-themed material that uses words like “stepmother,” “stepfather,” and “stepdaughter”) or what the industry calls “IR” (which stands for “interracial” and invariably means a larger, dark-skinned black man and a smaller light-skinned white woman, playing up supposed taboos via dialogue and scenarios). > > > > These particular “taboo” genres have existed since the early days of commercial American porn. For instance, see the stellar performance by black actor Johnnie Keyes as Marilyn Chambers’ orgy partner in 1972’s cinematic Behind the Green Door, or the VHS-era incest-focused sensation Taboo from 1980. But backed by online data of paid customers seemingly obsessed with these topics, the twenty-first-century porn industry—which this year, to much fanfare, was for the first time legally allowed to film performers born in this millennium—has seen a spike in titles devoted to these (frankly old-fashioned) fantasies. > > > > Most performers take any jobs their agents send them out for. The competition is fierce—the ever-replenishing supply of wannabe performers far outweighs the demand for roles—and they don’t want to be seen as “difficult” (particularly the women). Most of the time, the actors don’t see the scripts or know any specific details until they get to set. To the actors rolling their eyes at yet another prompt to declaim, “But you’re my stepdad!” or, “Show me your big black dick,” the directors shrug, point at the emailed instructions and say, “That’s what they want…” So my interpretation here is that it's not that there's suddenly a huge spike in people discovering they love incest in 2017 where they were clueless in 2016 or that they were all brainwashed to no longer enjoy vanilla that year, it's that that is when the hidden oligopoly turned on various analytics and started deliberately targeting those fetishes as a fleet-wide business decision. And this was because they had so thoroughly commodified regular porn to a price point of $0, that the only paying customers that are left are the ones with extreme fetishes who cannot be supplied by regular amateur or pro supply. They may or may not have increased in absolute number compared to pre-2017, but it doesn't matter, because everyone else vanished, and their relative importance skyrocketed: "If somebody consumed your stuff, they paid for it. But now it’s probably 10,000 to one, or something.” (For younger readers who may be confused by how a ratio like 10000:1 is even hypothetically possible because 'where did that 10k come from when no one pays for porn?', it's worth recalling that renting porn videos used to be big business that would be done by a lot of men, and it kept many non-Blockbuster video rental stores afloat and it was an ordinary thing for your local store to have a 'back room' that the kiddies were strictly forbidden from, and while it would certainly stock a lot of fetish stuff like interracial porn, it also rented out tons of normal stuff. If you have no idea what this was like, you may enjoy reading "True Porn Clerk Stories", Ali Davis 2002.) I think there is a similar effect with foot fetishes & furries: they are oddly well-heeled and pay a ton of money for new stuff, because they are under-supplied and demand new ones. There is not much 'organic' supply of women photographing their feet in various lascivious ways; it's not that it's hard, they just don't do it, but can be incentivized to do so. (I recall reading an article on Wikifoot where IIRC they interviewed a contributor who said he got some photos by simply politely emailing or DMing the woman to ask for her to take some foot photos, and she would oblige. "send foots kthnxbai" apparently works. And probably it's fairly easy to pay for or commission feet images/videos: almost everyone has two feet already, and you can work in feet into regular porn easily by simply choosing different angles or postures, and a closeup of a foot won't turn off regular porn consumers either, so you can have your cake & eat it too. Similarly for incest: saying "But you're my stepdad!" is cheap and easy and anyone can do it if the Powers That Be tell them to in case a few 'customers' will pay actual $$$ for it, while those 'consumers' not into that plot roll their eyes and ignore it as so much silly 'porn movie plot' framing as they get on with business.)

Text diffusion models are still LLMs, just not autoregressive.

The key question is whether you can find improvements which work at large scale using mostly small experiments, not whether the improvements work just as well at small scale. The 3 largest algorithmic advances discussed here (Transformer, MoE, and MQA) were all originally found at tiny scale (~1 hr on an H100 or ~1e19 FLOP[1] which is ~7 orders of magnitude smaller than current frontier training runs).[2] This paper looks at how improvements vary with scale, and finds the best improvements have returns which increase with scale. But, we care about predictability given careful analysis and scaling laws which aren't really examined. > We found that, historically, the largest algorithmic advances couldn't just be scaled up from smaller versions. They needed to have large amounts of compute to develop and validate This is false: the largest 3 advances they identify were all first developed at tiny scale. To be clear, the exact versions of these advances used in modern AIs are likely based on higher compute experiments. But, the returns from these more modern adaptations are unclear (and plausibly these adaptations could be found with small experiments using careful scaling analysis). ---------------------------------------- Separately, as far as I can tell, the experimental results in the paper shed no light on whether gains are compute-dependent (let alone predictable from small scale). Of the advances they experimentally test, only one (MQA) is identified as compute dependent. They find that MQA doesn't improve loss (at small scale). But, this isn't how MQA is supposed to help, it is supposed to improve inference efficiency which they don't test! So, these results only confirm that a bunch of innovations (RoPE, FA, LN) are in fact compute independent. Ok, so does MQA improve inference at small scale? The paper says: > At the time of its introduction in 2019, MQA was tested primarily on small models where memory constraints were not a major concern. As a result, its benefits were not immediately apparent. However, as model sizes grew, memory efficiency became increasingly important, making MQA a crucial optimization in modern LLMs Memory constraints not being a major concern at small scale doesn't mean it didn't help then (at the time, I think people didn't care as much about inference efficiency, especially decoder inference efficiency). Separately, the inference performance improvements at large scale are easily predictable with first principles analysis! The post misses all of this by saying: > MQA, then, by providing minimal benefit at small scale, but much larger benefit at larger scales —is a great example of the more-general class of a compute-dependent innovation. I think it's actually unclear if there was minimal benefit at small scale—maybe people just didn't care much about (decoder) inference efficiency at the time—and further, the inference efficiency gain at large scale is easily predictable as I noted! The post says: > compute-dependent improvements showed minimal benefit or actually hurt performance. But, as I've noted, they only empirically tested MQA and those results are unclear! The transformer is well known to be a huge improvement even at very small scale. (I'm not sure about MoE.) ---------------------------------------- FAQ: Q: Ok, but surely the fact that returns often vary with scale makes small scale experiments less useful? A: Yes, returns varying with scale would reduce predictability (all else equal), but by how much? If returns improve in a predictable way that would be totally fine. Careful science could (in principle) predict big gains at large scale despite minimal or negative gains at small scale. Q: Ok, sure, but if you actually look at modern algorithmic secrets, they are probably much less predictable from small to large scale. (Of course, we don't know that much with public knowledge.) A: Seems quite plausible! In this case, we're left with a quantitative question of how predictable things are, whether we can identify if something will be predictable, and if there are enough areas of progress which are predictable. ---------------------------------------- Everyone agrees compute is a key input, the question is just how far massively accelerated, much more capable, and vastly more prolific labor can push things. ---------------------------------------- This was also posted as a (poorly edited) tweet thread here. ---------------------------------------- 1. While 1e19 FLOP is around the scale of the final runs they included in each of these papers, these advances are pretty likely to have been initially found at (slightly) smaller scale. Like maybe 5-100x lower FLOP. The larger runs were presumably helpful for verifying the improvement, though I don't think they were clearly essential, probably you could have instead done a bunch of careful scaling analysis. ↩︎ 2. Also, it's worth noting that Transformer, MoE, and MQA are selected for being large single advances, making them unrepresentative. Large individual advances are probably typically easier to identify, making them more likely to be found earlier (and at smaller scale). We'd also expect large single improvements to be more likely to exhibit returns over a large range of different scales. But I didn't pick these examples, they were just the main examples used in the paper! ↩︎