I don't have much direct experience with transformers (I was part of some research with BERT once where we found it was really hard to use without adding hard-coded rules on top, but I have no experience with the modern GPT stuff). However, what you are saying makes a lot of sense to me based on my experience with CNNs and the attempts I've seen to explain/justify CNN behaviour with side channels (for instance this medical image classification system that also generates text as a side output). See also my comment on Facebook.
I think what you're saying makes a lot of sense. When assembling a good training data set, it's all about diversity.
Sorry, I missed that somehow. Thanks.
(cross posting this comment from E. S. Yudkowksy's Facebook with some edits / elaboration)Has anyone tried fine-tuning a transformer on small datasets of increasing size to get a sense of how large a dataset would be needed to do this well? I suspect it might have to be very large.
Note this is similar to the "self explaining AI" idea I explored in early 2020, which I threw together a paper on (I am hesitant to link to it because it's not that great of a paper and much of the discussion there is CNN specific, but here it is.). I can see how producing "thoug... (read more)
We're guessing 1000 steps per reasonably-completed run (more or less, doesn't have to be exact) and guessing maybe 300 words per step, mostly 'thought'. Where 'thoughts' can be relatively stream-of-consciousness once accustomed (we hope) and the dungeon run doesn't have to be Hugo quality in its plotting, so it's not like we're asking for a 300,000-word edited novel.
However I also could see the "thoughts" output misleading people - people might mistake the model's explanations as mapping onto the calculations going on inside the model to produce an output.
I think the key point on avoiding this is the intervening-on-the-thoughts part:"An AI produces thoughts as visible intermediates on the way to story text, allowing us to watch the AI think about how to design its output, and to verify that we can get different sensible outputs by intervening on the thoughts".
So the idea is that you train things in such a way that the thoughts do map onto the calculations going on inside the model.
Note: Pfizer started a trial in September to try to answer this question. We may know answer in a few months. In theory I don't see why it wouldn't work but with limited supply there's probably better uses at least in the next few months. Also, note the initial EUA application is asking it be approved for high-risk patients only, probably because Pfizer was told by FDA it wouldn't be EUA'd otherwise. Paxlovid must be taken with Ritonavir (otherwise Paxlovid breaks down to fast) which messes with liver enzymes and isn't a good choice for man... (read more)
Very cool, will take a look. This basically solves question 1. It seems the original Solomonoff work isn't published anywhere. By the way, the author, William H. Press, is a real polymath! I am curious if there is any extension of this work to agents with finite memory.. as an example, the same situation where you're screening a large number of people, but now you have a memory where you can store N results of prior screenings for reference. I'm going to look into it..
Here's another paper on small / non-robust features, but rather specific to patch-based vision transformers: Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation^ This work is very specific to patch-based methods. Whether patches are here to stay and for how long is unclear to me, but right now they seem to be on an ascendancy (?).
For what it's worth - I see value in votes being public by default. It can be very useful to see who upvoted or downvoted your comment. Of course then people will use the upvote feature just to indicate they read a post, but that's OK (we are familiar with that system from Facebook, Twitter, etc). I'm pretty apathetic about all the other proposals here. Reactions seem to me to be unnecessary distractions. [side note - emojiis are very ambiguous so it's good you put words next to each one to explain what they are supposed to mean]. The way I woul... (read more)
I'm curious why this comment has such low karma and has -1 alignment forum karma. If you think doom is very likely when AI reaches a certain level, than efforts to buy us time before then have the highest expected utility. The best way to buy time, arguably, is to study the different AI approaches that exist today and figure out which ones are the most likely to lead to dangerous AI. Then create regulations (either through government or at corporation level) banning the types of AI systems that are proving to be very hard to align. (For example we may... (read more)
Also... alignment is obviously continuum and of course 100% alignment with all human values is impossible. A different thing you could prove is whether it's possible to guarantee human control over an AI system as it becomes more intelligent. There's also a concern that a slightly unaligned system may become more and more aligned as its intelligence is scaled up (either by humans re-building/trianing it with more parameters/hardware or via recursive self-improvement). It would useful if someone could prove whether that is impossible to prev... (read more)
Roman Yampolsky has said recently (at a Foresight Salon event, the recording should be posted on YouTube soon) that it would be highly valuable if someone could prove that alignment is impossible. Given the high value for informing AI existential safety investment, I agree with Yampolsky we should have more people working on this (trying to prove theorems (or creating very rigorous arguments) as to whether alignment is possible or impossible). If we knew with very high certainty that alignment is impossible, than that would compel us to invest more r... (read more)
It's hard to imagine a "general intelligence" getting stuck at the level of a 10 year child in all areas -- certainly it will have an ability to interface with hardware that allows it to perform rapid calculations or run other super-human algorithms. But there are some arguments that suggest intelligence scaling at an exponential rate can't go on indefinitely and in fact limitations to exponential growth ("foom") may be hit very soon after AGI is developed, so basically foom is impossible. For instance, see this article by Francois Chollet: https... (read more)
we haven't solved the problem of deeper networks taking longer to train, right
My understanding is the vanishing gradient problem has been largely mitigated by introducing skip connections (first with resnet, and now standard in CNN architectures), allowing for networks with hundreds of layers.
It's too bad fully-connected networks don't scale.
I've heard people say vision transformers are sort of like going back to MLPs for vision. The disadvantage of going away from the CNN architecture (in particular weight sharing across receptive fields... (read more)
".. we just don't have very compelling example domains where ML systems understand important things in ways we can't. "
I'm guessing you mean in ways humans can't even in principle?
Regardless, here's something people might find amusing - researchers found that a simple VGG-like 3D CNN model can look at electron microscope images of neural tissue and do a task that humans don't know how to do. The network distinguishes neurons that specialize in certain neurotransmitters. From the abstract to this preprint:
"The network successfully discriminates
Great..Also I just realized that the "grokking" phenomena is relevant here. The "grokking" paper shows jumps during training, but it's similar. From the lens of the lottery ticket hypothesis, it's not surprising that grokking may be easier / more likely in larger models. I wonder how much "grokking" is new to transformers. I happened to stumble across an example in the literature where a CNN model "fails to grok" the Game of Life: https://arxiv.org/abs/2009.01398 .. I wonder what would happen if you used a transformer model instead..Also, please check... (read more)
Thanks.. I was looking for more graphs with discontinuous jumps and "# of parameters" on the x-axis... but I think "totally new and unexpected capabilities after going from GPT-2 to GPT-3" is a reasonable thing to point at, also. The scaling laws bibliography is super, super useful. I am just embarking on making my way through it now..
"If one looks at the performance of particular tasks, such as arithmetic on numbers of a certain size, across model sizes, one often observes points where larger models discontinuously become better at a task."
Is it accurate to say that one "often observes" this? The only examples I know of are in GPT-3 with the addition, multiplication, and symbolic substitution tasks. I'm not sure how concerned to be about this being a general phenomena. Does anyone have further examples? Does anyone have insights into whether the GPT-3 examples are special cases or not?
Here's my opinions on what deep learning can do, FWIW - 1 (abstraction) yes, but they aren't sample efficient ! 2. (generalization) eh, not if you define generalization as going out of distribution (note: that's not how it's normally defined in ML literature). Deep learning systems can barely generalize outside their training data distribution at all. The one exception I know is how GPT-3 learned addition but even then it broke down at large numbers. Some GPT-3 generalization failures can be seen here.3. (causality) maybe? 4. (long te... (read more)
I'm having trouble understanding the n-cut metric used in Filan's work. A more intuitive measure would be the sum of weights contained in edges that go between each subset of vertices divided by the total sum of weights in the graph as a whole. That's not quite what n-cut measures though, if you look at the equation - it isn't normalized that way. It would be nice if there were some figures of examples of modular graphs with different n-cut values to provide an intuitive understanding of what n-cut = 9 means vs n-cut = 5. Look at the latest ... (read more)
I want to point out that there has been some very small amounts of progress in the last 10 years on the problem of moving from connectome to simulation rather than no progress. First, there has been interesting work at the JHU Applied Physics Lab which extends what Busbice was trying to do when he tried to run as simulation of c elegans in a Lego Mindstorms robot (by the way, that work by Busbice was very much overhyped by Busbice and in the media, so it's fitting that you didn't mention it). They use a basic integrate and fire model to simulate the n... (read more)
Graphics have been dumped here: https://github.com/tliptrot/InternalMemos/tree/master/FDA_images?fbclid=IwAR2L_2ibD49JWkj4qrvD4aN-GbvDl7VfMOIfBwZUtCQKaDspoXmVrFb7xa4
Here's some updates on this: We have two Facebook event pages created for this: https://www.facebook.com/events/208338324360886 (35 RSVPs) https://www.facebook.com/events/1028113637697900 (18 RSVPs) This is great, but we need more people. It might be worth gently reminding people that it only takes a few minutes to set up a Twitter account. We have some big Twitter influencers who have signaled they are on our side but haven't yet used our hashtags. They should be our primary targets to get involved:
Not opposed, but just want to note we are planning a demonstration outside the FDA on sunday from 2-4pm. I shall post links to the Facebook and Meetup events soon. A bunch of local LWers can make it on weekends but not weekdays. I think this sequence works well - we do the protest and gets some pictures, and then share them as part of the Twitter storm on Monday.
As far as "playing the comments game", I admit I am guilty of that. At a deeper level it comes from a desire to connect with like-minded people. I may even be doing it right now.
We like to think people post because they are genuinely intellectually engaged in the material we've written, but the truth is people post comments for a myriad of different reasons, including wanting to score comment 'points' or 'karma' or engage in a back-and-forth with a figure they admire. People like getting attention. [even shy nerdy people who are socially isolate... (read more)
Its funny because 90+% of articles on Salon.com are 'godawful clickbait' in my opinion -- with this one being one of the exceptions.
Decent article but pretty basic. Still, a glimmer of reason in the dark pit of Salon.
Didn't know Y Combinator was doing a pilot. They don't mention how many people will be in the pilot in the announcement, but it will be interesting to see.
One thing I never understood is why it makes sense to do cash transfers to people that are already wealthy - or even above average income. A social safety net (while admittedly more difficult to manage) consisting solely of cash income seems to makes more sense. I guess the issue is with the practical implementation details of managing the system and making sure everyone who needs to be enrolled is.
Thanks for the review. I just submitted my application today (before I saw your post). I was a bit wary, due to fluttershy's post you mentioned, but more because of the lack of results (ie actual job placements) on their website compared to more established programs. The main benefit I see to this program is being in a space with other people who you can easily bounce ideas off (ie, the social experience). I tend to work bettered in a structured environment, also. Its also good to hear that it is useful for networking as well. I wasn't sure about that, because whereas other data science programs have working relationships with major companies, I didn't get that impression when reading about Signal.
I'd love to see some results as well, and I'm assuming as soon as you have them they'd be posted. I looked under 'projects' and looked at the available LinkedIn profiles, and it looks like three of the students got jobs (well, more specifically 2 jobs and an internship). Those students already had impressive resumes going into the program, but this is quite encouraging to see.
I am currently working from home and my laptop is now considerably more powerful than my desktop, which is 8 years old.
Anyone have a suggestion for a good external video card that would allow me to use two monitors to my new laptop? [it has a mini DisplayPort output and free USB]
The Skeptics Guide to the Universe podcast interviewed Grant Richey about this. He notes that some of the headlines were misleading, because the study did find that when flossing is performed by a dental hygienist on children, it has positive effect. So, a better encapsulation of the recent review is that improper flossing doesn't have any positive effect. On the other hand, its very unlikely to hurt you, unless you damage your gums in the process.
in case anyone wants a detailed review of the literature from before this study, Grant Richey did a blog post on it a few months ago: