"A team of 3 top research ML engineers with fine-tuning access to GPT-4o (including SFT and RL), $10 million in compute, and 1 year of time could use GPT-4o to surpass typical naive MTurk performance at ARC-AGI on the test set while using less than $100 per problem at runtime (as denominated by GPT-4o API costs)."
But doing so would tune that GPT-4o to be less good at other tasks, wouldn't it? Isn't this way of solving just Goodharting the metric and actually pushing the LLM away from being "General Intelligence"?
That being said, this is a great job you have done just over a week. Thank you for your contribution for science.
Children spending 300 hours per year learning math, on their own time and via well-designed engaging video-game-like apps (with eg AI tutors, video lectures, collaborating with parents to dispense rewards for performance instead of punishments for visible non-compliance, and results measured via standardized tests), at the fastest possible rate for them (or even one of 5 different paces where fewer than 10% are mistakenly placed into the wrong category) would probably result in vastly superior results among every demographic than the current paradigm of ~30-person classrooms.
This sounds great.
Googling "math tutor ai" already gives a bunch of competing suggestions. I wonder how well they work though.
Thank you for sharing. For me your original talk is more convincing, and your Death Star strike plan is the one I would be more willing to follow rather than movie one.
I suppose this is an area where one can have a strong conviction on how things ought to be done, assuming other smart people think the same way, but in reality the way they think about it is closer to the base rate, so communication towards them should be as such.
A lot of important technology do not exists yet for the above.
You assume independent movement and coordination of said movement in a hostile area, while we barely arrived at self-driving cars in well governed, 1st world areas (available only very few places afaik due to not being able to demonstrate high enough reliability to be convincing to lawmakers).
Point-to-point laser communication would be a great solution indeed, but that would also be a great solution for a bunch of other military applications. Yet it is not used, as we do not have reliable solution for working with it in case of moving objects (apart from satelites), too much coordination is needed to find the end points.
There is no software system currently that is even close to completing the requirements.
Let me offer a different point of view on the whole question:
A suicide drone is just a missile. Until recently there was just no way to propel and guide an explosive charge accurately, reliably and cheaply other than using a rocket engine.
A recon drone is just a helicopter. Until recently there was just no way to propel and guide a good enough observer and transmit the information reliably and cheaply other than a human carrying object using gas turbines.
What happened is now we have better battery and material technology, and way better (smaller) electronic devices (and optimized global supply chains). The cost advantage against missiles is mostly due to inertia in the military-industrial complex: most missiles in inventory were designed against targets with different size and performance parameters. You need a a million dollar AMRAAM missile to intercept an Su-30 flying 80 kms away, maybe flying at 2 mach or at 40 000 feet or pulling 9g maneuvers while dispensing chaff, flares and using EW. (and the missile carrying aircraft may have already pulled the same speed and maneuvers, and may have took off from the desert with 45 degrees Celsius and arrived to - 30 degrees Celsius flight attitude in the next couple of minutes, yet the missile must stay safe and reliable).
The infantryman did not get replaced. They just got missiles available in large numbers, and their own miniature recon helicopters in every bush. It is harder to hide and there are more precision fires to throw at you after getting spotted.
Thanks for writing the post.
Regarding the challenge: I suppose one of the errors made by you was using bad sources in some case, e.g. using the msn.com article for supporting Russian losses.
Regarding the article: although what is presented there is generally valid, it do not feel my understanding/defense increased in a meaningful way, as
I think what I really liked in your dialog with Isusr was it showing how such techniques looked like when someone was using them himself. I haven't experienced that before, especially not with that level of clarity and purposefulness.
Thanks for pinging me. Haven't noticed it yet, will read it now.
I think something along the lines of "Defense Against the Dark Arts" with actionable steps on recognizing and defusing them (and how to practice these) would be great. If you feel like you have the energy and time, more articles on offensive usage (practice) and on theoretical background (how to connect your practical experience to existing LW concepts/memes) would be also nice. But I think the first one (defense) would be the most useful for LW readers.
I think this is an important addition to the site. There had been articles before about the "dark" side/arts, but I think this is the first one where the examples are not thought experiments and abstractions, but actual real world experience from an actual user.
It is helpful for understanding politics.
How much time did they spend practicing with the kids? (Frequency and length of sessions. e.g. 7 times a week for 5-10 minutes)
I see that A Deepness in the Sky is a prequel to A Fire Upon the Deep, but was written later. Does it make any impact which one I read first? Deepness seems to be more interesting to me, but I generally I prefer to read things in the suggested order.