I'm a bit skeptical of AlphaFold 3

[-]Vanessa Kosoy1y81

This sounds like valid criticism, but also, isn't the task of understanding which proteins/ligands are similar enough to each other to bind in the same way non-trivial in itself? If so, exploiting such similarities would require the model to do something substantially more sophisticated than just memorizing?

[-]Oleg Trott1y40

(ligand = drug-like molecule, for anyone else reading)

Right, I didn't mean exact bitwise memory comparisons.

The dataset is redundant(ish), simply as an artifact of how it's constructed:

For example, if people know that X binds A, and X ≈ Y, and A ≈ B, they'll try to add X+B, Y+A and Y+B to the dataset also.

And this makes similarity-based predictions look artificially much more useful than they actually are, because in the "real world", you will need to make predictions about dissimilar molecules from some collection.

I hope this makes sense.

[-]wassname1y70

But if it's a poor split, wouldn't it also favour the baseline (your software). But they did beat the baseline. So if your concern is correct, they did outperform the baseline, but just didn't realistically measure generalisation to radically different structures.

So it's not fair to say 'it's only memorisation'. It seems fairer to say 'it doesn't generalise enough to be docking software, and this is not obvious at first due to a poor choice of train test split'.

[-]Ponder Stibbons1y52

I have similar concerns regarding the ligand sets used to test Alphafold3. I’ve had a cursory look at them and it seemed to me there were a lot phosphate containing molecules, a fair few sugars, and also some biochemical co-factors. I haven’t done a detailed analysis, so some caveats. But if true, there are two points here. Firstly there will be a lot of excellent crystallographic training material available on these essentially biochemical entities, so AlphaFold3 is more likely to get these particular ones right. Secondly, these are not drug-like molecules and docking programs are generally parameterized to dock drug-like molecules correctly, so are likely to have a lower success rate on these structures than on drug-like molecules.

I think a more in-depth analysis of performance of AF3 on the validation data is required, as the OP suggests. The problem here is that biochemical chemical space, which is very well represented by experimental 3D structure, is much smaller than potential drug-like chemical space, which is poorly represented by experimental 3D structure comparatively speaking. So inevitably AF3 will often be operating beyond the zone of applicability, for any new drug series. There are ways of getting round this data restriction, including creating physics compliant hybrid models (and thereby avoiding clashing atoms). I’d be very surprised if such approaches are not currently being pursued.

[-]Oleg Trott1y10

I'm surprised by how knowledgeable people are about this on this site!

BTW, there's some discussion of this happening on the CCL mailing list (limited to professionals in relevant fields) if you are interested.

[-]Yair Halberstadt1y42

This makes sense, but isn't alphafold available for use? Is it possible to verify this one way or another experimentally?

[-]Oleg Trott1y100

Determining 3D structures is expensive.

The most realistic thing one could do is repeat this work, with the same settings, but using k-fold cross-validation, where test and training sets are never related (like what I did at Columbia).

This will show how well (or poorly, as the case may be) the method generalizes to unrelated proteins.

I hope someone does it.

[-]ChristianKl1y20

But do you need 3D structures to test alphafold? If alphafold makes a prediction about whether a given ligand binds to a protein, I would expect that testing whether or not that ligand binds to the protein is much cheaper.

[-]Oleg Trott1y10

Unlike Vina, AF3 only predicts 3D structures, I believe. It does not predict binding affinities.

[-]ChristianKl1y20

AlphaFold 2 was only predicting 3D structures. From the abstract of the Alpha Fold 3 paper:

far greater accuracy for protein–ligand interactions compared with state-of-the-art docking tools

The FastCompany article says:

He said that for the interactions of proteins with small (drug-like) molecules AlphaFold 3 showed 76% accuracy in a benchmark test (versus 52% for the next best tool).

[-]Oleg Trott1y10

Right. The benchmark (their test set) just compares 3D structures.

Side note: 52% also seems low for Vina, but I haven't looked into this further. Maybe the benchmark is hard, or maybe the "search space" (user-specified) was too big.

On their other test (in their Extended Data), both Vina and AF3 do much better.

[-]CuriousEcho1y30

Since this shows up in homomeric proteins, maybe the issue is because symmetry breaking in quartinary structure happens late in training. This is just a guess tho... but aligns with my intuitions on training dynamics. I would also guess that this would get ironed out in the tails of the loss curve, if we push out further in training. I think the PLDDT loss would first learn to stack homomers on top of one another and then spread them out. I haven't used alphafold too much but the smoking gun plot id be interested in is clashes in heteromers vs clashes in homomers. It's a curious little detail, thanks for sharing this finding!

[-]Ponder Stibbons1y10

There is an additional important point that needs to be made. Alphafold3 is using predominantly “positive” data. By this I mean the training data encapsulates considerable knowledge of favourable atom-atom or group-group interactions and relative propensities can be deduced. But “negative” data, in other words repulsive electrostatic or Van der Waals interactions, are only encoded by absence because these are naturally not often found in stable biochemical systems. There are no relative propensities available for these interactions. So AF3 can be expected to not perform as well when applied to real-world drug design problems where such interactions have to be taken into account and balanced against each other and against favourable interactions. Again, this issue can be mitigated by creating hybrid physics compliant models.

It is worth also noting that ligand docking is not generally considered a high accuracy technique and, these days is often used to 1st pass screen large molecular databases. The hits from docking are then further assessed using an accurate physics-based method such as Free Energy Perturbation.

[-]Tim Bonzon1y10

So without knowing too much about AlphaFold 3 or AutoDock Vina, does this mean AlphaFold 3 essentially replaces AutoDock Vina? Does AutoDock Vina still fill a niche AlphaFold 3 doesn't?

LESSWRONG
LW

LESSWRONG
LW

87

I'm a bit skeptical of AlphaFold 3

87

87