1. CCS does not find the single linear probe with high accuracy: there are more than 20 orthogonal linear probes (i.e. using completely different information) that have similar accuracies as the linear probe found by CCS (for most datasets);

So what about an ensamble of the top 20 linear probes? Is it substantially better than using just the best one alone? I would expect so given that they are orthogonal, so they are using ~uncorrelated information.

The most important thing is approaching other points of view with an open mind, with epistemic humility , that is, knowing that something of what you think can be wrong, even if, from the inside, everything feels right.

On the object level:

• Carbohydrates can be: fruit/whole grains/normal bread/normal pasta or North American crazy industrial snacks and "sugar-cereals" and "sugar-bread". The first one is good, the second one is bad.
• No idea about optimal salt levels, just a note on your language: "potentially deadly" is too vague to have a useful discussion. Crossing the road is "potentially deadly" for some definitions of "potentially deadly". Try to quantify the risk instead, at least approximately. 
• For facemasks, a surgical mask that you re-use for several day, without cutting your beard (beard makes it impossible to have a perfect fit), with a random size, removing it constantly to eat and drink, of course it is going to work badly. If you wear a serious P100 masks, of appropriate shape and size, with no beard (cut down to zero with a high quality manual razor) and never remove it, of course it is going to work, it is basic physics at that point. If it is a good idea to wear it, considering costs and benefits, is another discussion.
• Please be really careful about psychoactive substances, they might have long term effects that are little known because they are little studied. (This last statement is based on random stuff read on the internet, take with a grain of salt).

Language Models are Few-Shot Learners

"""
That's when I discovered more effective ways to approach reading, including what I'll call "Guess-and-Check," the technique of scanning and making predictions. Instead of trying to read every word in a textbook, in Guess-and-Check you scan the material and make predictions about what you think the text is saying. This active reading process can help you better engage with the material and activate your prior knowledge. After making your prediction, be sure to confirm or correct it by checking it against the text.

"""
This is similar to the way GPT-3 was trained! Pretty cool that you also found it effective!

Yes, the tone of my comment could be improved. I appreciate him for publishing his lessons to the community and wanted to give some suggestions to improve (eventual) future ones, if he feels like the higher quality is worth the higher effort, and with no obligation. "Al caval donato non si guarda in bocca" (You should not look at the teeth of a gift horse (to learn about its age))

Some suggestions:

• Use a better marker, what you wrote on the whiteboard is almost unreadable for me.
• Expanding on the previous point, write bigger and make better use of the space on the board.
• If you have complex graphics, pre-make them accurately and print them out, put them on the whiteboard with weak tape. (What is the weird of bridge at the start?)
• Use Whisper to make subtitles to help non-native speakers (as another commenter suggested).
• Invest in a tripod to have the camera at a natural height instead of bottom to top.
• I did not watch the lectures, this feedback is from skimming around in the first one.
• In lecture 3, visualizing the Fourier matrix (Discrete Cosine Transform) matrix, could be really interesting.
• Are the lectures ordered by something or following a thread? If so you should make it clear by posting an overview lecture at the start.
• Do not wear glasses, it makes you look weird and lose connection to the viewer.
• Thanks a lot for your effort, we really need to spread mathematical education for the benefit of the whole society.

In Italy we have this, in the Ferragosto week (around the 15 of August) a huge percentage of people is on vacation. In general a lot of people take vacations in August and schools of all order and levels (including university) are closed the whole month.

In the simplest possible way to partecipate, yes, but a hackathon is made to elicit imaginative and novel ways to approach the problem (how? I do not know, it is the partecipants' job to find out).

Yes of course:

Models:

• https://paperswithcode.com/task/image-captioning 

Datasets:

• Laion 400 millions or other sizes: https://laion.ai/blog/laion-400-open-dataset/
• https://paperswithcode.com/dataset/coco-captions 
• Imagenet/any image classification dataset: just treat the labels as text, this should be used sparingly as otherwise the model will learn to just output single words.

Also in the performance metric, the sum of the performance of each layer should probably be weighted to give less importance to the initial layers, otherwise we encourage the models to do as much of the work as possible at the start instead of being gradual.

Probably even if not completely by hand, MNIST is so simple that hybrid human-machine optimization could be possible, maybe with a UI where you can see the effect on validation loss in (almost) real time of changing a particular weight with a slider. I do not know if it would be possible to improve the final score by changing the weights one by one. Or maybe the human can use instinctual vision knowledge to improve the convolutional filters.

On Cifar this looks very hard to do manually given that the dataset is much harder than Mnist.

I think that a too large choice of lambda is better than a too small one because if lambda is too big the results will still be interesting (which model architecture is the best under extremely strong regularization?) while if it is too small you will just get a normal architecture slightly more regularized.