“And then RNNs gave you a state machine within it”

“I know like the scaling laws paper shows that RNN's have worse curves for scaling but probably people would have been like I bet you could have built chat GBT with a very sophisticated RNN”

“Ilya Sutskever wrote the first paper that said you can just train a simple RNN model, scale it up and it'll beat all the phrase-based machine translation systems.”