MicrosoftResearch

What advice do you have for aspiring Undergraduates and others who want to pursue research in Reinforcement Learning?

The standard advice is to aim for a phd. Let me add some details to that. The most important element of a phd is your advisor(s) with the school a relatively distant second. I personally had two advisors, which I enjoyed---two different perspectives to learn from and two different ways to fund conference travel :-) Nevertheless, one advisor can be fine. Aside from finding a good advisor to work with, it's very good to maximize internship possibilities by visiting various others over the summers. Reinforcement Learning is a great topic, because it teaches you the value of exploration. Aside from these things to do, the most important thing to learn in my experience is how to constructively criticize existing research work. Papers are typically not very good at listing their flaws and you can't fix things you can't see. For research, you need to cultivate an eye for the limitations, most importantly the limitations of your own work. This is somewhat contradictory, because to be a great researcher, you need to both thoroughly understand the limitations of your work and be enthusiastic about it. - John

There seem to be two issues here - An engineering solution is to export code environments with all the hyperparameters (say in a Docker image), so that someone else can grab the Docker and run the code to exactly reproduce the plots in the paper. But this is a bandaid that is covering up a more serious issue - The more serious issue is that Deep RL algorithms are notoriously unstable and non-robust (A precursor problem is that DL itself is not very robust). Naturally this has an effect on reproducibility, but it also suggests that these methods have limited real-world potential. The way to address both of these issues is to develop more robust algorithms. -Akshay

The meaning of "human" is perhaps part of the debate here? There is much more that I-as-a-human can accomplish with a computer an an internet connection than I-as-a-human could do without. If our future looks more like man/machine hybrids that we choose to embrace, I don't fear it that future. On the other hand, we have not yet really seen AI-augmented warfare, which could be transformative in the same sense as nuclear or biological weapons. Real concerns here seem valid but it's a tricky topic in a multipolar world. One scenario that I worry about less is the 'skynet' situation where AI attacks humanity. As far as we can tell research-wise, AI never beats crypto. -John

This becomes easier with experience, but it is important to have a solid foundation. - Akshay

This depends a great deal on what you want to do programming-wise. If the goal is implementing things so that other people can use them (i.e. software engineering), then little background is needed as long as you can partner withone someone who understands the statistical side.

If the goal is creating your own algorithms, then it seems pretty essential to become familiar with the statistical side of machine learning. This could be an undergrad level course or there are many online courses available. For myself, I really enjoyed Yaser Abu-Mustafa's course as an undergrad---and this course is online now. Obviously, some mastery of the programming side is also essential, because ML often pushes the limits of hardware and embedding ML into other systems is nontrivial due to the stateful nature of learning processes. -John