Shape the Technology that will Shape your Future

    Nagymaros AI Retreat

    14 - 24 August 2023

  • Topics


    Large Language Models

    In last year's AI retreat, the GPT-3 language model was barely known outside AI research, and participants accessed it via an obscure AI. Fast forward one year, and ChatGPT is writing most of your classmates' homeworks. The space of massive language models trained on hundreds of millions of documents crawled from the Internet has developed tremendously over the last year and people now predict language-based AI will have a comparable impact to the Internet itself. We will look at how these models are progressing in their ability to solve high-school-level mathematics problems. We have some data from the Náboj competition to use for this purpose.

    AI for Science and Mathematics

    A few participants this year are particularly interested in physics and engineering, others in mathematics. We will look at some applications of machine learning in physical modelling, control engineering as well as mathematical problems. We will learn about reinforcement learning and train a control system for a lunar lander. We will look at how ML can be used to speed up expensive computations in modeling physical or chemical systems.

    What can possibly go wrong?

    AI capabilities develop at amazing speed. New models and breakthroughs in performance are announced almost on a monthly basis. If you're just starting university now, who knows what this technology will look like by the time you graduate. We will also talk a bit about AI safety and ethics. The field of AI safety looks at ways things can go seriously wrong as this technology gets more powerful. AI alignment tries to develop techniques and processes to ensure AI it's used for more good than bad and that it follows instructions in ways we intended it to. Meanwhile, AI Ethics is broadly concerned with ensuring that (dumb) AI that is already deployed everywhere can be trusted to effect society in a fair way and with respect to people's privacy.

  • Sponsorship

    This year's event is sponsored by G-Research. Thank you!

    broken image
  • Agenda

    Monday, 14 August

    • arrival at Mihály utca
    • 18:00 Welcome and Dinner

    Tuesday, 15 August - AI in Quantum Chemistry

    • presentation by Prof. Gábor Csányi
    • exercise: invariant many-body representations

    16 - 23 August - project work

    • 8:00-9:00 breakfast
    • generally working a lot
    • also boardgames, walks, sports, with bonus fireworks on 20 August

    Thursday, 24 August - demo day and closing remarks

    • retro meeting (mad, sad, glad format)
    • project presentations
    • saying goodbye

    Friday, 25 August - leaving home

  • This Year's Projects

    We worked on some pretty awesome problems this year:

    AI for Generating Mathematical Constructions

    In several branches of mathematics, we are interested in finding largest or smallest object with a certain property in a combinatorial search space. For example, in Ramsay theory we are interested in finding the largest graph that does not contain certain subgraphs. Following Ádám Wagner's work, in this project we leverage reinforcement learning to find such constructions. After a brief experimentation with Ramsay theory, we turned our attention to the no-three-in-line problem: selecting the maximum number of grid points on an n×n regular square grid such that no three of the chosen points fall on the same line.

    Q-Learning in the Moon Lander and Frozen Lake environments

    Another project explored a particular flavour of reinforcement learning in widely used benchmark environments. This project explored deep Q-learning, tabular Q-learning, curriculum learning (changing the strength of gravity for the lunar lander). Playing with these environments allowed the group to understand how and why Q-learning works, to gain first-hand experience with the difficulty of neural-network-based reinforcement learning, and to gear up to solving more difficult problems.

    LLMs to solve Náboj mathematics problems

    To everyone's big surprise, some large language models (LLMs) seem to be capable of mathematical problem-solving, even without explicit training. In this project we processed past problem sets from the Náboj high school mathematics competition, and evaluated publicly available LLMs on their ability to solve these medium difficulty problems. The team found that without creative prompting techniques even the likes of GPT4 are generally unable to do this. Only when using techniques like progressive hint prompting (PHP) did the models perform at reasonable level.

    A new invariant many-body descriptor for atomic environments

    on Day 2 Prof Gábor Csányi visited and gave a presentation about the use of AI techniques in quantum chemistry. He presented retreat participants with a challenge: design a descriptor of a set of points which (A) is invariant under permutations, translations and rotations, (B) varies smoothly with its inputs and (C) is invertible. We discussed why the multiset of atom-to-atom distances does not satisfy the invertibility criterion, and Gábor presented a pair of counterexamples that map to the same representation. He then left the group to think about a better solution, and we did. The result is a genuinely new idea, that solves the problem in a very simple and computationally tractable way.

  • Venue

    Nagymaros, Hungary

    broken image

    Mihály utca

    Our home for the week is a nice house in Nagymaros, close to the train station with lovely views over the river and surrounding hills. The house has communal areas, a kitchen, and rooms with 2-4 beds in each.

  • Speakers and Mentors

    full list of speakers TBD

    broken image

    Ferenc Huszár

    Associate Professor of ML and AI, Cambridge

    I studied Computer Engineering at BME (Budapest) and got into machine learning after joining computational neuroscience research group. I did a PhD in Probabilistic Machine Learning at Cambridge. I worked in tech for 10 years, including startups, venture capital, and four years at Twitter. I joined Cambridge as an Associate Professor in 2020, where my group does research on the theory of deep learning.

    broken image

    Botos Csabi

    DPhil Student in Computer Vision, Oxford

    Csabi started out as a neurobiologist at the institute of Experimental Medicine (MTA KOKI) in Budapest working on reconstructing 3D structures of brain cells responsible for short and long-term memory. To understand more about the underlying cognitive mechanisms he started his PhD in Oxford to work on continual learning theory in Philip Torr’s vision group. As a big fan of applied theory, he can do backflips and remember to fall face first when needed be.

    broken image

    Bea Benkő

    PhD student at Rényi Institute/ELTE

    I have been studying Mathematics and Computer Science at ELTE (Budapest), and I am currently a final-year PhD student working on continual learning, representation learning, out-of-distribution detection, machine unlearning. After working for over two years as a software developer during my undergraduate studies, I joined the AI research group at the Rényi Institute. As PhD student, I have participated in organizing machine learning journal club and deep learning seminars at ELTE, I have supervised BSc theses, and as AI module leader at the Milestone Institute, I have been mentoring high school students as well.

    broken image

    Gábor Csányi

    Professor of Molecular Modelling, U of Cambridge

    After and undergraduate degree in mathematics (Cambridge), and a PhD in physics (MIT), Gábor is worked in the Cavendish Laboratory and then accepted a faculty appointment at the Engineering Laboratory in 2007, where he is now Professor of Molecular Modelling. He is interested in molecular dynamics, molecular representations and machine learning, and also statistical mechanics.

    broken image

    Ádám Wagner

    Assistant Professor of Methematics at Worchester Polytechnic Institute

    I studied mathematics at Cambridge, and continued working in pure mathematics by doing a PhD in Illinois. I got interested in machine learning during my two postdocs at Zurich and Tel Aviv. Currently I am an assistant professor of mathematics at Worcester Polytechnic Institute, and a mathematical consultant for Google DeepMind.

  • Participants

    Full list of participants TBA

    broken image

    Béla Bálint

    Just finished 12th grade in Szeged and about to start physics-engineering at BME, Béla has a wide range of interests including coding and mathematics.

    Former intern at Zeto Eu kft.

    broken image

    Zsófi Keresztély

    I am about to start the 11th grade in a special mathematics class in Budapest. My main interests are competitive programming and mathematics, I spend a lot of my free time in maths and prog camps. I really love competing as well, I am both an EJOI and an EGOI medallist.

    broken image

    Ádám Vajda

    About to start 12th grade at ELTE Radnóti. Former TUDOK IT Grand Prize winner. A skilled coder and pytorch user, Ádám already has broad experience with convolutional networks for computer vision as well as reinforcement learning.

    broken image

    Franciska Tallián

    Competitive programmer who is about to start 11th grade in Szatmárnémeti, interested in learning and improving in CS, as well as understanding the intersection of technology and society.

    broken image

    Hédi Kovács-Bánhalmi

    About to start 11th grade. Robotics expert with over 6 years of experience and a programming enthusiast. Member of successful WRO and FLL teams, received the Kecskemét Excellence Award twice. Alongside technical pursuits, she enjoys teaching kids programming and has a keen interest in AI. She was also a finalist at the TUDOK 2023 Conference.

    broken image

    Boglárka Ecsedi

    Third year CS Student at Georgia Tech, Student researcher in Judy Hoffman’s Group. She holds a Stipendium Peregrinum Scholarship, and is a former ISEF and EUCYS finalist.


    broken image

    Benjámin Markovics

    About to start 12th grade at Veres Péter Gimnázium, interested in math and physics, likes competing, and plans to study engineering at university.

    broken image

    Georgina Anna Zsóri

    About to start grade 12 in Gyöngyös, interested in Mathematics, Physics and Computer Science, would like to study aerospace engineering at university.








    broken image

    Marcell Jenei

    About to start grade 12 in Kisvárda. Interested in Mathematics and Computer Science. Enjoys participating in Maths and Computing competitions.

    broken image

    Csanád Pászti

    Starting 12th Grade this September in Budapest.

    With for 4 years of prior programming experience, my main interests

    are CS and mathematics, and I look forward to improving

    in these subjects.

    broken image

    Erik Juhász-Molnár

    Starting 11th Grade this September in Budapest.

    Likes challenge programing, mathematics physics and football. Doesn't like appearing in photos.