Pushing the Frontier in AI

The Panellists (right to left) included:

Janusz Marecki - the founder and CEO of Fractal Brain. Janusz has had a 20 year distinguished career at the cutting edge of research at Frontier labs such as as IBM Watson and more recently Google Deep Mind before now branching out to start FractalBrain, a complete new innovative AI architecture that mimics some the processes of the human brain to achieve intelligence at "test-time". In our discussion it would be fair to say that Janusz does not believe that the route to Artificial General Intelligence is through either scaling LLM's through more data and more compute or by smart engineering such as "Context Engineering" (https://www.youtube.com/live/Egeuql3Lrzg) to optimise LLM performance on given tasks or by smart "scaffolds" to overcome LLM's limitations such as lack of memory,. Specifically Janusz believes LLM's transformer architecture will never have the ability to continuously learn at test time, when inference occurs, as humans do and as such the relatively puny Human brain ( with an approximate 2 million token context window) can still do many "intelligent tasks" better than the billion/ trillion token context windows of the latest and future LLM's. For Janusz the focus should be on adaptation over generalization in AI models. Adaptation in AI models emphasizes the ability to learn and adjust to new information or environments in real time, rather than solely relying on pre-existing data for generalisation. This will allows AI systems to handle novel situations and data distributions that were not present during training, enhancing their robustness and applicability in dynamic contexts. Focusing on adaptation can also lead to more efficient learning processes, as models can update their knowledge incrementally without needing extensive retraining on large datasets. For this and many other reasons Janusz has staked his future career on building a new architecture that can match some of the outstanding things that LLM's can do but also the amazing things we as humans are able to do, and this is what Fractal Brain hopes to start demonstrating in the next 12 months. In fact he is hoping in the next 12 months, to demonstrate that Fractal Brain will be able demonstrate a Lifelong Learning model that can build a knowledge base at "test time" that then can be trained and developed to have the intelligence capabilities of a Human Toddler . Watch this space https://fractalbrain.ai

Zhengyao Jiang - the Co-founder and CEO of WeCo, in my view the most exciting and interesting of the new startup Frontier Labs operating anywhere in the world. In 2024 Open AI released MLE Bench to evaluate Machine Learning Agents on Machine Learning Engineering tasks. In the paper they showed that all the then latest LLM models ( Gemini, Llama, Claude and GPT) significantly improved their performance when supplemented with WeCo AI Driven Exploration (AIDE) agent. The paper coined the phrase "scaffold" when referring to WeCo AIDE agent, which in the panel discussion, Zhengyao revealed at the time he was upset with the description of their cutting edge agent as a 'scaffold", for it implied that "Scaffolds" are temporary when in fact they are necessary "outer loop computation" (a phrase Zhengyao prefers) to enable LLM's to perform complex tasks such as Machine Learning experiments. Only last week, Meta's AI research lab released two papers that used Weco's AIDE to generate new innovative findings, and now all the frontier labs are heavy users of their open sourced AIDE API. On the panel Zhengyao praised the present and ongoing achievements of LLM's and forecast that some things in the next 12 months such as code generation by LLM's will surpass humans. Zhengyao was bullish on context engineering that goes beyond prompt engineering which attempts to efficiently find out what is within the LLM's context window, and instead focuses on what and how that window is being filled. It is about understanding the model’s exposome, the documents and information it sees, and the nature of that exposure, i.e. the systems for retrieving and recalling information. Nevertheless achieving AGI through LLM's is in Zhengyao view someway off mainly because "intelligence is not one-dimensional" and stubborn problems like LLM's lack of memory still need to be solved. Current AI has limitations in continuous memory. Currently LLM's lack the ability to retain information across sessions, resulting in a limited context window that hinders long-term learning and adaptation. The inability to update memory dynamically means that AI cannot effectively recall past interactions or knowledge, impacting its performance in tasks requiring continuity. This limitation restricts AI's ability to learn from new experiences in real-time, making it less effective in environments that require ongoing learning and adjustment.

Post the panel Zhengyao posted on X more details on his present thoughts on where is the frontier in AI. The post is a must read. ”https://x.com/zhengyaojiang/status/1942946852909056211?s=46 . Partly because he quotes Yann LeCun's famous observation that "Research in AI is like climbing a mountain range in the fog. Every time we reach what we thought was the summit, the clouds lift a bit and we realise we were only on a ridge with higher peaks ahead." but mostly because it succinctly states where we presently are. The graph below neatly capsulated his thoughts

The final point worth discussing is whether we are quickly approaching the "recursion point" where present AI creates the next generation of AI. Zhengyao was reluctant to say anything on the matter but hinted within the next 12 months WeCo may have something definitive to show that will settle the matter. Watch this space- www.weco.ai

Edward Grant - is an AI scientist with a PhD in machine Learning from UCL, where he cofounded AI in Drug Discovery startup Rahko that was acquired by Odyssey Therapeutics where he was head of AI until last year. Presently Ed is our AI expert in Residence at Twin Path, helping us carry out in-depth technical DD on AI first startups seeking our investment but in the Autumn he plans to lead a research project at UCL ( that might lead on to a new startup) on World Model in Robotics.

Ed recounted his experience of using the power of generative modelling in the exciting world of drug discovery. He reported there is a natural tension, between novelty of the molecules, compounds and protein discovered and their reliability or plausibility. The further away that a new found molecule is from the training data, the higher the likelihood of novelty. This is the new-to-nature stuff which all drug discovery labs dream of. However, the further away you get from the training data the harder it becomes to trust the answer. You have to take steps to show that your AI prediction engine is actually working. You have to test the results in wet-labs, refine and test again, and repeat again and again and in these cases, it is an interdisciplinary approach combining AI and biology expertise that wins by being able to generate a novel molecule that can be the basis of a patent protected drug that can be used to cure or mitigate a disease. Methods to create surrogate digital environments out of synthetic data and the use of Physics based models and techniques such as such as Density Functional Theory show promise but we are still in the foothills of frontier AI enabling humans to confront one its greatest foes - disease.

Ed also discussed its excitement in the recent Multimodal advancements that are enhancing AI's ability to process and understand complex data types, especially video. The development of Vision-Language-Action models allows LLM powered applications to interpret instructions and visual data simultaneously, facilitating real-time decision-making and control in dynamic environments. Ed believes the exciting VLAM's are powering a lot of research and investor interest in embodied AI- in self-controlled robots with the view that we may be able to create robots with a world model processing capability that can represent a specific past, present and future state and thereby learn and act like humans in the multimodal world to carry out complex and useful tasks. Yes the robots are coming and in the autumn Ed plans to work with UCL in their Robotic labs to see if he can be one of the first to overcome some of the outstanding challenges.

In the meantime he is working with us at Twin Path to source and select the best in Frontier AI

Cesc Cunillera Cesc C. - is the cofounder of an exciting stealth startup in the field of World Modules for Self Controlled robotics and autonomous vehicles. Cesc is an ex- AI Research Scientist at AI powered financial trading house and also happens to have a PhD in String Theory and one his more insightful comments that the lesson from Physics is that brute force methods ( more data, more compute, bigger transformer architectures etc) will not work as the sole means of achieving AGI. As Cesc posted out brute force approaches are highly inefficient. They often require an impractical amount of computational resources to explore vast parameter spaces, making them inefficient for complex problems. The complexity of physical systems often leads to an exponential growth in possible configurations, rendering brute force methods infeasible within reasonable timeframes. The unique set of fundamental laws governing the universe means that random sampling of parameters can lead to an astronomical number of possible outcomes, making brute force impractical. And yet Cesc and his team actually refer to Rich Suttons much cited paper The Bitter Lesson where the famous opening line is "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin" .http://www.incompleteideas.net/IncIdeas/BitterLesson.html . The reason being that they believe they can use the size and latest improvements in LLM's including their reasoning capabilities, the latest in VLAM's, alongside leveraging physical principles and calculations to significantly reduce the search space. The objective for Cesc's stealth startup is to use this power to improve the effectiveness of generating a large amount of diverse and accurate synthetic data derived from small samples of video data of robots working on tasks in order to create surrogate digital worlds on which you can use Reinforcement Learning techniques to train the next generation of self-controlled robots. The thing that Cesc believes is holding back embodied AI is the lack of robotic data ( tiny compared to all the data on the internet used to train LLM's) and that is why they think bigger and better data sets are required. So back to one of the constant rules of AI both now and at the frontier (if we stick to the present architectures) that progress and achievement is all about the data. Even better data on smaller LLM models can and will often outperform. Finding ways to overcome the limitations in the availability of data is an ever present challenge and opportunity for startups.

We concluded the discussion on where we believe there are opportunities in the next 12 months for AI first startups with novel SOTA solutions. The panel suggested these startups should focus on solving problems in:

• Multimodality including the development of AI systems that can integrate and process multiple types of data (e.g., text, images, and video) to enhance decision-making and improve user interactions.

• Efficient learning techniques, such as active learning and transfer learning, present opportunities for startups to create AI models that require less data and computational resources, leading to faster deployment and reduced operational costs.

• Context engineering - an emerging technique that allows for the creation of AI systems that can better understand and utilise external information, improving their accuracy and reliability in real-world applications, thus opening avenues for innovative applications in various industries.

The panel also thought there were opportunities for smart startups with smart solutions in developing

• Applications based on vision-language-action models in robotics.

• New scaffolding ecosystem that enhances AI agency by providing tools that enable decision-making and task execution. These tools should allow LLM systems to interact with their environment more effectively, improving their performance in complex tasks

• Adaptive AI

The discussion on Frontier AI was both illuminating and fascinating. It helps us at Twin Path form our investment thesis on why investing in world class technical AI talent to build the frontier in AI is both super exciting, challenging and potentially financially rewarding. If you are an AI startup pushing the frontier then do get in touch.