Japan is lagging behind in AI, but that might not be the case for long.

Today we sit down with Jad Tarifi, current founder of Integral AI and previously, founder of Google’s first Generative AI team, and we talk about some of Japan’s potential advantages in AI, the most likely path to AGI, and how small AI startups can compete against the over-funded AI giants.

It’s a great conversation, and I think you’ll enjoy it.

Podcast: Play in new window | Download | Embed

Subscribe: Apple Podcasts | RSS | More

Show Notes

  • Why Jad felt Google was not pursuing the best path toward AGI
  • The fundamental AI scaling problem and likely solutions
  • Why robotics is critical for the advancement of AI (and the not the other way around)
  • Why Japan is the ideal place to build a new AI startup
  • The reason it is so difficult for robotics startups to make money
  • Why humanoid robots are a dead-end
  • How AI startups can compete with the foundation-model comnpanies
  • How we get to AGI from our current AI
  • Solutions to the alignment problem
  • The challenge of making AI fundamentally benevolent
  • The biggest challenge in AI development is not technological

Links from our Guest

Transcript

Welcome to Disrupting Japan, Straight Talk from Japan’s most innovative founders and VCs.

I’m Tim Romero and thanks for joining me.

Japan is lagging behind in AI, but that was not always the case. And it won’t necessarily be the case in the future.

Today we sit down with Jad Tarifi, current founder of Integral AI, and previously founder of Google’s first generative AI team. We talk about his decision to leave Google after over a decade of groundbreaking research to focus on what he sees as a better, faster path to AGI or artificial general intelligence. And then to super intelligence.

It’s a fascinating discussion that begins very practically and gets more and more philosophical as we go on.

We talk about the key role robotics has to play in reaching AGI, how to leverage the overlooked AI development talent here in Japan, how small startups can compete against today’s AI giants, and then how we can live with AI, how to keep our interest aligned.

And at the end, one important thing Elon Musk shows us about our relationship to AI. And I guarantee it’s not what you, and certainly not what Elon thinks it is.

But you know, Jad tells that story much better than I can. So, let’s get right to the interview.

Interview

Tim: I am sitting here with Jad Tarifi, founder of Integral AI, so thanks for sitting down with me.

Jad: Thank you.

Tim: Integral AI, you guys are “unlocking, scalable, robust general intelligence.” Now that’s a pretty big claim, so let’s break that down. What exactly are you guys doing?

Jad: So, when we look at generative AI models right now, they usually operate as a black box. And because they have minimal assumptions on the data, they have to do a lot of work and they tend to be inefficient in terms of the amount of data they need and the amount of compute. We’re taking a different approach that’s inspired by the architecture of the neocortex, which roughly speaking follows a hierarchical design where different layers produce abstractions and then feed into higher layers that create abstractions of abstractions and so on.

Tim: Okay, so this is not an LLM architecture or is this a kind of LLM architecture?

Jad: When people talk about LLM, usually they talk about auto regressive transformer networks. So this would be a different type of architecture than that. However we can use transformers or other models like diffusion models as building blocks within that overall architecture.

Tim: It’s interesting that you took a different path than LLMs because you’re not new to AI. You led teams at Google for what? Nine years or so where you were working with transformer architecture. So, you know this technology deeply. What made you decide to not only leave Google and start a new startup, but to leave the LLM path and pursue a different technological architecture?

Jad: So, this all goes back to my PhD where we were exploring how the neocortex could work from an algorithmic perspective. And in fact, when I started the first generative AI team at Google, we were targeting how to have models that can imagine new things, which what we call generative AI right now. Transformers was one of the very exciting, scalable architectures to do so, but there were clearly limitations there that I cared about deeply because I do care about these models affecting the real world, and there was a bottleneck there in terms of reliability and in terms of efficiency. And from my work on the architecture of the neocortex, it was clear that there is a path that goes beyond the current models and I could pursue that path at Google, but it also felt that there’s a new class of applications that are going to be unlocked to have nothing to do with search that more about the physical world robotics movement, real time user interfaces, all of those exciting things that felt a little bit outside the box. And so it felt like it’ll be good to have a new company with a blank slate so we can move fast and make an impact.

Tim: Integral AI, let’s see, you guys launched in 2021, right? So this was a solid year before, well before generative AI became mainstream. So, has that prediction played out? So, generative AI in the last two years has increased significantly in accuracy and reliability, but do you think it’s going to hit a wall or has hit a wall in terms of how accurate it can be?

Jad: No, I don’t think generative AI will hit a wall at all. As someone who is one of the founders of generative AI, I think the sky’s dilemma, I think we are going to go to general intelligence and beyond human level all the way to super intelligence. In fact, I think the transformer architecture will continue to improve, but the rate of return on general pre-training of these architectures have reached diminishing returns right now. So, you need to spend 10 times more energy, 10 times more data to get the next step up. And if you give me infinite energy, infinite data, then I can do anything. But we are already seeing our models can do much better with far less resources and they have better scalable qualities. So you think about it as the slope of your scaling exponential.

Tim: So, I appreciate the thought that LLMs aren’t going to hit a wall, but if we do have that kind of a scaling issue, we can theoretically come up with 10 times more compute. But is there 10 times more data and is it, is it like quality data? I mean, sure we can train LLMs on YouTube videos and TikTok, but I’m not sure we really want to use those.

Jad: I’ll answer that in three different ways. One is by expanding to modalities. As you mentioned, YouTube vision is largely under explored. Just training on all the text on the internet is not enough. There’s a lot of other data sources including proprietary data sources, but also multimodal data like vision, like sound and all that stuff. But even that has fundamental limits. The next thing is about actually having humans create new data. And I think there’s a lot of ethical issues there. You know, a lot of the data is creating in poor countries, people who are underpaid. A lot of the labs are exploring this strategy and I think to some degree there’s some success there, but I don’t see it as entirely scalable long term. Third and most promising approach is the new paradigm of test time scale. So, these models can look at data reason and then generate new reasoning chains or plans, thereby creating better and better data for themselves. So, there’s this cycle in psychology, it’s system one and system two thinking. So when we think, when we make a plan, this plan becomes a new, fresh, high quality data that we can retrain our intuition. And an example would be chess. When you start off a chess, you need to maybe think about every single move. And as you become more experienced, those naturally come to you and you still have to plan, but you’re planning at the higher level. And so there’s this loop system one gives you a better system two, system two gives you a better system one. And so there’s a real sense in which this model can self-boost, trap and create their own data.

Tim: Okay, that makes sense. Let’s get back specifically to Integral AI and the work you guys are doing. So you mentioned the, the importance of multimodality and you and the team are doing a lot of work with robotics and with DENSO Wave.

Jad: Yeah, so ultimately the way AI is going to impact the real world is through taking physical action. And the form that computers take physical action in the world is robotics. So the way we define robotics is any controllable physical tool. So that includes cars, drones, but even elevators. So anything that you can move intelligently would be in our category, a robot.

Tim: Well, it sounds like just about anything with a physical manifestation of it would be a robot, anything that can interact with the world.

Jad: Right, that’s our expansive definition of the world. And if you are going to interact with the world, it’s really helpful to understand the world. And the most rich sense for understanding the world is the visual sense. As humans, you know, about 40% of our neo cortex is specialized for vision. So we tend to spend a lot of our energy processing the real world visually, compliments language very well because language is the natural modality for abstract thinking. So there’s the abstract thinking through language and there’s the real world grounding through vision. And so the world of abstraction is already compressed enough that you can get away with a very inefficient algorithm. But as soon as you go to the real world with vision, the data science becomes very large. The complexity becomes much larger in terms of the dimensionality of the problem. And so you need better and better algorithms. Our technology, of course, we can attack these language problems like LLMs, but where it shines is on these harder problems that occur in vision and definitely in the real world. So we thought that robotics is a really nice toy problem or like a specialization to focus the algorithms on. And so we collaborated with DENSO Wave, and many other companies like Honda and we try to find opportunities that this technology can affect their product line now.

Tim: The application of AI and robotics, the fact that there is a physical component to it seems not only difficult, but a unique opportunity for training data in the sense that the robots are interacting with the world, they’re getting their own feedback. They’re in a sense at least theoretically, can explore the world in analogous way to what we do. So in a sense, could they provide their own training data through their own experiences, their own interactions with the world?

Jad: Absolutely correct. I think you identified a key important insight that’s driving us. Our models now are reaching the point where they can collect their own training data. You can ask a question like invent a new drug for me and thinking, go out and do experiments, combine different molecules together, update a theory about how the system or how the drug interaction works, and then do other experiments and have that loop almost like automating the scientific process,. We call that internally active learning. So it’s a process of learning through taking action. It’s something we’re very excited about and we have simple versions of that right now already. We’re actually building up to a much more general version of this that’s going to come out in the next few weeks. The active learning group, we think is the key to achieving AGI.

Tim: I want to put a GI to the side for a second. I really want to get back to that, particularly as it applies to robotics. But before that, I want to ask you one question about your corporate structure. So you’re based in San Francisco, but you have offices here in Tokyo, I mean, we’re sitting here in Tokyo, you’ve got engineering teams here in Tokyo, I believe. So why so many companies are saying that there’s this talent shortage in Japan, they’ve got to go to San Francisco to find engineers, but you’re having success right here in Japan. So what are you seeing that so many other startups are missing?

Jad: The first is, well yes, there’s a lot of talented AI researchers in the Bay Area, but the robotics industry is relatively immature in the US compared to Japan. Japan is about 50% of the world’s industrial robot production. So we have a very strong ecosystem on the robotics side based in Tokyo. So our thought was to combine the primary area for AI, which is Silicon Valley and the primary area for robotics, which is Tokyo, Japan. This has been a really good bet because what we found is that there’s a lot of underappreciated engineering talent here. There is a language barrier. So one of the things we got really good at is helping young talented engineers get comfortable in an English speaking environment. We have a lot of best practices to help them kind of adapt to that. And usually it takes about three months to six months for them to get comfortable. The other thing is that we are not a research lab, so we don’t really need too many senior AI researchers exploring different ideas. We have a clear vision about what we want to build. And so it gets easier in the sense that we can afford to hire a lot more junior people. The value of someone with high energy, high integrity, great teamwork is much more than like industry experience for us at this stage.

Tim: So, any advice you have to all of those startup founders who think they need to go to the Bay Area to find talent?

Jad: I think to be open-minded is very important. It’s very attractive to say let’s go to the Bay Area. But in fact, a lot of the entrepreneurs I know in the Bay Area would say it’s too expensive in the Bay Area. Talent is too fickle. You hire someone, you train them for a few months and then they go to the next job. A bigger picture. I think we want more diverse opinions to be shaping AGI, we don’t want just one physical location, one monoculture. So I’d like to see that kind of spread out a bit more around the world.

Tim: So Japan is, as you mentioned, traditionally very strong in robotics, but robotics is hard. I mean, most startups fail, but I mean really almost all robotics startups fail. Why is it so hard to make money here?

Jad: This is a great question. I think we can spend an entire interview just on this question first. There’s this tendency when you are in a small startup to be overwhelmed by the robotics stack. There’s too many things, too many difficult things that you need to solve. And so what you end up doing is saying, I’m going to build this one product and let’s see robotic picking, and then I’m going to collaborate with this robot arm company and then collaborate with a system integrator and then sell to the company, there’s too many players and nobody has the complete ownership of on the product. And when no one has the complete ownership on the product, it’s hard to iterate.

Tim: Right. So you can’t like push something to market, right?

Jad: Yes.

Tim: It’s too much of a consensus building.

Jad: Exactly. So you end up in a state where it is just not favor startups that have to move fast, that that’s a huge bottleneck. And, and, and in order, if you want to own the whole thing, you need a lot of technical expertise and you need a lot of resources, which startups are not known to have. The second…

Tim: But wait, no, let me push back on that because even companies like look at Boston Dynamics. So they’ve got the resources, they’ve got the technical expertise, they have God it seems like unlimited funds, but they just can’t bring a product to market. I mean, they’ve got sales, but it’s all these kind of cool little pilot programs and even they can’t seem to like really be a robotics startup success story al although they have incredible tech.

Jad: I would challenge the incredible tech part. I think their tech is more traditional stuff. And I think they’re also adapting because they’re realizing the traditional control theory approach that they took is not really that scalable, but their demos are cool and impressive and inspiring. Now I like your pushback because you’re needing naturally to my second point. The second major point is, how attractive is it to have an automated solution that is specialized? So if you are a factory assembling phones in China, here’s the unfortunate reality. You can choose to buy a robot or you can choose to hire someone for $10,000 a year. And the metal on the robot itself is already more than $10,000 a year. So customer acquisition is very hard because again, there’s availability of cheap labor. Second, the sales cycle is very long. Sometimes I talk with some big name, very big name companies in Japan and they’re planning a factory. And I say, great, when are we going to go to production? And they say, oh, 2032.

Tim: That makes sense. They’re not going to re-engineer their line every year, right?

Jad: So you’re in the construction business, you’re not really in the technology business. The third component to why startups have failed in robotics is unfortunately the technology wasn’t there yet. You needed to have different technology per use case. So you need to basically get an engineering team, design a solution for every use case. And that adds up to the total cost of the system. But as we enter the generative AI era and AGI, we are going to have one solution that works across a wide variety of problem.

Tim: How universal is robotic training data. If I have data from an auto factory robot and I’ve got data from, I don’t know, Boston Diamonds Atlas and I have data from a robot that makes, makes and serves coffee, how much use is training data from one domain to another?

Jad: This is great question and it’s a common misunderstanding. In the investment community in Silicon Valley ecosystem, they assume that robotics is hard because there’s not enough data and that you need to go out and collect data for every robot environment. And that’s part of the story. For example, why human aids are great because you have one form and then you can collect data for, but actually a lot of how we learn is not through interaction, but in watching other people do something. In fact, what we found is that if you train a model to understand the world, it can apply across different bodies. You do need some interaction with the new body, but that’s just like stretching out before you go to the gym. And actually, it turns out the more the model that does the stretching generalizes well. So, if you trade on 10 types of robot, the 11th robot will be even easier. So we see a world where there’s going to be a whole Cambrian explosion of robot bodies, many, many different robot types doing different things.

Tim: Okay. But that sounds like a really great machine learning AI niche to be focused on right now. But if I could ask you a broader question about AI startups in general. So what are your thoughts, on how smaller AI startups can compete with these firms that are making foundation models like OpenAI, Anthropic, and Meta have unlimited funds, great talent, and without moving into things like hardware strictly within the software space, do you think there is defensible business models within generative AI right now?

Jad: I do. And I think your question can be divided into sub-questions. I’ll first answer how to compete against a larger company and then we’ll discuss the business model because that’s also very interesting. If you are just training a transformer model, I don’t think there’s much hope there because what you’re doing is clear to everyone. But if you do have a technology differentiation in terms of the architecture, the algorithms of what you’re doing, there is a chance. But that’s not enough because everything can be copied eventually. So what’s the startup advantage? It’s always the ability to make decisions quickly, iterate fast, so you have a more ability to explore an uncharted territory. And that becomes very important when you have a platform change. When you have a platform change, there’s a lot of opportunity and the winner is usually the one who can explore that faster.

Tim: To dig down on that, what would be some domains or some, some concrete examples where that might make sense? Because I see so many AI startups and most of them are like, wow, that’s a clever idea. But if that market ever got big enough, one of the big companies would take a small step sideways and take it from you.

Jad: It needs to be a platform change. It cannot be a feature, it has to be a fundamental platform change. That’s one Second, it has to have a strong defensibility to, so there has to be a moat. So right now we’re in the middle of a platform change and one of the platform changes towards LLMs. The classic example there is Open AI. Open AI was much less resourced in every way in terms of data, in terms of talent, in terms of financial resources than Google. And we are seeing right now, Google is struggling, I would say the least compared to Open AI. And this is not the first time. I mean, Google did that to Microsoft. So whenever you have a platform change, you have an opportunity for startups to win purely because they can maneuver faster. The second very important thing is that you have the capacity to build a moat. Clearly to me, in the physical world, there’s a huge moat when it comes to having models that continually grow with time and benefit from Gale on data. So if you have a model that is up to date that a lot of people already use, it already contains so much more information than what you can find publicly or you can train from scratch. Now with traditional generative AI, you can’t do that well because you have this model that you train once and then you run in inference mode. One of the breakthroughs we’ve unlocked in our company is the ability to do continuous training. The model can add new data, can add new experiences and not forget. So, there’s a problem in neural network called catastrophic forgetting. We’ve solved that problem in a clever way. And so this opens up the opportunity for a strong network effect on data.

Tim: Let me flip the question. Do you think there’s defensible value in building these foundation models? GPT, Lama, Claude, they all have different characters when you use them, but honestly the objective performance differences aren’t that obvious. And it’s all Google’s transformer architecture, it’s all open source. There’s not really IP that’s defensible here. So looking at Open AIs $157 billion valuation, safe super intelligence just raised a billion dollar pre-seed round. Is this defensible? I mean long term are these valuations going to be justified not for the technology mind you but for these specific companies or are we going to see lower cost competitors coming to the market?

Jad: So, first I want to make a distinction between the success of any one company versus a technology. The technology itself is world changing and if any one of those companies succeeds and manage to scale it up, then whoever invested in that is going to get the multiples and is going to justify the investment ecosystem. Is there something defensible about building a model from scratch? I would say no. the best it can give you it’s a head start. So even our technology is not transformer based is completely new yet we don’t believe its defensible if that’s the only thing. What’s defensible is always the ecosystem you can build, the relationship with the end user. And there is a bigger point here, this is such an important change for humanity maybe it doesn’t matter if many companies die trying.

Tim: That what I think might be a very likely scenario here, I mean I know its completely different but like the solar industry in the late 1990s, 2000s I mean so much money was lost. Most of those companies went out of business but the technology itself continued to advance and its having a transformative impact on the world. And I’m wondering if we might see the same thing with generative AI with the technology goes on to have a transformative effect and 20 years from now the investment community is looking back going my God what were we thinking.

Jad: I would agree that the vast majority will die but the bigger picture doesn’t matter because the social benefit will be there. And from an investor perspective there’s still some companies that will make it and those will be worth your investment.

Tim: Excellent, do you want to talk about AGI?

Jad: Sure.

Tim: All right, before we dive in let’s do definitions. How do you define AGI?

Jad: For me AGI is the ability to learn new skills, unseen skills and this ability to learn new skills has to be executed safely so without unintended side effects and second efficiently. So, with energy consumption at or below a human learning that same skill. Learning new skills as opposed to existing skills. Why? Because you can always brute force by training on a lot of human data. So the key is not existing skills, but the ability to acquire new skills.

Tim: But haven’t we already crossed that threshold? I mean, don’t we have both robots and software that can learn new skills and adapt?

Jad: If you limit the skill scope, you can have a model that can learn. But if you’re saying completely unseen skill, we can root force it by giving a lot of examples of that skill. And that gives you the whole lacking efficiency. Or we could have the robot just try every potential movement themselves. And that leads you to the lacking safety issue.

Tim: I mean, those are worthy goals of any type of AI. It seems like they should be kind of table stakes. You know, you don’t want to burn down the factory, you don’t want to waste resources. You want to learn new skills. But it seems like a definition of general intelligence should have some form of intention or autonomy. Like would it be learning skills that the software decides on its own, it wants to learn, or it just seems like there should be something more?

Jad: I would separate AGI from autonomous general intelligence. Artificial general intelligence is just the ability to solve problems across the world. So it’s like a neocortex, it has little to do with free will or autonomy. That’s something we can add on top. And I’m happy to discuss that.

Tim: Oh, no, let’s discuss. This is great. When I discuss AGI, I usually think of it in terms of some form of consciousness and self-awareness. Do you think that is a necessary component or is that something that can be mixed in later?

Jad: I will discuss it, but let me just one step back. Just because you don’t have intention and you just respond to the user’s request, doesn’t mean that the user can give you a very high level intention. Like find out the truth of the universe or become as powerful as possible. And then that becomes an inner sub goal. So there’s higher level goals, and if you give me a high enough goal, you can always create sub goals and sub goals. And from the outside it looks like completely autonomous being. So in a sense, what I’m saying is AGI doesn’t have to have intentionality. Intentionality is something you get for free. Do you need self-awareness and consciousness? I think we need to discuss the meaning of the words for self-awareness. If you mean having a model of yourself, you absolutely need to have that even Chat GPT has a model of itself. You can ask who are you? And Chat GPT consciousness. You can divide consciousness into two components. One is what we call an inner theater. And inner theater. You absolutely need to have inner theater just means I can imagine what the world is like. I can create kind of this internal representation of the world that I can play with. I can control, I can do experiments in my mind. And I think that’s necessary. There’s another component of consciousness that goes a little bit beyond that, and that’s usually discussed in the mind body problem or in the context of physics. For that, I think it really depends on how you define consciousness.

Tim: Okay. To drill down on something you said, you were talking about designing an intention and designing this kind of intelligence, but you also mentioned that like you get this intention for free, that it’s kind of an emergent property. And I think like intelligence itself and probably consciousness as well is an emergent property in humans and it quite likely will be in AGI as well. It seems to me it’s more likely that AGI will emerge rather than be designed in.

Jad: I think by AGI here you mean the inner theater.

Tim: The inner theater, yes.

Jad: This will be learned. Yes. I think having a world model, it’s you’re able to simulate universes in your mind. And that is something that is learned by the model. So I would say it’s something that will happen naturally when we have AGI. What I was mentioning about intentionality is a little bit different. So you can ask Chat GPT, hey, you are a doctor, act like a doctor. So now you’re giving it the intention of being a doctor. But Chat GPT is more powerful than any single intention. It lets you give it the intention.

Tim: But that’s still your intention. It’s not chat GPT’s intention of being a doctor.

Jad: Yes. The point is, the way we train the models right now is we give them across intentions so we can get to decide after the training. Hey, I want you to have this intention for now, and one of the things that I’ve been fascinating for years is what is the intention you want to give to an AGI to let it scale to super intelligence? It comes down to giving it the intention of amplifying what we call freedom or a sense of agency, not for itself, but for the entire ecosystem around it. If you define freedom carefully as a state of infinite agency, then you have a very strong story how this can be the right intention for AGI to achieve super intelligence. An intention that not only guides it, but also gives it alignment with human values and makes it beneficial.

Tim: I want to get to alignment, but when we’re talking about an intention, so if we have AGI that emerges, there’s no survival instinct, there’s no desire to reproduce, there’s no response to pleasure and pain. These are the things that pretty much all of human intelligence was evolved to deal with and to some level of abstraction or another. It’s still mostly what we humans are focused on. So would we even be able to recognize an intelligence that’s emergent from artificial intelligence?

Jad: So the way I like to think of AGI, it’s not replicating the entire brain, it’s just replicating our neocortex. And the neocortex, all it’s trying to do is minimize surprise. Now, the limbic system or other parts of your brain and body, they have expectations. You don’t want to be hungry. Being hungry is very surprising. All of these are fundamental drives that we have that we learn across evolutionary history. Neocortex comes in and says, I don’t know what these motivations are. I just don’t want them to be surprised. And then all the other stuff emerges from that. And in that sense, when we build AGI, what I’m building is the neocortex, and then we can add those drives if we want. All these drives can be adapted to human drives. For example, a neocortex that is focused on, you might adapt your drives as its own drives in that sense that AGI would be there as an augmentation of who you are.

Tim: In a sense, without all this emotional and evolutionary baggage, the AI intelligence is very likely to emerge as just a problem solving optimizing machine.

Jad: Exactly.

Tim: That’s pretty interesting. So the alignment problem is always talked about like, okay, how do we keep these AIs from rising up and killing us? But I think it’s simpler than that. I think it’s addressed really well by what you just brought up. So technology progresses incredibly quickly, and so the time period between when robots can do your dishes and when robots start wondering why they have to do your dishes is really small. But if I’m understanding what you’re saying correctly, is that they don’t have the emotional baggage to feel resentment or feel like they’re being exploited and they would just focus on problem solving and optimization.

Jad: Yeah. They can simulate feeling that way though.

Tim: If we put that into them?

Jad: We don’t have to put that into them. That’s the beauty of building products with AGI is that you get to integrate that with human civilization in a way that actually empowers us. But once you want to make them fully autonomous, then we are going to ask ourselves what fundamental intention or aspiration should they seek?

Tim: Well, actually, let me back up just a minute. One thing does occur to me, robots are a little bit different because robots can interact with their outside world. They would have a rudimentary pain and pleasure. They can break. They’re not self-contained beings like software is. So wouldn’t they start to develop some of their own evolutionary baggage, if you will?

Jad: Well, just because something can break doesn’t mean that you can feel pain or pleasure from breaking it.

Tim: I guess you’re right. I’m kind of putting my own biases and evolutionary baggage and projecting that onto the robots, aren’t I?

Jad: It’s natural that we do.

Tim: Okay. So, let’s dig into alignment or ethical AI. So, if AI truly is just this optimization execution, problem solving machine, is alignment even a problem for AI or is a problem really just preventing people from giving AI bad instructions?

Jad: Excellent question. Because this is exactly what I believe. In order to solve the alignment problem, we have to solve an even harder problem, which is aligning humans.

Tim: We’ve been working on that for a few millennia.

Jad: Yes. And so when you look at the history of philosophy, there’s a lot of work on that. And largely the history of the 20th century, especially the first half of the 20th century with world wars, with communism, is history of tragedy of these highest ideals, aspirations. We want equality, we want a good life for all, turn dark, the road to hell is painted with good intentions. We are really entering a thorny ground here. We have to be very careful. But the challenge of aligning AI and aligning humanity forces us to confront these problems. So we need to find a shared vision, something that we can kind of agree on, regardless of our diversity. And what I believe is that it’s a new notion of freedom. And that new notion of freedom is not about the absence of constraints, it’s about agency. And the way you can think of agency is by looking at what an agent can do. An agent can perceive, it can plan and decide things can act. So let’s take that all the way to infinity. So infinite knowledge, the capacity to make good decisions, which means benevolence, infinite power. And then because you’re embodied, you need to be able to maintain yourself. So infinite vitality or safe maintenance. So, this is kind of a loop where each component is reinforcing each other component. And this loop at infinity is what I call freedom. And this seems to be something that we can all agree we should aspire to move towards. And you can argue for that purely from an evolutionary perspective. If you don’t move towards freedom, you die. In fact, you can restate the theory of evolution as not the theory of the survival of the fittest, but the tendency for freer systems to survive. So the more agency you have in the world, the more you can survive. Otherwise…

Tim: Let me push back on that because I don’t think so. I mean, if you look at like the major extinction events throughout history, it’s not the little phytoplankton that gets wiped out. It’s the higher level animals. It’s the complex dinosaurs. It’s the complex animals that have much more freedom and can move around. Those are the ones that tend to get wiped out.

Jad: That’s very true. You need to define freedom at multi-time scales. This is also a problem with the naive formulation of natural selection, right? Because these dinosaur are supposed to be more fit.

Tim: I’ve always been somewhat annoyed by that phrasing of survival of the fittest because it’s a tautology, right? It’s like the survival of the fittest, but how do we define the fittest? It’s well, whatever survives. So it’s really the survival of the survivors.

Jad: Exactly. This is the problem. So can we add something a bit beyond that? And the question is, what does it mean to survive? So the means to survive is to manage the entropy in your environment. Are you able to minimize free energy? So basically the world is messy, it’s throwing stuff at you. Are you able to organize it enough so it doesn’t destroy you? And it’s impossible to do that because you cannot predict the future. The best thing you can do is instead of minimizing free energy directly, which you can’t do, is you can create a model of the world. So it’s an approximation of what you think the world’s free energy is and minimize the difference between your model of the world and the world itself. So you’re trying to improve your self-model of the world, and you can either minimize the difference by improving your model, which is what we call perception, or making the world closer to your model, which is what we call action. But this assumes no intentionality. So the question is, what is the intentionality that is ideal for distributed intelligence? It comes down to you want not only for yourself to have this freedom, but you want the freedom for the entire ecosystem. If you are going to take the freedom and destroy the whole ecosystem, in the process of you acquiring that freedom, you’re kind of shooting yourself in the foot because you’re optimizing at short time skills as opposed to long timescales. So, optimizing that free energy at every timescale forces you to introduce the concept of benevolence in there.

Tim: So, it’s says ecosystem wide optimization. It’s not an optimization for an individual organism within that ecosystem.

Jad: The beauty of it is benevolence means you can optimize for yourself, but because you’re benevolent by definition, you’re optimizing for the whole.

Tim: I don’t think humans live up to that standard. It’d be very hard to develop useful AI that follows those rules, wouldn’t it?

Jad: I think so. I think humans try.

Tim: I think we aspire to it. Some of us do.

Jad: While we’re not perfect, we try to be more benevolent, right? And we definitely try to grow,

Tim: Even if I grant you that humanity aspires to that which I think in our best moments we do. How do we keep one of the less benevolent members of humanity from convincing an AI to do something horrible? How does an AI distinguish between someone telling it to develop a cure for cancer versus a new bio weapon?

Jad: First, we want to agree on freedom as a shared goal. Second, we are going to put this freedom as a high level intention for AGI. So the AGI says, I’m going to support you, but I also want to keep in mind my highest intention, which is freedom for the world. So it’ll try to do both. So they’re going to be agents supporting you, agents supporting me, agents supporting everyone. But they have all in mind that shared goal. So then the question comes in, what if we have something that conflicts from that goal? What if I want to do unhealthy things for myself or for the world? Then comes in the concept of alignment economy. The alignment economy is a way to calculate the price of actions

Tim: Measured in dollars or measured in something else?

Jad: measured in dollars and whatever future currency we decide to have, the price is calculated as diversions from freedom. So if you’re moving alongside freedom, then actually the price should be negative. You should be paid to do that. And if you’re doing something that’s against freedom, that should be expensive.

Tim: I could see AI being able to calculate this. I could see AI being able to price incredibly complex externalities and provide an optimum way forward for human freedom and human happiness. What I can’t see is human beings going along with that. There’s so much of human history, the history of human governance and human hierarchy that just goes in a different direction from that ideal.

Jad: I agree with you that this is a challenge. So the theory makes sense. The challenge is the implementation. You don’t have to have everyone to agree to start even a single group can kind of agree to have these agents follow the freedom concept. And if you have enough critical mass that they can have a local economy, you can agree on these prices internally. So you can start kind of relatively small, an open door policy for, for the rest of the world to come in. That the beauty is that the more people join, the more attractive the whole thing is going to be. That system will outperform competing systems.

Tim: Give me a step by step of how we get to that. I am intrinsically skeptical of utopias.

Jad: Yeah, me too.

Tim: I’m not even going to question the technology because I think we have a clear roadmap to how we can get to technology that can do that. We’ve got the data, or at least a roadmap to the data. We’ve got the algorithms socially. What are the big steps we have to do to roll that out, to make that happen?

Jad: I believe the future is fundamentally open-ended. So there is no way I can tell you what’s going to happen in a year from now. I can’t tell you how we’re going to realize it. I can tell you the principles that which I am trying to apply and I hope to create some movement. One is, I written the book. I’m also creating a website and a lot of resources. I’m going around trying to present these ideas, getting people to talk about them. Ultimately, there’s something paradoxical about freedom. If you want to empower humanity with freedom, you can’t force it on them. They have to choose. Otherwise that’s not freedom, right? So all I can do is plant seeds, have these discussions and debates, and then maybe at some point start creating these concrete instantiations of systems at the local level and eventually persuade at the governmental level and at the international level.

Tim: I think that is an optimistic future to look forward to. So listen, Jad, thanks so much for sitting down with me. I really appreciate it.

Jad: Thank you.

Tim: It’s been fascinating.

Jad: Yeah, I’m happy. It’s good conversation

Outtro

And we are back.

I hope you enjoyed that conversation as much as I did.

In fact, after the mics were off, we continued talking about how hard it is to predict what the drives and motivations of an AGI would be. It’s almost impossible not to project your own motivations into it. I pointed out that how people think a super intelligent AI will think actually tells you a lot about how they think about the world.

For example, when a panicked Elon Musk warned that AGI represents an existential threat to humanity and called for all research on it to be paused, he was most likely just looking at the way he thinks about and treats the people beneath him, imagining a super intelligent AI treating him that way and then freaking out about it.

Jad agreed and added that this problem is endemic in AI research and something he’s been calling out for years.

Researchers will both consciously and subconsciously bring in mental models and motivations from human behavior, animal behavior or game theory that we have absolutely no reason to think will apply to AGI. The problem is that outside of human and animal intelligence, we don’t have much to go on.

Since AGI will not be subject to the survival and reproductive pressures that caused all known forms of intelligence to emerge. There is simply no way of knowing what internal motivations, if any, AGI will have.

It’s hard to even find a starting point without smuggling in our own evolutionary baggage and assumptions.

Jad’s solution is a clean one. Without the evolutionary need for those drives, they simply will not arise unless we decide to put them in. Super intelligent AI will be an autonomous learning and problem solving machine that works.

I mean that’s clearly the most logically consistent conclusion, but I’m not sure it’s the right one.

Human intelligence is an emergent phenomenon. And AGI almost certainly will be as well. I’m not comfortable concluding that self-motivation and self-direct and intention will not emerge spontaneously alongside intelligence. Perhaps because both the human created training data and the tasks we give AI are all intractably linked to human motivations or perhaps by the nature of self-awareness itself.

In fact, last month, Open AI and Apollo research published papers pointing out that open AI’s O1 model was beginning to show a kind of survival instinct. In about 5% of the cases when it was made aware that it might be shut down, it disabled its oversight mechanisms and then lied about doing so.

Now whether these motivations emerged from the prompts, the training data or some intrinsic aspect of intelligence is an interesting philosophical question. But the fact is the seeds of self-motivation and self-preservation are already emerging.

I wish I could tell you how I think all this will play out and make some reasonable predictions about what super intelligent AI will be like. But I just don’t know. Of course, most of the best minds in the field aren’t too sure either.

It’s been about 40,000 years since the Neanderthals died out and that was the last time we’ve had to share the planet with another species of similar intelligence. Of course, the Neanderthals shared our evolutionary history and drives. We were quite similar. AGI will be very different.

Hopefully we’ll be better at coexisting this time.

 

 

If you want to talk about the future of AI and come on, I know you do Jad, and I would love to talk with you. So come by disruptingJapan/show228 and let’s talk about it. And hey, if you enjoy disrupting Japan, share a link online or just, you know, tell people about it. Disrupting Japan is free forever and letting people know about is the absolute best way you can support the podcast.

But most of all, thanks for listening and thank you for letting people interested in Japanese startups and VCs know about the show.

I’m Tim Romero and thanks for listening to Disrupting Japan.