Over the next decade, ‘foundation models’ will be game changers, transforming strategy and business planning and powering novel offerings.
WHEN OPENAI REVEALED CHATGPT in late 2022, people clambered to test it. They asked complicated, open-ended questions, re-quested long-form narrative, and often got impressive results. Not even four months later, the company released GPT-4, the next generation of its AI software, which can respond to both image and text inputs and more nuanced instructions.
Even before ChatGPT came along, text-to-image generators like Stability AI’s Stable Diffusion and OpenAI’s DALL-E 2 stunned people by responding to written prompts with photorealistic images. ‘Two kangaroos waltzing in the style of Monet’ would get you pretty much that. Not surprisingly, arguments over the ethics of mimicking artists’ styles, legal risks and the impact on people’s livelihoods have flared.
While the impact on the art industry is clear, many leaders in other fields continue to view these tools as a mere novelty. That is a mistake, and in this article I will explain why.
A Step Change in AI
It all began in 2017 with a landmark innovation in AI model architecture by Google researchers who developed transformer architecture. Since then, tech companies and researchers have been supersizing AI, increasing the size of models by 10,000 times and the size of training sets, too. The result: powerful, pretrained models — called ‘foundation models’ — that offer unprecedented adaptability within the domains they are trained on, be it language, images or the structure of proteins.
With this adaptability, foundation models can complete a wide variety of tasks without needing task-specific training. What’s more, companies building foundation models are giving third parties access through application programming interfaces (APIs) or by open sourcing them, putting these advanced models in anyone’s hands. This is no everyday technology advancement.
While foundation models are not the only area of AI research that is growing, the magnitude of their potential and the speed at which they can be deployed is driving them to the top of companies’ innovation agendas. Indeed, 98 per cent of global executives we surveyed agree that AI foundation models will play an important role in their organizations’ strategies in the next three to five years.
One company taking advantage of this is CarMax, which is using ChatGPT to improve the car-buying experience. Know ing that there is a massive amount of information potential car buyers may want to read through before making a purchase decision, CarMax used Microsoft’s Azure OpenAI Service to ac-
cess a pre-trained GPT-3 model to quickly read and synthesize over 100,000 customer reviews for every vehicle make, model and year that they sell. From these reviews, the model generated 5,000 easy-to-read summaries — a task the company says would have taken its editorial team 11 years to complete.
The novel capabilities of foundation models have led some to see them as a step towards artificial general intelligence (AGI).
Other organizations are also experimenting with foundation models, adapting them for tasks ranging from powering customer service bots to generative product design to automated coding. As these models broaden and extend what we can do with AI, they are letting companies transform human-AI inter-
action and build an entirely new generation of AI applications and services. To be a part of this and to leverage these models to drive novel business solutions and offerings, companies need to understand their strengths and capabilities and track how they are advancing — starting today.
There are two key innovations making this particular wave of AI possible. The first is the aformentioned transformer models introduced by Google researchers. One of the newest classes of AI models, transformers are neural networks that identify and track relationships in sequential data (like the words in a sentence) to learn how they depend on and influence each other. They are typically trained via self-supervised learning, which for a large language model (LLM) could mean pouring through billions of blocks of text, hiding words from itself, guessing what they are based on surrounding context and repeating until
it can predict those words with high accuracy. This technique works well for other types of sequential data, too. Some multimodal text-to-image generators work by predicting clusters of pixels based on their surroundings.
The second innovation is scale — significantly increasing the size of models and, subsequently, the amount of ‘compute’ [computational power] used to train them. The size of a model is measured in parameters, which are the values or weights in a neural network that are trained to respond to various inputs or tasks in certain ways. Generally speaking, ‘more parameters’ lets a model soak up more information from its training data and make more accurate predictions later. But what OpenAI demonstrated with GPT-3 is that vastly increasing the number of parameters in a transformer model, and the computational power put into training it, leads not only to higher accuracy but also to the ability to learn tasks the model was never trained on.
This novel learning ability — also known as ‘few-shot’ and ‘zero-shot’ learning — means that foundation models can successfully complete new tasks given only a few or no task-specific training examples. DeepMind’s Flamingo — a multimodal visual-language model — is especially good at this. In a 2022 paper, DeepMind researchers demonstrated how Flamingo can conduct few-shot learning on a wide range of vision and language tasks, only being prompted by a few input/output examples and without the researchers needing to change or adapt the model’s weights. In six of 16 tasks they tested, Flamingo surpassed state-of-the-art models that had been trained on much more task-specific data, despite not having any retraining itself.
One of the most significant ways foundation models are evolving has to do with the data types they’re trained on — which, right now, are limited. Most of today’s foundation models are LLMs trained on natural language, and even multimodal models are typically language- and image-only. But some are working to expand to more data modalities. This can mean building standalone foundation models for new kinds of data.
Meta, for instance, developed a protein-folding model — an LLM that learned the ‘language of protein’ — accelerating protein structure predictions by up to 60 times. And a research team from the University of Texas at Austin, the Indian Institute of Technology and Google Research proposed Generalizable NeRF Transformer (GNT), a transformer-based architecture for NeRF reconstruction. A NeRF (Neural Radiance Field) is a neural network that can generate 3D scenes based on only partial 2D views — and experimenting with transformers to generate 3D data like this could have big metaverse implications.
Other organizations are working to incorporate more data types into a single model. Take Microsoft’s Florence, a foundation model built for general purpose computer vision tasks. While it was trained on a large data set of image-text pairs and has only a two-tower architecture, combining one language
encoder and one image encoder, its creators proposed a video adapter built off the image encoder. Extending to this additional data type is a key step towards a computer vision foundation model that could generalize across real-world vision tasks — and could drive applications in security, healthcare and more.
The amount of compute needed to train the largest AI models has grown exponentially — now doubling anywhere from every 10 months to every 3.4 months, according to various reports. And even after a model is trained, it’s expensive to run and host all of its downstream variations as it gets fine-tuned to handle different tasks. In today’s cloud computing setups, it’s slow to load foundation models each time they’re needed but expensive to keep many models online.
Anyscale — a unicorn that recently raised US$199 million — is working to lower these barriers. Anyscale was founded by a group of UC Berkeley researchers who developed Ray, an open-source framework that improves access to foundation models by making it easier to scale and distribute machine learning workloads. It is currently used to train the largest AI models coming out of OpenAI, like ChatGPT.
Elsewhere, Cohere, a start-up building an NLP developer toolkit, also uses Ray to train large language models. And IBM is using it to implement ‘zero-copy model loading,’ whereby they store model weights in shared memory and use Ray to instantly load and redirect cluster resources to whatever model an application requires in the moment. This frees users from needing to tune the number of model variations they keep loaded in memory and is expected to lead to much simpler foundation model adaptation and deployment.
The novel capabilities of foundation models have led some in the community to see them as a step towards artificial general intelligence (AGI) — an AI system capable of learning any intellectual task that a human can learn. Only time will tell if the technologies and methods behind foundation models are enough to achieve some form of truly general intelligence in the future. Nevertheless, the level of generalization foundation models have already achieved within certain data types is hugely significant and more than enough to revolutionize how and where enterprises use AI.
The question now for leaders shouldn’t be whether or not these models will impact their industry, but how. Foundation models are widely adaptable and could technically be used for a wide variety of tasks — so the decisions companies make around how to deploy them and what problems to address with them are where competitive differentiation will be found.
The Benefits of Foundation Models
Using foundation models for the right purposes starts with understanding what they truly change. This goes beyond technical capabilities — it’s about what these models let businesses do that they couldn’t do before. There are two major benefits here.
THE POTENTIAL TO DEEPLY TRANSFORM HUMAN-AI INTERACTION. Look at how some are calling ChatGPT the future of search and knowledge retrieval. It can write poems and essays, debug code and answer complicated questions because it’s trained on billions of text examples pulled from the internet. And it remembers previous conversations, so it can revise or elaborate on responses, making human-machine communication more sophisticated and natural.
This is important. Because many foundation models are (or contain) LLMs, they use natural language as their interface. That is a big part of why foundation models are giving rise to a new generation of AI applications: People can easily engage with them. Frame, for instance, is using an LLM capable of generating code to help teachers design 3D metaverse classrooms simply by describing out loud what they want in the room. And they’re not the only ones thinking along these lines. Jensen Huang, CEO of Nvidia, has said he expects LLMs to be a core technology for generating 3D images and shapes to populate the metaverse.
Another way foundation models are changing human-AI interaction is by transforming how work is done. At Accenture, for instance, we are leveraging generative AI across a number of functions. Currently, we are testing the use of OpenAI LLMs to improve developer efficiency by automatically generating documentation. This will allow coders to submit requests conveniently through a Microsoft Teams chat while they work. In return, accurately prepared documents are swiftly delivered, demonstrating how specific tasks, rather than entire jobs, can be enhanced and automated.
Foundational AI is also changing the way software engineers work at Google. They used a foundation model to develop a code completion tool that over 10,000 engineers tested for a three-month period. The results showed that coding iteration time was reduced by six per cent. The potential of these models to transform workflows and improve productivity, even in highly complex tasks, is undeniable. And soon, companies may start to use them in much more varied ways, augmenting tasks all across product development, business processes and more.
The question for leaders shouldn’t be whether or not these models will impact their industry, but how.
OPENING THE DOOR TO NEW AI APPLICATIONS AND SERVICES. Foundation models require massive amounts of data upfront, which is handled by their creators. But once a model is trained, organizations can adapt it to a range of downstream tasks, building new capabilities with just a few examples or fine-tuning a model with just a small training set. Rather than every new AI application requiring months of effort and investment, organizations will be
able to create and deploy them much more simply.
IBM, for instance, has been transitioning some of its Watson portfolio to use foundation models. It found that with pre-trained language models, Watson NLP could train sentiment analysis on a new language with only a few thousand sentences — a training set a hundred times smaller than what previous models required. Over about a year, the company was able to expand Watson NLP from 12 languages to 25.
Multimodal foundation models’ ability to recognize multiple data types and identify the relationships between them is also pushing the envelope of what AI is capable of, enabling powerful new systems. GPT-4, for example, is multimodal and accepts both image and text inputs, meaning that if someone were to show it a picture of the inside of their refrigerator, it could correctly identify the items inside, suggest meals that can be made with those ingredients and then provide step-by-step cooking instructions.
Meta has long seen the value of an AI system that can interpret content on its platform — especially when it comes to detecting hate speech. But this is a task that has historically been difficult for machines because people tend to communicate in multimodal ways on these platforms — using text and image together to tell a joke, for instance. Meta has launched a series of foundation and multimodal AI projects to help it analyze different types of communication — like text, image and video — simultaneously. The company created the Hateful Memes dataset to address the shortage of publicly available training data for classifying memes; it developed FLAVA, a multimodal foundation model that works across dozens of tasks; and it built Omnivore, a model that can operate across images, video and 3D data, doing things like detecting content in both videos and images.
It’s exciting to imagine the possibilities we’ll have in the future. What will we be able to do when multimodal models connect text, sound, image, video, 3D spatial data, sensor data from industrial equipment, environmental data and many other types of data? Early opportunities may start with generating marketing images and ad copy but could grow into sophisticated autogenerated code and new ways to search and access information. Analysts might use language to ask an AI system to describe patterns across thousands of satellite images. A piece of industrial equipment might use an AI system to translate data from dozens of sensors into a repair procedure for a mechanic. Or multimodal AI might help drastically improve the path planning and performance of robotic arms.
Leaders are now presented with huge opportunities and responsibilities fundamentally changing the roles of the C-suite. Tapping into the power of the underlying AI technology calls for a profound rethink of how the organization works, with multiple implications for enterprise IT architecture, organization, culture and more. One thing is certain: AI will become an ever-present co-pilot for every worker, boosting productivity by putting new kinds of hyper-personalized intelligence into human hands.
From Building AI to Building With AI
Foundation models are shifting conversations from ‘How can we build our own AI?’ to ‘What can we build with AI?’ The start-up Hugging Face is growing in popularity as a hub for foundation, transformer and other machine learning models. The community platform offers many pretrained models, allowing developers to work with new AI models without needing to collect data or train them themselves. The platform is popular with indie developers, but major companies — including Intel, eBay, Pfizer and Bloomberg — have started to use it as well. Hugging Face closed a $100 million funding round at a $2 billion valuation in May 2022 to help it become the premier location for building with pretrained AI models.
A ‘middle layer’ is also starting to appear alongside foundation model platforms, offering services like fine-tuning the models for more custom use. Though foundation models are pretrained and highly adaptable, some downstream tasks may require more specific fine-tuning to train it on new, task-specific data to change its output. Microsoft, for instance, is packaging and selling GPT-3 capabilities to customers, integrating it into its low-code app development platform Power Apps, and offering customers access to the model along with the enterprise-grade security, compliance, reliability and data privacy assurance of Azure.
To identify the right role for foundation models in your organization, it’s necessary to understand their best use cases. There are many AI applications that work with data types no foundation model can handle yet. Additionally, some use cases that a foundation model could feasibly attempt are still fundamentally better served by narrow AI, which is trained specifically for a task rather than across a modality.
What’s more, foundation models have some characteristics that make them questionable for certain situations, and this is adding an entirely new dimension to enterprises’ security efforts. For instance, bias in foundation models is a common concern due to homogenization as well as the fact that many are trained on large datasets from the Internet. Historic datasets that excluded certain populations, people and demographics can lead to undesirable outcomes. And some companies’ early efforts even found misinformation was affecting the outputs of these algorithms. When the same few models are used as the basis for many downstream applications, then any issues in the original models may propagate throughout the rest.
There have been efforts from the makers of foundation models to correct for some of these biases. GPT-3, for instance, was given an extra round of training on a more curated dataset after religious bias was discovered. But even so, businesses should carefully consider these risks when determining not only if they can use a foundation model, but whether they should. Just as parents hope to raise their children to be responsible and fair, AI systems need to be ‘raised’ with a diverse and inclusive set of inputs so that they reflect the broader business and societal norms of fairness and transparency.
As businesses narrow down where to use foundation models, they next need to decide how they will access them and where they will sit in the foundation model ecosystem. The base layer of this ecosystem consists of organizations with the expertise, access to data and computing infrastructure to train new foundation models — but not everyone needs to function at this level. For many, decisions will be around how directly they work with pre-trained models. A second layer of the ecosystem will be companies that access foundation models via APIs or model repositories and build on them to create their own applications and services. For simple applications, developers will be able to leverage the transfer learning of foundation models by simply building the user interface and interconnects around the model, while in more advanced cases, developers might leverage fine-tuning to hone the foundation model to a specific application.
This approach creates the opportunity for wide experimentation throughout the organization, as well as the chance to develop B2B foundation model offerings. But again, not everyone will want to do this: Even with natural language offering an easy-to-use interface, some software engineering knowledge is needed to successfully build applications around foundation models. Companies without those skillsets can still benefit from the technology though, and they make up the third layer of this evolving ecosystem.
In Closing
Foundation models are still in their infancy, but over the next decade their impact will grow to be overwhelming. They could drive new data practices, transforming the nature of knowledge sharing in the organization. They could be game-changing for digital strategies, writing sophisticated code and powering novel offerings. They could transform strategy and business planning, forecasting headcount needs or shifts in customer demand with greater ease and accuracy than we can currently imagine. And for organizations that build their own foundation models, they could bring industry-shifting advantages — new tools for scientific discovery, new methods for engineering or new industry-specific AI agents.
What if your organization builds the model that changes everything? 
Learn about your GP Leadership profile
Where do you rank in your Power, Challenge, Aspirational and Foundational traits?
