Self-supervised learning is the future of AI – The Next Web
Posted: April 11, 2020 at 12:49 am
Despite the huge contributions of deep learning to the field of artificial intelligence, theres something very wrong with it: It requires huge amounts of data. This is one thing that boththe pioneersandcritics of deep learningagree on. In fact, deep learning didnt emerge as the leading AI technique until a few years ago because of the limited availability of useful data and the shortage of computing power to process that data.
Reducing the data-dependency of deep learning is currently among the top priorities of AI researchers.
In hiskeynote speech at the AAAI conference, computer scientist Yann LeCun discussed the limits of current deep learning techniques and presented the blueprint for self-supervised learning, his roadmap to solve deep learnings data problem. LeCun is one of thegodfathers of deep learningand the inventor ofconvolutional neural networks (CNN), one of the key elements that have spurred a revolution in artificial intelligence in the past decade.
Self-supervised learning is one of several plans to create data-efficient artificial intelligence systems. At this point, its really hard to predict which technique will succeed in creating the next AI revolution (or if well end up adopting a totally different strategy). But heres what we know about LeCuns masterplan.
First, LeCun clarified that what is often referred to as the limitations of deep learning is, in fact, a limit ofsupervised learning. Supervised learning is the category of machine learning algorithms that require annotated training data. For instance, if you want to create an image classification model, you must train it on a vast number of images that have been labeled with their proper class.
[Deep learning] is not supervised learning. Its not justneural networks. Its basically the idea of building a system by assembling parameterized modules into a computation graph, LeCun said in his AAAI speech. You dont directly program the system. You define the architecture and you adjust those parameters. There can be billions.
Deep learning can be applied to different learning paradigms, LeCun added, including supervised learning,reinforcement learning, as well as unsupervised or self-supervised learning.
But the confusion surrounding deep learning and supervised learning is not without reason. For the moment, the majority of deep learning algorithms that have found their way into practical applications are based on supervised learning models, which says a lot aboutthe current shortcomings of AI systems. Image classifiers, facial recognition systems, speech recognition systems, and many of the other AI applications we use every day have been trained on millions of labeled examples.
Reinforcement learning and unsupervised learning, the other categories of learning algorithms, have so far found very limited applications.
Supervised deep learning has given us plenty of very useful applications, especially in fields such ascomputer visionand some areas of natural language processing. Deep learning is playing an increasingly important role in sensitive applications, such as cancer detection. It is also proving to be extremely useful in areas where the scale of the problem is beyond being addressed with human efforts, such aswith some caveatsreviewing the huge amount of content being posted on social media every day.
If you take deep learning from Facebook, Instagram, YouTube, etc., those companies crumble, LeCun says. They are completely built around it.
But as mentioned, supervised learning is only applicable where theres enough quality data and the data can capture the entirety of possible scenarios. As soon as trained deep learning models face novel examples that differ from their training examples, they start to behave in unpredictable ways. In some cases,showing an object from a slightly different anglemight be enough to confound a neural network into mistaking it with something else.
ImageNet vs reality: In ImageNet (left column) objects are neatly positioned, in ideal background and lighting conditions. In the real world, things are messier (source: objectnet.dev)
Deep reinforcement learning has shownremarkable results in games and simulation. In the past few years, reinforcement learning has conquered many games that were previously thought to off-limits for artificial intelligence. AI programs have already decimated human world champions atStarCraft 2, Dota, and the ancient Chinese board game Go.
But the way these AI programs learn to solve problems is drastically different from that of humans. Basically, a reinforcement learning agent starts with a blank slate and is only provided with a basic set of actions it can perform in its environment. The AI is then left on its own to learn through trial-and-error how to generate the most rewards (e.g., win more games).
This model works when the problem space is simple and you have enough compute power to run as many trial-and-error sessions as possible. In most cases, reinforcement learning agents take an insane amount of sessions to master games. The huge costs have limited reinforcement learning research to research labsowned or funded by wealthy tech companies.
Reinforcement learning agents must be trained on hundreds of years worth of session to master games, much more than humans can play in a lifetime (source: Yann LeCun).
Reinforcement learning systems are very bad attransfer learning. A bot that plays StarCraft 2 at grandmaster level needs to be trained from scratch if it wants to play Warcraft 3. In fact, even small changes to the StarCraft game environment can immensely degrade the performance of the AI. In contrast, humans are very good at extracting abstract concepts from one game and transferring it to another game.
Reinforcement learning really shows its limits when it wants to learn to solve real-world problems that cant be simulated accurately. What if you want to train a car to drive itself? And its very hard to simulate this accurately, LeCun said, adding that if we wanted to do it in real life, we would have to destroy many cars. And unlike simulated environments, real life doesnt allow you to run experiments in fast forward, and parallel experiments, when possible, would result in even greater costs.
LeCun breaks down the challenges of deep learning into three areas.
First, we need to develop AI systems that learn with fewer samples or fewer trials. My suggestion is to use unsupervised learning, or I prefer to call it self-supervised learning because the algorithms we use are really akin to supervised learning, which is basically learning to fill in the blanks, LeCun says. Basically, its the idea of learning to represent the world before learning a task. This is what babies and animals do. We run about the world, we learn how it works before we learn any task. Once we have good representations of the world, learning a task requires few trials and few samples.
Babies develop concepts of gravity, dimensions, and object persistence in the first few months after their birth. While theres debate on how much of these capabilities are hardwired into the brain and how much of it is learned, what is for sure is that we develop many of our abilities simply by observing the world around us.
The second challenge is creating deep learning systems that can reason. Current deep learning systems are notoriously bad at reasoning and abstraction, which is why they need huge amounts of data to learn simple tasks.
The question is, how do we go beyond feed-forward computation and system 1? How do we make reasoning compatible with gradient-based learning? How do we make reasoning differentiable? Thats the bottom line, LeCun said.
System 1 is the kind of learning tasks that dont require active thinking, such as navigating a known area or making small calculations. System 2 is the more active kind of thinking, which requires reasoning.Symbolic artificial intelligence, the classic approach to AI, has proven to be much better at reasoning and abstraction.
But LeCun doesnt suggest returning to symbolic AI or tohybrid artificial intelligence systems, as other scientists have suggested. His vision for the future of AI is much more in line with that of Yoshua Bengio, another deep learning pioneer, who introduced the concept ofsystem 2 deep learningat NeurIPS 2019 and further discussed it at AAAI 2020. LeCun, however, did admit that nobody has a completely good answer to which approach will enable deep learning systems to reason.
The third challenge is to create deep learning systems that can lean and plan complex action sequences, and decompose tasks into subtasks. Deep learning systems are good at providing end-to-end solutions to problems but very bad at breaking them down into specific interpretable and modifiable steps. There have been advances in creatinglearning-based AI systems that can decompose images, speech, and text. Capsule networks, invented by Geoffry Hinton, address some of these challenges.
But learning to reason about complex tasks is beyond todays AI. We have no idea how to do this, LeCun admits.
The idea behind self-supervised learning is to develop a deep learning system that can learn to fill in the blanks.
You show a system a piece of input, a text, a video, even an image, you suppress a piece of it, mask it, and you train a neural net or your favorite class or model to predict the piece thats missing. It could be the future of a video or the words missing in a text, LeCun says.
The closest we have to self-supervised learning systems are Transformers, an architecture that has proven very successful innatural language processing. Transformers dont require labeled data. They are trained on large corpora of unstructured text such as Wikipedia articles. And theyve proven to be much better than their predecessors at generating text, engaging in conversation, and answering questions. (But they are stillvery far from really understanding human language.)
Transformers have become very popular and are the underlying technology for nearly all state-of-the-art language models, including Googles BERT, Facebooks RoBERTa,OpenAIs GPT2, and GooglesMeena chatbot.
More recently, AI researchers have proven thattransformers can perform integration and solve differential equations, problems that require symbol manipulation. This might be a hint that the evolution of transformers might enable neural networks to move beyond pattern recognition and statistical approximation tasks.
So far, transformers have proven their worth in dealing with discreet data such as words and mathematical symbols. Its easy to train a system like this because there is some uncertainty about which word could be missing but we can represent this uncertainty with a giant vector of probabilities over the entire dictionary, and so its not a problem, LeCun says.
But the success of Transformers has not transferred to the domain of visual data. It turns out to be much more difficult to represent uncertainty and prediction in images and video than it is in text because its not discrete. We can produce distributions over all the words in the dictionary. We dont know how to represent distributions over all possible video frames, LeCun says.
For each video segment, there are countless possible futures. This makes it very hard for an AI system to predict a single outcome, say the next few frames in a video. The neural network ends up calculating the average of possible outcomes, which results in blurry output.
This is the main technical problem we have to solve if we want to apply self-supervised learning to a wide variety of modalities like video, LeCun says.
LeCuns favored method to approach supervised learning is what he calls latent variable energy-based models. The key idea is to introduce a latent variable Z which computes the compatibility between a variable X (the current frame in a video) and a prediction Y (the future of the video) and selects the outcome with the best compatibility score. In his speech, LeCun further elaborates on energy-based models and other approaches to self-supervised learning.
Energy-based models use a latent variable Z to compute the compatibility between a variable X and a prediction Y and select the outcome with the best compatibility score (image credit: Yann LeCun).
I think self-supervised learning is the future. This is whats going to allow to our AI systems, deep learning system to go to the next level, perhaps learn enough background knowledge about the world by observation, so that some sort of common sense may emerge, LeCun said in his speech at the AAAI Conference.
One of the key benefits of self-supervised learning is the immense gain in the amount of information outputted by the AI. In reinforcement learning, training the AI system is performed at scalar level; the model receives a single numerical value as reward or punishment for its actions. In supervised learning, the AI system predicts a category or a numerical value for each input.
In self-supervised learning, the output improves to a whole image or set of images. Its a lot more information. To learn the same amount of knowledge about the world, you will require fewer samples, LeCun says.
We must still figure out how the uncertainty problem works, but when the solution emerges, we will have unlocked a key component of the future of AI.
If artificial intelligence is a cake, self-supervised learning is the bulk of the cake, LeCun says. The next revolution in AI will not be supervised, nor purely reinforced.
This story is republished fromTechTalks, the blog that explores how technology is solving problems and creating new ones. Like them onFacebookhere and follow them down here:
Published April 5, 2020 05:00 UTC
Go here to see the original:
Self-supervised learning is the future of AI - The Next Web
- Mission Healthcare of San Diego Adopts Muse Healthcare's Machine Learning Tool - Southernminn.com - January 19th, 2021
- Deep Learning Outperforms Standard Machine Learning in Biomedical Research Applications, Research Shows - Georgia State University News - January 19th, 2021
- Project MEDAL to apply machine learning to aero innovation - The Engineer - January 19th, 2021
- Forecast On Machine Learning (ML) Intelligent Process Automation Market Witness the Growth of Great Billion by 2027 With Top Companies Like Automation... - January 19th, 2021
- Machine Learning Shown to Identify Patient Response to Sarilumab in Rheumatoid Arthritis - AJMC.com Managed Markets Network - January 19th, 2021
- Bangalore based Great Learning can help you unleash the potential of an M-Tech in Data Science & Machine - Times of India - January 19th, 2021
- CERC plans to embrace AI, machine learning to improve functioning - Business Standard - January 19th, 2021
- NTT Co-authored Papers at NeurIPS to Advance Machine Learning Efficiency and Performance - Business Wire - December 7th, 2020
- Why Intel believes confidential computing will boost AI and machine learning - VentureBeat - December 3rd, 2020
- Machine Learning Market to Grow Notably Attributed to Increasing Adoption of Analytics-driven Solutions by Developing Economies, says Fortune Business... - December 3rd, 2020
- Machine learning: The new language of data and analytics - ITProPortal - December 3rd, 2020
- Injecting Machine Learning And Bayesian Optimization Into HPC - The Next Platform - December 3rd, 2020
- QA Increasingly Benefits from AI and Machine Learning - RTInsights - December 3rd, 2020
- Everything to Know About Machine Learning as a Service (MLaaS) - Analytics Insight - December 3rd, 2020
- How the Food and Beverage Industry is Affected by Machine Learning and AI - IoT For All - December 3rd, 2020
- Amazon announces new machine learning tools to help customers monitor machines and worker safety - www.computing.co.uk - December 3rd, 2020
- Machine Learning and Location Data Applications Market 2020 Top Companies report covers, Industry Outlook, Top Countries Analysis & Top... - December 3rd, 2020
- Commentary: Chain of Demand applies AI, machine learning to retail supply chain profitability - FreightWaves - December 3rd, 2020
- Machine learning - it's all about the data - KHL Group - December 3rd, 2020
- Product Portfolio Analysis and Technological Development of Machine Learning in Medical Imaging Market during the forecasted period - Murphy's Hockey... - December 3rd, 2020
- Imaging AI and Machine Learning Beyond the Hype, Upcoming Webinar Hosted by Xtalks - PR Web - December 3rd, 2020
- Veritone aiWARE Now Supports NVIDIA CUDA for GPU-based AI and Machine Learning - Business Wire - December 3rd, 2020
- Exactech Launches Predict+, First Machine Learning-Based Software that Informs Surgeons with Patient-Specific Outcomes Predictions After Shoulder... - December 3rd, 2020
- How To Choose The Best Machine Learning Algorithm For A Particular Problem? - Analytics India Magazine - October 19th, 2020
- Lantronix Brings Advanced AI and Machine Learning to Smart Cameras With New Open-Q 610 SOM Based on the Powerful Qualcomm QCS610 System on Chip (SOC)... - October 19th, 2020
- AI and Machine Learning Technologies Expected to Play a Key Role in Expanding Multi Billion Dollar Digital Banking Sector: Report - Crowdfund Insider - October 19th, 2020
- AutoML Alleviates the Process of Machine Learning Analysis - Analytics Insight - October 19th, 2020
- Futurism Reinforces Its Next-Gen Business Commerce Platform With Advanced Machine Learning and Artificial Intelligence Capabilities - Yahoo Finance - October 19th, 2020
- Purebase Enhances Its Board of Advisors with An Expert on Machine Learning and Cheminformatics - GlobeNewswire - October 19th, 2020
- COVID-19 And The Role Of AI, Machine Learning In Logistics: A Conversation With Delhivery CTO Kapil Bharati - Mashable India - October 19th, 2020
- How to Beat Analysts and the Stock Market with Machine Learning - Knowledge@Wharton - October 19th, 2020
- AI and Machine Learning Can Help Fintechs if We Focus on Practical Implementation and Move Away from Overhyped Narratives, Researcher Says - Crowdfund... - October 19th, 2020
- Proximity matters: Using machine learning and geospatial analytics to reduce COVID-19 exposure risk - Healthcare IT News - September 20th, 2020
- PREDICTING THE OPTIMUM PATH - Port Strategy - September 20th, 2020
- What is 'custom machine learning' and why is it important for programmatic optimisation? - The Drum - September 20th, 2020
- How Machine Learning is Set to Transform the Online Gaming Community - Techiexpert.com - TechiExpert.com - September 20th, 2020
- Current and future regulatory landscape for AI and machine learning in the investment management sector - Lexology - September 20th, 2020
- Global Machine Learning Courses Market Research Report 2015-2027 of Major Types, Applications and Competitive Vendors in Top Regions and Countries -... - September 20th, 2020
- When AI in healthcare goes wrong, who is responsible? - Quartz - September 20th, 2020
- Is Wide-Spread Use of AI & Machine Intelligence in Manufacturing Still Years Away? - Automation World - September 20th, 2020
- How do we know AI is ready to be in the wild? Maybe a critic is needed - ZDNet - September 20th, 2020
- Solving the crux behind Apple's Silicon Strategy - Medium - September 20th, 2020
- Boost Your Animation To 60 FPS Using AI - Hackaday - September 20th, 2020
- 50 Latest Data Science And Analytics Jobs That Opened Last Week - Analytics India Magazine - September 20th, 2020
- Algorithms may never really figure us out thank goodness - The Boston Globe - September 20th, 2020
- Why Deep Learning DevCon Comes At The Right Time - Analytics India Magazine - September 20th, 2020
- Six notable benefits of AI in finance, and what they mean for humans - Daily Maverick - September 20th, 2020
- Twitter is looking into why its photo preview appears to favor white faces over Black faces - The Verge - September 20th, 2020
- 8 Trending skills you need to be a good Python Developer - iLounge - September 20th, 2020
- Automation Continuum - Leveraging AI and ML to Optimise RPA - Analytics Insight - September 20th, 2020
- UT Austin Selected as Home of National AI Institute Focused on Machine Learning - UT News | The University of Texas at Austin - August 27th, 2020
- Participation-washing could be the next dangerous fad in machine learning - MIT Technology Review - August 27th, 2020
- Getting to the heart of machine learning and complex humans - The Irish Times - August 27th, 2020
- Air Force Taps Machine Learning to Speed Up Flight Certifications - Nextgov - August 27th, 2020
- The Role of Artificial Intelligence and Machine Learning in the... - Insurance CIO Outlook - August 27th, 2020
- AI and Machine Learning Network Fetch.ai Partners Open-Source Blockchain Protocol Waves to Conduct R&D on DLT - Crowdfund Insider - August 27th, 2020
- AI may not predict the next pandemic, but big data and machine learning can fight this one - ZDNet - August 27th, 2020
- Machine Learning Artificial intelligence Market Size and Growth By Leading Vendors, By Types and Application, By End Users and Forecast to 2020-2027 -... - August 27th, 2020
- Explainable AI: From the peak of inflated expectations to the pitfalls of interpreting machine learning models - ZDNet - August 27th, 2020
- What is AutoML and Why Should Your Business Consider It - BizTech Magazine - August 27th, 2020
- Machine Learning Courses and Market 2020: Growing Tends in Global Regions with COVID-19 Pandemic Analysis, Growth Size, Share, Types, Applications,... - August 27th, 2020
- Focusing on ethical AI in business and government - FierceElectronics - August 27th, 2020
- CUHK Business School Research Looks at the Limitations of Using Artificial Intelligence to Pick Stocks - Taiwan News - August 27th, 2020
- Effects of the Alice Preemption Test on Machine Learning Algorithms - IPWatchdog.com - June 20th, 2020
- Googles latest experiment is Keen, an automated, machine-learning based version of Pinterest - TechCrunch - June 20th, 2020
- Deploying Machine Learning Has Never Been This Easy - Analytics India Magazine - June 20th, 2020
- This startup could be a dog owners best friend as it uses machine learning to help guide key decisions - GeekWire - June 20th, 2020
- How machine learning could reduce police incidents of excessive force - MyNorthwest.com - June 20th, 2020
- Adversarial attacks against machine learning systems everything you need to know - The Daily Swig - June 20th, 2020
- Trending News Machine Learning in Finance Market Key Drivers, Key Countries, Regional Landscape and Share Analysis by 2025|Ignite Ltd,Yodlee,Trill... - June 20th, 2020
- The startup making deep learning possible without specialized hardware - MIT Technology Review - June 20th, 2020
- Predicting and elucidating the etiology of fatty liver disease: A machine learning modeling and validation study in the IMI DIRECT cohorts. - DocWire... - June 20th, 2020
- This Startup Is Trying to Foster an AI Art Scene in Korea - Adweek - June 20th, 2020
- Scientists use AI and drone images to interpret crop health Earth.com - Earth.com - June 20th, 2020
- Machine learning - Wikipedia - June 2nd, 2020
- Announcing DataGroomr, the App that Utilizes Machine Learning to Find Duplicates in Salesforce Automatically - PRNewswire - June 2nd, 2020
- Expert Reaction On Forecast That Machine Learning Will Seriously Change The Automotive Industry And Its Security - ISBuzz News - June 2nd, 2020
- AI to machine learning: RILs $2-billion bet to be a tech tornado - Business Standard - June 2nd, 2020
- Barclaycard Payments Partners With Kount to Deliver Industry-leading Fraud Prevention and Prepare Businesses for SCA - AiThority - June 2nd, 2020
- Machine Learning Market Projected to Register 43.5% CAGR to 2030 Intel, H2Oai - Cole of Duty - June 2nd, 2020