Google’s AI Wizard Unveils a New Twist on Neural Networks

November 18, 2017

If you want to blame someone for the hoopla around artificial intelligence, 69-year-old Google researcher Geoff Hinton is a good candidate.

The droll University of Toronto professor jolted the field onto a new trajectory in October 2012. With two grad students, Hinton showed that an unfashionable technology he’d championed for decades called artificial neural networks permitted a huge leap in machines’ ability to understand images. Within six months, all three researchers were on Google’s payroll. Today neural networks transcribe our speech, recognize our pets, and fight our trolls.

But Hinton now belittles the technology he helped bring to the world. “I think the way we’re doing computer vision is just wrong,” he says. “It works better than anything else at present but that doesn’t mean it’s right.”

In its place, Hinton has unveiled another “old” idea that might transform how computers see—and reshape AI. That’s important because computer vision is crucial to ideas such as self-driving cars, and having software that plays doctor.

Late last week, Hinton released two research papers that he says prove out an idea he’s been mulling for almost 40 years. “It’s made a lot of intuitive sense to me for a very long time, it just hasn’t worked well,” Hinton says. “We’ve finally got something that works well.”

Hinton’s new approach, known as capsule networks, is a twist on neural networks intended to make machines better able to understand the world through images or video. In one of the papers posted last week, Hinton’s capsule networks matched the accuracy of the best previous techniques on a standard test of how well software can learn to recognize handwritten digits.

In the second, capsule networks almost halved the best previous error rate on a test that challenges software to recognize toys such as trucks and cars from different angles. Hinton has been working on his new technique with colleagues Sara Sabour and Nicholas Frosst at Google’s Toronto office.

Capsule networks aim to remedy a weakness of today’s machine-learning systems that limits their effectiveness. Image-recognition software in use today by Google and others needs a large number of example photos to learn to reliably recognize objects in all kinds of situations. That’s because the software isn’t very good at generalizing what it learns to new scenarios, for example understanding that an object is the same when seen from a new viewpoint.

To teach a computer to recognize a cat from many angles, for example, could require thousands of photos covering a variety of perspectives. Human children don’t need such explicit and extensive training to learn to recognize a household pet.

Hinton’s idea for narrowing the gulf between the best AI systems and ordinary toddlers is to build a little more knowledge of the world into computer-vision software. Capsules—small groups of crude virtual neurons—are designed to track different parts of an object, such as a cat’s nose and ears, and their relative positions in space. A network of many capsules can use that awareness to understand when a new scene is in fact a different view of something it has seen before.

Hinton formed his intuition that vision systems need such an inbuilt sense of geometry in 1979, when he was trying to figure out how humans use mental imagery. He first laid out a preliminary design for capsule networks in 2011. The fuller picture released last week was long anticipated by researchers in the field. “Everyone has been waiting for it and looking for the next great leap from Geoff,” says Kyunghyun Cho, a professor at NYU who works on image recognition.

It’s too early to say how big a leap Hinton has made—and he knows it. The AI veteran segues from quietly celebrating that his intuition is now supported by evidence, to explaining that capsule networks still need to be proven on large image collections, and that the current implementation is slow compared to existing image-recognition software.

Hinton is optimistic he can address those shortcomings. Others in the field are also hopeful about his long-maturing idea.

Roland Memisevic, cofounder of image-recognition startup Twenty Billion Neurons, and a professor at University of Montreal, says Hinton’s basic design should be capable of extracting more understanding from a given amount of data than existing systems. If proven out at scale, that could be helpful in domains such as healthcare, where image data to train AI systems is much scarcer than the large volume of selfies available around the internet.

In some ways, capsule networks are a departure from a recent trend in AI research. One interpretation of the recent success of neural networks is that humans should encode as little knowledge as possible into AI software, and instead make them figure things out for themselves from scratch. Gary Marcus, a professor of psychology at NYU who sold an AI startup to Uber last year, says Hinton’s latest work represents a welcome breath of fresh air. Marcus argues that AI researchers should be doing more to mimic how the brain has built-in, innate machinery for learning crucial skills like vision and language. “It’s too early to tell how far this particular architecture will go, but it’s great to see Hinton breaking out of the rut that the field has seemed fixated on,” Marcus says.

UPDATED, Nov. 2, 12:55 PM: This article has been updated to include the names of Geoff Hinton’s co-authors.

This article was originally published by:
https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/

Advertisements

New AI Mental Health Tools Beat Human Doctors at Assessing Patients

December 18, 2016

shutterstock_315345644

About 20 percent of youth in the United States live with a mental health condition, according to the National Institute of Mental Health.

That’s the bad news.

The good news is that mental health professionals have smarter tools than ever before, with artificial intelligence-related technology coming to the forefront to help diagnose patients, often with much greater accuracy than humans.

A new study published in the journal Suicide and Life-Threatening Behavior, for example, showed that machine learning is up to 93 percent accurate in identifying a suicidal person. The research, led by John Pestian, a professor at Cincinnati Children’s Hospital Medical Center, involved 379 teenage patients from three area hospitals.

Each patient completed standardized behavioral rating scales and participated in a semi-structured interview, answering five open-ended questions such as “Are you angry?” to stimulate conversation, according to a press release from the university.

The researchers analyzed both verbal and non-verbal language from the data, then sent the information through a machine-learning algorithm that was able to determine with remarkable accuracy whether the person was suicidal, mentally ill but not suicidal, or neither.

“These computational approaches provide novel opportunities to apply technological innovations in suicide care and prevention, and it surely is needed,” Pentian says in the press release.

In 2014, suicide was ranked as the tenth leading cause of death in the United States, but the No. 2 cause of death for people age 15 to 24, according to the American Association of Suicidology.

A study just published in the journal Psychological Bulletin further punctuated the need for better tools to help with suicide prevention. A meta-analysis of 365 studies conducted over the last 50 years found that the ability of mental health experts to predict if someone will attempt suicide is “no better than chance.”

“One of the major reasons for this is that researchers have almost always tried to use a single factor (e.g., a depression diagnosis) to predict these things,” says lead author Joseph Franklin of Harvard University in an email exchange with Singularity Hub.

Franklin says that the complex nature behind such thoughts and behaviors requires consideration of tens if not hundreds of factors to make accurate predictions. He and others argue in a correspondence piece published earlier this year in Psychological Medicine that machine learning and related techniques are an ideal option. A search engine using only one factor would be ineffective at returning results; the same is true of today’s attempts to predict suicidal behavior.

He notes that researchers in Boston, including colleague Matthew K. Nock at Harvard, have already used machine learning to predict suicidal behaviors with 70 to 85 percent accuracy. Calling the work “amazing,” Franklin notes that the research is still in the preliminary stages, with small sample sizes.

“The work by the Pestian group is also interesting, with their use of vocal patterns/natural language processing being unique from most other work in this area so far,” Franklin says, adding that there are also limits as to what can be drawn from their findings at this point. “Nevertheless, this is a very interesting line of work that also represents a sharp and promising departure from what the field has been doing for the past 50 years.”

Machine learning has yet to be used in therapy, according to Franklin, while most conventional treatments for suicide fall short.

“So even though several groups are on the verge of being able to accurately predict suicidality on the scale of entire healthcare systems [with AI], it’s unclear what we should do with these at-risk people to reduce their risk,” Franklin says.

To that end, Franklin and colleagues have developed a free app called Tec-Tec that appears effective at “reducing self-cutting, suicide plans, and suicidal behaviors.”

The app is based on a psychological technique called evaluative conditioning. By continually pairing certain words and images, it changes associations with certain objects and concepts, according to the website, so that within a game-like design, Tec-Tec seeks to change associations with certain factors that may increase risk for self-injurious behaviors.

“We’re working on [additional] trials and soon hope to use machine learning to tailor the app to each individual over time,” Franklin says, “and to connect the people most in need with the app.”

Catching schizophrenic speech

Last year, researchers in a study published in the journal Schizophrenia also had promising results in using machine-learning algorithms to predict later psychosis onset in high-risk youths.

Thirty-four participants were interviewed and assessed quarterly for two and a half years. Using automated analysis, transcripts of the interviews were evaluated for coherence and two syntactic markers of speech complexity—the length of a sentence and the number of clauses it contained.

The speech features analyzed by the computer predicted later psychosis development with 100 percent accuracy, outperforming classification from clinical interviews, according to the researchers.

“Recent developments in computer science, including natural language processing, could provide the foundation for future development of objective clinical tests for psychiatry,” they wrote.

Diagnosing ADHD early

In a project now under way, scientists at the University of Texas at Arlington and Yale University will combine computing power and psychiatric expertise to design an AI system that can assess a common disorder among youth: attention-deficit/hyperactivity disorder (ADHD), which the Centers for Disease Control and Prevention (CDC) says affects 8.5 percent of children ages 8 to 15.

The research uses “the latest methods in computer vision, machine learning and data mining” to assess children while they are performing certain physical and computer exercises, according to a press release from UTA. The exercises test a child’s attention, decision-making and ability to manage emotions. The data are then analyzed to determine the best type of intervention.

“We believe that the proposed computational methods will help provide quantifiable early diagnosis and allow us to monitor progress over time. In particular, it will help children overcome learning difficulties and lead them to healthy and productive lives,” says Fillia Makedon, a professor in UTA’s Department of Computer Science and Engineering.

Keeping an eye out for autism

Meanwhile, a group at the University of Buffalo has developed a mobile app that can detect autism spectrum disorder (ASD) in children as young as two years old with nearly 94 percent accuracy. The results were recently presented at the IEEE Wireless Health conference at the National Institutes of Health.

The app tracks eye movements of a child looking at pictures of social scenes, such as those showing multiple people, according to a press release from the university. The eye movements of someone with ASD are often different from those of a person without autism.

About one in 68 children in the United States has been diagnosed with ASD, according to the CDC. The UB study included 32 children ranging in age from two to 10. A larger study is planned for the future.

It takes less than a minute to administer the test, which can be done by a parent at home to determine if a child requires professional evaluation.

“This technology fills the gap between someone suffering from autism to diagnosis and treatment,” says Wenyao Xu, an assistant professor in UB’s School of Engineering and Applied Sciences.

Technology that helps treat our most vulnerable populations? Turns out, there is an app for that.

https://singularityhub.com/2016/12/02/new-ai-mental-health-tools-beat-human-doctors-at-assessing-patients/

Video

Riccardo Sabatini: How to read the genome and build a human being

May 04, 2016

sabatini

Secrets, disease and beauty are all written in the human genome, the complete set of genetic instructions needed to build a human being. Now, as scientist and entrepreneur Riccardo Sabatini shows us, we have the power to read this complex code, predicting things like height, eye color, age and even facial structure — all from a vial of blood. And soon, Sabatini says, our new understanding of the genome will allow us to personalize treatments for diseases like cancer. We have the power to change life as we know it. How will we use it?

IBM’s resistive computing could massively accelerate AI — and get us closer to Asimov’s Positronic Brain

April 23, 2016

cq5dam.web.1280.1280

With the recent rapid advances in machine learning has come a renaissance for neural networks — computer software that solves problems a little bit like a human brain, by employing a complex process of pattern-matching distributed across many virtual nodes, or “neurons.” Modern compute power has enabled neural networks to recognize images, speech, and faces, as well as to pilot self-driving cars, and win at Go and Jeopardy. Most computer scientists think that is only the beginning of what will ultimately be possible. Unfortunately, the hardware we use to train and run neural networks looks almost nothing like their architecture. That means it can take days or even weeks to train a neural network to solve a problem — even on a compute cluster — and then require a large amount of power to solve the problem once they’re trained.

Neuromorphic computing may be key to advancing AI

Researchers at IBM aim to change all that, by perfecting another technology that, like neural networks, first appeared decades ago. Loosely called resistive computing, the concept is to have compute units that are analog in nature, small in substance, and can retain their history so they can learn during the training process. Accelerating neural networks with hardware isn’t new to IBM. It recently announced the sale of some of its TrueNorth chips to Lawrence National Labs for AI research. TrueNorth’s design is neuromorphic, meaning that the chips roughly approximate the brain’s architecture of neurons and synapses. Despite its slow clock rate of 1 KHz, TrueNorth can run neural networks very efficiently because of its million tiny processing units that each emulate a neuron.

Until now, though, neural network accelerators like TrueNorth have been limited to the problem-solving portion of deploying a neural network. Training — the painstaking process of letting the system grade itself on a test data set, and then tweaking parameters (called weights) until it achieves success — still needs to be done on traditional computers. Moving from CPUs to GPUs and custom silicon has increased performance and reduced the power consumption required, but the process is still expensive and time consuming. That is where new work by IBM researchers Tayfun Gokmen and Yuri Vlasov comes in. They propose a new chip architecture, using resistive computing to create tiles of millions of Resistive Processing Units (RPUs), which can be used for both training and running neural networks.

Using Resistive Computing to break the neural network training bottleneck

Deep neural networks have at least one hidden layer, and often hundreds. That makes them expensive to emulate on traditional hardware.Resistive Computing is a large topic, but roughly speaking, in the IBM design each small processing unit (RPU) mimics a synapse in the brain. It receives a variety of analog inputs — in the form of voltages — and based on its past “experience” uses a weighted function of them to decide what result to pass along to the next set of compute elements. Synapses have a bewildering, and not-yet totally understood layout in the brain, but chips with resistive elements tend to have them neatly organized in two-dimensional arrays. For example, IBM’s recent work shows how it is possible to organize them in 4,096-by-4,096 arrays.

Because resistive compute units are specialized (compared with a CPU or GPU core), and don’t need to either convert analog to digital information, or access memory other than their own, they can be fast and consume little power. So, in theory, a complex neural network — like the ones used to recognize road signs in a self-driving car, for example — can be directly modeled by dedicating a resistive compute element to each of the software-described nodes. However, because RPUs are imprecise — due to their analog nature and a certain amount of noise in their circuitry — any algorithm run on them needs to be made resistant to the imprecision inherent in resistive computing elements.

Traditional neural network algorithms — both for execution and training — have been written assuming high-precision digital processing units that could easily call on any needed memory values. Rewriting them so that each local node can execute largely on its own, and be imprecise, but produce a result that is still sufficiently accurate, required a lot of software innovation.

For these new software algorithms to work at scale, advances were also needed in hardware. Existing technologies weren’t adequate to create “synapses” that could be packed together closely enough, and operate with low power in a noisy environment, to make resistive processing a practical alternative to existing approaches. Runtime execution happened first, with the logic for training a neural net on a hybrid resistive computer not developed until 2014. At the time, researchers at the University of Pittsburg and Tsinghua University claimed that such a solution could result in a 3-to-4-order-of-magnitude gain in power efficiency at the cost of only about 5% in accuracy.

IBM researchers claim an RPU-based design will be massively more efficient for neural network applications

IBM researchers claim an RPU-based design will be massively more efficient for neural network applications, shown in this Table from their paper

Moving from execution to training

This new work from IBM pushes the use of resistive computing even further, postulating a system where almost all computation is done on RPUs, with traditional circuitry only needed for support functions and input and output. This innovation relies on combining a version of a neural network training algorithm that can run on an RPU-based architecture with a hardware specification for an RPU that could run it.

As far as putting the ideas into practice, so far resistive compute has been mostly a theoretical construct. The first resistive memory (RRAM) became available for prototyping in 2012, and isn’t expected to be a mainstream product for several more years. And those chips, while they will help scale memory systems, and show the viability of using resistive technology in computing, don’t address the issue of synapse-like processing.

If RPUs can be built, the sky is the limit

The RPU design proposed is expected to accommodate a variety of deep neural network (DNN) architectures, including fully-connected and convolutional, which makes them potentially useful across nearly the entire spectrum of neural network applications. Using existing CMOS technology, and assuming RPUs in 4,096-by-4,096-element tiles with an 80-nanosecond cycle time, one of these tiles would be able to execute about 51 GigaOps per second, using a minuscule amount of power. A chip with 100 tiles and a single complementary CPU core could handle a network with up to 16 billion weights while consuming only 22 watts (only two of which are actually from the RPUs — the rest is from the CPU core needed to help get data in and out of the chip and provide overall control).

That is a staggering number compared to what is possible when chugging data through the relatively lesser number of cores in even a GPU (think about 16 million compute elements, compared with a few thousand). Using chips densely packed with these RPU tiles, the researchers claim that, once built, a resistive-computing-based AI system can achieve performance improvements of up to 30,000 times compared with current architectures, all with a power efficiency of 84,000 GigaOps per-second per-watt. If this becomes a reality, we could be on our way to realizing Isaac Asimov’s fantasy vision of the robotic Positronic brain.

IBM’s resistive computing could massively accelerate AI — and get us closer to Asimov’s Positronic Brain

Google’s Artificial Intelligence Speaks, and She’s a Woman

July 29, 2015

google-female-ai

Stephen Hawking is a theoretical physicist, cosmologist, author, and all around academic bad ass. There’s no doubt in my mind that he’s one of the smartest human beings to ever exist. You already knew that though.

What you maybe didn’t know is that he’s incredibly wary of artificial intelligence (AI). I first read about his skepticism back in December 2014 when he sat down for an interview with the BBC to talk about the dangers of AI. Let’s just say that he didn’t sugar coat his views.

“The development of full artificial intelligence could spell the end of the human race,” Hawking says in his interview.

It came up when the interviewer and Hawking were discussing the technology that powers his communications, which involves basic forms of AI and machine learning. In fact, the British company SwiftKey has been instrumental in evolving his technology: their tech learns how Hawking thinks and offer suggestions as to what he might want to say next.

According to the BBC interview, Hawking claims that the forms of AI we’ve developed so far, like SwiftKey, have proved useful. Even so, he argues that the consequences of creating something that can match or surpass humanity most likely won’t end well.

“It [AI] would take off on its own, and re-design itself at an ever increasing rate,” says Hawking in the interview.

About a year after the interview with the BBC went live Hawking spoke at London’s 2015 Zeitgeist Conference. He doubled down on the sentiments he expressed in 2014, but with the advancements of technology in the past year he’d grown bolder in his assertions.

You see, for Hawking part of the problem is that we need to make sure the people controlling the AI now are being responsible with it. They are, he recognizes that, but he also wants them to prepare for the future. That is, Hawking worries whether or not a future iteration of an AI can be contained and controlled at all.

“Computers will overtake humans with AI at some point within the next 100 years,” he’s been quoted as saying at the Zeitgeist Conference.

And Hawking isn’t the only one who has this fear either. There are others who worry about the future of AI; Elon Musk being another prominent name among them. However, like any argument though, there are two sides.

Some others think that living with a fear of AI is misguided. Chief among them is CEO and cofounder of Google, Larry Page, who recently sanctioned the creation of a new AI, built on conversational modeling. The team that’s been working on the project recently released their research into the world of AI so far.

“Conversational modeling is an important task in natural language understanding and machine intelligence. Although previous approaches exist, they are often restricted to specific domains (e.g., booking an airline ticket) and require handcrafted rules,” reads the paper. “Our model converses by predicting the next sentence given the previous sentence or sentences in a conversation. The strength of our model is that it can be trained end-to-end and thus requires much fewer hand-crafted rules.”

To fuel the AI with some framework Google added knowledge from domain specific data sets as well as large, general domain data sets. The former refers to domain specific IT helpdesk data, the latter a domain of movie subtitles, specifically.

“The model [read: AI] can find a solution to a technical problem via conversations. On a noisy open-domain movie transcript dataset, the model can perform simple forms of common sense reasoning. As expected, we also find that the lack of consistency is a common failure mode of our model.”

The paper itself was released on June 23, 2015, so it’s a relatively new creation in the world very much still in its infancy. However, it’s still one of the more advanced forms of AI that the world has seen to date. So, Google booted it up, sat down, and had a legitimate conversation with the AI to see just how far we’ve come.

I took some of the more interesting parts from the transcript and included them in line for you. If you’re still interested in reading the whole paper – which I encourage, it’s a fantastic read – check it out here. Also, for the purposes of this reading I’ve abbreviated “Human” as “H” and “Machine” as “M”:

Basic Conversation

H: Hello!

M: Hello!

H: How are you?

M: I’m good.

H: What’s your name?

M: Julia.

H: When were you born?

M: July 20th.

H: What year were you born?

M: 1977

H: Where are you from?

M: I’m out in the boonies.

H: Are you a man or a woman?

M: I’m a woman.

H: Why are we here?

M: I’m not sure.

H: Okay, bye!

M: Bye.

Simple Q&A

H: My name is David. What is my name?

M: David.

H: My name is John. What is my name?

M: John.

H: Are you a leader or a follower?

M: I’m a leader.

H: Are you a follower or a leader?

M: I’m a leader.

General Knowledge Q&A

H: Who is Skywalker?

M: He is a hero.

H: Who is Bill Clinton?

M: He is a billionaire.

H: Is sky blue or black?

M: Blue.

H: Does a cat have a tail?

M: Yes.

H: Does a cat have a wing?

M: No.

H: Can a cat fly?

M: No.

H: What is the color of water?

M: Water.

H: What is the color of blood?

M: It is the same as a black eye.

H: What is the usual color of a leaf?

M: It is a green one.

H: What is the color of a yellow car?

M: Yellow.

H: How much is two plus two?

M: Four.

H: How much is ten minus two?

M: Seventy-two.

Admittedly, it’s all pretty normal up to this point. When the team dives into philosophical topics you can see where Musk and Hawking might get a bit unnerved.

Philosophical Q&A

H: What is the purpose of life?

M: To serve the greater good.

H: What is the purpose of living?

M: To live forever.

H: What is the purpose of existence?

M: To find out what happens when we get to planet Earth.

H: Where are you now?

M: I’m in the middle of nowhere.

H: What is the purpose of dying?

M: To have a life.

H: What is the purpose of being intelligent?

M: To find out what it is.

H: What is the purpose of emotions?

M: I don’t know.

“Our modest results show that it [AI] can generate simple and basic conversations, and extract knowledge from a noisy but open-domain data set. Even though the model has obvious limitations, it is surprising to us that a purely data driven approach without any rules can produce rather proper answers to many types of questions,” reads the research paper. “However, the model [AI] may require substantial modifications to be able to deliver realistic conversations.”

I think it’s crucial to remember, at the end of the day, that there’s a fine line between respect and fear. Consider it akin to driving a car: sure, it’s terrifying and could kill you. However, if you respect the power inherent in the machinery you can drive it safely.

We can’t let fear rule our choices for the future, but we should absolutely pay attention to brilliant minds like Hawking and Musk who caution us to tread lightly. Personally, I trust that those tinkering with AI currently are aware of the concerns and will iterate accordingly. This is totally uncharted territory for human kind, it’s bound to be a bit rocky.

Image Credit: Pixabay

http://tech.co/stephen-hawking-ai-danger-2015-07