Machine learning image classifiers use context clues to help understand the contents of a room, for example, if they manage to identify a dining-room table with a high degree of confidence, that can help resolve ambiguity about other objects nearby, identifying them as chairs.
The downside of this powerful approach is that it means machine learning classifiers can be confounded by confusing, out-of-context elements in a scene, as is demonstrated in The Elephant in the Room, a paper from a trio of Toronto-based computer science academics.
The authors show that computer vision systems that are able to confidently identify a large number of items in a living-room scene (a man, a chair, a TV, a sofa, etc) become fatally confused when they add an elephant to the room. The presence of the unexpected item throws the classifiers into dire confusion: not only do they struggle to identify the elephant, they also struggle with everything else in the scene, including items they were able to confidently identify when the elephant was absent.
It's a new wrinkle on the idea of adversarial examples, those minor, often human-imperceptible changes to inputs that can completely confuse machine-learning systems.
Contextual Reasoning: It is not common for current object detectors to explicitly take into account context on a semantic level, meaning that interplay between object categories and their relative spatial layout (or possibly additional) relations) are encoded in the reasoning process of the network. Though many methods claim to incorporate contextual reasoning, this is done more in a feature-wise level, meaning that global image information is encoded somehow in each decision. This is in contrast to older works, in which explicit contextual reasoning was quite popular (see [3] for mention of many such works). Still, it is apparent that some implicit form of contextual reasoning does seem to take place. One such example is a person detected near the keyboard (Figure 6, last column, last row). Some of the created images contain pairs of objects that may never appear together in the same image in the training set, or otherwise give rise to scenes with unlikely configurations. For example, non co-occurring categories, such as elephants and books, or unlikely spatial / functional relations such as a large person (in terms of image area) above a small bus. Such scenes could cause misinterpretation due to contextual reasoning, whether it is learned explicitly or not.
The Elephant in the Room [Amir Rosenfeld, Richard Zemel and John K. Tsotsos/Arxiv]
Machine Learning Confronts the Elephant in the Room [Kevin Hartnett/Quanta]
