In 2016, Facebook unveiled their AI (artificial intelligence) technology, which attempts to automatically describe images to benefit people who are blind or visually impaired. While we are pretty excited about its potential, it’s important to understand where this technology might be helpful, and what its limitations are.

One of the most fundamental tasks when making a website accessible is ensuring that all of your images are also accessible to people who can’t see them. This generally involves adding alt attributes to the image tags. But what if, like Facebook, you allow your users to upload their own photos? How do you ensure that the thousands of images being uploaded each minute are properly described? Until recently, you couldn’t. But this solution is far from perfect.

To illustrate the problems with this technology, let’s look at a few examples of some randomly chosen images, along with what some AI image recognition services think they contain. You’ll notice that most of this output would need to be massaged into an appropriate format for an alt attribute, but that’s a relatively simple matter. The more important question is whether or not the information generated by these services sufficiently describes the images.

Image	Human Crafted Alt Attribute	Facebook Alt Attribute	Amazon Output	Google Output	Microsoft Output
	couple standing with labradoodle and holding 8 puppies	Image may contain: 2 people, people smiling, dog, tree, and outdoor	Animal Canine Dog Pet Puppy Flora Jar Plant Potted Plant Pottery Vase Golden Retriever Adorable Chow Blossom Flower Backyard Outdoors Yard Toy Tree Cherry Blossom	Golden Retriever Labrador Retriever Pit Bull Pug French Bulldog Puppy Keri Barker Animal Retriever Pet Dog breed Companion dog Image Cuteness Dog	Grass Dog Outdoor Person Brown Photo Posing Man Standing Bench Front Sitting Holding Truck Black Park Large White Hot Red Woman Field
	woman wearing sunglasses and smiling in swimming pool	Image may contain: 1 person, smiling, pool and outdoor	Pool Water Sport Sports Swimming Building Hotel Resort Swimming Pool House Housing Villa	Beirut Cairo Yara Swimming Pool Image Artist Swimming Film Resort Swimming Recreation Sabah Mahmoud Hemida Faten Hamama Lebanon	Water Pool Building Outdoor Swimming Table Blue Sitting Man Umbrella Boat Dock Pond Standing Bird Dog People Lake Top Large Body Woman White Young Beach Playing Frisbee Eating Board Riding Court Ocean
	new yorker style cartoon of man being visited in the hospital. He’s completely in traction and has a balloon tied to the foot of his bed. The caption says quote any improvement since I brought the balloon? end quote	Image may contain: drawing	Art Drawing Sketch Diagram Plan Doodle Text Label	The New Yorker Cartoon Humour Comics Drawing Image Sketch Magazine Cartoons Magazine Book Canvas Print Gary Larson Julia Suits Cartoon Cartoons Farley Katz	None
	graphite pencil sketch of a person’s face	Image may contain: drawing	Art Drawing Sketch	Sketch Drawing Sketchbook Visual Arts Art Pencil Hatching Illustration Figure Drawing Line Art Work of Art Portrait Portrait Drawing Cartoon Nose	Drawing
	blacktop parking spaces with the updated, forward leaning wheelchair accessible icons painted in each space	No automatic alt text available	Business Card Paper Text Asphalt Tarmac	Disability Disabled Parking Permit International Symbol of Access Car Park Accessibility Symbol Wheelchair Sign Americans with Disabilities Act of 1990 Don’t DIS my ABILITY Image Tactile Paving Independent Living Services and supports for people with disabilities Sticker	Road Building Tennis Man Court Holding Blue Street Playing Sign Parking Game City Player Ball Standing
	muscular man lifting barbell in gym	Image may contain: 1 person	Human People Person Exercise Fitness Gym Sport Sports Working Out Building Factory	Martini Power Lifting Barbell Squat Dumbbell Strength Athletics Deadlift Bodybuilding Bodypump Exercise Machine Physical Exercise Human Back Bar Calf Dorothy Parker	Indoor Sport Person Standing Woman Holding Black Young Man Kitchen Room Table Top Large Wearing Red Dog Display White Luggage Playing Ball Plane
	Star Wars droid BB-8	No automatic alt text available	Flora Food Plant Produce Pumpkin Squash Vegetable Inflatable Figurine Toy Bowl	BB-8 Chewbacca Jack-o-lantern Star Wars Pumpkin Lantern Droid Star Wars Opening Crawl Porg Halloween Trick-or-treating Carving Stencil Star Wars: The Last Jedi	Outdoor Sitting Small Orange White Table Lamp Top Sidewalk Vase Flower Black Large Green Street Standing Yellow Food Dirty Parked Plate
	row of parked school buses	Image may contain: outdoor	Bus Transportation Vehicle School Bus Cable Car Streetcar Tram Trolley	Greenville County School District School Bus Bus Elizabeth Public Schools School Transport School District School Bus Contractor Public Transport Field Trip School Bus Yellow Education Seat Belt Road South Carolina	Yellow Outdoor Road Transport Bus School Truck Parked Large Black Lot Street Sitting Parking Side City Driving

As you can see from the information in the previous table, even when AI recognizes what is in an image, the amount of detail provided is far from sufficient in most cases. These algorithms also have difficulty determining what elements in the image are important, and which are simply incidental. As a result, while you can sometimes tell which objects might be in the image, it can be quite difficult to derive what the image is trying to convey.

In short, while this technology does show a great deal of promise, it should, at present, only be used in situations where it is entirely impossible to provide descriptions in any other way. Most alt attributes on the web should continue to be written by humans for the foreseeable future. On the other hand, this technology is better than nothing, though perhaps only just.

For more on what Facebook has been doing in this regard recently, visit https://www.cnet.com/news/facebook-ai-blind-photo-recognition-caption/.

← Return to the blog