Table of Contents
Microsoft Targets Food Logging of Nutritional Meals
I’ve been trying to watch what I am eating and capture workouts on a stationary bike and with resistance bands. Some sites can track those, and they can estimate how many calories you consume with your meals, and how many of those you may burn off with exercise. I was surprised to see a patent from Microsoft that described a slightly different approach to tracking meals and nutritional values, using a process referred to as food logging. The focus of the patent is a visual one, taking photos of what you eat, and letting it try to track what you eat and analyze it. It sounds interesting, and I would like to try it out someday. Wondering if this may become a feature on Bing, or through another site run by Microsoft.
Related Content:
The patent granted this week covers improving a process of logging food intake by a person, to try to improve upon it. It starts by telling us about “Food Logging,” which is “monitoring food eaten by individuals along with various nutritional information associated with that food” and it tells us of some of the reasons why people engage in food logging. These include:
Reducing Obesity, which the patent tells us is “linked to conditions such as cardiovascular disease, diabetes, and cancer, and dramatically impacts both life expectancy and quality of life.” It also tells us that people have difficulty in making changes to “diet and exercise habits.” Also, that Food Logging helps support people making such changes. It then tells us that, “food logging is known to be well-correlated to increased initial weight loss and improved weight maintenance.”
How can Food Logging be Improved upon?
The effectiveness of Food Logging is limited by how inconvenient it can be. A technological approach that could help would be to make food logging more automatic, is by inferring nutritional information from a single food image. There are limits to how effective that may be done for several reasons. This patent focuses on using images such as those to gain nutritional information about meals. It seems that:
1. Some food may be hidden in images, causing information to be missed.
2., It is “unlikely that visual information alone conveys all the details of food preparation” such as the amount of oil, fat content of meats, sugar content, salt content, etc.)
3 It can be difficult to estimate volumes of food from single images, too.
Challenges of Tracking Nutrition from Images when Food Logging
Ideally, it can be difficult to estimate amounts of calories, fats, carbohydrates, etc.) from single images of realistic meals. Doing so from a single image can be challenging, and not very user friendly.
Often the focus is on “core computer vision challenges, using something like the “Pittsburgh Fast-Food Image Dataset”, or “user-supplied images and nutritional statistics to bootstrap classification.” This might provide limited information when it comes to portion sizes and nutritional values.
The patent also points to the possible use of “manual crowd-sourced assessments of nutritional information based on images of food being consumed.” This could take a fair amount of time and effort by people, but we are told that the results tend to be “similar to those supplied by a dietitian.”
Machine Learning of Models of Meals
Given the challenges of using images in Food Logging, the patent provides a different approach that involves “learning or training one or more image-based models (referred to herein as “meal models”) of the nutritional content of meals.” We are told that:
Training of meal models is based on one or more datasets of images of meals in combination with “meal features” that describe various parameters of the meal. Examples of meal features include but are not limited to, food type, meal contents, portion size, nutritional content (e.g., calories, vitamins, minerals, carbohydrates, protein, salt, etc.), a food source (e.g., specific restaurants or restaurant chains, grocery stores, particular pre-packaged foods, school meals, meals prepared at home, etc.), ingredients, etc. If the source or location of meals is known, the meal model may include source-specific classifiers that can be used to constrain recognition based on particular sources or locations.
A food logger who may be using those meal model images may be able to “add, remove, or modify any parameters (e.g., portion size, plate size, side dishes, drinks, etc.) used to estimate nutritional content of particular meals being consumed by the user.” A user interface would be capable of being able to do things like “adds a tablespoon of butter to a baked potato.”
These meal models could be based upon images and information from sources such as “particular restaurants, home-cooked meals, school meals, etc.”
We are told that a machine learning component of a food logger program might be used to learn about or train meal models, and information from those images might then be “either captured by the user (e.g., camera, cell phone, head-worn eyeglasses with one or more cameras or imaging devices, etc.) or captured via one or more cameras or imaging devices positioned to automatically capture images of food items of complete meals to be consumed by the user.”
Nutritional information captured by this system could then be studied later by a person using this system.
If a meal was eaten at a restaurant, the Food Logger program could use tracking or localization techniques, such as GPS,0 cell-tower based localization, RFID, or user specification of current location (e.g., typing restaurant name, using speech input, selecting from a list of favorite restaurants, etc.), to determine where the user is when the meal image is provided for recognition. As the patent tells us:
…for example, that the user is at the “Solo Grill” on King Street in Toronto, Canada, the Food Logger then constrains the meal image recognition process to nutritional information in the meal model for meals identified as originating from the Solo Grill.
Capturing meal model information from a place such as a restaurant may make it more likely that no food items were likely missed, and all nutritional information was captured, “by considering the meal as a whole entity on a per-restaurant basis, additional information such as ingredients (e.g., peanut sauce, shellfish, etc.) and preparation details (e.g., baked, fried, poached, etc.) can be encoded into the meal model along with corresponding nutritional information.”
This does sound like it would be helpful and convenient for someone trying to log their food intake.
The patent also describes some other possibilities of capturing information about meals:
Advantageously, text menus describing particular meals for most restaurants are generally easily available from a wide variety of online sources, e.g., sites such as Yelp.RTM. or Foursquare.com, or individual restaurants’ websites. Further, a high percentage of such text menus also include nutritional information (typically caloric content, at least, and often additional nutritional information). Note also that if not available in combination with the menu, estimates of nutritional information for use in constructing labeled examples for training meal models can be hand-coded, crowd-sourced, estimated based on similar meals from other restaurants or sources, etc.
The patent also tells us that it might be possible to get information about meals and their nutrition from searches on the Web.
The patent is:
Food logging from images
Inventors: Neel Suresh Joshi, Siddharth Khullar, T Scott Saponas, Daniel Morris, and Oscar Beijbom
Assignee: Microsoft Technology Licensing, LLC (Redmond, WA)
US Patent: 9,977,980
Granted: May 22, 2018
Filed: April 17, 2017
Abstract
A “Food Logger” provides various approaches for learning or training one or more image-based models (referred to herein as “meal models”) of the nutritional content of meals. This training is based on one or more datasets of images of meals in combination with “meal features” that describe various parameters of the meal. Examples of meal features include but are not limited to, food type, meal contents, portion size, nutritional content (e.g., calories, vitamins, minerals, carbohydrates, protein, salt, etc.), a food source (e.g., specific restaurants or restaurant chains, grocery stores, particular pre-packaged foods, school meals, meals prepared at home, etc.). Given the trained models, the Food Logger automatically provides estimates of nutritional information based on automated recognition of new images of meals provided by (or for) the user. This nutritional information is then used to enable a wide range of user-centric interactions relating to food consumed by individual users.
Take Aways involving Food Logging
This is one example of how a search provider might attempt to help many people using a machine learning approach, and using information that it could gain from external sources, or search for on the Web. If nutritional information was available for menu items online in structured data, a food logger system such as the one described in this patent might be even easier to develop.
It would be interesting seeing it come to life.
Search News Straight To Your Inbox
*Required
About Bill Slawski
With more than 26 years of SEO experience and a Juris Doctor Degree, Bill Slawski is the foremost expert on Google’s patents as related to SEO. Patent Exploration is one of the quickest and most detailed ways to find new information about SEO. Bill is the Editor of SEO by the Sea, a prominent search engine optimization blog, where he is the author of over 1,300 posts. Bill’s experience includes Fortune 500 brands and some of the largest websites in the world. Bill is a contributing author for Moz, Search Engine Land, and Search Engine Journal. In 2014-2021, he spoke at industry-leading international conferences about topics including search engine algorithms, universal and blended search, personalization in search, search and social, and duplicate content problems, structured data, and schemaJoin thousands of marketers to get the best search news in under 5 minutes. Get resources, tips and more with The Splash newsletter:
