The smartphone app uses computer vision to give visually impaired users a description of their surroundings and environment. Once downloaded, users point their iPhone camera at a person and let the AI take over. The app will say who the person is and their current emotion. Seeing AI will also work on items, such as products. In this instance, the app will tell the users what the product is. Microsoft says the AI it uses runs natively on the devices, so there is no need to be connected to the internet or cloud.
In terms of identifying people, the app can only do this positively if it has ‘seen’ them before. However, it can also guess the age of strangers and how all people are feeling. Household items can be recognized by scanning the barcodes. Additionally, Seeing AI can scan documents and understand the US dollar. Visually impaired users can pay for goods and not get confused by which bills they are using. Microsoft used artificial intelligence to give the app its power. Seeing AI uses a neural network to identify items and people in the real world. The company essentially took technology used for automated devices and vehicles and applied it to the app. While users can open and use the app in any circumstance for basic functions, some of the more experimental abilities need a connection. For example, Seeing AI will describe an entire environment and can even try to recognize handwriting. These two capabilities are still in a testing phase and will not be as robust as the basic feature set discussed above.
Availability
As mentioned, Seeing AI is so far limited to Apple’s mobile platform and is available in the United States, Canada, India, Hong Kong, New Zealand and Singapore. Microsoft has not said anything about bringing the app to Android or Windows 10 Mobile (wishful thinking). Equally, there is no official word on when the app will come to other countries. Indeed, it feels like Microsoft is seeing how this works before committing to a full roll out. If visually impaired users a positive towards the app, we guess it will spread its wings soon. Microsoft has been making headway in helping visually impaired users. Last year, the company opened Project Torino, a physical programming language that gives programming access to visually impaired children.
