Compatible with Android 9,10,11
Role: UI/UX Design and Unity Development
Client: Technological University Dublin
Duration: 24 weeks (May-Oct 2021)
Actave, an assistive technology application that combines augmented reality with sound tagging to enable people living with dementia to access contextual information in indoor environments. This was an individual Master's thesis project.
Design and develop an application that can play information based on spatial context which provides customised information in indoor spaces, specifically addressing Elderly People with Dementia.
Problem
Due to memory loss, dementia can severely impact one's ability to perform activities of daily living. In terms of moving around their homes, people with dementia are not able to obtain contextual information.
Target Group
Person living with dementia at home and personal carers of individuals living with dementia
Solution
Aim
An application that enables audio to be tagged with spatial context using Augmented reality. This means that, when the user is close to the area, the relevant information is played to support and enable independent living for people with dementia.
Design Thinking Process
IDEO Design thinking toolkit was utilized based on my need to explore emerging technology to integrate human needs into tailored technology for exploring and analyzing the users needs based on the emerging technology to explore the field in which I am new and the timeline for the project. I used the tool kit during Pandemic times to streamline the process and outline the research and execution phases for the duration of the project.
Primary research
Telephonic Interview: In the primary research, I contacted the stakeholders of people living with dementia, the stakeholders include clinical therapists, physiotherapists, and people who were part of the ecosystem in the government and non-government sectors. The covid-19 situation prevented me from meeting them in person, so I contacted them via telephone as well as sent them the concert note and the research that I'm doing.
Questionnaire: The survey link was sent to clinical therapists who provide care for dementia patients to better understand how technology and assistive applications are used by those living with dementia. I sent out a sample video explaining the AR application with visual UI that I had developed as an initial concept.
Survey Link: https://forms.gle/smVTtuemWHyKVnYz9
Insights
The use of assistive technology should never replace human contact or substitute for formal or informal social care.
-The information gathered by telecare systems must be stored safely and used only for the purposes for which it was intended
Observation
In order to better understand the pain points of living independently with dementia and to learn the eco-system of assistive technology for people living with dementia, I collaborated with a home care provider company. This helped me spend time with people who are living independently and diagnosed with dementia.
I visited 7 people living with dementia. This helped me spend time with people who are living independently and diagnosed with dementia. In addition to meeting people with dementia on a regular basis, I was able to interact with the family members of those individuals. The overall research took place over a period of 5 months with regular meetings. In some cases, I could see the progression of dementia.
When spending time with the patients in their private space, one gains a lot of insight and understanding about the people with dementia, as well as about the family members and Inner circle of the patient.
Learning
​
The person living with dementia believe that they should not be dependent on people hence they shy away from the problems
They aren't against using modern technology, but they don't want to learn or interact more with screens with complex steps and processes.
​
Care eco-system mapping
Secondary Research
As a secondary research method, data was collected through the internet and scholarly reports about assistive technology for people living with dementia.
- 81% of respondents were concerned about a decline in the person with dementia
- 54% of family carers reported a decline in their mental health, and 40% a decline in their physical health
- 28% of family carers reported considering a move to long term care for the person with dementia, with 65% saying
that this has become a consideration sooner due to the pandemic.
- The average weekly cost of residential care varies depending on the type and location. In public long-stay facilities,
it costs an average of €1,526 per week
Although it has potential benefits, AT may also have unintended impacts for users or their families and careful consideration should be given to the ethical issues involved
​​
The current technology is seriously affecting and in some cases compromising people's privacy and data because AR and VR applications literally look into the personal space of people, so there should be a serious consideration for privacy, and families should be informed about the data and what the application is doing with it.
Learnings derived from Secondary research
Identifying assistive technologies already available on the market and widely used. Currently, there is a great deal of research being conducted in this area where design and technology play a major role. By exploring this through desk research, I discovered smart sensor rooms that are very close to what I am seeking in terms of assistance.
Persona
Primary User
Secondary User
According to the research, family members and personal carers are crucial to the success of assistive technology. I created two personas based upon this insight: the primary one being people living with dementia, and the secondary one being their personal caregivers
Empathy Map
Touch Points
The parameter of high-to-low frustration was mapped on the basis of what could be the most frustrating and least frustrating thing that patients could do while experiencing memory loss during their everyday activities and how it could reduce the stress among patients during those moments from waking up to going to bed each day.
As a result of its intervention, the product should assist people with dementia in reducing their frustrations during their daily activities. This will be achieved when the product provides information at the moment when it is needed without zero to little effort, and it should feel natural to them. This graph was mapped when the early prototype was tested with the user and interviewed based on the their frustration level
Telecare Devices
Competitor analysis
Smart Sensors and IoT Devices (Smart homes)
Voice Assistant Devices
(Alexa and google home)
Dementia patients who have trouble remembering can benefit from these devices How to do certain things (or not to do them). They can be programmed with pre-recorded voice messages that automatically activate at set times (e.g. reminding the person it is time to eat). Other devices can have movement sensors incorporated so that a voice message is activated when the person passes the device. Some devices can display a video message or provide. Some devices require the user to press a button to hear a voice reminder. Installation of the device needs to be considered for it to be effective. Voice quality may be an issue. Battery life needs to be monitored. The use of text and images has to be appropriate to the person and take into account the culture of the user which is difficult for these devices.
Product Criteria
Based on the research I realised that these are the must-have product criteria for assistive technology that can solve the problem of the person living with dementia to live independently at home
Easy to use
Highly customisable for individual user needs
Why Augmented Reality
Cost-effective
Augmented reality technology will be cost-effective to scale and efficient both indoor and outdoor spaces. Customization for specific user needs will be a huge differentiator for the product and the ease of doing it so is an added advantage the use of Augmented reality and the technologies around it is growing.
With these factors in place and evaluating other technology based on the user needs Augmented reality application was selected to solve the problem of assisting people living with dementia.
Concept Ideation
After interacting with stakeholders, I had the hypothesis that I would use augmented reality to solve the problem of finding and using things for people living with dementia through both audio and visual user interface formats, and to that end, I visualised two functionalities: how to use something and where to find something. It occurred to me that this would make it easy for people with dementia to go about their daily activities
As a result of the interviews and observations, I concluded that visual augmentation can actually cause more problems than it solves, so audio was chosen as the method of augmentation.
Learning
Chances of hallucinations are high as dementia progresses and visual augmentation can add to the complication.
Visual assets are difficult to personalize since it is difficult to design visual assets for specific users, from both the company's and secondary users' perspectives (family members or caregivers). If it is not personalised for each user and uses generic assets, then it will not be an effective solution.
​
Both audio and visual information overload can occur for users.
​
Difficult to scale the solution.
​
Audio augmentation can be applied to any wearable device with a camera, whereas visual augmentation is limited to AR Glasses. This provides the user with the flexibility to customise the wearable based on their needs and requirements.
As a result of the above analysis and feedback from users, audio-only augmentation was implemented
User Journey
User Journey
Person living with dementia
Experience set-up flow (Personal carers)
Updated by Personal carers
The experience is initially set up by the secondary user, usually the caregiver from the patient's mobile phone. This is done by scanning the area and adding audio tags based on the needs of the patient. Then the person with dementia can use the application as and when they need it.
This application aims to keep the interaction between a person living with dementia and a mobile device as minimal as possible. This is partially achieved, and it is illustrated using the interaction between the user and the real world.
Tools Used for Application design & development
Experience
Primary users of the application are people with dementia who can access information tagged in indoor spaces, Family members or caretakers can use the patient's app to set up audio tags based on the patient's needs. This will be a one-time set, and the user will use the application only when there is a need to edit or change the existing audio tag.
Environment Design
Applying standard UX design principle for Augmented reality to evaluate the application and understand where the application fit in the AR spectrum
The experience happens in the personal space and the tag are triggered under 4 feet as the application aims to be precise with space context to help the users retrieve their information at home
Testing & Analysis
First Prototype testing
Aim: To know if the users are comfortable with the process of scanning the area, anchoring Cloud anchor and save the file
Task: Audio in the space will be tagged by the user through their mobile device. It basically involves the user opening the application and allowing the camera access then, they move the smartphone camera around to cover the area they want to tag They scan the area for 30 seconds, as recommended by ARCore.
A total of six people tested the application, including secondary users (family members and caregivers of people living with dementia) and people who are potential users of the application.
User feedback and observation
- There are only two buttons on the user interface, and it does not look overly complicated at the start.
- We need a few instructions so we know what to do.
- Holding on to the phone for a long time and having to move to set up things is painful. They will definitely do it once if it
helps the dependent patient but they were unsure if they would do it frequently
- It is interesting to hear the audio play in a certain area and then change to another without clicking anything on the phone.
Testing Phase 2
Some of the User feedback was addressed this build in terms of User interface and also the resolved the technical glitches.
Adding button to place the Audio anchor
There were two major benefits of this design: first, users were comfortable knowing that something would happen when they tapped on the button, and second, the issue of multiple anchors being placed in the space was solved since the user would not tap on the screen but on the button.
​
Using anchor indicator
A green color translucent indicator was added to show the user where the anchor will be positioned. Instead of allowing the user to tap on any point on the screen to position the anchor, the function was placed in a button
​
Loading animation
The loading animation ensures the user will press the button for three seconds to place the anchor. This is a UI addition to make sure the user wants to place the anchor in the specific location.
​
Feedback
The process of scanning and adding audio tags would be easier if there was an instruction page at the beginning of the screen.
Users felt that this was too large a target area for a small house because the audio was played from a far distance (the target area was 5 meters).
The user was confused by the technical text on the UI action buttons and tabs. Common terms are needed to make the actions easier to understand.
Branding
This octogon shape represents an octave. In Western music, an octave is an interval of eight notes in diatonic scales (major, minor, and modal). The information is retrieved as audio at different time intervals in a space, enabling individuals living with dementia to live independently. It's a shape representing the time interval that the information is retrieved.
Research has show that people with dementia express feelings and ideas. Encourages physical activity, dance, or movement. Music can trigger emotions, feelings, and memories, particularly when it's tied to personal experiences. The strength of music should somehow be reflected in the branding of a product with the core sensory of audio.
As a result of the application, people living with dementia can live independently and be active by doing their own chores. As a result, they are more active and engaged. In old age, being active is crucial, especially for those living with dementia. As a result of the word play between active and octave, the name of the application that was shortlisted is a name that resonates with the purpose of the product.
A bright colour palette is used to differentiate both physically illuminated objects and virtual content in AR. If the gradient is clear and distinct than a flat colour, then appearance differences due to scene illumination can be significant
User Interface
Login & Home Screen
The secondary user (the caregiver or relative) creates an account for the person living with dementia and create audio tags based on their specific needs. This is a one-time exercise. They don't have to repeat it again unless they want to edit the existing experience
The home screen has 3 buttons and the first two are particularly prominent. This is because they are the main functions that allow you to create the experience and edit it when needed. There are only a few elements on the home screen that are relevant to the app's function, and the layout is straightforward.
Scan area
It is easier to tag the anchor in accordance with the AR core guidelines if the area is scanned well. It is prudent to scan the area in all directions for at least 30 seconds. In order to make the user aware of this function, instructions are displayed to guide them through the process.
Adding audio tag
The user will be able to add the audio tag in the desired location where they want to pin the information. They can add as many tags as they want to suit their needs by hard pressing the "Add audio tag" button and placing the audio in the circle indicated above, the audio will be tagged. The circle indicates where the tag will be placed in the real-world environment.
Upon adding all the audio tags, the user can save the entire experience, that is, multiple audio tags in the scanned area, which can be used by the person living with dementia to retrieve information when needed during daily living chores.
Using the anchor name, the entire experience can be saved, including the tags and scanned area.
Testing Phase 3
As a result of feedback from the previous testing, the application's skin has been updated, and it now appears to be complete. User input was used to design the UI.
Improvements based on user feedback and observation
- The target area was reduced from 5 meters to 2 meters.
- An improved user interface to make editing easier and more intuitive
- A link was sent to the user explaining that the majority of the personal data is stored in the device's
internal storage and stored in ARCore.
The anchoring area was reduced because users tapped the screen to tag the anchor in the real world above their shoulder. A skewed proportion exists between the real world and mobile screens, which can create an illusion. To overcome this, the target anchor in the UI was designed to appear slightly off centre.
The tests included a number of elderly people and some people living with early stages of dementia. Feedback was positive, and the only complaint was having to carry their mobile phones all the time, and they preferred it to automatically play. The suggestions of AR glasses and wearable devices were also well received, both from the elderly and from their family members.
Conclusion
According to the user research and feedback, I received while working on the project so far. Integrating wearable devices with the application can be of huge benefit to people living with dementia. Despite the technological constraints during the development stage, the barriers were addressed as much as possible within the limited resources and time available.
During covid, I managed to meet the 7 elderly people living with dementia and their personal carers for research, I connected with the stakeholders who work in the capacity of researchers and organisations who work for people living with dementia which helped me gain a significant understanding of dementia and assistive technology.
In order to validate the idea around a virtual audio tag with spatial context to enable independent living for people with dementia, an application is built for AR-capable mobile phones. The learning outcomes from the project will help to improve the technology and design based on user validation, I will transfer the technology into a wearable AR product for people with dementia in the future.