12
Fuzzy reinforcement learning (RL) Autonomous Scent DISPENSER: for creating memorable customer experience of Long-Tail connected Venues 1 Chandrasekar Vuppalapati, 2 Anitha Ilapakurti, 3 Jayashankar Vuppalapati, 4 Santosh Kedari 1 Hanumayamma Innovations and Technologies, Inc. Email: [email protected] 2 Hanumayamma Innovations and Technologies, Inc. Email: [email protected] 3 Hanumayamma Innovations and Technologies Private Limited. Email: [email protected] 4 Hanumayamma Innovations and Technologies Private Limited. Email: [email protected] Abstract In today’s competitive business environment, creating memorable experiences and emotional connections with consumers is critical to win customer spending and long term brand loyalty. Brands want their customers to be in pleasing, subliminal scented environments. Even a few micro particles of scent can do a lot of marketing’s heavy lifting, from improving consumer perceptions of quality to increasing the number of visits. Hence, high roller venues such as Trump Towers and Caesars Palace of the World use digital connected intelligent scent disperser systems that deliver seamless olfactory experiences to connect with consumers on a deeper, emotional and personal level. The challenges, however, for venues with limited digital connected infrastructure and deficient intelligent systems are lack of engagement with patrons at a personal and emotional level and thus miss recurring business opportunities and sustained long-term brand loyalty. This research paper addresses the challenge by developing fuzzy Reinforcement Learning autonomous intelligent scent dispensers that bring connected intelligence to non-connected venues. The fuzzy reinforcement learning enables venue environment feedback loop that improves over customer experience. The paper presents prototyping solution design, as well as its application and certain experimental results. Keywords: Fuzzy Logic; Reinforcement Learning; Machine Learning; Internet Of Things; SARSA; Artificial Intelligent (AI) Systems; Long Tail; Sensors; iDispenser; Venue Analytics; 1.Introduction In today’s competitive business environment, creating memorable experiences and emotional connections with consumers is critical to win consumer spending and long-term brand loyalty [1]. In order to influence consumers, marketers have used several in-store venue stimuli: background music, venue lighting and ambient scent. The ambient scents have direct influence on improving the mood and increase various types of approach behavior of consumers [2] and thus have become dominant Brand promoting actor. With Ambient scent, brands want their customers to be in pleasing subliminal scented environments even a few micro particles of scent can do a lot of marketing’s heavy lifting, from improving consumer perceptions of quality to increasing the number of visits (Figure 1). Hence, high roller venues such as Trump Towers, New York, and Caesars Palace, Las Vegas, of the World use digitalconnected intelligent scent disperser systems that delivers seamless olfactory [4] experience to connect with consumers on a deeper emotional and personal level using data to create an unforgettable customer experience and five star Customer Relationship Management (CRM). [5] Figure 1: Scents work via physical environments resulting in emotion [3] International Journal of Pure and Applied Mathematics Volume 119 No. 12 2018, 12999-13009 ISSN: 1314-3395 (on-line version) url: http://www.ijpam.eu Special Issue ijpam.eu 12999

Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

Fuzzy reinforcement learning (RL) Autonomous Scent DISPENSER: for

creating memorable customer experience of Long-Tail connected Venues 1Chandrasekar Vuppalapati,

2Anitha Ilapakurti,

3Jayashankar Vuppalapati,

4Santosh Kedari

1Hanumayamma Innovations and Technologies, Inc.

Email: [email protected] 2Hanumayamma Innovations and Technologies, Inc.

Email: [email protected] 3Hanumayamma Innovations and Technologies Private Limited.

Email: [email protected] 4Hanumayamma Innovations and Technologies Private Limited.

Email: [email protected]

Abstract In today’s competitive business environment, creating memorable experiences and emotional

connections with consumers is critical to win customer spending and long term brand loyalty. Brands

want their customers to be in pleasing, subliminal scented environments. Even a few micro particles

of scent can do a lot of marketing’s heavy lifting, from improving consumer perceptions of quality to

increasing the number of visits. Hence, high roller venues such as Trump Towers and Caesars Palace

of the World use digital connected intelligent scent disperser systems that deliver seamless olfactory

experiences to connect with consumers on a deeper, emotional and personal level.

The challenges, however, for venues with limited digital connected infrastructure and deficient

intelligent systems are lack of engagement with patrons at a personal and emotional level and thus

miss recurring business opportunities and sustained long-term brand loyalty.

This research paper addresses the challenge by developing fuzzy Reinforcement Learning

autonomous intelligent scent dispensers that bring connected intelligence to non-connected venues.

The fuzzy reinforcement learning enables venue environment feedback loop that improves over

customer experience. The paper presents prototyping solution design, as well as its application and

certain experimental results.

Keywords: Fuzzy Logic; Reinforcement Learning; Machine Learning; Internet Of Things; SARSA;

Artificial Intelligent (AI) Systems; Long Tail; Sensors; iDispenser; Venue Analytics;

1.Introduction

In today’s competitive business environment, creating memorable experiences and

emotional connections with consumers is critical to win consumer spending and long-term brand

loyalty [1]. In order to influence consumers, marketers have used several in-store venue stimuli:

background music, venue lighting and ambient scent. The

ambient scents have direct influence on improving the

mood and increase various types of approach behavior of

consumers [2] and thus have become dominant Brand

promoting actor. With Ambient scent, brands want their

customers to be in pleasing subliminal scented

environments even a few micro particles of scent can do a

lot of marketing’s heavy lifting, from improving consumer

perceptions of quality to increasing the number of visits (Figure 1).

Hence, high roller venues such as Trump Towers, New York, and Caesars Palace, Las Vegas, of the

World use digitalconnected intelligent scent disperser systems that delivers seamless olfactory [4]

experience to connect with consumers on a deeper emotional and personal level – using data to create an unforgettable customer experience and five star Customer Relationship Management

(CRM). [5]

Figure 1: Scents work via physical environments

resulting in emotion [3]

International Journal of Pure and Applied MathematicsVolume 119 No. 12 2018, 12999-13009ISSN: 1314-3395 (on-line version)url: http://www.ijpam.euSpecial Issue ijpam.eu

12999

Page 2: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

1.1. The Long Tail1: issue at stake

Internet has changed the dynamics of the venue and hospitality market place [6]. Since the

development of the internet, all the attributes of the hospitality & venue industry has deeply

and continuously evolved: usage of connected networks, usage of consumer digital data &

mining of consumer preferences, exclusive recommendations gleaned from Web 2.0 & 3.0

data sources, Internet Of Things (IoT), wearable data sources and new business models

reshape the industry. This process has even

been sped up recently by the emergence of

disruptive innovations, from the evolutions of

web & mobile technologies to the Artificial

Intelligence (AI) platforms. The long tail is

used in this paper as a multitude of a small,

independent venue operators (Figure 2) with

limited digital connected infrastructure and

deficient intelligent systems can grasp and

synthesize this complexity.

This research paper addresses the challenge by developing fuzzy logic reinforcement-learning

autonomous intelligent scent dispenser that brings connected intelligence2 to non-connected and

limited compute processing venues. The paper presents prototyping solution design, as well as its

application and certain experimental results.

The structure of this paper is as follows. Section 2 discusses the basic concepts and methods about

Fuzzy Logic, Reinforcement Learning, Model Based, Model Free, Q Learning, Bluetooth Low Energy

Framework, Sensors and Fuzzy Control Process and Section 3 shows a case study. Conclusions and

future work are summarized in Section 4.

2. Understanding Fuzzy Logic & Reinforcement Learning for Intelligent Dispenser

2.1. nRF8001 Bluetooth Low Energy (BLE)

BLE is a low powered wireless personal area network that transmits small amount of

data over shorter distance as opposed to the class Bluetooth that consumes more power to transmit

large data over longer distance. The nRF8001 uses v4.0 BLE radio and features simple serial interface

that supports wide range of external application microcontroller. [7].

We have used Bluetooth not only for connectivity purposes but

also to identify number of people ―scanForPeripherals ― in a

venue. BLE scan provides a good view of foot traffic. For small

and medium venues, using Bluetooth to identify number of

people in a venue saves huge infrastructure cost (setting either

video cameras or infra red sensors for entry & exit strip

sensors).

2.2. Congestion and Number of Bluetooth Devices

There is linear relationship between number of people and number of Bluetooth devices in

avenue [8].Use count of Bluetooth Devices to Know the Number of People on New Year's Eve [9].

1 Long Tail in Tourism Industry - https://halshs.archives-ouvertes.fr/halshs-01248302/document

2 How sensory marketing applies to the hotel and restaurant industry in order to influence customer’s behavior in Thailand -

http://www.diva-portal.org/smash/get/diva2:426249/fulltext01

Figure 2: Long Tail

Figure 3: Count Number of People in New

Year's Eve [9]

International Journal of Pure and Applied Mathematics Special Issue

13000

Page 3: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

Bluetooth Central manager call for ―scanForDevices‖ is used to count number of devices in a room.

This popular method is being used to count number of people in New Year’s Eve (Figure 3).

2.3. Passive Infra Red Sensor (PIR)

PIR based motion detector is used to sense the

movement humans, animals and other living beings. For

instance, the Texas Instrument (TI) design TIDA –

010693 uses two lenses with different areas of detection

for each PIR Sensor: One lens for middle detection area

for a tall subject (like a human) and another for lower

detection for short subject (like a dog) (Figure 4).

The Crossing point distance (i.e., the certain distance the

beam of the middle and lower bean intersect)

X = 𝛿

tan 0.18 −tan 0.15

(1)

For TI Sensors𝛿𝑖𝑠𝑠𝑒𝑡𝑡𝑜10 cm. Therefore X = 2.96m;In Intelligent Dispenser case the PIR motion

index is a decimal number.

2.4. Microphone

Microphone data used to detect any human presence in a

venue. We have used Micro Electromechanical

Systems (MEMS) based A-weighted filter signals (Figure 5)

as these signals correspond to the human ear frequency

response4

2.5. Fuzzy Logic (FL)

Fuzzy Logic (FL) incorporates a simple rule based If X and Y Then Z approach to solving a control

problem than attempting to build a mathematical model. The FL

model is an empirically based relying on an operator's experience

rather than their technical knowhow of the system. For building

intelligent dispenser model, we have modeled Venue scent diffusion

system using predefined rules governing the behavior of venue

activity and developed activity indicator. Hence, Fuzzy Logic is the

most appropriate ML model [13].

Activity Indicator (Figure 6) is a combination variable that is

composite of # of people in a venue (based on the number of

people to BLE devices present in the venue) [8], the rate of

change of people, motion sensor value, the change or delta of motion sensor value [10] and Noise

Threshold Value, the delta of noise threshold [11]. Activity Indicator ranges from negative value to

positive value (denoting subsiding to active increase of human activity). In other words, negative

activity indicator indicates the number of people in the venue is in descending phase. The positive

value indicates the number of people in the venue is increased (more foot traffic).

3 TI PIR Sensor - http://www.ti.com/lit/ug/tiducv3b/tiducv3b.pdf 4 AN4426 Application Note:

http://www.st.com/content/ccc/resource/technical/document/application_note/46/0b/3e/74/cf/fb/4b/13/DM00103199.pdf/files/DM00103199.pdf/jcr:content/translations/en.DM00103199.pdf

Figure 4: TI PIR Sensor [10] - Fresnel Lenses View

Figure 5: A-weighted Filter Response [11]

Figure 6: Activity Algorithm

International Journal of Pure and Applied Mathematics Special Issue

13001

Page 4: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

Activity Indicator (t) =

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑃𝑒𝑜𝑝𝑙𝑒 𝑖𝑛 𝑡𝑕𝑒 𝑉𝑒𝑛𝑢𝑒 +

𝑚

𝑡=1

Δ 𝐶𝑕𝑎𝑛𝑔𝑒 𝑖𝑛 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑃𝑒𝑜𝑝𝑙𝑒 𝑖𝑛 𝑡𝑕𝑒 𝑉𝑒𝑛𝑢𝑒

Δ𝑡

+ max𝑡

𝑀𝑜𝑡𝑖𝑜𝑛 𝑆𝑒𝑛𝑠𝑜𝑟 𝑉𝑎𝑙𝑢𝑒 + Δ 𝑀𝑜𝑡𝑖𝑜𝑛 𝑆𝑒𝑛𝑠𝑜𝑟 𝑇𝑕𝑟𝑒𝑠𝑕𝑜𝑙𝑑

Δ𝑡

+ max𝑡

(𝑁𝑜𝑖𝑠𝑒 𝐿𝑒𝑣𝑒𝑙) + Δ 𝑁𝑜𝑖𝑠𝑒 𝐿𝑒𝑣𝑒𝑙 𝑇𝑕𝑟𝑒𝑠𝑕𝑜𝑙𝑑

Δ𝑡

(2)

2.5.1. Linguistic Variables

Fuzzy or Linguistic variables are the central to FL and were introduced by Professor LotfiZadeh in

1973. Succinctly put, Linguistic variables are language artifacts not numbers.

The control inputs, namely hardware sensors, in intelligent dispenser model are Fuzzy variables. For

instance, ―BLE Devices‖, ―Noise Level‖, ―Proximity Indicator – received signal strength indicator

(RSSI) Strength‖, ―Microphone5‖ and ―Passive Infra Red (PIR) Sensor

6‖ are modeled as Fuzzy

parameters. In FL error is just the difference, it can be thought of the same way as Fuzzy Variable

[12].

Another salient principle of FL is fuzzy variables (Table 1) are themselves adjectives that modify the

variable (e.g., "large positive" error, "small positive" error "zero" error, ―small negative" error, and

"large negative" error). As a minimum, one could simply have "positive", "zero", and "negative"

variables for each of the parameters. Additional ranges such as "very large burst of devices" and "very

small noise thresholds" are added to handle the BURST response or very nonlinear behavior.

Table 1: Fuzzy Variables

―N‖ = Negative Error or error-dot level

―Z‖ = Zero Error or error-dot level

―P‖ = Positive Error or error-dot level

―D‖ = ―Dispense‖ Output response

―-― = ―No Change‖ or Current Output

―ND‖ = No Dispense

2.5.2. Control Process

The goal of the control process is to model/control dispense

of Scent through the FL by detecting people activity in a

venue (as shown in the Figure 7): number of people in a

room (number of BLE devices), Motion Sensor detection and

Noise detected on the microphones.

2.5.2.1. Definitions:

INPUT# 1 System Status

- Error = Command-Feedback (Activity Indicator Eq.2)

- P = Increased Activity Index (Increase of venue occupancy or Noise or

Motion thresholds),

5Microphone - https://www.embedded.com/electronics-blogs/max-unleashed-and-unfettered/4441291/New-technology-provides-breakthrough-speech-recognition-in-noisy-

environments 6 PIR sensor - https://learn.adafruit.com/pir-passive-infrared-proximity-motion-sensor/how-pirs-work

Figure 7: Activity Indicator Control Process

International Journal of Pure and Applied Mathematics Special Issue

13002

Page 5: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

- Z = Muted or no Activity,

- N = Decreased Activity Index (Decrease of venue occupancy or Noise or

Motion thresholds)

INPUT# 2 System Status Error-dot = d (Error)/dt

- P = Getting Increased Activity Index

- Z = Not changing

- N = getting decreased activity index

OUTPUT Conclusion and System Response

Output:

- DISP = Dispense Scent

- = don't change

- NDISP = No Dispense

2.5.3. Matrix Rules

The fuzzy parameters of error (# BLE Devices with RSSI, PIR Sensor threshold and Microphone

Noise Levels) and error-dot (rate-of-change-of-error) were modified by the adjectives "negative",

"zero", and "positive" (Table 2).

In the Figure 9, the linguistic parameter ―BLE Devices‖ indicates number of devices within a venue

(negative value indicates people leaving the proximity of the iDispenser & positive implies entering

into proximity of the iDispenser).

Table 2: Rules

1. IF CMD_AI = N AND d(CMD_AI)/dt = N THEN OUTPUT = NDISP

2. IF CMD_AI = Z AND d(CMD_AI)/dt = N THEN OUTPUT = NDISP

3. IF CMD_AI = P AND d(CMD_AI)/dt = N THEN OUTPUT = DISP

4. IF CMD_AI = N AND d(CMD_AI)/dt = Z THEN OUTPUT = NDISP

5. IF CMD_AI = Z AND d(CMD_AI)/dt = Z THEN OUTPUT = NC

6. IF CMD_AI = P AND d(CMD_AI)/dt = Z THEN OUTPUT = DISP

7. IF CMD_AI = N AND d(CMD_AI)/dt = P THEN OUTPUT = DISP

8. IF CMD_AI = Z AND d(CMD_AI)/dt = P THEN OUTPUT = DISP

9. IF CMD_AI = P AND d(CMD_AI)/dt = P THEN OUTPUT = DISP

2.5.4. Matrix

The fuzzy parameters of error (Activity

Indicator) and error-dot (rate-of-change-of-

activity_Indicator) were modified by the

adjectives "negative", "zero", and

"positive". The following diagram provides

a 3-by-3 matrix (Figure 8) input and output

outcomes. The columns represent "negative

error", "zero error", and "positive error"

inputs from left to right. Please note:

―DISP‖ results into dispensing of the scent. NDISP – no Dispensing required and NC implies no

Change to the condition.

2.5.5. Scent Diffusion Membership Function

Figure 8: Matrix

International Journal of Pure and Applied Mathematics Special Issue

13003

Page 6: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

The membership function (Figure 9) is a graphical representation of the magnitude of participation of

each control input (#BLE Devices, Microphone

Thresholds and PIR Sensor Thresholds). It associates a

weighting with # BLE Devices, Microphone and PIR

Thresholds inputs, define functional overlap between

inputs (BLE Devices to Noise Level on Microphone)

(BLE Devices to PIR Thresholds) and (Microphone

Noise Levels to PIR Thresholds), and ultimately

determines an output response of dispenser (Scent

diffusion or not).

2.6. Reinforcement Learning (RL) [14]

RL [14] is learning process of an agent to act in

order to maximize its rewards. In RL learning is

by trial-and-error to map situations to actions

that maximize a numerical reward. The

following figure 10 has standard RL

architecture.

The agent is defined as a scent-dispensing algorithm, the action is the command to dispense or not

dispense, the environment is the target object, in this case activity indicator that is formed in a closed

venue, and the reward is performance improvement after applying the action. The goal of RL is to

choose an action in response to a current stat, based on activity indicator, to maximize the reward,

dispense scent or not. Generally, RL approaches learn estimates of the initialized Q Values Q(s,a),

which maps all system states s to their best action a. We initialize all Q(s,a) and during learning,

choose an action a for state s based on ∈ greedy policy to target platform. Then, we observe the new

state s’ and reward r and update Q-value of the last state-action pair Q(s,a), with respect to the

observed outcome state (s’) and reward r. However, in our case, we have used State-Action-Reward-

State-Action (SARSA).

2.6.1. Model Free vs. Model based

The Model stands for the simulation of the dynamics of the environment (dispense based on activity

indicator Eq. 2). That is, the model learns the transition probabilityT (𝑠1

(𝑠0,𝑎)) from the pair of current

state s0 and action a to the new state s1 (figure

11) [15]. To derive an action, an agent uses a

policy that aims to increase the future rewards

(indefinitely or long run environments). If the

transition probability successfully learned, the

agent will know how likely to enter a specific

state given current state and action. The model-

based approach becomes impractical if state

space and action space grows (S * S * A).

Trial-and-error process enables model-free systems to update knowledge. Model free-based systems

are more practical as the interactions of environment and agent are unknown in many physical

systems [16].

Figure 9: Member Functions [12]

Figure 10:The standard architecture of RL Algorithm

Figure 11: on-policy vs. off-policy agent

International Journal of Pure and Applied Mathematics Special Issue

13004

Page 7: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

2.6.2. On-policy and Off-policy

An on-policy agent, our current dispenser in case, learns the value based on an action derived from the

current policy and off-policy agent learns based on the action a* obtained from another policy (a

greedy policy in case of Q-learning).

2.6.3. Fuzzy SARSA Learning (FSL)

By using RL instead of using static matrix rules (Table 2) the performance of target environment can

be captured after applying each state decision. Since, the effectiveness of scenting can be fed back

from Venue managers by providing the input via mobile app. The combination of Fuzzy Logic

controller with SARSA is explained below:

1) Initialize the q-values: Each member of q-value

table assigned to a certain rule that describes some state-

action pairs and is updated during the learning

process. Please see table 2 and Figure 10. It can tell

us the performance of taking the action by taking

into account the reward value.

2) Select an action: to learn from the

system, we use Table 3 to gain the knowledge of the

system. This is called exploration/exploitation

strategy [14],∈ greedy is standard exploration policy.

3) Calculate the control action inferred by fuzzy logic controller: the fuzzy output is a

weighted average of the consequences of the rules:

𝑎 = 𝜇𝑖 𝑥 𝑥 𝑎𝑖𝑁

𝑖−1

Where N is number of rules (Scent dispenser N = 12),

𝜇𝑖 𝑥 𝑖𝑠 𝑡𝑕𝑒 𝑓𝑖𝑟𝑖𝑛𝑔 𝑑𝑒𝑔𝑟𝑒𝑒 𝑜𝑓 𝑡𝑕𝑒 𝑟𝑢𝑙𝑒 𝑓𝑜𝑟 𝑡𝑕𝑒 𝑖𝑛𝑝𝑢𝑡

𝑠𝑖𝑔𝑛𝑎𝑙 𝑥 𝑎𝑛𝑑 𝑎𝑖 𝑖𝑠 𝑡𝑕𝑒 𝑐𝑜𝑛𝑠𝑒𝑞𝑢𝑒𝑛𝑡 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑓𝑜𝑟 𝑡𝑕𝑒

𝑠𝑎𝑚𝑒 𝑟𝑢𝑙𝑒

4) Approximate the Q-function from the current q-values and the firing level of the

rules:

Q (s,a) = ( 𝑁𝑖=1 𝜇𝑖 𝑠 𝑥 𝑞[𝑖, 𝑎𝑖])

The action-value function Q (s,a) tells us how desirable it is to reach state s by taking action a by

allowing to take the action a many times and observe the return value.

5) Calculate the reward: the reward is calculated on following criteria: a) proper

dispensing of the scent.

6) Calculate the value of the new state s’:

Table 3: q-values table

International Journal of Pure and Applied Mathematics Special Issue

13005

Page 8: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

V (s’) = 𝑢𝑖 𝑠′ . max(𝑞 𝑖, 𝑎𝑘 )𝑁𝑖=1 where max(q[I,ak]) is the maximum of the q-values

applicable in the state s’.

3.Case Study

We have tested and deployed Fuzzy Logic (FL)

based

iDispenser(http://www.hanuinnotech.com/venueanalytics.html)

We have several Venue Users that are using the system and providing valuable venue and usage data

& improved CRM (Figure 12).

4. Conclusion

This paper presented a novel approach to creatingFuzzy Logic based reinforcement learning

intelligent scent dispenser that bridges the chasm between limited connected network and availability

of compute power to perform intelligent venue operations. As witnessed through our market reach, we

strongly conclude that there are still parts of the world that are not touched and blessed through the

data insights based intelligent devices – chiefly not have a connected network. With this research

paper and with the development of the device, we brought connected intelligence to non-connected

venues. We strongly believe that by implementing Reinforcement Learning FL based dispensing has a

huge positive impact in healthcare, travel, entertainment and other captive services business verticals.

Acknowledgements

We sincerely thank you the engineering teams, the marketing teams and the product management of

Hanumayamma Innovations and Technologies, Inc., and Hanumayamma Innovations and

Technologies Private Limited for their valuable contribution to product deployments in India and

USA and their active involvement in collecting customer data

References

Journals, Books and Conferences

[1] Martín-Ruiz, David, Carmen Barroso-Castro, and Isabel Mª Rosa-Díaz. "Creating customer value

through service experiences: An empirical study in the hotel industry." Tourism and Hospitality

Management 18, no. 1 (2012): 37-53.

[2] HuaMeng, ―THE EFFECTS OF AMBIENT SCENT ON CONSUMER BEHAVIOR: A REVIEW OF

THE LITERATURE‖, Aug. 2016, Accessed on: Feb 25, 2018,

https://etd.ohiolink.edu/!etd.send_file?accession=kent1468275876&disposition=inline

[3] Constant Berkhout, Retail Marketing Strategy: Delivering Shopper Delight, Publisher: Kogan Page

(November 28, 2015), ISBN-13: 978-0749476915

Figure 12: iDispenser Case Study

International Journal of Pure and Applied Mathematics Special Issue

13006

Page 9: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

[4] Making Sense of Scents: Smell and the Brain, Society for Neuroscience,

http://www.brainfacts.org/Sensing-Thinking-Behaving/Senses-and-Perception/Articles/2015/Making-

Sense-of-Scents-Smell-and-the-Brain, Creation Date: 27 Jan 2015 | Review Date: 27 Jan 2015

[5] David Norton, The High Roller Experience: How Caesars and Other World-Class Companies Are

Using Data to Create an Unforgettable Customer Experience, Publisher: McGraw-Hill Education; 1

edition (November 7, 2017), ISBN-13: 978-1259862953

[6] Christian Longhi and Sylvie Rochhia, Long Tails in the Tourism Industry: Towards Knowledge

Intensive Service Suppliers, https://halshs.archives-ouvertes.fr/halshs-01248302/document Access

Date: February 25, 2018

[7] Ilapakurti, J. S. Vuppalapati, S. Kedari, S. Kedari, C. Chauhan and C. Vuppalapati, "iDispenser — Big

Data Enabled Intelligent Dispenser," 2017 IEEE Third International Conference on Big Data

Computing Service and Applications (BigDataService), San Francisco, CA, 2017, pp. 124-130. doi:

10.1109/BigDataService.2017.53

[8] Streitz, Norbert, Markopoulos, Panos, Distributed, Ambient and Pervasive Interactions, Publisher:

Springer International Publishing, Softcover ISBN: 978-3-319-58696-0

[9] Cooking Hacks, Counting Bluetooth Devices to Know the Number of People on New Year's Eve,

https://www.cooking-hacks.com/blog/counting-bluetooth-devices-to-know-the-number-of-people-on-

new-years-eve/, Access date: February 28, 2018

[10] Texas Instrument, Advanced Motion Detector Using PIR Sensor Reference for False Trigger

Avoidance, Source: http://www.ti.com/lit/ug/tiducv3b/tiducv3b.pdf, Access date: February 28, 2018

[11] STMicroelectronics, AN4426 Application Note,

http://www.st.com/content/ccc/resource/technical/document/application_note/46/0b/3e/74/cf/fb/4b/13/

DM00103199.pdf/files/DM00103199.pdf/jcr:content/translations/en.DM00103199.pdf , Access date:

February 28, 2018

[12] Rajesh Kumar, Fundamental of Artificial Neural Network and Fuzzy Logic, Publisher: Laxmi

Publications (January 1, 2010) ISBN-13: 978-8131807101

[13] AnupamShukla, RituTiwari, Rahul Kala, Real Life Applications of Soft Computing, Publisher: CRC

Press; 1 edition (May 21, 2010), ISBN-13: 978-1439822876

[14] R.S. Sutton and A.G. Barto, ―Reinforcement Learning: An Introduction (Adaptive Computation and

Machine Learning)‖, A Bradford Book; 1st Edition edition (March 1, 1998), ISBN-13: 978-

0262193986

[15] Steeve Huang, ―Introduction to Various Reinforcement Learning Algorithms. Part I (Q-Learning,

SARSA, DQN, DDPG)‖, Source: https://towardsdatascience.com/introduction-to-various-

reinforcement-learning-algorithms-i-q-learning-sarsa-dqn-ddpg-72a5e0cb6287 Access Date: March 25,

2018

[16] P. Jamshidi, A. M. Sharifloo, C. Pahl, A. Metzger and G. Estrada, "Self-Learning Cloud Controllers:

Fuzzy Q-Learning for Knowledge Evolution," 2015 International Conference on Cloud and Autonomic

Computing, Boston, MA, 2015, pp. 208-211.doi: 10.1109/ICCAC.2015.35,

URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7312157&isnumber=7312127

Authors Biography

Chandrasekar Vuppalapati: Chandra is a seasoned Software IT Executive with diverse experience in

Software Technologies, Enterprise Software Architectures, Cloud Computing, Big Data

Business Analytics, Internet Of Things (IoT), and Software Product & Program Management.

Chandra held engineering and Product leadership roles at GE Healthcare, Cisco Systems, St.

Jude Medical, and Lucent Technologies, a Bell Laboratories Company. Chandra teaches

Software Engineering, Mobile Computing, Cloud Technologies, and Web & Data Mining for

Masters program in San Jose State University.

Anitha Ilapakurtihas wide variety of Software development project management experience in HealthCare,

Banking, Data Modeling, and Real Time Application systems. She has pioneered in software

development processes for the Business Agile systems and interacted with Fortune 500

business leaders in communicating and sharing her visions and product development.From her

International Journal of Pure and Applied Mathematics Special Issue

13007

Page 10: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

humble beginning as a programmer and analyst at a software development center in the Silicon Valley, Anitha

has handled several development and project management roles in the software application development life

cycles. Her most notable are: Application Architect and Business Manager before becoming a software

entrepreneur. She holds a B.SC, M.SC in Computer Sciences from Nagrjuna University, India, and Master of

Sciences in Software Engineering from San Jose State University, USA.

Santosh Kedarileads Indian operation in the capacity of Director of Software Services. Santosh co-founded

Sanjeevani Electronic Health Records (EHR) and passionate about democratizing EHR for

greater good. He conducts health camps in Southern India and plays a pivotal role in Sales

operation of Dairy Analytics. Apart from work, Santosh enjoys movie watching and wants to

produce healthcare documentaries and knowledge sharing small movies.

Jayashakar Vuppalapatiis a seasoned Executive with diverse experience in Software Technologies, Enterprise

Software Architectures, Big Data Business Analytics, Internet Of Things (IoT), Engineering

Management and new and innovative product designs. Jay held engineering and Product

leadership roles at FedEx, Anthem, Intel, HCL technologies, Health Net and other major

corporations. Currently, Jay holds CTO role and leads South Asian business markets

International Journal of Pure and Applied Mathematics Special Issue

13008

Page 11: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

13009

Page 12: Fuzzy reinforcement learning (RL) Autonomous Scent ... · and continuously evolved: usage of connected networks, u sage of consumer digital data & mining of consumer preferences,

13010