Autonomous Vehicle System based on Law and Case Law using

Qualitative Representation

Satoshi Nishimura1, Asaki Iwata2, Miwa Kurokawa2, Shun-ya Maruta2,

Daisuke Kaji2, Shinji Niwa2, Takuichi Nishimura1, Yo Ehara1 1 National Institute of Advanced Industrial Science and Technology, Tokyo, Japan

{satoshi.nishimura, takuichi.nishimura, y-ehara}@aist.go.jp 2 DENSO Corp.

{ASAKI_IWATA, MIWA_KUROKAWA, SHUNYA_MARUTA, DAISUKE_KAJI, SHINJI_NIWA}@denso.co.jp

Abstract

Recently, autonomous vehicle technology has been develop-ing quickly. Under the circumstances, we specifically ex-amine the problem of how an autonomous system under-stands traffic rules. We developed a knowledge model for safe driving using traffic statute law, case law, and incident movies.

Introduction

Autonomous vehicle systems are expected to reduce

traffic accidents, congestion, and fuel consumption and to

improve comfort and convenience. Technology related to

autonomous vehicles, such as that of Waymo,1 has been

developed by Alphabet Inc. Realizing such an autonomous

vehicle system requires numerous and diverse technologies

such as data collection for real environments, machine

learning for driving actions, and improved driving intelli-

gence (driving environment cognition, perception, etc.). In

addition, it is important to visualize reasoning processes by

which autonomous vehicle systems determine driving ac-

tions (Zhao et al., 2016).

Human drivers determine their driving actions according

to their cognition and traffic statute law. Autonomous ve-

hicle systems should drive a vehicle according to traffic

statute laws to ensure safe driving. However, traffic statute

laws are written for human beings. It is difficult for auton-

omous vehicle systems to understand laws expressed that

way. We regard that point as the key issue that hinders

autonomous vehicles’ understanding of traffic statute laws.

For example, traffic statute laws in Japan have the follow-

ing text. “(Safe Driving Obligations) Article 70 The driver

of a vehicle, etc. shall operate its equipment, including but

1 https://waymo.com/

not limited to its steering wheel and brakes, in a consistent

manner and shall drive the vehicle, etc. at a speed and in a

manner that pose no hazard to others taking into considera-

tion such situations as roads, traffic and the vehicle, etc.”

(Japanese Law Translation) Such consistent operation of

equipment and speed depend on the situation. Therefore,

we claim that the traffic statute laws are too ambiguous for

autonomous vehicle systems. To realize safe autonomous

vehicles, it is necessary to make the vehicles understand

the qualitative aspects of traffic statute laws. We regard the

qualitative perspective as important because human beings

understand laws qualitatively, not quantitatively. Therefore,

qualitative information is important for mutual communi-

cation between autonomous vehicle systems and human

beings.

Under the circumstances, we aim to develop a safe au-

tonomous vehicle system based on the following qualita-

tive knowledge base.

• Qualitative spatial representation for sensory data

• Traffic statute law and case law for decision making

• Driving action model

As described in this paper, we report a pilot study for safe

autonomous vehicle system. First, we explain the future

prospects of the our autonomous vehicle systems. Second,

we described driving actions using qualitative representa-

tion. The described action model will help us to clarify

what knowledge should be described and how to align

above qualitative knowledge bases for safe autonomous

driving system.

Future Prospects of Our Autonomous Vehicle

Overview of the Proposed System

Figure 1 presents an overview of the proposed autono-

mous vehicle system. Features of the system include a

qualitative knowledge base. The system senses the external

environment using sensors, maps, and communications

among vehicles in the cognition phase. The collected data

are then interpreted to qualitative spatial representations

such as “One pedestrian is standing diagonally forward

right. A traffic signal is in front of the car.” In the decision

phase, the autonomous vehicle systems determine the driv-

ing action based on traffic statute law and case law. We

divide the traffic law into statute law and case law because

of the granularity of the information. We describe related

details in section “Traffic Statute law and Case Law”. The

system also predicts risks of traffic environments based on

collected data in cognition phase and then determines the

actions. The system chooses actions based on the driving

action model. The model is built in a goal-oriented manner.

Therefore, the system chooses its actions rationally.

The key notion of the proposed system includes a quali-

tative knowledge base for communication between the

autonomous vehicle system and human beings. Therefore,

the system is useful not only for autonomous vehicle sys-

tems but also as a training system for novice drivers or

drivers who have caused accidents. The system ascertains

the rationale of driving actions according to such a

knowledge base.

Knowledge Base for the Proposed System

Qualitative Spatial Representation of Sensory Data

To represent the data from external environment, we

employ the method Qualitative Spatial Reasoning and Rep-

resentation (QSR) (Cohn, 2008), which is a method to rep-

resent spatial information in a qualitative manner, making

it comprehensible even for human beings. We will also

employ it for connection to other qualitative knowledge

bases.

Traffic Statute law and Case Law

We will transform traffic statute law to a machine-

readable but qualitative form. We regard traffic law as

comprising statute law and case law.

Followings are the reasons why we use case law. Other

autonomous vehicle systems are also developed by consid-

ering safe driving according to traffic statute laws. Howev-

er, other systems merely handle limited information related

to specific traffic statute laws such as speed limits and traf-

fic signs. In the traffic statute laws, some ambiguous texts

exist. For example, Japanese traffic statute laws include the

following text. “(Safe Driving Obligations) Article 70 The

driver of a vehicle, etc. shall operate its equipment, includ-

ing but not limited to its steering wheel and brakes, in a

consistent manner and shall drive the vehicle, etc. at a

speed and in a manner that pose no hazard to others taking

into consideration such situations as roads, traffic and the

vehicle, etc.” (Japanese Law Translation) We aim to pro-

cess not only information such as traffic signs and speed

limitations but also other ambiguous information in the

statute law. To handle such ambiguous information, we use

case law because of its granularity. The statute laws in-

clude abstract information, but case law includes more

fine-granular information.

However, case laws may be contradictory with each oth-

er and part of them has too concrete information for auton-

omous vehicles. To avoid the issue, we employed a book in

which law experts categorize traffic accidents into 338

categories, published by Hanrei Times Co., Ltd. (Hanrei

Times, 2014) Figure 2 shows the spatial information and

Autonomous Driving

Advice for Driving

Driving by Human

Cognition

3.5m

20.1m

4.2m

6.0m6.2m

Sensor data

Inter-vehicle communication

Map

Pattern recognition

DecisionPotential hazard

prediction

Keeping traffic laws

Driving Action

Knowledge BaseSpatial Information• One pedestrian stands diagonally forward right.

• One car stops diagonally forward left.

• Cross-walk is in front of me.• There are downtown near here, etc.

Traffic rules• Traffic laws• Case laws

Ontology based Action Knowledge• Action for turning right• Action for prevention of

collision, etc.

QSR* Traffic rules

Driving Action

Knowledge

Data transition using KB

*QSR:Qualitative Spatial Representation and Reasoning

Figure 1: Future prospect of autonomous vehicle system

from the perspective of safe driving.

comparative negligence ratio of one category in the book.

The category was built based on case laws. Moreover,

these categories are used in court in Japan as complement

information to judge. Thus, the use of this book's categori-

zation enables us to avoid contradictory case laws and

make decisions considering the information more concrete

than statute laws.

Case law presents advantages over statute law in terms

of its frequency of revision. Statute laws are revised once

every one or two years, but case law is produced continual-

ly. If the autonomous vehicle system can accommodate the

case law, then the system can follow recent situations.

Driving Action Model for Autonomous Vehicle

The driving action model describes the action that driv-

ers including human and autonomous vehicles choose and

take in a certain situation. It also provides structured

knowledge related to driving. The model is built in a man-

ner such that the driver cannot cause a traffic accident if

the driver obeys the model. The model will be connected to

QSR and traffic statute laws.

For this study, we used Convincing Human Action Ra-

tionalized Model (CHARM) (Nishimura et al., 2013) as

driving action model. We use knowledge explication

methodology (Nishimura et al., 2017) as the method of

knowledge building. Knowledge explication methodology

helps a knowledge holder build structured knowledge. It is

suited to the domain which presents knowledge depending

on a specific site or person. Regarding driving knowledge,

it also depends on the driver, so we choose knowledge ex-

plication methodology.

Built-up Driving Action Model

We accumulated “turning right at the intersection”

knowledge because it is a risky action according to statis-

tics (ITARDA, 2014). The Institute for Traffic Accident

Research and Data Analysis reported that the number of

accidents involving vehicles and humans (including pedes-

trians and bicycles) is not decreasing compared to the

numbers of other accidents such as accidents among cars

and self-inflicted injury accidents. Accidents involving

vehicles and humans also entail deadly risks. Zhao et al.

also specifically examine the driver behavior at rounda-

bouts, a type of intersection (Zhao et al., 2017). Therefore,

we specifically examined building the knowledge to reduce

accidents between vehicles and humans. The origin of the

knowledge is an unsafe incident database provided by To-

kyo University of Agriculture and Technology (TUAT).

We chose 36 incidents that occurred when the car turned

right at an intersection with pedestrians or bicycles.

Figure 3 portrays all knowledge of “turning right at an

intersection.” It is structured in a goal-oriented manner.

Each oval node shows an action which should be taken for

Turn right at an intersection

Prevent collision with

something

Prevent collision with standing

things

Prevent collision with moving things

Stop

way

way

way

Go to the intersection

Go away from the

intersection

Increase the

speed

Return the steering wheel

way

Accident pattern 14

risk

Accident pattern 12

Accident pattern 20

Decrease the speed

Turn the steering

wheel to right

way

Operate hand-brake

Operate foot-brake

way using foot-brake

way using hand-brake

Collide to something

safe

way of collision

Accident pattern 1

risk

Accident pattern 2

Condition: likely to collide

Inform

way

Turn on winkers

way

Move to a location where a car which turns right at

opposite side of the crossroad can look at

Get information(Recognize)

Look locations of

moving things

way of direct-viewing

See far away

Look states of moving

things

Keep a distance from the

moving things

way

Look states of moving objects on cross-walk

Look states of moving objects on intersection

Look states of moving objects

on road

way

Look locations of moving objects on cross-walk

Look locations of moving objects on intersection

Look locations of moving objects

on road

way

Avoid a blind

situation

way

Switch to high-beam

Condition: In dark

Look at the traffic signal

way of obeying traffic rules

Condition: the moving things obey

the traffic rules.

Condition: There is

traffic signal.

Accident pattern 135・・・ ・・・

Figure 4: Action for prevention of collisionFigure 5: Go through the intersection (go to and go away from the intersection)

Action A

Action B

Way A

Action C

Way B

Action D

Action E

Way C

Action F

Condition: A

Risk A

risk

Risk B

• These actions are operated in order from left to right.

• Both actions are necessary to achieve their goal.

• Either way is necessary to achieve the goal

Goal

Fine-granular action

Legends

• These actions can be operated independent with order.

• Both actions are necessary to achier the goal.

Figure 3: Entire driving action knowledge about turning right at an intersection.

Pedestrian started to cross a road during red signal.Car went into an intersection during blue signal.

Blue

BlueRed

Red

Red

Red

Comparative negligence ratio

At night

Main streetCrossing near car / stop on the road, going back

At residence / shopping district

Pedestrian is child / elderly

Pedestrian is toddler / handicapped

Group of pedestrians

Significant negligence

Serious negligence

No difference between driveway and sideway

Detailed description of the pattern

Figure 2: Spatial information and comparative negligence

ratio (Hanrei Times, 2014)

safe driving. The link from an upper-node to a lower-node

denotes the way to achieve the upper-node (goal action).

The lower-node and nodes connected horizontally are the

group of actions which are needed to achieve the goal ac-

tion. The ways are alternatives to achieve the same goal.

The horizontal links are classified into line and arrowhead.

The line denotes that the order of associated actions does

not matter. The arrowhead denotes that the order of the

connected actions is from left to right.

In this case, the topmost goal is “turn right.” The action

is achieved by following three actions: “prevent collision

with something” (see Figure 4), “go to the intersection”,

and “go away from the intersection” (see Figure 5). “Pre-

vent collision with something” can be interpreted as a goal

of fine-granular actions, “prevent collision with standing

things” and “prevent collision with moving things.” We

describe more details of “prevent collision with moving

things.” “Switching to high-beam” is a fine-granular action

in this tree. This action is performed in a “in dark” situa-

tion to achieve a goal, “see faraway.” In addition to that,

the risks are described in the tree. When the driver per-

forms “go to the intersection”, there are some risks de-

scribed as red rectangles (see rectangle in Figure 5). These

numbers denote an accident pattern extracted from the

book of Hanrei Times Co., Ltd. [Hanrei 2014]. This infor-

mation can be useful to choose driving actions for autono-

mous vehicle systems.

We describe a role of risks in the action model. When

modeling, one of the authors built the action model based

on unsafe-incident movies and his common sense. After

building the model, he linked the action model with acci-

dent patterns (we call it “risk” in the action model.) deter-

mined in (Hanrei Times, 2014). Thanks to the decomposi-

tion to each element of actions, it is easy to link the acci-

dent patterns with the corresponding action. On the other

hand, during operation of the system, the system checks

external situation and compares with the action model.

When the system found the action which the vehicle

should take, the system seeks risks by following the links

from the found action. If the action has high risks, then the

system seeks other action to accomplish its goal.

Prevent collision with

something

Prevent collision with standing

things

Prevent collision with moving things

Stop

way

way

Condition: likely to collide

Inform

way

Turn on winkers

way

Move to a location where a car which turns right at

opposite side of the crossroad can look at

Get information(Recognize)

Look locations of

moving things

way of direct-viewing

See far away

Look states of moving

things

Keep a distance from the

moving things

way

Look states of moving objects on cross-walk

Look states of moving objects on intersection

Look states of moving objects

on road

way

Look locations of moving objects on cross-walk

Look locations of moving objects on intersection

Look locations of moving objects

on road

way

Avoid a blind

situation

way

Switch to high-beam

Condition: In dark

Look at the traffic signal

way of obeying traffic rules

Condition: the moving things obey

the traffic rules.

Condition: There is

traffic signal.

Figure 4: Part of the action depicted in Figure 3 (Action for prevention of collision).

Go to the intersection

Go away from the

intersection

Increase the

speed

Return the steering wheel

way

Accident pattern 14

risk

Accident pattern 12

Accident pattern 20

Decrease the speed

Turn the steering

wheel to right

way

Operate hand-brake

Operate foot-brake

way using foot-brake

way using hand-brake

Collide to something

safe

way of collision

Accident pattern 1

risk

Accident pattern 2

Accident pattern 135・・・ ・・・

Figure 5: Part of the action shown in Figure 3 (go through the intersection).

How the Proposed System works

Finally, we present an image of how the proposed sys-

tem works (see Figure 6). The figure shows three situations,

(a) 6 s before the incident occurs, (b) 2 s before the inci-

dent occurs, and (c) the exact time when the incident oc-

curs. At each situation, the sensor collects the data of ex-

ternal situation and translates to qualitative representation.

At 6 s before the incident occurs, there is a traffic signal

with a blue signal turned on in front of the autonomous

vehicle. The system predicts risks using the accumulated

data and driving action knowledge including traffic statute

laws. This situation presents no risks. Therefore, the sys-

tem merely shows “no risks are detected.” The system de-

termines the actions which are hatched in the red screen in

Figure 6 according to the sensor collected data and

knowledge. The system performs the actions “go into the

intersection” at 2 s before the incident occurs. The sensor

detects “pedestrians standing on the cross-walk diagonally

forward right.” Based on its collected data, the system pre-

dicts the risk of “pedestrians running out into the road.”

Then the system chooses the actions “Care about the pe-

destrian. Turn right.” The red hatch, which is described in

the bottom of Figure 6(b), denotes actions to “prevent col-

lision” and emphasize the operation by the dense of color.

The deep red, rather than light red, denotes more important

actions. Finally, the system predicts the risk of “collision

with the pedestrians” and then determines the driving ac-

tion: “stop.”

Related work

Several studies related to this research have been con-

ducted. They are broadly divisible into two types: autono-

mous robot and autonomous vehicles with qualitative rep-

resentation.

Somani et al. presents an interface for collaborative tasks

involving a worker and an industrial robot (Somani et al.,

2013). The robot can learn the assembly plan through the

interface. The worker teaches the robot. The robot recog-

nizes the scenes before and after teaching. The robot ascer-

tains the taught plan with semantic description. Finally, the

worker and the robot can collaborate.

Rizaldi et al. also tackled the formalization of traffic

rules (Rizaldi et al., 2015). They formalize five rules from

the Vienna Convention on Road Traffic for autonomous

vehicles using first-order logic. Thereafter, they try to for-

malize the ambiguous rules, but they do not use case law to

concretize the rules.

As for modeling tasks, Erol et al., provide Hierarchical

Task Network (Erol et al., 1994a, 1994b). HTN, which is

acronym for hierarchical task network, was developed for

automatic planning. A network consists with some tasks

with their decomposition. The decomposition is done ac-

cording to satisfying constraints to accomplish its goal.

The network has primitive tasks, which is executable by

machine. We employed similar goal-oriented action model

CHARM, which is acronym for convincing human action

rationalized model (Nishimura et al., 2013). CHARM was

developed for human to understand their work processes

with goals. Modeling processes based on CHARM is like

HTN but it does not have primitive tasks so the decomposi-

tion of process is done recursively.

Decision to drive

Condition: Traffic signal is blue.Risk: No detectionActions: Decrease the speed and go to

the intersection.

-6s

Decided actions

Detected risks

Blue signal

(a) Decision making at 6 s before the incident

Decision to driveCondition: Traffic signal in front of the car is blue.

Traffic signal for pedestrians is red. Two pedestrians walk diagonally forward right of the cross-walk.

Risk: The pedestrian might run out to the road.Actions: Care about the pedestrians. Turn the

steering wheel to right.

-2s

Decided actions

Detected risks

Blue signal for car

Red signal for pedestrians

(b) Decision making at 2 s before the incident

Decision to driveCondition: Traffic signal for pedestrians

is red. The pedestrians are walking on the cross-walk.

Risk: Collision with the pedestrians.Actions: Stop to prevent the collision.

0s

Decided actions

Detected risks

Pedestrians

Red signal for pedestrians

(c) Decision making at the exact time when the incident oc-

curred

Figure 6: Image of the proposed system function.

Conclusion and Future Work

We proposed future prospects related to a safe autono-

mous vehicle system based on a qualitative knowledge

base, knowledge base requirements, and development of

the prototype. Finally, we presented an image of how the

proposed system works.

Future work will be undertaken to build a qualitative

knowledge base. Especially, we will try building qualita-

tive spatial representation for sensory data. We will also

tackle connections between the QSR and driving action

model.

Acknowledgements

This paper is partly based on results obtained from "Fu-

ture AI and Robot Technology Research and Development

Project" commissioned by the New Energy and Industrial

Technology Development Organization (NEDO).

References

Cohn, A. G., Renz, J. 2008. Chapter 13 Qualitative Spatial Repre-sentation and Reasoning, van Harmelen, F., Lifschitz, V., Porter, B. eds. Foundations of Artificial Intelligence Vol.3, Handbook of Knowledge Representation, pp.551-596.

Erol, K., Hendler, J., Nau, D. S. 1994. HTN Planning: Complexi-ty and Expressively, The Twelfth National Conference on Arti-ficial Intelligence (AAAI-94), pp. 1123-1128.

Erol, K., Hendler, J., Nau, D. S. 1994. UMCP: A Sound and Complete Procedure for Hierarchical Task-Network Planning, the Second International Conference on Artificial Intelligence Planning Systems (AIPS-94), pp. 249-254.

Institute for traffic accident research and data analysis. 2014. statistics of transportation accidents. http://www.itarda.or.jp/english/

Japanese Law Translation, Road Traffic Act. http://www.japaneselawtranslation.go.jp/law/detail/?ft=1&re=01&dn=1&x=0&y=0&co=01&ia=03&ky=%E9%81%93%E8%B7%AF%E4%BA%A4%E9%80%9A%E6%B3%95&page=11, 2017.05.18 accessed.

Nishimura, S., Kitamura, Y., Sasajima, M., Williamson, A., Ki-noshita, C., Hirao, A., Hattori, K., Mizoguchi, R. 2013. CHARM as Activity Model to Share Knowledge and Transmit Procedural Knowledge and its Application to Nursing Guide-lines Integration, Journal of Advanced Computational Intelli-gence and Intelligent Informatics, Vol.17, No. 2, pp.208-220.

Rizaldi, A., and Althoff, M. 2015. Formalising traffic rules for accountability of autonomous vehicles. 2015 IEEE 18th Inter-national Conference on Intelligent Transportation Systems (ITSC), pp. 1658-1665.

Smart Mobility Research Center, Drive recorder dataset. http://web.tuat.ac.jp/~smrc/drcenter.html, 2017.05.18 accessed (in Japanese)

Somani, N., Dean-Leon, E., Cai, C., and Knoll, A. 2013. Scene perception and recognition for human-robot co-operation. In-

ternational Conference on Image Analysis and Processing. pp. 50-59.

Special Interest Group of Civil Lawsuits about Transportation in Tokyo District Court. 2014. Scale of contributory negligence ratio for civil lawsuits about transportation. ISSN:0438-5896. Hanrei Times. (in Japanese)

Zhao, L., Ichise, R., Sasaki, Y., Liu, Z., and Yoshikawa, T. 2016. Fast Decision Making using Ontology-based Knowledge Base, 2016 IEEE Intelligent Vehicles Symposium. pp.173-178.

Zhao, M., Käthner, D., Jipp, M., Söffker, D., and Lemmer K. 2017. Modeling Driver Behavior at Roundabouts: Results from a Field Study, 2017 IEEE Intelligent Vehicles Symposium.