Parsing Giuseppe Attardi Università di Pisa. Parsing Calculate grammatical structure of program, like diagramming sentences, where: Tokens = “words” Tokens

ParsingParsing

Giuseppe AttardiGiuseppe Attardi

Università di PisaUniversità di Pisa

ParsingParsing

Calculate grammatical structure of Calculate grammatical structure of program, like diagramming program, like diagramming sentences, where:sentences, where:

Tokens = “words”Tokens = “words”

Programs = “sentences”Programs = “sentences”

For further information: Aho, Sethi, Ullman, “Compilers: Principles,

Techniques, and Tools” (a.k.a, the “Dragon Book”)

Outline of coverageOutline of coverage

Context-free grammarsContext-free grammarsParsingParsing

– Tabular Parsing Methods– One pass

• Top-down• Bottom-up

YaccYacc

Parser: extracts grammatical structure of programParser: extracts grammatical structure of program

function-def

name arguments stmt-list

mainstmt

expression

operatorexpression expression

variable string

cout

<<

“hello, world\n”

Context-free languagesContext-free languages

Grammatical structure defined by context-Grammatical structure defined by context-free grammarfree grammar

statementstatement labeled-statementlabeled-statement | | expression-statementexpression-statement | | compound-statementcompound-statementlabeled-statementlabeled-statement identident :: statementstatement | | casecase constant-expression constant-expression :: statementstatementcompound-statementcompound-statement {{ declaration-list statement-list declaration-list statement-list }}

terminalnon-terminal

“Context-free” = only one non-terminal in left-part

Parse treesParse trees

Parse tree = tree labeled with grammar Parse tree = tree labeled with grammar symbols, such that:symbols, such that:

If node is labeled A, and its children If node is labeled A, and its children are labeled are labeled xx11......xxnn, then there is a , then there is a productionproductionA A xx11......xxnn

““Parse tree from Parse tree from AA” = root labeled ” = root labeled with with AA

““Complete parse tree” = all leaves Complete parse tree” = all leaves labeled with tokenslabeled with tokens

Parse trees and sentencesParse trees and sentences

Frontier Frontier of tree = labels on leaves (in left-of tree = labels on leaves (in left-to-right order)to-right order)

Frontier of tree from Frontier of tree from SS is a is a sentential formsentential form Frontier of a complete tree from Frontier of a complete tree from SS is a is a

sentencesentence

L

E

a

L

; E

“Frontier”

ExampleExample

GG: L : L L L ;; E | E E | E E E aa | | bb

Syntax trees from start symbol (L):Syntax trees from start symbol (L):

a a;E a;b;b

L

E

a

L

E

a

L

; E L

E

a

L

; E

b

L

E

b

;

Sentential forms:Sentential forms:

DerivationsDerivations

Alternate definition of Alternate definition of sentencesentence:: Given Given , , in in VV*, say *, say is a is a derivation derivation

step step if if ’’’’ and ’’ and = = ’’’’ , where ’’ , where A A is a productionis a production

is a is a sentential form sentential form iff there exists a iff there exists a derivationderivation (sequence of derivation steps) (sequence of derivation steps) SS( alternatively, we say that ( alternatively, we say that SS))

Two definitions are equivalent, but note that there are many derivations corresponding to each parse tree

Another exampleAnother example

HH: L : L E E ;; L | E L | E E E aa | | bb

L

E

a

L

E

a

L

;E L

E

a

L

;E

b

L

E

b

;

AmbiguityAmbiguity

For some purposes, it is important to For some purposes, it is important to know whether a sentence can have more know whether a sentence can have more than one parse treethan one parse tree

A grammar is A grammar is ambiguous ambiguous if there is a if there is a sentence with more than one parse treesentence with more than one parse tree

Example: Example: EE E E++E | EE | E**E | E | idid

E

E

E

E

E

id id

id+

*

E

E

EE

Eid

id id

+

*

NotesNotes

If e then if b then d else fIf e then if b then d else f{ int x; y = 0; }{ int x; y = 0; }A.b.c = d;A.b.c = d; Id -> s | s.idId -> s | s.id

E -> E + T -> E + T + T -> T + T + T -> id E -> E + T -> E + T + T -> T + T + T -> id + T + T -> id + T * id + T -> id + id * id + T + T -> id + T * id + T -> id + id * id + T ->+ T ->id + id * id + idid + id * id + id

AmbiguityAmbiguity

Ambiguity is a function of the Ambiguity is a function of the grammar rather than the languagegrammar rather than the language

Certain ambiguous grammars may Certain ambiguous grammars may have equivalent unambiguous oneshave equivalent unambiguous ones

Grammar TransformationsGrammar Transformations

Grammars can be transformed Grammars can be transformed without affecting the language without affecting the language generatedgenerated

Three transformations are discussed Three transformations are discussed next:next:– Eliminating Ambiguity– Eliminating Left Recursion

(i.e.productions of the form AA )– Left Factoring

Eliminating AmbiguityEliminating Ambiguity

Sometimes an ambiguous grammar can Sometimes an ambiguous grammar can be rewritten to eliminate ambiguitybe rewritten to eliminate ambiguity

For example, expressions involving For example, expressions involving additions and products can be written as additions and products can be written as follows:follows:

EE E E ++T | TT | T TT T T ** idid | | idid The language generated by this grammar The language generated by this grammar

is the same as that generated by the is the same as that generated by the grammar in slide “Ambiguity”. Both grammar in slide “Ambiguity”. Both generate generate idid(+(+id|id|**idid)*)*

However, this grammar is not ambiguousHowever, this grammar is not ambiguous

Eliminating Ambiguity (Cont.)Eliminating Ambiguity (Cont.)

One advantage of this grammar is One advantage of this grammar is that it represents the precedence that it represents the precedence between operators. In the parsing between operators. In the parsing tree, products appear nested within tree, products appear nested within additionsadditions

E

T

TE

id

+

*

idT

id


An example of ambiguity in a An example of ambiguity in a programming language is the programming language is the dangling dangling elseelse

ConsiderConsider S S ifif thenthen SS elseelse SS | | ifif thenthen

SS | |


When there are two nested ifs and When there are two nested ifs and only one else..only one else..

S

ifif then S else S

if then S

S

ifif then S

ifif then S else S


In most languages (including C++ and Java), In most languages (including C++ and Java), each each elseelse is assumed to belong to the is assumed to belong to the nearest nearest ifif that is not already matched by an that is not already matched by an elseelse. This association is expressed in the . This association is expressed in the following (unambiguous) grammar:following (unambiguous) grammar:

S S MatchedMatched | Unmatched| Unmatched Matched Matched ifif thenthen Matched Matched elseelse Matched Matched | | Unmatched Unmatched ifif thenthen S S ||ifif thenthen Matched Matched elseelse Unmatched Unmatched


Ambiguity is a property of the Ambiguity is a property of the grammargrammar

It is undecidable whether a context It is undecidable whether a context free grammar is ambiguousfree grammar is ambiguous

The proof is done by reduction to The proof is done by reduction to Post’s correspondence problemPost’s correspondence problem

Although there is no general Although there is no general algorithm, it is possible to isolate algorithm, it is possible to isolate certain constructs in productions certain constructs in productions which lead to ambiguous grammarswhich lead to ambiguous grammars


For example, a grammar containing the For example, a grammar containing the production production AAAA |AA | would be ambiguous, would be ambiguous, because the substring because the substring has two parses: has two parses:

A

A A

A

A

A A

A

A

A

This ambiguity disappears if we use the productions This ambiguity disappears if we use the productions AAAB |AB | BB and and BB

or the productionsor the productions AABA |BA | BB and and BB..


Examples of ambiguous productions:Examples of ambiguous productions:AAAAA | AAA | AA

A CF language is inherently ambiguous if A CF language is inherently ambiguous if it has no unambiguous CFGit has no unambiguous CFG– An example of such a language is

L = {aibjcm | i=j or j=m} which can be generated by the grammar:

SAB | DC AaA | CcC | BbBc | DaDb |

Elimination of Left RecursionElimination of Left Recursion

A grammar is left recursive if it has a A grammar is left recursive if it has a nonterminal nonterminal AA and a derivation and a derivation A A AAfor some stringfor some string

– Top-down parsing methods cannot handle left-recursive grammars, so a transformation to eliminate left recursion is needed

Immediate left recursion (productions of Immediate left recursion (productions of the form the form A A AA) can be easily eliminated:) can be easily eliminated:

1. Group the A-productions as A A1 | A2 | … | Am | 1| 2 | … | n

where no i begins with A2. Replace the A-productions by A 1A’ | 2A’ | … | nA’

A’ 1A’ | 2A’| … | mA’ |

Elimination of Left Recursion (Cont.)Elimination of Left Recursion (Cont.)

The previous transformation, however, The previous transformation, however, does not eliminate left recursion does not eliminate left recursion involving two or more stepsinvolving two or more steps

For example, consider the grammarFor example, consider the grammar S Aa | b

A Ac | Sd |

S is left-recursive because S is left-recursive because S S AAaaSSdadabut it is not immediately left but it is not immediately left recursiverecursive


Algorithm. Eliminate left recursionAlgorithm. Eliminate left recursionArrange nonterminals in some order AArrange nonterminals in some order A11, A, A2 ,2 ,,…, A,…, Ann

for i = 1 to n {for i = 1 to n { for j = 1 to i - 1 {for j = 1 to i - 1 { replace each production of the form replace each production of the form AAi i AAjj by the production by the production AAi i 1 1 | | 2 2 | … | | … | nn where where AAj j 1 1 | | 2 2 |…| |…| nn are all the current Aare all the current Ajj--

productionsproductions }} eliminate the immediate left recursion among the Aeliminate the immediate left recursion among the Aii--

productionsproductions}}


To show that the previous algorithm actually To show that the previous algorithm actually works, notice that iteration works, notice that iteration ii only changes only changes productions with productions with AAii on the left-hand side. And on the left-hand side. And mm > > ii in all productions of the form in all productions of the form AAi i AAmm

Induction proof: Induction proof: – Clearly true for i = 1– If it is true for all i < k, then when the outer loop is

executed for i = k, the inner loop will remove all productions Ai Am with m < i

– Finally, with the elimination of self recursion, m in the AiAm productions is forced to be > i

At the end of the algorithm, all derivations of the At the end of the algorithm, all derivations of the form form AAi i AAmmwill have will have mm > > ii and therefore left and therefore left recursion would not be possiblerecursion would not be possible

Left FactoringLeft Factoring

Left factoring helps transform a grammar for Left factoring helps transform a grammar for predictive parsingpredictive parsing

For example, if we have the two productionsFor example, if we have the two productions S S ifif thenthen SS elseelse SS | | ifif thenthen SS

on seeing the input token on seeing the input token ifif, we cannot , we cannot immediately tell which production to choose to immediately tell which production to choose to expand expand SS

In general, if we have In general, if we have A A 1 1 || 22 and the input and the input begins with begins with , we do not know, we do not know (without looking (without looking further) which production to use to expand further) which production to use to expand AA

Left Factoring (Cont.)Left Factoring (Cont.)

However, we may defer the decision However, we may defer the decision by expanding A to by expanding A to A’A’

Then after seeing the input derived Then after seeing the input derived from from , we may expand A’ to , we may expand A’ to 1 1 or toor to

22

Left-factored, the original Left-factored, the original productions becomeproductions become

AA A’A’

A’A’1 1 | | 22

Non-Context-Free Language ConstructsNon-Context-Free Language Constructs

Examples of non-context-free languages are:Examples of non-context-free languages are:– L1 = {wcw | w is of the form (a|b)*}– L2 = {anbmcndm | n 1 and m 1 }– L3 = {anbncn | n 0 }

Languages similar to these that are context freeLanguages similar to these that are context free– L’1 = {wcwR | w is of the form (a|b)*} (wR stands for w

reversed) This language is generated by the grammar

S aSa | bSb | c

– L’2 = {anbmcmdn | n 1 and m 1 } This language is generated by the grammar

S aSd | aAdA bAc | bc

Non-Context-Free Language Constructs Non-Context-Free Language Constructs (Cont.)(Cont.)L”L”2 2 = {= {aannbbnnccmmddmm | | n n 1 1 andand m m 1 1 }} is generated by the grammaris generated by the grammar

S ABA aAb | abB cBd | cd

L’L’3 3 = {= {aannbbnn | | n n 1 1}} is generated by the grammaris generated by the grammar

S aSb | ab This language is not definable by any This language is not definable by any

regular expressionregular expression

Non-Context-Free Language Constructs Non-Context-Free Language Constructs (Cont.)(Cont.)

Suppose we could construct a DFSM Suppose we could construct a DFSM DD accepting accepting L’L’3. 3.

DD must have a finite number of states, say must have a finite number of states, say kk. . Consider the sequence of states Consider the sequence of states ss00, , ss11, , ss22, …, , …, sskk

entered by entered by DD having read having read , , aa, , aaaa, …, , …, aakk. . Since Since DD only has only has kk states, two of the states in the states, two of the states in the

sequence have to be equal. Say,sequence have to be equal. Say, s sii ssjj ( (ii jj). ). From From ssii, a sequence of , a sequence of ii bbs leads to an accepting s leads to an accepting

(final) state. Therefore, the same sequence of (final) state. Therefore, the same sequence of ii bbs s will also lead to an accepting state from will also lead to an accepting state from ssjj. . Therefore Therefore DD would accept would accept aajjbbii which means that which means that the language accepted by the language accepted by DD is not identical to L’ is not identical to L’33. . A contradiction.A contradiction.

ParsingParsing

The parsing problem is: Given string of The parsing problem is: Given string of tokens tokens ww, find a parse tree whose frontier , find a parse tree whose frontier is is ww. (Equivalently, find a derivation from . (Equivalently, find a derivation from ww))

A A parserparser for a grammar for a grammar GG reads a list of reads a list of tokens and finds a parse tree if they form tokens and finds a parse tree if they form a sentence (or reports an error otherwise)a sentence (or reports an error otherwise)

Two classes of algorithms for parsing:Two classes of algorithms for parsing:– Top-down– Bottom-up

Parser generatorsParser generators

A A parser generator parser generator is a program that reads is a program that reads a grammar and produces a parsera grammar and produces a parser

The best known parser generator is The best known parser generator is yaccyacc It It produces bottom-up parsersproduces bottom-up parsers

Most parser generators - including yacc - Most parser generators - including yacc - do not work for every CFG; they accept a do not work for every CFG; they accept a restricted class of CFG’s that can be restricted class of CFG’s that can be parsed efficiently using the method parsed efficiently using the method employed by that parser generatoremployed by that parser generator

Top-down parsingTop-down parsing

Starting from parse tree containing Starting from parse tree containing just just SS, build tree down toward input. , build tree down toward input. Expand left-most non-terminal.Expand left-most non-terminal.

Algorithm: (next slide)Algorithm: (next slide)

Top-down parsing (cont.)Top-down parsing (cont.)

Let input = aLet input = a11aa22...a...ann

current sentential form (csf) = current sentential form (csf) = SS

loop {loop {

suppose csf = asuppose csf = a11…a…akkAA

based on abased on akk+1+1…, choose production…, choose production

A A csf becomes acsf becomes a11…a…akk

}}

Top-down parsing exampleTop-down parsing example

Grammar: Grammar: HH: L : L E E ;; L | E L | E E E aa | | bb

Input: Input: a;ba;bParse tree Sentential form Parse tree Sentential form Input Input

L a;b

E;L a;b

L

LE L;

LE L;

a

a;L a;b

Top-down parsing example Top-down parsing example (cont.)(cont.)

Parse tree Sentential form InputParse tree Sentential form Input

a;E a;bLE L;

a E

LE L;

a E

b

a;b a;b

LL(1) parsingLL(1) parsing

Efficient form of top-down parsingEfficient form of top-down parsingUse only first symbol of remaining Use only first symbol of remaining

input (input (aakk+1+1) to choose next ) to choose next

production. That is, employ a production. That is, employ a function M: function M: N N P in “choose P in “choose production” step of algorithm.production” step of algorithm.

When this is possible, grammar is When this is possible, grammar is called LL(1)called LL(1)

LL(1) examplesLL(1) examples

Example 1:Example 1: H: L E ; L | E

E a | b

Given input a;b, so next symbol is a.

Which production to use? Can’t tell.

H not LL(1)

LL(1) examplesLL(1) examples

Example 2:Example 2: Exp Term Exp’

Exp’ $ | + Exp

Term id(Use $ for “end-of-input” symbol.)

Grammar is LL(1): Exp and Term have only one production; Exp’ has two productions but only one is applicable at any time.

Grammar is LL(1): Exp and Term have only one production; Exp’ has two productions but only one is applicable at any time.

Nonrecursive predictive parsingNonrecursive predictive parsing

Maintain a stack explicitly, rather Maintain a stack explicitly, rather than implicitly via recursive callsthan implicitly via recursive calls

Key problem during predictive Key problem during predictive parsing: determining the production parsing: determining the production to be applied for a non-terminalto be applied for a non-terminal

Nonrecursive predictive parsingNonrecursive predictive parsing

Algorithm. Algorithm. Nonrecursive predictive parsingNonrecursive predictive parsing Set Set ipip to point to the first symbol of to point to the first symbol of ww$.$. repeatrepeat Let Let XX be the top of the stack symbol and a the symbol pointed to be the top of the stack symbol and a the symbol pointed to

by by ipip ifif XX is a terminal or $ is a terminal or $ thenthen ifif XX == == aa thenthen pop pop XX from the stack and advance from the stack and advance ipip elseelse error() error() elseelse // // XX is a nonterminal is a nonterminal ifif MM[[X,aX,a] == ] == XXYY11 Y Y22 … Y … Y kk thenthen pop pop XX from the stack from the stack push Ypush YkkY Y k-1k-1, …, Y, …, Y11 onto the stack with Y onto the stack with Y11 on top on top (push nothing if (push nothing if YY11 Y Y22 … Y … Y kk is is ) ) output the production output the production XXYY11 Y Y22 … Y … Y kk elseelse error() error() untiluntil X == $ X == $

LL(1) grammarsLL(1) grammars

No left recursionNo left recursionA A : If this production is chosen,

parse makes no progress.No common prefixesNo common prefixes

A |

Can fix by “left factoring”:A A’

’|

LL(1) grammars (cont.)LL(1) grammars (cont.)

No ambiguityNo ambiguityPrecise definition requires that

production to choose be unique (“choose” function M very hard to calculate otherwise)

Top-down ParsingTop-down Parsing

Input tokens: <t0,t1,…,ti,...>L

E0 … En

Start symbol androot of parse tree

Input tokens: <ti,...>L

E0 … En

...From left to right,“grow” the parsetree downwards

Checking LL(1)-nessChecking LL(1)-ness

For any sequence of grammar symbols For any sequence of grammar symbols , , define set FIRST(define set FIRST() ) to be to be

FIRST(FIRST() = { ) = { aa | | * * aa for some for some } }

LL(1) definitionLL(1) definition

Define: Grammar G = (N, Define: Grammar G = (N, , P, S) is LL(1), P, S) is LL(1) iff whenever there iff whenever there are two left-most derivations (in which the leftmost non-are two left-most derivations (in which the leftmost non-terminal is always expanded first) terminal is always expanded first) SS * * wAwA ww * * wtxwtx SS * * wAwA ww * * wtywty

it follows that it follows that = =

In other words, given In other words, given 1. a string1. a string wA wA in V* and in V* and 2. t, the first terminal symbol to be derived from 2. t, the first terminal symbol to be derived from AA there is at most one production that can be applied to there is at most one production that can be applied to AA to to yield a derivation of any terminal string beginning with yield a derivation of any terminal string beginning with wtwt

FIRST sets can often be calculated by inspectionFIRST sets can often be calculated by inspection

FIRST SetsFIRST Sets

Exp Term Exp’Exp’ $ | + Exp Term id

(Use $ for “end-of-input” symbol)

FIRST($) = {$}FIRST(+ Exp) = {+}

FIRST($) FIRST(+ Exp) = {}

grammar is LL(1)

FIRST($) = {$}FIRST(+ Exp) = {+}

FIRST($) FIRST(+ Exp) = {}

grammar is LL(1)

FIRST SetsFIRST Sets

L E ; L | EE a | b

FIRST(E ; L) = {a, b} = FIRST(E)FIRST(E ; L) FIRST(E) {} grammar not LL(1).

FIRST(E ; L) = {a, b} = FIRST(E)FIRST(E ; L) FIRST(E) {} grammar not LL(1).

Computing FIRST SetsComputing FIRST Sets

Algorithm. Compute FIRST(X) for all grammar Algorithm. Compute FIRST(X) for all grammar symbols Xsymbols X

forall X forall X V do FIRST(X) = {} V do FIRST(X) = {} forall X forall X (X is a terminal) do FIRST(X) = {X} (X is a terminal) do FIRST(X) = {X} forall productions X forall productions X do FIRST(X) = FIRST(X) U { do FIRST(X) = FIRST(X) U {}} repeatrepeat c: forall productions c: forall productions X X YY11YY22 … Y … Ykk do do forall i forall i [1,k] do [1,k] do FIRST(X) = FIRST(X) U (FIRST(FIRST(X) = FIRST(X) U (FIRST(YYii) - {) - {}) })

if if FIRST( FIRST(YYii) then continue c) then continue c FIRST(X) = FIRST(X) U {FIRST(X) = FIRST(X) U {} } until no more terminals or until no more terminals or are added to any FIRST are added to any FIRST

setset

FIRST Sets of Strings of SymbolsFIRST Sets of Strings of Symbols

FIRST(XFIRST(X11XX22…X…Xnn) is the union of ) is the union of

FIRST(XFIRST(X11) and all FIRST(X) and all FIRST(Xii) such that ) such that

FIRST( FIRST(XXkk) for ) for kk = 1, 2, …, = 1, 2, …, ii-1-1

FIRST(XFIRST(X11XX22…X…Xnn) contains ) contains iff iff

FIRST(FIRST(XXkk) for k = 1, 2, …, ) for k = 1, 2, …, nn

FIRST Sets do not SufficeFIRST Sets do not Suffice

Given the productionsGiven the productions A A T x T x A A T y T y

T T ww

T T TTww should be applied when the next should be applied when the next

input token is w.input token is w. TTshould be applied whenever the should be applied whenever the

next terminal is either x or ynext terminal is either x or y

FOLLOW SetsFOLLOW Sets

For any nonterminal For any nonterminal XX, define the set , define the set FOLLOW(FOLLOW(XX) ) as as

FOLLOW(FOLLOW(XX) = {) = {aa | S | S * * XXaa}}

Computing the FOLLOW SetComputing the FOLLOW Set

Algorithm. Compute FOLLOW(X) for all nonterminals Algorithm. Compute FOLLOW(X) for all nonterminals XX

FOLLOW(S) ={$}FOLLOW(S) ={$} forall productions A forall productions A BB do do

FOLLOW(B)=Follow(B) FOLLOW(B)=Follow(B) (FIRST( (FIRST() - {) - {})}) repeatrepeat forall productions A forall productions A B or A B or A BB with with

FIRST(FIRST() do) do FOLLOW(B) = FOLLOW(B) FOLLOW(B) = FOLLOW(B)

FOLLOW(A) FOLLOW(A) until all FOLLOW sets remain the sameuntil all FOLLOW sets remain the same

Construction of a predictive parsing tableConstruction of a predictive parsing table

Algorithm. Construction of a predictive parsing Algorithm. Construction of a predictive parsing tabletable

M[:,:] = {}M[:,:] = {} forall productions A forall productions A do do forall a forall a FIRST( FIRST() do ) do M[A,a] = M[A,a] U {A M[A,a] = M[A,a] U {A } } if if FIRST( FIRST() then ) then forall b forall b FOLLOW(A) do FOLLOW(A) do M[A,b] = M[A,b] U {A M[A,b] = M[A,b] U {A } } Make all empty entries of M be errorMake all empty entries of M be error

Another Definition of LL(1)Another Definition of LL(1)

Define: Grammar G is LL(1)Define: Grammar G is LL(1) if for every if for every

AA N with productions A N with productions A

11nn

FIRST(FIRST(ii FOLLOW(A)) FOLLOW(A)) FIRST( FIRST(j j

FOLLOW(A) ) = {} for all FOLLOW(A) ) = {} for all ii, , jj

Regular LanguagesRegular Languages

Definition. A Definition. A regularregular grammar is one grammar is one whose productions are all of the whose productions are all of the type:type:– A aB– A a

A A Regular ExpressionRegular Expression is either: is either:– a– R1 | R2

– R1 R2

– R*

Nondeterministic Finite State Nondeterministic Finite State AutomatonAutomaton

0 1 2 3

a

b

a b bstart

Regular LanguagesRegular Languages

Theorem. The classes of languagesTheorem. The classes of languages– Generated by a regular grammar– Expressed by a regular expression– Recognized by a NDFS automaton– Recognized by a DFS automaton

coincide.coincide.

Deterministic Finite AutomatonDeterministic Finite Automaton

space, tab, new line

digit

OPERATOR

KEYWORD

digit

=, +, -, /, (, )

letter

START

NUM

$ $ $

circle state

double circle accept state

arrow transition

bold, cap labels state names

lower case labels transition characters

Scanner codeScanner code state := startstate := start looploop if no input character buffered then read one, and add it to the accumulated tokenif no input character buffered then read one, and add it to the accumulated token case state ofcase state of start: start: case input_char ofcase input_char of A..Z, a..z : state := idA..Z, a..z : state := id 0..9 : state := num0..9 : state := num else ...else ... endend id:id: case input_char ofcase input_char of A..Z, a..z : state := idA..Z, a..z : state := id 0..9 : state := id0..9 : state := id else ...else ... endend num:num: case input_char ofcase input_char of 0..9: ...0..9: ... ...... else ...else ... endend ...... end;end; end;end;

Table-driven DFATable-driven DFA

0-start 1-num 2-id 3-operator 4-keyword

white space 0 exit exit exit exit

letter 2 error 2 exit error

digit 1 1 2 exit error

operator 3 exit exit exit exit

$ 4 error error exit 4

L0

CFL [NPA]

Language ClassesLanguage Classes

LR(1)

LL(1)RL

[DFA=NFA]

L0

CSL

QuestionQuestion

Are regular expressions, as provided Are regular expressions, as provided by Perl or other languages, sufficient by Perl or other languages, sufficient for parsing nested structures, e.g. for parsing nested structures, e.g. XML files?XML files?

Recursive Descent ParserRecursive Descent Parser

stat stat → var → var == expr expr ;;

expr → term [expr → term [++ expr] expr]

term → factor [term → factor [** factor] factor]

factor → factor → (( expr expr )) | var | constant | var | constant

var → identifiervar → identifier

ScannerScanner

public class Scanner {public class Scanner { private StreamTokenizer input;private StreamTokenizer input; private Type lastToken;private Type lastToken;

public enum Type { INVALID_CHAR, NO_TOKEN , PLUS,public enum Type { INVALID_CHAR, NO_TOKEN , PLUS,// etc. for remaining tokens, then:// etc. for remaining tokens, then:EOFEOF

};};

public Scanner (Reader r) {public Scanner (Reader r) { input = new StreamTokenizer(r);input = new StreamTokenizer(r); input.resetSyntax();input.resetSyntax(); input.eolIsSignificant(false);input.eolIsSignificant(false); input.wordChars('a', 'z');input.wordChars('a', 'z'); input.wordChars('A', 'Z');input.wordChars('A', 'Z'); input.ordinaryChar('+');input.ordinaryChar('+'); input.ordinaryChar('*');input.ordinaryChar('*'); input.ordinaryChar('=');input.ordinaryChar('='); input.ordinaryChar('(');input.ordinaryChar('('); input.ordinaryChar(')');input.ordinaryChar(')'); input.whitespaceChars('\u0000', ' ');input.whitespaceChars('\u0000', ' '); }}

ScannerScanner

public int nextToken() {public int nextToken() { Type token;Type token; try {try { switch (input.nextToken()) {switch (input.nextToken()) { case StreamTokenizer.TT_EOF:case StreamTokenizer.TT_EOF:

token = EOF;token = EOF;break;break;

case Type.TT_WORD:case Type.TT_WORD:if (input.sval.equalsIgnoreCase("false"))if (input.sval.equalsIgnoreCase("false")) token = FALSE;token = FALSE;else if (input.sval.equalsIgnoreCase("true"))else if (input.sval.equalsIgnoreCase("true")) token = TRUE;token = TRUE;elseelse token = VARIABLE;token = VARIABLE;break;break;case '+':case '+': token = PLUS;token = PLUS;break;break;// etc.// etc.

}} } catch (IOException ex) { token = EOF; }} catch (IOException ex) { token = EOF; } return token;return token; }}}}

ParserParser

public class Parser {public class Parser { private LexicalAnalyzer lexer;private LexicalAnalyzer lexer; private Type token;private Type token;

public Expr parse(Reader r) throws public Expr parse(Reader r) throws SyntaxException {SyntaxException {

lexer = new LexicalAnalyzer(r);lexer = new LexicalAnalyzer(r);nextToken(); // assigns tokennextToken(); // assigns token

Statement stat = statement();Statement stat = statement(); expect(LexicalAnalyzer.EOF);expect(LexicalAnalyzer.EOF); return stat;return stat; }}

StatementStatement

// stat ::= variable '=' expr ';'// stat ::= variable '=' expr ';' private Statement stat() throws private Statement stat() throws

SyntaxException {SyntaxException { Expr var = variable();Expr var = variable(); expect(LexicalAnalyzer.ASSIGN);expect(LexicalAnalyzer.ASSIGN); Expr exp = expr();Expr exp = expr(); Statement stat = new Statement(var, exp);Statement stat = new Statement(var, exp); expect(LexicalAnalyzer.SEMICOLON);expect(LexicalAnalyzer.SEMICOLON); return stat;return stat; }}

ExprExpr

// expr ::= term ['+' expr]// expr ::= term ['+' expr] private Expr expr() throws SyntaxException private Expr expr() throws SyntaxException

{{ Expr exp = term();Expr exp = term(); while (token == LexicalAnalyzer.PLUS) {while (token == LexicalAnalyzer.PLUS) { nextToken();nextToken(); exp = new Exp(exp, expression());exp = new Exp(exp, expression()); }} return exp;return exp; }}

TermTerm

// term ::= factor ['*' term ]// term ::= factor ['*' term ]

private Expr term() throws private Expr term() throws SyntaxException {SyntaxException {

Expr exp = factor();Expr exp = factor();

// Rest of body: left as an exercise.// Rest of body: left as an exercise.

}}

FactorFactor

// factor ::= ( expr ) | var// factor ::= ( expr ) | var private Expr factor() throws S.Exception {private Expr factor() throws S.Exception { Expr exp = null;Expr exp = null; if (token == LexicalAnalyzer.LEFT_PAREN) {if (token == LexicalAnalyzer.LEFT_PAREN) { nextToken();nextToken(); exp = expression();exp = expression(); expect(LexicalAnalyzer.RIGHT_PAREN);expect(LexicalAnalyzer.RIGHT_PAREN); } else {} else {

exp = variable();exp = variable();}}

return exp;return exp; }}

VariableVariable

// variable ::= identifier// variable ::= identifier private Expr variable() throws S.Exception {private Expr variable() throws S.Exception {

if (if (token == LexicalAnalyzer.ID) {token == LexicalAnalyzer.ID) { Expr exp = new Variable(lexer.getString());Expr exp = new Variable(lexer.getString()); nextToken();nextToken(); return exp;return exp; }} }}

ConstantConstant

private Expr constantExpression() throws private Expr constantExpression() throws S.Exception {S.Exception {

Expr exp = null;Expr exp = null;

// Handle the various cases for constant// Handle the various cases for constant

// expressions: left as an exercise.// expressions: left as an exercise.

return exp;return exp;

}}

UtilitiesUtilities

private void expect(Type t) throws private void expect(Type t) throws SyntaxException {SyntaxException {

if (token != t) { // throw SyntaxException...if (token != t) { // throw SyntaxException... }} nextToken();nextToken(); }}

private void nextToken() {private void nextToken() { token = lexer.nextToken();token = lexer.nextToken(); }}}}

Documents

Parsing Giuseppe Attardi Università di Pisa. Parsing Calculate grammatical structure of program, like diagramming sentences, where: Tokens = “words” Tokens