Lexical Analysis (2)

(1)

Lexical Analysis (2)

Sukree Sinthupinyo 1

1

Department of Computer Engineering Chulalongkorn University

14 July 2012

(2)

Outline

1 Recognition of Tokens Transition Diagrams

Completion of the Running Example

Architecture of a Transition-Diagram-Based Lexical Analyzer

Recognizing a keyword

2 Lexical-Analyzer Generator Lex Use of Lex

Structure of Lex

Lex Demonstration

(3)

Example

We will use this grammar as our first example

(4)

Example

(5)

Tokens, Patterns, and Attribute Values

(6)

Outline

2 Lexical-Analyzer Generator Lex Use of Lex

Structure of Lex

Lex Demonstration

(7)

Transition Diagrams

We’ll convert the regular-expression to transition diagrams Nodes represent states.

Edges represent methods of how to change the state.

If we find an edge labeled a connecting from current state

to another state and the next symbol is a, we will move to

such state.

(8)

Transition Diagrams (cont.)

Conventions about transition diagrams

Accepting or final state indicates that a lexeme has been found. It’s represented by a double circle.

If it is necessary to retract the forward pointer one position, then we shall place a * near that accepting state.

There is one start state, or initial state.

(9)

Transition Diagram for relop

(10)

Recognition of Reserved Words and Identifiers

How to recognize keywords, such as if, then and, else.

Install the reserved words in the symbol table initially. Any new identifier will be an id.

Create separate transition diagrams for each keyword.

(11)

Outline

Structure of Lex

Lex Demonstration

(12)

A transition diagram for unsigned numbers.

A transition diagram for whitespace.

(13)

Outline

Structure of Lex

Lex Demonstration

(14)

Example Code for relop

(15)

Architecture of a Transition-Diagram-Based Lexical

Analyzer (cont.)

(16)

Architecture of a Transition-Diagram-Based Lexical

Analyzer (cont.)

(17)

Architecture of a Transition-Diagram-Based Lexical Analyzer (cont.)

Try sequentially. The function fail() will reset the pointer forward and starts the next transition diagram.

Run in parallel. Feed the next input to all of them and allowing each one to make whatever transition it required.

Combine all the transition diagrams into one.

We combine states 0, 9, 12, and 22 into one start state.

(18)

Outline

2 Lexical-Analyzer Generator Lex Use of Lex

Structure of Lex

Lex Demonstration

(19)

KMP Algorithm

Knuth, Morris, and Pratt algorithm

Recognize a single keyword b

₁

b

₂

. . . b

_n

in a text string.

For example, the diagram for the keyword ababaa is

We must find the failure function, f (s) computed as follow.

(20)

Failure Function

ababaa

(21)

Failure Function cont.

To find whether a keyword b 1 b 2 . . . b n is a substring of a

string a 1 a 2 . . . a m , we use the following algorithm.

(22)

Outline

Structure of Lex

Lex Demonstration

(23)

Use of Lex

Process of Lex

(24)

Outline

Structure of Lex

Lex Demonstration

(25)

Form of Lex

declarations

%%

translation rules

%%

auxiliary functions

(26)

Declarations

(27)

Translations Rules

(28)

Auxiliary Functions

(29)

Outline

Structure of Lex

Lex Demonstration