像其他语言的PHP那样的解析器令牌？

Question

Short version:

Is there something similar to PHP parser tokens in other interpreted languages (Python, Ruby, etc.) and compiled languages (the C family, Java, etc.)?

Long Version:

On the CPP Rocks website there is an article showing a visual comparison of language complexity by means of a graph that breaks down the various building blocks of a language into categories. The graph for Coffeescript looks like this:

I wanted to make such a graph for PHP using the parser tokens as a starting point (to make sure I don't miss anything and because I'm lazy). I was wondering if there is something similar to these tokens in other in other interpreted languages (Python, Ruby, etc.) and compiled languages (the C family, Java, etc.).

Findings thus far:

• Java: the Chapters of the Language Specification describing Syntax and Lexical Structure seem a good place to start.
• Python: Chapter 2 of the manual does describe Python's lexical structure.
• Ruby: the token list for Ruby.

Answer 1

All parsers make the input into tokens. The language may or may not show what those tokens are, and of course, the actual meaning and names of tokens vary, and of course, since different languages have different syntax, set of reserved words and other constructs, each language will have a slightly different set of tokens.

A token here is just a "named representation of the actual symbol in the language specification". So for example, the parser will see the word break as input, and make it into the token T_BREAK.

For the type of graph you are looking at, you need to know what the different language constructs are, categorise and then show them graphically - I'm not sure looking at the list of tokens is the best way to achieve that.