C# Token

Understand the concept of tokens and how programs are made out of tokens.

Token. What is a token in the C# programming language? A token is a specific part of a C# program. The specification defines a token using the C# grammar.

A description. Tokens are generally any unit that is not whitespace or a comment. They are part of the text of a program. We examine tokens at the level of the C# language.Comment

Example. Let us explore the concept of tokens by looking at this C# program. We see the program and then break it up into individual tokens.

Types: Types of tokens include keywords (class), identifiers (Program), literals (2) and operators (=).

Program: C# class Program { static void Main() { int test = 2; float item = 5.5f; char unit = 'e'; string basic = "c#"; } } Tokens class Program { static void Main ( ) { int test = 2 ; float item = 5.5f ; char unit = 'e' ; string basic = "c#" ; } } Token types identifier: Identifiers are arbitrary names in your C# program. This includes Program, Main, test, item, unit and basic. keyword: Many keywords are reserved by the C# language. These include class, static, void, int, float, char and string. integer-literal: real-literal: character-literal: string-literal: Literals are constant values you can use in your program. These include 2, 5.5f, 'e' and "c#". Literals include the enclosing quotation marks. For the literal 5.5f, the numeric suffix is included in the literal. This is a real-literal. operator-or-punctuator: These are characters in your program that are part of the scope structure, or assignment and arithmetic statements. Some operators, such as && are considered a single token. The characters () are separate tokens.

Specification. For more material like this on the C# language, the specification is available. This material was adapted from page 61 of the C# annotated specification.

A summary. We now know what a token is in the C# language. This can be helpful in understanding how the language is parsed—and how to fix compile-time errors.

Any time you see more closely how the compiler approaches a language, you gain insight into how it should be used. This description provides insight into how the C# compiler views programs.
Dot Net Perls
© 2007-2020 Sam Allen. Every person is special and unique. Send bug reports to