C# Token Explanation

What exactly is a token in the C# programming language? The specification defines a token using the C# grammar. Tokens are generally any unit that is not whitespace or a comment.

This C# article explains the concept of tokens. Programs are made out of tokens.

Example

Let's explore the concept of tokens by looking at this simple C# program. First, we see the program, and then we break the program up into individual tokens. Tokens include keywords (class); identifiers (Program); literals (2); and operators (=).

Program [C#]

class Program
{
    static void Main()
    {
	int test = 2;
	float item = 5.5f;
	char unit = 'e';
	string basic = "c#";
    }
}

Tokens

class
Program
{
static
void
Main
(
)
{
int
test
=
2
;
float
item
=
5.5f
;
char
unit
=
'e'
;
string
basic
=
"c#"
;
}
}

identifier Identifiers are arbitrary names in your C# program. This includes Program, Main, test, item, unit, and basic.

keyword Keywords are reserved by the C# language. These include class, static, void, int, float, char, and string.

integer-literal real-literal character-literal string-literal Literals are constant values you can use in your program. These include 2, 5.5f, 'e', and "c#". Literals include the enclosing quotation marks. For the literal 5.5f, the numeric suffix is included in the literal; this is a real-literal.

operator-or-punctuator These are characters in your program that are part of the scope structure, or assignment and arithmetic statements. Some operators, such as && are considered a single token. However, the characters () are separate tokens.

Summary

We now know exactly what a token is in the C# programming language. This can be helpful in understanding how the language is parsed. Any time you can see more closely how the compiler approaches a language, you can gain insight into how it should be used. This material was adapted from page 61 of the C# annotated specification.