Token. What is a token in the C# programming language? A token is a specific part of a C# program. The specification defines a token using the C# grammar.
Tokens are generally any unit that is not whitespace or a comment. They are part of the text of a program. We examine tokens at the level of the C# language.
Example. Let us explore the concept of tokens by looking at this C# program. We see the program and then break it up into individual tokens.
Detail Types of tokens include keywords (class), identifiers (Program), literals (2) and operators (=).
class Program
{
static void Main()
{
int test = 2;
float item = 5.5f;
char unit = 'e';
string basic = "c#";
}
}class
Program
"{"
static
void
Main
(
)
"{"
int
test
=
2
;
float
item
=
5.5f
;
char
unit
=
'e'
;
string
basic
=
"c#"
;
"}""}"identifier:
Identifiers are arbitrary names in your C# program.
This includes Program, Main, test, item, unit and basic.
keyword:
Many keywords are reserved by the C# language.
These include class, static, void, int, float, char and string.
integer-literal:
real-literal:
character-literal:
string-literal:
Literals are constant values you can use in your program.
These include 2, 5.5f, 'e' and "c#".
Literals include the enclosing quotation marks.
For the literal 5.5f, the numeric suffix is included in the literal.
This is a real-literal.
operator-or-punctuator:
These are characters in your program that are part of the scope
structure, or assignment and arithmetic statements.
Some operators, such as && are considered a single token.
The characters () are separate tokens.
Specification. For more material like this on the C# language, the specification is available. This material was adapted from page 61 of the C# annotated specification.
A summary. We now know what a token is in the C# language. This can be helpful in understanding how the language is parsed—and how to fix compile-time errors.
Any time you see more closely how the compiler approaches a language, you gain insight into how it should be used. This description provides insight into how the C# compiler views programs.
Dot Net Perls is a collection of pages with code examples, which are updated to stay current. Programming is an art, and it can be learned from examples.
Donate to this site to help offset the costs of running the server. Sites like this will cease to exist if there is no financial support for them.
Sam Allen is passionate about computer languages, and he maintains 100% of the material available on this website. He hopes it makes the world a nicer place.