Class TokenizerRuntime

Namespace

Utils.Parser.Runtime

Assembly

Utils.Parser.dll

Provides reusable tokenization mechanics for parser-oriented consumers.

public sealed class TokenizerRuntime

Inheritance

object

TokenizerRuntime

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Extension Methods

DataUtils.ToDbValue(object?)

MathEx.Clamp<T>(T, T, T, IComparer<T>)

FluentExtensions.Failure<T>(T)

FluentExtensions.IsEqualTo<T, CT>(T, CT)

FluentExtensions.IsGreaterOrEqualsThan<T, CT>(T, CT)

FluentExtensions.IsGreaterThan<T, CT>(T, CT)

FluentExtensions.IsIn<T>(T, params T[])

FluentExtensions.IsLowerOrEqualsThan<T, CT>(T, CT)

FluentExtensions.IsLowerThan<T, CT>(T, CT)

FluentExtensions.IsNotIn<T>(T, params T[])

FluentExtensions.IsNotNull<T>(T)

FluentExtensions.IsNull<T>(T)

FluentExtensions.Success<T>(T)

FluentExtensions.Test<T>(T, Func<T, bool>)

ObjectUtils.Between<T>(T, IComparer<T>, T, T)

ObjectUtils.Between<T>(T, IComparer<T>, T, T, bool, bool)

ObjectUtils.DoAsync<T, Result>(T, Func<T, Task<Result>>, Func<Task<Result>>)

ObjectUtils.DoAsync<T, Result>(T, Func<T, Task<Result>>, Result)

ObjectUtils.DoAsync<T, Result>(T, Func<T, Result>, Func<Result>)

ObjectUtils.DoAsync<T, Result>(T, Func<T, Result>, Result)

ObjectUtils.Do<T, Result>(T, Func<T, Result>, Func<Result>)

ObjectUtils.Do<T, Result>(T, Func<T, Result>, Result)

ObjectUtils.In<T>(T, params T[])

ObjectUtils.NotIn<T>(T, params T[])

Validations.ArgMustBe<T>(T, Func<T, bool>, string, string)

Validations.ArgMustNotBe<T>(T, Func<T, bool>, string, string)

Validations.Arg<T>(T, string)

Validations.ValueMustNotBeNull<T>(T, string)

Validations.Variable<T>(T, string)

Constructors

TokenizerRuntime(string, IEnumerable<char>, IEnumerable<string>, IEnumerable<RuntimeTryReadToken>, IEnumerable<RuntimeStringTransformer>)

Initializes a new instance of the TokenizerRuntime class.

public TokenizerRuntime(string content, IEnumerable<char> whiteSpaces, IEnumerable<string> symbols, IEnumerable<RuntimeTryReadToken> tokenReaders, IEnumerable<RuntimeStringTransformer> stringTransformers)

Parameters

content string

Source content to tokenize.

whiteSpaces IEnumerable<char>

Whitespace characters to skip when requested.

symbols IEnumerable<string>

Symbols to match when token readers do not consume input.

tokenReaders IEnumerable<RuntimeTryReadToken>

Custom token readers.

stringTransformers IEnumerable<RuntimeStringTransformer>

Custom string transformers.

Properties

DefineString

Gets the most recently defined string value.

public string DefineString { get; }

Property Value

string

Position

Gets the current tokenizer position.

public RuntimeTokenizerPosition Position { get; }

Property Value

RuntimeTokenizerPosition

Methods

DiscardPosition()

Discards the last saved position.

public void DiscardPosition()

PeekToken()

Peeks the next token without consuming it.

public string? PeekToken()

Returns

string

The next token, or null.

PopPosition()

Restores the last saved position.

public void PopPosition()

PushPosition()

Saves the current position.

public void PushPosition()

ReadSymbol(string, bool)

Reads a symbol at the current position.

public bool ReadSymbol(string symbol, bool throwExceptionIfError)

Parameters

symbol string

Symbol expected at current location.

throwExceptionIfError bool

Indicates whether a mismatch throws.

Returns

bool

true when symbol matched; otherwise false.

ReadToken(bool)

Reads the next token.

public string? ReadToken(bool ignoreWhiteSpace = true)

Parameters

ignoreWhiteSpace bool

When true, skips configured whitespace characters.

Returns

string

The token text, or null when end-of-content is reached.

Reset()

Resets tokenizer state.

public void Reset()