pic_tokenizer Module

Splitting of character data into tokens and parsing of those tokens into numbers with explicit error reporting through error_t.

The parsing routines never fail silently: every malformed input produces an ERROR_PARSE error with a human readable message, and the output value is set to zero. They deliberately do not use list directed internal reads, because those accept input that is almost never what the caller meant ("1,2" reads as 1, "3*7" is a repeat count, "nan" and "infinity" are accepted, and trailing junk after a separator is ignored). Every accepted form is checked against an explicit grammar first.

Empty field policy of split

split keeps empty fields, so the number of returned elements is always the number of delimiter occurrences plus one:

  • split("a,,b", ",") returns three elements, "a", "" and "b"
  • split("", ",") returns one element, the empty string
  • split(",a", ",") returns "" and "a", split("a,", ",") returns "a" and ""
  • split("abc", ",") returns one element, "abc"

This makes join(split(text, delim), delim) reproduce text exactly (for fields without trailing blanks) and keeps positional data such as CSV records aligned. Use tokenize for the other common policy, where runs of separators collapse and empty fields are dropped.

A zero length delimiter cannot match anything, so split(text, "") returns the whole of text as a single element.


Uses

  • module~~pic_tokenizer~~UsesGraph module~pic_tokenizer pic_tokenizer module~pic_ascii pic_ascii module~pic_tokenizer->module~pic_ascii module~pic_error pic_error module~pic_tokenizer->module~pic_error module~pic_string_type pic_string_type module~pic_tokenizer->module~pic_string_type module~pic_strings pic_strings module~pic_tokenizer->module~pic_strings module~pic_types pic_types module~pic_tokenizer->module~pic_types module~pic_ascii->module~pic_types module~pic_error->module~pic_types module~pic_global_definitions pic_global_definitions module~pic_error->module~pic_global_definitions module~pic_string_type->module~pic_ascii module~pic_string_type->module~pic_types module~pic_optional_value pic_optional_value module~pic_string_type->module~pic_optional_value module~pic_strings->module~pic_ascii module~pic_strings->module~pic_string_type module~pic_strings->module~pic_types iso_c_binding iso_c_binding module~pic_strings->iso_c_binding module~pic_strings->module~pic_optional_value module~pic_types->iso_c_binding iso_fortran_env iso_fortran_env module~pic_types->iso_fortran_env module~pic_global_definitions->module~pic_types module~pic_optional_value->module~pic_types

Used by

  • module~~pic_tokenizer~~UsedByGraph module~pic_tokenizer pic_tokenizer module~pic_cli pic_cli module~pic_cli->module~pic_tokenizer

Variables

Type Visibility Attributes Name Initial
character(len=*), private, parameter :: EXPONENT_MARKERS = "eEdD"

Exponent letters accepted by parse_real; q/Q is not accepted
character(len=*), private, parameter :: SIGNS = "+-"

Characters accepted as a leading sign of a number or of an exponent

Interfaces

public interface parse_int

Parse a string into an integer, reporting failures through error_t.

The accepted grammar is optional surrounding whitespace, an optional + or - sign and one or more decimal digits, with nothing else. An empty or blank string, a missing digit, any stray character (including trailing junk such as "12abc") and a value too large for the requested kind all set ERROR_PARSE and leave value at zero.

Specific procedures exist for int32 and int64, so an argument declared integer(default_int) resolves correctly in both the default and the USE_INT8 build.

  • private pure subroutine parse_int_int32_char(str, value, err)

    Parse a character string into a 32 bit integer.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: str
    integer(kind=int32), intent(out) :: value

    Fixed width on purpose: the narrow specific procedure of the generic.
    type(error_t), intent(out) :: err

  • private pure subroutine parse_int_int32_string(str, value, err)

    Parse a string_type into a 32 bit integer.

    Arguments

    Type IntentOptional Attributes Name
    type(string_type), intent(in) :: str
    integer(kind=int32), intent(out) :: value
    type(error_t), intent(out) :: err

  • private pure subroutine parse_int_int64_char(str, value, err)

    Parse a character string into a 64 bit integer.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: str
    integer(kind=int64), intent(out) :: value

    Fixed width on purpose: this is the widest integer the parser accumulates in, and the narrower specific procedure range checks against it.
    type(error_t), intent(out) :: err

  • private pure subroutine parse_int_int64_string(str, value, err)

    Parse a string_type into a 64 bit integer.

    Arguments

    Type IntentOptional Attributes Name
    type(string_type), intent(in) :: str
    integer(kind=int64), intent(out) :: value
    type(error_t), intent(out) :: err

public interface parse_real

Parse a string into a real, reporting failures through error_t.

The accepted grammar is optional surrounding whitespace, an optional sign, a mantissa holding at least one digit and at most one decimal point (so both ".5" and "5." are accepted, "." is not), and an optional exponent introduced by e, E, d or D with an optional sign and at least one digit.

Everything else is rejected, in particular the spellings that a list directed read would happily accept: "nan", "inf", "infinity", repeat counts such as "3*7", comma separated values such as "1,2", and any trailing junk. Values that overflow the requested kind are rejected as well instead of becoming infinity.

Specific procedures exist for sp and dp.

  • private subroutine parse_real_sp_char(str, value, err)

    Parse a character string into a single precision real.

    The conversion is done in double precision and then range checked, so a value that is finite in dp but too large for sp is reported as an error instead of becoming infinity.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: str
    real(kind=sp), intent(out) :: value
    type(error_t), intent(out) :: err

  • private subroutine parse_real_sp_string(str, value, err)

    Parse a string_type into a single precision real.

    Arguments

    Type IntentOptional Attributes Name
    type(string_type), intent(in) :: str
    real(kind=sp), intent(out) :: value
    type(error_t), intent(out) :: err

  • private subroutine parse_real_dp_char(str, value, err)

    Parse a character string into a double precision real.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: str
    real(kind=dp), intent(out) :: value
    type(error_t), intent(out) :: err

  • private subroutine parse_real_dp_string(str, value, err)

    Parse a string_type into a double precision real.

    Arguments

    Type IntentOptional Attributes Name
    type(string_type), intent(in) :: str
    real(kind=dp), intent(out) :: value
    type(error_t), intent(out) :: err

public interface split

Split a string on every occurrence of a delimiter, keeping empty fields.

The result always holds one more element than there are delimiter occurrences, so an empty input yields one empty element. See the module documentation for the full policy.

  • private pure function split_char_char(string, delimiter) result(parts)

    Split a character string on a character delimiter.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: string
    character(len=*), intent(in) :: delimiter

    Return Value type(string_type), allocatable, (:)

  • private pure function split_char_string(string, delimiter) result(parts)

    Split a character string on a string_type delimiter.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: string
    type(string_type), intent(in) :: delimiter

    Return Value type(string_type), allocatable, (:)

  • private pure function split_string_char(string, delimiter) result(parts)

    Split a string_type on a character delimiter.

    Arguments

    Type IntentOptional Attributes Name
    type(string_type), intent(in) :: string
    character(len=*), intent(in) :: delimiter

    Return Value type(string_type), allocatable, (:)

  • private pure function split_string_string(string, delimiter) result(parts)

    Split a string_type on a string_type delimiter.

    Arguments

    Type IntentOptional Attributes Name
    type(string_type), intent(in) :: string
    type(string_type), intent(in) :: delimiter

    Return Value type(string_type), allocatable, (:)

public interface tokenize

Split a string into whitespace separated words.

Runs of whitespace are treated as a single separator and empty fields are discarded, so a string that is empty or entirely whitespace yields a zero sized result. Whitespace is the set defined by pic_ascii, that is space, horizontal tab, vertical tab, carriage return, line feed and form feed.

  • private pure function tokenize_char(string) result(tokens)

    Split a character string into whitespace separated words.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: string

    Return Value type(string_type), allocatable, (:)

  • private pure function tokenize_string(string) result(tokens)

    Split a string_type into whitespace separated words.

    Arguments

    Type IntentOptional Attributes Name
    type(string_type), intent(in) :: string

    Return Value type(string_type), allocatable, (:)


Functions

private pure function split_char_char(string, delimiter) result(parts)

Split a character string on a character delimiter.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: string
character(len=*), intent(in) :: delimiter

Return Value type(string_type), allocatable, (:)

private pure function split_char_string(string, delimiter) result(parts)

Split a character string on a string_type delimiter.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: string
type(string_type), intent(in) :: delimiter

Return Value type(string_type), allocatable, (:)

private pure function split_string_char(string, delimiter) result(parts)

Split a string_type on a character delimiter.

Arguments

Type IntentOptional Attributes Name
type(string_type), intent(in) :: string
character(len=*), intent(in) :: delimiter

Return Value type(string_type), allocatable, (:)

private pure function split_string_string(string, delimiter) result(parts)

Split a string_type on a string_type delimiter.

Arguments

Type IntentOptional Attributes Name
type(string_type), intent(in) :: string
type(string_type), intent(in) :: delimiter

Return Value type(string_type), allocatable, (:)

private pure function tokenize_char(string) result(tokens)

Split a character string into whitespace separated words.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: string

Return Value type(string_type), allocatable, (:)

private pure function tokenize_string(string) result(tokens)

Split a string_type into whitespace separated words.

Arguments

Type IntentOptional Attributes Name
type(string_type), intent(in) :: string

Return Value type(string_type), allocatable, (:)


Subroutines

private pure subroutine check_exponent(text, pos, err)

Validate the exponent part of a real, starting at pos.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: text
integer(kind=default_int), intent(in) :: pos
type(error_t), intent(inout) :: err

private pure subroutine parse_int_int32_char(str, value, err)

Parse a character string into a 32 bit integer.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: str
integer(kind=int32), intent(out) :: value

Fixed width on purpose: the narrow specific procedure of the generic.
type(error_t), intent(out) :: err

private pure subroutine parse_int_int32_string(str, value, err)

Parse a string_type into a 32 bit integer.

Arguments

Type IntentOptional Attributes Name
type(string_type), intent(in) :: str
integer(kind=int32), intent(out) :: value
type(error_t), intent(out) :: err

private pure subroutine parse_int_int64_char(str, value, err)

Parse a character string into a 64 bit integer.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: str
integer(kind=int64), intent(out) :: value

Fixed width on purpose: this is the widest integer the parser accumulates in, and the narrower specific procedure range checks against it.
type(error_t), intent(out) :: err

private pure subroutine parse_int_int64_string(str, value, err)

Parse a string_type into a 64 bit integer.

Arguments

Type IntentOptional Attributes Name
type(string_type), intent(in) :: str
integer(kind=int64), intent(out) :: value
type(error_t), intent(out) :: err

private pure subroutine parse_integer_wide(str, value, err)

Core integer parser, accumulating in int64.

Read more…

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: str
integer(kind=int64), intent(out) :: value
type(error_t), intent(out) :: err

private subroutine parse_real_dp_char(str, value, err)

Parse a character string into a double precision real.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: str
real(kind=dp), intent(out) :: value
type(error_t), intent(out) :: err

private subroutine parse_real_dp_string(str, value, err)

Parse a string_type into a double precision real.

Arguments

Type IntentOptional Attributes Name
type(string_type), intent(in) :: str
real(kind=dp), intent(out) :: value
type(error_t), intent(out) :: err

private subroutine parse_real_sp_char(str, value, err)

Parse a character string into a single precision real.

Read more…

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: str
real(kind=sp), intent(out) :: value
type(error_t), intent(out) :: err

private subroutine parse_real_sp_string(str, value, err)

Parse a string_type into a single precision real.

Arguments

Type IntentOptional Attributes Name
type(string_type), intent(in) :: str
real(kind=sp), intent(out) :: value
type(error_t), intent(out) :: err

private subroutine parse_real_wide(str, value, err)

Core real parser.

Read more…

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: str
real(kind=dp), intent(out) :: value
type(error_t), intent(out) :: err

private pure subroutine skip_sign(text, pos)

Advance pos past a leading + or - if one is present.

Arguments

Type IntentOptional Attributes Name
character(len=*), intent(in) :: text
integer(kind=default_int), intent(inout) :: pos