matt/agila_open_tool
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial project import

Browse Source
This commit is contained in:
Agila Forge Importer committed 2026-07-13 15:20:32 +00:00
commit 23f2d29e32
11957 files changed
+1754577
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tools/v/vlib/regex/README.md
+829
View File
@@ -0,0 +1,829 @@
# Description
`regex` is a small but powerful regular expression library,
written in pure V.
> **Note**
> `regex` is *not* PCRE compatible.
[TOC]
## Introduction
Here are the assumptions made during the writing of the implementation, that
are valid for all the `regex` module features:
1. The matching stops at the end of the string, *not* at newline characters.
2. The basic atomic elements of this regex engine are the tokens.
In a query string a simple character is a token.
## Differences with PCRE:
> **Note**
> We must point out that the **V-Regex module is not PCRE compliant** and thus
> some behaviour will be different. This difference is due to the V philosophy,
> to have one way and keep it simple.
The main differences can be summarized in the following points:
- The basic element **is the token not the sequence of symbols**, and the most
simple token, is a single character.
- `|` **the OR operator acts on tokens,** for example `abc|ebc` is not
`abc` OR `ebc`. Instead it is evaluated like `ab`, followed by `c OR e`,
followed by `bc`, because the **token is the base element**,
not the sequence of symbols.
Note: **Two char classes with an `OR` in the middle is a syntax error.**
- The **match operation stops at the end of the string**. It does *NOT* stop
at new line characters.
## Tokens
The tokens are the atomic units, used by this regex engine.
They can be one of the following:
### Simple char
This token is a simple single character like `a` or `b` etc.
### Match positional delimiters
`^` Matches the start of the string.
`$` Matches the end of the string.
### Char class (cc)
The character classes match all the chars specified inside. Use square
brackets `[ ]` to enclose them.
The sequence of the chars in the character class, is evaluated with an OR op.
For example, the cc `[abc]`, matches any character, that is `a` or `b` or `c`,
but it doesn't match `C` or `z`.
Inside a cc, it is possible to specify a "range" of characters, for example
`[ad-h]` is equivalent to writing `[adefgh]`.
A cc can have different ranges at the same time, for example `[a-zA-Z0-9]`
matches all the latin lowercase, uppercase and numeric characters.
It is possible to negate the meaning of a cc, using the caret char at the
start of the cc like this: `[^abc]` . That matches every char that is NOT
`a` or `b` or `c`.
A cc can contain meta-chars like: `[a-z\d]`, that match all the lowercase
latin chars `a-z` and all the digits `\d`.
It is possible to mix all the properties of the char class together.
> **Note**
> In order to match the `-` (minus) char, it must be preceded by
> a backslash in the cc, for example `[\-_\d\a]` will match:
> - `-` minus,
> - `_` underscore,
> - `\d` numeric chars,
> - `\a` lower case chars.
### Meta-chars
A meta-char is specified by a backslash, before a character.
For example `\w` is the meta-char `w`.
A meta-char can match different types of characters.
- `\w` matches a word char `[a-zA-Z0-9_]`
- `\W` matches a non word char
- `\d` matches a digit `[0-9]`
- `\D` matches a non digit
- `\s` matches a space char, one of `[' ','\t','\n','\r','\v','\f']`
- `\S` matches a non space char
- `\a` matches only a lowercase char `[a-z]`
- `\A` matches only an uppercase char `[A-Z]`
- `\x41` match a byte of value 0x41, `A` in ascii code
- `\X414C` match two consecutive bytes of value 0x414c, `AL` in ascii code
### Quantifier
Each token can have a quantifier, that specifies how many times the character
must be matched.
#### **Short quantifiers**
- `?` matches 0 or 1 time, `a?b` matches both `ab` or `b`
- `+` matches *at least* 1 time, for example, `a+` matches both `aaa` or `a`
- `*` matches 0 or more times, for example, `a*b` matches `aaab`, `ab` or `b`
#### **Long quantifiers**
- `{x}` matches exactly x times, `a{2}` matches `aa`, but not `aaa` or `a`
- `{min,}` matches at least min times, `a{2,}` matches `aaa` or `aa`, not `a`
- `{,max}` matches at least 0 times and at maximum max times,
for example, `a{,2}` matches `a` and `aa`, but doesn't match `aaa`
- `{min,max}` matches from min times, to max times, for example
`a{2,3}` matches `aa` and `aaa`, but doesn't match `a` or `aaaa`
A long quantifier, may have a `greedy off` flag, that is the `?`
character after the brackets. `{2,4}?` means to match the minimum
number of possible tokens, in this case 2.
### Dot char
The dot is a particular meta-char, that matches "any char".
It is simpler to explain it with an example:
Suppose you have `abccc ddeef` as a source string, that you want to parse
with a regex. The following table shows the query strings and the result of
parsing source string.
| query string | result |
| ------------ | ----------- |
| `.*c` | `abc` |
| `.*dd` | `abcc dd` |
| `ab.*e` | `abccc dde` |
| `ab.{3} .*e` | `abccc dde` |
The dot matches any character, until the next token match is satisfied.
> Important Note: Consecutive dots, for example `...`, are not allowed.
> This will cause a syntax error. Use a quantifier instead.
### OR token
The token `|`, means a logic OR operation between two consecutive tokens,
i.e. `a|b` matches a character that is `a` or `b`.
The OR token can work in a "chained way": `a|(b)|cd ` means test first `a`,
if the char is not `a`, then test the group `(b)`, and if the group doesn't
match too, finally test the token `c`.
> **Note**
> Unlike in PCRE, the OR operation works at token level!
> It doesn't work at concatenation level!
> **Note**
> Two char classes with an `OR` in the middle is a syntax error.
That also means, that a query string like `abc|bde` is not equal to
`(abc)|(bde)`, but instead to `ab(c|b)de`.
The OR operation works only for `c|b`, not at char concatenation level.
### Groups
Groups are a method to create complex patterns with repetitions of blocks
of tokens. The groups are delimited by round brackets `( )`. Groups can be
nested. Like all other tokens, groups can have a quantifier too.
`c(pa)+z` match `cpapaz` or `cpaz` or `cpapapaz` .
`(c(pa)+z ?)+` matches `cpaz cpapaz cpapapaz` or `cpapaz`
Lets analyze this last case, first we have the group `#0`, that is the most
outer round brackets `(...)+`. This group has a quantifier `+`, that says to
match its content *at least one time*.
Then we have a simple char token `c`, and a second group `#1`: `(pa)+`.
This group also tries to match the sequence `pa`, *at least one time*,
as specified by the `+` quantifier.
Then, we have another simple token `z` and another simple token ` ?`,
i.e. the space char (ascii code 32) followed by the `?` quantifier,
which means that the preceding space should be matched 0 or 1 time.
This explains why the `(c(pa)+z ?)+` query string,
can match `cpaz cpapaz cpapapaz`.
In this implementation the groups are "capture groups". This means that the
last temporal result for each group, can be retrieved from the `RE` struct.
The "capture groups" are stored as indexes in the field `groups`,
that is an `[]int` inside the `RE` struct.
**example:**
```v oksyntax
import regex
text := 'cpaz cpapaz cpapapaz'
query := r'(c(pa)+z ?)+'
mut re := regex.regex_opt(query) or { panic(err) }
println(re.get_query())
// #0(c#1(pa)+z ?)+
// #0 and #1 are the ids of the groups, are shown if re.debug is 1 or 2
start, end := re.match_string(text)
// [start=0, end=20] match => [cpaz cpapaz cpapapaz]
mut gi := 0
for gi < re.groups.len {
if re.groups[gi] >= 0 {
println('${gi / 2} :[${text[re.groups[gi]..re.groups[gi + 1]]}]')
}
gi += 2
}
// groups captured
// 0 :[cpapapaz]
// 1 :[pa]
```
> **Note**
> To show the `group id number` in the result of the `get_query()`
> the flag `debug` of the RE object must be `1` or `2`
In order to simplify the use of the captured groups, it is possible to use the
utility function: `get_group_list`.
This function returns a list of groups using this support struct:
```v oksyntax
pub struct Re_group {
pub:
start int = -1
end int = -1
}
```
Here is an example of use:
```v oksyntax
import regex
/*
This simple function converts an HTML RGB value with 3 or 6 hex digits to
an u32 value, this function is not optimized and it is only for didatical
purpose. Example: #A0B0CC #A9F
*/
fn convert_html_rgb(in_col string) u32 {
mut n_digit := if in_col.len == 4 { 1 } else { 2 }
mut col_mul := if in_col.len == 4 { 4 } else { 0 }
// this is the regex query, it uses the V string interpolation to customize the regex query
// Note: If you want to use escaped code you must use the r"" (raw) strings,
// *** please remember that the V interpoaltion doesn't work on raw strings. ***
query := '#([a-fA-F0-9]{${n_digit}})([a-fA-F0-9]{${n_digit}})([a-fA-F0-9]{${n_digit}})'
mut re := regex.regex_opt(query) or { panic(err) }
start, end := re.match_string(in_col)
println('start: ${start}, end: ${end}')
mut res := u32(0)
if start >= 0 {
group_list := re.get_group_list() // this is the utility function
r := ('0x' + in_col[group_list[0].start..group_list[0].end]).int() << col_mul
g := ('0x' + in_col[group_list[1].start..group_list[1].end]).int() << col_mul
b := ('0x' + in_col[group_list[2].start..group_list[2].end]).int() << col_mul
println('r: ${r} g: ${g} b: ${b}')
res = u32(r) << 16 | u32(g) << 8 | u32(b)
}
return res
}
```
Other utility functions are `get_group_by_id` and `get_group_bounds_by_id`
that get directly the string of a group using its `id`:
```v ignore
txt := 'my used string....'
for g_index := 0; g_index < re.group_count; g_index++ {
println('#${g_index} [${re.get_group_by_id(txt, g_index)}] \
bounds: ${re.get_group_bounds_by_id(g_index)}')
}
```
More helper functions are listed in the **Groups query functions** section.
### Groups Continuous saving
In particular situations, it is useful to have a continuous group saving.
This is possible by initializing the `group_csave` field in the `RE` struct.
This feature allows you to collect data in a continuous/streaming way.
In the example, we can pass a text, followed by an integer list,
that we wish to collect. To achieve this task, we can use the continuous
group saving, by enabling the right flag: `re.group_csave_flag = true`.
The `.group_csave` array will be filled then, following this logic:
`re.group_csave[0]` - number of total saved records
`re.group_csave[1+n*3]` - id of the saved group
`re.group_csave[2+n*3]` - start index in the source string of the saved group
`re.group_csave[3+n*3]` - end index in the source string of the saved group
The regex will save groups, until it finishes, or finds that the array has no
more space. If the space ends, no error is raised, and further records will
not be saved.
```v ignore
import regex
fn main() {
txt := 'http://www.ciao.mondo/hello/pippo12_/pera.html'
query := r'(?P<format>https?)|(?P<format>ftps?)://(?P<token>[\w_]+.)+'
mut re := regex.regex_opt(query) or { panic(err) }
// println(re.get_code()) // uncomment to see the print of the regex execution code
re.debug = 2 // enable maximum log
println('String: ${txt}')
println('Query : ${re.get_query()}')
re.debug = 0 // disable log
re.group_csave_flag = true
start, end := re.match_string(txt)
if start >= 0 {
println('Match (${start}, ${end}) => [${txt[start..end]}]')
} else {
println('No Match')
}
if re.group_csave_flag == true && start >= 0 && re.group_csave.len > 0 {
println('cg: ${re.group_csave}')
mut cs_i := 1
for cs_i < re.group_csave[0] * 3 {
g_id := re.group_csave[cs_i]
st := re.group_csave[cs_i + 1]
en := re.group_csave[cs_i + 2]
println('cg[${g_id}] ${st} ${en}:[${txt[st..en]}]')
cs_i += 3
}
}
}
```
The output will be:
```
String: http://www.ciao.mondo/hello/pippo12_/pera.html
Query : #0(?P<format>https?)|{8,14}#0(?P<format>ftps?)://#1(?P<token>[\w_]+.)+
Match (0, 46) => [http://www.ciao.mondo/hello/pippo12_/pera.html]
cg: [8, 0, 0, 4, 1, 7, 11, 1, 11, 16, 1, 16, 22, 1, 22, 28, 1, 28, 37, 1, 37, 42, 1, 42, 46]
cg[0] 0 4:[http]
cg[1] 7 11:[www.]
cg[1] 11 16:[ciao.]
cg[1] 16 22:[mondo/]
cg[1] 22 28:[hello/]
cg[1] 28 37:[pippo12_/]
cg[1] 37 42:[pera.]
cg[1] 42 46:[html]
```
### Named capturing groups
This regex module supports partially the question mark `?` PCRE syntax for groups.
`(?:abcd)` **non capturing group**: the content of the group will not be saved.
`(?P<mygroup>abcdef)` **named group:** the group content is saved and labeled
as `mygroup`.
The label of the groups is saved in the `group_map` of the `RE` struct,
that is a map from `string` to `int`, where the value is the index in
`group_csave` list of indexes.
Here is an example for how to use them:
```v ignore
import regex
fn main() {
txt := 'http://www.ciao.mondo/hello/pippo12_/pera.html'
query := r'(?P<format>https?)|(?P<format>ftps?)://(?P<token>[\w_]+.)+'
mut re := regex.regex_opt(query) or { panic(err) }
// println(re.get_code()) // uncomment to see the print of the regex execution code
re.debug = 2 // enable maximum log
println('String: ${txt}')
println('Query : ${re.get_query()}')
re.debug = 0 // disable log
start, end := re.match_string(txt)
if start >= 0 {
println('Match (${start}, ${end}) => [${txt[start..end]}]')
} else {
println('No Match')
}
for name in re.group_map.keys() {
println("group:'${name}' \t=> [${re.get_group_by_name(txt, name)}] \
bounds: ${re.get_group_bounds_by_name(name)}")
}
}
```
Output:
```
String: http://www.ciao.mondo/hello/pippo12_/pera.html
Query : #0(?P<format>https?)|{8,14}#0(?P<format>ftps?)://#1(?P<token>[\w_]+.)+
Match (0, 46) => [http://www.ciao.mondo/hello/pippo12_/pera.html]
group:'format' => [http] bounds: (0, 4)
group:'token' => [html] bounds: (42, 46)
```
In order to simplify the use of the named groups, it is possible to
use a name map in the `re` struct, using the function `re.get_group_by_name`.
Here is a more complex example of using them:
```v oksyntax
import regex
// This function demonstrate the use of the named groups
fn convert_html_rgb_n(in_col string) u32 {
mut n_digit := if in_col.len == 4 { 1 } else { 2 }
mut col_mul := if in_col.len == 4 { 4 } else { 0 }
query := '#(?P<red>[a-fA-F0-9]{${n_digit}})' + '(?P<green>[a-fA-F0-9]{${n_digit}})' +
'(?P<blue>[a-fA-F0-9]{${n_digit}})'
mut re := regex.regex_opt(query) or { panic(err) }
start, end := re.match_string(in_col)
println('start: ${start}, end: ${end}')
mut res := u32(0)
if start >= 0 {
red_s, red_e := re.get_group_by_name('red')
r := ('0x' + in_col[red_s..red_e]).int() << col_mul
green_s, green_e := re.get_group_by_name('green')
g := ('0x' + in_col[green_s..green_e]).int() << col_mul
blue_s, blue_e := re.get_group_by_name('blue')
b := ('0x' + in_col[blue_s..blue_e]).int() << col_mul
println('r: ${r} g: ${g} b: ${b}')
res = u32(r) << 16 | u32(g) << 8 | u32(b)
}
return res
}
```
Other utilities are `get_group_by_name` and `get_group_bounds_by_name`,
that return the string of a group using its `name`:
```v ignore
txt := 'my used string....'
for name in re.group_map.keys() {
println("group:'${name}' \t=> [${re.get_group_by_name(txt, name)}] \
bounds: ${re.get_group_bounds_by_name(name)}")
}
```
### Groups query functions
These functions are helpers to query the captured groups
```v ignore
// get_group_bounds_by_name get a group boundaries by its name
pub fn (re RE) get_group_bounds_by_name(group_name string) (int, int)
// get_group_by_name get a group string by its name
pub fn (re RE) get_group_by_name(group_name string) string
// get_group_by_id get a group boundaries by its id
pub fn (re RE) get_group_bounds_by_id(group_id int) (int,int)
// get_group_by_id get a group string by its id
pub fn (re RE) get_group_by_id(in_txt string, group_id int) string
struct Re_group {
pub:
start int = -1
end int = -1
}
// get_group_list return a list of Re_group for the found groups
pub fn (re RE) get_group_list() []Re_group
```
## Flags
It is possible to set some flags in the regex parser, that change
the behavior of the parser itself.
```v ignore
// example of flag settings
mut re := regex.new()
re.flag = regex.f_bin
```
- `f_bin`: parse a string as bytes, utf-8 management disabled.
- `f_ci`: match ASCII letters without case sensitivity.
- `f_efm`: exit on the first char matches in the query, used by the
find function.
- `f_ms`: matches only if the index of the start match is 0,
same as `^` at the start of the query string.
- `f_me`: matches only if the end index of the match is the last char
of the input string, same as `$` end of query string.
- `f_nl`: stop the matching if found a new line char `\n` or `\r`
## Functions
### Initializer
These functions are helpers that create the `RE` struct.
A `RE` struct can be created manually if needed.
#### **Simplified initializer**
```v ignore
// regex create a regex object from the query string and compile it
pub fn regex_opt(in_query string) ?RE
```
#### **Base initializer**
```v ignore
// new create a RE of small size, usually sufficient for ordinary use
pub fn new() RE
```
### Compiling
After an initializer is used, the regex expression must be compiled with:
```v ignore
// compile_opt compile RE pattern string, returning an error if the compilation fails
pub fn (mut re RE) compile_opt(pattern string) !
```
### Matching Functions
These are the matching functions
```v ignore
// match_string try to match the input string, return start and end index if found else start is -1
pub fn (mut re RE) match_string(in_txt string) (int, int)
```
## Find and Replace
There are the following find and replace functions:
#### Find functions
```v ignore
// find try to find the first match in the input string
// return start and end index if found else start is -1
pub fn (mut re RE) find(in_txt string) (int, int)
// find_all find all the "non overlapping" occurrences of the matching pattern
// return a list of start end indexes like: [3,4,6,8]
// the matches are [3,4] and [6,8]
pub fn (mut re RE) find_all(in_txt string) []int
// find_all_str find all the "non overlapping" occurrences of the match pattern
// return a list of strings
// the result is like ['first match','secon match']
pub fn (mut re RE) find_all_str(in_txt string) []string
```
#### Replace functions
```v ignore
// replace return a string where the matches are replaced with the repl_str string,
// this function supports groups in the replace string
pub fn (mut re RE) replace(in_txt string, repl string) string
```
replace string can include groups references:
```v ignore
txt := 'Today it is a good day.'
query := r'(a\w)[ ,.]'
mut re := regex.regex_opt(query)?
res := re.replace(txt, r'__[\0]__')
```
in this example we used the group `0` in the replace string: `\0`, the result will be:
```
Today it is a good day. => Tod__[ay]__it is a good d__[ay]__
```
> **Note**
> In the replace strings can be used only groups from `0` to `9`.
If the usage of `groups` in the replace process, is not needed, it is possible
to use a quick function:
```v ignore
// replace_simple return a string where the matches are replaced with the replace string
pub fn (mut re RE) replace_simple(in_txt string, repl string) string
```
If it is needed to replace N instances of the found strings it is possible to use:
```v ignore
// replace_n return a string where the first `count` matches are replaced with the repl_str string
// `count` indicate the number of max replacements that will be done.
// if count is > 0 the replace began from the start of the string toward the end
// if count is < 0 the replace began from the end of the string toward the start
// if count is 0 do nothing
pub fn (mut re RE) replace_n(in_txt string, repl_str string, count int) string
```
#### Custom replace function
For complex find and replace operations, you can use `replace_by_fn` .
The `replace_by_fn`, uses a custom replace callback function, thus
allowing customizations. The custom callback function is called for
every non overlapped find.
The custom callback function must be of the type:
```v ignore
// type of function used for custom replace
// in_txt source text
// start index of the start of the match in in_txt
// end index of the end of the match in in_txt
// --- the match is in in_txt[start..end] ---
fn (re RE, in_txt string, start int, end int) string
```
The following example will clarify its usage:
```v ignore
import regex
// customized replace functions
// it will be called on each non overlapped find
fn my_repl(re regex.RE, in_txt string, start int, end int) string {
g0 := re.get_group_by_id(in_txt, 0)
g1 := re.get_group_by_id(in_txt, 1)
g2 := re.get_group_by_id(in_txt, 2)
return '*${g0}*${g1}*${g2}*'
}
fn main() {
txt := 'today [John] is gone to his house with (Jack) and [Marie].'
query := r'(.)(\A\w+)(.)'
mut re := regex.regex_opt(query) or { panic(err) }
result := re.replace_by_fn(txt, my_repl)
println(result)
}
```
Output:
```
today *[*John*]* is gone to his house with *(*Jack*)* and *[*Marie*]*.
```
## Debugging
This module has few small utilities to you write regex patterns.
### **Syntax errors highlight**
The next example code shows how to visualize regex pattern syntax errors
in the compilation phase:
```v oksyntax
query := r'ciao da ab[ab-]'
// there is an error, a range not closed!!
mut re := new()
re.compile_opt(query) or { println(err) }
// output!!
// query: ciao da ab[ab-]
// err : ----------^
// ERROR: ERR_SYNTAX_ERROR
```
### **Compiled code**
It is possible to view the compiled code calling the function `get_query()`.
The result will be something like this:
```
========================================
v RegEx compiler v 1.0 alpha output:
PC: 0 ist: 92000000 ( GROUP_START #:0 { 1, 1}
PC: 1 ist: 98000000 . DOT_CHAR nx chk: 4 { 1, 1}
PC: 2 ist: 94000000 ) GROUP_END #:0 { 1, 1}
PC: 3 ist: 92000000 ( GROUP_START #:1 { 1, 1}
PC: 4 ist: 90000000 [\A] BSLS { 1, 1}
PC: 5 ist: 90000000 [\w] BSLS { 1,MAX}
PC: 6 ist: 94000000 ) GROUP_END #:1 { 1, 1}
PC: 7 ist: 92000000 ( GROUP_START #:2 { 1, 1}
PC: 8 ist: 98000000 . DOT_CHAR nx chk: -1 last! { 1, 1}
PC: 9 ist: 94000000 ) GROUP_END #:2 { 1, 1}
PC: 10 ist: 88000000 PROG_END { 0, 0}
========================================
```
`PC`:`int` is the program counter or step of execution, each single step is a token.
`ist`:`hex` is the token instruction id.
`[a]` is the char used by the token.
`query_ch` is the type of token.
`{m,n}` is the quantifier, the greedy off flag `?` will be showed if present in the token
### **Log debug**
The log debugger allow to print the status of the regex parser when the
parser is running. It is possible to have two different levels of
debug information: 1 is normal, while 2 is verbose.
Here is an example:
*normal* - list only the token instruction with their values
```ignore
// re.flag = 1 // log level normal
flags: 00000000
# 2 s: ist_load PC: i,ch,len:[ 0,'a',1] f.m:[ -1, -1] query_ch: [a]{1,1}:0 (#-1)
# 5 s: ist_load PC: i,ch,len:[ 1,'b',1] f.m:[ 0, 0] query_ch: [b]{2,3}:0? (#-1)
# 7 s: ist_load PC: i,ch,len:[ 2,'b',1] f.m:[ 0, 1] query_ch: [b]{2,3}:1? (#-1)
# 10 PROG_END
```
*verbose* - list all the instructions and states of the parser
```ignore
flags: 00000000
# 0 s: start PC: NA
# 1 s: ist_next PC: NA
# 2 s: ist_load PC: i,ch,len:[ 0,'a',1] f.m:[ -1, -1] query_ch: [a]{1,1}:0 (#-1)
# 3 s: ist_quant_p PC: i,ch,len:[ 1,'b',1] f.m:[ 0, 0] query_ch: [a]{1,1}:1 (#-1)
# 4 s: ist_next PC: NA
# 5 s: ist_load PC: i,ch,len:[ 1,'b',1] f.m:[ 0, 0] query_ch: [b]{2,3}:0? (#-1)
# 6 s: ist_quant_p PC: i,ch,len:[ 2,'b',1] f.m:[ 0, 1] query_ch: [b]{2,3}:1? (#-1)
# 7 s: ist_load PC: i,ch,len:[ 2,'b',1] f.m:[ 0, 1] query_ch: [b]{2,3}:1? (#-1)
# 8 s: ist_quant_p PC: i,ch,len:[ 3,'b',1] f.m:[ 0, 2] query_ch: [b]{2,3}:2? (#-1)
# 9 s: ist_next PC: NA
# 10 PROG_END
# 11 PROG_END
```
the columns have the following meaning:
`# 2` number of actual steps from the start of parsing
`s: ist_next` state of the present step
`PC: 1` program counter of the step
`=>7fffffff ` hex code of the instruction
`i,ch,len:[ 0,'a',1]` `i` index in the source string, `ch` the char parsed,
`len` the length in byte of the char parsed
`f.m:[ 0, 1]` `f` index of the first match in the source string, `m` index that is actual matching
`query_ch: [b]` token in use and its char
`{2,3}:1?` quantifier `{min,max}`, `:1` is the actual counter of repetition,
`?` is the greedy off flag if present.
### **Custom Logger output**
The debug functions output uses the `stdout` as default,
it is possible to provide an alternative output, by setting a custom
output function:
```v oksyntax
// custom print function, the input will be the regex debug string
fn custom_print(txt string) {
println('my log: ${txt}')
}
mut re := new()
re.log_func = custom_print
// every debug output from now will call this function
```
## Example code
Here an example that perform some basically match of strings
```v ignore
import regex
fn main() {
txt := 'http://www.ciao.mondo/hello/pippo12_/pera.html'
query := r'(?P<format>https?)|(?P<format>ftps?)://(?P<token>[\w_]+.)+'
mut re := regex.regex_opt(query) or { panic(err) }
start, end := re.match_string(txt)
if start >= 0 {
println('Match (${start}, ${end}) => [${txt[start..end]}]')
for g_index := 0; g_index < re.group_count; g_index++ {
println('#${g_index} [${re.get_group_by_id(txt, g_index)}] \
bounds: ${re.get_group_bounds_by_id(g_index)}')
}
for name in re.group_map.keys() {
println("group:'${name}' \t=> [${re.get_group_by_name(txt, name)}] \
bounds: ${re.get_group_bounds_by_name(name)}")
}
} else {
println('No Match')
}
}
```
More examples are available in the test code for the `regex` module,
see `vlib/regex/regex_test.v`.
tools/v/vlib/regex/pcre/README.md
+173
View File
@@ -0,0 +1,173 @@
# regex.pcre Module Documentation
The `regex.pcre` module is a high-performance **Virtual Machine (VM)**
based regular expression engine for V.
### Key Features
- **Non-recursive VM**: Safe execution that avoids stack overflows on complex patterns.
- **Zero-Allocation Search**: Uses a pre-allocated `Machine` workspace for search operations.
- **Fast ASCII Path**: Optimized path for characters < 128 to bypass heavy UTF-8 decoding.
- **Bitmap Lookups**: ASCII character classes use a 128-bit bitset for $O(1)$ matching.
- **Instruction Merging**: Consecutive character matches are merged
into string blocks for faster execution.
- **Bitmap lookups**: ASCII character classes use a 128-bit bitset for O(1) matching.
- **NFA Virtual Machine**: Executes bytecode instructions to simulate pattern matching.
- **Dynamic Stack Growth**: Automatically expands the backtracking stack to prevent false negatives.
- **Zero-Allocation Search**: Reuses a pre-allocated Machine workspace for search operations.
- **Anchored Optimization**: Patterns starting with '^' skip the scanning loop.
- **Prefix Skipping**: Uses Boyer-Moore-like skipping for literal prefixes.
## Supported Syntax
| Feature | Syntax | Description |
| :--- | :--- | :--- |
| **Literals** | `abc` | Matches exact characters (UTF-8 supported). |
| **Wildcard** | `.` | Matches any character (excluding `\n` unless `(?s)` flag is used). |
| **Alternation** | `|` | Matches the left OR right expression (e.g., `cat|dog`). |
| **Quantifiers** | `*`, `+`, `?` | Matches 0+, 1+, or 0-1 times. |
| **Lazy** | `*?`, `+?`, `??` | Non-greedy versions of the above. |
| **Repetition** | `{m,n}` | Matches between `m` and `n` times. `{m,}` for m or more. |
| **Groups** | `(...)` | Capturing group. |
| | `(?:...)` | Non-capturing group. |
| | `(?P<name>...)` | Named capturing group. |
| **Anchors** | `^`, `$` | Start/End of string (or line with `(?m)`). |
| | `\b`, `\B` | Word boundary and Non-word boundary. |
| **Classes** | `[abc]`, `[^abc]` | Character set and Negated character set. |
| | `[a-z]` | Range of characters. |
| | `\w`, `\W` | Word / Non-word (`[a-zA-Z0-9_]`). |
| | `\d`, `\D` | Digit / Non-digit. |
| | `\s`, `\S` | Whitespace / Non-whitespace (` \t\n\r\v\f`). |
| | `\a`, `\A` | Lowercase / Uppercase ASCII character class. |
| **Flags** | `(?i)` | Case-insensitive matching. |
| | `(?m)` | Multiline mode (`^` and `$` match start/end of lines). |
| | `(?s)` | Dot-all mode (`.` matches newlines). |
---
## Structs
### Regex
The compiled regular expression object.
```v ignore
pub struct Regex {
pub:
pattern string // The original pattern
prog []Inst // Compiled VM bytecode
total_groups int // Number of capture groups
group_map map[string]int // Map for named groups
}
```
### Match
Represents the result of a successful search.
```v ignore
pub struct Match {
pub:
text string // The full substring that matched
start int // Byte index where match starts
end int // Byte index where match ends
groups []string // Text captured by each group
}
```
---
## Core Functions
### `compile`
Compiles a pattern into a `Regex` object.
```v ignore
fn compile(pattern string) !Regex
```
### `find`
Finds the first match in the text. Returns `none` if no match is found.
```v ignore
fn (r Regex) find(text string) ?Match
```
### `find_all`
Returns all non-overlapping matches in a string.
```v ignore
fn (r Regex) find_all(text string) []Match
```
### `replace`
Replaces the first match in `text` with `repl`.
Supports backreferences like `$1`, `$2`.
```v ignore
fn (r Regex) replace(text string, repl string) string
```
### `max_stack_depth` (configuration field)
Controls the maximum backtracking depth for the VM.
Default is `2048`. Increase this value if complex patterns return `none` prematurely due to
deep backtracking; decrease it to limit memory usage.
```v ignore
r := pcre.compile(pattern)!
r.max_stack_depth = 4096
```
---
## Named Groups Example
```v
import regex.pcre
fn main() {
r := pcre.compile(r'(?P<year>\d{4})-(?P<month>\d{2})')!
m := r.find('Date: 2026-02') or { return }
year := r.group_by_name(m, 'year')
month := r.group_by_name(m, 'month')
println('Year: ${year}, Month: ${month}') // Year: 2026, Month: 02
}
```
---
## PCRE Compatibility Layer
To facilitate easier migration from other engines, a compatibility layer is provided:
| Function | Equivalent To |
| :--- | :--- |
| `new_regex(pattern, flags)` | `compile(pattern)` |
| `r.match_str(text, start, flags)` | `r.find_from(text, start)` |
| `m.get(idx)` | Retrieves match text (`0`) or capture group (`1+`). |
| `m.get_all()` | Returns `[full_match, group1, group2, ...]` |
**Example:**
```v
import regex.pcre
r := pcre.new_regex(r'(\w+) (\w+)', 0)!
if m := r.match_str('hello world', 0, 0) {
println(m.get(0)?) // "hello world"
println(m.get(1)?) // "hello"
println(m.get(2)?) // "world"
}
```
## Performance Note
Here is a clear summary of the optimizations implemented in the code:
* **Raw Pointer Access:** The VM bypasses standard array bounds checking by using `unsafe`
pointer arithmetic for both the instruction set and the string text, significantly speeding up
the hot loop.
* **Zero-Allocation Search:** The `Machine` struct pre-allocates the backtracking stack and
capture arrays, ensuring that running a search (finding a match) creates no new heap allocations
(garbage collection pressure is zero).
* **Fast ASCII Path:** The code checks if a byte is `< 128` before decoding. If it is ASCII, it
skips the expensive UTF-8 decoding logic entirely.
* **Bitmap Class Lookups:** Character classes (like `\w`, `\d`, `[a-z]`) use a 128-bit bitset.
Checking if an ASCII character matches a class is a single O(1) bitwise operation.
* **Instruction Merging:** The compiler groups consecutive literal characters into a single
`string` instruction (e.g., `a`, `b`, `c` becomes `"abc"`), reducing the number of VM cycles
required.
* **Prefix Skipping:** If a pattern starts with a literal string, the engine scans ahead for
that substring (Boyer-Moore style) before initializing the VM, avoiding useless execution.
* **Anchored Optimization:** If the pattern starts with `^`, the engine only attempts a match at
the start of the string (or line), skipping the character-by-character scan of the rest of the text.
tools/v/vlib/regex/pcre/regex.v
+1314
View File
File diff suppressed because it is too large. Load diff
tools/v/vlib/regex/pcre/regex_test.v
+709
View File
@@ -0,0 +1,709 @@
/*
regex_test.v
Copyright (c) 2026 Dario Deledda. All rights reserved.
Use of this source code is governed by an MIT license
that can be found in the LICENSE file.
*/
import regex.pcre
fn main() {
println('Running pcre tests...\n')
test_regex()
test_complex_quantifiers()
test_range_quantifiers()
test_anchors()
test_word_boundaries()
test_flags()
test_named_groups()
test_non_capturing_groups()
// New features tests
test_find_all()
test_find_from()
test_replace()
// test_stress_vm()
println('\nAll tests passed!')
}
// --- New Feature Tests ---
fn test_find_all() {
println('\n--- Testing find_all() ---')
// Basic extraction
tst_find_all(r'\d+', '123 abc 456', ['123', '456'])
tst_find_all(r'\w+', 'hi there', ['hi', 'there'])
// No matches
tst_find_all(r'\d+', 'no numbers', [])
// Pattern matching empty strings (e.g., boundaries)
// Note: Behavior depends on engine implementation regarding empty matches.
// Current VM advances index if match length is 0 to avoid infinite loop.
// Pattern \b matches at 0 (start), 3 (after 123), 4 (before abc), 7 (after abc)
// But find_all usually returns non-overlapping text. \b returns empty string.
// tst_find_all(r'\b', '123 abc', ['', '', '', '']) // Commented out, specific implementation detail
// Anchored find_all (should only match once if anchored at start)
tst_find_all(r'^\w+', 'word word word', ['word'])
// Overlapping logic check (find_all is typically non-overlapping)
// "ana" in "banana". Indices: 1 ("ana"). Next search starts at 4 ("na").
tst_find_all(r'ana', 'banana', ['ana'])
}
fn test_find_from() {
println('\n--- Testing find_from() ---')
text := 'test test test'
// Start from 0 (finds first)
tst_find_from(r'test', text, 0, 0, 'test')
// Start from 1 (skips first, finds second)
tst_find_from(r'test', text, 1, 5, 'test')
// Start from 6 (finds third)
tst_find_from(r'test', text, 6, 10, 'test')
// Start from end (finds nothing)
tst_find_from(r'test', text, 11, -1, 'none')
// Out of bounds
tst_find_from(r'test', text, 50, -1, 'none')
// Start exactly at match position
tst_find_from(r'test', text, 5, 5, 'test')
}
fn test_replace() {
println('\n--- Testing replace() ---')
// Simple replacement
tst_replace(r'\d+', 'abc 123 def', 'NUM', 'abc NUM def')
// Group substitution
tst_replace(r'(\w+), (\w+)', 'Doe, John', '$2 $1', 'John Doe')
// Multiple replacements?
// The current replace() implementation in the provided code
// usually replaces the FIRST occurrence (based on find()).
// Let's verify:
tst_replace(r'a', 'bananas', 'o', 'bonanas')
tst_replace(r'(^[#.]+)|([#.]+$)', r'_#abc.#_ab#', '*', '_#abc.#_ab*')
// Invalid group index (should ignore or remove)
tst_replace(r'(\d+)', '123', 'Num: $9', 'Num: ')
}
/*
fn test_stress_vm() {
println('\n--- Testing VM Stability (Stress Test) ---')
// Recursive engines often crash on patterns like (a*)* or very long strings
// if not carefully managed. The VM should handle this via heap stack.
long_text := 'a'.repeat(2000)
tst_find(r'a+', long_text, long_text)
println(' [Pass] Long string match')
// Backtracking stress
// Pattern: (a+)+b matching aaaaa....a (fails)
// This forces extensive backtracking.
short_text := 'a'.repeat(25)
mut r := pcre.compile(r'(a+)+b') or { panic(err) }
r.max_stack_depth = 4000 // increase the stack depth for this test
res := r.find(short_text)
assert res == none
println(' [Pass] Backtracking stress test')
}
*/
// --- Existing Tests ---
fn test_flags() {
println('\n--- Testing Flags ((?i), (?m), (?s)) ---')
// 1. Case Insensitive (?i)
tst_find('(?i)cat', 'Cat', 'Cat')
tst_find('(?i)CAT', 'cat', 'cat')
tst_find('(?i)[a-z]+', 'UPPER', 'UPPER') // char class expansion
tst_find('(?i)x', 'X', 'X')
tst_find('(?i)x', 'y', 'none')
// Mixed (flag applies to subsequent tokens)
tst_find('c(?i)at', 'cAT', 'cAT')
tst_find('c(?i)at', 'Cat', 'none') // first 'c' is case-sensitive
// 2. Multiline (?m)
// ^ matches start of line
tst_find('(?m)^line2', 'line1\nline2', 'line2')
tst_find('^line2', 'line1\nline2', 'none') // Default: matches only start of string
// $ matches end of line
tst_find('(?m)line1$', 'line1\nline2', 'line1')
tst_find('line1$', 'line1\nline2', 'none') // Default: matches only end of string
// 3. Dot-all / Singleline (?s)
// . matches newline
tst_find('(?s)a.b', 'a\nb', 'a\nb')
tst_find('a.b', 'a\nb', 'none') // Default: . does not match \n
// 4. Combined Flags (?im)
tst_find('(?im)^line2', 'LINE1\nLINE2', 'LINE2')
// --- Negative Tests (Flags) ---
tst_find('(?i)cat', 'dog', 'none')
tst_find('(?m)^line2', 'line1 line2', 'none') // Not at start of line
}
fn test_word_boundaries() {
println('\n--- Testing Word Boundaries (\\b and \\B) ---')
// 1. Word Boundary (\b)
tst_find('\\bcat', 'cat', 'cat')
tst_find('\\bcat', 'concat', 'none')
tst_find('\\bcat', 'catapult', 'cat')
tst_find('cat\\b', 'cat', 'cat')
tst_find('cat\\b', 'concat', 'cat')
tst_find('cat\\b', 'catapult', 'none')
tst_find('\\bcat\\b', 'cat', 'cat')
tst_find('\\bcat\\b', 'a cat is here', 'cat')
tst_find('\\bcat\\b', 'concat', 'none')
tst_find('\\bcat\\b', 'catapult', 'none')
tst_find('\\btest\\b', 'test.', 'test')
tst_find('\\btest\\b', '(test)', 'test')
// 2. Non-Word Boundary (\B)
tst_find('a\\B', 'ab', 'a')
tst_find('a\\B', 'a.', 'none')
tst_find('\\Bcat', 'concat', 'cat')
tst_find('\\Bcat', 'cat', 'none')
tst_find('cat\\B', 'catapult', 'cat')
// --- Negative Tests (Word Boundaries) ---
tst_find('\\b\\w+\\b', '... ...', 'none')
tst_find('\\B', 'a', 'none')
}
fn test_anchors() {
println('\n--- Testing Anchors (^ and $) ---')
// 1. Start of String (^)
tst_find('^abc', 'abc', 'abc')
tst_find('^abc', 'abcdef', 'abc')
tst_find('^abc', 'abc abc', 'abc')
tst_find('^\\d+', '123 text', '123')
// 2. End of String ($)
tst_find('xyz$', 'xyz', 'xyz')
tst_find('xyz$', 'abcxyz', 'xyz')
tst_find('\\d+$', 'text 123', '123')
// 3. Both Anchors (^...$)
tst_find('^hello$', 'hello', 'hello')
// 4. Zero-width matches
tst_find('^', 'abc', '')
tst_find('$', 'abc', '')
// 5. Anchors with Alternation
tst_find('^a|b$', 'apple', 'a')
tst_find('^a|b$', 'blob', 'b')
// 6. Anchors with Groups
tst_find('^(abc)+$', 'abcabc', 'abcabc')
// --- Negative Tests (Anchors) ---
tst_find('^abc', 'xyzabc', 'none')
tst_find('^\\d+', 'text 123', 'none')
tst_find('xyz$', 'xyzabc', 'none')
tst_find('\\d+$', '123 text', 'none')
tst_find('^hello$', 'hello world', 'none')
tst_find('^hello$', 'say hello', 'none')
tst_find('^a|b$', 'cba', 'none')
tst_find('^(abc)+$', 'abcabcx', 'none')
tst_find('^$', 'a', 'none')
}
fn test_regex() {
println('\n--- Testing Basic Features ---')
tst_find('a?b', 'ab', 'ab')
tst_find('a?b', 'b', 'b')
tst_find('a+b', 'aaab', 'aaab')
tst_find('a*b', 'b', 'b')
tst_find('\\d+', '123 abc', '123')
println('\n--- Testing Character Classes ---')
tst_find('\\w+', 'word1_ and', 'word1_')
tst_find('\\W+', ' and', ' ')
tst_find('\\s+', ' start', ' ')
tst_find('\\d{3}-\\d{4}', 'call 555-1234 now', '555-1234')
tst_find('\\D+', 'call 555', 'call ')
tst_find('\\a+', 'lowercase', 'lowercase')
tst_find('\\A+', 'UPPER', 'UPPER')
println('\n--- Testing Alternation (|) ---')
tst_find('cat|dog', 'the dog says meow', 'dog')
tst_find('a(b|c)d', 'acd', 'acd')
tst_find('apple|apply', 'I want to apply', 'apply')
println('\n--- Testing Custom Character Classes ([...]) ---')
tst_find('[aeiou]', 'hello world', 'e')
tst_find('gr[ae]y', 'the color grey', 'grey')
tst_find('[^aeiou]+', 'rhythm', 'rhythm')
tst_find('[a-z]+', 'lowercase123', 'lowercase')
tst_find('[a-zA-Z0-9_]+', 'word_1_with_everything', 'word_1_with_everything')
println('\n--- Testing Unicode ---')
tst_find('日本語', 'Text containing 日本語.', '日本語')
tst_find('h.llo', 'héllo wørld', 'héllo')
tst_find('(é)+', 'cafééé', 'ééé')
tst_find('😀+', 'Happy 😀😀 day', '😀😀')
println('\n--- Testing fullmatch() ---')
tst_fullmatch(r'\d+', '12345', '12345')
tst_fullmatch('(?s).*', 'Any content including 😀', 'Any content including 😀')
// --- Negative Tests (Basic) ---
tst_find('abc', 'ab', 'none')
tst_find('abc', 'acb', 'none')
tst_find('a+b', 'b', 'none')
tst_find('\\d+', 'abc', 'none')
tst_find('\\D+', '123', 'none')
tst_find('\\w+', '@#$', 'none')
tst_find('\\s+', 'Text', 'none')
tst_find('\\a+', 'UPPERCASE', 'none')
tst_find('\\A+', 'lowercase', 'none')
tst_find('cat|dog', 'bird', 'none')
tst_find('[0-9]', 'a', 'none')
tst_find('[^0-9]', '1', 'none')
tst_fullmatch(r'\d+', '12345abc', 'none')
tst_fullmatch(r'\d+', 'abc12345', 'none')
println('\n--- Testing Compilation Errors ---')
tst_compile_error('a++')
tst_compile_error('[a-z')
tst_compile_error('a|')
}
fn test_complex_quantifiers() {
println('\n--- Testing Complex Quantifiers (+, *, ?) ---')
tst_find('a+', 'aaaaa', 'aaaaa')
tst_find('a+b', 'aaaaab', 'aaaaab')
tst_find('x*y', 'y', 'y')
tst_find('x*y', 'xy', 'xy')
tst_find('x*y', 'xxxy', 'xxxy')
tst_find('colou?r', 'color', 'color')
tst_find('colou?r', 'colour', 'colour')
tst_find('x?y', 'xy', 'xy')
tst_find('x?y', 'y', 'y')
tst_find('(ab)+', 'ababab', 'ababab')
tst_find('(ha)+', 'hahaha!', 'hahaha')
tst_find('(cat|dog)+', 'catdogcat', 'catdogcat')
tst_find('(a|b)+', 'abaabbba', 'abaabbba')
tst_find('[0-9]+', 'Order 12345', '12345')
tst_find('[a-z]*', '123', '')
// --- Negative Tests (Complex Quantifiers) ---
tst_find('a+', '', 'none')
tst_find('a+', 'b', 'none')
tst_find('a+b', 'aaac', 'none')
tst_find('x?y', 'x', 'none')
tst_find('(ab)+', 'ac', 'none')
tst_find('[0-9]+', 'abc', 'none')
}
fn test_range_quantifiers() {
println('\n--- Testing Range Quantifiers {m,n} ---')
tst_find('a{3}', 'aaa', 'aaa')
tst_find('a{3}', 'aaaa', 'aaa')
tst_find('a{2,}', 'aa', 'aa')
tst_find('a{2,}', 'aaaaa', 'aaaaa')
tst_find('a{,3}', 'aaaa', 'aaa')
tst_find('a{,3}', 'aa', 'aa')
tst_find('a{,3}', '', '')
tst_find('a{2,4}', 'aa', 'aa')
tst_find('a{2,4}', 'aaa', 'aaa')
tst_find('a{2,4}', 'aaaa', 'aaaa')
tst_find('a{2,4}', 'aaaaa', 'aaaa')
tst_find(r'\d{2,4}-\w{2}', '123-ab', '123-ab')
tst_find(r'\d{2,4}-\w{2}', '12345-ab', '2345-ab')
// --- Negative Tests (Range Quantifiers) ---
tst_find('a{3}', 'aa', 'none')
tst_find('a{2,}', 'a', 'none')
tst_find('a{2,4}', 'a', 'none')
tst_find(r'\d{2,4}-\w{2}', '1-ab', 'none')
tst_find(r'\d{2,4}-\w{2}', '123-a', 'none')
}
fn test_named_groups() {
println('\n--- Testing Named Groups ---')
pattern := '(?P<year>\\d{4})-(?P<month>\\d{2})'
text := 'Date: 2025-01'
r := pcre.compile(pattern) or { panic(err) }
m := r.find(text) or { panic('Match not found') }
assert m.groups[0] == '2025'
assert m.groups[1] == '01'
assert r.group_by_name(m, 'year') == '2025'
assert r.group_by_name(m, 'month') == '01'
assert r.group_by_name(m, 'missing') == ''
nested_pat := '(?P<entry>key: (?P<val>\\d+))'
nested_txt := 'List [ key: 99 ]'
r_nested := pcre.compile(nested_pat) or { panic(err) }
m_nested := r_nested.find(nested_txt) or { panic('Match not found') }
println('Nested: entry="${r_nested.group_by_name(m_nested, 'entry')}", val="${r_nested.group_by_name(m_nested,
'val')}"')
assert r_nested.group_by_name(m_nested, 'entry') == 'key: 99'
assert r_nested.group_by_name(m_nested, 'val') == '99'
pattern_mixed := '(?P<key>\\w+): (\\d+)'
text_mixed := 'Price: 100'
r_mixed := pcre.compile(pattern_mixed) or { panic(err) }
m_mixed := r_mixed.find(text_mixed) or { panic('Match not found') }
assert m_mixed.groups[0] == 'Price'
assert m_mixed.groups[1] == '100'
assert r_mixed.group_by_name(m_mixed, 'key') == 'Price'
p_seq := '(?P<a>a)(?P<b>b)(?P<c>c)'
t_seq := 'abc'
r_seq := pcre.compile(p_seq) or { panic(err) }
m_seq := r_seq.find(t_seq) or { panic('Match not found') }
assert r_seq.group_by_name(m_seq, 'a') == 'a'
assert r_seq.group_by_name(m_seq, 'b') == 'b'
assert r_seq.group_by_name(m_seq, 'c') == 'c'
// --- Negative Tests (Named Groups) ---
if _ := r.find('Date: 99-01') {
assert false, 'Should not match'
} else {
println('Found: none (Expected: none)')
}
tst_find('(?P<id>\\d+)', 'abc', 'none')
}
fn test_non_capturing_groups() {
println('\n--- Testing Non-Capturing Groups ---')
tst_find_with_groups('(?:a|b)c', 'ac', 'ac', [])
tst_find_with_groups('(a)(?:b)(c)', 'abc', 'abc', ['a', 'c'])
tst_find_with_groups('(a(?:b)c)', 'abc', 'abc', ['abc'])
tst_find_with_groups('(?:header): (\\d+)', 'header: 123', 'header: 123', ['123'])
// --- Negative Tests (Non-Capturing Groups) ---
tst_find('(?:a|b)c', 'dc', 'none')
tst_find('(?:a)b', 'c', 'none')
}
// --- Helper Functions ---
fn tst_find(pattern string, text string, expected string) {
print('[find] Pattern: "${pattern}", Text: "${text}" -> ')
r := pcre.compile(pattern) or {
println('Compile error: ${err}')
assert false, 'Unexpected compile error: ${err}'
return
}
match_res := r.find(text)
check_result(match_res, expected)
}
fn tst_find_all(pattern string, text string, expected []string) {
print('[find_all] Pattern: "${pattern}", Text: "${text}" -> ')
r := pcre.compile(pattern) or { panic(err) }
matches := r.find_all(text)
mut res_strs := []string{}
for m in matches {
res_strs << m.text
}
println('Found: ${res_strs}')
assert res_strs == expected
}
fn tst_find_from(pattern string, text string, start int, expected_pos int, expected_text string) {
print('[find_from] Pattern: "${pattern}", Start: ${start} -> ')
r := pcre.compile(pattern) or { panic(err) }
match_res := r.find_from(text, start)
if match_res != none {
println('Found: "${match_res.text}" at ${match_res.start}')
assert match_res.text == expected_text
assert match_res.start == expected_pos
} else {
println('Found: none')
assert expected_text == 'none'
}
}
fn tst_replace(pattern string, text string, repl string, expected string) {
print('[replace] Pattern: "${pattern}", Repl: "${repl}" -> ')
r := pcre.compile(pattern) or { panic(err) }
res := r.replace(text, repl)
println('Result: "${res}"')
assert res == expected
}
fn tst_fullmatch(pattern string, text string, expected string) {
print('[fullmatch] Pattern: "${pattern}", Text: "${text}" -> ')
r := pcre.compile(pattern) or {
println('Compile error: ${err}')
assert false, 'Unexpected compile error: ${err}'
return
}
match_res := r.fullmatch(text)
check_result(match_res, expected)
}
fn check_result(match_res ?pcre.Match, expected string) {
if match_res != none {
println('Found: "${match_res.text}" (Expected: "${expected}")')
assert match_res.text == expected
} else {
println('Found: none (Expected: "${expected}")')
assert expected == 'none'
}
}
fn tst_find_with_groups(pattern string, text string, expected_match string, expected_groups []string) {
print('[find+groups] Pattern: "${pattern}", Text: "${text}" -> ')
r := pcre.compile(pattern) or {
println('Compile error: ${err}')
assert false, 'Unexpected compile error: ${err}'
return
}
match_res := r.find(text)
if match_res != none {
println('Found: "${match_res.text}", Groups: ${match_res.groups}')
assert match_res.text == expected_match
assert match_res.groups == expected_groups
} else {
println('Found: none')
assert false // Should have found a match_res
}
}
fn tst_compile_error(pattern string) {
print('[compile_error] Pattern: "${pattern}" -> ')
_ := pcre.compile(pattern) or {
println('Caught expected error: ${err}')
return
}
println('Error: Did not get a compilation error!')
assert false
}
fn test_non_greedy_quantifiers() {
println('\n--- Testing Non-Greedy Quantifiers (*?, +?, ??, {m,n}?) ---')
// 1. Lazy Star (*?)
// Should stop at the first closing '>' (minimal match)
tst_find(r'<.*?>', '<div>content</div>', '<div>')
// Contrast with greedy (default) which consumes until the last '>'
tst_find(r'<.*>', '<div>content</div>', '<div>content</div>')
// 2. Lazy Plus (+?)
// Should match minimal characters (1 'a') to satisfy the constraint
tst_find(r'a+?', 'aaaaa', 'a')
// Forced expansion: Must match all 'a's to finally match 'b' (backtracking test)
tst_find(r'a+?b', 'aaab', 'aaab')
// 3. Lazy Question Mark (??)
// Should match empty string (prefers 0 occurrences over 1)
tst_find(r'a??', 'a', '')
// Contextual: 'u' is lazy (prefers skip), matches 'color' immediately
tst_find(r'colou??r', 'color', 'color')
// Contextual: 'u' is lazy, tries skip, fails to match 'r', backtracks to match 'u'
tst_find(r'colou??r', 'colour', 'colour')
// 4. Lazy Range ({m,n}?)
// Should match minimum required (2 digits)
tst_find(r'\d{2,5}?', '123456789', '12')
// Contrast with greedy which matches maximum (5 digits)
tst_find(r'\d{2,5}', '123456789', '12345')
// 5. Complex/Real-world Case (User report)
// Escaped characters + lazy capture group
// Should match only '$t(common.hello)', not the span to the second ')'
tst_find(r'\$t\((.*?)\)', r'$t(common.hello) dear $t(common.name)', r'$t(common.hello)')
// --- Negative / Edge Cases ---
// Lazy quantifier with no termination in string should match nothing/min if possible,
// but since it's "find", it grabs the first valid match.
tst_find(r'x.*?y', 'x123y456y', 'x123y') // Stops at first y
// Anchor interaction: ^.*?b
// Matches from start, .*? expands lazily until it hits 'b'
tst_find(r'^.*?b', '123b', '123b')
// Ensure lazy doesn't cause failure when a greedy match would succeed (correct backtracking)
// Pattern wants to match "a" lazily, but must consume "a" to satisfy the final "a"
tst_find(r'a?a', 'a', 'a')
tst_find(r'a??a', 'a', 'a')
}
fn test_compatibility_layer() {
// Test new_regex (alias for compile)
// Passing '0' as the second argument to simulate the ignored C-flag argument
pattern := r'(\w+)\s+(\d+)'
re := pcre.new_regex(pattern, 0) or {
assert false, 'new_regex failed to compile: ${err}'
return
}
text := 'item 42 ignored item 99'
// Test match_str (alias for find_from)
// We start searching from index 0. The third argument '0' is the ignored option flag.
// This should match "item 42"
m1 := re.match_str(text, 0, 0) or {
assert false, 'match_str failed to find match'
return
}
// Test get()
// Index 0 should be the full text of the match
full_match := m1.get(0) or { '' }
assert full_match == 'item 42'
// Index 1 should be the first capture group (\w+)
group_1 := m1.get(1) or { '' }
assert group_1 == 'item'
// Index 2 should be the second capture group (\d+)
group_2 := m1.get(2) or { '' }
assert group_2 == '42'
// Index 3 should be none (out of bounds)
if _ := m1.get(3) {
assert false, 'get(3) should return none for 2 groups'
}
// Test get_all()
// Should return ['item 42', 'item', '42']
all_captures := m1.get_all()
assert all_captures.len == 3
assert all_captures[0] == 'item 42'
assert all_captures[1] == 'item'
assert all_captures[2] == '42'
// Test match_str with a specific start index
// Start searching after "item 42" (length is 7)
// This should match "item 99"
m2 := re.match_str(text, 7, 0) or {
assert false, 'match_str failed to find second match from offset'
return
}
assert m2.get(0) or { '' } == 'item 99'
assert m2.get(2) or { '' } == '99'
// Test match_str failure case
// Start searching at the very end of string
no_match := re.match_str(text, text.len, 0)
if _ := no_match {
assert false, 'match_str should return none when no match is found'
}
}
fn test_hex_escapes() {
// \xHH — two hex digits
tst_find(r'\x41', 'ABC', 'A') // 0x41 = 'A'
tst_find(r'\x61', 'abc', 'a') // 0x61 = 'a'
tst_find(r'\x41+', 'AAAB', 'AAA')
tst_find(r'\x20\x41', ' A test', ' A') // space + 'A'
// \XHHHH — four hex digits (Unicode codepoint)
tst_find(r'\X0041', 'ABC', 'A') // U+0041 = 'A'
tst_find(r'\X0061', 'abc', 'a') // U+0061 = 'a'
tst_find(r'\X03B1', 'αβγ', 'α') // U+03B1 = 'α'
// Mix with other escapes
tst_find(r'\x48\x65\x6C\x6C\x6F', 'Hello World', 'Hello') // \x48\x65\x6C\x6C\x6F = "Hello"
// Invalid hex escape compile errors
tst_compile_error(r'\x4') // only 1 digit
tst_compile_error(r'\xGG') // invalid hex chars
tst_compile_error(r'\X004') // only 3 digits
}
fn test_duplicate_named_groups() {
// Compile error: same name used twice
tst_compile_error(r'(?P<id>\d+)-(?P<id>\w+)')
// Different names are fine
r := pcre.compile(r'(?P<a>\d+)-(?P<b>\w+)') or {
assert false, 'Should compile: ${err}'
return
}
m := r.find('12-abc') or {
assert false, 'Should match'
return
}
assert r.group_by_name(m, 'a') == '12'
assert r.group_by_name(m, 'b') == 'abc'
}
fn test_invalid_quantifier_ranges() {
// min > max is an error
tst_compile_error(r'a{3,1}')
tst_compile_error(r'a{5,2}')
// negative min-like patterns (parsed as 0)
// {0,0} should compile and match empty string
r := pcre.compile(r'a{0,0}b') or {
assert false, 'Should compile: ${err}'
return
}
m := r.find('b') or {
assert false, 'Should match'
return
}
assert m.text == 'b'
}
fn test_find_all_utf8_safety() {
// find_all with an empty-matching pattern must not get stuck inside a multi-byte rune
r := pcre.compile(r'x*') or { panic(err) }
matches := r.find_all('aé') // 'é' is 2 bytes (0xC3 0xA9)
// Every result start/end must align on a rune boundary
for m in matches {
text_bytes := 'aé'.bytes()
if m.start < text_bytes.len {
// byte at start must not be a UTF-8 continuation byte
assert (text_bytes[m.start] & 0xC0) != 0x80, 'Misaligned match start at ${m.start}'
}
}
// find_all should not infinite-loop on emoji
r2 := pcre.compile(r'y*') or { panic(err) }
matches2 := r2.find_all('😀!')
assert matches2.len > 0
}
tools/v/vlib/regex/regex.v
+2792
View File
File diff suppressed because it is too large. Load diff
tools/v/vlib/regex/regex_anchor_test.v
+44
View File
@@ -0,0 +1,44 @@
import regex
fn test_anchor_start() {
mut re := regex.regex_opt(r'^\w+') or { panic(err) }
start, end := re.find('id.')
assert start == 0
assert end == 2
}
fn test_anchor_end() {
mut re := regex.regex_opt(r'\w+$') or { panic(err) }
start, end := re.find('(id')
assert start == 1
assert end == 3
}
fn test_anchor_both() {
mut re := regex.regex_opt(r'^\w+$') or { panic(err) }
start, end := re.find('(id)')
assert start == -1
assert end == -1
}
fn test_anchor_both_find_multiline() {
text := 'TITLE\n\nThis is a test.'
mut re := regex.regex_opt(r'^\w+$') or { panic(err) }
start, end := re.find(text)
assert start == 0
assert end == 5
}
fn test_anchor_both_find_all_multiline() {
text := 'TITLE\n\nThis is a test.'
mut re := regex.regex_opt(r'^\w+$') or { panic(err) }
res := re.find_all(text)
assert res == [0, 5]
}
fn test_anchor_both_find_all_str_multiline() {
text := 'TITLE\n\nThis is a test.'
mut re := regex.regex_opt(r'^\w+$') or { panic(err) }
res := re.find_all_str(text)
assert res == ['TITLE']
}
tools/v/vlib/regex/regex_complex_test.v
+54
View File
@@ -0,0 +1,54 @@
import regex
fn test_complex_matching001() {
tmp_text := r"@[export:'something']
const (
some_const = [1114111, 127, 2047, 65535, 1114111]!
)
@[export:'something_empty']
const (
empty_const = ()
)
"
query := r'\[export:\S+\]\sconst\s\(\s+\S+\s{3}=\s\('
mut re := regex.regex_opt(query) or { panic(err) }
mut a := re.find_all(tmp_text)
assert a == [88, 140]
}
fn test_complex_matching002() {
text := '<?xml version="1.0" encoding="utf-8"?>
<resources>
<string name="v_app_name">V0</string>
<!-- comment -->
<string name="v_lib_name">v1</string>
<string name="v_lib_name2">v2</string>
</resources>
'
mut re := regex.regex_opt(r'.*<resources>.+<string name="v_lib_name">([^<]+)') or { panic(err) }
start, end := re.match_string(text)
if start >= 0 && re.groups.len > 0 {
// check that we have obtained our 'v1' value
assert text#[re.groups[0]..re.groups[1]] == 'v1'
return
}
assert false
}
fn test_complex_matching003() {
text := 'abcdefgt1234abcd<tag name="mio_tag">value</tag>'
mut re := regex.regex_opt(r'\w*<tag\s*name\s*="mio_tag">([^<]+)') or { panic(err) }
start, end := re.match_string(text)
if start >= 0 && re.groups.len > 0 {
println('found ${text#[start..end]}')
println('group: ${text#[re.groups[0]..re.groups[1]]}')
return
}
assert false
}
tools/v/vlib/regex/regex_opt.v
+53
View File
@@ -0,0 +1,53 @@
module regex
import strings
// compile_opt compile RE pattern string
pub fn (mut re RE) compile_opt(pattern string) ! {
re_err, err_pos := re.impl_compile(pattern)
if re_err != compile_ok {
mut err_msg := strings.new_builder(300)
err_msg.write_string('\nquery: ${pattern}\n')
line := '-'.repeat(err_pos)
err_msg.write_string('err : ${line}^\n')
err_str := re.get_parse_error_string(re_err)
err_msg.write_string('ERROR: ${err_str}\n')
return error_with_code(err_msg.str(), re_err)
}
}
// new create a RE of small size, usually sufficient for ordinary use
pub fn new() RE {
// init regex
mut re := RE{}
re.prog = []Token{len: max_code_len + 1} // max program length, can not be longer then the pattern
re.cc = []CharClass{len: max_code_len} // can not be more char class the length of the pattern
re.group_csave_flag = false // enable continuos group saving
re.group_max_nested = 128 // set max 128 group nested
re.group_max = max_code_len >> 1 // we can't have more groups than the half of the pattern legth
re.group_stack = []int{len: re.group_max, init: -1}
re.group_data = []int{len: re.group_max, init: -1}
return re
}
// regex_opt create new RE object from RE pattern string
pub fn regex_opt(pattern string) !RE {
// init regex
mut re := RE{}
re.prog = []Token{len: pattern.len + 1} // max program length, can not be longer then the pattern
re.cc = []CharClass{len: pattern.len} // can not be more char class the length of the pattern
re.group_csave_flag = false // enable continuos group saving
re.group_max_nested = pattern.len >> 1 // set max 128 group nested
re.group_max = pattern.len >> 1 // we can't have more groups than the half of the pattern legth
re.group_stack = []int{len: re.group_max, init: -1}
re.group_data = []int{len: re.group_max, init: -1}
// compile the pattern
re.compile_opt(pattern)!
return re
}
tools/v/vlib/regex/regex_test.v
+1012
View File
File diff suppressed because it is too large. Load diff
tools/v/vlib/regex/regex_util.v
+596
View File
@@ -0,0 +1,596 @@
/*
regex 1.0 alpha
Copyright (c) 2019-2024 Dario Deledda. All rights reserved.
Use of this source code is governed by an MIT license
that can be found in the LICENSE file.
*/
module regex
import strings
/******************************************************************************
*
* Inits
*
******************************************************************************/
// regex_base returns a regex object (`RE`) generated from `pattern` string and
// detailed information in re_err, err_pos, if an error occurred.
pub fn regex_base(pattern string) (RE, int, int) {
// init regex
mut re := RE{}
re.prog = []Token{len: pattern.len + 1} // max program length, can not be longer then the pattern
re.cc = []CharClass{len: pattern.len} // can not be more char class the length of the pattern
re.group_csave_flag = false // enable continuos group saving
re.group_max_nested = pattern.len >> 1 // set max 128 group nested
re.group_max = pattern.len >> 1 // we can't have more groups than the half of the pattern legth
re.group_stack = []int{len: re.group_max, init: -1}
re.group_data = []int{len: re.group_max, init: -1}
re_err, err_pos := re.impl_compile(pattern)
return re, re_err, err_pos
}
/******************************************************************************
*
* Utilities
*
******************************************************************************/
// get_group_bounds_by_name get a group boundaries by its name
pub fn (re &RE) get_group_bounds_by_name(group_name string) (int, int) {
if group_name in re.group_map {
tmp_index := re.group_map[group_name] - 1
start := re.groups[tmp_index * 2]
end := re.groups[tmp_index * 2 + 1]
return start, end
}
return -1, -1
}
// get_group_by_name get a group boundaries by its name
pub fn (re &RE) get_group_by_name(in_txt string, group_name string) string {
if group_name in re.group_map {
tmp_index := re.group_map[group_name] - 1
start := re.groups[tmp_index * 2]
end := re.groups[tmp_index * 2 + 1]
if start >= 0 && end > start {
return in_txt[start..end]
}
}
return ''
}
// get_group_by_id get a group string by its id
pub fn (re &RE) get_group_by_id(in_txt string, group_id int) string {
if group_id < (re.groups.len >> 1) {
index := group_id * 2
start := re.groups[index]
end := re.groups[index + 1]
if start >= 0 && end > start {
return in_txt[start..end]
}
}
return ''
}
// get_group_by_id get a group boundaries by its id
pub fn (re &RE) get_group_bounds_by_id(group_id int) (int, int) {
if group_id < re.group_count {
index := group_id * 2
return re.groups[index], re.groups[index + 1]
}
return -1, -1
}
pub struct Re_group {
pub:
start int = -1
end int = -1
}
// get_group_list return a list of Re_group for the found groups
pub fn (re &RE) get_group_list() []Re_group {
mut res := []Re_group{len: re.groups.len >> 1}
mut gi := 0
// println("len: ${re.groups.len} groups: ${re.groups}")
for gi < re.groups.len {
if re.groups[gi] >= 0 {
txt_st := re.groups[gi]
txt_en := re.groups[gi + 1]
// println("#${gi/2} start: ${re.groups[gi]} end: ${re.groups[gi + 1]} ")
if txt_st >= 0 && txt_en > txt_st {
tmp := Re_group{
start: re.groups[gi]
end: re.groups[gi + 1]
}
// println(tmp)
res[gi >> 1] = tmp
} else {
res[gi >> 1] = Re_group{}
}
}
gi += 2
}
return res
}
/******************************************************************************
*
* Matchers
*
******************************************************************************/
// match_string Match the pattern with the in_txt string
@[direct_array_access]
pub fn (re &RE) match_string(in_txt string) (int, int) {
unsafe {
start, mut end := re.match_base(in_txt.str, in_txt.len + 1)
if end > in_txt.len {
end = in_txt.len
}
if start >= 0 && end >= start {
ok, _ := re.check_anchors(in_txt, start, end)
if ok {
return start, end
}
return no_match_found, 0
}
return start, end
}
}
// matches_string Checks if the pattern matches the in_txt string
pub fn (re &RE) matches_string(in_txt string) bool {
start, _ := re.match_string(in_txt)
return start != no_match_found
}
/******************************************************************************
*
* Finders
*
******************************************************************************/
@[direct_array_access; inline]
fn (re &RE) check_anchors(in_txt string, start int, end int) (bool, bool) {
// `^` means start of the source string.
if (re.flag & f_ms) != 0 && start > 0 {
return false, true
}
// `$` means end of the source string, or right before a newline.
if (re.flag & f_me) != 0 && end < in_txt.len {
if in_txt[end] in new_line_list {
return true, false
}
// When `^` is also present, scanning forward can never recover.
return false, (re.flag & f_ms) != 0
}
return true, false
}
/*
// find internal implementation HERE for reference do not remove!!
@[direct_array_access]
fn (mut re RE) find_imp(in_txt string) (int,int) {
old_flag := re.flag
re.flag |= f_src // enable search mode
start, mut end := re.match_base(in_txt.str, in_txt.len + 1)
//print("Find [${start},${end}] '${in_txt[start..end]}'")
if end > in_txt.len {
end = in_txt.len
}
re.flag = old_flag
if start >= 0 && end > start {
return start, end
}
return no_match_found, 0
}
*/
// find try to find the first match in the input string
@[direct_array_access]
pub fn (mut re RE) find(in_txt string) (int, int) {
// old_flag := re.flag
// re.flag |= f_src // enable search mode
mut i := 0
for i <= in_txt.len {
mut s := -1
mut e := -1
unsafe {
// tmp_str := tos(in_txt.str + i, in_txt.len - i)
// println("Check: [${tmp_str}]")
s, e = re.match_base(in_txt.str + i, in_txt.len - i + 1)
if s >= 0 && e >= s {
abs_start := i + s
abs_end := i + e
ok, stop_scan := re.check_anchors(in_txt, abs_start, abs_end)
if !ok {
if stop_scan {
break
}
i++
continue
}
// println("find match in: ${i+s},${i+e} [${in_txt[i+s..i+e]}]")
// re.flag = old_flag
mut gi := 0
for gi < re.groups.len {
re.groups[gi] += i
gi++
}
return abs_start, abs_end
}
i++
}
}
// re.flag = old_flag
return -1, -1
}
// find try to find the first match in the input string strarting from start index
@[direct_array_access]
pub fn (mut re RE) find_from(in_txt string, start int) (int, int) {
old_flag := re.flag
// re.flag |= f_src // enable search mode
mut i := start
if i < 0 {
return -1, -1
}
for i <= in_txt.len {
//--- speed references ---
mut s := -1
mut e := -1
unsafe {
tmp_str := tos(in_txt.str + i, in_txt.len - i)
s, e = re.match_string(tmp_str)
}
//------------------------
// s,e = re.find_imp(in_txt[i..])
//------------------------
if s >= 0 && e >= s {
abs_start := i + s
abs_end := i + e
ok, stop_scan := re.check_anchors(in_txt, abs_start, abs_end)
if !ok {
if stop_scan {
break
}
i++
continue
}
// println("find match in: ${i+s},${i+e} [${in_txt[i+s..i+e]}]")
re.flag = old_flag
mut gi := 0
for gi < re.groups.len {
re.groups[gi] += i
gi++
}
return abs_start, abs_end
} else {
i++
}
}
re.flag = old_flag
return -1, -1
}
// find_all find all the non overlapping occurrences of the match pattern and return the start and end index of the match
//
// Usage:
// ```v
// blurb := 'foobar boo steelbar toolbox foot tooooot'
// mut re := regex.regex_opt('f|t[eo]+')?
// res := re.find_all(blurb) // [0, 3, 12, 15, 20, 23, 28, 31, 33, 39]
// ```
@[direct_array_access]
pub fn (mut re RE) find_all(in_txt string) []int {
// old_flag := re.flag
// re.flag |= f_src // enable search mode
mut i := 0
mut res := []int{}
for i <= in_txt.len {
mut s := -1
mut e := -1
unsafe {
// tmp_str := in_txt[i..]
// tmp_str := tos(in_txt.str + i, in_txt.len - i)
// println("Check: [${tmp_str}]")
s, e = re.match_base(in_txt.str + i, in_txt.len + 1 - i)
if s >= 0 && e >= s {
abs_start := i + s
abs_end := i + e
ok, stop_scan := re.check_anchors(in_txt, abs_start, abs_end)
if !ok {
if stop_scan {
break
}
i++
continue
}
res << abs_start
res << abs_end
if e > s {
i += e
} else {
i++
}
continue
}
/*
if e > 0 {
i += e
continue
}
*/
i++
}
}
// re.flag = old_flag
return res
}
// split returns the sections of string around the regex
//
// Usage:
// ```v
// blurb := 'foobar boo steelbar toolbox foot tooooot'
// mut re := regex.regex_opt('f|t[eo]+')?
// res := re.split(blurb) // ['bar boo s', 'lbar ', 'lbox ', 't ', 't']
// ```
pub fn (mut re RE) split(in_txt string) []string {
pos := re.find_all(in_txt)
mut sections := []string{cap: pos.len / 2 + 1}
if pos.len == 0 {
return [in_txt]
}
for i := 0; i < pos.len; i += 2 {
if i == 0 {
sections << in_txt[..pos[i]]
} else {
sections << in_txt[pos[i - 1]..pos[i]]
}
}
sections << in_txt[pos[pos.len - 1]..]
return sections
}
// find_all_str find all the non overlapping occurrences of the match pattern, return a string list
@[direct_array_access]
pub fn (mut re RE) find_all_str(in_txt string) []string {
// old_flag := re.flag
// re.flag |= f_src // enable search mode
mut i := 0
mut res := []string{}
for i <= in_txt.len {
mut s := -1
mut e := -1
unsafe {
// tmp_str := in_txt[i..]
// tmp_str := tos(in_txt.str + i, in_txt.len - i)
// println("Check: [${tmp_str}]")
s, e = re.match_base(in_txt.str + i, in_txt.len + 1 - i)
if s >= 0 && e >= s {
abs_start := i + s
abs_end := i + e
ok, stop_scan := re.check_anchors(in_txt, abs_start, abs_end)
if !ok {
if stop_scan {
break
}
i++
continue
}
tmp_str := tos(in_txt.str + i, in_txt.len - i)
mut tmp_e := if e > tmp_str.len { tmp_str.len } else { e }
// println("Found: ${s}:${e} [${tmp_str[s..e]}]")
res << tmp_str[s..tmp_e]
if e > s {
i += e
} else {
i++
}
continue
}
}
/*
if e > 0 {
i += e
continue
}
*/
i++
}
// re.flag = old_flag
return res
}
/******************************************************************************
*
* Replacers
*
******************************************************************************/
// replace_simple return a string where the matches are replaced with the replace string
pub fn (mut re RE) replace_simple(in_txt string, repl string) string {
pos := re.find_all(in_txt)
if pos.len > 0 {
mut res := ''
mut i := 0
mut s1 := 0
mut e1 := in_txt.len
for i < pos.len {
e1 = pos[i]
res += in_txt[s1..e1] + repl
s1 = pos[i + 1]
i += 2
}
res += in_txt[s1..]
return res
}
return in_txt
}
// type of function used for custom replace
// in_txt source text
// start index of the start of the match in in_txt
// end index of the end of the match in in_txt
// the match is in in_txt[start..end]
pub type FnReplace = fn (re RE, in_txt string, start int, end int) string
// replace_by_fn return a string where the matches are replaced with the string from the repl_fn callback function
pub fn (mut re RE) replace_by_fn(in_txt string, repl_fn FnReplace) string {
mut i := 0
mut res := strings.new_builder(in_txt.len)
mut last_end := 0
for i < in_txt.len {
// println("Find Start. ${i} [${in_txt[i..]}]")
s, e := re.find_from(in_txt, i)
// println("Find End.")
if s >= 0 && e > s {
// println("find match in: ${s},${e} [${in_txt[s..e]}]")
if last_end < s {
res.write_string(in_txt[last_end..s])
}
/*
for g_i in 0 .. re.group_count {
re.groups[g_i * 2] += i
re.groups[(g_i * 2) + 1] += i
}
*/
repl := repl_fn(re, in_txt, s, e)
// println("repl res: ${repl}")
res.write_string(repl)
// res.write_string("[[${in_txt[s..e]}]]")
last_end = e
i = e
} else {
break
// i++
}
// println(i)
}
if last_end >= 0 && last_end < in_txt.len {
res.write_string(in_txt[last_end..])
}
return res.str()
}
fn (re &RE) parsed_replace_string(in_txt string, repl string) string {
str_lst := repl.split('\\')
mut res := str_lst[0]
mut i := 1
for i < str_lst.len {
tmp := str_lst[i]
// println("tmp: ${tmp}")
if tmp.len > 0 && tmp[0] >= `0` && tmp[0] <= `9` {
group_id := int(tmp[0] - `0`)
group := re.get_group_by_id(in_txt, group_id)
// println("group: ${group_id} [${group}]")
res += '${group}${tmp[1..]}'
} else {
res += '\\' + tmp
}
i++
}
return res
}
// replace return a string where the matches are replaced with the repl_str string,
// this function supports groups in the replace string
pub fn (mut re RE) replace(in_txt string, repl_str string) string {
mut i := 0
mut res := strings.new_builder(in_txt.len)
mut last_end := 0
for i < in_txt.len {
// println("Find Start. ${i} [${in_txt[i..]}]")
s, e := re.find_from(in_txt, i)
// println("Find End.")
if s >= 0 && e > s {
// println("find match in: ${s},${e} [${in_txt[s..e]}]")
if last_end < s {
res.write_string(in_txt[last_end..s])
}
/*
for g_i in 0 .. re.group_count {
re.groups[g_i * 2] += i
re.groups[(g_i * 2) + 1] += i
}
*/
// repl := repl_fn(re, in_txt, s, e)
repl := re.parsed_replace_string(in_txt, repl_str)
// println("repl res: ${repl}")
res.write_string(repl)
// res.write_string("[[${in_txt[s..e]}]]")
last_end = e
i = e
} else {
break
// i++
}
// println(i)
}
if last_end >= 0 && last_end < in_txt.len {
res.write_string(in_txt[last_end..])
}
return res.str()
}
// replace_n return a string where the first count matches are replaced with the repl_str string,
// if count is > 0 the replace began from the start of the string toward the end
// if count is < 0 the replace began from the end of the string toward the start
// if count is 0 do nothing
pub fn (mut re RE) replace_n(in_txt string, repl_str string, count int) string {
mut i := 0
mut index := 0
mut i_p := 0
mut res := strings.new_builder(in_txt.len)
mut lst := re.find_all(in_txt)
if count < 0 { // start from the right of the string
lst = unsafe { lst#[count * 2..] } // limitate the number of substitions
} else if count > 0 { // start from the left of the string
lst = unsafe { lst#[..count * 2] } // limitate the number of substitions
} else if count == 0 { // no replace
return in_txt
}
// println("found: ${lst}")
for index < lst.len {
i = lst[index]
res.write_string(in_txt[i_p..i])
res.write_string(repl_str)
index++
i_p = lst[index]
index++
}
i = i_p
res.write_string(in_txt[i..])
return res.str()
}