[Macro]
do-scans (match-start match-end reg-starts reg-ends regex target-string &optional result-form &key start end) declaration* statement* => result*

A macro which iterates over target-string and tries to match regex as often as possible evaluating statement* with match-start, match-end, reg-starts, and reg-ends bound to the four return values of each match (see SCAN) in turn. After the last match, returns result-form if provided or NIL otherwise. An implicit block named NIL surrounds DO-SCANS; RETURN may be used to terminate the loop immediately. If regex matches an empty string the scan is continued one position behind this match.

This is the most general macro to iterate over all matches in a target string. See the source code of DO-MATCHES, ALL-MATCHES, SPLIT, or REGEX-REPLACE-ALL for examples of its usage.

[Macro]
do-matches (match-start match-end regex target-string &optional result-form &key start end) declaration* statement* => result*

Like DO-SCANS but doesn't bind variables to the register arrays.

Example:

* (defun foo (regex target-string &key (start 0) (end (length target-string)))
    (let ((sum 0))
      (cl-ppcre:do-matches (s e regex target-string nil :start start :end end)
        (incf sum (- e s)))
      (format t "~,2F% of the string was inside of a match~%"
                ;; note: doesn't check for division by zero
                (float (* 100 (/ sum (- end start)))))))

FOO

* (foo "a" "abcabcabc")
33.33% of the string was inside of a match
NIL
* (foo "aa|b" "aacabcbbc")
55.56% of the string was inside of a match
NIL

[Macro]
do-matches-as-strings (match-var regex target-string &optional result-form &key start end sharedp) declaration* statement* => result*

Like DO-MATCHES but binds match-var to the substring of target-string corresponding to each match in turn. If sharedp is true, the substrings may share structure with target-string.

Example:

* (defun crossfoot (target-string &key (start 0) (end (length target-string)))
    (let ((sum 0))
      (cl-ppcre:do-matches-as-strings (m :digit-class
                                         target-string nil
                                         :start start :end end)
        (incf sum (parse-integer m)))
      (if (< sum 10)
        sum
        (crossfoot (format nil "~A" sum)))))

CROSSFOOT

* (crossfoot "bar")
0

* (crossfoot "a3x")
3

* (crossfoot "12345")
6

Of course, in real life you would do this with DO-MATCHES and use the start and end keyword parameters of PARSE-INTEGER.

[Macro]
do-register-groups var-list (regex target-string &optional result-form &key start end sharedp) declaration* statement* => result*

Iterates over target-string and tries to match regex as often as possible evaluating statement* with the variables in var-list bound to the corresponding register groups for each match in turn, i.e. each variable is either bound to a string or to NIL. You can use the same shortcuts and abbreviations as in REGISTER-GROUPS-BIND. The number of variables in var-list must not be greater than the number of register groups. For each element of var-list which is NIL there's no binding to the corresponding register group. After the last match, returns result-form if provided or NIL otherwise. An implicit block named NIL surrounds DO-REGISTER-GROUPS; RETURN may be used to terminate the loop immediately. If regex matches an empty string the scan is continued one position behind this match. If sharedp is true, the substrings may share structure with target-string.

Example:

* (do-register-groups (first second third fourth)
      ("((a)|(b)|(c))" "abababc" nil :start 2 :sharedp t)
    (print (list first second third fourth)))
("a" "a" NIL NIL) 
("b" NIL "b" NIL) 
("a" "a" NIL NIL) 
("b" NIL "b" NIL) 
("c" NIL NIL "c")
NIL

* (let (result)
    (do-register-groups ((#'parse-integer n) (#'intern sign) whitespace)
        ("(\\d+)|(\\+|-|\\*|/)|(\\s+)" "12*15 - 42/3")
      (unless whitespace
        (push (or n sign) result)))
    (nreverse result))
(12 * 15 - 42 / 3)

[Function]
all-matches regex target-string &key start end => list

Returns a list containing the start and end positions of all matches of regex against target-string, i.e. if there are N matches the list contains (* 2 N) elements. If regex matches an empty string the scan is continued one position behind this match.

Examples:

* (cl-ppcre:all-matches "a" "foo bar baz")
(5 6 9 10)

* (cl-ppcre:all-matches "\\w*" "foo bar baz")
(0 3 3 3 4 7 7 7 8 11 11 11)

[Function]
all-matches-as-strings regex target-string &key start end sharedp => list

Like ALL-MATCHES but returns a list of substrings instead. If sharedp is true, the substrings may share structure with target-string.

Examples:

* (cl-ppcre:all-matches-as-strings "a" "foo bar baz")
("a" "a")

* (cl-ppcre:all-matches-as-strings "\\w*" "foo bar baz")
("foo" "" "bar" "" "baz" "")

[Function]
split regex target-string &key start end limit with-registers-p omit-unmatched-p sharedp => list

Matches regex against target-string as often as possible and returns a list of the substrings between the matches. If with-registers-p is true, substrings corresponding to matched registers are inserted into the list as well. If omit-unmatched-p is true, unmatched registers will simply be left out, otherwise they will show up as NIL. limit limits the number of elements returned - registers aren't counted. If limit is NIL (or 0 which is equivalent), trailing empty strings are removed from the result list. If regex matches an empty string the scan is continued one position behind this match. If sharedp is true, the substrings may share structure with target-string.

Beginning with CL-PPCRE 0.2.0, this function also tries hard to be Perl-compatible - thus the somewhat peculiar behaviour. But note that it hasn't been as extensively tested as SCAN.

Examples:

* (cl-ppcre:split "\\s+" "foo   bar baz
frob")
("foo" "bar" "baz" "frob")

* (cl-ppcre:split "\\s*" "foo bar   baz")
("f" "o" "o" "b" "a" "r" "b" "a" "z")

* (cl-ppcre:split "(\\s+)" "foo bar   baz")
("foo" "bar" "baz")

* (cl-ppcre:split "(\\s+)" "foo bar   baz" :with-registers-p t)
("foo" " " "bar" "   " "baz")

* (cl-ppcre:split "(\\s)(\\s*)" "foo bar   baz" :with-registers-p t)
("foo" " " "" "bar" " " "  " "baz")

* (cl-ppcre:split "(,)|(;)" "foo,bar;baz" :with-registers-p t)
("foo" "," NIL "bar" NIL ";" "baz")

* (cl-ppcre:split "(,)|(;)" "foo,bar;baz" :with-registers-p t :omit-unmatched-p t)
("foo" "," "bar" ";" "baz")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::")
("a" "b" "c" "d" "e" "f" "g")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 1)
("a:b:c:d:e:f:g::")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 2)
("a" "b:c:d:e:f:g::")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 3)
("a" "b" "c:d:e:f:g::")

* (cl-ppcre:split ":" "a:b:c:d:e:f:g::" :limit 1000)
("a" "b" "c" "d" "e" "f" "g" "" "")

[Function]
regex-replace regex target-string replacement &key start end preserve-case simple-calls => list

Try to match target-string between start and end against regex and replace the first match with replacement.

replacement can be a string which may contain the special substrings "\&" for the whole match, "\`" for the part of target-string before the match, "\'" for the part of target-string after the match, "\N" or "\{N}" for the Nth register where N is a positive integer.

replacement can also be a function designator in which case the match will be replaced with the result of calling the function designated by replacement with the arguments target-string, start, end, match-start, match-end, reg-starts, and reg-ends. (reg-starts and reg-ends are arrays holding the start and end positions of matched registers (or NIL) - the meaning of the other arguments should be obvious.)

If simple-calls is true, a function designated by replacement will instead be called with the arguments match, register-1, ..., register-n where match is the whole match as a string and register-1 to register-n are the matched registers, also as strings (or NIL). Note that these strings share structure with target-string so you must not modify them.

Finally, replacement can be a list where each element is a string (which will be inserted verbatim), one of the symbols :match, :before-match, or :after-match (corresponding to "\&", "\`", and "\'" above), an integer N (representing register (1+ N)), or a function designator.

If preserve-case is true (default is NIL), the replacement will try to preserve the case (all upper case, all lower case, or capitalized) of the match. The result will always be a fresh string, even if regex doesn't match.

Examples:

* (cl-ppcre:regex-replace "fo+" "foo bar" "frob")
"frob bar"

* (cl-ppcre:regex-replace "fo+" "FOO bar" "frob")
"FOO bar"

* (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob")
"frob bar"

* (cl-ppcre:regex-replace "(?i)fo+" "FOO bar" "frob" :preserve-case t)
"FROB bar"

* (cl-ppcre:regex-replace "(?i)fo+" "Foo bar" "frob" :preserve-case t)
"Frob bar"

* (cl-ppcre:regex-replace "bar" "foo bar baz" "[frob (was '\\&' between '\\`' and '\\'')]")
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"

* (cl-ppcre:regex-replace "bar" "foo bar baz"
                          '("[frob (was '" :match "' between '" :before-match "' and '" :after-match "')]"))
"foo [frob (was 'bar' between 'foo ' and ' baz')] baz"

[Function]
regex-replace-all regex target-string replacement &key start end preserve-case simple-calls => list

Like REGEX-REPLACE but replaces all matches.

Examples:

* (cl-ppcre:regex-replace-all "(?i)fo+" "foo Fooo FOOOO bar" "frob" :preserve-case t)
"frob Frob FROB bar"

* (cl-ppcre:regex-replace-all "(?i)f(o+)" "foo Fooo FOOOO bar" "fr\\1b" :preserve-case t)
"froob Frooob FROOOOB bar"

* (let ((qp-regex (cl-ppcre:create-scanner "[\\x80-\\xff]")))
    (defun encode-quoted-printable (string)
      "Convert 8-bit string to quoted-printable representation."
      ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
             (declare (ignore start end match-end reg-starts reg-ends))
             (format nil "=~2,'0x" (char-code (char target-string match-start)))))
        (cl-ppcre:regex-replace-all qp-regex string #'convert))))
Converted ENCODE-QUOTED-PRINTABLE.
ENCODE-QUOTED-PRINTABLE

* (encode-quoted-printable "Fête Sørensen naïve Hühner Straße")
"F=EAte S=F8rensen na=EFve H=FChner Stra=DFe"

* (let ((url-regex (cl-ppcre:create-scanner "[^a-zA-Z0-9_\\-.]")))
    (defun url-encode (string)
      "URL-encode a string."
      ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (target-string start end match-start match-end reg-starts reg-ends)
             (declare (ignore start end match-end reg-starts reg-ends))
             (format nil "%~2,'0x" (char-code (char target-string match-start)))))
        (cl-ppcre:regex-replace-all url-regex string #'convert))))
Converted URL-ENCODE.
URL-ENCODE

* (url-encode "Fête Sørensen naïve Hühner Straße")
"F%EAte%20S%F8rensen%20na%EFve%20H%FChner%20Stra%DFe"

* (defun how-many (target-string start end match-start match-end reg-starts reg-ends)
    (declare (ignore start end match-start match-end))
    (format nil "~A" (- (svref reg-ends 0)
                        (svref reg-starts 0))))
HOW-MANY

* (cl-ppcre:regex-replace-all "{(.+?)}"
                              "foo{...}bar{.....}{..}baz{....}frob"
                              (list "[" 'how-many " dots]"))
"foo[3 dots]bar[5 dots][2 dots]baz[4 dots]frob"

* (let ((qp-regex (cl-ppcre:create-scanner "[\\x80-\\xff]")))
    (defun encode-quoted-printable (string)
      "Convert 8-bit string to quoted-printable representation.
Version using SIMPLE-CALLS keyword argument."
      ;; ;; won't work for Corman Lisp because non-ASCII characters aren't 8-bit there
      (flet ((convert (match)
               (format nil "=~2,'0x" (char-code (char match 0)))))
        (cl-ppcre:regex-replace-all qp-regex string #'convert
                                    :simple-calls t))))

Converted ENCODE-QUOTED-PRINTABLE.
ENCODE-QUOTED-PRINTABLE

* (encode-quoted-printable "Fête Sørensen naïve Hühner Straße")
"F=EAte S=F8rensen na=EFve H=FChner Stra=DFe"

* (defun how-many (match first-register)
    (declare (ignore match))
    (format nil "~A" (length first-register)))
HOW-MANY

* (cl-ppcre:regex-replace-all "{(.+?)}"
                              "foo{...}bar{.....}{..}baz{....}frob"
                              (list "[" 'how-many " dots]")
                              :simple-calls t)

"foo[3 dots]bar[5 dots][2 dots]baz[4 dots]frob"

[Function]
regex-apropos regex &optional packages &key case-insensitive => list

Like APROPOS but searches for interned symbols which match the regular expression regex. The output is implementation-dependent. If case-insensitive is true (which is the default) and regex isn't already a scanner, a case-insensitive scanner is used.

Here are examples for CMUCL:

* *package*
#<The COMMON-LISP-USER package, 16/21 internal, 0/9 external>

* (defun foo (n &optional (k 0)) (+ 3 n k))
FOO

* (defparameter foo "bar")
FOO

* (defparameter |foobar| 42)
|foobar|

* (defparameter fooboo 43)
FOOBOO

* (defclass frobar () ())
#<STANDARD-CLASS FROBAR {4874E625}>

* (cl-ppcre:regex-apropos "foo(?:bar)?")
FOO [variable] value: "bar"
    [compiled function] (N &OPTIONAL (K 0))
FOOBOO [variable] value: 43
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "(?:foo|fro)bar")
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "(?:foo|fro)bar" 'cl-user)
FROBAR [class] #<STANDARD-CLASS FROBAR {4874E625}>
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "(?:foo|fro)bar" '(pcl ext))
PCL::|COMMON-LISP-USER::FROBAR class predicate| [compiled closure]

* (cl-ppcre:regex-apropos "foo")
FOO [variable] value: "bar"
    [compiled function] (N &OPTIONAL (K 0))
FOOBOO [variable] value: 43
|foobar| [variable] value: 42

* (cl-ppcre:regex-apropos "foo" nil :case-insensitive nil)
|foobar| [variable] value: 42

[Function]
regex-apropos-list regex &optional packages &key upcase => list

Like APROPOS-LIST but searches for interned symbols which match the regular expression regex. If case-insensitive is true (which is the default) and regex isn't already a scanner, a case-insensitive scanner is used.

Example (continued from above):

* (cl-ppcre:regex-apropos-list "foo(?:bar)?")
(|foobar| FOOBOO FOO)

[Special variable]
*regex-char-code-limit*

This variable controls whether scanners take into account all characters of your CL implementation or only those the CHAR-CODE of which is not larger than its value. It is only relevant if the regular expression contains certain character classes. The default is CHAR-CODE-LIMIT, and you might see significant speed and space improvements during scanner creation if, say, your target strings only contain ISO-8859-1 characters and you're using an implementation like AllegroCL, LispWorks, or CLISP where CHAR-CODE-LIMIT has a value much higher than 255. The test suite will automatically set *REGEX-CHAR-CODE-LIMIT* to 255 while you're running the default test.

Here's an example with LispWorks:

CL-USER 23 > (time (cl-ppcre:create-scanner "[3\\D]"))
Timing the evaluation of (CL-PPCRE:CREATE-SCANNER "[3\\D]")

user time    =      0.443
system time  =      0.001
Elapsed time =   0:00:01
Allocation   = 546600 bytes standard / 2162611 bytes fixlen
0 Page faults
#<closure 20654AF2>

CL-USER 24 > (time (let ((cl-ppcre:*regex-char-code-limit* 255)) (cl-ppcre:create-scanner "[3\\D]")))
Timing the evaluation of (LET ((CL-PPCRE:*REGEX-CHAR-CODE-LIMIT* 255)) (CL-PPCRE:CREATE-SCANNER "[3\\D]"))

user time    =      0.000
system time  =      0.000
Elapsed time =   0:00:00
Allocation   = 3336 bytes standard / 8338 bytes fixlen
0 Page faults
#<closure 206569DA>

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *REGEX-CHAR-CODE-LIMIT* is bound at that time. The default value should always yield correct results unless you play dirty tricks with implementation-dependent behaviour, though.

[Special variable]
*use-bmh-matchers*

Usually, the scanners created by CREATE-SCANNER (or implicitely by other functions and macros) will use fast Boyer-Moore-Horspool matchers to check for constant strings at the start or end of the regular expression. If *USE-BMH-MATCHERS* is NIL (the default is T), the standard function SEARCH will be used instead. This will usually be a bit slower but can save lots of space if you're storing many scanners. The test suite will automatically set *USE-BMH-MATCHERS* to NIL while you're running the default test.

Note: Due to the nature of LOAD-TIME-VALUE and the compiler macro for SCAN some scanners might be created in a null lexical environment at load time or at compile time so be careful to which value *USE-BMH-MATCHERS* is bound at that time.

[Special variable]
*allow-quoting*

If this value is true (the default is NIL) CL-PPCRE will support \Q and \E in regex strings to quote (disable) metacharacters. Note that this entails a slight performance penalty when creating scanners because (a copy of) the regex string is modified (probably more than once) before it is fed to the parser. Also, the parser's syntax error messages will complain about the converted string and not about the original regex string.

* (cl-ppcre:scan "^a+$" "a+")
NIL

* (let ((cl-ppcre:*allow-quoting* t))
    (cl-ppcre:scan "^\\Qa+\\E$" "a+"))
0
2
#()
#()

* (let ((cl-ppcre:*allow-quoting* t))
    (cl-ppcre:scan "\\Qa()\\E(?#comment\\Q)a**b" "()ab"))

Quantifier '*' not allowed at position 19 in string "a\\(\\)(?#commentQ)a**b"

Note how in the last example the regex string in the error message is different from the first argument to the SCAN function. Also note that the second example might be easier to understand (and Lisp-ier) if you write it like this:

* (cl-ppcre:scan '(:sequence :start-anchor
                             "a+" ;; no quoting necessary
                             :end-anchor)
                 "a+")
0
2
#()
#()

Make sure you also read the relevant section in "Bugs and problems."

[Function]
quote-meta-chars string => string'

This is a simple utility function used when *ALLOW-QUOTING* is true. It returns a string STRING' where all non-word characters (everything except ASCII characters, digits and underline) of STRING are quoted by prepending a backslash similar to Perl's quotemeta function. It always returns a fresh string.

* (cl-ppcre:quote-meta-chars "[a-z]*")
"\\[a\\-z\\]\\*"

[Condition type]
ppcre-error

Every error signaled by CL-PPCRE is of type PPCRE-ERROR. This is a direct subtype of SIMPLE-ERROR without any additional slots or options.

[Condition type]
ppcre-invocation-error

Errors of type PPCRE-INVOCATION-ERROR are signaled if one of the exported functions of CL-PPCRE is called with wrong or inconsistent arguments. This is a direct subtype of PPCRE-ERROR without any additional slots or options.

[Condition type]
ppcre-syntax-error

An error of type PPCRE-SYNTAX-ERROR is signaled if CL-PPCRE's parser encounters an error when trying to parse a regex string or to convert a parse tree into its internal representation. This is a direct subtype of PPCRE-ERROR with two additional slots. These denote the regex string which HTML-PPCRE was parsing and the position within the string where the error occured. If the error happens while CL-PPCRE is converting a parse tree both of these slots contain NIL. (See the next two entries on how to access these slots.)

As many syntax errors can't be detected before the parser is at the end of the stream, the row and column usually denote the last position where the parser was happy and not the position where it gave up.

* (handler-case
    (cl-ppcre:scan "foo**x" "fooox")
    (cl-ppcre:ppcre-syntax-error (condition)
      (format t "Houston, we've got a problem with the string ~S:~%~
                 Looks like something went wrong at position ~A.~%~
                 The last message we received was \"~?\"."
              (cl-ppcre:ppcre-syntax-error-string condition)
              (cl-ppcre:ppcre-syntax-error-pos condition)
              (simple-condition-format-control condition)
              (simple-condition-format-arguments condition))
      (values)))
Houston, we've got a problem with the string "foo**x":
Looks like something went wrong at position 4.
The last message we received was "Quantifier '*' not allowed".

[Function]
ppcre-syntax-error-string condition => string

If condition is a condition of type PPCRE-SYNTAX-ERROR this function will return the string the parser was parsing when the error was encountered (or NIL if the error happened while trying to convert a parse tree). This might be particularly useful when *ALLOW-QUOTING* is true because in this case the offending string might not be the one you gave to the CREATE-SCANNER function.

[Function]
ppcre-syntax-error-pos condition => number

If condition is a condition of type PPCRE-SYNTAX-ERROR this function will return the position within the string where the error occured (or NIL if the error happened while trying to convert a parse tree).

Filters

Note that filters are currently considered an experimental feature and their API might change in the future.

A filter is defined by its filter function which must be a function of one argument. During the parsing process this function might be called once or several times or it might not be called at all. If it's called its argument is an integer pos which is the current position within the target string. The filter can either return NIL (which means that the subexpression represented by this filter didn't match) or an integer not smaller than pos for success. A zero-length assertion should return pos itself while a filter which wants to consume N characters should return (+ POS N).

If you supply the optional value length and it is not NIL then this is a promise to the regex engine that your filter will always consume exactly length characters. The regex engine might use this information for optimization purposes but it is otherwise irrelevant to the outcome of the matching process.

The filter function can access the following special variables from its code body:

• CL-PPCRE::*STRING*: The target (a string) of the current matching process.
• CL-PPCRE::*START-POS* and CL-PPCRE::*END-POS*: The start and end (integers) indices of the current matching process. These correspond to the START and END keyword parameters of SCAN.
• CL-PPCRE::*REAL-START-POS*: The initial starting position. This is only relevant for repeated scans (as in DO-SCANS) where CL-PPCRE::*START-POS* will be moved forward while CL-PPCRE::*REAL-START-POS* won't. For normal scans the value of this variable is NIL.
• CL-PPCRE::*REG-STARTS* and CL-PPCRE::*REG-ENDS*: Two simple vectors which denote the start and end indices of registers within the regular expression. The first register is indexed by 0. If a register hasn't matched yet then its corresponding entry in CL-PPCRE::*REG-STARTS* is NIL.

Note that the names of the variables are not exported from the CL-PPCRE package because there's currently no guarantee that they will be available in future releases.

Here are some filter examples:

* (defun my-info-filter (pos)
    "Show some info about the matching process."
    (format t "Called at position ~A~%" pos)
    (loop with dim = (array-dimension cl-ppcre::*reg-starts* 0)
          for i below dim
          for reg-start = (aref cl-ppcre::*reg-starts* i)
          for reg-end = (aref cl-ppcre::*reg-ends* i)
          do (format t "Register ~A is currently " (1+ i))
          when reg-start
               (write-string cl-ppcre::*string* nil
            do (write-char #\')
               (write-string cl-ppcre::*string* nil
                     :start reg-start :end reg-end)
               (write-char #\')
          else
            do (write-string "unbound")
          do (terpri))
    (terpri)
    pos)
MY-INFO-FILTER

* (scan '(:sequence
           (:register
             (:greedy-repetition 0 nil
                                 (:char-class (:range #\a #\z))))
           (:filter my-info-filter 0) "X")
        "bYcdeX")
Called at position 1
Register 1 is currently 'b'

Called at position 0
Register 1 is currently ''

Called at position 1
Register 1 is currently ''

Called at position 5
Register 1 is currently 'cde'

2
6
#(2)
#(5)

* (scan '(:sequence
           (:register
             (:greedy-repetition 0 nil
                                 (:char-class (:range #\a #\z))))
           (:filter my-info-filter 0) "X")
        "bYcdeZ")
NIL

* (defun my-weird-filter (pos)
    "Only match at this point if either pos is odd and the character
  we're looking at is lowerrcase or if pos is even and the next two
  characters we're looking at are uppercase. Consume these characters if
  there's a match."
    (format t "Trying at position ~A~%" pos)
    (cond ((and (oddp pos)
                (< pos cl-ppcre::*end-pos*)
                (lower-case-p (char cl-ppcre::*string* pos)))
           (1+ pos))
          ((and (evenp pos)
                (< (1+ pos) cl-ppcre::*end-pos*)
                (upper-case-p (char cl-ppcre::*string* pos))
                (upper-case-p (char cl-ppcre::*string* (1+ pos))))
           (+ pos 2))
          (t nil)))
MY-WEIRD-FILTER

* (defparameter *weird-regex*
                `(:sequence "+" (:filter ,#'my-weird-filter) "+"))
*WEIRD-REGEX*

* (scan *weird-regex* "+A++a+AA+")
Trying at position 1
Trying at position 3
Trying at position 4
Trying at position 6
5
9
#()
#()

* (fmakunbound 'my-weird-filter)
MY-WEIRD-FILTER

* (scan *weird-regex* "+A++a+AA+")
Trying at position 1
Trying at position 3
Trying at position 4
Trying at position 6
5
9
#()
#()

For more ideas about what you can do with filters see this thread on the mailing list.
 

Testing CL-PPCRE

* (mk:compile-system "cl-ppcre-test")
; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
; Loading #p"/home/edi/cl-ppcre/packages.x86f".
; Loading #p"/home/edi/cl-ppcre/specials.x86f".
; Loading #p"/home/edi/cl-ppcre/util.x86f".
; Loading #p"/home/edi/cl-ppcre/errors.x86f".
; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
; Loading #p"/home/edi/cl-ppcre/parser.x86f".
; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
; Loading #p"/home/edi/cl-ppcre/convert.x86f".
; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
; Loading #p"/home/edi/cl-ppcre/closures.x86f".
; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
; Loading #p"/home/edi/cl-ppcre/api.x86f".
; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".
NIL

* (cl-ppcre-test:test)

;; ....
;; (a list of incompatibilities with Perl)

With LispWorks, SCL, and SBCL (starting from version 0.8.4.8) you can also call CL-PPCRE-TEST:TEST with a keyword argument argument THREADED which - in addition to the usual tests - will also check whether the scanners created by CL-PPCRE are thread-safe.

Note that the file testdata provided with CL-PPCRE was created on a Linux system with Perl 5.8.0. You can (and you should if you're on Mac OS or Windows) create your own testdata with the Perl script perltest.pl:

edi@bird:~/cl-ppcre > perl perltest.pl < testinput > testdata

Compatibility with Perl

Empty strings instead of undef in $1, $2, etc.

Strange scoping of embedded modifiers

Inconsistent capturing of $1, $2, etc.

Captured groups not available outside of look-aheads and look-behinds

Alternations don't always work from left to right

"\r" doesn't work with MCL

What about "\w"?

Performance

Benchmarking

edi@bird:~/cl-ppcre > echo "/((a{0,5}){0,5})*[c]/
aaaaaaaaaaaac

/((a{0,5})*)*[c]/
aaaaaaaaaaaac" | perl perltest.pl .5 > timedata
1
2

edi@bird:~/cl-ppcre > cmucl -quiet
; Loading #p"/home/edi/.cmucl-init".

* (mk:compile-system "cl-ppcre-test")
; Loading #p"/home/edi/cl-ppcre/cl-ppcre.system".
; Loading #p"/home/edi/cl-ppcre/packages.x86f".
; Loading #p"/home/edi/cl-ppcre/specials.x86f".
; Loading #p"/home/edi/cl-ppcre/util.x86f".
; Loading #p"/home/edi/cl-ppcre/errors.x86f".
; Loading #p"/home/edi/cl-ppcre/lexer.x86f".
; Loading #p"/home/edi/cl-ppcre/parser.x86f".
; Loading #p"/home/edi/cl-ppcre/regex-class.x86f".
; Loading #p"/home/edi/cl-ppcre/convert.x86f".
; Loading #p"/home/edi/cl-ppcre/optimize.x86f".
; Loading #p"/home/edi/cl-ppcre/closures.x86f".
; Loading #p"/home/edi/cl-ppcre/repetition-closures.x86f".
; Loading #p"/home/edi/cl-ppcre/scanner.x86f".
; Loading #p"/home/edi/cl-ppcre/api.x86f".
; Loading #p"/home/edi/cl-ppcre/ppcre-tests.x86f".
NIL

* (cl-ppcre-test:test :file-name "/home/edi/cl-ppcre/timedata")
   1: 0.5559 (1000000 repetitions, Perl: 4.5330 seconds, CL-PPCRE: 2.5200 seconds)
   2: 0.4573 (1000000 repetitions, Perl: 4.5922 seconds, CL-PPCRE: 2.1000 seconds)
NIL

Here are some more benchmarks (done with Perl 5.6.1 and CMUCL 18d+):

Test case	Repetitions	Perl (sec)	CL-PPCRE (sec)	Ratio CL-PPCRE/Perl
"@{['x' x 100]}" =~ /(.)*/s	100000	0.1394	0.0700	0.5022
"@{['x' x 1000]}" =~ /(.)*/s	100000	0.1628	0.0600	0.3685
"@{['x' x 10000]}" =~ /(.)*/s	100000	0.5071	0.0600	0.1183
"@{['x' x 100000]}" =~ /(.)*/s	10000	0.3902	0.0000	0.0000
"@{['x' x 100]}" =~ /.*/	100000	0.1520	0.0800	0.5262
"@{['x' x 1000]}" =~ /.*/	100000	0.3786	0.5400	1.4263
"@{['x' x 10000]}" =~ /.*/	10000	0.2709	0.5100	1.8826
"@{['x' x 100000]}" =~ /.*/	1000	0.2734	0.5100	1.8656
"@{['x' x 100]}" =~ /.*/s	100000	0.1320	0.0300	0.2274
"@{['x' x 1000]}" =~ /.*/s	100000	0.1634	0.0300	0.1836
"@{['x' x 10000]}" =~ /.*/s	100000	0.5304	0.0300	0.0566
"@{['x' x 100000]}" =~ /.*/s	10000	0.3966	0.0000	0.0000
"@{['x' x 100]}" =~ /x*/	100000	0.1507	0.0900	0.5970
"@{['x' x 1000]}" =~ /x*/	100000	0.3782	0.6300	1.6658
"@{['x' x 10000]}" =~ /x*/	10000	0.2730	0.6000	2.1981
"@{['x' x 100000]}" =~ /x*/	1000	0.2708	0.5900	2.1790
"@{['x' x 100]}" =~ /[xy]*/	100000	0.2637	0.1500	0.5688
"@{['x' x 1000]}" =~ /[xy]*/	10000	0.1449	0.1200	0.8282
"@{['x' x 10000]}" =~ /[xy]*/	1000	0.1344	0.1100	0.8185
"@{['x' x 100000]}" =~ /[xy]*/	100	0.1355	0.1200	0.8857
"@{['x' x 100]}" =~ /(.)*/	100000	0.1523	0.1100	0.7220
"@{['x' x 1000]}" =~ /(.)*/	100000	0.3735	0.5700	1.5262
"@{['x' x 10000]}" =~ /(.)*/	10000	0.2735	0.5100	1.8647
"@{['x' x 100000]}" =~ /(.)*/	1000	0.2598	0.5000	1.9242
"@{['x' x 100]}" =~ /(x)*/	100000	0.1565	0.1300	0.8307
"@{['x' x 1000]}" =~ /(x)*/	100000	0.3783	0.6600	1.7446
"@{['x' x 10000]}" =~ /(x)*/	10000	0.2720	0.6000	2.2055
"@{['x' x 100000]}" =~ /(x)*/	1000	0.2725	0.6000	2.2020
"@{['x' x 100]}" =~ /(y|x)*/	10000	0.2411	0.1000	0.4147
"@{['x' x 1000]}" =~ /(y|x)*/	1000	0.2313	0.0900	0.3891
"@{['x' x 10000]}" =~ /(y|x)*/	100	0.2336	0.0900	0.3852
"@{['x' x 100000]}" =~ /(y|x)*/	10	0.4165	0.0900	0.2161
"@{['x' x 100]}" =~ /([xy])*/	100000	0.2678	0.1800	0.6721
"@{['x' x 1000]}" =~ /([xy])*/	10000	0.1459	0.1200	0.8227
"@{['x' x 10000]}" =~ /([xy])*/	1000	0.1372	0.1100	0.8017
"@{['x' x 100000]}" =~ /([xy])*/	100	0.1358	0.1100	0.8098
"@{['x' x 100]}" =~ /((x){2})*/	10000	0.1073	0.0400	0.3727
"@{['x' x 1000]}" =~ /((x){2})*/	10000	0.9146	0.2400	0.2624
"@{['x' x 10000]}" =~ /((x){2})*/	1000	0.9020	0.2300	0.2550
"@{['x' x 100000]}" =~ /((x){2})*/	100	0.8983	0.2300	0.2560
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/	100000	0.2829	0.2300	0.8129
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/	10000	0.1859	0.1700	0.9143
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /[a-z]*FOOBARBAZ/	1000	0.1420	0.1700	1.1968
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/	1000000	0.9196	0.4600	0.5002
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/	100000	0.2166	0.2500	1.1542
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/	10000	0.1465	0.2300	1.5696
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/	100000	0.2917	0.2600	0.8915
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/	10000	0.1811	0.1800	0.9942
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z])*FOOBARBAZ/	1000	0.1424	0.1600	1.1233
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z])*FOOBARBAZ/	1000000	0.9154	0.7400	0.8083
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z])*FOOBARBAZ/	100000	0.2170	0.2800	1.2901
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z])*FOOBARBAZ/	10000	0.1497	0.2300	1.5360
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/	10000	0.4359	0.1500	0.3441
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/	1000	0.5456	0.1500	0.2749
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}FOOBARBAZ" =~ /([a-z]|ab)*FOOBARBAZ/	10	0.2039	0.0600	0.2943
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/	1000000	0.9311	0.7400	0.7947
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/	100000	0.2162	0.2700	1.2489
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /([a-z]|ab)*FOOBARBAZ/	10000	0.1488	0.2300	1.5455
"@{[join undef, map { chr(ord('a') + rand 26) } (1..100)]}NOPE" =~ /[a-z]*FOOBARBAZ/i	1000	0.1555	0.0000	0.0000
"@{[join undef, map { chr(ord('a') + rand 26) } (1..1000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i	10	0.1441	0.0000	0.0000
"@{[join undef, map { chr(ord('a') + rand 26) } (1..10000)]}NOPE" =~ /[a-z]*FOOBARBAZ/i	10	13.7150	0.0100	0.0007

As you might have noticed, Perl shines if it can reduce significant parts of the matching process to cases where it can advance through the target string one character at a time. This leads to C code where you can very efficiently test and increment a pointer into a string in a tight loop and can hardly be beaten with CL. In almost all other cases, the CMUCL/CL-PPCRE combination is usually faster than Perl - sometimes a lot faster.

As most of the examples above were chosen to make Perl look good here's another benchmark - the result of running perltest.pl against the full testdata file with a time limit of 0.1 seconds, CL-PPCRE 0.1.2 on CMUCL 18e-pre vs. Perl 5.6.1. CL-PPCRE is faster than Perl in 1511 of 1545 cases - in 1045 cases it's more than twice as fast.

Note that Perl as well as CL-PPCRE keep the rightmost matches in registers - keep that in mind if you benchmark against other regex implementations. Also note that CL-PPCRE-TEST:TEST automatically skips test cases where Perl and CL-PPCRE don't agree.

Other performance issues

However, beginning with version 0.5.2, CL-PPCRE uses a compiler macro and LOAD-TIME-VALUE to make sure that the scanner is only built once if the first argument to SCAN, SCAN-TO-STRINGS, SPLIT, REGEX-REPLACE, or REGEX-REPLACE-ALL is a constant form. (But see the notes for *REGEX-CHAR-CODE-LIMIT* and *USE-BMH-MATCHERS*.)

Here's an example of its effect

* (trace cl-ppcre::convert)
(CL-PPCRE::CONVERT)
* (defun foo (string) (cl-ppcre:scan "(?s).*" string))
FOO
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

  0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
  0: CL-PPCRE::CONVERT returned
       #<CL-PPCRE::SEQ {48B033C5}>
       0
       #<CL-PPCRE::EVERYTHING {48B031D5}>
Evaluation took:
  0.0 seconds of real time
  0.00293 seconds of user run time
  9.77e-4 seconds of system run time
  0 page faults and
  11,408 bytes consed.
0
19
#()
#()
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

  0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
  0: CL-PPCRE::CONVERT returned
       #<CL-PPCRE::SEQ {48B14C4D}>
       0
       #<CL-PPCRE::EVERYTHING {48B14B65}>
Evaluation took:
  0.0 seconds of real time
  0.00293 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  10,960 bytes consed.
0
19
#()
#()
* (compile 'foo)
  0: (CL-PPCRE::CONVERT #<lambda-list-unavailable>)
  0: CL-PPCRE::CONVERT returned
       #<CL-PPCRE::SEQ {48B1FEC5}>
       0
       #<CL-PPCRE::EVERYTHING {48B1FDDD}>
Compiling LAMBDA (STRING): 
Compiling Top-Level Form: 
FOO
NIL
NIL
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

Evaluation took:
  0.0 seconds of real time
  0.0 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  0 bytes consed.
0
19
#()
#()
* (time (foo "The quick brown fox"))
Compiling LAMBDA NIL: 
Compiling Top-Level Form: 

Evaluation took:
  0.0 seconds of real time
  0.0 seconds of user run time
  0.0 seconds of system run time
  0 page faults and
  0 bytes consed.
0
19
#()
#()
* 

Of course, the usual rules for creating efficient regular expressions apply to CL-PPCRE as well although it can optimize a couple of cases itself. The most important rule is probably that you shouldn't use capturing groups if you don't need the captured information, i.e. use "(?:a|b)*" instead of "(a|b)*" if you don't need to refer to the register. (In fact, in this particular case CL-PPCRE will be able to optimize away the register group, but it won't if you replace "a|b" with, say, "a|bc".)

Another point worth mentioning is that you definitely should use single-line mode if you have long strings without #\Newline (or where you don't care about the line breaks) and plan to use regular expressions like ".*". See the benchmarks for comparisons between single-line mode and normal mode with such target strings.

Another thing to consider is that, for performance reasons, CL-PPCRE assumes that most of the target strings you're trying to match are simple strings and coerces non-simple strings to simple strings before scanning them. If you plan on working with non-simple strings mostly you might consider modifying the CL-PPCRE source code. This is easy: Change all occurences of SCHAR to CHAR and redefine the macro in util.lisp where the coercion takes place - that's all.
 

Bugs and problems

Stack overflow

Here's one example with CLISP:

[1]> (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
TARGET

[2]> (cl-ppcre:scan "a*" (target 1000))
0 ;
1000 ;
#() ;
#()

[3]> (cl-ppcre:scan "(?:a|b)*" (target 1000))
0 ;
1001 ;
#() ;
#()

[4]> (cl-ppcre:scan "(a|b)*" (target 1000))
0 ;
1001 ;
#(1000) ;
#(1001)

[5]> (cl-ppcre:scan "(a|b)*" (target 10000))
0 ;
10001 ;
#(10000) ;
#(10001)

[6]> (cl-ppcre:scan "(a|b)*" (target 100000))
0 ;
100001 ;
#(100000) ;
#(100001)

[7]> (cl-ppcre:scan "(a|b)*" (target 1000000))
0 ;
1000001 ;
#(1000000) ;
#(1000001)

;; No problem until now - but...

[8]> (cl-ppcre:scan "(a|)*" (target 100000))
*** - Lisp stack overflow. RESET

[9]> (cl-ppcre:scan "(a|)*" (target 3200))
*** - Lisp stack overflow. RESET

With CMUCL the situation is better and worse at the same time. It will take a lot longer until CMUCL gives up but if it gives up the whole Lisp image will silently die (at least on my machine):

[Note: This was true for CMUCL 18e - CMUCL 19a behaves in a much nicer way and gives you a chance to recover.]

* (defun target (n) (concatenate 'string (make-string n :initial-element #\a) "b"))
TARGET

* (cl-ppcre:scan "(a|)*" (target 3200))
0
3200
#(3200)
#(3200)

* (cl-ppcre:scan "(a|)*" (target 10000))
0
10000
#(10000)
#(10000)

* (cl-ppcre:scan "(a|)*" (target 100000))
0
100000
#(100000)
#(100000)

* (cl-ppcre:scan "(a|)*" (target 1000000))
0
1000000
#(1000000)
#(1000000)

;; No problem until now - but...

* (cl-ppcre:scan "(a|)*" (target 10000000))
edi@bird:~ >

You might want to compare this to the way Perl handles the same situation. It might lie to you:

edi@bird:~ > perl -le '$_="a" x 32766 . "b"; /(a|)*/; print $1'

edi@bird:~ > perl -le '$_="a" x 32767 . "b"; /(a|)*/; print $1'
a

edi@bird:~ > perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
Complex regular subexpression recursion limit (32766) exceeded at -e line 1.
a

edi@bird:~ > /opt/perl-5.8/bin/perl -lwe '$_="a" x 32767 . "b"; /(a|)*/; print $1'
Segmentation fault

"\Q" doesn't work, or does it?

#!/usr/bin/perl -l

$a = '\E*';
print 1
  if '\E*\E*' =~ /(?:\Q$a\E){2}/;

* (let ((cl-ppcre:*allow-quoting* t)
        (a "\\E*"))
    (cl-ppcre:scan (concatenate 'string "(?:\\Q" a "\\E){2}") "\\E*\\E*"))
Quantifier '*' not allowed at position 3 in string "(?:*\\E){2}"

What gives? "\\Q...\\E" in CL-PPCRE should only be used in literal strings. If you want to quote arbitrary strings try CL-INTERPOL or use QUOTE-META-CHARS:

* (let ((a "\\E*"))
    (cl-ppcre:scan (concatenate 'string
                                "(?:" (cl-ppcre:quote-meta-chars a) "){2}")
                   "\\E*\\E*"))
0
6
#()
#()

* (let ((a "\\E*"))
    (cl-ppcre:scan `(:greedy-repetition 2 2 ,a)
                   "\\E*\\E*"))
0
6
#()
#()

Backslashes may confuse you...

* (let ((a "y\\y"))
    (cl-ppcre:scan a a))
NIL

'y\y' =~ /y\y/;

$a = 'y\y';
$a =~ /$a/;

Still confused? You might want to try CL-INTERPOL.
 

Remarks

All test cases and benchmarks in this document where performed on an IBM Thinkpad T23 laptop (Pentium III 1.2 GHz, 768 MB RAM) running Gentoo Linux 1.1a.
 

AllegroCL compatibility mode

The advantage of this mode is that you'll get a much smaller image and most likely faster code. (But note that CL-PPCRE needs to do a small amount of work to massage AllegroCL's output into the format expected by CL-PPCRE.) The downside is that your code won't be fully compatible with CL-PPCRE anymore. Here are some of the differences (most of which probably don't matter very often):

• The AllegroCL engine doesn't offer parse tree synonyms and filters.
• The AllegroCL engine will choke on some regular expressions involving curly braces that are accepted by Perl and CL-PPCRE's native engine.
• The AllegroCL engine's case-folding mode switch (which is used instead of CL-PPCRE's :CASE-INSENSITIVE keyword parameter) is currently only effective for ASCII characters.
• CL-PPCRE's engine doesn't understand the named register groups provided by AllegroCL.
• The AllegroCL engine doesn't support quoting of metacharacters.
• In AllegroCL compatibility mode compiled regular expressions (as returned by CREATE-SCANNER) aren't functions but structures.

To use the AllegroCL compatibility mode you have to

(push :use-acl-regexp2-engine *features*)

Acknowledgements

The asdf system definitions were kindly provided by Marco Baringer. Hannu Koivisto provided patches to make the .system files more usable. Thanks to Kevin Rosenberg and Douglas Crosher for pointing out how to be friendly to case-sensitive ACL images. Thanks to Karsten Poeck and JP Massar for their help in making CL-PPCRE work with Corman Lisp. JP Massar and Kent M. Pitman also helped to improve/fix the test suite and the compiler macro.

Thanks to the guys at "Café Olé" in Hamburg where I wrote most of the code and thanks to my wife for lending me her PowerBook to test CL-PPCRE with MCL and OpenMCL.

$Header: /lingo/CVS/lkb/src/ppcre/ppcre.html,v 1.3 2006/02/08 14:02:56 oe Exp $

BACK TO MY HOMEPAGE