RegEx字表现：\ w vs [a-zA-Z0-9_]

Question

I'd like to know the list of chars that \w passes, is it just [a-zA-Z0-9_] or are there more chars that it might cover?

I'm asking this question, because based on this, \d is different with [0-9] and is less efficient.

\w vs [a-zA-Z0-9_]: which one might be faster in large scale?

Answer 1

[ This answer is Perl-specific. The information within may not apply to PCRE or the engine used by the other languages tagged. ]

/\w/aa (the actual equivalent of /[a-zA-Z0-9_]/) is usually faster, but not always. That said, the difference is so minimal (less than 1 nanosecond per check) that it shouldn't be a concern. To put it in to context, it takes far, far longer to call a sub or start the regex engine.

What follows covers this in detail.

---

First of all, \w isn't the same as [a-zA-Z0-9_] by default. \w matches every alphabetic, numeric, mark and connector punctuation Unicode Code Point. There are 119,821 of these!^[1] Determining which is the fastest of non-equivalent code makes no sense.

However, using \w with /aa ensures that \w only matches [a-zA-Z0-9_]. So that's what we're going to be using for our benchmarks. (Actually, we'll use both.)

(Note that each test performs 10 million checks, so a rate of 10.0/s actually means 10.0 million checks per second.)

---

ASCII-only positive match
               Rate [a-zA-Z0-9_]      (?u:\w)     (?aa:\w)
[a-zA-Z0-9_] 39.1/s           --         -26%         -36%
(?u:\w)      52.9/s          35%           --         -13%
(?aa:\w)     60.9/s          56%          15%           --

When finding a match in ASCII characters, ASCII-only \w and Unicode \w both beat the explicit class.

/\w/aa is ( 1/39.1 - 1/60.9 ) / 10,000,000 = 0.000,000,000,916 s faster on my machine

---

ASCII-only negative match
               Rate      (?u:\w)     (?aa:\w) [a-zA-Z0-9_]
(?u:\w)      27.2/s           --          -0%         -12%
(?aa:\w)     27.2/s           0%           --         -12%
[a-zA-Z0-9_] 31.1/s          14%          14%           --

When failing to find a match in ASCII characters, the explicit class beats ASCII-only \w.

/[a-zA-Z0-9_]/ is ( 1/27.2 - 1/31.1 ) / 10,000,000 = 0.000,000,000,461 s faster on my machine

---

Non-ASCII positive match
               Rate      (?u:\w) [a-zA-Z0-9_]     (?aa:\w)
(?u:\w)      2.97/s           --        -100%        -100%
[a-zA-Z0-9_] 3349/s      112641%           --          -9%
(?aa:\w)     3664/s      123268%           9%           --

Whoa. This tests appears to be running into some optimization. That said, running the test multiple times yields extremely consistent results. (Same goes for the other tests.)

When finding a match in non-ASCII characters, ASCII-only \w beats the explicit class.

/\w/aa is ( 1/3349 - 1/3664 ) / 10,000,000 = 0.000,000,000,002,57 s faster on my machine

---

Non-ASCII negative match
               Rate      (?u:\w) [a-zA-Z0-9_]     (?aa:\w)
(?u:\w)      2.66/s           --          -9%         -71%
[a-zA-Z0-9_] 2.91/s          10%           --         -68%
(?aa:\w)     9.09/s         242%         212%           --

When failing to find a match in non-ASCII characters, ASCII-only \w beats the explicit class.

/[a-zA-Z0-9_]/ is ( 1/2.91 - 1/9.09 ) / 10,000,000 = 0.000,000,002,34 s faster on my machine

---

Conclusions

• I'm surprised there's any difference between /\w/aa and /[a-zA-Z0-9_]/.
• In some situation, /\w/aa is faster; in others, /[a-zA-Z0-9_]/.
• The difference between /\w/aa and /[a-zA-Z0-9_]/ is very minimal (less than 1 nanosecond).
• The difference is so minimal that you shouldn't be concerned about it.
• Even the difference between /\w/aa and /\w/u is quite small despite the latter matching 4 orders of magnitude more characters than the former.

---

use strict;
use warnings;
use feature qw( say );

use Benchmarks qw( cmpthese );

my %pos_tests = (
   '(?u:\\w)'     => '/^\\w*\\z/u',
   '(?aa:\\w)'    => '/^\\w*\\z/aa',
   '[a-zA-Z0-9_]' => '/^[a-zA-Z0-9_]*\\z/',
);

my %neg_tests = (
   '(?u:\\w)'     => '/\\w/u',
   '(?aa:\\w)'    => '/\\w/aa',
   '[a-zA-Z0-9_]' => '/[a-zA-Z0-9_]/',
);

$_ = sprintf( 'use strict; use warnings; our $s; for (1..1000) { $s =~ %s }', $_)
   for
      values(%pos_tests),
      values(%neg_tests);

local our $s;

say "ASCII-only positive match";
$s = "J" x 10_000;
cmpthese(-3, \%pos_tests);

say "";

say "ASCII-only negative match";
$s = "!" x 10_000;
cmpthese(-3, \%neg_tests);

say "";

say "Non-ASCII positive match";
$s = "\N{U+0100}" x 10_000;
cmpthese(-3, \%pos_tests);

say "";

say "Non-ASCII negative match";
$s = "\N{U+2660}" x 10_000;
cmpthese(-3, \%neg_tests);

---

1. Unicode version 11.