It hasn't so much to do with IBM-850, that's just a codepage filling out the 8th bit beyond ASCII to give a representation to the bytes you'll end up with.
The key here is the bitwise not operator ~
which flips all the bits - 0 becomes 1, and 1 becomes 0. If you dump ~"/^(.*?[aeiouy]+)/i"
to a file and open it up as 850 it'll look like:
ðíÎÐı└ñ×ÜûÉèåóÈÍðû
And likewise ~"$1 $1 $0"
looks like:
█╬▀█╬▀█¤
So you see where this is headed.
In PHP an undefined constant is assumed to have a string value matching its name. For example:
var_dump(foo);
Outputs string(3) "foo"
(as well as the notice "Use of undefined constant foo - assumed 'foo'", if notices are on.)
When either of the two gibberish strings above are put in a PHP script without quotes they're taken as undefined constants with their names assumed for their values as well.
Now prepend each with ~
to flip their bits back and you've got the original regex and replacement strings:
preg_filter("/^(.*?[aeiouy]+)/i","$1 $1 $0",$argn)
Only those parameters had their bits flipped because they're the only string literals, which is what this trick applies to. For each string it's shaving off a pair of quotes in exchange for taking on only a single tilde.
The bit flipping had to be done because either of the original strings on their own without quotes would've landed parse errors.
Clever way to net two bytes.