Your first rule with .* and an optional / is not strict enough and is matching for both URLs.
This updated rule should allow for the URLs you are expecting. The [] is a character class if there are more characters you need to allow add them in there. The {} is a range, first number is minimum, second maximum.
RewriteRule ^category_([a-zA-Z]{4,20})/?$ site/category.php?category=$1 [NC,L]
RewriteRule ^category_(.*)_(.*)/?$ site/category.php?category=$1&page=$2 [NC,L]
A good place to test regex's is regex101. It shows what the regex is matching and explains each part of it.
Original example: https://regex101.com/r/zO7lI8/1
New example: https://regex101.com/r/zO7lI8/2
That should resolve your .htaccess issue.
Note on regex101 the delimiter is / so all /s need to be escaped, depending on the language/software the regex is being used a delimiter may not be needed. The / is not a special character unless it is the delimiter. So it doesn't need to be escaped.
This:
$category = $_GET["category"];
$result = "SELECT * FROM my".$category." ORDER BY id_".$category.";
Is still open to a sql injection though. A user could input anything into $_GET["category"] and get contents from your database. A malicious user could possibly even pull all usernames, passwords, and emails.
Take a look at:
How can I prevent SQL injection in PHP?
http://php.net/manual/en/security.database.sql-injection.php
https://www.owasp.org/index.php/SQL_Injection_Prevention_Cheat_Sheet
for more information on SQL injections and how to prevent them.
