There doesn't appear to be a way to list multiple testsuites from a phpunit.xml file, but then only run one. However, if you do have some control over a fuller integration and testing environment where you can configure things more exactly, you can have more than one phpunit configuration files, and set one (or more) with more involved environments to set the command-line parameter
--configuration <file> option with a configuration that will do more. This at least makes sure that the simplest config will run in the easiest fashion.
The two files can be called whatever you like if you run them specifically, but it may be worth thinking about having the quick-to-run file called the default
phpunit.xml, and the specifically named and extended filem as
phpunit.xml.dist. The .dist file will be automatically run by default if the original plain
.xml does not exist. Another option is to have the
phpunit.xml.dist file in a code repository, but then copy it to a
phpunit.xml file, with less testsuite's, which is not in itself checked into version control, and is only kept locally (it would probably also be marked as ignored in a .gitignore file, or similar).