Try increasing the size of the segment size used by APC. Use only one segment. Also access the wp admin interface from a subdomain you create.
If there are other vhosts on your server which does not need opcode caching, you can disable APC for these sites. You can do it on vhost level by setting
apc.cache_by_default=0 in the apc.ini file, and put
php_flag apc.cache_by_default On in the .htaccess file on your wp root directory. That should be the reason for the fragmentation.
Changes in the files also can cause fragmentation. The edited file will be deleted and the new file will be added to the cache. If you haven't done already, you should also set
apc.stat=0. This will improve the overall performance by not checking everytime if the files are changed or not.
If you don't want WP Admin to be cached you can create a subdomain like admin.example.com and you can access the admin panel. By doing like this you will be able to disable opcode caching. Which will also decrease fragmentation.
Update: Disabling object caching and db caching help reducing the fragmentation. Us'ng redis or memcached for object caching and APC for only opcode caching solves the problem.