It definately sounds like a PHP Limitation, raising the memory limit probably isn't the best way to do it though, that leads to nothing but hassle, trust me.
Best method i can think of from the top of my head is to use Envoy (the server script method not the deployment service) to put together an SSH task, that way your job is being executed outside of PHP so you're not subject to the same memory limitations. Your Envoy script (
envoy.blade.php in your project root) would probably look something like this;
@servers(['your_server_name' => 'your.server.ip'])
@task('upload', ['on' => ['your_server_name']])
// perform your FTP setup, login etc.
I've only got one of these set up for a deployment job which is called from Jenkins so i'm not sure if you can launch it from within Laravel, but i launch from the command line like this;
vendor/bin/envoy run myJobName
Like i said the only thing i can't quite remember is if you can run Envoy from within Laravel itself, and the docs seem a little hazy on it, Definately an option worth checking out though :)