在 node.js 中复制文件的最快方式

Project that I am working at (node.js) implies a lot of operations with file system (copying/reading/writing etc). I'd like to know, what methods are the fastest, and I'd be happy for some advice.

转载于:https://stackoverflow.com/questions/11293857/fastest-way-to-copy-file-in-node-js

csdnceshi72
谁还没个明天 The only correct answer on the page is this one. None of the other answers actually copy files. Files on MacOS and Windows have other metadata that is lost by just copying bytes. Examples of data not copied by any other answer on this page, windows and macos. Even on Unix the other answers don't copy the creation date, something that's often important when copying a file.
接近 2 年之前 回复
csdnceshi63
elliott.david Mostly we're just fresh new and excited about this whole "files" business after years of normalizing browsers.
接近 6 年之前 回复
csdnceshi54
hurriedly% It's a good question, though it is interesting that it gets 25 upvotes when other similar format questions will get 3 or 4 downvotes right away for not meeting the SO "standards" (maybe the javascript tag is crawled by kinder people :)
大约 7 年之前 回复

13个回答

This is a good way to copy a file in one line of code using streams:

var fs = require('fs');

fs.createReadStream('test.log').pipe(fs.createWriteStream('newLog.log'));
csdnceshi56
lrony* this is not "copying a file". Files have things like creation dates (not copied with the code above). Files on MacOS and Windows also have other data streams (not copied with the code above). See the fs.copyFile answer. Looking in the node source fs.copyFile uses the OS level copy on MacOS and Windows and so should actually copy the files where as the code above mearly creates a new file and copies to bytes to the new file.
接近 2 年之前 回复
csdnceshi58
Didn"t forge Mikhail 's answer below uses Node's internal fs.copyFile function and is the preferred solution: stackoverflow.com/a/46253698
大约 2 年之前 回复
csdnceshi59
ℙℕℤℝ Note: "copy a file" implies a complete clone of the file, not just the data within the file. Copying a file would include the creation date of the original. When you're streaming from one file to another, you are copying the data, not the file.
大约 3 年之前 回复
csdnceshi72
谁还没个明天 I had the same problem. I used the solution given below by Tester and it worked.
接近 4 年之前 回复
csdnceshi65
larry*wei I make a copy of a file just after having created it, and it gives a blank file as a result. I'm not sure that using streams to copy files is a good practice. Best is to use fs-extra.copySync (see other answer in this thread), it works a lot better.
大约 4 年之前 回复
csdnceshi74
7*4 This is a common way. But if I want do things after copy, I must listen to an event "end". This is not convenient.
4 年多之前 回复
csdnceshi60
℡Wang Yan your code as written cannot be used together with process.exit because the latter terminates all IO without waiting for the streams to finish their data exchange
4 年多之前 回复
weixin_41568196
撒拉嘿哟木头 I had the same issue. The file would be blank and node would just hold a handle to that file forever. I used the fs-extra module's copy method instead
5 年多之前 回复
csdnceshi75
衫裤跑路 I used this method and all I got was a blank file on write. any ideas why? fs.createReadStream('./init/xxx.json').pipe(fs.createWriteStream('xxx.json'));
大约 6 年之前 回复
weixin_41568208
北城已荒凉 Unfortunately on my system using streams is extremely slow compared to child_process.execFile('/bin/cp', ['--no-target-directory', source, target]).
接近 7 年之前 回复
csdnceshi80
胖鸭 Both streams are closed by default. readstream is closed as such by node. writestream can be remained open if you pass { end: false} to pipe, otherwise will be closed by default. See here nodejs.org/api/…
大约 7 年之前 回复
csdnceshi78
程序go How about closing the files, does this code keep them opened after copy completed?
大约 7 年之前 回复
weixin_41568131
10.24 Well copy is not portable on Window, contrary to a full Node.js solution.
大约 7 年之前 回复
csdnceshi61
derek5. How much faster/slower is this than executing the raw cp test.log newLog.log via require('child_process').exec?
7 年多之前 回复
csdnceshi64
游.程 sure, usually i wrap such code in try catch construction.
大约 8 年之前 回复
csdnceshi69
YaoRaoLov Just remember that in real life, you'd want to check both the createReadStream and createWriteStream for errors, so you wouldn't get a one-liner (though it would still be just as fast).
大约 8 年之前 回复
csdnceshi64
游.程 You're right, i like this way of coding.
大约 8 年之前 回复

Same mechanism, but this adds error handling:

function copyFile(source, target, cb) {
  var cbCalled = false;

  var rd = fs.createReadStream(source);
  rd.on("error", function(err) {
    done(err);
  });
  var wr = fs.createWriteStream(target);
  wr.on("error", function(err) {
    done(err);
  });
  wr.on("close", function(ex) {
    done();
  });
  rd.pipe(wr);

  function done(err) {
    if (!cbCalled) {
      cb(err);
      cbCalled = true;
    }
  }
}
csdnceshi53
Lotus@ .destroy() is now the official way to close a readable stream as per the documentation - nodejs.org/api/stream.html#stream_readable_destroy_error
大约 2 年之前 回复
csdnceshi76
斗士狗 I understood why. Yes it can’t be twice at one. So basically what I did: I created a recursive function that executes the calls one after each other. There goes your performance :D
大约 3 年之前 回复
csdnceshi76
斗士狗 Schilling, yes, that is fast. But if I try to copy lots of files one after each other within a short time period I get ERROR: There are some read requests waiting on finished stream. I guess a stream is unique and instead of waiting for one to finish before starting it just breaks. Any ideas on how to handle this? Thanks
大约 3 年之前 回复
weixin_41568183
零零乙 that's a very good idea, regarding John Poe's question. I can't really remember well anymore, but I think I was talking about something different. Any error while or after opening the source or target file doesn't close the other stream. You can't really only rely on the open event. I think in my case I actually wanted to move the file and there was an accidental write error at some point, so copyFile dutifully returned the error. My app then tried it again successfully and unlinked the source file. But it still showed up in readdir because of the open handle.
3 年多之前 回复
csdnceshi78
程序go you could always listen to the open event rd.on('open', function() {}), and create the write stream there.
3 年多之前 回复
csdnceshi75
衫裤跑路 'cb' stands for "callback". You should pass in a function.
5 年多之前 回复
csdnceshi52
妄徒之命 what does the cb stand for? what should we pass in as the third argument?
5 年多之前 回复
csdnceshi62
csdnceshi62 Wish someone could clarify @Robert point.
5 年多之前 回复
weixin_41568183
零零乙 I think an error in the WriteStream will only unpipe it. You would have to call rd.destroy() yourself. At least that's what happened to me. Sadly there's not much documentation except from the source code.
大约 6 年之前 回复
weixin_41568196
撒拉嘿哟木头 How do you handle the error if the source file doesn't exist? Destination file still gets created in that case.
6 年多之前 回复
csdnceshi59
ℙℕℤℝ It is worth noting that cbCalled flag is needed because pipe errors trigger an error on both streams. Source and destination streams.
6 年多之前 回复

I was not able to get the createReadStream/createWriteStream method working for some reason, but using fs-extra npm module it worked right away. I am not sure of the performance difference though.

fs-extra

npm install --save fs-extra

var fs = require('fs-extra');

fs.copySync(path.resolve(__dirname,'./init/xxx.json'), 'xxx.json');
weixin_41568126
乱世@小熊 I tried using fs.copySync(path.resolve(__dirname,'./init/xxx.json'), 'xxx.json'); also I tried using copy method , also createReadStream solution discussed earlier in this thread . Still getting the blank file . File does get copied to the desired folder . If i rename it , it will delete the original file but I need to make multiple copies of the same file . Any ideas how can this be achieved ?
大约 4 年之前 回复
csdnceshi56
lrony* Thank you. Best solution.
大约 4 年之前 回复
csdnceshi67
bug^君 yes, I think if you need performance or non blocking for larger files for example you would definitely want to look into the async methods.
4 年多之前 回复
csdnceshi51
旧行李 reservation. The one place to absolutely avoid synchronous functions is when responsiveness is paramount (such as when writing a server). Even then, you could utilize synchronous code by forking a new VM with require('child').fork(...) since it wouldn't block the main event loop. It's all about context and what you're trying to achieve.
接近 5 年之前 回复
csdnceshi60
℡Wang Yan I was - am - under the impression that if there is only 1 task at hand it does not make any difference if I use sync or async code ... just checking my knowledge nothing more, nothing less...
接近 5 年之前 回复
csdnceshi69
YaoRaoLov fs-extra also has asynchronous methods, i.e. fs.copy(src, dst, callback);, and these should resolve @mvillar's concern.
接近 5 年之前 回复
csdnceshi57
perhaps? Fastest to implement or fastest to execute? Differing priorities mean this is a valid answer.
接近 5 年之前 回复
csdnceshi66
必承其重 | 欲带皇冠 Oh please... The question is about fastest method to copy a file. While fastest is always subjective, I don't think a synchronous piece of code has any business here.
大约 5 年之前 回复
csdnceshi76
斗士狗 Fortunately performance is not always an issue.
5 年多之前 回复
csdnceshi65
larry*wei Using syncronous code in node kills your application performance.
5 年多之前 回复
csdnceshi62
csdnceshi62 This is the best option now
5 年多之前 回复

Fast to write and convenient to use, with promise and error management.

function copyFile(source, target) {
  var rd = fs.createReadStream(source);
  var wr = fs.createWriteStream(target);
  return new Promise(function(resolve, reject) {
    rd.on('error', reject);
    wr.on('error', reject);
    wr.on('finish', resolve);
    rd.pipe(wr);
  }).catch(function(error) {
    rd.destroy();
    wr.end();
    throw error;
  });
}

Same with async/await syntax:

async function copyFile(source, target) {
  var rd = fs.createReadStream(source);
  var wr = fs.createWriteStream(target);
  try {
    return await new Promise(function(resolve, reject) {
      rd.on('error', reject);
      wr.on('error', reject);
      wr.on('finish', resolve);
      rd.pipe(wr);
    });
  } catch (error) {
    rd.destroy();
    wr.end();
    throw error;
  }
}
csdnceshi62
csdnceshi62 wha happens in that case? I mean the file is not written and a "finish" event will be fired?
3 年多之前 回复
csdnceshi76
斗士狗 It seems that the "finish" event is unfortunately also send if the stream could not be written because of access restrictions. "finish" doesn't mean "success"
接近 4 年之前 回复
weixin_41568196
撒拉嘿哟木头 According to API: "One important caveat is that if the Readable stream emits an error during processing, the Writable destination is not closed automatically. If an error occurs, it will be necessary to manually close each stream in order to prevent memory leaks."
接近 4 年之前 回复
csdnceshi70
笑故挽风 you are right. 'close' should be 'finish' according to official API.
5 年多之前 回复
csdnceshi56
lrony* And if you wonder why your application never closes after pipe errors on /dev/stdin, that is a bug github.com/joyent/node/issues/25375
5 年多之前 回复
csdnceshi56
lrony* By the way, close should be finish for Writable streams.
5 年多之前 回复
csdnceshi56
lrony* I just tested new Promise(function(resolve, reject) { resolve(1); resolve(2); reject(3); reject(4); console.log("DONE"); }).then(console.log.bind(console), function(e){console.log("E", e);}); and looked up the spec on this and you are right: Attempting to resolve or reject a resolved promise has no effect. Perhaps you could extend your answer and explain why you have written the function in this way? Thanks :-)
5 年多之前 回复
weixin_41568127
?yb? The promise is resolved once the copy succeed. If it's rejected, its state is settled and calling reject multiple times won't make no difference.
5 年多之前 回复
csdnceshi56
lrony* What happens when no more input exists (broken network share), but the write still succeeds? Will both reject (from read) and resolve (from write) be called? What if both read/write fails (bad disk sectors during read, full disk during write)? Then reject will be called twice. A Promise solution based on Mike's answer with a flag (unfortunately) seems to be the only viable solution that properly considers error handling.
5 年多之前 回复

Mike Schilling's solution with error handling with a short-cut for the error event handler.

function copyFile(source, target, cb) {
  var cbCalled = false;

  var rd = fs.createReadStream(source);
  rd.on("error", done);

  var wr = fs.createWriteStream(target);
  wr.on("error", done);
  wr.on("close", function(ex) {
    done();
  });
  rd.pipe(wr);

  function done(err) {
    if (!cbCalled) {
      cb(err);
      cbCalled = true;
    }
  }
}

Since Node.js 8.5.0 we have new fs.copyFile and fs.copyFileSync methods.

Usage Example:

var fs = require('fs');

// destination.txt will be created or overwritten by default.
fs.copyFile('source.txt', 'destination.txt', (err) => {
    if (err) throw err;
    console.log('source.txt was copied to destination.txt');
});
weixin_41568127
?yb? This is the only correct answer on the page. None of the other answers actually copy files. Files on MacOS and Windows have other metadata that is lost by just copying bytes. Examples of data not copied by any other answer on this page, windows and macos. Even on Unix the other answer don't copy the creation date, something that's often important when copying a file.
接近 2 年之前 回复
csdnceshi59
ℙℕℤℝ This should be the correct answer
大约 2 年之前 回复

If you don't care about it being async, and aren't copying gigabyte-sized files, and would rather not add another dependency just for a single function:

function copySync(src, dest) {
  if (!fs.existsSync(src)) {
    return false;
  }

  var data = fs.readFileSync(src, 'utf-8');
  fs.writeFileSync(dest, data);
}
weixin_41568110
七度&光 I've added a "and aren't copying gigabyte-sized files" caveat.
接近 3 年之前 回复
weixin_41568131
10.24 and requires as much memory as the file content... I am amazed by the count of upvotes there.
3 年多之前 回复
weixin_41568127
?yb? I like this answer. Clear and simple.
3 年多之前 回复

Well, usually it is good to avoid asynchronous file operations. Here is the short (i.e. no error handling) sync example:

var fs = require('fs');
fs.writeFileSync(targetFile, fs.readFileSync(sourceFile));
csdnceshi58
Didn"t forge Asynchronous operations are appropriate on web servers, where node.js can handle other http requests while the file is being copied. Using synchronous operations is easier, though, and they are appropriate in command-line scripts in which your script is the only one running in the process.
大约 2 年之前 回复
csdnceshi77
狐狸.fox the *Sync being deprecated ?... hum... that's why fs.exists() is deprecated and fs.existsSync() is not. lol.
2 年多之前 回复
csdnceshi77
狐狸.fox OFC it won't see the config file if you write the require just after the asynchronous method, that is how wrong you may have design your process in the first place, the whole asynchronous philosophy is "do something when I finish", that means you have to call your require when the Promise resolves...
2 年多之前 回复
csdnceshi56
lrony* Yeah I don't think it's helpful to EVER say something like 'a synchronous op is preferable in nodejs'. That's contrary to what is pretty much rule #1 in node: make everything async where possible. Emphasis is on "where possible". If your file copy is required for subsequent logic then yes, it should be sync (and even then, maybe not). But if you're spitting out a log that will be used later or something unrelated to the current application then it's 100% preferable. For those beginners out there, just understand, contrary to this answer, IT IS USUALLY BAD TO AVOID ASYNC OPS IN NODE.
接近 3 年之前 回复
csdnceshi80
胖鸭 i downvoted this but later ended up using it as best answer - apologies - it wont let me upvote - if u edit it it will let me!
3 年多之前 回复
csdnceshi76
斗士狗 for tools and build operations it just bloats your code to use syncronous stuff. good solution.
3 年多之前 回复
csdnceshi62
csdnceshi62 Sync methods are fine when you are interacting with another sync operation or what you want is to perform sequential operation (ie. you would be emulating sync anyway). If the operations are sequential just avoid the callback hell (and/or promise soup) and use the sync method. In general they should be used with caution on servers but are fine for most cases that involve CLI scripts.
大约 5 年之前 回复
weixin_41568183
零零乙 To give an example for using synchronous copying: I have a npm script that needs a config file. If none exists yet, I copy a default one to the config file location before require'ing the config file. When using the asynchronous copy, the require doesn't see the new config file..
大约 5 年之前 回复
csdnceshi70
笑故挽风 Keep in mind that this works ONLY if sourceFile fits into memory (i.e.: don't do this with huge files).
5 年多之前 回复
csdnceshi78
程序go I'm not aware of them being deprecated. Sync methods are almost always a terrible idea on a web server but sometimes ideal in something like node-webkit where it only locks up action in the window while files are copying. Throw up a loading gif and maybe a load bar that updates at certain points and let sync methods block all action until the copying is done. It's not really a best practice thing so much as a when and where they have their place thing.
接近 6 年之前 回复
csdnceshi59
ℙℕℤℝ The only reason I can think of for using them is for simplicity - if you are writing a quick script that you will only use once, you probably aren't going to be all that bothered about blocking the process.
接近 6 年之前 回复
csdnceshi75
衫裤跑路 using the *Sync methods are totally against nodejs' philosphy! I also think they are slowly being deprecated. The whole idea of nodejs is that it's single threaded and event-driven.
接近 6 年之前 回复
csdnceshi57
perhaps? To say that in general is extremely false, particularly since it leads to people re-slurping files for every request made to their server. This can get expensive.
6 年多之前 回复
weixin_41568110
七度&光 Why is it good to avoid asynchronous file operations?
6 年多之前 回复

benweet's solution checking visibility of the file before copy:

function copy(from, to) {
    return new Promise(function (resolve, reject) {
        fs.access(from, fs.F_OK, function (error) {
            if (error) {
                reject(error);
            } else {
                var inputStream = fs.createReadStream(from);
                var outputStream = fs.createWriteStream(to);

                function rejectCleanup(error) {
                    inputStream.destroy();
                    outputStream.end();
                    reject(error);
                }

                inputStream.on('error', rejectCleanup);
                outputStream.on('error', rejectCleanup);

                outputStream.on('finish', resolve);

                inputStream.pipe(outputStream);
            }
        });
    });
}

Mike's solution, but with promises:

const FileSystem = require('fs');

exports.copyFile = function copyFile(source, target) {
    return new Promise((resolve,reject) => {
        const rd = FileSystem.createReadStream(source);
        rd.on('error', err => reject(err));
        const wr = FileSystem.createWriteStream(target);
        wr.on('error', err => reject(err));
        wr.on('close', () => resolve());
        rd.pipe(wr);
    });
};
共13条数据 1 尾页
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问