Exec failing with internal error when piping large output

[ealmansi]

Node version (or tell us if you're using electron or some other framework):

8

ShellJS version (the most recent version/Github branch you see the bug on):

0.8.0

Operating system:

Ubuntu 16.04.3

Description of the bug:

Piping medium to large amounts of output between two exec's causes the second one to fail with 'Error: exec: internal error'.

Example ShellJS command to reproduce the error:

var shell = require('shelljs')
shell.config.fatal = true
shell.exec('cat small.txt', { silent: true }).exec('wc -l', { silent: true }).to('few-lines.txt')
shell.exec('cat big.txt', { silent: true }).exec('wc -l', { silent: true }).to('many-lines.txt')

Here, small.txt is a small file (under 1kb), while big.txt is a 2MB file.
The first command executes perfectly well, while the second one fails with exec: internall error.
Without fatal = true, the second exec on the second command simply returns null and the command fails due to to not being a property of null. (Likely related: #724)

Note

I did some digging and it seems the problem is here:

https://github.com/shelljs/shelljs/blob/master/src/exec.js#L68

That call to execFileSync receives execArgs, which contains the serialized output from the first exec. So it's essentially calling a process with an argument which is over 2MB, which is breaking a limit on my OS. That call to execFileSync ends with an Error: spawnSync /usr/bin/node E2BIG being thrown by node.

[nfischer]

Thanks for digging into this! Yeah, it makes sense that there's a limit on argument size. Did this reproduce on 0.7.x?

It seems like #807 was wrong, and we may be able to fix this by reverting.

[nfischer]

I'll release this in 0.8.1 soon

[FatihEbibelot]

I'm getting the same error with node v6.11.1 and shelljs v0.8.1,

Code is;

var shell = require('shelljs')
var executionCommand = "/home/ubuntu/Audiveris/bin/Audiveris -batch"
shell.exec(executionCommand, function(error, stdout, stderr) {
})