bash: no job control in this shell

Communities CommandBox CLI
1

Hi Commandbox Gurus,
We use the ‘run’ system command (actually, the shortcut ‘!’ bang followed by the shell command) to run various commands that operate in the native shell. (mainly, stuff like !git status, !git commit, !svn info etc.).

With commandBox 4.0 in bash shells, the following line is always printed as the first line of the output:

bash: no job control in this shell

Following that line, the output of the command does display as expected.

This behavior is only observed on Linux / Mac OS machines running Bash.

I’m sure it has something to do with the JLine3 libraries. My only question is if it’s possible to suppress this warning?

We have some commandbox tasks that have several external commands that execute and this additional line is causing minor issues for us. If not, I’m sure we can work around it but it would be nice to have the same output from windows and *nix based systems.

Thanks in advance!
–Joel

2

I should mention that the same line is printed when using the command DSL to execute an external, native shell command.

3

This is probably unrelated to JLine. I made some changes to how the run commands work. I’ve dealt with this silly message for years and for the life of me I’ve never found out exactly why it happens or how to stop it from happening. It’s one of those things that doesn’t seem to happen on every OS, just some of them. YOu mentioned, Linux and Mac OS. Did you specifically test on both? What distros versions of each?

Feel free to try and figure it out and submit a pull. I’ve spent hours upon hours messing with the run command on *nix in the past. Of course, I use Windows and I don’t have those silly errors there :slight_smile: You might get some mileage out of piping the output through sed to try and remove it. Just keep in mind, that won’t allow interactive binaries to run.

the same line is printed when using the command DSL

Well yes, that’s because it’s the same command that runs in the background.

4

Hi Brad,

I think I found the issue. It’s regarding java.lang.ProcessBuilder’s redirectErrorStream() function. When it’s set to true, the stderr stream is null (as it is redirected to stdout).
Bash job.c has a check where if stderr is not attached to a tty, it disables job control.

I replaced that line in run.cfc with

.inheritIO()

which is equivalent to:

`

.redirectInput(Redirect.INHERIT)
.redirectOutput(Redirect.INHERIT)
.redirectError(Redirect.INHERIT)

`

I’m still testing this and will submit a pull request in the AM if it seems to work and still give us stderr.

Thanks,
Joel

5

No, I don’t want to use inheritIO(). I used to use that but purposefully changed it back in the 4.0 release. The reason is that there is no way to capture the output of a command when you inherit the IO, so instead I manually handle that so I can capture the output if necessary. Check out the logic that uses the interactive flag. This allows you to do this:

!java -version > myFile.txt

Thanks for checking into the code in Job.c, at least that explains why that message came back. Is there any other way to get around that?

6

Brad,
Ok, scratch inheritIO(). How about:

.redirectError(Redirect.INHERIT)

instead of

.redirectErrorStream( true )

.

This enables job control, the shell is interactive, aliases are expanded and can be used in commandBox (aka ‘!ll’ works), redirection works (to a file or otherwise) and from what I see, stdErr is printed to the shell when an error occurs.

And, of course, the “bash: no job control in this shell” message is NOT printed before the command output.

Is there any functionality that I’m missing that .redirectErrorStream( true ) provides?

Thanks again!
Joel

7

I assume that means you won’t be able to capture error output though, right?

8

I assume that means you won’t be able to capture error output though, right?

doh. yep. you’re right.

I think I’m starting to understand your frustrations with this “Bash: No job control…” message :slight_smile:

9

Yes, I’ve fought back and forth with that message for a long time now. At one point I had a check in that would just remove that text, but that’s a lot harder now that I’m streaming output through a buffer one character at a time.

10

Hi Brad,

I just realized something. Even with
.redirectErrorStream( true )

I can not redirect stderr to a file. It always prints errors in the console. I tried this on my mac as well as in Windows 7 command prompt. In both OSes, if the run command output is redirected to a file (using ‘> filename.txt’), it simply ignores it. Even if I echo the command and pipe it through run, such as “echo ‘binary_that_errors > test.txt’ | run”, it prints the error in my shell. Here’s some actual output, as seen in cmd.exe. I’m using svn info here from a directory that is not an svn repo. (but any external binary seems to behave this way) :

`
CommandBox:svn> !svn info > test.txt
svn: E155007: ‘C:\Users\joel\webroots\svn’ is not a working copy

ERROR (4.0.0+01015)

Command returned failing exit code [1]

CommandBox:svn> echo ‘svn info > test.txt’ | run
svn: E155007: ‘C:\Users\joel\webroots\svn’ is not a working copy

ERROR (4.0.0+01015)

Command returned failing exit code [1]

CommandBox:svn>

`

In both cases, test.txt file does not contain the error. Does this work in your shell?

–Joel

11

I’m not sure if I followed what you were doing there, but for what it’s worth, that will work if you combine the error and starndard output streams together like this:

echo ‘git status > status.txt 2>&1’ | run

That works on Windows, I didn’t test it on *nix right off.

12

I’m sorry for the confusion, Brad. I know that ‘2>&1’ will redirect stderr to stdout. But I thought that was already happening in the java.lang.ProcessBuilder class, due to the “.redirectErrorStream( true)”. Based on that assumption, I figured that any errors would show up when I simply redirect stdout to a file.

I assume that means you won’t be able to capture error output though, right?

!binary that errors > jobResults.txt

If that isn’t the case and one still needs to redirect stderr to stdout using “2>&1”, then my previous suggestion to fix the “bash: no job control in this shell” works beautifully.

That is, replacing “.redirectErrorStream()” with “.redirectError(Redirect.INHERIT)”. That removes the annoying message, enables job control, expands aliases, and all functionality regarding redirection of output and errors works as it does today in 4.0.0.

Unless I’m missing something else, which could be the case.

Thanks,

–Joel

13

Brad, I submitted a pull request and COMMANDBOX-808 https://ortussolutions.atlassian.net/browse/COMMANDBOX-808

14

But I thought that was already happening

Well, yes and no. It was already happening, but at a different place. Let’s assume you run a binary that returns output to both the standard output stream AND the error stream. If you ask the native shell to redirect the error stream, the from Java’s perspective, it never gets anything on the error stream. It only sees the input on the standard input. But if you ask process builder to do the redirect, then that means the operating system sends output on both streams back to Java and the process builder class combines them for you.

So to think through this out loud, I think we’re saying that if we redirect the error stream to the standard output on the OS side and set the error stream to inherit in process builder then we’ll never actually GET anything on the error stream, but for the sake of job control, bash will THINK that it’s error stream is connected to a TTY so it won’t complain. Does that sound correct?

15

I see your pull, but won’t that prevent us from being able to capture the error output of a command that fails?

!git command that fails > myOutput.txt

I would think we’d want to also combine this with a change to redirect the error stream on the OS side as well. Also, I think we’ll only want to make this change for *nix. It might work the same on Windows, but I’d like to leave Windows alone if we can.

16

I see your pull, but won’t that prevent us from being able to capture the error output of a command that fails?

!git command that fails > myOutput.txt

I mentioned this in a previous reply but I think my example was confusing. The command above only prints the error to the screen. The error never makes it into the file. This is with or without my changes, in bash or in windows. (running Commandbox 4.0) When you redirect the output of a command or binary to a file, and that command fails, the error doesn’t get redirected to the file so you don’t end up capturing anything. But, you do see the error printed in the terminal.

Also, I think we’ll only want to make this change for *nix.

Ok. I can fix that.

–Joel

17

The command above only prints the error to the screen. The error never makes it into the file. This is with or without my changes, in bash or in windows.

Ahh yes, this caught me off guard just now but when I looked at the code I realized it’s “working as designed”. Basically when a native command returns a failing exit code, the “run” command itself returns an error and inherits the same failing exit code. As such, the entire command chain (meaning piped commands or redirected input) all get aborted and execution stops right there. And, as is customary when any command fails, anything in the print buffer for that command gets dumped out to the screen after it errors (which is why you see the error text on the screen).

So what this means is that you basically can’t pipe or redirect the output of a failing command anywhere as CommandBox has been designed to just abort right there, print the error message and return to the shell. Makes sense now that I think about it.

So I guess that brings up a good question which is whether that’s how we want to treat failing commands. I guess it makes sense in one regard (and I did write it that way on purpose) though I suppose I could imagine a scenario where I want to run something that I know may error but I want to capture whatever happens. Ignoring that for now, this means my test of seeing if we could capture the error stream is rendered invalid if my binary ALSO returns a failing exit code. I need to test a binary that returns text to the error stream but doesn’t fail. I concocted a quick Java class that just does this:

System.out.println( “This is the out stream” );
System.err.println( “This is the error stream” );

And compiled it into a jar and ran it like so:

!java -jar testapp.jar

echo “java -jar testapp.jar” | run > brad.txt

In the first case, both lines were output (straight passthrough to the shell) and in the second example both lines were written to the text file (both streams captured by the “run” command). I’m attaching a zip of the jar in case you want to try it (rename it to remove the .txt-- stupid Gmail!!). I think we want to make sure we preserve the behavior of capturing both streams in all cases. I’ll also add a note here that I’ve seen a lot of “naughty” apps that return text on the error stream even though they are totally not failing. Embedded Jetty comes to mind…

In the case that there is output on one or both streams but the binary fails, I think I’m tempted to say that that’s just working as designed that it basically aborts and dumps all output so we don’t need to worry about that case.

testapp.jar.txt (1.05 KB)

18

Brad,
Here are the results of executing your jar file on my Mac. I ran a 3rd test case where I redirect stderr to stdout in the native shell. (echo “java -jar testapp.jar 2>&1” | run > brad.txt). After each test, I cat the file so you can see what was captured and what was printed on the screen.

Here are the results.

`

`

`

/* WITH redirectErrorStream(true) (without the changes in my pull request) */

`

`

`

`

/*here, we see the output of both streams printed to the screen. */

`

`

CommandBox:webroots> !java -jar testapp.jar

`

`

bash: no job control in this shell

`

`

This is the out stream

`

`

This is the error stream

`

`

/I didn’t bother showing the content of the brad.txt file after the above command./

`

`

`

`

/*now we redirect to brad.txt. */

`

`

CommandBox:webroots> echo “java -jar testapp.jar” | run > brad.txt

`

`

/above command didn’t print either stream to the console./

`

`

/*and both streams ended up in the txt file: */

`

`

CommandBox:webroots> cat brad.txt

`

`

bash: no job control in this shell

`

`

This is the out stream

`

`

This is the error stream

`

`

`

`

/*what if we redirect stderr to stdout in our native shell, bash? */

`

`

CommandBox:webroots> echo “java -jar testapp.jar 2>&1” | run > brad.txt

`

/*above command didn't print either stream to the console.*/

/*and both streams ended up in the txt file: */

`

CommandBox:webroots> cat brad.txt

`

`

bash: no job control in this shell

`

`

This is the out stream

`

`

This is the error stream

`

`

`

/*round 1 tests complete! It seems the last two tests in this round did the exact same thing. Nothing printed to console and both streams captured in text file. */

`

`

/* Now we will apply the change in my pull request and run the same tests again */

`

/* WITH redirectError(REDIRECT.inherit): */

`

`

`

`

CommandBox:webroots> r

`

`

`

`

Reloading shell…

`

`

/*here, we see the output of both streams printed to the screen. */

`

`

CommandBox:webroots> !java -jar testapp.jar

`

`

This is the error stream

`

`

This is the out stream

`

`

`

`

/the next command prints the stderr stream to the console:/

`

`

CommandBox:webroots> echo “java -jar testapp.jar” | run > brad.txt

`

`

This is the error stream

`

`

`

`

/and the stdout stream was redirected to our txt file:/

`

`

CommandBox:webroots> cat brad.txt

`

`

This is the out stream

`

`

`

`

/and the 3rd test case, redirect stderr from bash and stdout in commandBox prints nothing in the console:/

`

`

CommandBox:webroots> echo “java -jar testapp.jar 2>&1” | run > brad.txt

`

`

`

`

/and now both streams are redirected to our txt file:/

`

`

CommandBox:webroots> cat brad.txt

`

`

This is the out stream

`

`

This is the error stream

`

`

`

Besides making the bash shell interactive (which removes the “no job control” nag and includes some additional “benefits” that may or may not apply when running native shell commands from commandBox), the change in my pull request seems to provide a bit more flexibility when redirecting stderr. It also more closely matches the way the bash shell natively handles the stderr stream when you redirect to a file. That is, stderr is printed on the screen while stdout is redirected to the file. In bash, if you want both streams captured in a file, you would do something like, ‘some command 1>some_file 2>1’. Then, you will get nothing printed on the screen and both streams in some_file.

Without my pull request, stderr will always get redirected when you redirect stdout, which may be exactly what you are going for. But, this behavior may surprise some people because when you execute ‘some_command > some_file’ , what you are really doing is ‘some_command 1>some_file’. (redirecting stdout, or stream “1”, to a file)

So the question is, do you want redirection to a file to behave more like the native shell’s redirection, or do you intentionally want to redirect both streams (stderr and stdout) at the same time?

–Joel

19

Thanks for your tests, I think it shows I was correct that inheriting the error stream would make it not possible to capture the error stream in CommandBox. As far as how I want that to work, CommandBox has never had a notion of a separate out and error stream so I’ve always preferred the native processes to be combined and captured together. Going back to where I was a few E-mails ago, I think this may work like we both want if CommandBox combines the streams on the bash side and then inherits the error stream in process builder. The error stream will always be empty coming from bash but that will at least trick the job control message from not being there.

20

Brad,
Ok. I’ve issued a 3rd commit which does what you’ve asked. The change is to line 62 of run.cfc which adds the stderr redirection to stdout in the commandArray for bash.

`
commandArray = [ nativeShell, ‘-i’, ‘-c’, arguments.command & ’ 2>&1 && ( exit $? > /dev/null )’ ];

`

This seems to hit the middle ground we are looking for. Executing your jar and redirecting the output to a file in commandBox puts both streams in the file and nothing on the screen. With the changes from the other commits, nothing should change for Windows users. They still get “.redirectErrorStream(true)”.

Let me know if you see any issues with that.

Thanks again for the awesome support and expertise, Brad! I appreciate you taking the time to explain all of this to me and providing the handy jar file which was monumental in troubleshooting and understanding the behavior.

–Joel