Mike Slinn
ScalaCourses.com

Process Control

— Draft —

Published 2014-06-03. Last modified 2019-07-17.
Time to read: 7 minutes.

This lecture introduces processes, standard input and output and some predefined implicit helper classes.

This lecture discusses Scala’s built-in runtime capability. Ammonite-Ops is an alternative (here is an operator reference).

The code examples for this lecture are provided in ProcessIO.scala.

Let’s start by having some fun with the REPL. :sh is a REPL command that runs a shell command and returns a value to an automatically assigned variable. The -u option for the ps command lists the current user’s foreground processes that have a tty.

Scala REPL
scala> :sh ps -u
res0: scala.tools.nsc.interpreter.ProcessResult = `ps u` (7 lines, exit 0)

:sh returned a ProcessResult called res0. ProcessResult has a property called lines of type Array[String] which contains the output returned by the process. Foreach item in the collection called res0 we can do something, like invoke println, which prints the item on its own line.

Scala REPL (continued)
scala> res0.lines foreach println
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mslinn     460  0.0  0.0  12568  3228 pts/6    S+   14:10   0:00 bash /usr/bin/scala -Dscala.color=true
mslinn     532  0.0  0.0  18484  2684 pts/6    R+   14:10   0:00 ps -u
mslinn    9279  0.0  0.0  25072  8040 pts/4    Ss+  Jun18   0:00 -bash
mslinn   23872  0.0  0.0  24184  7080 pts/5    Ss   09:26   0:00 -bash
mslinn   32322  0.0  0.0  12568  3232 pts/5    S+   13:58   0:00 bash /usr/bin/scala -Dscala.color=true
mslinn   32416  0.0  0.0  24104  6932 pts/6    Ss   13:59   0:00 -bash

BTW, Scala’s Process-related classes were ported from SBT. The implementation in SBT differs slightly from the Scala implementation. If you use sbt console to run the REPL instead of the scala compiler-driver you do not need to specify the lines property of ProcessResult. The code example will work fine using sbt console whether or not you specify the lines property.

Scala REPL (continued)
scala> res0 foreach println
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
mslinn     460  0.0  0.0  12568  3228 pts/6    S+   14:10   0:00 bash /usr/bin/scala -Dscala.color=true
mslinn     532  0.0  0.0  18484  2684 pts/6    R+   14:10   0:00 ps -u
mslinn    9279  0.0  0.0  25072  8040 pts/4    Ss+  Jun18   0:00 -bash
mslinn   23872  0.0  0.0  24184  7080 pts/5    Ss   09:26   0:00 -bash
mslinn   32322  0.0  0.0  12568  3232 pts/5    S+   13:58   0:00 bash /usr/bin/scala -Dscala.color=true
mslinn   32416  0.0  0.0  24104  6932 pts/6    Ss   13:59   0:00 -bash

You can run these examples by typing:

Shell
$ sbt "runMain ProcessIO"

ProcessBuilder

Scala programs also have ways of running shell processes. The method I am about to show you does exactly the same thing as the REPL example above, but it does not rely on the REPL’s :sh command and so can be incorporated into your program. Instead, it uses the Scala ProcessBuilder DSL.

Scala REPL
scala> import sys.process._
import sys.process._

I imported sys.process._, which is the same as writing scala.sys.process._; these imports provide implicits that provide some Scala magic. We learned about implicits in the Implicit Values and Implicit Conversions lectures earlier in this course. We can now run an operating system command or another program and obtain standard output (stdout). Let’s look at /etc/passwd. BTW, if you typing along and are running Microsoft Windows and use Git for Windows you will not have this file. In that case, pick another file to display, such as ~/.profile, ~/.bashrc .

Scala REPL
scala> val passwds = "cat /etc/passwd" !!
passwds: String =
"root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
..."

!! (often pronounced "bang! bang!") is an implicit method defined in scala.sys.process.Process that executes the preceding String as a system command, and returns the output string to the variable passwds. You can see that the standard output from the cat process was truncated by the Scala REPL.

!! was used with a postfix method invocation, and the Scala compiler sometimes has difficulty in figuring out if a line ending with a postfix method invocation should continue across newlines. This means it is often necessary to terminate lines ending with postfix method invocations with a semicolon. The ProcessIO.scala version of this code example shows this clearly.

We can capture file names from ProcessBuilder like this. Notice that the REPL also echoes the value stored into fileNames to the screen.

Scala REPL
scala> val fileNames = "ls" !!
fileNames: String =
"build.sbt
project
src
target
test
"

scala> {% noselect scala> fileNames
res0: String =
"build.sbt
project
src
target
test
" %}

ProcessBuilder provides several ways to pipe data into processes. The following code executes the bash printf command, which creates a three-line string literal which looks like this:

Scala REPL
scala> printf "xray\\nyankee\\nzulu"
xray
yankee
zulu

Those three lines are then piped into grep, which displays any lines containing x.

Scala REPL
scala> printf "xray\\nyankee\\nzulu" | grep x

The backslashes are doubled inside the String literal because escape characters are processed before invoking printf.

Scala REPL
scala> printf "xray\\nyankee\\nzulu" | grep x
xray

Here is similar code, written in Scala.

Scala REPL
scala> "printf xray\\nyankee\\nzulu" #> "grep x" !!
res1: String
"xray
"

Notice that there is an extra linefeed in the output. Here is how you can remove it.

Scala REPL
scala> ("printf xray\\nyankee\\nzulu" #> "grep x" !!) trim
res2: String = xray

Process.apply

You can invoke Process.apply instead of using ProcessBuilder for special situations. Multiple overloaded versions of Process.apply are available, the simplest of which merely accepts a list of tokens. Here is an example that uses the bash find command to list all Scala source files in the current directory tree:

Shell
$ cd courseNotes/

$ scala
Welcome to Scala 2.12.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation.
Or try :help.

scala> import sys.process._
import sys.process._

scala> Process(List("find", ".", "-name", "*.scala")).!!.trim
res1: String =
./src/main/scala/Structural.scala
./src/main/scala/ParametricBounds.scala
./src/main/scala/ForFun.scala
./src/main/scala/ConfigFun.scala
./src/main/scala/cache/Memoize.scala
./src/main/scala/ProcessIO.scala
./src/main/scala/multi/futures/FuturesUtil.scala
./src/main/scala/multi/futures/TinyFuture.scala
./src/main/scala/multi/futures/FutureComb.scala
./src/main/scala/multi/futures/package.scala
./src/main/scala/multi/futures/FutureAwaitSignal.scala
./src/main/scala/multi/futures/FutureRace.scala
./src/main/scala/multi/futures/FutureWord.scala
./src/main/scala/multi/futures/FutureFor.scala
./src/main/scala/multi/futures/FutureCallback.scala
./src/main/scala/multi/MonkeyActors.scala
./src/main/scala/multi/ActorFun.scala
./src/main/scala/multi/Parall...

Here is a handy method that makes invoking this flavor of Process.apply more convenient. It simplifies the syntax by using varargs syntax, discussed in the Classes Part 2 lecture of the Introduction to Scala course.

Scala code
def run(cmd: String*): String = Process(cmd).!!.trim

Let’s use the run method to invoke find as before.

Scala REPL
scala> def run(cmd: String*): String = Process(cmd).!!.trim
run: (cmd: String*)String
scala>  run("find", ".", "-name", "*.scala")
res2: String =
./src/main/scala/Structural.scala
./src/main/scala/ParametricBounds.scala
./src/main/scala/ForFun.scala
./src/main/scala/ConfigFun.scala
./src/main/scala/cache/Memoize.scala
./src/main/scala/ProcessIO.scala
./src/main/scala/multi/futures/FuturesUtil.scala
./src/main/scala/multi/futures/TinyFuture.scala
./src/main/scala/multi/futures/FutureComb.scala
./src/main/scala/multi/futures/package.scala
./src/main/scala/multi/futures/FutureAwaitSignal.scala
./src/main/scala/multi/futures/FutureRace.scala
./src/main/scala/multi/futures/FutureWord.scala
./src/main/scala/multi/futures/FutureFor.scala
./src/main/scala/multi/futures/FutureCallback.scala
./src/main/scala/multi/MonkeyActors.scala
./src/main/scala/multi/ActorFun.scala
./src/main/scala/multi/Parall...

There is another version of Process.apply that allows you to specify a list of tokens for the command to execute along with its arguments, the current directory to execute from, and an arbitrary number of tuples to specify environment variables as name/value pairs.

This example shows how this version of Process.apply can be used. The following bash code example will be replicated in Scala. This example calls psql with a query. psql has an awkward security feature: the password cannot be supplied to the command, so in order to invoke psql via a bash script, you can set the PGPASSWORD environment variable to the password, and then invoke psql. The following is a one-line bash command that does this.

Shell
PGPASSWORD=mypass psql -h localhost -U postgres -p 5432 -c "select * from pg_tables where schemaname=’public’"

Let’s recreate this command using Scala. In the following code, notice that each token is specified as part of a List so psql receives its arguments properly.

Scala REPL
scala> import sys.process._
import sys.process._

scala> val cmd = List("psql", "-h", "localhost", "-U", "postgres", "-p", "5432", "-c", "select * from pg_tables where schemaname=’public’")
cmd: List[String] = List(/usr/bin/psql, -h, localhost, -U, postgres, -p, 5432, -c, select * from pg_tables where schemaname=’public’)

scala> %}val cwd = None // current working directory is not required
cwd: None.type = None

scala> %}val extraEnv = ("PGPASSWORD", "mypass")
extraEnv: (String, String) = (PGPASSWD,mypass)

scala> %}Process(cmd, cwd, extraEnv).!!.trim
res3: String =
schemaname | tablename | tableowner | tablespace | hasindexes | hasrules | hastriggers
------------+-----------+------------+------------+------------+----------+-------------
(0 rows) %}

If a problem occurs while attempting to run psql, an Exception will be thrown. It would be better to call psql this way.

Scala code
try {
  val result = Process(cmd, cwd, extraEnv).!!.trim
  println(s"psql tables are:\n$result")
} catch {
  case e: RuntimeException =>

  case e: java.io.IOException =>
    println(s"${e.getMessage}\nPerhaps PostgreSQL is not installed on your system, or psql is not on the path?")

  case e: Exception =>
    println(e.getMessage)
}

Helper Traits

ProcessBuilder has some helper traits that can create Processes, such as URLBuilder and FileBuilder. Both of these helper traits have methods with similar signatures. For example, each defines a method called #>, which is overloaded to accept a File instance to write to. Explore the Scaladoc to learn about the other methods available in these helper classes.

The scala.sys.process package object (Scaladoc) extends ProcessImplicits (Scaladoc, defined in Process.scala) thereby bringing all the implicits that it defines into scope.

URLBuilder

Here is a little program that downloads a web page into a file called scalaCourses.html.

Scala code
object URLBuilderDemo extends App {
  import java.io.File
  import java.net.URL
  import sys.process._

  new URL("https://scalacourses.com") #> new File("scalaCourses.html") !;
  println("cat scalaCourses.html" !!)
}

Both the ! and !! methods start a process, block until it exits, and return something; standard output and error are sent to the console. The ! method returns the exit code as an Int, while the !! method returns output as a String, and throws an exception if the return code is not 0.

Output is the HTML source for this web site. It works because ProcessImplicits contains an implicit conversion from a URL instance to a URLBuilder.

Scala code
implicit def urlToProcess(url: URL): URLBuilder

You can run this program by typing:

Shell
$ sbt "runMain URLBuilderDemo"

FileBuilder

This little program uses the #> operator to pipe the output of the bash ls command to a file called dirContents.txt and prints it out. It works because ProcessImplicits contains an implicit conversion from a File instance to a URLBuilder.

Scala code
object FileBuilderDemo extends App {
  import java.io.File
  import sys.process._

  "ls" #> new File("dirContents.txt") !;
  println("cat dirContents.txt" !!)
}

You can run this program by typing:

Shell
$ sbt "runMain FileBuilderDemo"

Composing ProcessBuilders

ProcessBuilders can be composed, which means that you can chain sequences of ProcessBuilders that pipe data between themselves, and you can also specify stderr handling. The following method constructs a ProcessBuilder using the cat method.

Scala code
def readUrlPBuilder(url: String, fileName: String): ProcessBuilder = new URL(url) #> new File(fileName) cat

We can call readUrlPBuilder like in the REPL like this:

Scala REPL
scala> readUrlPBuilder("https://scalacourses.com", "scalaCourses.html") !
res4: Int = 0

scala> :sh cat scalaCourses.html
res5: scala.tools.nsc.interpreter.ProcessResult = `cat scalacourses.html` (457 lines, exit 0)

readUrlPBuilder returns a ProcessBuilder, which is executed by the ! method. We can also invoke readUrlPBuilder such that it encounters an error.

Scala REPL
scala> readUrlPBuilder("https://n_o_s_u_c_h.com", "nosuch.html") !
java.net.UnknownHostException: n_o_s_u_c_h.com
  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
  at java.net.Socket.connect(Socket.java:579)
  at java.net.Socket.connect(Socket.java:528)
  at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
  at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
  at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
  at sun.net.www.http.HttpClient.New(HttpClient.java:308)
  at sun.net.www.http.HttpClient.New(HttpClient.java:326)
  at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
  at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
  at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
  at java.net.URL.openStream(URL.java:1037)
  at scala.sys.process.ProcessBuilderImpl$URLInput$$anonfun$$init$$1.apply(ProcessBuilderImpl.scala:30)
  at scala.sys.process.ProcessBuilderImpl$URLInput$$anonfun$$init$$1.apply(ProcessBuilderImpl.scala:30)
  at scala.sys.process.ProcessBuilderImpl$IStreamBuilder$$anonfun$$init$$5.apply(ProcessBuilderImpl.scala:44)
  at scala.sys.process.ProcessBuilderImpl$IStreamBuilder$$anonfun$$init$$5.apply(ProcessBuilderImpl.scala:44)
  at scala.sys.process.ProcessBuilderImpl$ThreadBuilder$$anonfun$1.apply$mcV$sp(ProcessBuilderImpl.scala:57)
  at scala.sys.process.ProcessImpl$Spawn$$anon$1.run(ProcessImpl.scala:22)

Let’s compose two ProcessBuilders such that the second is only executed if the first succeeds.

Scala REPL
scala> readUrlPBuilder("https://scalacourses.com", "scalacourses.html") #&& "echo yes" !!
yes
res7: String = yes

Now let’s compose two ProcessBuilders such that the second is only executed if the first fails.

Scala REPL
scala> readUrlPBuilder("https://n_o_s_u_c_h.com", "nosuch.html") #|| "echo no" !!
no
res8: String = no

Useful ProcessBuilder Operators

The helper classes also define some of the same operators as ProcessBuilder.

! Execute the command and return the exit value (by convention, 0 means success, anything else means failure). This is a blocking call.

!! Like !, but also captures Scala process standard output.

#| Pipe output to Scala process standard output.

### Sequence two operations.

#&& Do the following operation if the previous operation succeeded.

#|| Do the following operation if the previous operation failed.

#<(b: ProcessBuilder): ProcessBuilder Reads the output of the given ProcessBuilder into this process’s input stream.

#<(b: InputStream): ProcessBuilder Reads the contents of the given InputStream into this process’s input stream.

#<(b: URL): ProcessBuilder Reads the contents of the given URL into this process’s input stream.

#<(b: File): ProcessBuilder Reads the contents of the given File into this process’s input stream.

#>(b: ProcessBuilder): ProcessBuilder Writes the output of this process to the given ProcessBuilder’s input stream..

#>(b: OutputStream): ProcessBuilder Writes the output of this process to the given OutputStream.

#>(b: File): ProcessBuilder Writes the output of this process to the given ProcessBuilder’s input stream..

#>> Create a new ProcessBuilder and connect its input to the output of the previous ProcessBuilder

There are many more operators.

Exercise – Reading Process Output into Memory

Write a short program that capitalizes every word from this text file and prints it out: https://raw.github.com/mslinn/ExecutorBenchmark/master/README.md.

Hints

Solution

Scala code
object MemoryProcess extends App {
  val p = Process(new URL(https://raw.github.com/mslinn/ExecutorBenchmark/master/README.md))
  val os = new java.io.ByteArrayOutputStream
  p #> os !;
  os.toString.split(" ") foreach { w =>
    print(s" ${ w.capitalize }")
  }
}

The variable os contains a string, and that string should be trimmed before use because the ! operator introduces extra whitespace around the returned string from the external process. Regarding the Scala idiom: s" ${ w.capitalize }", the way to read this code is from the inside out.

  1. The text in w is capitalized
  2. ${} invokes toString, which is essentially a no-op when applied to a String
  3. The results of the previous step are inserted into the interpolated string, which is preceded by a space.

The above means that this code fragment is just a way of prepending a space in front of a token. This is a Scala idiom.

The solution is provided in solutions/MemoryProcess.scala. Run it like this:

Shell
$ sbt "runMain solutions.MemoryProcess"

Here is a more functional way to write the same code. The Combinators lecture later on in this course introduces this functional style of writing.

Scala code
println(os.toString
        .trim
        .split(" ")
        .map(_.capitalize)
        .mkString(" ")
)

The disadvantage of the above is that the entire string must be assembled into memory prior to printing it out. For very long strings this might use more memory than is available. Here is another incantation that does not require the entire string to be assembled:

Scala code
os.toString
  .trim
  .split(" ")
  .map(_.capitalize)
  .foreach( word => println(s" $word") )

Exercise – Managing Processes

Use the REPL or a Scala worksheet and ProcessBuilder to discover running Scala programs. Here is how you could write this entirely in bash.

Shell
$ ps ef | grep scala

If you are unsure what it does, try typing it into a bash shell.

Hint

  • You cannot create a Scala Process that contains a pipe, so this will not work:
    Scala code
    $ "ps ef | grep scala" !!
  • Instead, use the Process piping capability to chain two Processes.

Solution

Run the following in the REPL:

Scala code
import scala.sys.process._
"ps ef" #> "grep scala" !!
Previous lecture:

Next lecture:
* indicates a required field.

Please select the following to receive Mike Slinn’s newsletter:

You can unsubscribe at any time by clicking the link in the footer of emails.

Mike Slinn uses Mailchimp as his marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp’s privacy practices.