Unit testing With ScalaTest and Specs2

Published 2014-01-14. Last modified 2017-01-30.
Time to read: 7 minutes.

Scala has two popular unit testing frameworks: ScalaTest and Specs2. We explore the common aspects of both test frameworks in this lecture. The unit tests in this lecture run from the command line, using IntelliJ IDEA versions 13 onwards on Ubuntu, Mac OS/X and Windows using Scala 2.12+. Visual Studio Code is also supported.

ScalaTest and Specs2 are both comprehensive test frameworks. This lecture is merely intended to give you an introduction to common aspects of these test frameworks. Both of these unit testing frameworks are much more powerful than JUnit, however they interoperate with JUnit and can use the JUnit test runner. A nice feature of both frameworks is the ability to write unit tests using a DSL. One way that projects gradually introduce Scala into an all-Java code base is to write unit tests using one of these frameworks.

SBT follows the Maven convention of placing unit test source files in the project’s src/test folder. You can use SBT to run them from the command line using the test, testOnly and testQuick tasks. IntelliJ IDEA and Visual Studio Code both offer integrated unit test support. You cannot use the REPL, Scala scripts or worksheets to write unit tests.

Project Setup

Specs2

Specs2 for Scala 2.11 and 2.12 can be specified for a build.sbt file like this:

libraryDependencies ++= Seq(
  "org.specs2" %% "specs2-core"  % "3.8.6" % Test,
  "org.specs2" %% "specs2-junit" % "3.8.6" % Test
)

Dependencies that have % "test" or % Test following the dependency versions will only be included by SBT when running unit tests.

ScalaTest

ScalaTest can be included into a build.sbt file like this:

build.sbt fragment

libraryDependencies ++= Seq (
  "org.scalatest" %% "scalatest" % "3.0.0" % Test
)

Again, notice the % Test that follows the dependency names.

ScalaTest and Specs2 In the Same Project

You can add dependencies for both Specs2 and ScalaTest in the same build.sbt file. The necessary libraryDependencies are:

build.sbt fragment

libraryDependencies ++= Seq(
  "org.specs2"    %% "specs2-core"  % "3.8.6" % Test,
  "org.specs2"    %% "specs2-junit" % "3.8.6" % Test,
  "org.scalatest" %% "scalatest"    % "3.0.0" % Test,
)

Because these dependencies are only required for tests, they do not add overhead to your deployed project.

Quick and Easy Test Specifications

This lecture demonstrates unit tests written for both frameworks using similar DSLs. With these DSLs, test classes contain should or must statements which themselves contain one or more in statements. If a problem occurs, the error messages are quite helpful. Place your unit tests in the test/scala or test/java directories. Tests written with these DSLs read like:

Sample test

The MumbleFratz should obliviate the FramminJammin in {
  /* unit tests here */
}

Specs2

This file is provided in courseNotes/src/test/scala/Specs2Demo.scala. Two of the imports are specified just so the JUnit test runner can be used. The @RunWith annotation is so Scala-IDE can run the tests.

The test class extends Specification, which defines the DSL for writing unit tests. Three unit tests are contained within this Specification. The Specification DSL groups unit tests by blocks of code that start with a descriptive String followed by the word should or must, followed by zero or more unit tests contained within curly braces.

This Specification instance contains one should container that reads something like:

Specification instance

"The 'Hello world' string" should {
  /* unit tests here */
}

Each unit test starts with a descriptive String followed by the word in, followed by the body of the unit test contained within curly braces. The words should, must and in are actually method calls.

Complete Specs2 test file

import org.specs2.mutable._
import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner

@RunWith(classOf[JUnitRunner])
class Specs2Demo extends Specification {
  "The ’Hello world’ string" should {
    "contain 11 characters" in {
      "Hello world" must have size 11
    }

    "start with ’Hello’" in {
      "Hello world" must startWith("Hello")
    }

    "end with ’world’" in {
      "Hello world" must endWith("world")
    }
  }
}

You read the unit tests starting with the outer group description and appending the unit test description, followed by the body of the unit test which implements the test. For example, these unit tests are read this way:

The "Hello world" string should contain 11 characters –
followed by the DSL which implements the test:
"Hello world" must have size 11
The "Hello world" string should start with "Hello" in –
followed by the DSL which implements the test:
"Hello world" must startWith("Hello")
The "Hello world" string should end with "world" in –
followed by the DSL which implements the test:
"Hello world" must endWith("world")

See the Specs2 documentation for more information.

ScalaTest

ScalaTest is extremely flexible in how unit tests can be structured and offers several DSLs: FeatureSpec, FlatSpec, FreeSpec, FunSpec, PropSpec, and WordSpec. This example was intended to resemble the Specs2 unit test above as closely as possible, so this test class extends WordSpec, which provides a DSL for writing unit tests that is quite similar to Specs2’s Specification.

The sample code for this lecture can be found in courseNotes/src/test/scala/TestScalaTest.scala.

Notice that:

The structure of the unit tests are almost identical for the Specs2 Specification and ScalaTest WordSpec DSLs – the only difference is how the body of the unit tests are actually written.
WordSpec uses the word should instead of must.
Triple equals (===) is used to assert that two values must be equal.

import org.junit.runner.RunWith
import org.scalatest.junit.JUnitRunner
import org.scalatest._
import org.scalatest.Matchers._

@RunWith(classOf[JUnitRunner])
class TestScalaTest extends WordSpec {
  "The ’Hello world’ string" should {
    "contain 11 characters" in {
      "Hello world".length === 11
    }

    "start with ’Hello’" in {
      "Hello world" should startWith("Hello")
    }

    "end with ’world’" in {
      "Hello world" should endWith("world")
    }
  }
}

The ScalaTest documentation discusses additional test types. For more information.

Lets look at some specific ScalaTest features now.

Testing Strings

The ScalaTest DSL is quirky; the syntax changes depending on whether there are an even or odd number of words used in the comparison.

You can test all the things you might expect: equality, length, startsWith, endsWith, if a substring is present in (include), and regular expressions (regexes).

ScalaTest example test

"Strings" should {
  val string = """Thank you for your order.
                 |You enrolled in the best course ever!
                 |Go study and become successful.
                 |""".stripMargin
  "compare normally" in {
    string === string
    string.length shouldBe 96
    string should startWith("Thank you for your order")
    string should include("the best course ever!")
    string should endWith("successful.\n")
  }
  "compare with regexes" in {
    string should fullyMatch regex "(?s)Thank .* best .*"
    string should startWith regex "Thank .*"
    string should include regex "best course .*"
    string should endWith regex "successful.\n"
}
}

We can also use regular expressions to compare strings.

Strings can fully match a regex by using fullyMatch:
```
string should fullyMatch regex "(?s)Thank .* best .*"
```
The (?s) inside the regex enables the DOTALL flag, which tells the regex engine built into the JVM to treat newlines as regular characters.
Strings can be tested to ensure that the beginning of the string matches a regular expression:
```
string should startWith regex "Thank .*"
```
Strings can be tested to ensure that they contain a substring that matches a regex:
```
string should include regex "best course .*"
```
Strings can be tested to ensure that the end of the string matches a regular expression:
```
string should endWith regex "Thank .*"
```

Working With `OptionValues`

The ScalaTest documentation describes a trait called org.scalatest.OptionValues that makes working with Option values more convenient. The OptionValues companion object is documented in the ScalaTest Scaladoc; this object allows you to import the OptionValues behavior instead of mixing in the OptionValues trait into the test class. The OptionValues import is automatically performed if you import org.scalatest.Matchers._. Let’s look at the following code.

ScalaTest OptionValues example

"OptionValue" should {
  "work for Some values" in {
    val option = Some(3)
    option shouldBe defined
    option.value shouldBe 3
    option.value should be < 7
    option should contain oneOf (3, 5, 7, 9)
    List(3, 5, 7, 9) should contain (option.value)
    option should not contain oneOf (7, 8, 9)
    List(5, 7, 9) should not contain option.value
    option should contain noneOf (7, 8, 9)
  }

  "work for None" in {
    val option: Option[Int] = None
    // the following are all equivalent:
    option shouldEqual None
    option shouldBe None
    option should === (None)
    option shouldBe empty
  }
}

The first test, "OptionValue should work for Some values", defines an immutable variable called option, of type Option[Int] with value Some(3).

This verifies that option is defined, or in other words that the Option container is not empty. Notice that ScalaTest DSL requires shouldBe to be written as one word.
ScalaTest shouldBe example 1
```
option shouldBe defined
```
This verifies that the value of option must be exactly 3.
ScalaTest shouldBe example 2
```
option.value shouldBe 3
```
This verifies that the value of the option is less than 7. The value of the Option container is compared to the test value by writing option.value. In this case, the ScalaTest DSL requires should be to be written as two words. I find it hard to remember when to write shouldBe as one word or should be as two words, but my IDE tells me when I guess wrong.
```
option.value should be < 7
```
This verifies that the option value appears in the sequence 3, 5, 7, or 9, expressed as literals using the ScalaTest DSL. The value of the Option container is compared to the test values by writing option, without qualifying with .value.
```
option should contain oneOf (3, 5, 7, 9)
```
This does the same test, but shows the syntax necessary in order to work with a collection instead of a literal sequence. Notice the parentheses around (option.value), and that the .value qualifier was required.
```
List(3, 5, 7, 9) should contain (option.value)
```
This does the opposite test. This time the .value qualifier was not required.
```
option should not contain oneOf (7, 8, 9)
```
This again does the opposite test, but shows the syntax necessary in order to work with a collection instead of a literal sequence. Notice there is no parentheses around option.value.
```
List(3, 5, 7, 9) should not contain option.value
```
This shows the syntax necessary to ensure that the option value is not in the sequence of values.
```
option should contain noneOf (7, 8, 9)
```

The second test, "OptionValue should work for None", could have also been written any of these ways.

ScalaTest equivalent should examples

option shouldEqual None
option shouldBe None
option should === (None)
option shouldBe empty

Working with `EitherValues`

The ScalaTest documentation describes a trait called org.scalatest.EitherValues that makes working with Either values more convenient. The EitherValues object is documented in the ScalaTest Scaladoc; this object allows you to import the EitherValues behavior instead of mixing in the EitherValues trait into the test class. Unlike the OptionValues import, the EitherValues import is NOT automatically performed if you import org.scalatest.Matchers._. Let’s look at the following code.

ScalaTest either examples

"EitherValues" should {
  import org.scalatest.EitherValues._
  val either: Either[String, Int] = Right(3)
  "work for Right values" in {
    either.right.value shouldBe 3
    either shouldBe right
    either should not be left
  }
}

Again we find that the ScalaTest DSL is quirky when working with Either values. First we define an immutable variable called either of type Either[String, Int], and give it the value Right(3).

The test "EitherValues should work for Right values" verifies the value of the either variable.

This verifies that the variable is a Right and that its value is 3.
ScalaTest either example
```
either.right.value shouldBe 3
```
This simply verifies that the variable is a Right.
ScalaTest either example
```
either shouldBe ’right
```
This simply verifies that the variable is not a Left.
ScalaTest either example
```
either should not be ’left
```

Testing a Value Against Multiple Predicates

You can test a value against multiple predicates by enclosing them within parentheses and combining them with and and or.

ScalaTest example with multiple predicates

"Multiple predicates" should {
  "combine" in {
    val string = """Thank you for your order.
                   |You enrolled in the best course ever!
                   |Go study and become successful.
                   |""".stripMargin
    string should (
      include("Thank you for your order") and
      include("You enrolled in")
    )
    string should (
      include("Thank you for your order") or
      include("You enrolled in")
    )
  }
}

Running Unit Tests

You can run unit tests in three environments:

From the command line using SBT
From IntelliJ IDEA’s test runners
From Visual Studio Code’s test runners

We will explore each of these now.

SBT

SBT has a good test runner. You can run all unit tests by using the sbt test command, as shown below. Recall that the leading tilde (~) causes the tests to be rerun any time a source file is changed.

Shell

$ sbt ~test
[info] Loading global plugins from /home/mslinn/.sbt/plugins
[info] Loading project definition from /var/work/course_scala_intro_code/courseNotes/project
[info] Set current project to scalaIntroCourse (in build file:/var/work/course_scala_intro_code/courseNotes/)
[success] Total time: 33 s, completed Sep 2, 2013 5:14:36 PM
1.
Waiting for source changes...
(press enter to interrupt)

You can also just run a specific test class, like this (note the quotes):

Shell

$ sbt ~"testOnly ScalaTestDemo"
[info] Loading global plugins from /home/mslinn/.sbt/plugins
[info] Loading project definition from /var/work/course_scala_intro_code/courseNotes/project
[info] Set current project to scalaIntroCourse (in build file:/var/work/course_scala_intro_code/courseNotes/)
[success] Total time: 2 s, completed Sep 2, 2013 5:16:42 PM
1.
Waiting for source changes...
(press enter to interrupt)

You can also run a specific unit test within a test class by using the -z argument within sbt:

Shell

$ sbt
[info] Loading global plugins from /home/mslinn/.sbt/0.13/plugins
[info] Loading project definition from /var/work/course_scala_intro_code/courseNotes/project
[info] Set current project to scalaIntroCourse (in build file:/var/work/course_scala_intro_code/courseNotes/)
scala> testOnly ScalaTestDemo -z "The ’Hello world’ string should contain 11 characters"
[info] Updating {file:/var/work/course_scala_intro_code/courseNotes/}coursenotes...
[info] Resolving org.ccil.cowan.tagsoup#tagsoup;1.2 ...
[info] Done updating.
[info] Compiling 4 Scala sources to /var/work/course_scala_intro_code/courseNotes/target/scala-2.11/test-classes...
[info] ScalaTestDemo:
[info] The "Hello world" string
[info] - should contain 11 characters
[info] - should start with "Hello"
[info] - should end with "world"
[info] ScalaCheck
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] ScalaTest
[info] Run completed in 203 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 3, Failed 0, Errors 0, Passed 3
[success] Total time: 5 s, completed Nov 28, 2014 9:25:20 PM

Notice how the test name was constructed: the name of the should clause was concatenated with should, followed by the name of the in clause, to form the full test name: "The ’Hello world’ string should contain 11 characters".

Here is how you can specify all of this on one line. We discussed this syntax in the .sbtrc section of the SBT Global Setup lecture. Notice that the embedded quotes are escaped:

Shell

$ sbt ";testOnly ScalaTestDemo -z \"The ’Hello world’ string should contain 11 characters\""
[info] Loading global plugins from /home/mslinn/.sbt/0.13/plugins
[info] Loading project definition from /var/work/course_scala_intro_code/courseNotes/project
[info] Set current project to IntroScalaCourse (in build file:/var/work/course_scala_intro_code/courseNotes/)
[info] Updating {file:/var/work/course_scala_intro_code/courseNotes/}coursenotes...
[info] Resolving org.ccil.cowan.tagsoup#tagsoup;1.2 ...
[info] Done updating.
[info] Compiling 4 Scala sources to /var/work/course_scala_intro_code/courseNotes/target/scala-2.11/test-classes...
[info] ScalaTestDemo:
[info] The "Hello world" string
[info] - should contain 11 characters
[info] - should start with "Hello"
[info] - should end with "world"
[info] ScalaCheck
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[info] ScalaTest
[info] Run completed in 203 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 3, Failed 0, Errors 0, Passed 3
[success] Total time: 8 s, completed Jan 26, 2016 11:22:51 AM

A leading tilde in combination with testQuick causes only those tests affected by the most recently modified change to be recompiled and tested. This is a very useful incantation.

Shell

$ sbt ~testQuick
[info] Loading global plugins from /home/mslinn/.sbt/plugins
[info] Loading project definition from /var/work/course_scala_intro_code/courseNotes/project
[info] Set current project to scalaIntroCourse (in build file:/var/work/course_scala_intro_code/courseNotes/)
[success] Total time: 2 s, completed Sep 2, 2013 5:16:42 PM
1.
Waiting for source changes...
(press enter to interrupt)

IntelliJ IDEA

Running ScalaTest and Specs 2 unit tests with IDEA is easy. Simply right-click on the unit test, and select a runner. You can see the name of the test runner in the lower left corner of the window when it is selected in the popup menu. IDEA creates a run configuration based on your selection and launches the test. Here is what IDEA looks like after running the unit tests in this lecture.

Exercise – Write Specs2 and ScalaTest Unit Tests

Create a new SBT project.
Write a unit test using both Specs2 and ScalaTest that fetches the contents of https://scalacourses.com and does a case-insensitive search that verifies the word scala is present.
Use any IDE, or no IDE.

Hints

You can read the contents of a URL as follows:

Scala code

io.Source.fromURL("https://scalacourses.com").getLines.mkString

You can convert a String to lower case via the String.toLowerCase method.

Solutions

The ScalaTest solution is provided in courseNotes/test/scala/solutions/ScalaTestSolution.scala.

Scala code

package solutions

import org.scalatest._

class ScalaTestSolution extends WordSpec {
  "ScalaCourses.com" should {
    "contain the word scala" in {
      val contents = io.Source.fromURL("https://scalacourses.com").getLines.mkString
      contents.toLowerCase.contains("scala")
    }
  }
}

Run it like this:

Shell

$ sbt "testOnly solutions.ScalaTestSolution"

The Specs2 solution is provided in courseNotes/src/test/scala/solutions/Specs2Solution.scala.

ScalaTest example

package solutions

import org.specs2.mutable._

class Specs2Solution extends Specification {
  "ScalaCourses.com" should {
    "contain the word scala" in {
      val contents = io.Source.fromURL("https://scalacourses.com").getLines().mkString
      contents.toLowerCase must contain("scala")
    }
  }
}

Run it like this:

Shell

$ sbt "testOnly solutions.Specs2Solution"

Previous lecture: Algebraic Data Types

Next lecture: Recursion and Functional Programming

© Copyright 1994-2024 Michael Slinn. All rights reserved.
If you would like to request to use this copyright-protected work in any manner, please send an email.

This website was made using Jekyll and Mike Slinn’s Jekyll Plugins.

Project Setup

Specs2

ScalaTest

ScalaTest and Specs2 In the Same Project

Quick and Easy Test Specifications

Specs2

ScalaTest

Testing Strings

Working With OptionValues

Working with EitherValues

Testing a Value Against Multiple Predicates

Running Unit Tests

SBT

IntelliJ IDEA

Exercise – Write Specs2 and ScalaTest Unit Tests

Solutions

Working With `OptionValues`

Working with `EitherValues`