HOCON Exercise and Extended Example

Published 2014-06-21.
Time to read: 4 minutes.

This lecture contains an exercise and an extended code example for HOCON configuration.

Exercise – Working with Application Defaults and Nested Settings

Akka and Play use nested HOCON settings to store configuration data. You can get a list of all default values by invoking ConfigFactory.load.

Write a program that converts the default values returned by ConfigFactory.load as nested settings to a ’flat’ representation. For example, convert settings from this format:
HOCON
```
a = {
  b = {
    c = 1
    d = hello
  }
}
```
Into this:
HOCON
```
a.b.c = 1
a.b.d = hello
```
Save the flattened version to ~/application.conf, with keys sorted alphabetically.
Load the flattened version (using ConfigFactory.parseResources) and verify that every flattened setting is found in the original hierarchical configuration. This is relatively easy to kinda sorta make work, but fairly difficult to do properly. You will learn a lot more if you soldier bravely forward and get this to work properly!

Hints.

Config entries are key/value pairs, just the same as maps.
You can get a Set of all entries from a Config instance by calling Config.entrySet, just the same as for maps.

You can obtain a properly formatted value from a Config entry by using ConfigRenderOptions like this:

Scala code

val renderOptions = ConfigRenderOptions.concise.setComments(false)
val formattedValue = entry.getValue.render(renderOptions)

HOCON string values may be surrounded by double quotes, but this is optional. The render method does not consistently enclose strings in double quotes. Deal with it.
You can reference a File called application.conf in your home directory by invoking:
Scala code
```
new java.io.File(System.getProperty("user.home"), "application.conf")
```
Reading the Config documentation will help you do this exercise, but to do a proper job you will probably need to poke around in the Config source code. Such is the open source programmer’s reality. Happily, Config is neatly broken into interface specification and implementation, and you only need to program against the interfaces.

Solution

The complete solution is too long to show all at once. We will walk through it bit by bit here.

The complete program is provided in courseNotes/src/main/scala/solutions/ConfigFlatNest.scala.

We store the default settings into configApp, which has type Config. It is created by ConfigFactory.

Scala code

import com.typesafe.config._
import scala.collection.JavaConverters._

val configApp: Config = ConfigFactory.load

Now we convert the Config entries to a List of tuples – more precisely List[(String, Any)]. The List is sorted by the first tuple element and result is stored into tuples.

Scala code

def flatKeyValueTuples(config: Config): List[(String, Any)] = {
  for {
    entry <- config.entrySet.asScala.toList
  } yield (entry.getKey, entry.getValue.render)
}
val tuples = flatKeyValueTuples(configApp).sortBy(_._1)

Next the List of tuples is transformed into a List[String], where each String item is of the form name = value. The entire List[String] is then turned into a giant String, with newlines between each item and an extra newline at the end.

Scala code

val result1 = tuples.map( x => s"${x._1} = ${x._2}" )
                    .mkString("", "\n", "\n")

Now we write the giant String to ~/applications.conf using File and PrintWriter

Scala code

import java.io.{File, PrintWriter}
val file = new File(System.getProperty("user.home"), "application.conf")
val writer = new PrintWriter(file)
writer.write(result1)
writer.close()

It is easy to read contents the file back into a new Config called confFlat:

Scala code

val confFlat = ConfigFactory.parseFile(file)

Next we walk through all of the entries in confFlat and compare the values to corresponding values in the original configApp Config. This code takes a while to figure out, so we’ll start with an overview. The forall combinator is used for this purpose; inside the forall expression the comparesEqual variable is set if an entry in confFlat matches the item in configApp. Since a value could be a scalar such as an Int, a String, a Double, etc or it might be a collection we need to have a recursive transformation of values.

Scala code

val renderOptions = ConfigRenderOptions.concise.setComments(false)
val originalEntrySet = configApp.entrySet
val matched = confFlat.entrySet.asScala.forall { kv =>
  val flatKey: String = kv.getKey
  val flatValue: String = kv.getValue // need to transform this value into something useful
  val originalValue: String = configApp.getAnyRef(flatKey).toString.replace("\n", "\\n")
  val comparesEqual: Boolean = originalValue.toString == flatValue
  if (!comparesEqual)
    println(s"$flatKey: $originalValue (${originalValue.getClass.getName}) != $flatValue (${flatValue.getClass.getName})")
  comparesEqual
}

Here is the flatValue transformation:

Scala code

val flatValue = kv.getValue match {
  case list: ConfigList =>
    list.unwrapped.asScala.map { renderObject }.mkString("[", ", ", "]")

  case value =>
    removeOuterParens(value.render(renderOptions))
}

Now we can provide the implementation of the three methods:

Scala code

def removeOuterParens(string: String) =
  if (string.startsWith("\"") && string.endsWith("\"")) string.substring(1, string.length-1) else string

def render(value: ConfigValue): String =
  value.unwrapped match {
    case l: List[_] =>
      l.map { case item: ConfigValue => render(item) }.mkString(", ")

    case configString: String =>
      removeOuterParens(value.render(renderOptions))

    case _ =>
      value.render(renderOptions)
  }

def renderObject(value: Object): String =
  value match {
    case jList: java.util.List[_] =>
      jList.asScala.map { case obj: Object => renderObject(obj) } .mkString(", ")
     case string: String =>
      removeOuterParens(string)
     case _ =>
      value.toString
  }

And now we conclude with this:

Scala code

println(s"""Flat version in ${file.getAbsolutePath} ${ if (matched) "matches" else "does not match" } original hierarchical version""")

You can run this solution by typing:

Shell

$ sbt "runMain solutions.ConfigFlatNest"

AWS Authentication Example

The sample code for this lecture can be found in courseNotes/src/main/scala/ConfigAWS.scala.

Here is a practical example: your application requires an AWS access key and secret key; there are a default values for development, but production values must be supplied via environment variables. The following fragment of an application.conf file is all that is needed to specify these values. Let’s set up a layering of configuration files.

Scala code

import com.typesafe.config.{Config, ConfigFactory}

val defaultStr = """aws {
  accessKey = "stringAccessKey"
  secretKey = "stringSecretKey"
}""".stripMargin
val strConf = ConfigFactory.parseString(defaultStr).resolve
val appConf = ConfigFactory.parseResources("application.conf").resolve
val libConf = ConfigFactory.parseResources("library.conf").resolve
val defConf = ConfigFactory.load
val combined: Config =
  strConf                  // highest priority
    .withFallback(appConf) // second highest priority
    .withFallback(libConf) // priority 3
    .withFallback(defConf) // lowest priority

I added the following to the courseNotes/src/main/resources/application.conf configuration file:

application.conf

aws {
  accessKey = "applicationAccessKey"
  accessKey = ${?ACCESS_KEY}

  secretKey = "applicationSecretKey"
  secretKey = ${?SECRET_KEY}
}

The above syntax defines default values for aws.accessKey and aws.secretKey, which will be overridden if environment variables called ACCESS_KEY or SECRET_KEY are defined. Because bash does not allow environment variables to contain a dot (period), the convention is to name them in all capital letters, using underscores (_).

The ConfigAWS class in the courseNotes contains the above code in a runnable program. If the environment variables are not set, then the values specified in application.conf are used.

We need to be able to display the configuration values, so we’ll write a method that accepts a configuration name and a Config object, then check to see if there is a key/value pair for each of the keyList items implicitly passed in. The Config class does not provide a direct way to obtain the available keys in an instance, so we use the asScala method provided by collection.JavaConverters to wrap the entrySet of key/value pairs into a Scala equivalent collection, and then use map to transform the key/value pairs into a Set of keys. We then use the contains method to detect if the desired keys are defined.

Scala code

import collection.JavaConverters._

def showValues(configName: String, config: Config)(implicit keyList: List[String]): Unit = {
  val keySet: Set[String] = config.entrySet.asScala.map(_.getKey).toSet

  def show(key: String): Unit =
    if (keySet contains key) {
      val accessKey = config.getString(key)
      println(s"$configName defines $key=$accessKey")
    } else
      println(s"$configName does not define $key as one of its ${keyList.size} keys")

  if (keySet.isEmpty) {
    println(s"$configName is empty")
  } else {
    keyList foreach show
  }
  println()
}

The showValues method displays the presence or absence of each of the items in keyList for each Config instance.

Scala code

def showAll(implicit keyList: List[String]): Unit = {
  showValues("defaultStr",       strConf)
  showValues("library.conf",     libConf)
  showValues("application.conf", appConf)
  showValues("Default Config",   defConf)
  showValues("Combined Config",  combined)
}

showAll(List("aws.accessKey", "aws.secretKey", "akka.version", "user.home"))

Once the above code is wrapped in an App instance called ConfigAWS we can run it, without setting environment variables. You will find the completed program in the courseNotes directory.

Shell

$ sbt "run-main ConfigAWS"
[info] Loading global plugins from /home/mslinn/.sbt/plugins
[info] Loading project definition from /home/mslinn/work/course_scala_intermediate_code/courseNotes/project
[info] Set current project to IntermediateScala Course (in build file:/home/mslinn/work/course_scala_intermediate_code/courseNotes/)

defaultStr defines aws.accessKey=stringAccessKey
defaultStr defines aws.secretKey=stringSecretKey
defaultStr does not define akka.version as one of its 4 keys
defaultStr does not define user.home as one of its 4 keys

library.conf is empty

application.conf does not define aws.accessKey as one of its 4 keys
application.conf does not define aws.secretKey as one of its 4 keys
application.conf defines akka.version=2.2.1
application.conf does not define user.home as one of its 4 keys

Default Config does not define aws.accessKey as one of its 4 keys
Default Config does not define aws.secretKey as one of its 4 keys
Default Config defines akka.version=2.2.1
Default Config defines user.home=/home/mslinn

Combined Config defines aws.accessKey=stringAccessKey
Combined Config defines aws.secretKey=stringSecretKey
Combined Config defines akka.version=2.2.1
Combined Config defines user.home=/home/mslinn

[success] Total time: 1 s, completed Sep 12, 2013 9:19:44 PM

If you define the environment variables and run the program, the output changes to:

Shell

$ export ACCESS_KEY=AXWAJ8K7JK3JDRL2JOQ
$ export SECRET_KEY=K9LSJDKEJ738373LSLS923821
$ sbt "runMain ConfigAWS"
[info] Loading global plugins from /home/mslinn/.sbt/plugins
[info] Loading project definition from /home/mslinn/work/course_scala_intermediate_code/courseNotes/project
[info] Set current project to IntermediateScala Course (in build file:/home/mslinn/work/course_scala_intermediate_code/courseNotes/)

defaultStr defines aws.accessKey=stringAccessKey
defaultStr defines aws.secretKey=stringSecretKey
defaultStr does not define akka.version as one of its 4 keys
defaultStr does not define user.home as one of its 4 keys

library.conf is empty

application.conf defines aws.accessKey=AXWAJ8K7JK3JDRL2JOQ
application.conf defines aws.secretKey=K9LSJDKEJ738373LSLS923821
application.conf defines akka.version=2.2.1
application.conf does not define user.home as one of its 4 keys

Default Config defines aws.accessKey=AXWAJ8K7JK3JDRL2JOQ
Default Config defines aws.secretKey=K9LSJDKEJ738373LSLS923821
Default Config defines akka.version=2.2.1
Default Config defines user.home=/home/mslinn

Combined Config defines aws.accessKey=stringAccessKey
Combined Config defines aws.secretKey=stringSecretKey
Combined Config defines akka.version=2.2.1
Combined Config defines user.home=/home/mslinn

[success] Total time: 6 s, completed Sep 12, 2013 9:18:36 PM

Previous lecture: HOCON Configuration

Next lecture: Parametric Variance and Bounds