by Bruno Dias
Thanks to the generosity of PROCJAM's kickstarter backers, I'm here to teach you the basics about Improv. This is a semi-advanced tutorial; some comfort with text editors, the Javascript programming language, and command-line tools is expected, but I won't be delving into anything too complicated (I hope).
Improv is a tool for generating text procedurally from a grammar. A grammar is a set of rules that define how to write a text by assembling it from bits and pieces. The major disadvantage of grammars is that they're laborious; they have to be written by hand or generated by some other method, maybe from structured data. But grammars can generate text that's structured and consistent, and they're more accessible to use (and get good results with) than markov chains or predictive text or neural networks or other fancy machine learning techniques.
Improv owes its basic structure to Kate Compton's Tracery and to the methodology Emily Short developed to write The Annals of the Parrigues. Improv was designed for Voyageur, and as such it had to be capable of generating fairly varied paragraph-long descriptions of places and things that were self-consistent and described something with complex variable attributes. Voyageur is a space exploration and trading game, in which players hop from planet to planet. In Voyageur, a planet has independent factors such as ecology, ideology, and economy; and it has to be given a description that touches on all of those factors.
We need an environment where we can run JavaScript. In practice, most things you might use Improv for would run in the browser, or in a browser-like environment like Electron or Cordova. But Improv can run on Node.js just as well, which is what I'll be doing in this tutorial for simplicity's sake; we need Node anyway to download a copy of Improv through npm, Node's package manager utility. So, on a machine with Node.js installed (and using either the default bash shell on Linux/MacOS or Powershell on Windows), you can set up a simple Improv environment by doing:
mkdir improv-tutorial
cd improv-tutorial
npm install fs-jetpack js-yaml improvnpm will complain a bunch about you not having a package.json file in your project, but it will install the packages. You'll want to create an improv-tutorial.js file here, and a grammar.yaml file.
Here's our simple script for running Improv:
// Require libraries
const Improv = require('improv');
const yaml = require('js-yaml');
const fs = require('fs-jetpack');
// Load our data from a file
const grammarData = yaml.load(fs.read('grammar.yaml'));
// Create a generator object from this data
const generator = new Improv(grammarData, {
filters: [Improv.filters.mismatchFilter()],
reincorporate: true
});
// Generate text and print it out
console.log(generator.gen('root', {}));
Improv, imported from the library, is a constructor (so use it with new) that creates and returns a generator object. It takes two parameters, a grammar and another object configuring the generator. There are a lot of options here but for purposes of this tutorial we're only using filters and reincorporate; I'll get back to what that means in a bit.
You can run this with just node improv-tutorial.js, but it will throw an error; grammar.yaml doesn't exist yet, so let's create it.
Improv is a JavaScript library, so grammars in Improv are javascript objects, but they have a pretty deeply nesting structure. For ease of reading and to preserve my own sanity, I like to write them in YAML instead. A basic grammar looks like this:
root:
groups:
-
tags: []
phrases:
- "The [:prefix] [:name] is a [:class] commissioned in 1888"
prefix:
groups:
-
tags: []
phrases:
- "HMS"
name:
groups:
-
tags: []
phrases:
- "Invincible"
class:
groups:
-
tags: []
phrases:
- "cutter"
Save that as grammar.yaml and run the script again, and you should get exactly this output: "The HMS Invincible is a cutter commissioned in 1888."
There's a fair bit going on here so let me go through it step by step: Each key in this object (like root, prefix) is a rule. Each rule contains groups (in this example, only one), and each group has a list of tags and a list of phrases. Each phrase is an individual chunk of text that might contain directives ([:prefix] for example) that point at other rules.
Here's how Improv operates: We ask it to generate the root rule (generator.gen('root', {}) back in the JavaScript file). Improv goes over all the groups for that rule; of course, there's only one in this file. There's a filtering step first, which we'll talk about in more detail later, which selects what groups it will use. It then collates all of the phrases in the groups, and picks one at random.
Each phrase is then templated; Improv looks for directives surrounded by [brackets]. Directives starting with a colon, like [:prefix], are the bread and butter of Improv text generation; they're references to other rules, which then get inserted in place. When Improv encounters one of those, it recurs, generating that rule.
Right now this is decidedly unimpressive; we have the same output every time, since all our rules only have one phrase in them. We can add variations by adding more phrases:
root:
groups:
-
tags: []
phrases:
- "The [:prefix] [:name] is a [:class] commissioned in [#1880-1910]"
prefix:
groups:
-
tags: []
phrases:
- "HMS"
name:
groups:
-
tags: []
phrases:
- "Invincible"
- "Unstoppable"
- "Indefatigable"
- "Inevitable"
- "Inerrant"
class:
groups:
-
tags: []
phrases:
- "cutter"
- "torpedo boat"
- "light cruiser"
- "heavy cruiser"
- "battleship"
And just like that, we now have variations:
The HMS Unstoppable is a battleship commissioned in 1898
The HMS Invincible is a cutter commissioned in 1891
The HMS Indefatigable is a light cruiser commissioned in 1906
Note the [#1880-1910]; that's a special directive that produces a random number between 1880 and 1910.
Say we want to include both civilian and military vessels in our grammar. Let's add civilian variants, with their own names, classes, and prefix:
root:
groups:
-
tags: []
phrases:
- "The [:prefix] [:name] is a [:class] commissioned in [#1880-1910]"
prefix:
groups:
-
tags:
- ['type', 'military']
phrases:
- "HMS"
-
tags:
- ['type', 'civilian']
phrases:
- "SS"
name:
groups:
-
tags:
- ['type', 'military']
phrases:
- "Invincible"
- "Unstoppable"
- "Indefatigable"
- "Inevitable"
- "Inerrant"
-
tags:
- ['type', 'civilian']
phrases:
- "Dromedary"
- "Mule"
- "Camel"
- "Ox"
class:
groups:
-
tags:
- ['type', 'military']
phrases:
- "cutter"
- "torpedo boat"
- "light cruiser"
- "heavy cruiser"
- "battleship"
-
tags:
- ['type', 'civilian']
phrases:
- "steamship"
- "cargo ship"
- "ferry"
We now have groups of phrases for both civilian and military ships; and we're using the tags property. The important thing to notice here is that tags is a list of lists. Each individual tag is a list, like ['type', 'military']; that's one tag, not two. This is important; it means that tags form a hierarchy, and will be relevant for filtering later.
The first way tags get used is in reincorporation, which we set to true earlier. Note that when we called Improv.gen(), we gave it two parameters; the first one, 'root', is the name of the rule we want to start from. The second one was just {}, a brand new, empty object. This is our model, an object that holds data about the text we're generation. Every time Improv selects and uses a phrase, assuming reincorporation is turned on, it gets added to the model. We can make a small change to our script so we can inspect the model afterwards:
const model = {};
// Generate text and print it out
console.log(generator.gen('root', model));
console.log(model);Example output:
The HMS Inevitable is a light cruiser commissioned in 1894
{ tags: [ [ 'type', 'military' ] ] }
We can see that the tag we used was added to our empty object. You can generate a model ahead of time using some other method and pass it to Improv, and you can save a model after generating text (as a variable) to reuse it, generating different sets of text with the same tags.
So far, we haven't touched on Improv's most important feature, filtering. Remember filters: [Improv.filters.mismatchFilter()]? What that line is doing is configuring the generator to use Improv.filters.mismatchFilter() as its only filter. A filter is just a function used to help Improv select what groups it should use when following a rule. Improv.filters contains a bunch of built-in functions that return ready-made filters to be used by Improv, but nothing stops you from writing your own. The Improv documentation has a list of built-in filters and what they're used for, but for our purposes, we'll just be looking at the *mismatch filter.*
What the mismatch filter does is look for tags that contradict the model's tags, and discards groups that have those tags. It's the most basic and useful filter; it stops Improv from contradicting itself, assuming that the text in the grammar has been tagged so as to prevent that.
The definition of "contradicts" here is:
- The model and the group have a tag with the same first element; - But not every element is identical.So, 'type', 'military' is a mismatch with 'type', 'civilian'. But it allows exact matches ('type', 'military'), as well as tags that are totally different ('propulsion', 'sail', for example).
This is why tags are lists; the first element is a sort of category, and subsequent elements define sub-categories. Tags state what aspect of the thing they represent, so that contradictions can be identified; a ship can't be both military and civilian or big and small; but it can be military and have sails, or it can be civilian and be a large ship.
You can add more tags and variations to expand this grammar and make ever more complex descriptions; the demo in the Improv github repository contains a very detailed worked example of generating fantasy military ships.
This is only really scraping the surface of Improv's functionality. Most filters don't outright discard groups; instead, they return a number (negative or positive), and numbers returned from all filters get tallied up to give each group a *salience score.* Improv then chooses what to use based on salience score. This can be used to do more than just stop text from contradicting itself. Voyageur uses a complex "secret sauce" of filtering methods to produce its text. One thing it selects for is breadth of description, trying to use tags it hasn't used already to produce a paragraph that goes over most aspects of what it's describing. It also tries to look for specificity, valuing bits of text associated with very specific conditions, so they show up on the rare occasions where they're appropriate.