Configuration
Configuration can be written i json or yaml format.
YAML basics
YAML is a human-readable data serialization language. It is commonly used for configuration files, but could be used in many applications where data is being stored [...] YAML 1.2 is a superset of JSON, another minimalist data serialization format where braces and brackets are used instead of indentation.
- Whitespace indentation is used to denote structure; however tab characters are never allowed as indentation.
- Comments begin with the number sign (#).
- List members are denoted by a leading hyphen (-) with one member per line.
- Associative arrays are represented using the colon space (: ) in the form key: value, either one per line or enclosed in curly braces ({ }) and separated by comma space (, ).
- Strings (scalars) are ordinarily unquoted, but may be enclosed in double-quotes ("), or single-quotes (').
(Source: Wikipedia)
Examples:
products:
- sku : BL394D
quantity : 4
description : Basketball
price : 450.00
- sku : BL4438H
quantity : 1
description : Super Hoop
price : 2392.00 # Oh my!
products:
- { sku: BL394D, quantity: 4, description: Basketball, price: 450.00 }
- { sku: BL4438H, quantity: 1, description: Super Hoop, price: 2392.00 }
mailbody: >
Wrapped text
will be folded
into a single
paragraph
Blank lines denote
paragraph breaks
A useful tool is Yaml lint for verifying that your yaml is valid.
Foopipes configuration file structure
version: 2
plugins: <list of plugin specifications>
services: <associative array of services>
pipelines: <list of pipelines>
Plugins
Convention based plugin loading. Foopipes will try its best to resolve and load the plugin:
plugins:
- Elasticsearch
Explicit plugin loading:
plugins:
- { path: <filepath>, filename: <assembly filename>, assemblyName: <assembly name> }
Writing custom .NET plugins is a way to extend the functionallity of Foopipes. Often the functionallity that is needed can be implemented with node.js modules, but plugins is also a way to package functionallity for a specific use case. See Writing Plugins for more information.
Services
The Services section contains the definition of all event sources and services. You can then reference a service later in the pipeline section. A service may also support variable data binding, see Variable Binding.
services:
scheduler:
type: scheduler
myWebhook:
type: httplistener
path: myWebhook
elasticsearch:
url: "http://${elasticsearch|localhost}:9200",
termMappings:
-
index: entries
typename: entry
field: fields.url
In this example we have a scheduler event source, a http listener on the path /myWebhook
, and an Elasticsearch configuration.
Also we define a not_analyzed term mapping for Elasticsearch which is created on startup. Term mappings are not needed for the pipelines in this example, but are useful when querying for data using exact values.
The url to Elasticsearch is obtained from the environment variable elasticsearch
, with a fallback to localhost.
Automatically created services
There are a couple of services that are automatically created when the application starts unless overridden in this config section.
Most important is the queue
service as Foopipes is built around message passing.
They are:
* queue
- An in-memory message queue
* file
- a default file storage.
* http
- for sending http request with the http
task.
Pipelines
Pipelines are processing that starts after a service fires an event.
pipelines:
-
when:
- { service: myWebhook }
from:
- { task: http, url: "http://myurl/entries" }
to:
- { task: publish, service: queue, topic: entry }
-
when:
- { service: queue, topic: entry }
do:
- { task: node, module: mymodule }
to:
- { task: store, service: elasticsearch, index: myindex, dataType: entry, key: "#{entryId}" }
What happens here is when an event is fired from the service myWebhook
, a message with the contents of the request body is:
- Matched to the first pipeline, as it subscribes to events from
myWebhook
. - Json is loaded from an url.
- All entries from the previous task are passed on to a message queue with the topic
entry
. - A cascade of new events are fired from the message queue, and as the topic
entry
matches the second pipeline's when criterias, the second pipeline is starts for each entry. - The node.js module
mymodule
's default export is invoked to process the entry. - The outputs from the previous step are stored to the
elasticsearch
service with the key specified in theentryId
field of each entry.
Config shorthand format
There's an option to write pipeline configuration in a shorthand format where the first key specify the task's name: taskName: defaultarg
instead of having to specify task: taskName
for each step in the pipeline.
The defaultarg
is specific for each task type. For instance, the http
task's default argument is url
.
See Tasks reference for built in tasks and what their default argument is.
when:
- { service: [defaultarg], [arg: value] }
- service: [defaultarg]
- <service>
do:
- { <task>: [<defaultarg>], [arg: value] }
- <task>: [<defaultarg>]
- <task>
For example, the following config is valid:
pipelines:
-
when:
- queue: started
- scheduler
from:
- { http: "https://jsonplaceholder.typicode.com/posts", method: get }
to:
- { file: "post_#{id}.json", path: "." }
finally:
- exit
Which is the same as:
pipelines:
-
when:
- { service: queue, topic: started }
- { service: scheduler }
from:
- { task: http, url: "https://jsonplaceholder.typicode.com/posts", method: get }
to:
- { task: file, filename: "post_#{id}.json", path: "." }
finally:
- { task: exit }
Here are four ways to write the same thing:
to:
- { task: file, filename: "post_#{id}.json", path: "." }
- { file: "post_#{id}.json", path: "." }
- file: "post_#{id}.json"
path: "."
- task: file
filename: "post_#{id}.json"
path: "."
Or:
finally:
- { task: exit }
- { exit }
- exit