FsShelter is a library for implementation of Apache Storm components and topologies in F#. FsShelter is based on and a major rewrite of FsStorm. It departs from FsStrom in significant ways and therefore has been split into itsown project.
Overall, the librabry provides "batteries included" experience with wrappers for Nimbus API as well as support for packaging and exporting:
- bundle and submit a topology for execution w/o needing JDK or Storm CLI
- include Storm-side serializer along
- kill a running topology
- generate a topology graph as part of your build
The topology and the components could be implemented in a single EXE project and are executed by Storm via its multilang protocol as separate processes - one for each task/instance. Corresponding ProtoShell Storm-side library facilitates Protobuf serialization, which improve throughput of FsShelter topologies as compared to standard JSON. See samples to learn how to bundle the assemblies and a serializer for upload to Storm.
Bring your own, if you need it:
- command line parser
- custom serializer
While Storm tuples are dynamically typed and to a large extend the types are transparent to Storm itself, they are not types-less. Mistakes and inconsistencies between declared outputs and tuple consumers could easily lead to errors detectable at run-time only and may be frustrating to test, detect and fix. FsShelter introduces concept of topology schema, defined as F# discriminated union:
1: 2: 3:
where every DU case becomes a distinct stream in the topology. The fields of each DU case will become tuple fields in Storm streams.
It is often handy to define a type that's shared across streams and FsShelter supports defining cases with records:
1: 2: 3: 4: 5: 6:
It is also common to join/zip tuples from multiple streams and FsShelter supports defining cases with records adjoined:
1: 2: 3:
Other than safety of working with statically-verified schema the reason we care about structure of the tuple is because we reference them in Storm grouping definitions. FsShelter "flattens" the first immediate "layer" of the DU case so that all the fields, weither they come from the embedded record or the DU case itself, are available for grouping expressions.
Some of the flexibility of Storm has been hidden to provide simple developer experience for authoring event-driven solutions. For exmple, FsShelter components are implemeted as simple functions:
The async body of a spout is expected to return an option if there's a tuple to emit or None if there's nothing to emit at this time.
Bolts can get a tuple on any number of streams, and so we pattern match:
1: 2: 3: 4: 5: 6: 7: 8:
The bolt can also emit at any time, and we can hold on to the passed emit function (with caveates). Also, there can be as many arguments for the component functions as needed, the specifics will be determined when the components are put together in a topology.
1: 2: 3: 4: 5: 6: 7:
Storm topology is a graph of spouts and bolts connected via streams. FsShelter provides an embedded DSL for defining the topologies, which allows for mix and match of native Java, external shell and FsShell components:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26:
Storm will start (a copy of) the same EXE for every component instance in the topology and will assign each instance a task it supposed to execute.
The topology can be packaged with all its dependecies and submitted using embedded Nimbus client, see the examples for details.
Once the number of components grows beyond trivial it is often handy to be able to visualize them and FsShelter includes a simple way to export the topology into a graph:
See the samples included for further details.
WordCount contains a "unreliable" spout example - emitted tuples do not require ack, could be lost in case of failure.
Guaranteed contains a "reliable" spout example - emitted tuples have unique ID and require ack.
API Reference contains automatically generated documentation for public types, modules and functions in the library.
$ docker run --name fsshelter-samples -d -p 8080:8080 prolucid/fsshelter-samples
The project is hosted on GitHub where you can report issues, fork the project and submit pull requests. If you're adding a new public API, please also consider adding samples that can be turned into a documentation. You might also want to read the library design notes to understand how it works.
The library is available under Apache 2.0 license, which allows modification and redistribution for both commercial and non-commercial purposes. For more information see the License file in the GitHub repository.
| Original of int
| Incremented of int
Full name: Index.BasicSchema
val int : value:'T -> int (requires member op_Explicit)
Full name: Microsoft.FSharp.Core.Operators.int
type int = int32
Full name: Microsoft.FSharp.Core.int
type int<'Measure> = int
Full name: Microsoft.FSharp.Core.int<_>
Full name: Index.Number
val string : value:'T -> string
Full name: Microsoft.FSharp.Core.Operators.string
type string = String
Full name: Microsoft.FSharp.Core.string
| Original of int
| Described of Number
| Translated of Number
Full name: Index.RecordSchema
| Original of Number
| Doubled of Number * Number
Full name: Index.RecordsSchema
Full name: Index.numbers
Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.async
Full name: Index.addOne
Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.failwithf
Full name: Index.logResult
Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.sprintf
Full name: Index.source
type Random =
new : unit -> Random + 1 overload
member Next : unit -> int + 2 overloads
member NextBytes : buffer:byte -> unit
member NextDouble : unit -> float
Full name: System.Random
Random() : unit
Random(Seed: int) : unit
Random.Next(maxValue: int) : int
Random.Next(minValue: int, maxValue: int) : int
Full name: Index.sampleTopology
Full name: FsShelter.DSL.topology
Full name: Microsoft.FSharp.Core.Operators.log
Full name: FsShelter.DSL.runBolt
Full name: FsShelter.DSL.withParallelism
| Trace = 0
| Debug = 1
| Info = 2
| Warn = 3
| Error = 4
Full name: FsShelter.Multilang.LogLevel
static member on : case:Expr<('a0 -> 't)> -> (bool -> ComponentId -> ComponentId -> Stream<'t>)
Full name: FsShelter.DSL.shuffle
Full name: FsShelter.DotGraph.writeToConsole