Quarks API
Package	Description
quarks.execution	Execution of Quarks topologies and graphs.
quarks.execution.mbeans	Management MBeans for execution.
quarks.execution.services	Execution services.
quarks.function	Functional interfaces for lambda expressions.
quarks.topology	Functional api to build a streaming topology.
quarks.topology.json	Utilities for use of JSON in a streaming topology.
quarks.topology.plumbing	Plumbing for a streaming topology.
quarks.topology.tester	Testing for a streaming topology.

Quarks Providers
Package	Description
quarks.providers.development	Execution of a streaming topology in a development environment .
quarks.providers.direct	Direct execution of a streaming topology.

Quarks Connectors
Package	Description
quarks.connectors.file	File stream connector.
quarks.connectors.http	HTTP stream connector.
quarks.connectors.iot	Quarks device connector API to a message hub.
quarks.connectors.iotf	IBM Watson IoT Platform stream connector.
quarks.connectors.jdbc	JDBC based database stream connector.
quarks.connectors.kafka	Apache Kafka enterprise messing hub stream connector.
quarks.connectors.mqtt	MQTT (lightweight messaging protocol for small sensors and mobile devices) stream connector.
quarks.connectors.mqtt.iot	An MQTT based IotDevice connector.
quarks.connectors.pubsub	Publish subscribe model between jobs.
quarks.connectors.pubsub.oplets	Oplets supporting publish subscribe service.
quarks.connectors.pubsub.service	Publish subscribe service.
quarks.connectors.serial	Serial port connector API.
quarks.connectors.wsclient	WebSocket Client Connector API for sending and receiving messages to a WebSocket Server.
quarks.connectors.wsclient.javax.websocket	WebSocket Client Connector for sending and receiving messages to a WebSocket Server.
quarks.connectors.wsclient.javax.websocket.runtime

Quarks Samples
Package	Description
quarks.samples.apps	Support for some more complex Quarks application samples.
quarks.samples.apps.mqtt	Base support for Quarks MQTT based application samples.
quarks.samples.apps.sensorAnalytics	The Sensor Analytics sample application demonstrates some common continuous sensor analytic application themes.
quarks.samples.connectors	General support for connector samples.
quarks.samples.connectors.file	Samples showing use of the File stream connector.
quarks.samples.connectors.iotf	Samples showing device events and commands with IBM Watson IoT Platform.
quarks.samples.connectors.jdbc	Samples showing use of the JDBC stream connector.
quarks.samples.connectors.kafka	Samples showing use of the Apache Kafka stream connector.
quarks.samples.connectors.mqtt	Samples showing use of the MQTT stream connector.
quarks.samples.console	Samples showing use of the Console web application.
quarks.samples.topology	Samples showing creating and executing basic topologies .
quarks.samples.utils.metrics
quarks.samples.utils.sensor

Quarks Analytics
Package	Description
quarks.analytics.math3.json	JSON analytics using Apache Commons Math.
quarks.analytics.math3.stat	Statistical algorithms using Apache Commons Math.
quarks.analytics.sensors	Analytics focused on handling sensor data.

Quarks Utilities
Package	Description
quarks.metrics	Metric utility methods, oplets, and reporters which allow an application to expose metric values, for example via JMX.
quarks.metrics.oplets

Quarks Low-Level API
Package	Description
quarks.graph	Low-level graph building API.
quarks.oplet	Oplets API.
quarks.oplet.core	Core primitive oplets.
quarks.oplet.core.mbeans	Management beans for core oplets.
quarks.oplet.functional	Oplets that process tuples using functions.
quarks.oplet.plumbing	Oplets that control the flow of tuples.
quarks.oplet.window	Oplets using windows.
quarks.window	Window API.

Other Packages
Package	Description
quarks.javax.websocket	Support for working around JSR356 limitations for SSL client container/sockets.
quarks.javax.websocket.impl	Support for working around JSR356 limitations for SSL client container/sockets.
quarks.runtime.etiao	A runtime for executing a Quarks streaming topology, designed as an embeddable library so that it can be executed in a simple Java application.
quarks.runtime.etiao.graph
quarks.runtime.etiao.graph.model
quarks.runtime.etiao.mbeans
quarks.runtime.jmxcontrol
quarks.runtime.jsoncontrol
quarks.test.svt

Quarks provides an programming model and runtime for executing streaming analytics at the edge

Quarks v0.4

Overview
Programming Model
Getting Started

Overview

Quarks provides an programming model and runtime for executing streaming analytics at the edge. Quarks is focusing on two edge cases:

Internet of Things (IoT) - Widely distributed and/or mobile devices.
Enterprise Embedded - Edge analytics within an enterprise, such as local analytic applications of eash system in a machine room, or error log analytics in application servers.

In both cases Quarks applications analyze live data and send results of that analytics and/or data intermittently to back-end systems for deeper analysis. A Quarks application can use analytics to decide when to send information to back-end systems, such as when the behaviour of the system is outside normal parameters (e.g. an engine running too hot).
Quarks applications do not send data continually to back-end systems as the cost of communication may be high (e.g. cellular networks) or bandwidth may be limited.

Quarks applications communicate with back-end systems through some form of message hub as there may be millions of edge devices. Quarks supports these message hubs:

MQTT - Messaging standard for IoT
IBM Watson IoT Platform - Cloud based service providing a device model on top of MQTT
Apache Kafka - Enterprise message bus

Back-end analytic systems are used to perform analysis on information from Quarks applications that cannot be performed at the edge. Such analysis may be:

Running complex analytic algorithms than require more resources (cpu, memory etc.) than are available at the edge.
Maintaining more state per device that can exist at the edge, e.g. hours of state for patients' medical sensors.
Correlating device information with multiple data sources:

Weather data
Social media data
Data of record (e.g patients' medical histories, trucking manifests).
Other devices
etc.

Back-end systems can interact or control devices based upon their analytics, by sending commands to specific devices, e.g. reduce maximum engine revs to reduce chance of failure before the next scheduled service, or send an alert of an accident ahead.

Programming Model

Quarks applications are streaming applications in which each tuple (data item or event) in a stream of data is processed as it occurs. Additionally, you can process windows (logical subsets) of data. For example, you could analyze the last 90 seconds of data from a sensor to identify trends in the data

Topology functional API

Overview

The primary api is Topology which uses a functional model to build a topology of streams for an application.
TStream is a declaration of a stream of tuples, an application will create streams that source data (e.g. sensor readings) and then apply functions that transform those streams into derived streams, for example simply filtering a stream containg engine temperator readings to a derived stream that only contains readings thar are greater than 100°C.
An application terminates processing for a stream by sinking it. Sinking effectively terminates a stream by applying processing to each tuple on the stream (as it occurs) that does not produce a result. Typically this sinking is transmitting the tuple to an external system, for example the messgae hub to send the data to a back-end system, or locally sending the data to a user interface.

This programming style is typical for streaming systems and similar APIs are supported by systems such as Apache Flink, Apache Spark Streaming, IBM Streams and Java 8 streams.

Functions

Quarks supports Java 8 and it is encouraged to use Java 8 as functions can be easily and clearly written using lambda expressions.

Arbitrary Topology

Simple applications may just be a pipeline of streams, for example, logically:
source --> filter --> transform --> aggregate --> send to MQTT
However Quarks allows arbitrary topologies including:

Multiple source streams in an application
Multiple sinks in an application
Multiple processing including sinks against a stream (fan-out)
Union of streams (fan-in)
Correlation of streams by allowing streams to be joined (to be added)

Graph API

Overview

The graph API is a lower-level API that the topology api is built on top of. A graph consists of oplet invocations connected by streams. The oplet invocations contain the processing applied to each tuple on streams connected to their input ports. Processing by the oplet submits tuples to its output ports for subsequent processing by downstream connected oplet invocations.

Getting Started

Below, <quarks-target> refers to a Quarks release's platform target directory such as .../quarks/java8.

A number of sample Java applications are provided that demonstrate use of Quarks.
The Java code for the samples is under <quarks-target>/samples.

Shell scripts to run the samples are <quarks-target>/scripts. See the README there.

Summary of samples:

Sample	Description	Focus
`HelloWorld`	Prints Hello World! to standard output.	Basic mechanics of declaring a topology and executing it.
`PeriodicSource`	Polls a random number generator for a new value every second and then prints out the raw value and a filtered and transformed stream.	Polling of a data value to create a source stream.
`SensorsAggregates`	Demonstrates partitioned aggregation and filtering of simulated sensors that are bursty in nature, so that only intermittently is the data output to `System.out`	Simulated sensors with windowed aggregation
`SimpleFilterTransform`
File	Write a stream of tuples to files. Watch a directory for new files and create a stream of tuples from the file contents.	Use of the File stream connector
IotfSensors, IotfQuickstart	Sends simulated sensor readings to an IBM Watson IoT Platform instance as device events.	Use of the IBM Watson IoT Platform connector to send device events and receive device commands.
JDBC	Write a stream of tuples to an Apache Derby database table. Create a stream of tuples by reading a table.	Use of the JDBC stream connector
Kafka	Publish a stream of tuples to a Kafka topic. Create a stream of tuples by subscribing to a topic and receiving messages from it.	Use of the Kafka stream connector
MQTT	Publish a stream of tuples to a MQTT topic. Create a stream of tuples by subscribing to a topic and receiving messages from it.	Use of the MQTT stream connector
SensorAnalytics	Demonstrates a Sensor Analytics application that includes: configuration control, a device of one or more sensors and some typical analytics, use of MQTT for publishing results and receiving commands, local results logging, conditional stream tracing.	A more complete sample application demonstrating common themes.

Other samples are also provided but have not yet been fully documented. Feel free to explore them.

Building Applications

You need to include one or more Quarks jars in your classpath depending on what features your application uses.

Include one or both of the following:

<quarks-target>/lib/quarks.providers.direct.jar - if you use the DirectProvider
<quarks-target>/lib/quarks.providers.development.jar - if you use the DevelopmentProvider

Include the jar of any Quarks connector you use:

<quarks-target>/connectors/file/lib/quarks.connectors.file.jar
<quarks-target>/connectors/jdbc/lib/quarks.connectors.jdbc.jar
<quarks-target>/connectors/iotf/lib/quarks.connectors.iotf.jar
<quarks-target>/connectors/kafka/lib/quarks.connectors.kafka.jar
<quarks-target>/connectors/mqtt/lib/quarks.connectors.mqtt.jar
<quarks-target>/connectors/wsclient-javax.websocket/lib/quarks.connectors.wsclient.javax.websocket.jar [*]

[*] You also need to include a javax.websocket client implementation if you use the wsclient connector. Include the following to use an Eclipse Jetty based implementation:

<quarks-target>/connectors/javax.websocket-client/lib/javax.websocket-client.jar

Include jars for any Quarks utility features you use:

<quarks-target>/utils/metrics/lib/quarks.utils.metrics.jar - for the quarks.metrics package

Quarks uses slf4j for logging, leaving the decision of the actual logging implementation to your application (e.g., java.util.logging or log4j). For java.util.logging you can include:

<quarks-target>/ext/slf4j-1.7.12/slf4j-jdk-1.7.12.jar