Interface TStream<T>
-
- Type Parameters:
T
- Tuple type.
- All Superinterfaces:
- TopologyElement
public interface TStream<T> extends TopologyElement
ATStream
is a declaration of a continuous sequence of tuples. A connected topology of streams and functional transformations is built usingTopology
.
Generic methods on this interface provide the ability tofilter
,map (or transform)
orsink
this declared stream using a function.TStream
is not a runtime representation of a stream, it is a declaration used in building a topology. The actual runtime stream is created once the topology issubmitted
to a runtime.
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method and Description TStream<java.lang.String>
asString()
Convert this stream to a stream ofString
tuples by callingtoString()
on each tuple.TStream<T>
filter(Predicate<T> predicate)
Declare a new stream that filters tuples from this stream.<U> TStream<U>
flatMap(Function<T,java.lang.Iterable<U>> mapper)
Declare a new stream that maps tuples from this stream into one or more (or zero) tuples of a different typeU
.java.util.Set<java.lang.String>
getTags()
Returns the set of tags associated with this stream.<K> TWindow<T,K>
last(int count, Function<T,K> keyFunction)
Declare a partitioned window that continually represents the lastcount
tuples on this stream for each partition.<K> TWindow<T,K>
last(long time, java.util.concurrent.TimeUnit unit, Function<T,K> keyFunction)
Declare a partitioned window that continually represents the lasttime
seconds of tuples on this stream for each partition.<U> TStream<U>
map(Function<T,U> mapper)
Declare a new stream that maps (or transforms) each tuple from this stream into one (or zero) tuple of a different typeU
.TStream<T>
modify(UnaryOperator<T> modifier)
Declare a new stream that modifies each tuple from this stream into one (or zero) tuple of the same typeT
.TStream<T>
peek(Consumer<T> peeker)
Declare a stream that contains the same contents as this stream while peeking at each element usingpeeker
.<U> TStream<U>
pipe(Pipe<T,U> pipe)
Declare a stream that contains the output of the specifiedPipe
oplet applied to this stream.TSink<T>
print()
Utility method to print the contents of this stream toSystem.out
at runtime.TSink<T>
sink(Consumer<T> sinker)
Sink (terminate) this stream using a function.TSink<T>
sink(Sink<T> oplet)
Sink (terminate) this stream using a oplet.java.util.List<TStream<T>>
split(int n, ToIntFunction<T> splitter)
Split a stream's tuples amongn
streams as specified bysplitter
.TStream<T>
tag(java.lang.String... values)
Adds the specified tags to the stream.TStream<T>
union(java.util.Set<TStream<T>> others)
Declare a stream that will contain all tuples from this stream and all the streams inothers
.TStream<T>
union(TStream<T> other)
Declare a stream that will contain all tuples from this stream andother
.-
Methods inherited from interface quarks.topology.TopologyElement
topology
-
-
-
-
Method Detail
-
filter
TStream<T> filter(Predicate<T> predicate)
Declare a new stream that filters tuples from this stream. Each tuplet
on this stream will appear in the returned stream iffilter.test(t)
returnstrue
. Iffilter.test(t)
returnsfalse
then thent
will not appear in the returned stream.Examples of filtering out all empty strings from stream
s
of typeString
TStream<String> s = ... TStream<String> filtered = s.filter(t -> !t.isEmpty());
- Parameters:
predicate
- Filtering logic to be executed against each tuple.- Returns:
- Filtered stream
-
map
<U> TStream<U> map(Function<T,U> mapper)
Declare a new stream that maps (or transforms) each tuple from this stream into one (or zero) tuple of a different typeU
. For each tuplet
on this stream, the returned stream will contain a tuple that is the result ofmapper.apply(t)
when the return is notnull
. Ifmapper.apply(t)
returnsnull
then no tuple is submitted to the returned stream fort
.Examples of transforming a stream containing numeric values as
String
objects into a stream ofDouble
values.// Using lambda expression TStream<String> strings = ... TStream<Double> doubles = strings.map(v -> Double.valueOf(v)); // Using method reference TStream<String> strings = ... TStream<Double> doubles = strings.map(Double::valueOf);
- Parameters:
mapper
- Mapping logic to be executed against each tuple.- Returns:
- Stream that will contain tuples of type
U
mapped from this stream's tuples.
-
flatMap
<U> TStream<U> flatMap(Function<T,java.lang.Iterable<U>> mapper)
Declare a new stream that maps tuples from this stream into one or more (or zero) tuples of a different typeU
. For each tuplet
on this stream, the returned stream will contain all non-null tuples in theIterator<U>
that is the result ofmapper.apply(t)
. Tuples will be added to the returned stream in the order the iterator returns them.
If the return is null or an empty iterator then no tuples are added to the returned stream for input tuplet
.Examples of mapping a stream containing lines of text into a stream of words split out from each line. The order of the words in the stream will match the order of the words in the lines.
TStream<String> lines = ... TStream<String> words = lines.flatMap( line -> Arrays.asList(line.split(" ")));
- Type Parameters:
U
- Type of mapped input tuples.- Parameters:
mapper
- Mapper logic to be executed against each tuple.- Returns:
- Stream that will contain tuples of type
U
mapped and flattened from this stream's tuples.
-
split
java.util.List<TStream<T>> split(int n, ToIntFunction<T> splitter)
Split a stream's tuples amongn
streams as specified bysplitter
.For each tuple on the stream,
splitter.applyAsInt(tuple)
is called. The return valuer
determines the destination stream:if r < 0 the tuple is discarded else it is sent to the stream at position (r % n) in the returned array.
Each split
TStream
is exposed by the API. The user has full control over the each stream's processing pipeline. Each stream's pipeline must be declared explicitly. Each stream can have different processing pipelines.An N-way
split()
is logically equivalent to a collection of Nfilter()
invocations, each with a predicate to select the tuples for its stream.split()
is more efficient. Each tuple is analyzed only once by a singlesplitter
instance to identify the destination stream. For example, these are logically equivalent:List<TStream<String>> streams = stream.split(2, tuple -> tuple.length()); TStream<String> stream0 = stream.filter(tuple -> (tuple.length() % 2) == 0); TStream<String> stream1 = stream.filter(tuple -> (tuple.length() % 2) == 1);
Example of splitting a stream of log records by their level attribute:
TStream<LogRecord> lrs = ... List<<TStream<LogRecord>> splits = lrr.split(3, lr -> { if (SEVERE.equals(lr.getLevel())) return 0; else if (WARNING.equals(lr.getLevel())) return 1; else return 2; }); splits.get(0). ... // SEVERE log record processing pipeline splits.get(1). ... // WARNING log record processing pipeline splits.get(2). ... // "other" log record processing pipeline
- Parameters:
n
- the number of output streamssplitter
- the splitter function- Returns:
- List of
n
streams - Throws:
java.lang.IllegalArgumentException
- ifn <= 0
-
peek
TStream<T> peek(Consumer<T> peeker)
Declare a stream that contains the same contents as this stream while peeking at each element usingpeeker
.
For each tuplet
on this stream,peeker.accept(t)
will be called.- Parameters:
peeker
- Function to be called for each tuple.- Returns:
this
-
sink
TSink<T> sink(Consumer<T> sinker)
Sink (terminate) this stream using a function. For each tuplet
on this streamsinker.accept(t)
will be called. This is typically used to send information to external systems, such as databases or dashboards.If
sinker
implementsAutoCloseable
, itsclose()
method will be called when the topology's execution is terminated.Example of terminating a stream of
String
tuples by printing them toSystem.out
.TStream<String> values = ... values.sink(t -> System.out.println(tuple));
- Parameters:
sinker
- Logic to be executed against each tuple on this stream.- Returns:
- sink element representing termination of this stream.
-
sink
TSink<T> sink(Sink<T> oplet)
Sink (terminate) this stream using a oplet. This provides a richer api for a sink thansink(Consumer)
with a full life-cycle of the oplet as well as easy access toruntime services
.- Parameters:
oplet
- Oplet processes each tuple without producing output.- Returns:
- sink element representing termination of this stream.
-
pipe
<U> TStream<U> pipe(Pipe<T,U> pipe)
Declare a stream that contains the output of the specifiedPipe
oplet applied to this stream.- Type Parameters:
U
- Tuple type of the returned stream.- Parameters:
pipe
- ThePipe
oplet.- Returns:
- Declared stream that contains the tuples emitted by the pipe oplet.
-
modify
TStream<T> modify(UnaryOperator<T> modifier)
Declare a new stream that modifies each tuple from this stream into one (or zero) tuple of the same typeT
. For each tuplet
on this stream, the returned stream will contain a tuple that is the result ofmodifier.apply(t)
when the return is notnull
. The function may return the same reference as its inputt
or a different object of the same type. Ifmodifier.apply(t)
returnsnull
then no tuple is submitted to the returned stream fort
.Example of modifying a stream
String
values by adding the suffix 'extra
'.TStream<String> strings = ... TStream<String> modifiedStrings = strings.modify(t -> t.concat("extra"));
This method is equivalent to
map(Function<T,T> modifier
).- Parameters:
modifier
- Modifier logic to be executed against each tuple.- Returns:
- Stream that will contain tuples of type
T
modified from this stream's tuples.
-
asString
TStream<java.lang.String> asString()
Convert this stream to a stream ofString
tuples by callingtoString()
on each tuple. This is equivalent tomap(Object::toString)
.- Returns:
- Declared stream that will contain each the string representation of each tuple on this stream.
-
print
TSink<T> print()
Utility method to print the contents of this stream toSystem.out
at runtime. Each tuple is printed usingSystem.out.println(tuple)
.- Returns:
TSink
for the sink processing.
-
last
<K> TWindow<T,K> last(int count, Function<T,K> keyFunction)
Declare a partitioned window that continually represents the lastcount
tuples on this stream for each partition. Each partition independently maintains the lastcount
tuples for each key seen on this stream. If no tuples have been seen on the stream for a key then the corresponding partition will be empty.
The window is partitioned by each tuple's key, obtained bykeyFunction
. For each tuple on the streamkeyFunction.apply(tuple)
is called and the returned value is the tuple's key. For any two tuplesta,tb
in a partitionkeyFunction.apply(ta).equals(keyFunction.apply(tb))
is true.
The key function must return keys that implementequals()
andhashCode()
correctly.To create a window partitioned using the tuple as the key use
identity()
as the key function.To create an unpartitioned window use a key function that returns a constant, by convention
unpartitioned()
is recommended.- Type Parameters:
K
- Key type.- Parameters:
count
- Number of tuples to maintain in each partition.keyFunction
- Function that defines the key for each tuple.- Returns:
- Window on this stream representing the last
count
tuples for each partition.
-
last
<K> TWindow<T,K> last(long time, java.util.concurrent.TimeUnit unit, Function<T,K> keyFunction)
Declare a partitioned window that continually represents the lasttime
seconds of tuples on this stream for each partition. If no tuples have been seen on the stream for a key in the lasttime
seconds then the partition will be empty. Each partition independently maintains the lastcount
tuples for each key seen on this stream.
The window is partitioned by each tuple's key, obtained bykeyFunction
. For each tuple on the streamkeyFunction.apply(tuple)
is called and the returned value is the tuple's key. For any two tuplesta,tb
in a partitionkeyFunction.apply(ta).equals(keyFunction.apply(tb))
is true.
The key function must return keys that implementequals()
andhashCode()
correctly.To create a window partitioned using the tuple as the key use
identity()
as the key function.To create an unpartitioned window use a key function that returns a constant, by convention
unpartitioned()
is recommended.- Type Parameters:
K
- Key type.- Parameters:
time
- Time to retain a tuple in a partition.unit
- Unit fortime
.keyFunction
- Function that defines the key for each tuple.- Returns:
- Partitioned window on this stream representing the last
count
tuple.
-
union
TStream<T> union(TStream<T> other)
Declare a stream that will contain all tuples from this stream andother
. A stream cannot be unioned with itself, in this casethis
will be returned.- Parameters:
other
-- Returns:
- A stream that is the union of
this
andother
.
-
union
TStream<T> union(java.util.Set<TStream<T>> others)
Declare a stream that will contain all tuples from this stream and all the streams inothers
. A stream cannot be unioned with itself, in this case the union will only contain tuples from this stream once. Ifothers
is empty or only containsthis
thenthis
is returned.- Parameters:
others
- Stream to union with this stream.- Returns:
- A stream that is the union of
this
andothers
.
-
tag
TStream<T> tag(java.lang.String... values)
Adds the specified tags to the stream. Adding the same tag to a stream multiple times will not change the result beyond the initial application.- Parameters:
values
- Tag values.- Returns:
- The tagged stream.
-
getTags
java.util.Set<java.lang.String> getTags()
Returns the set of tags associated with this stream.- Returns:
- set of tags
-
-