Rolling window from time-series
Date : March 29 2020, 07:55 AM
will help you I am wondering if there is a simple and neat way to create a rolling window representation from a time-series data, using Pandas (etc)? , How about: pd.DataFrame({i: x.shift(-i) for i in range(5)}).dropna()
|
Time series with scala and spark. Rolling window
Tag : scala , By : Thomas Plunkett
Date : March 29 2020, 07:55 AM
help you fix your problem This is a perfect application for window-functions. By using rangeBetween you can set your sliding window to 20s. Note that in the example below no partitioning is specified (no partitionBy). Without a partitioning, this code will not scale: import ss.implicits._
val df = Seq(
(225, 1.5),
(245, 0.5),
(300, 2.4),
(319, 1.2),
(320, 4.6)
).toDF("seconds", "value")
val window = Window.orderBy($"seconds").rangeBetween(-20L, 0L) // add partitioning here
df
.withColumn("num_row_in_window", sum(lit(1)).over(window))
.withColumn("sum_values_in_window", sum($"value").over(window))
.show()
+-------+-----+-----------------+--------------------+
|seconds|value|num_row_in_window|sum_values_in_window|
+-------+-----+-----------------+--------------------+
| 225| 1.5| 1| 1.5|
| 245| 0.5| 2| 2.0|
| 300| 2.4| 1| 2.4|
| 319| 1.2| 2| 3.6|
| 320| 4.6| 3| 8.2|
+-------+-----+-----------------+--------------------+
|
Rolling window over irregular time series
Date : March 29 2020, 07:55 AM
|
optimized rolling functions on irregular time series with time-based window
Date : March 29 2020, 07:55 AM
|
R: Compute a rolling sum on irregular time series grouped by id variables with time-based window
Date : March 29 2020, 07:55 AM
|