  How to get insights from relational data streams in a timely manner is a hot
research topic. Data streams can present unique challenges, such as
distribution drifts, outliers, emerging classes, and changing features, which
have recently been described as open environment challenges for machine
learning. While existing studies have been done on incremental learning for
data streams, their evaluations are mostly conducted with synthetic datasets.
Thus, a natural question is how those open environment challenges look like and
how existing incremental learning algorithms perform on real-world relational
data streams. To fill this gap, we develop an Open Environment Benchmark named
OEBench to evaluate open environment challenges in real-world relational data
streams. Specifically, we investigate 55 real-world relational data streams and
establish that open environment scenarios are indeed widespread, which presents
significant challenges for stream learning algorithms. Through benchmarks with
existing incremental learning algorithms, we find that increased data quantity
may not consistently enhance the model accuracy when applied in open
environment scenarios, where machine learning models can be significantly
compromised by missing values, distribution drifts, or anomalies in real-world
data streams. The current techniques are insufficient in effectively mitigating
these challenges brought by open environments. More researches are needed to
address real-world open environment challenges. All datasets and code are
open-sourced in https://github.com/sjtudyq/OEBench.
