# Composable Data Systems

- Page: https://shared.btbytes.com/p/composable-data-systems
- Date: 2024-07-16
- Tags: data-engineering

[Wes McKinney - The Road to Composable Data Systems: Thoughts on the Last 15 Years and the Future](https://wesmckinney.com/blog/looking-back-15-years/);  ([Pedreira et al. 2023](https://www.vldb.org/pvldb/vol16/p2679-pedreira.pdf)) [PDF](https://www.vldb.org/pvldb/vol16/p2679-pedreira.pdf)

The stack proposed by this article:

- [apache-arrow](https://shared.btbytes.com/p/apache-arrow) -- language-independent compute and data interchange layer and supporting systems infrastructure across important programming languages
- ibis and dplyr --  backend-agnostic data frame interfaces
- RAPIDS -- GPU-accelerated libraries for data analytics and machine learning
- [duckdb](https://shared.btbytes.com/p/duckdb) and velox --  embeddable systems providing fast columnar query processing
- Substrait -- - language-independent intermediate representation (IR) middleware for analytical computing to assist in decoupling user interfaces from compute engines

## References

- Pedreira, Erling, Karanasos, Schneider, McKinney, Valluri, Zait, Nadeau (2023). [The Composable Data Management System Manifesto](https://www.vldb.org/pvldb/vol16/p2679-pedreira.pdf).