# Data Engineering

- Page: https://shared.btbytes.com/p/data-engineering
- Date: 2018-02-20
- Tags: data-engineering

## News

* <date>2022-10-24</date>: [The Evolution of The Data Engineer: A Look at The Past, Present & Future | Airbyte](https://airbyte.com/blog/data-engineering-past-present-and-future); via [HN](https://news.ycombinator.com/item?id=33317126)

* <date>2020-12-14</date>: [What Is Data Engineering and Is It Right for You? – Real Python](https://realpython.com/python-data-engineer/)
* <date>2024-02-07</date>: [A tale of three data platforms](https://www.fivetran.com/blog/a-tale-of-three-data-platforms) -- <q>The convergent evolution of BigQuery, Snowflake and Databricks demonstrates a broader tendency in the development of cloud platforms</q>


## Software and Libraries

- [Weld](http://web.archive.org/web/20170331032535/http://weld.stanford.edu/) is a runtime for improving the performance of data-intensive applications. It optimizes across libraries and functions by expressing the core computations in libraries using a small common intermediate representation, similar to CUDA and OpenCL.
- [Naiad](https://dl.acm.org/citation.cfm?id=2522738) is a distributed system for executing data parallel, cyclic dataflow programs. It offers the high throughput of batch processors, the low latency of stream processors, and the ability to perform iterative and incremental computations.
- [frankmcsherry/timely-dataflow: A modular implementation of timely dataflow in Rust](https://github.com/frankmcsherry/timely-dataflow)
- [DataFusion: Big Data Platform for Rust](https://www.datafusion.rs/)

## Articles

- [Take the tools out of 'Data', but don't take the data out of the tools](https://csvbase.com/blog/1)

## Books

- [📖 Data Engineering Design Patterns (DEDP)](https://www.dedp.online/); WIP; Dec 2023. The [discussion at HN](https://news.ycombinator.com/item?id=38557715) around how the definition of *data engineer* has changed is an interesting one.
- [Data Engineering With Rust](https://datawithrust.com/)
- [Data Engineering Cookbook](https://github.com/andkret/Cookbook)


## Courses

[Data Engineering Concentration - Data Science, MS | University of San Francisco](https://www.usfca.edu/arts-sciences/programs/graduate/data-science/data-engineering-concentration)