Archive

The PEP-723 lays out the intentions and how the metadata that can be embedded into python scripts to assist launchers, IDEs etc to interact with the script. uv was the first tool (to my knowledge) that brought this into wide attention.

Since then many python ecosystem tools have started to leverage this.

• Marimo -- Serializing package requirements in marimo notebooks
• nanodjango uses this illustrate how a single file can used to run a web application with the help of PEP-723 inline script metadata.

Scikit-learn provides libraries for - classification, regression, clustering, dimensionality reduction, model selection, preprocessing etc.

Scitkit-learn can be (should be) installed into a python virtualenv with pip install -U scikit-learn (the -U flag stands for "upgrade"). This installation in addition may requireother popular python data science libraries like numpy, pandas, matplotlib, cython, polars, .. etc for full functionality. The basic install only installed the following.

# 2024-08-27
$ pip freeze
joblib==1.4.2
numpy==2.1.0
scikit-learn==1.5.1
scipy==1.14.1
threadpoolctl==3.5.0

See also: pytorch, and tensorflow

Django from first principles

Static site generator for Django - Django Distill

While making https://thesummarist.net, I came across https://django-distill.com, which is THE BEST way to create static sites, bar none. You just make a standard, dynamic Django site, then you run a command and boom! It's static. I love it. -- via Stravos

Django SQLite Production Config

# yourproject/settings.py
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "OPTIONS": {
            "transaction_mode": "IMMEDIATE",
            "timeout": 5,  # seconds
            "init_command": """
                PRAGMA journal_mode=WAL;
                PRAGMA synchronous=NORMAL;
                PRAGMA mmap_size = 134217728;
                PRAGMA journal_size_limit = 27103364;
                PRAGMA cache_size=2000;
            """,
        },
    }
}

Write complete web applications in Python without writing javascript. There are a large class of problems for which this is an excellent solution, without having to dip into the javascript tarpit.

The advantage of something like this over using templates is that you can think of "widgets" vs "templates".

Some frameworks are:

Reflex (formerly known as pynecone). Uses FastAPI + React.

NiceGUI. Uses FastAPI + Vue.js + Quasar, tailwind.

Pydantic/FastUI. Uses React. still early in development.

Gradio -- "Build & Share Delightful Machine Learning Apps"

(Moritz et al. 2018) PDF

Ray unifies task-parallel and actor programming models in a single dynamic task graph and employs a scalable architecture enabled by the global control store and a bottom-up distributed scheduler. The programming flexibility, high throughput, and low latencies simultaneously achieved by this architecture is particularly important for emerging artificial intelligence work- loads, which produce tasks diverse in their resource requirements, duration, and functionality.

Learning Ray by Max Pumperla, Edward Oakes, and Richard Liaw. ( PDF ).

In short, Ray sets up and manages clusters of computers so that you can run distributed tasks on them. A Ray Cluster consists of nodes that are connected to each other via a network. You program against the so-called driver, the program root, which lives on the head node. The driver can run jobs, a collection of tasks, that are run on the nodes in the cluster. Specifically, the individual tasks of a job are run on worker processes on worker nodes.

This cluster can also be local, on your laptop. The default number of worker processes is the number of CPUs available on your machine.

Github - https://github.com/ray-project/ray

$ pip install "ray[rllib, serve, tune]==2.10.0"

import ray
ray.init()

Libraries:

• data processing -- apache-arrow
• model training -- TensorFlow, pytorch
• hyperparameter tuning
• model serving

News:

• Amazon’s Exabyte-Scale Migration from Apache Spark to Ray on Amazon EC2 | AWS Open Source Blog

References

• Moritz, Nishihara, Wang, Tumanov, Liaw, Liang, Elibol, Yang, Paul, Jordan, Stoica (2018). Ray: A Distributed Framework for Emerging AI Applications.