here you goThings I’ve made, read and saved — shared here.

Archive

9,621 entries: 57 from X, 8,925 from Pinboard, saved 1998 to 2026.

Browse by tag, site, author, folder or language

Tags

Sites

Authors

Folders

Languages

Showing 351–400 of 547Folder notes ×

2023

Apache Arrow

Apache Arrow Project: https://arrow.apache.org/

Sub Projects

See also

Articles

Apache Arrow Uses
Apache Arrow Uses
[via](https://x.com/Dipankartnt/status/1773903690371326436).

notes#analytics#arrow#bigdata

Processing

Home page: Processing.org

The Nature of Code (A CC by NC licensed #book) uses processing.

an online resource and print book exploring the unpredictable evolutionary and emergent properties of nature in software via the creative coding framework Processing. 2024 version with a new and rebooted version of this book built around JavaScript and the p5.js library. The book has a few new coding tricks this time, but it’s the same old nature—birds still flap their wings, and apples still fall on our heads.

Welcome to py5! is a new version of Processing for Python 3.9+. The goal of py5 is to create a version of Processing that is integrated into the Python ecosystem. Built into the library are thoughtful choices about how to best get py5 to work with other popular Python libraries and tools such as Jupyter, numpy, shapely, trimesh,matplotlib, and Pillow. needs Java 17

notes#processing#visualization

htmx

Home page -- </> htmx - high power tools for html

Papers

[@10.1145/3648188.3675127]; PDF

Articles

Presentations

Videos

Resources

Related

See Also

Golang

Python

Django

Elixir

  • phoenix liveview

Ruby

Javascript

From the creator of Ahastack = Astro.js + HTML + Alpine.js

Every feature I add to a new webapp I'm building with Astro + Alpine + htmx + Tailwind + Pocketbase just brings me joy in how simple this stuff can really be. It's delightful. via

Plainweb is a framework using HTMX, SQLite and TypeScript.

plainweb learned from and is heavily inspired by the Grug Brained Developer, SQLite, HTMX, PocketBase, Remix, Litestream and some of the Go proverbs.

dotnet

HTMX for ASP.NET Core Developers - JetBrains Guide

CSS

Franken/UI -- HTML-first, framework-agnostic implementation of shadcn/ui - franken/ui.

notes#html#htmx#javascript#web

Making Websites

What to keep in mind while making websites -- Rediscovering the Small Web - Neustadt.fr

What should a personal website be? - ratfactor

100 things you can do on your personal website | James' Coffee Blog

notes#web

Clickhouse

Introduction

Home page -- Fast Open-Source OLAP DBMS - ClickHouse

clickhouse-local is an easy-to-use version of ClickHouse that is ideal for developers who need to perform fast processing on local and remote files using SQL without having to install a full database server. With clickhouse-local, developers can use SQL commands (using the ClickHouse SQL dialect) directly from the command line, providing a simple and efficient way to access ClickHouse features without the need for a full ClickHouse installation. One of the main benefits of clickhouse-local is that it is already included when installing clickhouse-client. This means that developers can get started with clickhouse-local quickly, without the need for a complex installation process.

curl https://clickhouse.com/ | sh will download a single clickhouse binary, which can be used as clickhouse-local:

./clickhouse local -q "SELECT * FROM file('reviews.tsv', 'TabSeparated')"
# to see the inferred schema
./clickhouse local -q "DESCRIBE file('reviews.tsv')"
# a query
./clickhouse local -q "SELECT
argMax(product_title,star_rating),
max(star_rating)
FROM file('reviews.tsv')"

This can also be used to query data in a parquet file on S3 storage:

./clickhouse local -q "
SELECT count()
FROM s3('https://datasets-documentation.s3.eu-west-3.amazonaws.com/house_parquet/house_0.parquet')"

chDB

embedded SQL OLAP Engine powered by ClickHouse

The birth of chDB - auxten -- "rocket engine on a bicycle". github.

  • In-process SQL OLAP Engine, powered by ClickHouse
  • No need to install ClickHouse
  • Minimized data copy from C++ to Python with python memoryview
  • Input&Output support Parquet, CSV, JSON, Arrow, ORC and 60+more formats, samples
  • Support Python DB API 2.0, example
  • pip install chdb
query = "SELECT distinct city from 'employees.csv' "
res = chdb.query(query, "CSV")
print(res)

See Trying chDB, an embeddable ClickHouse engine for more code examples.

TODO: What's the comparison to duckdb and where would one shine over the other?

Comparisons

[ClickHouse or StarRocks? Here is a DETAILED Comparison | by Tianyi Wang | Dec, 2021 | Medium]; (https://archive.is/llUmZ) (archive).

Articles

Papers

ClickHouse - Lightning Fast Analytics for Everyone ( PDF ) (2024). Youtube Talk.

This paper presents an overview of ClickHouse, a popular open- source OLAP database designed for high-performance analytics over petabyte-scale data sets with high ingestion rates. Its storage layer combines a data format based on traditional log-structured merge (LSM) trees with novel techniques for continuous trans- formation (e.g. aggregation, archiving) of historical data in the background. Queries are written in a convenient SQL dialect and processed by a state-of-the-art vectorized query execution engine with optional code compilation. ClickHouse makes aggressive use of pruning techniques to avoid evaluating irrelevant data in queries. Other data management systems can be integrated at the table function, table engine, or database engine level. Real-world bench- marks demonstrate that ClickHouse is amongst the fastest analyti- cal databases on the market.

Internals

Related

ByConity is a fork of clickhouse by tiktok; it is a distributed cloud-native SQL data warehouse engine, that excels in interactive queries and Ad-Hoc queries, featuring support for querying multiple tables, cluster expansion without sensation, and unified aggregation of offline batch data and real-time data streams.

observability

Clickhouse is winning the Observability Wars July 2026 - Mat Duggan.

  • The amazing thing is that ClickHouse scales without changing shape.
  •  ClickHouse at 10 TB a day looks like ClickHouse at 1 TB a day with more shards. That's it. That's the pitch.

notes#database

typescript

Typescript was created at Microsoft.

Learn

TypeScript Deep Dive by basarat is a good tutorial for TS. It helped me understand various terminology like what what package.json, tsconfig.json are. commonjs modules, exports; setting up projects for various target - node.js vs browsers etc.

Typescript Book by Simone Poggiali.

Cloud

Val Town -- If GitHub Gists could run And AWS Lambda was fun. Val Town is a social website to write and deploy TypeScript. Build APIs and schedule functions from your browser.

notes#javascript#programming-language

pandas

Top level notes

  • Pandas Documentation Homepage
  • Installing (2.0) -- pip install pandas==2.0
    • installing pandas will also install numpy
  • Pandas helps in working with tabular data (databases, spreadsheets)
  • Helps = explore, clean and proess
  • Anything you can do with SQL you can also do with Pandas.
    • Group by operations
  • Summary Statistics. eg., mean, median, std..
  • The data table is called a DataFrame
  • pandas can read (read_ methods) from various file formats like csv, xls, parquet, hdf5, json, sql (db) and generate output (to_ methods) in the above formats.
  • matplotlib is used to generated plots from data tables.
  • split-apply-combine approach. Hmm.. map-reduce?
  • melt() to convert from wide -> long/tidy form and pivot() to convert from long to wide format.
  • combine multiple tables using concat() function for column-wise or row-wise joining
  • for database like joining/merging, use the merge function. pd.merge(tbone, tbtwo, how="left", left_on="col1", right_on="col2"). also merge_asof
  • pandas supports inner, outer, right and left joins.
  • use shape() to find the rowxcolumn shape.
  • TODO: how vectorization is done in pandas
  • TODO: where numpy is faster than pandas
  • TODO: Pyspark with pandas

Quick intro

import numpy as np
import pandas as pd

# create new Series
s = pd.Seres([1,2,3, pd.nan, 8, 121])
# pd.nan = not a number

# pd.date_range("<YYYYMMDD", periods=n) produces a
# a date range starting at YYYYMMDD for the period `n`

# dataframes are created with

df = pd.DataFrame(nparray, index=<indexvec>, columns=list(names))
# the nparray is a 2-d numpy array of mxn dimensions

# the df can also be created by passing a dict with keys as the
# column names and the value as the series.

Articles

notes#python

Obsidian

Obsidian is a knowledge base that works on top of your local folder of plain text files. Think of logseq and roamresearch.

Obsidian Graph
Obsidian Graph, manually exported from the Graph View as a screenshot - 2024-04-05

How I'm using Obsidian

  • This website is orgnized using markdown
  • I open up the content directory as a vault (obsidian word).
  • I edit them as any other markdown file
  • The hotkeys and / commands are handy in writing
  • I use WikiLinks (double square brackets) to link to peer pages, these are converted to HTML links via the Python-Markdown extension. wikilinks = WikiLinkExtension(base_url="/", end_url=".html")

Templates

Creating and using templates in Obsidian.

Use case: I write my notebooks and posts using markdown and ergo obsidian. The two types have a frontmatter that looks like this (taken from this document):

---
title: Obsidian
kind: notebook
date: 2023-04-12
tags:
  - pkm
  - markdown
  - text-editor
---

Using instructions from the templates page, i created _obstemplates directory and added notebook.md as a file with the following content:

---
title: {{title}}
date: {{date}}
kind: post
description: TODO
tags:
  - TODO
---

To use this template, use the "ribbon" and select "Insert Template". voila!

Plugins

I'm using these plugins

  • dataview
  • emoji-toolbar
  • style-settings
  • periodic-notes
  • table-editor

Alfred Workflows

Obsidian published websites

Examples of interesting obsidian published websites

Templater


Related:

Introducing My Knowledge Lakehouse

notes#markdown#pkm#text-editor

Distributed Systems

News

Learning

Learning about distributed systems: where to start? from Prof Murat Demirbas.

MIT 6.824 Distributed Systems (Spring 2020) Videos and Schedule

Dan Creswell's Distributed Systems Reading List I often argue that the toughest thing about distributed systems is changing the way you think. The below is a collection of material I've found useful for motivating these changes.

Articles

Blogs

College Courses

notes#distributed-systems

SVG

TinyVG -- binary format with 10 commands and a readable 20-page spec; can render a large subset of SVG (and has some good criticisms of SVG qua format) 🐥

A Deep Dive Into SVG Path Commands If you've ever looked at the SVG code for an icon before, you might have noticed that they're usually made up of a bunch of path elements, each with a cryptic dattribute.

NN SVG -- Publication-ready NN-architecture schematics; github.

Making noisy SVGs — Daniel Immke via HN (good discussion, tangents and links)

Grainy Gradients | CSS-Tricks - CSS-Tricks

Intro - Notes on SVG parsing

Sites built using SVG

notes#graphics

ConTeXt

Context installation

2022-03-22

If you want to run ConTeXt everywhere, you need to adapt the path, like:

  export PATH=/Users/pradeep/context/tex/texmf-osx-arm64/bin:$PATH

If you run from an editor you can specify the full path to mtxrun:

  /Users/pradeep/context/tex/texmf-osx-arm64/bin/mtxrun --autogenerate --script context --autopdf ...

If you also want to run LuaTeX and MkIV you can provide --luatex but first the file
database has to be generated (this time with context and not mtxrun):

  context --luatex --generate
  context --luatex --make

The following settings were used:

  server   : lmtx.contextgarden.net,lmtx.pragma-ade.com,lmtx.pragma-ade.nl
  instance : install-lmtx
  extras   :
  ownpath  : /Users/pradeep/context
  platform : osx-arm64

notes#typesetting

Vercel

Notes

  • Vercel Markup Many of Vercel's products are built on top of another service. They usually add a significant markup that may not outweigh the convenience. This documents the markup percentage for each service based on their reported pricing, although some numbers are estimates.; via HN

My Vercel hosted pages and apps

Connects using gitlab account.

Below are not deployed using git{hub, lab} integration. direct deployment.

2023-05-01vercel announced storage including KV( redis ), postgresql, and blob powered by cloudflare r2

An alternative for the "expensive" vercel redis is upstash.

notes#vercel#webhosting

Minify HTML

wilsonzlin/minify-html : Extremely fast and smart HTML + JS + CSS minifier, available for Rust, Deno, Java, Node.js, Python, Ruby, and WASM

The Python library on pypi.

Using the Python library:

# code/example_minifyhtml.py
#!/usr/bin/env python

import minify_html

with open("output/index.html") as f:
    t = f.read()
    print(t)
    o = minify_html.minify(
        t,
        do_not_minify_doctype=True,
        keep_spaces_between_attributes=True,
        ensure_spec_compliant_unquoted_attribute_values=True,
    )
    print(o)

notes#html#python

Procastination

Why you procrastinate even when it feels bad

(below is the transcript of the above YouTube video with my highlights)

It’s 5 p.m. and you’ve just realized that report you’ve been putting off is due tomorrow. It’s time to buckle down, open your computer... and check your phone. Maybe catch up on your favorite YouTube channel? Actually, you should probably make dinner first. You usually like cooking, though it’s hard to enjoy with this work hanging over your head, and oh— it’s actually pretty late! Maybe you should just try again in the morning? This is the cycle of procrastination, and I promise you, we have all been there. But why do we keep procrastinating even when we know it’s bad for us? To be clear, putting something off isn’t always procrastinating. Responsible time management requires deciding which tasks are important and which ones can wait.

Procrastination is when we avoid a task we said we would do, for no good reason, despite expecting our behavior to bring negative consequences.&

Obviously, it’s irrational to do something you expect to harm you.

But ironically, procrastination is the result of our bodies trying to protect us, specifically by avoiding a task we see as threatening.

When you realize you need to write that report, your brain responds like it would to any incoming threat. Your amygdala, a set of neurons involved in emotional processing and threat identification, releases hormones including adrenaline that kick off a fear response. This stress-induced panic can overpower the impulses from your prefrontal cortex, which typically help you think long term and regulate your emotions. And it’s in the midst of this fight, flight, or freeze response that you decide to handle the threat by avoiding it in favor of some less stressful task.

This response might seem extreme— after all, it’s just a deadline, not a bear attack. But we’re most likely to procrastinate tasks that evoke negative feelings, were more likely to put off tasks they perceived as stressful or challenging. In one experiment, students were given reminders to study throughout the day. While they were studying, most reported that it wasn’t so bad. But when they were procrastinating, Because procrastination is motivated by our negative feelings, and those who struggle with low self-esteem are much more likely to procrastinate, regardless of how good they are at time management.

However, it's a common misconception that all procrastinators are lazy. In the body and brain, laziness is marked by no energy and general apathy. When you’re feeling lazy, you’re more likely to sit around doing nothing than distract yourself with unimportant tasks. In fact, many people procrastinate because they care too much.

Procrastinators often report a high fear of failure, putting things off because they’re afraid their work won’t live up to their high standards.

Whatever the reason for procrastination, the results are often the same. Frequent procrastinators are likely to suffer from anxiety and depression, ongoing feelings of shame, higher stress levels and physical ailments associated with high stress. Worst of all, while procrastination hurts us in the long run, it does temporarily reduce our stress level, reinforcing it as a bodily response for coping with stressful tasks. So, how can we break the cycle of procrastination? Traditionally, people thought procrastinators needed to cultivate discipline and practice strict time management. But today, many researchers feel the exact opposite.

Being too hard on yourself can layer additional bad emotions onto a task, making the threat even more intense.

To short-circuit this stress response, we need to address and reduce these negative emotions. Some simple strategies include

  • breaking a task into smaller elements
  • or journaling about why it's stressing you out and addressing those underlying concerns.
  • try removing nearby distractions that make it easy to impulsively procrastinate.
  • and more than anything, it helps to cultivate an attitude of self-compassion, forgiving yourself, and making a plan to do better next time.

Because a culture that perpetuates this cycle of stress and procrastination hurts all of us in the long term.

notes#productivity

LLM

Introductory Materials

Generative AI exists because of the transformer -- A visual story from Financial Times; Sept 2023.

Large language models, explained with a minimum of math and jargon

A High-level Overview of Large Language Models - Borealis AI

Tiny LLM - LLM Serving in a Week -- uses MLX (on MacOS) to  ... implement an LLM serving project from scratch -- with only matrix manipulations APIs, so that I can understand what it takes to load those LLM model parameters and do the math magic to generate text.

Books

Study Guides

-Mastering LLMs workshop. See Takeaways from Mastering LLMs Course from SwaroopCH.

Blogs and websites

Papers

Videos

An observation on Generalization - YouTube by Ilya Sutskever (OpenAI); Aug 14, 2023.

  • Supervised Learning - precise mathematical condition under which learning should succeed, which is - Low training error + more training data than "degrees of freedom" = low test error

Prompt Engineering

Models

See this page - Models Table – Dr Alan D. Thompson – Life Architect for a visual representation of models, and a table of various attributes of models.

Stuff you can run on your computer

Everything I've learned so far about running local LLMs; Nov 2024.

smol-ai/developer: with 100k context windows on the way, it's now feasible for every dev to have their own smol developer

How is LLaMa.cpp possible? how can we run llama.cpp on local machines when the expectation is that large models need expensive GPUS (eg: A100) to run

Introducing Code Llama, a state-of-the-art large language model for coding Code Llama is a code-specialized version of Llama 2 that was created by further training Llama 2 on its code-specific datasets, sampling more data from that same dataset for longer. Essentially, Code Llama features enhanced coding capabilities, built on top of Llama 2. It can generate code, and natural language about code, from both code and natural language prompts (e.g., “Write me a function that outputs the fibonacci sequence.”) It can also be used for code completion and debugging. It supports many of the most popular languages being used today, including Python, C++, Java, PHP, Typescript (Javascript), C#, and Bash.

Ask HN: Cheapest way to run local LLMs? | Hacker News

See also Perplexity Labs where they have multiple models to try from

 Perplexity labs models  
Perplexity labs models; Apr 2024

LLMs in your language

All languages are NOT created (tokenized) equal

Small Language Models

  • (Eldan and Li 2023)

Building LLMs

Using LLMs

Fine-tuning has one huge advantage though: it is far more effective at guiding a model's behavior than prompting, so you can often get away with a much smaller model. That gets you faster responses and lower inference costs. A fine-tuned Llama 7B model is 50x cheaper than GPT-3.5 on a per-token basis, and for many use cases can produce results that are as good or better! For example, classifying the 2M recipes at https://huggingface.co/datasets/corbt/all-recipes with GPT-4 would cost $23k. Even with GPT-3.5 it would cost over $1k. The model we fine-tuned performs similarly to GPT-4 and costs just $19 to run over the entire dataset. OpenBMB/ToolBench: An open platform for training, serving, and evaluating large language model for tool learning.

Frameworks

LlamaIndex 🦙 0.8.13

Haystack | Haystack Open-source LLM framework to build production-ready applications.

Use the latest LLMs: hosted models by OpenAI or Cohere, open-source LLMs, or other pre-trained models All tooling in one place: preprocessing, pipelines, agents & tools, prompts, evaluation and finetuning Choose your favorite database: Elasticsearch, OpenSearch, Weaviate, Pinecone, Qdrant, Milvus and more Scale to millions of documents: use Haystack's proven retrieval architecture Compare it to LangChainAI

GPT4All A free-to-use, locally running, privacy-aware chatbot. No GPU or internet required.

AI Proxy

an AI proxy that lets you use a variety of providers (OpenAI, Anthropic, LLaMa2, Mistral, and others) behind a single interface w/ caching & API key management.

MLC LLM Machine Learning Compilation for Large Language Models (MLC LLM) is a high-performance universal deployment solution that allows native deployment of any large language models with native APIs with compiler acceleration. The mission of this project is to enable everyone to develop, optimize and deploy AI models natively on everyone’s devices with ML compilation techniques.

Project Overview Project Overview of MLC LLM

Multimodal Learning

todo

Security

OWASP | Top 10 for Large Language Models

Abliteration

Uncensor any LLM with abliteration

Operational Issues

GGUF

GGUF and GGML are file formats used for storing models for inference, especially in the context of language models like GPT

Benchmarking LLMs

LLM Benchmark Report for: NousResearch/Redmond-Puffin-13B

Linkdump

2023

2024

2026

References


See also: Multimodal Learning, Multi Agent Frameworks, AI Agent Framework, generative-ai, smol-llm, transformer-math, LlamaIndex, RAG, local-llm, llm-embedding, AI SaaS, LLM Training, Reward Models, AI Code Assistants, Building LLM Based Systems, LLMs in Data Management

notes#chatgpt#deep-learning#llm

Java

Modern Java - an online book; a 2026 approach to learning Java in bite-sized lessons. Don't have to start in an IDE for one.

Internals

Articles

Interesting libraries

Permazen Permazen is a ground-up rethink of Java persistence. It has been in production use in commercial software since 2015.

 Instead of starting from the storage technology side, it starts from the programming language side, asking the simple question, "What are the issues that are inherent to persistence programming, regardless of programming language or database storage technology, and how can they be addressed at the language level in the simplest, most correct, and most language-natural way?"

notes#java#programming-language

JAX

JAX is a Python library which augments numpy and Python code with function transformations which make it trivial to perform operations common in machine learning programs. Concretely, this makes it simple to write standard Python/numpy code and immediately be able to

Compute the derivative of a function via a successor to autograd Just-in-time compile a function to run efficiently on an accelerator via XLA Automagically vectorize a function, so that e.g. you can process a “batch” of data in parallel. learn more ... "You don't know Jax" (2019)

Jax's website - google/jax: Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more

JAX and Equinox: What are they and why should I bother?

notes#deep-learning#python

Git

Learning Git

Why Git is hard Confusing git terminology; Nov 2023 by Julia Evans.

Things I wish everyone knew about Git ) and (Part II) by Mark Dominus ( plover.com )

Inside Git by jvns.

eficode-academy/git-katas, A set of exercises for deliberate Git Practice

Oh My Git, learn git through playing games.

Artisanal Handcrafted Git Repositories | drew's dev blog -- "I’m going to talk about how to handmake your git repositories without using these silly git commands. You might also learn a bit more about how git works under the hood during the process, or whatever."

So You Think You Know Git - FOSDEM 2024 - YouTube by Scott Chacon, now creator of Git Butler

USEFUL CONFIG STUFF

$ git config --global alias.staash 'stash --all'
$ git clean

https://gist.gihub.com/schacon

$ git config --global alias.bb !better-branch.sh

Conditional Configs
[includeif "gitdir:~/projects/work/"]
    path = ~/projects/work/.gitconfig
[includeif "gitdir:~/projects/oss/"]
    path = ~/projects/oss/.gitconfig

OLDIES BUT GOODIES

$ git blame -L
git blame a portion of the file

$ git log -L 15,26:path/to/file
works the same way with git log

$ git blame -w
ignore whitespace

$ git blame -w -C
.. and detect lines or copies in the same commit

$ git blame -w -C -C
.. or the commit that created the file

$ git blame -w -C -C -C
.. or any commit at all

$ git log -S files_watcher -p
the pickaxe

$ git reflog

$ git diff --word-diff

$ git config --global rerere.enabled true
REuse REcorded REsolution

NEW STUFF

$ git branch --column

$ git config --global column.ui auto
$ git config --global branch.sort -committerdate

$ git push --force-with-lease
This one is better than using --force by default

signing commits with ssh
$ git config gpg.format ssh

$ git push --signed
github and gitlab do not support it.

$ git maintenance

gc(disabled),
commit-graph and prefetch (hourly),
loose-objects incremental-repack (daily)
pack-refs (none)

BIG REPO STUFF

Half a million files in working directory

windows rpeo - 300gb, 3.5 million files
linux kernel - 4.5gb in comparison

prefetching
$ git for-each-ref | grep prefe

commit-graph
$ git config --global fetch.writeCommitGraph true

filesystem monitor
$ git config core.untrackedcache true
$ git config core.fsmonitor true

partial cloning
$ git clone --filter=blob:none
$ git clone --filter=tree:0

GITHUB STUFF

auto merge

merge queue

$ git ls-remote

"all mypull requests are actual refs"

$ git config remote.origin.fetch '+refs/pull/*:refs/remotes/origin/pull/*'

$ git fetch


GITBUTLER

virtual branches

Tools

jesseduffield/lazygit: simple terminal UI for git commands

https://gitless.com -- A Git-compatible VCS that does away with the staging area (and stashes), and instead has separate working dirs for every branch (branch = automatically stashed). They have a detailed analysis of Git that explains their design: What's wrong with Git? A conceptual design analysis #pdf

chshersh/zbg: ✨ Zero bullshit git, porcelain for git in ocaml

Git man page generator; funny.

Jujutsu/jj - A Git-compatible DVCS that is both simple and powerful -- It combines features from Git (data model, speed), Mercurial (anonymous branching, simple CLI free from "the index", revsets, powerful history-rewriting), and Pijul/Darcs (first-class conflicts), with features not found in most of them (working-copy-as-a-commit, undo functionality, automatic rebase, safe replication via rsync, Dropbox, or distributed file system). Basic jj Workflows

GitHub - jnsahaj/lumen: Beautiful git diff, Instant AI Git Commit message, Git changes summary from the CLI (no API key required) -- A command-line tool that uses AI to streamline your git workflow - generate commit messages, view diff and explain changes, all without requiring an API key.

Git workflows

Hosted git

Cloudflare Artifacts --

notes#git

GPT

GPT = Generative Pre-trained Transformer

See: AutoGPT

References

notes#deep-learning#gpt#llm

Computer Science Education

The Missing Semester of Your CS Education -- You should do this self taught course from MIT whether if you have a degree in CS or not. This is my top recommendation for learning the craft of programming/development.

Classes teach you all about advanced topics within CS, from operating systems to machine learning, but there’s one critical subject that’s rarely covered, and is instead left to students to figure out on their own: proficiency with their tools. We’ll teach you how to master the command-line, use a powerful text editor, use fancy features of version control systems, and much more!

Open Source Society University - computer-science: Path to a free self-taught education in Computer Science! -- links to various MOOCs for pretty much all the topics covered in a CS undergraduate curriculum.

A Functional Introduction To Computer Science (Part I) and Part II

CS Primer Courses (This is a paid course) -- The TL;DR is:

  • Start wherever looks interesting
  • Focus on solving problems
  • Use as much or as little of the problem walkthroughs and explainer videos as you need to keep making progress and having fun
  • Keep going!

notes#computer-science#learning

Archive.org

Some links from Archive.org

notes#books#website

Deep learning

Books

Papers

Articles

Learning Paths

Courses

Sargur Shrihari at Uni Buffalo

Videos

Blogs

People

References

  • Liquet, Moka, Nazarathy (2024). Mathematical Engineering of Deep Learning.

notes#deep-learning

System Design

Attention Conservation Notice

This specifically refers to software systems. Perhaps even more narrowly to Distributed Systems that involve Data, Web etc., Very unlikely hardware/embedded etc., because I do not deal in those.

Everything I know about good system design by sean goedecke. This article links to a few others he has writte on the topic.

software design is how you assemble lines of code, system design is how you assemble services. The primitives of software design are variables, functions, classes, and so on. The primitives of system design are app servers, databases, caches, queues, event buses, proxies, and so on.


Popular Service Level Indicators (SLIs):

  • Correctness
  • Availability
  • Throughput
  • Response Time ( p50, p99, etc)
  • Functional requirements describe the SLI for correctness.
  • Non-functional requirements describe the SLIs for availability, throughput, response time etc.

Service Level Objective (SLO):

  • A target value (or range) for an SLI. (e.g.: SLI <= target).
  • non-functional requirements for scaling, like - data size, bulky service time, geolocation, and data hotspots, to satisfy SLOs.

Service Level Agreement (SLA) is a contract with users on what the SLOs are, including the consequences of meeting/missing the SLOs. (e.g.: penalty/rebate).


videos on system design

Distributed Systems in One Lesson by Tim Berglund

Google Systems Design Interview With An Ex-Googler

My System Design Template - LeetCode Discuss

ByteByteGo | Ace Your Next System Design Interview

How to crawl a quarter billion webpages in 40 hours – DDI

CS 329S - Machine Learning Systems Design; "an iterative framework for developing real-world machine learning systems that are deployable, reliable, and scalable."

notes#system-design

ML Ops

MLOps Requirements

MLOPs reqs from abacus.ai


visenger/awesome-mlops: A curated list of references for MLOps

MLOps References

Papers

Operationalizing machine learning: An interview study (Shankar et al. 2022)

The process of operationalizing ML, or MLOps, consists of a continual loop of (i) data collection and labeling, (ii) experimentation to improve ML performance, (iii) evaluation throughout a multi-staged deployment process, and (iv) monitoring of performance drops in production. When considered together, these responsibilities seem staggering -- how does anyone do MLOps, what are the unaddressed challenges, and what are the implications for tool builders? We conducted semi-structured ethnographic interviews with 18 MLEs working across many applications, including chatbots, autonomous vehicles, and finance. Our interviews expose three variables that govern success for a production ML deployment: Velocity, Validation, and Versioning. We summarize common practices for successful ML experimentation, deployment, and sustaining production performance.

Unsolved Problems in MLOps - ACM Queue by Niall Murphy and Todd Underwood

Platforms

Frameworks

Chronon  is an open source end-to-end feature platform that allows Machine Learning (ML) teams to easily build, deploy, manage and monitor data pipelines for machine learning. It’s currently used to power all major ML applications within Airbnb, as well as major use cases at Stripe. Airbnb and Stripe jointly manage and maintain the project.

Chronon Schematic
Chronon Schematic
[via ](https://docs.google.com/document/d/10o75QTLfkaHCiiT2BftBld8ImuTojefZ03AYm__L9yw/edit) / [HN](https://news.ycombinator.com/item?id=41709547)

Blog posts

References

notes#machine-learning

Csharp

Why Tracebit is written in C#; Jan 2025. "Tracebit deploys and maintains tailored security canaries, proactively detecting intrusions across your organization.  From cloud infrastructure, to identity and endpoints."


bflat - C# as you know it but with Go-inspired tooling (small, selfcontained, and native executables)

  • Comes with two standard libraries: one (DotNet) that is compatible with .NET and supports everything from console to JSON, and another (Zero) that is stripped to bare minimum and doesn't even have a GC.
  • Run C# with no operating system, on bare metal
  • Build native code or .NET bytecode
  • Based on parts of the .NET 7 SDK
  • Invoke bflat directly from terminal, integrate with a shell script, Makefile, or even MSBuild.

notes#dotnet#programming-language

2022

Deno

Deno — A modern runtime for JavaScript and TypeScript. Learn it -- Deno by Example

How Deno works especially the bits about Extensions, web workers, and building your own runtime.

Why Deno

Applications written in Typescript

Libraries

Tooling

Frameworks

notes#deno#javascript#typescript

Webhosting

Show HN: SinglePage – Quickly and anonymously publish a page to the web | Hacker News; see some interesting options to publish single pages on the web -- telegra.ph, carrd.co, Write.as, Neocities, nekoweb, mmm.page — Dead Simple, Drag & Drop Websites. Of these carrd, and mmm* have visual tools, and the rest are simple text publishing tools.

I recommend Neocities if all you want to do is upload HTML files (whether handwritten or produced by an editor).

If you are someone familiar with using git, you also have the option of using github pages, and codeberg pages, of these github will give you CI/CD system in case you want to build a website from source files (eg: markdown -> HTML)

Cloudflare pages also has the CI option, in addition giving lot more features in dealing with DNS, caching, routing etc.. This is what I would recommend to a technical savvy person.

None of the above systems support (afaik), any kind of dynamic server side programming (eg: PHP, Python, Node), nor do they have traditional databases like MySQL support.


pgs.sh -- A zero-install static site hosting service for hackers

notes#webhosting

Podcasts

A note on various podcast episodes I have heard (partially or fully). Starting in Aug 2025, I will try to make an attempt to note what I have heard.

Recommendations to check out

Episodes Recommended by others from the CoRecursive Podcast:

Corecursive episodes by topic: Episodes organized by theme. - CoRecursive Podcast

Podcast Tech

  • Transistor - podcast hosting for creatives, brands, professionals; about $190/yr. Found via Oxide and Friends
  • 2025-08-02I recently got curious about how popular podcasters host thier shows, specifically after listening to Naval Ravikant's short 3-5 minute episodes. I noticed that he hosted the podcast on Libsyn. It does appear to be quite affordable for $5/mo for the "bare essentials".

notes#podcast

Wasm

About

Ecosystem

WasmEdge is a lightweight webassembly runtime for cloud native, edge and decentralized applications. It can be used to power serverless apps, embedded functions, microservices, smart contracts, and IOT devices.

Wasmer - The Universal WebAssembly Runtime "run any code on any client" with webassembly and wasmer.

2023-05-30: Announcing WASIX

WASIX is the long term stabilization and support of the existing WASI ABI _plus_additional non-invasive syscall extensions that complete the missing gaps sufficiently enough to enable real, practical and useful applications to be compiled and used now. It aims to speed up the ecosystem around WASI so that the Wasm’ification of code bases around the world can really start _today!_And it does so without any breaking changes to wasi_preview1.

WASIX:

  • It’s a toolchain:
    • Rust
    • C (upcoming support for Zig!)
    • AssemblyScript (expect upstream support soon!)
  • It's a specification: https://wasix.org/docs/api-reference
  • Runtime support: starting with Wasmer, but we expect more runtimes joining the initiative soon!
  • Is fully runnable in browsers

Projects using WASM

Runtimes

Chicory, a JVM native WebAssembly runtime, which allows you to do things like:

WasmModule module = Parser.parse(new File("./factorial.wasm"));
Instance instance = Instance.builder(module).build();

notes#wasm#web

Selfhosted

Why Selfhosted?

What is Self Hosted? What is a Stack? | Armin Ronacher's Thoughts and Writings

Candidates for self hosting

Challenges

  • moderation
  • availability
  • backups
  • technical upkeep (upgrades, patches)

Links

Development

Frameworks

Software

PaaS

notes#fediverse#paas#self-hosting

CS230

2022-11-27

AI Classes often discuss - Deep learning, Machine learning, Probabilistic Graphical Models, Planning algorithms (eg: motion planning), Search algorithms, Knowledge representation (knowledge graphs), Game theory.

Deep learning was much more attractive brand compared to Neural networks.

The DL/ML field has taken off, driven by more data and more computation, and the positive feedback loop by users; lot of Algorithmic innovations.

Software Design Book by John Ousterhout

"collect more data" vs "tune hyper parameters" - systematic approach to making this decision is the distinction between a beginner and a experienced person.

notes#ai#deep-learning

Containers

The What, Why and How of Containers – Annwan; HN

--

Lima. Lima launches Linux virtual machines with automatic file sharing and port forwarding (similar to WSL2), and containerd. Lima can be considered as a some sort of unofficial "containerd for Mac". Lima is expected to be used on macOS hosts, but can be used on Linux hosts as well. colima is a container environment on macOS with minimal setup that uses lima-vm.

Create an LXD container in Macpine - Macpine

GoogleContainerTools/distroless: 🥑 Language focused docker images, minus the operating system.

Wolfi "is a community Linux OS designed for the container and cloud-native era. Chainguard started the Wolfi project to enable building Chainguard Images, our collection of curated distroless images that meet the requirements of a secure software supply chain. This required a Linux distribution with components at the appropriate granularity and with support for both glibc and musl, something that was not yet available in the cloud-native Linux ecosystem. Wolfi is a stripped-down distro designed for the cloud-native era. It doesn't have a kernel of its own, instead relying on the environment (such as the container runtime) to provide one. This separation of concerns in Wolfi means it is adaptable to a range of environments."

Chainguard Images contain only what is required to build or run your application. This results in fewer CVEs over time compared to other base images and on average an 80% reduction in overall size.

  • Signed with Sigstore
  • SBOMs included

lucavallin/barco: Linux containers from scratch in C.

abiosoft/colima: Container runtimes on macOS (and Linux) with minimal setup is an alternative for podman

Bottlerocket is a Linux-based operating system optimized for hosting containers.  

  • Bottlerocket is installed as the base operating system on the machine or instance where your containers themselves are running.
  • It is designed to work with container orchestrator like kubernetes to automate the lifecycle of the containers running in the cluster.
  • Bottlerocket runs in the cloud or in your datacenter.
  • Bottlerocket is designed to be updatable but it doesn’t have a package manager. It doesn’t need one. Updates are delivered via an image that is downloaded to a specific partition. When you’re ready to update, let your orchestrator drain the node and then tell Bottlerocket to apply the update and reboot when ready. Bottlerocket will swap the partitions and boot with the new version atomically.
  • Bottlerocket’s unique functionality is written in Rust and a little bit of Golang.
  • The root filesystem of Bottlerocket is immutable.
  • Bottlerocket has an always-enabled, enforced, restrictive SELinux policy
  • It does not have a shell
  • You can still interact with the system through privileged “host” containers (that do have shells)

notes#containers#docker#kubernetes#linux#lxd

Blogging

Get Blogging! -- how to start blogging.

ooh.directory -- directory of blogs.

I give you feedback on your blog post draft but you don't send it to me

It's still worth blogging in the age of AI :: Giles' blog; Feb 2025

blogging isn't just about being read -- it's about learning and thinking, and having a durable proof that you can do both.

Long term blogging experiences

notes#blogging

Helix

Config file: ~/.config/helix/config.toml

hx --health shows the health like this (on MacOS):

Config file: default
Language file: default
Log file: /Users/pgowda/.cache/helix/helix.log
Runtime directory: /opt/homebrew/Cellar/helix/22.08.1/libexec/runtime
Clipboard provider: pbpaste+pbcopy
System clipboard provider: pbpaste+pbcopy

Language               LSP                    DAP                    Highlight              Textobject             Indent
awk                    ✘ awk-language-serv…   None                   ✓                      ✓                      ✘
bash                   ✘ bash-language-ser…   None                   ✓                      ✘                      ✘
beancount              None                   None                   ✓                      ✘                      ✘
c                      ✓ clangd               ✘ lldb-vscode          ✓                      ✓                      ✓
c-sharp                ✘ OmniSharp            None                   ✓                      ✓                      ✘

Notes

Comparisons

notes#editor

Typesetting

Typesetting systems

Polytype – Home -- A Rosetta stone for typesetting engines. This project's goal is to provide a chrestomathy for typesetting similar to what Rosetta Code does for programming languages. The samples here are designed to compare and/or contrast the approaches taken to various typesetting situations by different typesetting engines.

Classic

New

Typst: Compose papers faster


See also: A Quick How-to on using LuaTeX with ConTeXt, pandoc, typesetting, groff.

PDF

pdf.tocgen is a set of command-line tools for automatically extracting and generating the table of contents (ToC) of a pdf file. It uses the embedded font attributes and position of headings to deduce the basic outline of a pdf file.

notes#typesetting

Postscript

Mathematical Illustrations an introduction to mathematical illustrations using postscript programming language. This is one of the best showcases of postscript

Don Lancaster (tinaja.com) and his many notes on postscript publishing; Obituary.

Don Lancaster passing away notice
Don Lancaster passed away in June 2023.

PostScript: A Digital Printing Press | Hacker News. Main discussion about PostScript: A Digital Printing Press from the Computer History Museum article, but the HN page has lots of links to PS stuff; Dec 2022.

The origins of Postscript by John Warnock, founder of Adobe in IEEE Annals of History of Computing (2018) via gwern.

A First Guide to PostScript by Peter Weingartner. Online book.

PSChess - A Chess Engine in PostScript

PostScript® 1.0 - A Code Study | ℤ→ℤ

nasa/pvslib: NASA PVS Library of Formal Developments with 18% postscript code (among common-lisp, python etc )

nst/PSTris: A Tetris implementation in PostScript

--

See also: My Hello Postscript page.

notes#graphics#postscript#programming#typesetting