here you goThings I’ve made, read and saved — shared here.

Archive

9,565 entries: 46 from X, 8,923 from Pinboard, saved 1998 to 2026.

Browse by tag, site, author, folder or language

Tags

Sites

Authors

Folders

Languages

46 entries#sql ×

2025

Evidence

Evidence is "Business Intelligence as Code"; which means it allows you to build reports, and standard business data visualization ( charts, and basic graphics) using sql with markdown as the glue between various data and visualization elements. You can put to gether a dynamic report with a few interactive elements (e.g: filter data, change date ranges) that can be exported as "static" web pages. The exported web pages will also have interactive elements (e.g: tooltip on hover, charts that re-render when you change the data criteria). You can see the github; and examples

datasette is another player in this space -- "data applications using sql" space, but evidence goes much further by supporting various data backends, and having first class support for thinking in terms of reports, and standard visualization components like charts. Support for markdown is naturally expected.

The combination of markdown + code does put it in a group with jupyter, but the availability of dynamic controls also make you think of streamlit, but does not go as far as observable because observable, given it's pedigree with d3 etc., is a lot more capable visulization tool (think maps, and advanced dataviz techniques).

notes#dataapps#sql#visualization

The DuckDB Local UI

duckdb.org

$ duckdb -ui

can have multiple databases including in memory database attached. Can explore the databases using Table summary, the notebooks are useful in organizing work into notebooks. SQL is the language of the notebooks like [[python]] is for [[jupyter]] notebooks. The notebook cells support syntax highlighting and autocompletion.

The results can be visualized using the column explorer, which shows the summary of the results. It is "type aware", and thus can visulize things like percentiles, median etc.

Interestingly, the the UI extension is all written in [[cpp]] - https://github.com/duckdb/duckdb-ui

Pinboard Database Schema

static.pinboard.in

the mysql database schema that powers pinboard.in. useful if you are trying to create your own pinboard like service.

-- MySQL dump 10.13   
--  
-- Pinboard database schema  
-- Feel free to use this however you like. Send questions to support@pinboard.in. 
-- For best results, use the Percona version of MySQL! http://www.percona.com/software/percona-server/  


CREATE TABLE `bookmarks` (  
    `id` int(11) NOT NULL,  
    `url` mediumtext,        -- verbatim URL (may differ from actual URL referenced by url_id)  
    `title` varchar(255),  
    `description` mediumtext,  
    `user_id` int(11) NOT NULL,  
    `toread` tinyint(1) DEFAULT '0',  
    `private` binary(1) DEFAULT '0',  
    `url_id` int(11),     
    `slug` char(20),         -- opaque token for use in URLs  
    `snapshot_id` int(11),  
    `code` char(3),          -- http response code (if crawled)  
    `source` smallint(6),    -- numeric     
    `added_at` datetime,     -- date added to Pinboard  
    `created_at` datetime,   -- stated creation date  
    `updated_at` datetime,  
  
    PRIMARY KEY (`id`),  
    UNIQUE KEY `slug` (`slug`),     
    UNIQUE KEY `bookmark` (`user_id`,`url`(255)),  
    KEY `created` (`created_at`),  
    KEY `user` (`user_id`),  
    KEY `private` (`private`),  
    KEY `url` (`url_id`),  
    KEY `toread` (`toread`),  
    KEY `updated` (`updated_at`),  
    KEY `snapshot` (`snapshot_id`),  
    KEY `code` (`code`),  
    KEY `multi` (`user_id`,`private`,`toread`,`created_at`)  
) ENGINE=InnoDB DEFAULT CHARSET=utf8;  
     
  
-- Table structure for table `btags` (short for 'bookmark tags')  

  
CREATE TABLE `btags` (  
    `id` int(11) NOT NULL,     
    `user_id` int(11) NOT NULL,  
    `bookmark_id` int(11) NOT NULL,  
    `url_id` int(11),  
    `tag` varchar(255),  
    `created_at` datetime,     
    `private` tinyint(1),     
    `seq` tinyint(4),         -- preserve display order  
    PRIMARY KEY (`id`),  
    UNIQUE KEY `btag` (`user_id`,`bookmark_id`,`tag`),  
    KEY `user` (`user_id`),  
    KEY `tag` (`tag`),  
    KEY `bookmark` (`bookmark_id`),  
    KEY `url` (`url_id`),  
    KEY `private` (`private`),  
    KEY `usertag` (`user_id`,`tag`)  
) ENGINE=InnoDB DEFAULT CHARSET=utf8;   
     
  
  
  
-- Table structure for table `snapshots`  
-- (snapshots are directories created by wget)  
  
CREATE TABLE `snapshots` (  
    `id` int(11) NOT NULL,  
    `url_id` int(11) NOT NULL,  
    `crawled_at` datetime,     
    `slug` varchar(30),  
    `content_type` varchar(200),     
    `etag` varchar(255),                  -- from Etag header     
    `last_modified` varchar(255),         -- from Last-Modified header     
    `code` char(3),                       -- http status code  
    `content_length` varchar(20),         -- from Content-Length header  
    `num_files` smallint(6),                      
    `size` int(11),                       -- size in bytes of snapshot files  
    `filename` varchar(255),              -- which file in this snapshot to serve  
    `updated_at` datetime,  
    `user_id` int(11),  
    `remote_backup` datetime,                     
    `flagged` binary(1) DEFAULT '0',  
    `actual_url_id` int(11),              -- where we ended up after redirects  
    `server` varchar(200),                -- where this snapshot is stored  
    `charset` varchar(20),                -- detected charset (don't trust servers!)  
    PRIMARY KEY (`id`),  
    UNIQUE KEY `user_url` (`url_id`,`user_id`),  
    KEY `size` (`size`),  
    KEY `content_length` (`content_length`),  
    KEY `content_type` (`content_type`),  
    KEY `slug` (`slug`),  
    KEY `user_id` (`user_id`),  
    KEY `code` (`code`),  
    KEY `crawled` (`crawled_at`)  
) ENGINE=InnoDB DEFAULT CHARSET=utf8;  
     
  
  
     
--  
-- Table structure for table `urls`  
--  
  
CREATE TABLE `urls` (  
    `id` int(11) NOT NULL,  
    `url` mediumtext CHARACTER SET latin1,  -- latin1 reduces storage requirement  
    `create

2024

LLMs in Data Management

[@10.14778/3611479.3611527]; PDF

We argue that the disruptive influence that LLMs will have on data management will come from two angles. (1) A number of hard database problems, namely, entity resolution, schema matching, data discovery, and query synthesis, hit a ceiling of automation because the system does not fully understand the semantics of the underlying data. Based on large training corpora of natural language, structured data, and code, LLMs have an unprece- dented ability to ground database tuples, schemas, and queries in real-world concepts. We will provide examples of how LLMs may completely change our approaches to these problems. (2) LLMs blur the line between predictive models and information retrieval systems with their ability to answer questions. We will present examples showing how large databases and information retrieval systems have complementary functionality.

(Shi et al. 2024)

writing SQL queries requires specialized knowledge, which poses a challenge for non-professional users trying to access and query databases. Text-to-SQL parsing solves this issue by converting natural language queries into SQL queries, thus making database access more accessible for non-expert users. To take advantage of the recent developments in Large Language Models (LLMs), a range of new methods have emerged, with a primary focus on prompt engineering and fine-tuning. This survey provides a comprehensive overview of LLMs in text-to-SQL tasks, discussing benchmark datasets, prompt engineering, fine-tuning methods, and future research directions.

References

notes#data-management#database#llm#sql

Data Apps

Data applications (usually), web application frameworks that make it easy to publish data as dynamic reports that have an element of interactivity to them that is not possible in static reports, they are often purpose-built to tell a story or present findings backed by data.

Frameworks

evidence is a lightweight framework for building data apps using SQL. It's open source and free to get started.

observable is another example -- that uses javascript

Briefer | The collaborative data platform -- "jupyter notebooks meet Notion-like documents. Briefer is the multiplayer notebook that let's you run SQL and Python, and turn notebooks into dashboards and Data Apps.

Mercury – Build Data Web Apps in Jupyter Notebook You don't need to know frontend frameworks, HTML, CSS or JavaScript. All in pure Python without leaving Jupyter Notebook.

my mercury account

notes#dataapps#database#framework#sql

2023

Query Languages

There is of course sql.. and

PRQL -- Pipelined Relational Query Language, pronounced “Prequel”. The web page has good examples and the advantages; probably the most promising looking of these query languages I have known so far (aug 2023).

And there is hydromatic/morel: Standard ML interpreter, with relational extensions, implemented in Java.. the reasoning behind the creation of Morel is given here -- Morel: A functional language for data | Julian Hyde on Streaming Data, Open Source OLAP. And stuff.. The creator of Morel is also behind Apache Calcite, Mondrian (OLAP tool, IIRC).

notes#sql

2022

PostgreSQL

Why PostgreSQL

PostgreSQL is Enough; and Just Use Postgres for Everything.

Just use Postgres

Quickstart

Pedram Navid | A Fast, Sample Postgres Database with Docker

Learning PostgreSQL

Internals

Articles

Appliances

The Elephant Shed PostgreSQL Appliance bundles and integrates proven components, required for easy management of a PostgreSQL server. Proven tools for all relevant aspects are already preinstalled and preconfigured. The majority of these tools can be controlled via a comfortable web interface. Even experienced PostgreSQL administrators will hardly find an area not covered by Elephant Shed PostgreSQL appliance.

Migrating and upgrading PostgreSQL

  • How we migrated our PostgreSQL database with 11 seconds downtime - Government Digital Service; 2024
  • xataio/pgroll -- is an open source command-line tool that offers safe and reversible schema migrations for PostgreSQL by serving multiple schema versions simultaneously. It takes care of the complex migration operations to ensure that client applications continue working while the database schema is being updated. This includes ensuring changes are applied without locking the database, and that both old and new schema versions work simultaneously (even when breaking changes are being made!). This removes risks related to schema migrations, and greatly simplifies client application rollout, also allowing for instant rollbacks.. Read introductory blogpost on xata.io

Tips and tricks

PostgreSQL and Full Text Search

This is one of the things PG does a good job. Keeping notes.

Using PostgreSQL as a queue / Kafka

So that’s what we built: a new queue backed by a single, and surprisingly simple, Postgres table. Our publishers write to it. Our consumers (workers) read from it. We maintain things like queue ordering by adding an ORDER BY clause in the query that consumers use to read from it (groundbreaking, we know). And we guarantee that jobs won’t be picked up by more than one worker through simple read/write row-level locks. The new system is actually kind of absurdly simple when you look at it. And that’s a good thing. It’s also behaved flawlessly so far.

It comes with several, significant upsides for our team. For one, the application state is no longer spread out over two systems (RabbitMQ storage and Postgres). It’s now centralized in our application database. This makes disaster recovery that much easier, and increases the resiliency of the overall system by removing moving pieces.

To make all of this run smoothly, we enqueue and dequeue thousands of jobs every day.

(the load isn't all that big)


Sequin "A Postgres alternative to Kafka. Get a feature-rich message stream with none of the operational overhead". See Build your own SQS or Kafka with Postgres blog post.

PostgreSQL and ML

  • Home – PostgresML -- An open source Postgres based platform for your interactive AI powered applications. Instantly scale your workloads with dedicated replicas or our serverless GPU accelerated cloud.

PostgreSQL on MacOS

Installing PostgreSQL on a Mac with Homebrew - Daily Dev Tips -- includes tips on how to create a non-postgres user.

Self Hosting

Go ahead, self-host Postgres | Pierce Freeman

PostgreSQL on the cloud

  1. Supabase -- "an open source Firebase alternative. Postgres database, Authentication, instant APIs, Edge Functions, Realtime subscriptions, Storage, and Vector embeddings".
    1. postgres.new: In-browser Postgres with an AI interface by Supabase; HN
    2. Postgres Sandbox
  2. Xata -- "serverless data platform for PostgreSQL."
  3. Tembo -- "Tembo is the Postgres developer platform for building every data service."

PostgreSQL-ish databases

Apart from the ones above ("on the cloud"),

Embedded PostgreSQL

PGlite "Run a full Postgres database locally in WASM with reactivity and live sync."

  • under 3MB Gzipped.
  • Dynamic extension loading mechanism, including support for pgvector and PostGIS
  • Built in support for data loading, synchronisation and live query primitives.

notes#database#postgresql#sql

2021

2020

SQL

SQL Words
SQL Words (via)
## On SQL

Learning resources

Select Star SQL "... is an interactive book which aims to be the best place on the internet for learning SQL. It is free of charge, free of ads and doesn't require registration or downloads. It helps you learn by running queries against a real-world dataset to complete projects of consequence. It is not a mere reference page — it conveys a mental model for writing SQL". Very nicely laid out (credits M. Buttrick of "Practical Typography").

SQL Teaching - The easiest tutorial to learn SQL

SQLBolt - Learn SQL - Introduction to SQL ... a series of interactive lessons and exercises designed to help you quickly learn SQL right in your browser. Related website - regexone

Advanced SQL window functions quiz

SQL style guide by Simon Holywell

How Does a Database Work? | Let’s Build a Simple Database

Learn SQL while solving crimes! SQL Police Department

SQL for Data Scientists in 100 Queries

SQL visualization/query/notebook tools

Redash -- can be self hosted.

Franchise: a sql notebook -- runs local proxy on your system. the UI itself is on franchise.cloud website. Can also run locally.

apache/incubator-superset: Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application

forbesmyester/esqlate -- Build minimum viable admin panels quickly with just SQL. uses node.js proxy behind nginx. template queries are written using JSON and the "holes" are filled by user to get the results back. Users can also add new records.

The Log File Navigator --

  • An advanced log file viewer for the small-scale
  • Watch and analyze your log files from a terminal.
  • No server. No setup. Still featureful.
  • Many logging tools, like Splunk, provide great features but are optimized for large-scale deployments. They require installing and configuring servers before they can be effectively used. There is still a need for a robust log file analyzer for the terminal.
  • Just point lnav to a directory and it will take care of the rest. File formats are automatically detected and compressed files are unpacked on the fly.
  • Log files are a wealth of information, lnav can help highlight the parts that are important and filter out the noise.

Editors / IDEs

Command line tools

Translation tools

Frameworks

LLM

A Survey Paper (Shi et al. 2024)

Text-to-SQL parsing solves this issue by converting natural language queries into SQL queries, thus making database access more accessible for non-expert users. To take advantage of the recent developments in Large Language Models (LLMs), a range of new methods have emerged, with a primary focus on prompt engineering and fine-tuning. This survey provides a comprehensive overview of LLMs in text-to-SQL tasks, discussing benchmark datasets, prompt engineering, fine-tuning methods, and future research directions.


See also Query Languages

References

notes#database#sql

2019

2017

2015

2014

2013

2010

SQL Toolbelt - All our SQL Server tools in a single installer

red-gate.com

according to this comment: http://www.reddit.com/r/programming/comments/eie3z/a_table_that_should_exist_in_all_projects_with_a/c18bpbs
on reddit, "This is not a sales pitch, but we use this toolkit to do all of our database comparisons. It allows us to develop freely on our Dev servers without too much concern about losing track of changes.
Sure, its reasonably expensive, but it takes seconds to compare and sync database structures and data (if necessary).
If you do a lot of progressive development with multiple release drops, then it's great. It can even prepare you upgrade scripts which you can run as part of patches and upgrades."

red-gate.comPinboardPermalink

#sql

2009

2007

2006

2005

2004