Weston Pace

Data engineer from the open source space, working on LanceDb, Arrow, Substrait

Olympia, WA

2 Articles

Columnar File Readers in Depth: Parallelism without Row Groups

Recently, I shared our plans for a new file format, Lance v2. As I'm creating a file reader for this new format I

7 min read

Lance v2: A columnar container format for modern data

Why a new format? Lance was invented because readers and writers for existing column formats did not handle AI/ML workloads efficiently. Lance v1 solved

12 min read