Tutorial: Artifacts

Biology is measured in samples that generate batches of data.

LaminDB provides a framework to transform these batches into more useful representations: validated, queryable datasets, machine learning models, and analytical insights.

The tutorial has two parts, each is a Jupyter notebook:

  1. Tutorial: Artifacts - register & access

  2. Tutorial: Features & labels - validate & annotate

Setup

Install the lamindb Python package:

pip install 'lamindb[jupyter,aws]'

Init a LaminDB instance with a directory ./lamin-tutorial for storing data:

!lamin init --storage ./lamin-tutorial  # or "s3://my-bucket" or "gs://my-bucket"
Hide code cell output 
! using anonymous user (to identify, call: lamin login)

→ connected lamindb: anonymous/lamin-tutorial
What else can I configure during setup?

  1. Instead of the default SQLite database, use PostgreSQL:

    db=postgresql://<user>:<pwd>@<hostname>:<port>/<dbname>

  2. Instead of a default instance name derived from storage, provide a custom name:

    name=myinstance

  3. Beyond the core schema, use bionty and other schemas:

    schema=bionty,custom1,template1

For more, see Install & setup.

Track data transformations

The code that generates a dataset is a transform (Transform). It could be a script, a notebook, a pipeline or a UI interaction like an upload.

Let’s track the notebook that’s being run:

import lamindb as ln

# --> `ln.track()` generates a uid for your code
# --> `ln.track(uid)` initiates a tracked run
ln.track("NJvdsWWbJlZS0000")
Hide code cell output 
→ connected lamindb: anonymous/lamin-tutorial

→ notebook imports: lamindb==0.76.15

→ created Transform('NJvdsWWb'), started new Run('5oqBFfLR') at 2024-11-07 12:39:11 UTC

By calling track(), the notebook is automatically linked as the source of all data that’s about to be saved!

What happened under the hood?

  1. Imported package versions of current notebook were detected

  2. Notebook metadata was detected and stored in a Transform record

  3. Run metadata was detected and stored in a Run record

The Transform class registers data transformations: a notebook, a pipeline or a UI operation.

The Run class registers executions of transforms. Several runs can be linked to the same transform if executed with different context (time, user, input data, etc.).

How do I track a pipeline instead of a notebook?
transform = ln.Transform(name="My pipeline")
transform.version = "1.2.0"  # tag the version
ln.context.track(transform)
Why should I care about tracking notebooks?

If you can, avoid interactive notebooks: Anything that can be a deterministic pipeline, should be a pipeline.

Just: much insight generated from biological data is driven by computational biologists interacting with it.

A notebook that’s run a single time on specific data is not a pipeline: it’s a (versioned) document that produced insight or some other form of data representation (with parallels to an ELN in the wetlab).

Because humans are in the loop, most mistakes happen when using notebooks: track() helps avoiding some.

(An early blog post on this is here.)

Manage artifacts

We’ll work with a toy collection of image files and transform them into higher-level features for downstream analysis.

(For other data types: see Data types.)

Consider 3 directories storing images & metadata of Iris flowers, generated in 3 subsequent studies:

# we use anon=True here in case no aws credentials are configured
ln.UPath("s3://lamindata/iris_studies", anon=True).view_tree()
Hide code cell output 
3 sub-directories & 151 files with suffixes '.csv', '.jpg'
s3://lamindata/iris_studies
├── study0_raw_images/
│   ├── iris-0337d20a3b7273aa0ddaa7d6afb57a37a759b060e4401871db3cefaa6adc068d.jpg
│   ├── iris-0797945218a97d6e5251b4758a2ba1b418cbd52ce4ef46a3239e4b939bd9807b.jpg
│   ├── iris-0f133861ea3fe1b68f9f1b59ebd9116ff963ee7104a0c4200218a33903f82444.jpg
│   ├── iris-0fec175448a23db03c1987527f7e9bb74c18cffa76ef003f962c62603b1cbb87.jpg
│   ├── iris-125b6645e086cd60131764a6bed12650e0f7f2091c8bbb72555c103196c01881.jpg
│   ├── iris-13dfaff08727abea3da8cfd8d097fe1404e76417fefe27ff71900a89954e145a.jpg
│   ├── iris-1566f7f5421eaf423a82b3c1cd1328f2a685c5ef87d8d8e710f098635d86d3d0.jpg
│   ├── iris-1804702f49c2c385f8b30913569aebc6dce3da52ec02c2c638a2b0806f16014e.jpg
│   ├── iris-318d451a8c95551aecfde6b55520f302966db0a26a84770427300780b35aa05a.jpg
│   ├── iris-3dec97fe46d33e194520ca70740e4c2e11b0ffbffbd0aec0d06afdc167ddf775.jpg
│   ├── iris-3eed72bc2511f619190ce79d24a0436fef7fcf424e25523cb849642d14ac7bcf.jpg
│   ├── iris-430fa45aad0edfeb5b7138ff208fdeaa801b9830a9eb68f378242465b727289a.jpg
│   ├── iris-4cc15cd54152928861ecbdc8df34895ed463403efb1571dac78e3223b70ef569.jpg
│   ├── iris-4febb88ef811b5ca6077d17ef8ae5dbc598d3f869c52af7c14891def774d73fa.jpg
│   ├── iris-590e7f5b8f4de94e4b82760919abd9684ec909d9f65691bed8e8f850010ac775.jpg
│   ├── iris-5a313749aa61e9927389affdf88dccdf21d97d8a5f6aa2bd246ca4bc926903ba.jpg
│   ├── iris-5b3106db389d61f4277f43de4953e660ff858d8ab58a048b3d8bf8d10f556389.jpg
│   ├── iris-5f4e8fffde2404cc30be275999fddeec64f8a711ab73f7fa4eb7667c8475c57b.jpg
│   ├── iris-68d83ad09262afb25337ccc1d0f3a6d36f118910f36451ce8a6600c77a8aa5bd.jpg
│   ├── iris-70069edd7ab0b829b84bb6d4465b2ca4038e129bb19d0d3f2ba671adc03398cc.jpg
│   ├── iris-7038aef1137814473a91f19a63ac7a55a709c6497e30efc79ca57cfaa688f705.jpg
│   ├── iris-74d1acf18cfacd0a728c180ec8e1c7b4f43aff72584b05ac6b7c59f5572bd4d4.jpg
│   ├── iris-7c3b5c5518313fc6ff2c27fcbc1527065cbb42004d75d656671601fa485e5838.jpg
│   ├── iris-7cf1ebf02b2cc31539ed09ab89530fec6f31144a0d5248a50e7c14f64d24fe6e.jpg
│   ├── iris-7dcc69fa294fe04767706c6f455ea6b31d33db647b08aab44b3cd9022e2f2249.jpg
│   ├── iris-801b7efb867255e85137bc1e1b06fd6cbab70d20cab5b5046733392ecb5b3150.jpg
│   ├── iris-8305dd2a080e7fe941ea36f3b3ec0aa1a195ad5d957831cf4088edccea9465e2.jpg
│   ├── iris-83f433381b755101b9fc9fbc9743e35fbb8a1a10911c48f53b11e965a1cbf101.jpg
│   ├── iris-874121a450fa8a420bdc79cc7808fd28c5ea98758a4b50337a12a009fa556139.jpg
│   ├── iris-8c216e1acff39be76d6133e1f549d138bf63359fa0da01417e681842210ea262.jpg
│   ├── iris-92c4268516ace906ad1ac44592016e36d47a8c72a51cacca8597ba9e18a8278b.jpg
│   ├── iris-95d7ec04b8158f0873fa4aab7b0a5ec616553f3f9ddd6623c110e3bc8298248f.jpg
│   ├── iris-9ce2d8c4f1eae5911fcbd2883137ba5542c87cc2fe85b0a3fbec2c45293c903e.jpg
│   ├── iris-9ee27633bb041ef1b677e03e7a86df708f63f0595512972403dcf5188a3f48f5.jpg
│   ├── iris-9fb8d691550315506ae08233406e8f1a4afed411ea0b0ac37e4b9cdb9c42e1ec.jpg
│   ├── iris-9ffe51c2abd973d25a299647fa9ccaf6aa9c8eecf37840d7486a061438cf5771.jpg
│   ├── iris-a2be5db78e5b603a5297d9a7eec4e7f14ef2cba0c9d072dc0a59a4db3ab5bb13.jpg
│   ├── iris-ad7da5f15e2848ca269f28cd1dc094f6f685de2275ceaebb8e79d2199b98f584.jpg
│   ├── iris-bc515e63b5a4af49db8c802c58c83db69075debf28c792990d55a10e881944d9.jpg
│   ├── iris-bd8d83096126eaa10c44d48dbad4b36aeb9f605f1a0f6ca929d3d0d492dafeb6.jpg
│   ├── iris-bdae8314e4385d8e2322abd8e63a82758a9063c77514f49fc252e651cbd79f82.jpg
│   ├── iris-c175cd02ac392ecead95d17049f5af1dcbe37851c3e42d73e6bb813d588ea70b.jpg
│   ├── iris-c31e6056c94b5cb618436fbaac9eaff73403fa1b87a72db2c363d172a4db1820.jpg
│   ├── iris-ca40bc5839ee2f9f5dcac621235a1db2f533f40f96a35e1282f907b40afa457d.jpg
│   ├── iris-ddb685c56cfb9c8496bcba0d57710e1526fff7d499536b3942d0ab375fa1c4a6.jpg
│   ├── iris-e437a7c7ad2bbac87fef3666b40c4de1251b9c5f595183eda90a8d9b1ef5b188.jpg
│   ├── iris-e7e0774289e2153cc733ff62768c40f34ac9b7b42e23c1abc2739f275e71a754.jpg
│   ├── iris-e9da6dd69b7b07f80f6a813e2222eae8c8f7c3aeaa6bcc02b25ea7d763bcf022.jpg
│   ├── iris-eb01666d4591b2e03abecef5a7ded79c6d4ecb6d1922382c990ad95210d55795.jpg
│   ├── iris-f6e4890dee087bd52e2c58ea4c6c2652da81809603ea3af561f11f8c2775c5f3.jpg
│   └── meta.csv
├── study1_raw_images/
│   ├── iris-0879d3f5b337fe512da1c7bf1d2bfd7616d744d3eef7fa532455a879d5cc4ba0.jpg
│   ├── iris-0b486eebacd93e114a6ec24264e035684cebe7d2074eb71eb1a71dd70bf61e8f.jpg
│   ├── iris-0ff5ba898a0ec179a25ca217af45374fdd06d606bb85fc29294291facad1776a.jpg
│   ├── iris-1175239c07a943d89a6335fb4b99a9fb5aabb2137c4d96102f10b25260ae523f.jpg
│   ├── iris-1289c57b571e8e98e4feb3e18a890130adc145b971b7e208a6ce5bad945b4a5a.jpg
│   ├── iris-12adb3a8516399e27ff1a9d20d28dca4674836ed00c7c0ae268afce2c30c4451.jpg
│   ├── iris-17ac8f7b5734443090f35bdc531bfe05b0235b5d164afb5c95f9d35f13655cf3.jpg
│   ├── iris-2118d3f235a574afd48a1f345bc2937dad6e7660648516c8029f4e76993ea74d.jpg
│   ├── iris-213cd179db580f8e633087dcda0969fd175d18d4f325cb5b4c5f394bbba0c1e0.jpg
│   ├── iris-21a1255e058722de1abe928e5bbe1c77bda31824c406c53f19530a3ca40be218.jpg
│   ├── iris-249370d38cc29bc2a4038e528f9c484c186fe46a126e4b6c76607860679c0453.jpg
│   ├── iris-2ac575a689662b7045c25e2554df5f985a3c6c0fd5236fabef8de9c78815330c.jpg
│   ├── iris-2c5b373c2a5fd214092eb578c75eb5dc84334e5f11a02f4fa23d5d316b18f770.jpg
│   ├── iris-2ecaad6dfe3d9b84a756bc2303a975a732718b954a6f54eae85f681ea3189b13.jpg
│   ├── iris-32827aec52e0f3fa131fa85f2092fc6fa02b1b80642740b59d029cef920c26b3.jpg
│   ├── iris-336fc3472b6465826f7cd87d5cef8f78d43cf2772ebe058ce71e1c5bad74c0e1.jpg
│   ├── iris-432026d8501abcd495bd98937a82213da97fca410af1c46889eabbcf2fd1b589.jpg
│   ├── iris-49a9158e46e788a39eeaefe82b19504d58dde167f540df6bc9492c3916d5f7ca.jpg
│   ├── iris-4b47f927405d90caa15cbf17b0442390fc71a2ca6fb8d07138e8de17d739e9a4.jpg
│   ├── iris-5691cad06fe37f743025c097fa9c4cec85e20ca3b0efff29175e60434e212421.jpg
│   ├── iris-5c38dba6f6c27064eb3920a5758e8f86c26fec662cc1ac4b5208d5f30d1e3ead.jpg
│   ├── iris-5da184e8620ebf0feef4d5ffe4346e6c44b2fb60cecc0320bd7726a1844b14cd.jpg
│   ├── iris-66eee9ff0bfa521905f733b2a0c6c5acad7b8f1a30d280ed4a17f54fe1822a7e.jpg
│   ├── iris-6815050b6117cf2e1fd60b1c33bfbb94837b8e173ff869f625757da4a04965c9.jpg
│   ├── iris-793fe85ddd6a97e9c9f184ed20d1d216e48bf85aa71633eff6d27073e0825d54.jpg
│   ├── iris-850229e6293a741277eb5efaa64d03c812f007c5d0f470992a8d4cfdb902230c.jpg
│   ├── iris-86d782d20ef7a60e905e367050b0413ca566acc672bc92add0bb0304faa54cfc.jpg
│   ├── iris-875a96790adc5672e044cf9da9d2edb397627884dfe91c488ab3fb65f65c80ff.jpg
│   ├── iris-96f06136df7a415550b90e443771d0b5b0cd990b503b64cc4987f5cb6797fa9b.jpg
│   ├── iris-9a889c96a37e8927f20773783a084f31897f075353d34a304c85e53be480e72a.jpg
│   ├── iris-9e3208f4f9fedc9598ddf26f77925a1e8df9d7865a4d6e5b4f74075d558d6a5e.jpg
│   ├── iris-a7e13b6f2d7f796768d898f5f66dceefdbd566dd4406eea9f266fc16dd68a6f2.jpg
│   ├── iris-b026efb61a9e3876749536afe183d2ace078e5e29615b07ac8792ab55ba90ebc.jpg
│   ├── iris-b3c086333cb5ccb7bb66a163cf4bf449dc0f28df27d6580a35832f32fd67bfc9.jpg
│   ├── iris-b795e034b6ea08d3cd9acaa434c67aca9d17016991e8dd7d6fd19ae8f6120b77.jpg
│   ├── iris-bb4a7ad4c844987bc9dc9dfad2b363698811efe3615512997a13cd191c23febc.jpg
│   ├── iris-bd60a6ed0369df4bea1934ef52277c32757838123456a595c0f2484959553a36.jpg
│   ├── iris-c15d6019ebe17d7446ced589ef5ef7a70474d35a8b072e0edfcec850b0a106db.jpg
│   ├── iris-c45295e76c6289504921412293d5ddbe4610bb6e3b593ea9ec90958e74b73ed2.jpg
│   ├── iris-c50d481f9fa3666c2c3808806c7c2945623f9d9a6a1d93a17133c4cb1560c41c.jpg
│   ├── iris-df4206653f1ec9909434323c05bb15ded18e72587e335f8905536c34a4be3d45.jpg
│   ├── iris-e45d869cb9d443b39d59e35c2f47870f5a2a335fce53f0c8a5bc615b9c53c429.jpg
│   ├── iris-e76fa5406e02a312c102f16eb5d27c7e0de37b35f801e1ed4c28bd4caf133e7a.jpg
│   ├── iris-e8d3fd862aae1c005bcc80a73fd34b9e683634933563e7538b520f26fd315478.jpg
│   ├── iris-ea578f650069a67e5e660bb22b46c23e0a182cbfb59cdf5448cf20ce858131b6.jpg
│   ├── iris-eba0c546e9b7b3d92f0b7eb98b2914810912990789479838807993d13787a2d9.jpg
│   ├── iris-f22d4b9605e62db13072246ff6925b9cf0240461f9dfc948d154b983db4243b9.jpg
│   ├── iris-fac5f8c23d8c50658db0f4e4a074c2f7771917eb52cbdf6eda50c12889510cf4.jpg
│   └── meta.csv
└── study2_raw_images/
    ├── iris-01cdd55ca6402713465841abddcce79a2e906e12edf95afb77c16bde4b4907dc.jpg
    ├── iris-02868b71ddd9b33ab795ac41609ea7b20a6e94f2543fad5d7fa11241d61feacf.jpg
    ├── iris-0415d2f3295db04bebc93249b685f7d7af7873faa911cd270ecd8363bd322ed5.jpg
    ├── iris-0c826b6f4648edf507e0cafdab53712bb6fd1f04dab453cee8db774a728dd640.jpg
    ├── iris-10fb9f154ead3c56ba0ab2c1ab609521c963f2326a648f82c9d7cabd178fc425.jpg
    ├── iris-14cbed88b0d2a929477bdf1299724f22d782e90f29ce55531f4a3d8608f7d926.jpg
    ├── iris-186fe29e32ee1405ddbdd36236dd7691a3c45ba78cc4c0bf11489fa09fbb1b65.jpg
    ├── iris-1b0b5aabd59e4c6ed1ceb54e57534d76f2f3f97e0a81800ff7ed901c35a424ab.jpg
    ├── iris-1d35672eb95f5b1cf14c2977eb025c246f83cdacd056115fdc93e946b56b610c.jpg
    ├── iris-1f941001f508ff1bd492457a90da64e52c461bfd64587a3cf7c6bf1bcb35adab.jpg
    ├── iris-2a09038b87009ecee5e5b4cd4cef068653809cc1e08984f193fad00f1c0df972.jpg
    ├── iris-308389e34b6d9a61828b339916aed7af295fdb1c7577c23fb37252937619e7e4.jpg
    ├── iris-30e4e56b1f170ff4863b178a0a43ea7a64fdd06c1f89a775ec4dbf5fec71e15c.jpg
    ├── iris-332953f4d6a355ca189e2508164b24360fc69f83304e7384ca2203ddcb7c73b5.jpg
    ├── iris-338fc323ed045a908fb1e8ff991255e1b8e01c967e36b054cb65edddf97b3bb0.jpg
    ├── iris-34a7cc16d26ba0883574e7a1c913ad50cf630e56ec08ee1113bf3584f4e40230.jpg
    ├── iris-360196ba36654c0d9070f95265a8a90bc224311eb34d1ab0cf851d8407d7c28e.jpg
    ├── iris-36132c6df6b47bda180b1daaafc7ac8a32fd7f9af83a92569da41429da49ea5b.jpg
    ├── iris-36f2b9282342292b67f38a55a62b0c66fa4e5bb58587f7fec90d1e93ea8c407a.jpg
    ├── iris-37ad07fd7b39bc377fa6e9cafdb6e0c57fb77df2c264fe631705a8436c0c2513.jpg
    ├── iris-3ba1625bb78e4b69b114bdafcdab64104b211d8ebadca89409e9e7ead6a0557c.jpg
    ├── iris-4c5d9a33327db025d9c391aeb182cbe20cfab4d4eb4ac951cc5cd15e132145d8.jpg
    ├── iris-522f3eb1807d015f99e66e73b19775800712890f2c7f5b777409a451fa47d532.jpg
    ├── iris-589fa96b9a3c2654cf08d05d3bebf4ab7bc23592d7d5a95218f9ff87612992fa.jpg
    ├── iris-61b71f1de04a03ce719094b65179b06e3cd80afa01622b30cda8c3e41de6bfaa.jpg
    ├── iris-62ef719cd70780088a4c140afae2a96c6ca9c22b72b078e3b9d25678d00b88a5.jpg
    ├── iris-819130af42335d4bb75bebb0d2ee2e353a89a3d518a1d2ce69842859c5668c5a.jpg
    ├── iris-8669e4937a2003054408afd228d99cb737e9db5088f42d292267c43a3889001a.jpg
    ├── iris-86c76e0f331bc62192c392cf7c3ea710d2272a8cc9928d2566a5fc4559e5dce4.jpg
    ├── iris-8a8bc54332a42bb35ee131d7b64e9375b4ac890632eb09e193835b838172d797.jpg
    ├── iris-8e9439ec7231fa3b9bc9f62a67af4e180466b32a72316600431b1ec93e63b296.jpg
    ├── iris-90b7d491b9a39bb5c8bb7649cce90ab7f483c2759fb55fda2d9067ac9eec7e39.jpg
    ├── iris-9dededf184993455c411a0ed81d6c3c55af7c610ccb55c6ae34dfac2f8bde978.jpg
    ├── iris-9e6ce91679c9aaceb3e9c930f11e788aacbfa8341a2a5737583c14a4d6666f3d.jpg
    ├── iris-a0e65269f7dc7801ac1ad8bd0c5aa547a70c7655447e921d1d4d153a9d23815e.jpg
    ├── iris-a445b0720254984275097c83afbdb1fe896cb010b5c662a6532ed0601ea24d7c.jpg
    ├── iris-a6b85bf1f3d18bbb6470440592834c2c7f081b490836392cf5f01636ee7cf658.jpg
    ├── iris-b005c82b844de575f0b972b9a1797b2b1fbe98c067c484a51006afc4f549ada4.jpg
    ├── iris-bfcf79b3b527eb64b78f9a068a1000042336e532f0f44e68f818dd13ab492a76.jpg
    ├── iris-c156236fb6e888764485e796f1f972bbc7ad960fe6330a7ce9182922046439c4.jpg
    ├── iris-d99d5fd2de5be1419cbd569570dbb6c9a6c8ec4f0a1ff5b55dc2607f6ecdca8f.jpg
    ├── iris-d9aae37a8fa6afdef2af170c266a597925eea935f4d070e979d565713ea62642.jpg
    ├── iris-dbc87fcecade2c070baaf99caf03f4f0f6e3aa977e34972383cb94d0efe8a95d.jpg
    ├── iris-e3d1a560d25cf573d2cbbf2fe6cd231819e998109a5cf1788d59fbb9859b3be2.jpg
    ├── iris-ec288bdad71388f907457db2476f12a5cb43c28cfa28d2a2077398a42b948a35.jpg
    ├── iris-ed5b4e072d43bc53a00a4a7f4d0f5d7c0cbd6a006e9c2d463128cedc956cb3de.jpg
    ├── iris-f3018a9440d17c265062d1c61475127f9952b6fe951d38fd7700402d706c0b01.jpg
    ├── iris-f47c5963cdbaa3238ba2d446848e8449c6af83e663f0a9216cf0baba8429b36f.jpg
    ├── iris-fa4b6d7e3617216104b1405cda21bf234840cd84a2c1966034caa63def2f64f0.jpg
    ├── iris-fc4b0cc65387ff78471659d14a78f0309a76f4c3ec641b871e40b40424255097.jpg
    └── meta.csv

Our goal is to turn these directories into a validated & queryable dataset that can be used alongside many other datasets.

Register an artifact

LaminDB uses the Artifact class to manage datasets & models that are stored as files, folders, or arrays. Artifact is a registry to manage search, queries, validation & storage access.

It is possible to register local files or folders, AWS S3 paths ("s3://..."), Google Cloud Storage paths ("gs://...") and Hagging Face paths ("hf://..."). Here we work with an AWS S3 path.

Let’s create a Artifact object for one of the studies:

artifact = ln.Artifact(
    "s3://lamindata/iris_studies/study0_raw_images"
)
artifact
Hide code cell output 
! calling anonymously, will miss private instances

Artifact(uid='Ap7tVclB3xCkIZWf0000', is_latest=True, key='iris_studies/study0_raw_images', suffix='', size=658465, hash='IVKGMfNwi8zKvnpaD_gG7w', n_objects=51, _hash_type='md5-d', visibility=1, _key_is_virtual=False, storage_id=2, transform_id=1, run_id=1, created_by_id=1)
Which fields are populated when creating an artifact record?

Basic fields:

  • uid: universal ID

  • key: storage key, a relative path of the artifact in storage

  • description: an optional string description

  • storage: the storage location (the root, say, an S3 bucket or a local directory)

  • suffix: an optional file/path suffix

  • size: the artifact size in bytes

  • hash: a hash useful to check for integrity and collisions (is this artifact already stored?)

  • hash_type: the type of the hash (usually, an MD5 or SHA1 checksum)

  • created_at: time of creation

  • updated_at: time of last update

Provenance-related fields:

  • created_by: the User who created the artifact

  • transform: the Transform (pipeline, notebook, instrument, app) that was run

  • run: the Run of the transform that created the artifact

For a full reference, see Artifact.

Upon .save(), artifact metadata is written to the database:

artifact.save()
Hide code cell output 
Artifact(uid='Ap7tVclB3xCkIZWf0000', is_latest=True, key='iris_studies/study0_raw_images', suffix='', size=658465, hash='IVKGMfNwi8zKvnpaD_gG7w', n_objects=51, _hash_type='md5-d', visibility=1, _key_is_virtual=False, storage_id=2, transform_id=1, run_id=1, created_by_id=1, created_at=2024-11-07 12:39:16 UTC)
What happens during save?

In the database: A artifact record is inserted into the artifact registry. If the artifact record exists already, it’s updated.

In storage:

  • If the default storage is in the cloud, .save() triggers an upload for a local artifact.

  • If the artifact is already in a registered storage location, only the metadata of the record is saved to the artifact registry.

We can get an overview of all artifacts in the database by calling df():

ln.Artifact.df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 Ap7tVclB3xCkIZWf0000 None True None iris_studies/study0_raw_images None 658465 IVKGMfNwi8zKvnpaD_gG7w 51 None md5-d None 1 False 2 1 1 2024-11-07 12:39:16.350102+00:00 1

View data lineage

Visualize data lineage with view_lineage():

artifact.view_lineage()
Hide code cell output
_images/48fe5bab814cbed62bffde8f96e7aa00e52a5d0249b2ba7dc0157769637130b9.svg

Or directly access its linked Transform & Run records:

artifact.transform
Hide code cell output 
Transform(uid='NJvdsWWbJlZS0000', is_latest=True, name='Tutorial: Artifacts', key='tutorial.ipynb', type='notebook', created_by_id=1, created_at=2024-11-07 12:39:11 UTC)

artifact.run
Hide code cell output 
Run(uid='5oqBFfLRhVzSCKlPsPGu', started_at=2024-11-07 12:39:11 UTC, is_consecutive=True, transform_id=1, created_by_id=1, created_at=2024-11-07 12:39:11 UTC)

(For a comprehensive example with data lineage through UI uploads, pipelines & notebooks of multiple data types, see Project flow.)

Access an artifact

path gives you the file path, a UPath object:

artifact.path
Hide code cell output 
S3Path('s3://lamindata/iris_studies/study0_raw_images')

Typically, your artifact is in cloud storage - to cache it locally, call cache():

artifact.cache()
Hide code cell output 
PosixUPath('/home/runner/.cache/lamindb/lamindata/iris_studies/study0_raw_images')

If the data is large, you’ll likely want to query it via open() or shard the adata across many array-like artifacts. For more on this, see: Query arrays.

How do I update an artifact?

If you’d like to update metadata:

artifact.description = "My new description"
artifact.save()  # save the change to the database

If you’d like to replace the underlying stored object, use replace().

Filter & search artifacts

You can search artifacts directly based on the Artifact registry:

ln.Artifact.search("iris").df().head()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 Ap7tVclB3xCkIZWf0000 None True None iris_studies/study0_raw_images None 658465 IVKGMfNwi8zKvnpaD_gG7w 51 None md5-d None 1 False 2 1 1 2024-11-07 12:39:16.350102+00:00 1

You can also query & search the artifact by any metadata combination.

For instance, look up a user with auto-complete from the User registry:

users = ln.User.lookup()
users.anonymous
Hide code cell output 
User(uid='00000000', handle='anonymous', created_at=2024-11-07 12:39:06 UTC)
How do I act non-anonymously?

  1. Sign up for a free account (see more info) and copy the API key.

  2. Log in on the command line:

    lamin login <email> --key <API-key>

Filter the Transform registry for a name:

transform = ln.Transform.get(name__icontains="Artifacts")  # get exactly one result
transform
Hide code cell output 
Transform(uid='NJvdsWWbJlZS0000', is_latest=True, name='Tutorial: Artifacts', key='tutorial.ipynb', type='notebook', created_by_id=1, created_at=2024-11-07 12:39:11 UTC)
What does a double underscore mean?

For any field, the double underscore defines a comparator, e.g.,

  • name__icontains="Martha": name contains "Martha" when ignoring case

  • name__startswith="Martha": name starts with "Martha

  • name__in=["Martha", "John"]: name is "John" or "Martha"

For more info, see: Query & search registries.

Use these results to filter the Artifact registry:

ln.Artifact.filter(
    created_by=users.anonymous,
    transform=transform,
).df().head()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 Ap7tVclB3xCkIZWf0000 None True None iris_studies/study0_raw_images None 658465 IVKGMfNwi8zKvnpaD_gG7w 51 None md5-d None 1 False 2 1 1 2024-11-07 12:39:16.350102+00:00 1

You can also query for directories using key__startswith (LaminDB treats directories like AWS S3, as the prefix of the storage key):

ln.Artifact.filter(key__startswith="iris_studies/study0_raw_images").df().head()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
1 Ap7tVclB3xCkIZWf0000 None True None iris_studies/study0_raw_images None 658465 IVKGMfNwi8zKvnpaD_gG7w 51 None md5-d None 1 False 2 1 1 2024-11-07 12:39:16.350102+00:00 1

Note

You can look up, filter & search any registry (Record).

You can chain filter() statements and search(): ln.Artifact.filter(suffix=".jpg").search("my image")

An empty filter returns the entire registry: ln.Artifact.filter()

For more info, see: Query & search registries.

Filter & search on LaminHub

Describe artifacts

Get an overview of what happened:

artifact.describe()
Hide code cell output 
Artifact(uid='Ap7tVclB3xCkIZWf0000', is_latest=True, key='iris_studies/study0_raw_images', suffix='', size=658465, hash='IVKGMfNwi8zKvnpaD_gG7w', n_objects=51, _hash_type='md5-d', visibility=1, _key_is_virtual=False, created_at=2024-11-07 12:39:16 UTC)
  Provenance
    .storage = 's3://lamindata'
    .transform = 'Tutorial: Artifacts'
    .run = 2024-11-07 12:39:11 UTC
    .created_by = 'anonymous'

artifact.view_lineage()
Hide code cell output
_images/48fe5bab814cbed62bffde8f96e7aa00e52a5d0249b2ba7dc0157769637130b9.svg

Version artifacts

You can create new versions of artifacts, collections & transforms when you pass an older version to revises.

new_artifact = ln.Artifact(data, revises=old_artifact)

Alternatively, you can set a key to append to a version family in the same way you’d do it on AWS S3.

File-like artifacts

Above, we looked at a folder-like artifact. Here is a file-like artifact.

local_filepath = ln.core.datasets.file_jpg_paradisi05()
artifact = ln.Artifact(local_filepath, description="My single image").save()
artifact.load()
_images/8b5b1a910da8862bfc63129e335c264f8ce470b5b2152d26247f9a5df0d0e36f.jpg

Collections

Often times, several artifacts together represent a collection.

Let’s seed a growing Collection of artifacts:

collection = ln.Collection(
    artifact,
    name="Iris collection",
    description="Iris study 0",
)
collection.save()
Hide code cell output 
Collection(uid='0ySyD0ZdhjfhVxRH0000', is_latest=True, name='Iris collection', description='Iris study 0', hash='cCm_OMdLmd9D2zdZ7yoN1g', visibility=1, created_by_id=1, transform_id=1, run_id=1, created_at=2024-11-07 12:39:19 UTC)

Now, we collect more data in subsequent studies.

We want to keep track of their data as part of a growing versioned collection:

artifacts = [artifact]
for folder_name in ["study1_raw_images", "study2_raw_images"]:
    # create an artifact for the folder
    new_artifact = ln.Artifact(f"s3://lamindata/iris_studies/{folder_name}").save()
    artifacts.append(new_artifact)
    # create a new version of the collection
    collection = ln.Collection(
        artifacts, revises=collection, description=f"Now includes {folder_name}"
    )
    collection.save()

See all artifacts:

ln.Artifact.df()
Hide code cell output
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
4 PLPi2o1ySZxR619r0000 None True None iris_studies/study2_raw_images None 667449 BfyPwSCYPaHRKe4bJk7yRw 51.0 None md5-d None 1 False 2 1 1 2024-11-07 12:39:19.984330+00:00 1
3 g4JrI3AdtF7c8SYR0000 None True None iris_studies/study1_raw_images None 642480 Iip0GzbvjACYC2O7ZrtZiQ 49.0 None md5-d None 1 False 2 1 1 2024-11-07 12:39:19.850076+00:00 1
2 hOkZJTa0k5p3nxbf0000 None True My single image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g NaN None md5 None 1 True 1 1 1 2024-11-07 12:39:19.663486+00:00 1
1 Ap7tVclB3xCkIZWf0000 None True None iris_studies/study0_raw_images None 658465 IVKGMfNwi8zKvnpaD_gG7w 51.0 None md5-d None 1 False 2 1 1 2024-11-07 12:39:16.350102+00:00 1

See all collections:

ln.Collection.df()
Hide code cell output
uid version is_latest name description hash reference reference_type visibility transform_id meta_artifact_id run_id created_at created_by_id
id
3 0ySyD0ZdhjfhVxRH0002 None True Iris collection Now includes study2_raw_images mn1F-fZyCexIIz6ZvD2-NA None None 1 1 None 1 2024-11-07 12:39:19.995089+00:00 1
2 0ySyD0ZdhjfhVxRH0001 None False Iris collection Now includes study1_raw_images 9E8-h6W3X74nUHt3HnK2RQ None None 1 1 None 1 2024-11-07 12:39:19.859437+00:00 1
1 0ySyD0ZdhjfhVxRH0000 None False Iris collection Iris study 0 cCm_OMdLmd9D2zdZ7yoN1g None None 1 1 None 1 2024-11-07 12:39:19.701887+00:00 1

Most functionality that you just learned about artifacts - e.g., queries & provenance - also applies to Collection.

Collections become powerful if you directly leverage them for training models: Train a machine learning model on a collection.

View changes

With view(), you can see the latest changes to the database:

ln.view()  # link tables in the database are not shown
Hide code cell output 
Artifact
uid version is_latest description key suffix type size hash n_objects n_observations _hash_type _accessor visibility _key_is_virtual storage_id transform_id run_id created_at created_by_id
id
4 PLPi2o1ySZxR619r0000 None True None iris_studies/study2_raw_images None 667449 BfyPwSCYPaHRKe4bJk7yRw 51.0 None md5-d None 1 False 2 1 1 2024-11-07 12:39:19.984330+00:00 1
3 g4JrI3AdtF7c8SYR0000 None True None iris_studies/study1_raw_images None 642480 Iip0GzbvjACYC2O7ZrtZiQ 49.0 None md5-d None 1 False 2 1 1 2024-11-07 12:39:19.850076+00:00 1
2 hOkZJTa0k5p3nxbf0000 None True My single image None .jpg None 29358 r4tnqmKI_SjrkdLzpuWp4g NaN None md5 None 1 True 1 1 1 2024-11-07 12:39:19.663486+00:00 1
1 Ap7tVclB3xCkIZWf0000 None True None iris_studies/study0_raw_images None 658465 IVKGMfNwi8zKvnpaD_gG7w 51.0 None md5-d None 1 False 2 1 1 2024-11-07 12:39:16.350102+00:00 1 
Collection
uid version is_latest name description hash reference reference_type visibility transform_id meta_artifact_id run_id created_at created_by_id
id
3 0ySyD0ZdhjfhVxRH0002 None True Iris collection Now includes study2_raw_images mn1F-fZyCexIIz6ZvD2-NA None None 1 1 None 1 2024-11-07 12:39:19.995089+00:00 1
2 0ySyD0ZdhjfhVxRH0001 None False Iris collection Now includes study1_raw_images 9E8-h6W3X74nUHt3HnK2RQ None None 1 1 None 1 2024-11-07 12:39:19.859437+00:00 1
1 0ySyD0ZdhjfhVxRH0000 None False Iris collection Iris study 0 cCm_OMdLmd9D2zdZ7yoN1g None None 1 1 None 1 2024-11-07 12:39:19.701887+00:00 1 
! No records found

! No records found

Run
uid started_at finished_at is_consecutive reference reference_type transform_id report_id environment_id parent_id created_at created_by_id
id
1 5oqBFfLRhVzSCKlPsPGu 2024-11-07 12:39:11.244707+00:00 None True None None 1 None None None 2024-11-07 12:39:11.244775+00:00 1 
Storage
uid root description type region instance_uid run_id created_at created_by_id
id
2 yK5ZLDem7Bol s3://lamindata None s3 us-east-1 None None 2024-11-07 12:39:16.231945+00:00 1
1 P31JqwXA8qe5 /home/runner/work/lamindb/lamindb/docs/lamin-t... None local None 5WuFt3cW4zRx None 2024-11-07 12:39:06.073238+00:00 1 
Transform
uid version is_latest name key description type source_code hash reference reference_type _source_code_artifact_id created_at created_by_id
id
1 NJvdsWWbJlZS0000 None True Tutorial: Artifacts tutorial.ipynb None notebook None None None None None 2024-11-07 12:39:11.234641+00:00 1 
! No records found

User
uid handle name created_at
id
1 00000000 anonymous None 2024-11-07 12:39:06.069097+00:00

Save notebook & scripts

When you’ve completed the work on a notebook or script, you can save the source code and, for notebooks, an execution report to your storage location like so:

ln.finish()

This enables you to query execution report & source code via transform.latest_run.report and transform._source_code_artifact.

If you registered the instance on LaminHub, you can share it like here.

Get notebooks & scripts

If you want to cache a notebook or script, call:

lamin load https://lamin.ai/laminlabs/lamindata/transform/PtTXoc0RbOIq65cN

Read on

Now, you already know about 6 out of 9 LaminDB core classes! The two most central are:

And the four registries related to provenance:

  • Transform: transforms of artifacts

  • Run: runs of transforms

  • User: users

  • Storage: storage locations like S3/GCP buckets or local directories

If you want to validate data, label artifacts, and manage features, read on: Tutorial: Features & labels.