Keep artifacts local in a cloud instance¶

If you want to default to keeping artifacts local in a cloud instance, enable keep_artifacts_local.

# !pip install 'lamindb[jupyter,aws]'
import lamindb as ln

ln.track("l9lFf83aPwRc0000")

→ connected lamindb: testuser1/keep-artifacts-local

→ notebook imports: lamindb==0.76.15

→ created Transform('l9lFf83a'), started new Run('Uc2WP5OH') at 2024-11-07 12:42:37 UTC

You can register a managed local storage location as follows:

ln.settings.storage_local = "./my_storage_local"

→ defaulting to local storage: /home/runner/work/lamindb/lamindb/docs/faq/my_storage_local

Now, you have two storage locations: one in the S3 bucket, and the other locally.

ln.Storage.df()

Show code cell output Hide code cell output

	uid	root	description	type	region	instance_uid	run_id	created_at	created_by_id
id
2	wRsX11D0OisR	/home/runner/work/lamindb/lamindb/docs/faq/my_...	None	local	None	6uGWmLpZlNoJ	None	2024-11-07 12:42:39.095740+00:00	1
1	PCUu60lwb6pa	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	None	2024-11-07 12:42:27.858789+00:00	1

Update storage description¶

You can add a description to the storage by using the description parameter:

storage_record = ln.Storage.get(root=ln.settings.storage_local.root)
storage_record.description = "Files stored locally in site X on server Y for reason ABC"
storage_record.save()
ln.Storage.df()

	uid	root	description	type	region	instance_uid	run_id	created_at	created_by_id
id
2	wRsX11D0OisR	/home/runner/work/lamindb/lamindb/docs/faq/my_...	Files stored locally in site X on server Y for...	local	None	6uGWmLpZlNoJ	None	2024-11-07 12:42:39.095740+00:00	1
1	PCUu60lwb6pa	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ	None	2024-11-07 12:42:27.858789+00:00	1

Use local storage¶

If you save an artifact, by default, it’s stored in local storage.

original_filepath = ln.core.datasets.file_fcs()
artifact = ln.Artifact(original_filepath, description="My fcs file").save()
local_path = artifact.path
local_path

You’ll see the .fcs file named by the uid in your .lamindb/ directory under ./my_storage_local/:

ln.settings.storage_local.root.view_tree()

If you’d like to upload an artifact, you pass upload=True to the save() method.

artifact.save(upload=True)

You now see the artifact in the S3 bucket:

ln.setup.settings.storage.root.view_tree()

And it’s no longer present in local storage:

ln.setup.settings.instance.storage_local.root.view_tree()

Direct upload¶

You can also directly upload a file by passing upload=True:

filepath = ln.core.datasets.file_mini_csv()
artifact2 = ln.Artifact(filepath, description="My csv file").save(upload=True)
artifact2.path

Show code cell output Hide code cell output

! record with similar description exists! did you mean to load it?

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
1	G9lSYv3x0wJWkcGP0000	None	True	My fcs file	None	.fcs	None	6785467	KCEXRahJ-Ui9Y6nksQ8z1A	None	None	md5	None	1	True	1	1	1	2024-11-07 12:42:39.495106+00:00	1

S3Path('s3://lamindb-ci/keep-artifacts-local/.lamindb/UVygdLsFJ3z8Loz40000.csv')

Now we have two files on S3:

ln.Artifact.df(include="storage__root")

Show code cell output Hide code cell output

	storage__root	uid	version	is_latest	description	key	suffix	type	size	hash	...	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
2	s3://lamindb-ci/keep-artifacts-local	UVygdLsFJ3z8Loz40000	None	True	My csv file	None	.csv	None	11	z1LdF2qN4cN0M2sXrcW8aw	...	None	md5	None	1	True	1	1	1	2024-11-07 12:42:41.124788+00:00	1
1	s3://lamindb-ci/keep-artifacts-local	G9lSYv3x0wJWkcGP0000	None	True	My fcs file	None	.fcs	None	6785467	KCEXRahJ-Ui9Y6nksQ8z1A	...	None	md5	None	1	True	1	1	1	2024-11-07 12:42:39.495106+00:00	1

2 rows × 21 columns

Pre-existing artifacts¶

Assume we already have a file in our registered local storage location:

1 sub-directory & 2 files with suffixes '', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── .lamindb/
│   └── _is_initialized
└── output.bam

If we create an artifact from it, it remains where it is during saving:

my_existing_file = ln.Artifact("./my_storage_local/output.bam", description="my existing file").save()
ln.UPath("my_storage_local/").view_tree()

Show code cell output Hide code cell output

! records with similar descriptions exist! did you mean to load one of them?

	uid	version	is_latest	description	key	suffix	type	size	hash	n_objects	n_observations	_hash_type	_accessor	visibility	_key_is_virtual	storage_id	transform_id	run_id	created_at	created_by_id
id
1	G9lSYv3x0wJWkcGP0000	None	True	My fcs file	None	.fcs	None	6785467	KCEXRahJ-Ui9Y6nksQ8z1A	None	None	md5	None	1	True	1	1	1	2024-11-07 12:42:39.495106+00:00	1
2	UVygdLsFJ3z8Loz40000	None	True	My csv file	None	.csv	None	11	z1LdF2qN4cN0M2sXrcW8aw	None	None	md5	None	1	True	1	1	1	2024-11-07 12:42:41.124788+00:00	1

1 sub-directory & 2 files with suffixes '', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── .lamindb/
│   └── _is_initialized
└── output.bam

The storage path of the artifact is constructed using key because key_is_virtual=False:

my_existing_file

However, if we decide to upload the artifact, we’ll use the uid for constructing the storage path and switch key_is_virtual=True:

my_existing_file.save(upload=True)

Here is the remote path of the artifact:

my_existing_file.path

And here are the contents of the storage locations:

# the path on S3
ln.setup.settings.storage.root.view_tree()
# the local path
ln.setup.settings.instance.storage_local.root.view_tree()

Delete the test instance¶

Delete the artifacts:

artifact.delete(permanent=True)
artifact2.delete(permanent=True)
my_existing_file.delete(permanent=True)

Delete the instance:

ln.setup.delete("keep-artifacts-local", force=True)