Venice Single-Datacenter Docker Quickstart
Follow this guide to set up a simple venice cluster using docker images provided by Venice team.
Step 1: Install and set up Docker Engine
Follow https://docs.docker.com/engine/install/ to install docker and start docker engine
Step 2: Download Venice quickstart Docker compose file
wget https://raw.githubusercontent.com/linkedin/venice/main/quickstart/docker-compose.yaml
Step 3: Run docker compose
This will download and start containers for kafka, zookeeper, venice-controller, venice-router, venice-server, and venice-client. Once containers are up and running, it will create a test cluster, namely, venice-cluster
.
Note: Make sure the docker-compose.yaml
downloaded in step 2 is in the same directory from which you will run the following command.
docker compose up -d
Step 4: Access venice-client
container’s bash shell
From this container, we will create a store in venice-cluster
, which was created in step 3, push data to it and run queries against it.
docker exec -it venice-client bash
Step 5: Create a venice store
The below script uses venice-admin-tool
to create a new store: venice-store
. We will use the following key and value schema for store creation.
key schema:
{
"name": "id",
"type": "string"
}
value schema:
{
"name": "name",
"type": "string"
}
Let’s create a venice store:
bash create_store.sh
Step 6: Push data to the store
Venice supports multiple ways to write data to the store. For more details, please refer to Write Path section in README. In this example, we will use batch push mode and push 100 records.
key: 1 to 100
value: test_name_1 to test_name_100
Let’s push the data:
bash batch_push_data.sh
Step 7: Read data from the store
Let’s query some data from venice-store
, using venice-thin-client
which is included in venice container images. (key: 1 to 100
)
bash query_data.sh <key>
For example:
$ bash query_data.sh 1
key=1
value=test_name_1
$ bash query_data.sh 100
key=100
value=test_name_100
# Now if we do get on non-existing key, venice will return `null`
$ bash query_data.sh 101
key=101
value=null
Step 8: Update and add some new records using Incremental Push
Venice supports incremental push which allows us to update values of existing rows or to add new rows in an existing store. In this example, we will
- update values for keys from
51-100
. For example, the new value of100
will betest_name_100_v1
- add new rows (key:
101-150
)
bash inc_push_data.sh
Step 9: Read data from the store after Incremental Push
Incremental Push updated the values of keys 51-100 and added new rows 101-150. Let’s read the data once again.
# Value of 1 changed remains unchanged
$ bash query_data.sh 1
key=1
value=test_name_1
# Value of 100 changed from test_name_100 to test_name_100_v1
$ bash query_data.sh 100
key=100
value=test_name_100_v1
# Incremental Push added value for previously non-existing key 101
$ bash query_data.sh 101
key=101
value=test_name_101_v1
Step 10: Exit venice-client
exit
Step 11: Stop docker
Tear down the venice cluster
docker compose down
Next steps
Venice is a feature rich derived data store. It offers features such as write-compute, read-compute, streaming ingestion, multi data center active-active replication, deterministic conflict resolution, etc. To know more about such features please refer README and reach out to the Venice team.