Package com.linkedin.davinci.compression

Class UrnDictV1

java.lang.Object

com.linkedin.davinci.compression.UrnDictV1

public class UrnDictV1
extends Object

This class implements a URN dictionary for encoding and decoding URNs. The dictionary maps URN types to integer IDs to reduce the size of URNs when stored in a data store or transmitted over the network.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	UrnDictV1.EncodedUrn

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_MAX_DICT_SIZE
static final RecordDeserializer<Object>	DICT_DESERIALIZER
static final org.apache.avro.Schema	DICT_SCHEMA
static final RecordSerializer<Object>	DICT_SERIALIZER
static final int	UNKNOWN_URN_TYPE_ID
static final String	URN_PREFIX
static final int	URN_PREFIX_LENGTH
static final Character	URN_SEPARATOR

Method Summary

Modifier and Type	Method	Description
String	decodeUrn(UrnDictV1.EncodedUrn urn)
UrnDictV1.EncodedUrn	encodeUrn(String urn)
UrnDictV1.EncodedUrn	encodeUrn(String urn, boolean updateDictByUnknownUrn)
Map<String,Integer>	getUrnToIdMap()
static UrnDictV1	loadDict(byte[] dict, int maxDictSize)
static UrnDictV1	loadDict(ByteBuffer dict)
static UrnDictV1	loadDict(ByteBuffer dict, int maxDictSize)
static UrnDictV1	loadDict(Map<String,Integer> dict)
static UrnDictV1	loadDict(Map<String,Integer> dict, int maxDictSize)
byte[]	serializeDict()
static UrnDictV1	trainDict(List<String> urns, int dictSize)
static void	validateDict(Map<String,Integer> dict)	This function will validate the dictionary in the following ways: 1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_MAX_DICT_SIZE

public static final int DEFAULT_MAX_DICT_SIZE

See Also:

• Constant Field Values

URN_PREFIX

public static final String URN_PREFIX

See Also:

• Constant Field Values

URN_SEPARATOR

public static final Character URN_SEPARATOR

URN_PREFIX_LENGTH

public static final int URN_PREFIX_LENGTH

UNKNOWN_URN_TYPE_ID

public static final int UNKNOWN_URN_TYPE_ID

See Also:

• Constant Field Values

DICT_SCHEMA

public static final org.apache.avro.Schema DICT_SCHEMA

DICT_SERIALIZER

public static final RecordSerializer<Object> DICT_SERIALIZER

DICT_DESERIALIZER

public static final RecordDeserializer<Object> DICT_DESERIALIZER

Method Details

validateDict

public static void validateDict(Map<String,Integer> dict)

This function will validate the dictionary in the following ways: 1. Make sure the dictionary is not null. 2. Make sure the mapped ids are in the range of [0, dict.size() - 1). 3. Make sure there are no duplicate ids. When validation fail, it will throw an VeniceException.

getUrnToIdMap

public Map<String,Integer> getUrnToIdMap()

loadDict

public static UrnDictV1 loadDict(Map<String,Integer> dict)

loadDict

public static UrnDictV1 loadDict(Map<String,Integer> dict,
 int maxDictSize)

loadDict

public static UrnDictV1 loadDict(ByteBuffer dict)

loadDict

public static UrnDictV1 loadDict(ByteBuffer dict,
 int maxDictSize)

loadDict

public static UrnDictV1 loadDict(byte[] dict,
 int maxDictSize)

serializeDict

public byte[] serializeDict()

trainDict

public static UrnDictV1 trainDict(List<String> urns,
 int dictSize)

encodeUrn

public UrnDictV1.EncodedUrn encodeUrn(String urn)

encodeUrn

public UrnDictV1.EncodedUrn encodeUrn(String urn,
 boolean updateDictByUnknownUrn)

decodeUrn

public String decodeUrn(UrnDictV1.EncodedUrn urn)