External dictionaries

Feedback
1

Describe the problem you’re experiencing?

There are a lot of small (<30k records) dictionaries kept in our current Greenplum database.
Most of the data that we load into MemSQL reference these dictionaries by ID, and in order to get the data from these dictionaries, we have to copy them into MemSQL with a separate ETL process for each dictionary.

What is your ideal solution? What are you looking for?

It would be quite handy to introduce “External dictionary” functionality into MemSQL, which is essestially a cached view referencing a table in an external database via JBDC/ODBC.
This way, we would not have to copy-paste all dictionaries into MemSQL and keep them updated.

There is a similar functionality available in Clickhouse DB, for the reference: https://clickhouse.yandex/docs/en/query_language/dicts/external_dicts/
What version(s) of MemSQL or related tools is this affecting?
All versions

2

Thanks for the feature request.

We are looking at adding “external table” support in a feature version which would allow querying other databases (or blob stores).

Likely the closest thing to a clickhouse dictionary in memsql today is the ENUM datatype (you can query it via enumCol = intValue yet select enmCol will return the string value of the ENUM).