Driver Interface Changelog
Metabase 0.46.0
-
The process for building a driver has changed slightly in Metabase 0.46.0. Your build command should now look something like this:
# Example for building the driver with bash or similar # switch to the local checkout of the Metabase repo cd /path/to/metabase/repo # get absolute path to the driver project directory DRIVER_PATH=`readlink -f ~/sudoku-driver` # Build driver. See explanation in sample Sudoku driver README clojure \ -Sdeps "{:aliases {:sudoku {:extra-deps {com.metabase/sudoku-driver {:local/root \"$DRIVER_PATH\"}}}}}" \ -X:build:sudoku \ build-drivers.build-driver/build-driver! \ "{:driver :sudoku, :project-dir \"$DRIVER_PATH\", :target-dir \"$DRIVER_PATH/target\"}"
Take a look at our build instructions for the sample Sudoku driver for an explanation of the command.
Note that while this command itself is quite a lot to type, you no longer need to specify a
:build
alias in your driver’sdeps.edn
file.Please upvote https://ask.clojure.org/index.php/7843/allow-specifying-aliases-coordinates-that-point-projects , which will allow us to simplify the driver build command in the future.
-
The multimethod
metabase.driver/table-rows-sample
has been added. This method is used in situations where Metabase needs a limited sample from a table, like when fingerprinting. The default implementation defined in themetabase.db.metadata-queries
namespace runs an MBQL query using the regular query processor to produce the sample rows. This is good enough in most cases, so this multimethod should not be implemented unless really necessary. Currently, the only case when a special implementation is used is for BigQuery, which does not respect limit clauses. -
The multimethod
metabase.driver.sql.query-processor/datetime-diff
has been added. This method is used by implementations of->honeysql
for the:datetime-diff
clause. It is recommended to implement this if you want to use the default SQL implementation of->honeysql
for the:datetime-diff
, which includes validation of argument types across all units. -
The multimethod
metabase.query-processor.util.add-alias-info/field-reference
has been added. This method is used to produce a reference to a field by theadd-alias-info
middleware. (Note that this middleware is optional, currently it is only used by the SQL and MongoDB drivers.) The default implementation returns the name of the field instance. It should be overridden if just the name is not a valid a valid reference. For example, MongoDB supports nested documents and references to nested fields should contain the whole path. See the namespacemetabase.driver.mongo.query-processor
for an alternative implementation. -
The multimethod
metabase.driver.sql-jdbc.sync.interface/syncable-schemas
(aliased asmetabase.driver.sql-jdbc.sync/syncable-schemas
), which was deprecated in 0.43.0, has been removed. Implementmetabase.driver.sql-jdbc.sync.interface/filtered-syncable-schemas
instead. See 0.43.0 notes below for more details. -
The multimethod
metabase.driver/format-custom-field-name
, which was deprecated in 0.42.0, has been removed. Implementmetabase.driver/escape-alias
instead. See 0.42.0 notes below for more information. -
The multimethod
metabase.driver.sql-jdbc.execute/read-column
, which was deprecated in 0.35.0, has been removed. Implementmetabase.driver.sql-jdbc.execute/read-column-thunk
instead. See 0.35.0 notes below for more information.
Honey SQL 2
The following only applies to SQL drivers; you can ignore it for non-SQL drivers.
Prior to Metabase 0.46.0, SQL drivers used Honey SQL 1 as an intermediate target when compiling queries. In 0.46.0 we have began the process of migrating to Honey SQL 2 as our new intermediate target.
We plan to continue to support use of Honey SQL 1 until Metabase 0.49.0. Please be sure to migrate your drivers before then.
In Metabase 0.46.x, 0.47.x, and 0.48.x, you can specify which version of Honey SQL you driver should use by
implementing the metabase.driver.sql.query-processor/honey-sql-version
multimethod:
(require '[metabase.driver.sql.query-processor :as sql.qp])
;;; use Honey SQL 2 for :my-driver
(defmethod sql.qp/honey-sql-version :my-driver
[_driver]
2)
This method must return either 1
or 2
. Currently, the default implementation returns 1
. Effectively this means
you currently have to opt-in to Honey SQL 2 compilation. It’s a good idea to do this sooner rather than later so your
driver is prepared for 0.49.0 well in advance.
In Metabase 0.47.x or 0.48.x we will likely change the default Honey SQL version to 2
to ensure everyone is aware of
the upcoming breaking changes in 0.49.0 and give them one or two release cycles to update their drivers to target
Honey SQL 2. You will still be able to opt-in to using Honey SQL 1 until 0.49.0 by implementing
sql.qp/honey-sql-version
and returning 1
.
What You Need to Change
Our Honey SQL utility namespace, metabase.util.honeysql-extensions
, commonly aliased as hx
, has been updated to
generate forms appropriate for either Honey SQL 1 or Honey SQL 2. This is done automatically based on your driver’s
honey-sql-version
. metabase.driver.sql.query-processor
itself also supports both targets in the same way.
The actual changes you will need to make to your driver code will probably be fairly small. The most important things to note when porting your driver:
-
Avoid use of things from Honey SQL 1 namespaces like
honeysql.core
orhoneysql.format
. If you must, use Honey SQLhoney.sql
instead; you may not need either. -
While you can continue to use
metabase.util.honeysql-extensions
in the short term, since it can target either version of Honey SQL, we will probably remove this namespace at some point in the future. Update your code to usemetabase.util.honey-sql-2
instead. The namespaces implement an almost identical set of helper functions, so all you should need to switch is which one you:require
in yourns
form. -
honeysql.core/call
no longer exists; instead of a form like(hsql/call :my_function 1 2)
, you simply return a plain vector like[:my_function 1 2]
.(hsql/raw "x")
is now[:raw "x"]
. New handlers can be registered with Honey SQL 2 withhoney.sql/register-fn!
. There is no equivalent of the Honey SQL 1honeysql.format./ToSql
protocol, so you should no longer define one-off types to implement custom SQL compilation rules. Usehoney.sql/register-fn!
instead. -
In Honey SQL 1 you were able to register functions to a more limited extent by implementing the multimethod
honeysql.format/fn-handler
. Metabase registered the functions:extract
,:distinct-count
, and:percentile-cont
in this way. For Honey SQL 2, we’ve registered these functions as qualified keywords in themetabase.util.honey-sql-2
namespace, to prevent confusion as to where they’re defined. Thus you’ll need to update the keyword if you’re using these functions.;;; Honey SQL 1 (hsql/call :distinct-count expr)
becomes
;;; Honey SQL 2 (require '[metabase.util.honey-sql-2 :as h2x]) [::h2x/distinct-count expr]
-
Because custom expressions are now just plain vectors like
[:my_function 1]
, you may need to wrap expressions in an additional vector if they appear inside:select
,:from
, or other places where a vector could be interpreted as[expression alias]
. e.g.;; Honey SQL 1 (honeysql.core/format {:select [[:my_function 1]]}) ;; => ["SELECT my_function AS 1"] ;; Honey SQL 2 ;; ;; WRONG (honey.sql/format {:select [[:my_function 1]]}) ;; => ["SELECT my_function AS ?" 1] ;; CORRECT (honey.sql/format {:select [[[:my_function 1]]]}) ;; => ["SELECT MY_FUNCTION(?)" 1]
The SQL query processor does this automatically for forms it generates, so you only need to worry about this if you’re overriding the way it generates
:select
or other top-level clauses. -
Numbers are parameterized by default, e.g.
{:select [1]}
becomesSELECT ?
rather thanSELECT 1
. You can use:inline
to force the SQL to be generated inline instead:{:select [[[:inline 1]]]}
becomesSELECT 1
. Numbers generated by the SQL query processor code should automatically be inlined, but you may need to make sure any numbers you generate are wrapped in:inline
if they can end up as expressions inside aGROUP BY
clause. Some databases can recognize expressions as being the same thing only when they are not parameterized:-- This is okay SELECT x + 1 FROM table GROUP BY x + 1 -- Bad: DB doesn't know whether the two x + ? expressions are the same thing SELECT x + ? FROM table GROUP BY x + ?
Exercise caution when :inline
ing things – take care not to use it on untrusted strings or other avenues for SQL
injection. Only inlining things that are a number?
is a safe bet.
Please read Differences between Honey SQL 1.x and 2.x for more information on the differences between the library versions.
Breaking Changes in 0.46.0 related to the Honey SQL 2 transition
Note: these breaking changes will hopefully be fixed before 0.46.0 ships. This will be updated if they are.
The classes metabase.util.honeysql_extensions.Identifer
and metabase.util.honeysql_extensions.TypedHoneySQLForm
have been moved to metabase.util.honey_sql_1.Identifer
and metabase.util.honey_sql_1.TypedHoneySQLForm
,
respectively. On the off chance that your driver directly referencing these class names, you may need to update things
to use the new class names.
Similarly, metabase.util.honeysql-extensions/->AtTimeZone
has been removed; use
metabase.util.honeysql-extensions/at-time-zone
instead.
Metabase 0.45.0
metabase.driver.sql-jdbc.connection/details->connection-spec-for-testing-connection
has been removed in Metabase 0.45.0, because it leaked SSH tunnels. See #24445. If you are using this function, please update your code to usemetabase.driver.sql-jdbc.connection/with-connection-spec-for-testing-connection
instead, which properly cleans up after itself.
New methods
-
metabase.driver.sql-jdbc.sync.describe-table-fields
has been added. Implement this method if you want to override the default behavior for fetching field metadata (such as types) for a table. -
->honeysql [<driver> :convert-timezone]
has been added. Implement this method if you want your driver to support theconvertTimezone
expression. This method takes 2 or 3 arguments and returns atimestamp without time zone
column.
Metabase 0.43.0
-
The
:expressions
map in an MBQL query now uses strings as keys rather than keywords (see #14647). You only need to be concerned with this if you are accessing or manipulating this map directly. Drivers deriving from:sql
implementing->honeysql
for[<driver> :expression]
may need to be updated. A utility function,metabase.mbql.util/expression-with-name
, has been available since at least Metabase 0.35.0 and handles both types of keys. It is highly recommended that you use this function rather than accessing:expressions
directly, as doing so can make your driver compatible with both 0.42.0 and with 0.43.0 and newer. -
There is now a
describe-nested-field-columns
method undersql-jdbc.sync
namespace which returns an instance of NestedFCMetadata. This is in order to allow JSON columns in Postgres and eventually other DB’s which are usually ordinary RDBMS’s but then sometimes they have a denormalized column with JSON or some other semantics. Given a table with denormalized columns which have nested field semantics (so, typed sub-fields which are still denormalized but stable in type between rows), return value should be a NestedFCMetadata, a map of flattened key paths to the detected sub-field. Field detection in syncing will then be enriched with those nested types. This is materially different from the way we do it for mongo because every kind of JSON column is different, but it’s going to run every sync so it can’t be too slow, even on enormous tables and enormous denormalized columns on those enormous tables.
Metabase 0.42.0
Changes in Metabase 0.42.0 affect drivers that derive from :sql
(including :sql-jdbc
).
Non-SQL drivers likely will require no changes.
0.42.0 introduces several significant changes to the way the SQL query processor compiles and determines aliases for
MBQL :field
clauses. For more background, see pull request
#19384.
If you were manipulating Field or Table aliases, we consolidated a lot of overlapping vars and methods, which means you may need to delete deprecated method implementations.
Significant changes
-
The
metabase.driver.sql.query-processor/->honeysql
method for Field instances, e.g.(defmethod sql.qp/->honeysql [:my-driver (class Field)] [driver field] ...)
is no longer invoked. All compilation is now handled by the MBQL
:field
clause method, e.g.(defmethod sql.qp/->honeysql [:my-driver :field] [driver field-clause] ...)
If you were doing something special here, you’ll need to move that special login into
[<driver> :field]
instead. (You may no longer need this special logic, however – see below.) -
:field
,:expression
, and:aggregation-options
clauses now contain information about what aliases you should use to refer to them on both the left-hand side and right-hand side of SQLAS
or elsewhere in a query. See PR #19610 for a detailed discussion about the new information, hereafter referred to as the /#19610 information/. -
If you have a custom implementation of
->honeysql
for:field
or(class Field)
:->honeysql
methods for:field
should use or replace the #19610 information rather than attempting to determine or override it in some other fashion.
New methods
-
metabase.driver/escape-alias
(moved frommetabase.driver.sql.query-processor/escape-alias
, which was introduced in 0.41.0) is now used to generate the #19610 information and used consistently across the SQL QP code. If you need to transform generated Field aliases for any reason (such as escaping disallowed characters), implement this method. -
metabase.driver.sql-jdbc.sync.interface/filtered-syncable-schemas
has been added, and will eventually replacemetabase.driver.sql-jdbc.sync.interface/syncable-schemas
. It serves a similar purpose, except that it’s also passed the inclusion and exclusion patterns (ex:auth*,data*
) to further filter schemas that will be synced.
Deprecated methods and vars
The following methods and vars are slated for removal in Metabase 0.45.0 unless otherwise noted.
-
metabase.driver/format-custom-field-name
is now unused. Implementmetabase.driver/escape-alias
instead. -
metabase.driver.sql.query-processor/escape-alias
has been renamed tometabase.driver/escape-alias
. Everything else is the same. -
metabase.driver.sql.query-processor/field-clause->alias
no longer uses the optional parameterunique-name-fn
. Aliases are now made unique automatically, after escaping them; implementmetabase.driver/escape-alias
if you need to do something special before they are made unique. (Unique aliases are also escaped a second time if you need to do something /really/ special.) -
metabase.driver.sql.query-processor/field->alias
, which was deprecated in 0.41.0, is now unused in 0.42.0. Implementing this method no longer has any effect. Implementmetabase.driver/escape-alias
instead if you need to do something special; use the #19610 information if you need to escape the alias for one reason or another. This method is still slated for removal in Metabase 0.44.0. -
metabase.driver.sql.query-processor/*field-options*
is now unused and is no longer bound automatically. If you need field options for some reason, see our SQL Server driver for an example on how to create it. -
metabase.driver.sql.query-processor/*table-alias*
is now unused and is no longer bound automatically. Use or override:metabase.query-processor.util.add-alias-info/source-table
from the #19610 information instead. -
metabase.driver.sql.query-processor/*source-query*
is now unused and is no longer bound automatically. Usemetabase.driver.sql.query-processor/*inner-query*
instead, which is always bound, even if we aren’t inside of a source query. -
metabase.driver.sql.query-processor/field->identifier
is now unused. Implementing this method should no longer be necessary under any circumstances. Override->honeysql
for[<driver> :field]
and manipulate the #19610 information if you need to do something special here. metabase.driver.sql.query-processor/prefix-field-alias
is no longer used. Previously, it was made available to give drivers a chance to escape automatically generated aliases for joined Fields. This is no longer necessary, becausemetabase.driver/escape-alias
is called on automatically generates aliases. Implementmetabase.driver/escape-alias
if you need to do something special.metabase.driver.sql-jdbc.sync.interface/syncable-schemas
has been deprecated in favor ofmetabase.driver.sql-jdbc.sync.interface/filtered-syncable-schemas
(see above). The existing default implementation ofsyncable-schemas
currently callsfiltered-syncable-schemas
(withnil
filters, i.e. the filtering operation is actually a no-op), but it will eventually be removed.
Removed Methods and Vars
metabase.mbql.schema/DatetimeFieldUnit
, deprecated in 0.39.0, is now removed.
Older versions
Before 0.42.0, this information was tracked in our Wiki. You can find changes for versions before 0.42.0 in the table below:
Version | Wiki page |
---|---|
0.41.0 | changes |
0.40.0 | No changes. |
0.39.0 | No changes. |
0.38.0 | changes |
0.37.0 | changes |
0.36.0 | changes |
0.35.0 | changes |