Development environment
The Metabase application has two basic components:
- A backend written in Clojure which contains a REST API as well as all the relevant code for talking to databases and processing queries.
- A frontend written as a Javascript single-page application which provides the web UI.
Both components are built and assembled together into a single JAR file. In the directory where you run the JAR, you can create a JAR file (if Metabase hasn’t already created it) and add drivers in there (the drivers are also JARs).
Quick start
To spin up a development environment, you’ll need to start two terminal sessions: one for the frontend and one for the backend.
Frontend
Metabase depends on third-party libraries to run, so you’ll need to keep those up to date. The Clojure CLI will automatically fetch the dependencies when needed. With JavaScript dependencies, however, you’ll need to kick off the installation process manually.
# javascript dependencies
$ yarn
Start the frontend build process with
yarn build-hot
See Frontend development.
Backend
Run your backend development server with
clojure -M:run
See backend development.
Frontend development
We use these technologies for our FE build process to allow us to use modules, es6 syntax, and css variables.
- webpack
- babel
- cssnext
Frontend tasks are executed using yarn
. All available tasks can be found in package.json
under scripts.
To build the frontend client without watching for changes, you can use:
$ yarn build
If you’re working on the frontend directly, you’ll most likely want to reload changes on save, and in the case of React components, do so while maintaining state. To start a build with hot reloading, use:
$ yarn build-hot
Note that at this time if you change CSS variables, those changes will only be picked up when a build is restarted.
There is also an option to reload changes on save without hot reloading if you prefer that.
$ yarn build-watch
Some systems may have trouble detecting changes to frontend files. You can enable filesystem polling by uncommenting the watchOptions
clause in webpack.config.js
. If you do this it may be worth making git ignore changes to webpack config, using git update-index --assume-unchanged webpack.config.js
We exclude ESLint loader in dev mode for seven times quicker initial builds by default. You can enable it by exporting an environment variable:
$ USE_ESLINT=true yarn build-hot
By default, these build processes rely on a memory cache. The build process with ESLint loader enabled uses a large amount of memory and may take a considerable amount of time to start (1 - 2 minutes or more). FE developers (or anyone else who frequently restarts FE builds) are encouraged to use webpack’s filesystem cache option for much better start-up performance:
$ FS_CACHE=true yarn build-hot
When using FS_CACHE=true
you may need to remove the node_modules/.cache
directory to fix scenarios where the build may be improperly cached, and you must run rm -rf node_modules/.cache
in order for the build to work correctly when alternating between open source and enterprise builds of the codebase.
Frontend testing
Run all unit and Cypress end-to-end tests with
yarn test
Cypress tests and some unit tests are located in frontend/test
directory. New unit test files are added next to the files they test.
If you are using FS_CACHE=true
, you can also use FS_CACHE=true
with yarn test
.
Frontend debugging
By default, we use a simple source mapping option that is optimized for speed.
If you run into issues with breakpoints, especially inside jsx, please set env variable BETTER_SOURCE_MAPS
to true before you run the server.
Example:
BETTER_SOURCE_MAPS=true yarn dev
Cypress end-to-end tests
End-to-end tests simulate realistic sequences of user interactions. Read more about how we approach end-to-end testing with Cypress.
Cypress end-to-end tests use an enforced file naming convention <test-suite-name>.cy.spec.js
to separate them from unit tests.
Jest unit tests
Unit tests are focused around isolated parts of business logic.
Unit tests use an enforced file naming convention <test-suite-name>.unit.spec.js
to separate them from end-to-end tests.
yarn test-unit # Run all tests at once
yarn test-unit-watch # Watch for file changes
Backend development
Clojure REPL is the main development tool for the backend. There are some directions below on how to setup your REPL for easier development.
And of course your Jetty development server is available via
clojure -M:run
Building drivers
Most of the drivers Metabase uses to connect to external data warehouse databases are separate projects under the
modules/
subdirectory. When running Metabase via clojure
, you’ll need to build these drivers in order to have access
to them. You can build drivers as follows:
# Build the 'mongo' driver
./bin/build-driver.sh mongo
(or)
# Build all drivers
./bin/build-drivers.sh
Including driver source paths for development or other tasks
For development when running various Clojure tasks you can add the drivers
and drivers-dev
aliases to merge the
drivers’ dependencies and source paths into the Metabase project:
# Install dependencies, including for drivers
clojure -P -X:dev:ci:drivers:drivers-dev
Running unit tests
Run unit tests with
# OSS tests only
clojure -X:dev:test
# OSS + EE tests
clojure -X:dev:ee:ee-dev:test
or a specific test (or test namespace) with
# run tests in only one namespace (pass in a symbol)
clojure -X:dev:test :only metabase.api.session-test
# run one specific test (pass in a qualified symbol)
clojure -X:dev:test :only metabase.api.session-test/my-test
# run tests in one specific folder (test/metabase/util in this example)
# pass arg in double-quotes so Clojure CLI interprets it as a string;
# our test runner treats strings as directories
clojure -X:dev:test :only '"test/metabase/util"'
Testing drivers
By default, the tests only run against the h2
driver. You can specify which drivers to run tests against with the env var DRIVERS
:
DRIVERS=h2,postgres,mysql,mongo clojure -X:dev:drivers:drivers-dev:test
Some drivers require additional environment variables when testing since they are impossible to run locally (such as Redshift and Bigquery). The tests will fail on launch and let you know what parameters to supply if needed.
Running the linters
clj-kondo
must be installed separately.
# Run Eastwood
clojure -X:dev:ee:ee-dev:drivers:drivers-dev:eastwood
# Run the namespace checker
clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test:namespace-checker
# Run clj-kondo
clj-kondo --parallel --lint src shared/src enterprise/backend/src --config .clj-kondo/config.edn
Continuous integration
All frontend and backend linters and tests can be executed with
$ yarn ci
It is also possible to execute front-end and back-end checks separately
$ yarn ci-frontend
$ yarn ci-backend