Configuring with URLs

Cells configuration for all storages is done using a "scheme-based" pattern, that fully describes a given driver used by Cells.

Flags and URLs

Let's have a look at some of the available Flags advertised by the command (output below is filtered):

% CELLS_DISPLAY_HIDDEN_FLAGS=true ./cells start --help

Flags:
      --broker string                Pub/sub service URL for events broadcast. Supported schemes: grpc|mem|nats|rabbit (default "mem://")
      --cache string                 Sharded Cache (default "bigcache://")
      --discovery string             Combine registry, config and pub/sub discovery service (default "mem://")
      --registry string              Registry URL used to manage services. Supported schemes: etcd|file|grpc|mem (default "mem://?cache=shared")
      --shortcache string            Short cache (default "pm://")

Global Flags:
      --certs_store string   Certificates Store URL. Can be switched to vault://host:port/secretPath (default "file://${CELLS_WORKING_DIR}/certs")
      --config string        Configuration storage URL. Supported schemes: etcd|file|grpc|mem|vault|vaults (default "file://${CELLS_WORKING_DIR}/pydio.json")
      --keyring string       Keyring URL. Can be switched to vault://host:port/secretPath?key=storeKey (default "file:///${CELLS_WORKING_DIR}/cells-vault-key?keyring=true")
      --vault string         Vault location, automatically detected from config url, unless an URL is provided (same schemes as config) (default "detect")

As you can see, most default values use a "mem://" (a.k.a memory) or "file://" scheme. This shows that data attached to these store will be maintained in memory or on-file. If we want one of these store to another technology, we can use one of the 'supported scheme' to pass a connection string and additional informations required by Cells with a different URL. For example, assuming an ETCD cluster is running and maintained aside in the cluster, we can easily start Cells with

% ./cells start --registry=etcd://:2379

Or as all our Flags can be configured by their corresponding CELLS_[FLAG_NAME_UPPER_CASE] environment variable, we could use :

% export CELLS_REGISTRY=etcd://:2379
% ./cells start

Supported Schemes

We will explain below the supported schemes for each flag

Registry

The registry maintains a list of all Cells services (started or not) available in the cluster.

Flag --registry or environment CELLS_REGISTRY

Scheme	Example	Comment
mem	mem://?cache=shared	This is the default. It provides an in-memory registry that is exposed via gRPC by the pydio.grpc.registry service
etcd	etcd://:2379/[prefix]	Instead storing registry state in-memory, store it in an ETCD K/V store. This should be the default in Cluster mode.
grpc	grpc://:8002	This is the default for fork processes (whether the main process is using mem:// or etcd://). Connects to a running pydio.grpc.registry locally on the 8002 port

Broker

The broker provides Publish/Subscribe pattern for broadcasting events.

Flag --broker or environment CELLS_BROKER

Scheme	Example	Comment
mem	mem://	This is the default. It uses google cloud golang pub/sub library under the hood. It exposes itself as gRPC vie the pydio.grpc.broker service
nats	nats://:4222/[prefix]	Connect to a Nats.io cluster for messaging. This should be the default in Cluster mode.
grpc	grpc://:8002	This is the default for fork processes (whether the main process is using mem:// or nats://). Connects to a running pydio.grpc.broker service locally on the 8002 port

Config, Vault, Certificates

A watchable key/value API that is common to config and vault flags.

Flag --config or environment CELLS_CONFIG : unencrypted configuration, as they can be found in the pydio.json file
Flag --vault or environment CELLS_VAULT : specific configurations that must be encrypted using a master key (stored in the Keyring, see below).
Flag --certs_store or environment CELLS_CERTS_STORE : Encrypted store for Caddy to store generated certificates. Uses a master key (stored in the Keyring, see below).

Scheme	Example	Comment
file	file:///{path/to/pydio.json}	This is the default. Stores configurations as JSON inside a local file.
grpc	grpc://:8002	This is the default for fork processes. Connects to a running pydio.grpc.config service locally on the 8002 port
etcd	etcd://:2379/[prefix]	Used in cluster mode writing/reading configuration from an ETCD cluster
vault	vault://localhot:8200/[prefix]	Used in cluster mode for encrypted values, points to a Hashicorp Vault key/value secret storage exposed on https://localhost:8200
vaults	vaults://localhot:8200/[prefix]	Same as previous but with TLS connection

Discovery

The discovery flag is simply a shortcut for --broker, --registry and --config to shorten the fork processes command.
If you look at the cells process, you will see all forks running with --discovery grpc://:8002, as the main process on a given machine always starts a set of grpc services to expose configs, broker and registry.

Cache

Caching layer to heavy data requests.

Flag --cache or environment CELLS_CACHE

Scheme	Example	Comment
bigcache	bigcache://	Uses BigCache Go library (in memory, with pre-allocation of memory segments).
pm	pm://	Uses patrickmm/cache Go library (in memory, simple synchronised maps)
redis	redis://:6379	Connects to a Redis cluster for caching. Should be the default for Cluster mode.

Keyring

As explained above, Vault and Certificates Store require a master key for encrypting data. This one is stored in the keyring.

Flag --keyring or environment CELLS_KEYRING

Scheme	Example	Comment
file	file://{path-to-cells_vault_key}	Stores generated key on-file
vault	vault://localhost:8200/kv	Stores keyring data inside HashiCorp vault

TLS Connection with client certificates

TLS connections can be configured to secure the communication between Cells and external services.

You first need to import the certificate via the command line to make it available to the application :

$ cells admin cert import --uuid my-client-cert myclientcert.pem
$ cells admin cert import --uuid my-client-cert-key myclientcertkey.pem
$ cells admin cert import --uuid my-client-rootca myclientrootca.pem

The following external systems can be configured with TLS :

Scheme	Parameter used	Example
`mysql+tls`	mysql dsn (cells configure / pydio.json)	`mysql+tls://root@tcp(localhost:3306)/cells` `?parseTime=true&ssl=true&tlsCertCAUUID=my-client-rootca&tlsCertKeyUUID=my-client-cert-key&tlsCertUUID=my-client-cert`
`mongodb`	mongodb dsn (cells configure / pydio.json)	`mongodb://cells-mongodb.cells.svc.cluster.local:27017/cells` `?ssl=true&tlsCertCAUUID=my-client-rootca&tlsCertKeyUUID=my-client-cert-key&tlsCertUUID=my-client-cert`
`etcd+tls`	--registry / --config (cells start flag) CELLS_REGISTRY / CELLS_BROKER (environment variable)	`etcd+tls://:2379?ssl=true&tlsCertCAUUID=my-client-rootca&tlsCertKeyUUID=my-client-cert-key&tlsCertUUID=my-client-cert`
`nats`	--broker (cells start flag) CELLS_BROKER (environment variable)	`nats://:4222?ssl=true&tlsCertCAUUID=my-client-rootca&tlsCertKeyUUID=my-client-cert-key&tlsCertUUID=my-client-cert`
`redis+tls`	--cache (cells start flag) CELLS_CACHE (environment variable)	`redis+tls://:6379?ssl=true&tlsCertCAUUID=my-client-rootca&tlsCertKeyUUID=my-client-cert-key&tlsCertUUID=my-client-cert`
`vaults`	--keyring (cells start flag) CELLS_KEYRING (environment variable)	`vaults://:8200?ssl=true&tlsCertCAUUID=my-client-rootca&tlsCertKeyUUID=my-client-cert-key&tlsCertUUID=my-client-cert`

by using the following standard parameters :

Parameter	Type	Description
`tlsCertStoreName`	bool	Store Name used in the server certificate to match
`tlsCertInsecureHost`	string	Skip the store name verification
`tlsCertUUID`	string (cert)	UUID of the imported certificate
`tlsCertKeyUUID`	string (cert)	UUID of the imported certificate key
`tlsCertCAUUID`	string (cert)	UUID of the imported certificate root ca

Pydio v8