Pydio Cells 1.5.0 - Performances, features, security

This release improves performances again, provides new features and security fixes. Upgrade is highly recommended, see instructions at the end of this document. Thanks to @fa-jo and @rossbeazley for their contributions.

Main Changes

• We got our first vulnerabilities reported by Loginsoft Research (CVEs to be submitted). It is very good news to see that the community is auditing the code! 
• Indexing an existing storage with huge amount of files is now much faster, micro-services on-file indexes are taking less space, CPU usage is much better as well. 
• WebDAV support is stabilized.
• Encrypted datasources support streaming, and copy/move to/from another datasource. Inter-workspaces copy and move operations can be done by drag'n'drop.
• Cells Enterprise now provides a way to create groups of users who cannot "see" each other (group tenancy).

Fig1: Copy progress report in UX​

Details

Massive performance improvements

• Improve indexation speed, compute checksums in parallel, batch events in many services
• Fix DB prepared statements count going too high
• Use new index format for Bleve-based services (needs manual reindexation, see Upgrade instructions)
• Fix idle CPU usage: can be as low as 2% for main process, below 0.5% for sub-processes

Fig 2: Idle CPU usage

Datasources

• Allow creating a local FS pointing to a Symbolic Link
• Fix Streaming on Encrypted Datasources
• Show specific status when starting datasources
• Prevent enabling Encryption with no Master Key
• Fix inter-datasource copy/move operations (especially between encrypted/clear datasources)

UX / APIs

• Easy Drag'n'drop between Workspaces and/or Cells. Dropping to a Cell shows a dialog to either move the file/folder to the Cell or add it as a new shared root.
• Handle numeric sorting of Cells and Files. Add a quick filter on Cells.
• Diaporama supports left/right arrow keys for navigation, Esc for closing, fix sorting
• Add Markdown Preview in code editor
• Fix issue with thumbnails size on switching workspace
• Update VideoJS version
• Publish Share as Activity
• Additional parameters for upload requests timeout, for super-slow connections
• Add generic validation mechanism on Protos and apply automatically when reading pb entity in Rest API. Applied to DSName as an example.
• Fake synchronous API for WebDAV
• Allow unicode string usage in user attribute values (e.g. display name of users can use emoji).

​
Fig 3: Drag'n'drop files on an existing Cell

Scheduler

• Copy/move task reports progress in UX
• New Trigger Event on User modification
• Send groups deletion as background task, and pass a channel for handling objects as they are deleted
• Optimize events received by scheduler: add a NodesSelector at the Jobs level to avoid creating unnecessary tasks
• Scheduler actions: use TasksLogger instead of Logger

Cells Enterprise

• Implement simple per-group tenancy by adjusting users policies.
• Fix corner cases on LDAP synchronization
• Change heuristic for computing license stat

Other

• Vulnerabilities reported by 'Loginsoft Research' (CVEs to be submitted)
• Mailer: Sender control feature, Default sender display name, PR #58
• Better JSON serialization for exporting to ELK stack
• Add various subscribers for cleaning resources on user deletion. By default, personal folder is now renamed to a folder name with uuid.
• Reload Jobs automatically when a task is stuck in Pending Mode

Upgrade

Upgrade Binary

Upgrade should appear in the administration Console. Run the upgrade to download latest binary and restart Cells. Do not forget to run the 'setcap' command on the new binary if you are binding to port 80 or 443.

Migrating Indexes

A fresh install is now using a new format for the on-file indexes used by some microservices (logs, search engine, etc). For technical details, we use Bleve library and turned on the new Scorch indexation format.

At upgrade, re-indexation is not triggered automatically to avoid long running tasks at migration time, so you have to perform this manually after upgrading. If you notice that your APP_DIR/pydio/cells/services/ folder is taking too much space, you may want to recreate the indexes using the new format, using the cells-ctl (or cells-enterprise-ctl) tool and the following commands (while Cells is running):

> ./cells-ctl data sync --service pydio.grpc.log
> ./cells-ctl data sync --service pydio.grpc.search
> ./cells-ctl data sync --service pydio.grpc.docstore

For Cells Enterprise, perform also: 

> ./cells-ctl data sync --service pydio.grpc.audit --path=logs

These commands will launch tasks in background, they can take some time, when it's finished you should see a big difference in space usage for the services/ directory afterward (~ divided by 6).