How we improved Pydio performances using Blackfire.io

by Charles du Jeu - Lead Developer

Introduction

Pydio

Pydio is an open-source file sharing & sync platform written in PHP. It was once called AjaXplorer, started almost 8 years ago, and had a nice traction over the years to transform itself from a simple webFTP to a fully-featured on-premise "dropbox" replacement.

The server was written using PHP, and although this language does have some drawbacks that everyone knows, it was a successful choice for getting more attention from contributors, and for allowing an easy deployment on almost 99% servers in the world. Of course, as the feature-list grew, so did the number of Lines Of Code and the complexity, and it was regularly necessary to audit the code for best performances.

In this article, we will describe how we used the new tool Blackfire.io to look for perfs bottlenecks for Pydio 6.2.0.

Profiling PHP

Looking for bottlenecks in one's code, especially when it's yours, can be a really rewarding activity, as you generally know where to start, but don't know where you will end: your guts tell you where the slow part is, but using efficient profiling tool often leads you to discover unexpected anti-patterns that you missed. No one is perfect. Fixing those can bring great results, as you generally find the problems inside loops where they can be repeated many times...

I've always been a big fan of the XDebug / WinCacheGrind combo for these hard-core profiling session. I must admit that it was still an efficient set of tools, until I entered Blackfire.io : no more need to load my 50Mb profiles in a not-so-friendly app displaying a rough tree of method calls, Blackfire php extension + online graphical UI just does the magic for you. It really lets you focus on how to fix the issues instead of spending time finding them.

Here is what using XDebug+WinCacheGrind looked like: usable, but not so sexy :-)

First glance: Houston, we've got a registry problem!

Pydio Plugins and XML Registry

In a constant quest for modularity, Pydio was built since a long-time on a fully plugin-oriented architecture. Even the basic features (like configuration management, authentication, etc.) are plugins-based and provide different implementations to fit administrator's needs. The so-called "core" is a super-thin layer in charge of maintaining a pool of available plugins and loading them and their depencencies depending on the current state of the application.

Each plugin is fully "self-declarative", and this is done through a manifest written in XML (for old-schoolers out there, yes it was inspired by the Eclipse OSGI architecture). Once the application decides to load a given set of plugins, these XML branches are merged into one big DOMDocument that we call the registry. This gives a very high flexibility on how plugins can depend and interact one with an other, and being shared by both the server and the clients, it can be easily queried using XPath. This is still a huge advantage over JSON deep structures.

As an example, the server-side uses the registry to detect active plugins and dynamically call (or not) some specific event-driven callbacks. And the Web UI is using that same registry to collect various template parts, rebuilding itself depending on the currently active plugins. To learn more about the registry, please read this article in our developer guide on our website.

Building and querying the registry: our bottleneck.

Starting the Blackfire.io session showed very quickly that all these XML operations were time-consuming and should definitely be optimized. Some caching layers already existed to avoid the permanent "scanning" of the plugins folder. But as the registry itself changes in time depending on the application state, it is necessary to rebuild it frequently.

To build the registry, we have to loop over the active plugins, call their loadRegistryContributions method (that returns XML nodes) and attach these branches to the main tree. The mergeNodes method is itself using XPath to merge the XML nodes intelligently.

As you can see, the findActionAndApply method (the main entry point in the server controller) will first triggerbuildXmlRegistry to find what plugin and callback it must call, then apply the callback, which can in turn trigger some internal hooks (another manifest-based event driven aspect of the app) that will search as well.

=> Everything is leading to XPath, and this is definitely bad for us. XPath syntax is powerful, but costly.

As we've been taking performance matter very seriously for a long time, there were not a lot of pure "code optimization" that could be done to improve performances. So we turned to expanding our caching mechanisms.

Caching XPath queries results.

The graph above showed that apart from this "building" operation, the registry is also heavily sollicited by external XPath queries: they are super-handy, but cpu-consuming. So we can simply cache the results wherever possible. Serializing/unserializing XML is unefficient, so we always transform those results into a more serializable-friendly php-object/array.

In the code below, we use loadFromPluginQueriesCache/storeToPluginQueriesCache to cache on file some XPath query results, and avoid rerunning the query whenever possible.

$exposed = array(); $cacheHasExposed = AJXP_PluginsService::getInstance()->loadFromPluginQueriesCache("//server_settings/param[contains(@scope,'repository') and @expose='true']"); if ($cacheHasExposed !== null && is_array($cacheHasExposed)) { $exposed = $cacheHasExposed; } else { $exposed_props = AJXP_PluginsService::searchAllManifests("//server_settings/param[contains(@scope,'repository') and @expose='true']", "node", false, false, true); foreach($exposed_props as $exposed_prop){ $pluginId = $exposed_prop->parentNode->parentNode->getAttribute("id"); $paramName = $exposed_prop->getAttribute("name"); $paramDefault = $exposed_prop->getAttribute("default"); $exposed[] = array("PLUGIN_ID" => $pluginId, "NAME" => $paramName, "DEFAULT" => $paramDefault); } AJXP_PluginsService::getInstance()->storeToPluginQueriesCache("//server_settings/param[contains(@scope,'repository') and @expose='true']", $exposed); }

On the graph below, we use the COMPARE feature of blackfire to see the differences between two profiling sessions. With many XPath queries cached, we can see that we have a 24ms gain, which was already almost a 10% gain on each request.

Expanding cache: from files to session to kvstores

Limiting the registry possible states

The problem with our registry concept is precisely that it evolves over the time, depending on the state of application. At one point, some plugins were actively modifying their XML contributions based on their own internal state. But going through the existing codebase, we decided that we could really narrow down the "state" concept to the following variables :

1. Is a user logged (and if so, who): this will add to the XML tree a set of information about the current user, but also this will load the role of a user, which can in turn filter-out or modify some nodes inside the registry.
2. On which workspace is the user currently logged: this is the main source of plugin differentiation, as a workspace is primarily defined by its access driver (what type of storage) plugin, and a set of features (like indexation, metadata, user comments, etc...) which are plugins as well.
3. Registry mode: for internal purpose, we can build the registry in a "light" or an "extended" mode to speed thing.

In that condition, we can create a matrix of possible values of the registry, and cache these pre-built registry versions inside local files.

Writing to / reading from XML files

As stated above, we were already implementing a file-based cache for the main pool of plugins, as well as for the XPath queries results (although not on all of them). We generalized the latter, but loading a file content and unserializing its data on each request is still insufficient on high server load.

The Profiling session above shows a simple /changes/ query sent to the server. This is used by the sync client to test if there were recent changes in a given workspace. As you can see, loading Registry from cached XML file (on the left) is still a time-consuming operation. Logging the user as well, and finally the findRestActionAndApply, which is the effective code of this request, remains as the smallest part of the query.

Caching in the session

As our WebUI is entirely session-based, a first level of optimization was to store more data directly inside the PHP session. Data still requires and unserialize() operation on session load, but at least it's done only once from one source and optimized by PHP. But the REST API does not use session...

Rest API performances requires kvstore caching.

The Rest API, introduced in Pydio 6, is used by our Desktop Sync client and as such it required a particular attention: deploying many desktop sync clients and having them continually talk with the server for real-time synchronization can of course tremendously raise the server load, and the various caching methods must take that into account.

Of course, its name says it all: the REST api does not use session. Any session-based caching done there is useless. This means that for example on each request, we have to check if a user is correctly authenticated, load the registry from cached file, check permissions on the current workspace, etc. This can be seen in the previous image as well.

For that reason we introduced the support of Key/Value Store systems. Currently, we just implemented an interface with APC, but our API respects the Doctrine component signature and we will shortly add a generic support for Memcache or other equivalent systems. Below is a sample code using our KVCaching internal API:

private function loadRegistryFromCache(){ if((!defined("AJXP_SKIP_CACHE") || AJXP_SKIP_CACHE === false)){ $reqs = AJXP_Utils::loadSerialFile(AJXP_PLUGINS_REQUIRES_FILE); if (count($reqs)) { foreach ($reqs as $fileName) { if (!is_file($fileName)) { // Cache is out of sync return false; } require_once($fileName); } $kvCache = ConfService::getInstance()->getKeyValueCache(); $test = $kvCache->fetch("plugins_registry"); if($test !== FALSE) { $this->registry = $test; }else{ $res = AJXP_Utils::loadSerialFile(AJXP_PLUGINS_CACHE_FILE); $this->registry = $res; $kvCache->save("plugins_registry", $res); } // Refresh streamWrapperPlugins foreach ($this->registry as $plugs) { foreach ($plugs as $plugin) { if (method_exists($plugin, "detectStreamWrapper") && $plugin->detectStreamWrapper(false) !== false) { $this->streamWrapperPlugins[] = $plugin->getId(); } } } return true; }else{ return false; } }else{ return false; } }

And the first results are promising, as shown on the Comparison chart below:

Optimizing DB load

Caching users roles

Once the registry and its heavy XPath got out the way, we started to notice that eventually, loading a user and all its data from the database could introduce another important loss in performances. As Pydio provides a flexible role-oriented permissions managements, a user "effective" role is always a "merge" of its parent group(s) role(s), the role(s) the admin may have assigned manually, and its personnal role. All those loaded from the DB, to e.g. display the current display name of a user. Oops.

Caching loaded workspaces

As for the users, at one point, loading a workspace data from the DB should be cached.

Putting it all together in a KVStore

The following image shows the comparison of simple REST /changes/ calls without and with KVCaching. Performance gain is great.

Now we're talking!

Conclusion

Basically, here are the results: a simple query time is divided by 4,6. We went from 370ms to 80ms. This is a much more acceptable response time for such a request.

Of course, this will still depend on each requests: these optimisations were done on the overall framework, listing a folder with tons of files in it will have a bigger portion of its code dedicated to the actual listing than to the framework load. But clearly, every requests being more optimized immediately resulted in a much smoother user experience in the UI, as well as a lower server load when using the sync clients.

Blackfire.io is an ideal tool for easy profiling and optimization, and we just actually used a portion of what they can do. Many features were released since then, have a look at their website Blackfire.io.

Note on Key/Value Caching

As it currently requires a correct APC/APCu underlying configuration, this feature is by default NOT activated on a vanilla Pydio installation. This can be done by changing the following key in the conf/bootstrap_context.phpfile: set AJXP_SKIP_KVCACHE to false instead of true. Also, if you have many Pydio instance running on the same server, use the AJXP_KVCACHE_PREFIX value to make sure they are differentiated in the memory-based cache.