Tuesday, September 17, 2013

A personal matter

Many of you already know this, but I wanted to make an 'official' statement on me going into part time retirement regarding deegree.

For personal reasons, I had to give up being self employed. Fortunately, I quickly found a new home at the WhereGroup, where I'll be working (apart from other things) on Mapbender.

Besides switching sides from server to client that also means I'll be switching from deegree to Mapbender. That doesn't necessarily mean that I'll never use or develop something for deegree ever again. But it for sure means that I won't make new 100+ commits pull requests any time soon.

Since I'll only be doing minor contributions from now on, it doesn't make any sense for me to remain a member of the TMC, so I'll step back from that position.

I sincerely wish things had worked out differently. With over 40 percent of the commits it almost feels like my third child. I wish you all the best of luck!

See you around (perhaps at the FOSSGIS code sprint in Essen?)!

Friday, June 7, 2013

Bolsena #4 - More JAXB fun

Adding to the JAXB fun I had yesterday, I've decided to fix related problems when binding our schemas at compile time, where the maven-jaxb2-plugin tried (sometimes) to load schemas from the net, although a XML catalog file was provided and configured.

The first thing I noticed was that only some of the related modules had that problem. Very strange, so I ran maven in debug mode (-X) for a module where it worked properly, and one module where it failed.

The plugin was configured identically in both cases (we manage plugins from our parent pom.xml). The only difference I've noticed was that in the working module all project dependencies were passed to the xjc call, whereas in the other module only the plugin's own dependencies were added. So obviously the schemas could not be loaded from the classpath.

I was not able to find out a reason for this, but just adding the project dependencies as plugin dependencies fortunately fixed the problem. This means that apart from the unit tests deegree can now also be compiled offline without problems.

If you want to know the details on xjc + catalogs, read this guide. It might be a little out of date, but most of it is still valid.

Thursday, June 6, 2013

Bolsena #3 - JAXB fun

In deegree we make heavy use of JAXB for unmarshalling configuration files. That works quite well, but had a drawback when making use of schema inclusion. The included schemas were always loaded from the internet.

Using the JAXB SchemaFactory I thought it was pretty easy to work around loading schemas from the net and using the ones included in our .jars instead. But somehow that didn't work out, the base schemas were still loaded through the internet.

Smart people found out that the order of the schema URLs when giving them to the SchemaFactory plays a role, and it turns out to be true! I've just opened a pull request that fixes the problem in deegree.

For it to work the schemas need to be in reverse order of inclusion, so put the schema without dependencies first and so on.

Bolsena #2 - coordinate system ramblings

Time for an update, even though there's not much to discuss on actual progress.

First for some good news, the resource dependencies pull request is finally here! It would be great if many of you would test it out, with 189 commits it's the biggest change since our move to GitHub yet, and although the unit and integration tests are working some details might still need some tuning.

Since my last post, I've been thinking and experimenting with the coordinate subsystem in deegree. The API is currently heavily based on the GML representation of coordinate systems, with references everywhere, for  axis, datums, ellipsoids and so on.

While I'm a friend of models that are 'complete', I'm not sure whether that's the best approach for coordinate systems. There are two major use cases for the CRS package. One is to keep the information on what system is being used with what identifier, the other is to transform coordinates from one system to another. Other use cases would be to import/export coordinate system definitions from/to GML, WKT, proj4 etc.

A review of the code revealed that all identifiers are stored in lower case. So exporting to GML or finding out exactly what identifiers exist for a given system is impossible, because the proper identifiers do not exist. The convenient use case of having eg. your layer configuration set up with the correct CRS identifier from the datastore also becomes impossible sometimes, in case the underlying data store is not configured explicitly with a CRS identifier.

So to summarize, the current package has several shortcomings. First is the identifier mess, which is not easily fixed short of a complete reimport of CRS definitions (which would override some manually modified definitions). Second is the model, which is just too complex and makes it hard to compare two definitions. Third, transformations are slow, sometimes not thread safe, sometimes they are synchronized and thus not scalable on multicore machines. Four, the whole system is statically initialized and makes heavy use of global static state.

I've tried unsuccessfully to fix some of these issues the past days, but I fear a complete rewrite is the only thing that will do the trick.

Monday, June 3, 2013

Bolsena 2013 #1

It's that time of year again, where OSGeo hackers from around the globe meet in a former monastery to collaborate and code under the Italian sun.

Things are a little different this year. Looking at the past years we've got a record number of people attending this time! Also, sadly, the Italian sun is missing. I hope that people are right when they tell me it's going to get better...

So back to business. I'm in the process of completing the resource dependencies branch/pull request, I can probably create the pull request today or tomorrow. Things are looking good, the web console is already adapted, no tests are failing and it works.

The Mapbender people have installed deegree on their computers and are working on integrating a simple workflow to create a new WMS/layer based on a shape file in a remote deegree instance using our REST API. That's already working, and we're currently trying to get it running in a more user friendly way. Not a bad start!

In a related note, I've created a new pull request that adds some more features to the REST API, like querying for all supported coordinate systems, checking if a coordinate system is supported by deegree and an experimental call to retrieve all known identifiers for a WKT encoded coordinate system.

In theory the equality relation is defined on coordinate systems (not taking identifiers into account obviously), but in practice I was not able to compare the Utah UTM zone (EPSG:26912) to a WKT encoded variant. I guess that's another reason why a rewrite/cleanup of the CRS package is needed.

Stay tuned for more!

Wednesday, May 22, 2013

Hans Moleman issues

XML, especially GML schema validation can be hard. The mysterious Xerces 'honor all schema locations' flag springs to mind (this is a mystery yet to be fully understood). Often, slow schema validation processes (which seem to fetch schemas from the web) can be traced to Hans Moleman. No, sorry, wrong link, to Hans Moleman.

So what's happening? And what does Hans Moleman have to do with it?

As the GML experts among you may know, GML application schemas depend on the GML schema, which in turn consists of many (varies amongst versions) schemas, depending on other schemas like for example the W3C XLinks schema, which in turn includes the W3C XML schema (the schema for the xml namespace itself: http://www.w3.org/XML/1998/namespace).

So even when validating a feature collection against a local version of a GML application schema, the schema parser might still get to a point where it needs to fetch dependent schemas from the internet. And since the xml.xsd is the last one in the chain, it's also the one that gets requested the most.

According to W3C people, they had ~130 million accesses to this file per day, and since decided to completely block eg. the Java default HTTP UserAgent and others. Apparently they later had a change of heart, and don't block it any more, but the xml.xsd URL has a delay of several seconds upon loading (see http://www.w3.org/2001/xml.xsd).

So when validating multiple documents, which all need the xml.xsd, with all schemas loaded freshly every time, you'll get a delay of several seconds where your computer seems to do nothing at all.

We've thought about the problem of remote schemas quite a while ago, and made use of a custom Xerces entity resolver to load OGC and W3C schemas from a local artifact which we ship with deegree. There would also be other solutions, our JAXB schema generation for example makes use of standard XML catalog files to avoid fetching schemas from the web.

But unfortunately the CITE WFS 1.0.0 tests (and others) do not (although newer versions tend to load required schemas from the classpath as well).

Using reverse engineering using an eclipse plugin (see the other post from today) I was able to fix this (they were already using a custom entity resolver, loading everything from the web all the time). Now a complete deegree build including integration tests runs only needs 13 minutes on fast machine!

For those interested, have a look at our deegree-compliance-tests module.

All the library sources

One of the nicer features in eclipse is that it is able to automatically browse not only through your own sources, but through library sources as well, if they're available. But what if they're not?

In that case eclipse shows the class in a byte code view, with a button to attach a source .jar. Unfortunately it  is often the case that you don't have the sources, either because they were not uploaded to maven central, or because they're closed altogether.

In any case, it is obviously often desirable while debugging to see what a library function actually expects, or why it fails. Contrary to popular belief, the actual code is always the ultimate documentation, because it's up to date even if human language docs are not.

So recently, while chasing after a Hans Moleman issue (more on that later), I needed sources from binary classes. Of course my first thought was 'decompiler', so I searched the eclipse marketplace for 'decompiler'.

I found JadClipse for eclipse 4.x, installed it, and voila: when I double click on a class file with no sources attached, the decompiler automatically decompiles it and shows me the source. Now that's what I call a plugin without hassle! That's a perfect counterpart to the -DdownloadSources flag for the maven eclipse mojo, now I never need to go without a library source again.

Wednesday, April 10, 2013

deegree developers to the front

Now that 3.2 has been released with our great new users handbook, I think it's time to start working on developer documentation as well.

We've got plenty of outdated development pages in the old wiki (which I'm not going to link here, it's too embarassing :-)). Instead of bringing the old docs up to speed, I've decided not to describe the status quo, but instead to describe things which are to come.

I've been working on a new implementation of the workspace and resource concept in deegree, which fixes a couple of outstanding issues with the old one. In particular, it adds resource level dependencies.

In case you want to have a look at the code, check out the resource-dependencies branch on GitHub.

Since we decided to have developer documentation close by the code, I've added the description of the new concepts to the GitHub wiki. Have a look at the workspace documentation which describes the new concepts.

I'm unsure whether the code will make it into 3.3 (it'll probably be released in around four weeks or so), but 3.4 is definitely the target.

I invite all interested developers to have a look at the code and the docs. Let's make the workspace right this time!

Wednesday, March 20, 2013

Setting up eclipse working sets using maven

Setting up eclipse with maven using a combination of the eclipse plugin for maven and the deegree maven plugin works pretty good. Unfortunately, we've got more than a hundred modules in the deegree code right now, which means the list gets really long in eclipse.

Of course, eclipse supports working sets, which can be used to group projects. But it's a lot of click work to organize all those projects into working sets, especially if you remove and re-add the projects often (eg. if you switch branches often).

Since I really dislike clicking often, I've decided to add a new mojo to the deegree maven plugin which saves me the trouble of configuring working sets and adding projects to them.

Since deegree module names follow a certain pattern (deegree-<modulegroup>-<name>, eg. deegree-featurestore-sql) it's a good guess to use the module groups as working set names. For example all deegree-core-* modules will be put into the d3-core working set, all deegree-featurestore-* modules into d3-featurestore.

To set up the working sets using the plugin, run the following line in the deegree root:

mvn org.deegree:deegree-maven-plugin:1.19-SNAPSHOT:setup-eclipse-working-sets -Declipse.workspace=$HOME/workspaces/test/

Use -Declipse.workspace to specify the workspace to modify, and optionally -Dworkingset-prefix to configure something other than d3- as a working set name prefix.

If you're generally interested in what the deegree maven plugin can do for you, check out the documentation.

Please note that this plugin is highly experimental, and has only been tested on the latest eclipse version. It might just destroy your eclipse workspace configuration! Use at your own risk!

Please also note that 1.19 of the deegree maven plugin is not yet released, you'll have to checkout my feature branch at GitHub and build it yourself (1.19 is probably going to be released on 2nd of April).

Edit:

1.19 has been released, you can use this out of the box now. Please note that it is indeed experimental, sometimes the working sets seem to be lost for some reason. In that case, re-doing the procedure works (for me).

Tuesday, January 29, 2013

deegree/Occam Labs in academia

Jürgen Weichand, author of the WFS 2.0 support for the Quantum GIS, recently published his master thesis (German only) on the topic of implementing and using INSPIRE DownloadServices.

While a large focus of the thesis lies on the client side (the WFS 2.0 plugin for Quantum GIS was developed in the context of the thesis), some server side solutions were also evaluated. Since UMN mapserver does not have a WFS 2.0 implementation, only GeoServer and deegree were considered OpenSource-wise, the GO Publisher WFS was chosen as closed-source example.

Looking at table 6.7 on page 67 it seems we're doing pretty well. I'm sure the upcoming 3.2 release will improve the situation even further. And it's one of the first publications explicitly mentioning us at Occam Labs as developers of deegree!

We probably could have made an even more prominent place in the thesis if we had a nicer web administration console ready (chapter 6.2.6 mentions this as the probable reason why GeoServer was chosen for an example implementation, simple features only!). At least with the deegree handbook nearing completion it will allow future thesis writers (and all others of course) to better understand how easy it is to set up a WFS serving rich features with deegree.

Edit: I did not mean to imply that GeoServer can't serve complex features (as Andrea pointed out it has been able to do that for years), it's just not as easy to do using the web interface (which Jürgen Weichand also pointed out in this thesis).

Tuesday, January 15, 2013

deegree goes GitHub

There's breaking news in the deegree world, as deegree (at least deegree as in the deegree webservices) is now at GitHub! Have a look at the official repository yourself. And help yourself to a fork...

Our main reason for moving to GitHub (besides moving to git as a tool) was to make it easier for people to contribute. And that's exactly what happened! We hadn't even figured out how to properly work with pull requests and the famous git cheap branches and there was our first pull request from an external contributor!

I also extracted the deegree maven plugin into a separate repository. Have a look at the documentation I added in its wiki.

We decided to (gradually) move all developer related documentation to the wiki of the deegree webservices code base, so that should be the primary source of information for now. Links to the old wiki were added where appropriate.

So I hope to see more of those pull requests coming!

In a related note, on Friday (the 18th of January) I'll move some more deegree related projects like the old deegree 2 code base to GitHub as well, and then the svn repository will be shut down/made read only. In case some old project picks up again, I offer to migrate any of the old projects to GitHub, but for building the unchanged code base, the read-only svn should be good enough.

Friday, June 15, 2012

Bolsena 2012 #3: deegree rendering OpenStreetMap

deegree has been using OpenStreetMap as a background map for a couple of years now in its layer preview. That's already quite nice, but there's so much more that can be done with the OpenStreetMap data. To make a start I've begun to set up a workspace that makes use of OpenStreetMap data imported by using imposm.

To set up the data, I just followed the imposm tutorial. The workspace is configured using the production table schema, so make sure you run imposm --deploy-production-tables once you're done importing the data.

The workspace is located in the deegree svn (deegree-workspace-osm), so if you want to have a look, go ahead. It uses the GeoServer styles that were used in the FOSS4G benchmarking 2011. It still needs a little tweaking to improve the map quality (labels and stuff), but the styles worked out of the box. The labels along roads/waterways seem to be repeating without taking the gap configuration into account, so analysing the deegree renderer here seems to be in order.

Apart from that, this enables you to create your own maps based on the OpenStreetMap data. The new workspace is the place to start, you can just adapt the styles for your own needs or create your own layers any way you want.

We at Occam Labs plan to provide an online demo using the German and Dutch OSM extracts from Geofabrik. We already did some tests on these datasets, and the result was quite promising. So stay tuned for updates on this one!

Edit: Forget the comment about the deegree renderer not taking the gap into account. It didn't, but now it does, another successful Bolsena bugfix ;-)

Wednesday, June 13, 2012

Bolsena 2012 #2: Lenient SLD/SE evaluation in deegree

In preparation of setting up a new demo workspace (more on this later), I've been working on making the SLD/SE filter and expression evaluation more lenient. Until now, the parser already ignored invalid SLD/SE bindings within style files, but the evaluator of expressions (like PropertyNames) and filters was strict.

The new evaluation will try to fix missing namespace bindings for PropertyNames in expressions and filters. Such broken bindings can be hard to find, because even though the schemas require you to use qualified names/xpaths, normal schema validators will not find these errors.

One nice (and intended) side effect is that SLD files used to configure GeoServer can now be used without modification (unless some special GeoServer constructs are used).

Since it is still desirable to write valid documents, you can set a class to DEBUG in your log4j.configuration to see what PropertyNames are being repaired:

log4j.logger.org.deegree.filter.Filters = DEBUG

Monday, June 11, 2012

Bolsena 2012 #1: deegree on the couch 2

Following up on the 2011 Bolsena event, we got news from Volker Mische that we could try to adapt the GeoCouch feature store created then to the new 2.0 version of Couchbase.

This would allow for easy, transparent use of multiple Couchbase machines as a backend, and might also improve the speed situation of our last experiments.

Turns out it was only half a day of work to adapt the existing approach to the Couchbase one. With a little URL/URI fiddling the indexes could be used same as before, with only minor changes. We now used the approach to store the GML blob directly in the JSON document (base64 encoded).

Actually, using the Java libraries for Couchbase was pretty straightforward, and resulted in much cleaner (and much less) code than before for inserting. Since the Java libs use a custom protocol, inserting also seemed quite a bit faster.

We left the query interface to still use the old HTTP interface (with only minor URL/URI fiddling), and querying is around 50% slower than our PostGIS blob storage. We expect that using a few performance tweaks would result in approximately the same speed (we're not using any compression now, and using the Couchbase Java libs would probably also be an improvement).

So all said, the first day was already quite a success. Although the weather started out with rain and comparatively low temperature, the sun came out around the same time we did our performance tests. Must be an omen ;-)

In case you're wondering, Markus is currently working on the documentation. We're going to try to expose some test subjects here to the handbook... Actually, it seems that quite a lot of the people here have already used or are using deegree, so maybe we can actually find someone willing :-)

Stay tuned for more Bolsena news!

Friday, May 4, 2012

deegree updates

Hello everyone, it's been a while. Much has changed in deegree since.

From a project and community perspective, the new website would be the most obvious. We are also finally working on a proper documentation handbook for deegree 3, the latest work in progress can be found at http://download.occamlabs.de/deegree-webservices-handbook/.

Aside from that, our new company has started out quite nicely, we were sometimes so busy we had no time to write blog posts. I know, a lousy excuse :-)

Let's see what else has changed in deegree. The layers/themes refactoring/rewriting has taken place, and is now in a usable state. In fact, it's now the preferred way of configuring your service content.

Speaking of services, there's also a new member of deegree's little service family, the WMTS. Which means we also have a couple of tile stores to provide data.

I plan to pick out a few of the new features and post about them, but of course you're free to play around with it yourself.

Last but not least, we will of course be attending the OSGeo Bolsena Code Sprint 2012 this year. We haven't finalized our plans yet, but I'll be sure to post updates on our activities here, so stay tuned!

Tuesday, September 27, 2011

Xerces ClassCastExceptions in multiple deployments

Today I'd like to talk about strange XML ClassCastExceptions one sometimes gets when deploying the same web application twice in the same Tomcat.

The problem occurs when you deploy for example a newer xerces library in your webapp's WEB-INF/lib directory. The problem has to do with the mechanism the XML factories are instantiated dynamically. I've always thought (and wondered) why this is so, because obviously the xerces classes have been loaded properly by the webapp classloader specific for the webapp.

After not having to deal with the problem in deegree 2 (the xerces was not specifically needed there, and the problem could be solved to remove it from WEB-INF/lib) I ran into the problem again with deegree 3, where xerces is a central and necessary library to parse XML Schemas.

I've found that there is a debug setting for the JVM -Djaxp.debug=true, which was essential for finding the problem. It showed that the ClassCastException did not actually occur when instantiating the DocumentBuilderFactory for the second time, but when XSLT was used. So what happened?

The built-in XSLT-solution in Java is xerces' counterpart, xalan. Like the built-in xerces this is an older version, apparently preconfigured to use the built-in xerces. I think what probably happens is that the used class is somewhere cached within the parent class loader (one of the global class loaders of Tomcat), but loaded with the webapp class loader of the first deployed webapp. Once the second webapp tries to use XSLT, it fails, because a different (and inaccessible) class loader was used to instantiate the parser in question. But that's pure speculation...

... which led to a hunch on my side. I thought that if each webapp had its own xalan as well, it might solve the problem, because the dynamic loading mechanism for XML factories prefers factories that can be loaded via SPI. So I deployed a recent xalan as well (plus the xalan serializer jar) and the problem was gone.

To conclude, it seems good practice to deploy recent versions of all needed (directly or indirectly) XML factories in each webapp. An indirect need for XSLT can be created as quickly as converting a OMElement to an XMLStream, so it's a good guess you need it.

PS: Another solution would be to deploy all the XML factory libs centrally in the Tomcat, but that solution has obvious disadvantages (having to fiddle with the Tomcat, not being able to use different versions across webapps).

Thursday, September 22, 2011

Maven, Java and Yet Another Memory Problem

Everyone working with Java has probably experienced an OutOfMemoryError at one time or another. Since the default values of the JVM are often ridiculously low (used to be 64MB for a standard JVM), increasing the value through startup options usually solved the problems.

Experienced users know that there are different kinds of OutOfMemory errors, the most common being 'Java heap space'. People playing with Tomcat and redeploying webapps a couple of times run into the 'PermGen space' variant pretty fast.

But recently we encountered yet another memory problem, which even resulted in a vm crash, with problems in libjvm.so. This was reproducible on our build server, where we also release new versions using the maven-release-plugin.

It seems that during the maven run within maven (when running mvn release:prepare) the reserved code cache memory area runs full. The documentation about -XX:ReservedCodeCacheSize is a bit unclear on its workings:
Reserved code cache size (in bytes) - maximum code cache size. [Solaris 64-bit, amd64, and -server x86: 48m; in 1.5.0_06 and earlier, Solaris 64-bit and and64: 1024m.]
But it just so happens that increasing it to half a gigabyte fixed the problem for good.

I'm all for being able to configure the JVM in all ways imaginable, but being forced to control it just can't be the right way. To run the release plugin we now have to configure three different memory settings. Can't we have a maximum memory setting that limits heap, stack, permgen and code cache size?

Thursday, September 8, 2011

Raster Pyramids in deegree

Today I would like to talk about raster pyramids in deegree. Gone are the times where you have to use our custom tools to create pyramids and manually configure the different solutions.

In order to make the handling of raster pyramids easy we now support using standard GeoTIFF pyramids with overlays. Let me show you how to prepare your data and use it as coverage data source in deegree.


To prepare your data, you can use standard tools like GDAL. There are a couple of requirements that the processed data needs to meet:

  • it must be a GeoTIFF, containing the extent and coordinate system of the data
  • overlays must be multiples of 2
Assume you have a couple of raster files lying around in a directory, eg. png with .wld. First, build a GDAL virtual raster:

gdalbuildvrt virtual.vrt *png

Then, build the base GeoTIFF using gdalwarp:

gdalwarp -t_srs EPSG:26912 -co BIGTIFF=YES -co TILED=YES virtual.vrt merged.tif

The BIGTIFF option enables you to create files bigger than 4GB. There are a whole lot of other options (you can also compress the TIFF if you want), see the GDAL documentation for details.

Finally, calculate the overlays using gdaladdo. Take care to only use multiples of 2 for the overlays, like in this example:

gdaladdo -r average merged.tif 2 4 8 16 32

That's it, now you have a GeoTIFF ready to run. To configure it in deegree, simply add a new coverage data source using the web console and configure the location of your file.

All done! Now you can use it like any other coverage data source to configure your layers (or a WCS). The files can be very big, I've already tested it with ~150GB files (my lack of disk space prevented testing bigger files).

The implementation was done using the standard TIFF driver from imageio-ext, which has support for BigTIFF without the need for JNI libraries (imageio-ext also has support for GDAL using JAI, but I didn't use that one). Many thanks to the developers for making this easy!

Edit: Removed references to images. I was wondering where all my screenshots went. Seems it had something to do with joining Google+. But I'm not taking the effort to recreate the screenshots here (you should read the deegree handbook anyway, and use the new tile stores instead of this one).

Friday, August 26, 2011

Cascaded Layers and Themes

Today I want to talk about an experimental way to configure cascaded layers.

I already talked about a new layers and themes concept. In order to test out how this can work, I've now implemented an experimental way to cascade a WMS. This is a different approach than the one I already talked about, it focuses on cascading a complete service rather than a single layer.

So we have a couple of different resources here. The first and most basic resource is a remote OWS (OGC Web Service). There are now two different remote OWS resource types, with the old one still being used by the traditional cascading layers concept. The new one is currently a lot simpler to configure:

<?xml version="1.0"?>
<RemoteWMS xmlns="http://www.deegree.org/remoteows/wms"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.deegree.org/remoteows/wms remotewms.xsd"
configVersion="3.1.0">
  <CapabilitiesDocumentLocation
    location="http://deegree3-demo.deegree.org/deegree-utah-demo/services?request=GetCapabilities&amp;service=WMS&amp;version=1.1.1" />
</RemoteWMS>

Please note that the namespace differs from the traditional remote WMS store. The only thing to configure here is the location of the WMS capabilities document.

The next resource we have is a LayerStore. A remote WMS layer store can be used to provide a 'copy' of all the layers a remote WMS has. The configuration simply consists of telling the store which remote WMS to use:

<?xml version="1.0"?>
<RemoteWMSLayers
xmlns="http://www.deegree.org/layers/remotewms"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.deegree.org/layers/remotewms remotewms.xsd"
configVersion="3.1.0">
<RemoteWMSStoreId>d3</RemoteWMSStoreId>
</RemoteWMSLayers>
 
Next is the configuration of a Theme. The configuration of a theme is also pretty straightforward. It also needs a remote WMS, and it needs to have layer stores:

<?xml version="1.0"?>
<RemoteWMSThemes
xmlns="http://www.deegree.org/themes/remotewms"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.deegree.org/themes/remotewms remotewms.xsd"
configVersion="3.1.0">
<RemoteWMSId>d3</RemoteWMSId>
<LayerStoreId>d3</LayerStoreId>
</RemoteWMSThemes> 
 
The interesting thing here is what happens here. As you might remember, a theme combines sub-themes and layers to what is traditionally known as layer trees. You can now use theme references to configure your WMS layer structure (as we will see below).

The remote WMS theme creates a theme structure which is identical to the layer structure of the remote WMS. It then tries to find matching layers from the available layer stores, and inserts them at the appropriate places. If the layer store coincides with the remote WMS layer store (as in our case) you'll get the same structure as the cascaded WMS.

The WMS configuration is then as simple as this:

<wms:deegreeWMS
xmlns:wms="http://www.deegree.org/services/wms"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
configVersion="3.1.0"
xsi:schemaLocation="http://www.deegree.org/services/wms http://scemas.deegree.org/services/wms/3.1.0/wms_configuration.xsd">
<wms:ServiceConfiguration>                                                                                                                         
    <wms:ThemeId>d3</wms:ThemeId>                                                                                                                                                     
  </wms:ServiceConfiguration>
</wms:deegreeWMS>

And that's it. True, you had to edit four files, but all are very simple to understand. Cascading a WMS with 524 layers used to be harder...

If you do a quick GetCapabilities, you'll see that bounding boxes, metadata etc. are just copied from the remote service. In the future it will of course be possible to manually create a custom theme, which can then be used to only select a couple of layers from the remote WMS, mix them with other layers and add/edit metadata.

I think this proves that the layers/themes concept can work. To further test it, a couple of prototypic other layer stores will be implemented soon.

Friday, July 29, 2011

deegree Themes

After talking about layers I recently read the OGC WMTS specification. A new concept called themes was introduced there, and that lead to me being able to understand the problem I'm trying to solve here more thoroughly.

Layers in WMTS are no longer hierarchical. All that remains of the old WMS layer trees is a single list of layers. In order to create order in a potentially huge linear list of layers, the theme concept was introduced.

Themes in WMTS can be hierarchical, and each theme can reference any number of layers. Layers can even be referenced multiple times, and you can have multiple top-level themes.

That actually makes a beautiful distinction between structure and data. Speaking in deegree workspace language, it suddenly makes a lot of sense to have not only a bunch of layers configured in one place, but also have a bunch of themes, referencing layers, in another place. A WMS would then only reference themes.

Other ideas include simple layer collections (which aggregate layers similar to current logical layers within other layers).

I still have to figure out a lot of the details, but I can feel that this is going to make things simpler. When configuring a layer you only think about what data it's going to use, and when configuring a theme you only think about the bigger picture (where do my layers belong).

I'm always open for suggestions, so if you have an idea about how to make things good in the end, please speak up.