TerraFlux - Prototyping a New Webbased Earth Observation Application
- Marco Peters
- 21 hours ago
- 3 min read
Hello EOMasters,
Some time ago I already teased this project on social-media. Now I want to talk a bit more about it.
It all started with the idea that it is quite tedious to visit all the different data providers separately. Why not having one common search interface and search across all data providers with a single request? This idea evolved. I thought it would be good to keep the data in the cloud and also directly do processing on the data. And then one should be able to analyse and visualise the data. And finally, download the data locally if needed.
Sounds like a 3-year project done by 10 people. However, I started on it.
After doing the first steps, I think it it still a big project. But I started it. Not sure where it leads to, at least I will learn new things.
So, I give you a little report on what I have achieved so far.
First, the baby needs to have a name and a face (logo). This is usually one of the first things I have to do for a new project. So I came up with the name TerraFlux. The Earth is passed on in stream through the different stages. It is a flux of Earth data. Also it reminded me of the flux capacitor in Back to the Future. As logo I wanted something which represents this flux around the Earth. So this is the story of the name and the logo.
Next I started to develop accessing the data provider. They all use different APIs to access their catalogs and to search for data. So I had to define an interface which provides a generic access to different APIs but also also allows to include certain specialities of some data providers.
This is now setup and implemented for CDSE. It is possible to search for Sentinel-1, Sentinel-2 and Sentinel-3 at once. The APIs of EarthExplorer, EUMETSAT or Sinergise and Planet can be used too.
After searching for data, the results are listed. For each result a quick look image is shown, the footprint on earth and the most important metadata information. On click, more detailed information can be displayed. And at the moment, the data can be downloaded locally.
This is actually the current status. On the way different small feature have been implemented. A Language system to support multiple languages for the front end, User management to manage the different user settings, a database had to be set up to hold the data. The next step I'm working on is the processing engine.
This engine shall be usable also from the command line and not only in the cloud. So, first I develop a more or less traditional processing system. Those who know ESA SNAP will recognise similarities. Yes, why not keeping the good things and exchange and improve what is needed. What is similar is that you have single operations which you can combine to a graph. The graph definition will be a bit simpler because YAML will be used instead of XML as description language.
The operation implementation can have annotated methods which allow to process them independently in parallel. Cache and memory management is in particular taken care of. The backend of the processing engine is my own implementation. It will not use any old processing tool like JAI or other 3rd party processing systems. Often these systems have quite some overhead and you always depend on the work of someone else. My solution is tailored to my needs. I've implemented it, I have the control and I can fix it if needed. For such a central component this is important, I think.
The first processing results are promising. But it is still to early to give you numbers. But I can say performance and memory consumption are in a good shape.
Currently I have a little GUI tool to view and analyse the results and work with the data.
I even have a little expression editor to compute new bands using math expressions. And it has syntax highlighting.
Would you be interested in such a tool and join me on this journey? Even it is still limited and get's extended step by step? Let me know in the Groups or on social-media You can also contact me directly.
That's it for now. Let’s see where this journey takes me. I will report again if new milestones are reached and something noteworthy can be told.
Tschüss & Goodbye Marco