Students capstone project

[CamilleGaillard]

Hello,

I'm considering proposing a capstone projects for CS students similar to SlideRule, but for other RS data.
Based on your experience developing SlideRule, how big of a project should it be to fit in 1 year for 4 CS students?

Also, would it make sense to discuss how to synergise it with SlideRule?

Thnks,

Camille

[jpswinski]

Hi @CamilleGaillard, that sounds like a very interesting project. I'd be happy to provide some information relating to the size of the effort with regards to SlideRule and also some possible collaboration with the SlideRule team.

Can you provide some additional details about what you would like to build? SlideRule has a lot of different components that have taken a few years to build, but there are parts of the SlideRule system that I could see being a good fit for a 1 year project.

[CamilleGaillard]

@jpswinski
The overall idea is to have a pipeline to access RS data in "real-time" (when new data becomes available), to process/format it automatically for integration in a mechanistic model, or any other research project, as long as the requirements are specified. Integration in a model with automatic updating/downloading would allow for the model output to be always up to date with latest inputs.
Technically there are two main ideas:
-Automatically downloading and formatting data based on research requirements
-Improving "user-friendliness" for data access
We might also want to do that with data coming from community science projects/apps.
We provide the question and frame, but the students will have to figure out the precise methodology.

[jpswinski]

Now that I have a better idea of what you are looking to do, it may be that SlideRule is not the best fit. SlideRule is primarily an on-demand data processing system that provides real-time APIs to generate custom products. From what you are describing, it looks like you would like to build a pipeline that detects new data being available in a given dataset and then ingesting and processing that data to update a local model. If that is correct, you might want to look at tools that run and orchestrate pipelines in the cloud. For instance, the HySDS project is something I've come across recently that sounds like it might fit your requirements better: hysds.github.io.

If I've misunderstood, and your goal is to provide an on-demand service like SlideRule which is driven by user-exposed REST-like endpoints, then I think being able to select a very narrow algorithm to be implemented, along with one or two source datasets, would be a good place to start. Once the source datasets are selected, then the way of accessing the data can be fleshed out. A lot of the roadblocks and potentially costly development efforts are likely to come from how the data is accessed.

[CamilleGaillard]

Thanks for the link to HySDS. It seems that our idea might be a bit in-between the two. One of the ideas of our approach would be to have enough flexibility to be useful for multiple projects and to update them automatically. SlideRule could be a reference for accessing the data and handling the storage of the raw data.
I agree that the students should limit themselves to demonstrating the capability of what they'll do on only one or two datasets.
What kind of roadblocks would you reckon could come from accessing data?