puzzle of not reproducable initial runs from restarts

[suvarchal]

Here is what i mean by the title, I am using (branch of) the master branch, the experiment is as follows:

• a cold-start run starting at 0 1 1958 run for 20 steps.
• a restart run until 20th step started from cold-start like above run initially for 10 steps and restarted for 10 next steps.

now comparing output sst (and sss) quickly say using global sum mean square difference as function of time:

exactly around 10th step the runs starts to diverge, I would surprised and assume that it shouldn't be that way. Is is some quirk of time stepping that we cannot save the state exactly?(unlikely i presume). Anyway I feel somewhat model is reading/replacing current time value in restart run from climatology (do_ic3d to read is used to read clim even in restart) then from restart file could that be true?

of course one can even reduce experiments to 2 steps to replicate the behavior.

[koldunovn]

The problem is forcing. When you run consequently the forcing is interpolated between forcing time intervals (e.g. 3 hours). When you doing restart, the closest forcing time step is taken.

[suvarchal]

@koldunovn thanks.

and thanks to @patrickscholz @dsidoren, found a way to test if forcing interpolation are causing this, so we forced the forcings to be constant: 0.0 in gen_forcings_couple.F90 using atmdata=0.0 just above this line

fesom2/src/gen_forcing_couple.F90

Line 255 in 00df069

u_wind = atmdata(i_xwind,:)

and returning immediately after the block of initialization.

did a new set of same experiment as above, the differences are significantly less (blue line below) not but there is probably one more issue somewhere else.

I guess we should at some point fix this by interpolating consistently with whats done in regular time stepping loop or by less elegantly saving forcings in the restarts like @patrickscholz suggests would be easy thing to do.

[JanStreffing]

Is this something we should invest time in still? See also: #350