Stream: pinatubo-LE

Topic: CESM-LE integration

view this post on Zulip Matt Long (Nov 12 2020 at 21:02):

@all, we're planning to repeat a portion of the CESM-LE with different forcing (no Pinatubo). Who has run the LENS tag on Cheyenne? How much do we need to worry about validation?

view this post on Zulip Allison Baker (Nov 12 2020 at 22:35):

I've only run that tag on Yellowstone (same with @Sheri Mickelson I think).

If you are concerned about differences between CESM-LE on Cheyenne vs. Yellowstone, then I would not be concerned (but I would disable AVX2 (FMA) for the Cheyenne compiler) - I did a fair bit of testing with our CESM-ECT tool at the time Cheyenne was new and all was good...

view this post on Zulip Matt Long (Nov 12 2020 at 23:16):

@nan, have you run the LENS tag on Cheyenne?

view this post on Zulip Keith Lindsay (Nov 12 2020 at 23:37):

I think https://svn-ccsm-models.cgd.ucar.edu/cesm1/exp_tags/cesm1_1_2_LENS_n21/ has known problems resolved on cheyenne.
I have a copy at /glade/work/klindsay/cesm1_tags/cesm1_1_2_LENS_n21.

view this post on Zulip Nan Rosenbloom (Nov 17 2020 at 20:46):

Hi Matt. Yes, I've run lots of LENS-tag simulations on cheyenne, using the same tag that Keith Lindsay pointed you to. I've also coordinated several LEs with this tag; let me know if I can help in any way.

view this post on Zulip Michael Levy (Dec 07 2020 at 16:50):

If you are concerned about differences between CESM-LE on Cheyenne vs. Yellowstone, then I would not be concerned (but I would disable AVX2 (FMA) for the Cheyenne compiler) - I did a fair bit of testing with our CESM-ECT tool at the time Cheyenne was new and all was good...

Thanks for this tip! How do you disable AVX2? I'm trying to look at the n21 tag to see if it has already been done, but I don't think it has been

view this post on Zulip Michael Levy (Dec 07 2020 at 16:56):

it looks like maybe I want to add the -no-fma flag, and that it isn't already included? Tagging @Allison Baker because I forgot to do that when I quoted her in a reply :)

view this post on Zulip Michael Levy (Dec 07 2020 at 18:47):

Is it possible that -no-fma is set as a default somewhere for intel 18.0.5? I created two cases and ran them both for one month:

  1. /glade/work/mlevy/codes/pinatubo-LE/cases/b.e11.B20TRC5CNBDRD.f09_g16.001
  2. /glade/work/mlevy/codes/pinatubo-LE/cases/b.e11.B20TRC5CNBDRD.f09_g16.002

I didn't make any changes to the first case, but for the second case I added FFLAGS += -no-fma to Macros; this change shows up in the build logs

$ zgrep fma /glade/scratch/mlevy/b.e11.B20TRC5CNBDRD.f09_g16.001/bld/ocn.bldlog.*.gz | wc -l
0
$ zgrep fma /glade/scratch/mlevy/b.e11.B20TRC5CNBDRD.f09_g16.002/bld/ocn.bldlog.*.gz | wc -l
112

but as far as I can tell, the two runs are bit-for-bit identical.

view this post on Zulip Allison Baker (Dec 07 2020 at 19:06):

So as long as you don't have the -xCORE-AVX2 flag then you may be OK. Since -no-fma did not do anything, then you are probably not using avx2. (If you have -xCORE-AVX2 , then it automatically does fma. It is possible to have both flags -xCORE-AVX2 -no-fma)

view this post on Zulip Michael Levy (Dec 07 2020 at 19:51):

Just for those following along at home -- I talked with @Allison Baker and she agrees that it looks like the current default does not use fma; we think at one point early in the process of porting to cheyenne CSEG had enabled AVX2, which turned fma on by default, but it was turned off when it caused issues with the verification tool and is not enabled in the n21 LENS tag. So we can still compare to the yellowstone output and I don't need to modify anything in the build system.

Last updated: May 16 2025 at 17:14 UTC