Stream: fisheries

Topic: DPLE

Kristen Krumhardt (May 28 2021 at 20:28):

Hello @Colleen Petrik I've found my notebook that processes DPLE and does the drift correction. I think it should still work just fine. However, I need to get the unprocessed DPLE data. Before (when I was processing a bunch of DPLE variables) I usually grabbed the data files off tape for the variables I needed (I think they were compressed tar files) and put it on my scratch space (then deleted them after I processed them). However, now I can't remember where it is on tape (and my post-it that has the path to the data on tape is hanging on my office wall at NCAR-- not too useful!) @Matt Long do you know where i can find the orginal dple output? Not sure if it would be only on tape ... or if it's located elsewhere on glade..

Michael Levy (May 28 2021 at 21:00):

@Kristen Krumhardt -- there's a DPLE directory on campaign: /glade/campaign/cesm/collections/CESM1-DPLE I don't know much about the experiment, so no idea how complete the data set is... it could be the case that Gary is still moving data from HPSS over

Kristen Krumhardt (May 28 2021 at 21:05):

Thanks @Michael Levy !! I think that directory should have all the data we need (and the files are already in netcdf rather than tar - yay! one less step:))

Matt Long (May 29 2021 at 13:24):

DPLE data is here

/glade/campaign/cesm/collections/CESM1-DPLE

@Stephen Yeager or @Elizabeth Maroon have you advanced the state-of-the-art in DPLE processing/drift correction? Could you point @Colleen Petrik and @Kristen Krumhardt to a latest example code? We're spinning up to assess predictability of fishes.

Kristen Krumhardt (May 29 2021 at 13:45):

The notebooks I have come from @Elizabeth Maroon .

Elizabeth Maroon (May 29 2021 at 18:23):

The notebooks that @Kristen Krumhardt has are pretty old, and while they do get the job done, Steve, Who, and I have been iterating on a cleaner/better notebook over the past few months. Adding a drift correction function to them is currently on my desk. I'll put that on the short list of things to get done next week and get back to you soon.

For now, you can find the current example notebook that does everything except drift correction here: https://github.com/NCAR/RAPCDI-analysis/blob/main/notebooks/DPLE_ENSO_check.ipynb

Matt Long (May 29 2021 at 20:28):

Thanks!

Kristen Krumhardt (Jun 21 2021 at 16:36):

Hello, just wanted to provide an update here. I used @Elizabeth Maroon 's new notebook for processing the DPLE and was able to get some of the necessary variables for FEISTY processed. Right now I'm just trying to process 2 year (24 month) forecasts from 1970 to 2018 (just as a starting point). I've combined Liz's notebook (with her new drift correction) with some functions from the proc-cesm-dple-fields.ipynb notebook from the fish-offline repo. I successfully processed 150m mean temp, and depth integrated spC, diatC, zooC, and zoo loss variables. the output is here: /glade/work/kristenk/fish-offline/ . However, I'm having trouble with making the bottom field variables for TEMP and POC_FLUX_IN. My notebook is here and you can see the error I'm getting concerning the field_at_bottom function when I try to make a quick map of the anomalies (I get the same error when trying to load the dso_anoms before writing out the netcdf): https://github.com/kristenkrumhardt/fish-offline/blob/kristens_branch/notebooks/DPLE_process_FEISTY_fields.ipynb

I worked with @Michael Levy briefly on friday to try to find a solution (our attempts are in a copy of the above notebook, not yet in my branch of the fish-offline repo), but we weren't able to fix it yet...

@Matt Long do you have any suggestions on how to proceed?

Kristen Krumhardt (Jul 06 2021 at 14:45):

Hello! I am finally through processing the DPLE variables necessary for FEISTY. You'll find monthly drift-corrected anomalies for 150m integrated spC, diatC, zooC, and zoo_loss, 150m mean TEMP, and bottom fields for TEMP and POC_FLUX_IN here:

/glade/scratch/kristenk/fish-offline

I did all forecast start years (1954-2017), 10 year (122 month) forecasts. @Colleen Petrik let me know if you have questions! Thanks :)

Colleen Petrik (Jul 14 2021 at 21:46):

Thanks @Kristen Krumhardt ! Does this DPLPE use a different gridspec than the FOSI with grid-data-POP_gx1v6.nc? I don't think it does, but I'm trying to understand why the variables seem to have a different number of non-nan grid cells from that and from each other. Do you know why only pelagic temperature agrees?
For example,
bottom temp 86199
bottom POC 86199
diatC 85813
pelagic temp 86212
zooC 85813
zoo loss 85813
POP 86212

Kristen Krumhardt (Jul 14 2021 at 22:08):

hmmm that is a good question! I will look into it and let you know:)

Matt Long (Jul 15 2021 at 12:58):

The DPLE and FOSI use the same grid. What do you mean by

Do you know why only pelagic temperature agrees?

Also, what is that list of numbers?

Do you see differences in the NaNs if you manually mask our land? You can do this as follows:

field_masked = field.where(KMT > 0)

for example.

Colleen Petrik (Jul 16 2021 at 23:07):

I used the region_mask in the POP grid data file to create the land_mask, but I just tested your method using KMT instead. Both of those methods give 86212 ocean cells. I like to verify that all of these grid cells contain non-nan (or non-FillValue) values for each forcing variable otherwise it doesn't make sense to run FEISTY in those cells.

All the numbers I listed are the number of non-nan grid cells for that variable for a given time. For example,

sum( find( ~isnan( diatC(:,:,1) ) ) ) = 85813

Which means that there are some ocean cells of the DPLE files missing variables even though they aren't land. This was not the case for the FOSI run where all the forcing variables listed had 86212 non-nan cells, the same number as the land-masked cells.

Matt Long (Jul 22 2021 at 19:20):

@Colleen Petrik, here's a path to the CESM-LENS tag code base:
/glade/p/cgd/oce/people/mclong/cesm/cesm1_1_2_LENS_n19/models/ocn/pop2/source

Matt Long (Jul 22 2021 at 19:21):

Here is the equation for zoo_loss

zoo_loss = z_mort2 * Zprime**1.4_r8 + z_mort * Zprime

Matt Long (Jul 22 2021 at 19:28):

   thres_z1          = 100.0e2_r8
   thres_z2          = 200.0e2_r8
   loss_thres_zoo    = 0.2_r8

  parm_z_mort_0  = 0.08_r8 * dps
  parm_z_mort2_0  = 0.42_r8 * dps

   z_mort2   = parm_z_mort2_0   * Tfunc
   z_mort    = parm_z_mort_0    * Tfunc

   if (zt(k) > thres_z1) then
      if (zt(k) < thres_z2) then
         f_loss_thres = (thres_z2 - zt(k))/(thres_z2 - thres_z1)
      else
         f_loss_thres = c0
      endif
   else
      f_loss_thres = c1
   endif

   C_loss_thres = f_loss_thres * loss_thres_zoo
   Zprime = max(zooC_loc - C_loss_thres, c0)

  zoo_loss = z_mort2 * Zprime**1.4_r8 + z_mort * Zprime

Colleen Petrik (Jul 22 2021 at 21:38):

More on the discrepancies in the number of non-NaN grid cells in the DPLE. The integrated variables (diatC, spC, zooC, zoo_loss) have extra NaNs in the Baltic, Black, Caspian, and Red Seas. However, the bottom variables (TEMP_bottom and POC_flux) have extra NaNs in 13 distinct locations:
Lat Lon
_______ ______
-76.015 182.19
-76.015 183.31
-76.015 184.44
-74.947 325.06
-74.947 326.19
-74.947 327.31
-74.947 328.44
60.869 355.98
61.279 355.73
61.688 355.46
67.058 334.03
67.455 333.73
67.854 333.41

Matt Long (Jul 23 2021 at 18:54):

Ok, so we should be ignoring "marginal seas," which are designated by negative values in the REGION_MASK variable.

The points you note as different are likely "src" locations for the "overflow" parameterization, which "pops" up KMT. The pop_tools.get_grid method does not have these "pop up" locations, so where we use KMT from that tool, we're probably indexing KMT + 1 and getting a NaN.

@Kristen Krumhardt, can you work with @Colleen Petrik to get these discrepancies reconciled?

Matt Long (Jul 23 2021 at 18:57):

Ideally, we would add code to pop_tools.get_grid that reads the overflow input file here and applies the KMT modifications.

Colleen Petrik (Aug 02 2021 at 22:33):

The DPLE outputs Kristen kindly post-processed are drift-corrected anomalies. Can I get actual values for forcing FEISTY by adding these anomalies to the appropriate FOSI year and month?

Matt Long (Aug 03 2021 at 15:27):

@Colleen Petrik, I think there is an order of operations question here and I apologize for not catching this sooner. It seems to me that the most appropriate approach would be to compute the fish on the DPLE output—and then apply the drift correction procedure to generate forcast anomalies in the fish.

Kristen Krumhardt (Aug 03 2021 at 15:44):

I can easily generate non-drift corrected DPLE output for the variables needed for FEISTY. Then we can do the drift correction on the FEISTY output later on.. @Colleen Petrik , I'll let you know when I've got these files ready for you:)

Colleen Petrik (Aug 04 2021 at 01:54):

Thanks @Kristen Krumhardt and @Matt Long .

Matt Long (Aug 05 2021 at 20:21):

Welcome @Zhuomin Chen to Zulip!

@Zhuomin Chen just started as a researcher at UCONN and will be analyzing the DPLE

Zhuomin Chen (Aug 05 2021 at 21:26):

Matt Long said:

Welcome Zhuomin Chen to Zulip!

Zhuomin Chen just started as a researcher at UCONN and will be analyzing the DPLE

Thank you Matt! I am new to Zulip, and very happy to join this forum!

Colleen Petrik (Aug 06 2021 at 00:57):

@Matt Long , I couldn't find 3 parameters in the ecosys code: c0 (zero?), c1 (one?), and T0_Kelvin (273.15?). Just want to be sure before moving forward on the quadratic mortality back calculation.

Matt Long (Aug 06 2021 at 15:01):

you got it, except for some reason, POP uses T0_Kelvin = 273.16, as you can see here. The other constants are defined in that module too.

Kristen Krumhardt (Aug 12 2021 at 17:59):

Hi @Colleen Petrik ! I prepared the non-drift corrected FEISTY variables. They are here: /glade/scratch/kristenk/fish-offline/*_nodriftcorr.nc

Colleen Petrik (Aug 13 2021 at 01:07):

Excellent, thanks @Kristen Krumhardt ! I'll start working with these. Could you also help me process additional FOSI outputs? I tried to modify the proc-cesm-dple-fields.ipynb code to include all 3 phytoplankton production individual outputs (integrated over top 150m), but I'm having troubling starting a dask cluster.

Kristen Krumhardt (Aug 13 2021 at 16:04):

Hi @Colleen Petrik , I can definitely help with this. So, just to clarify, do you need the photoC_sp, photoC_diat, and photoC_diaz terms depth integrated from the FOSI? these are net primary production from each phytoplankton functional type. I ran my version of proc_cesm-dple-fields.ipynb (with an updated dask part) on those three variables. Output is here: /glade/work/kristenk/fish-offline/g.e11_LENS.GECOIAF.T62_g16.009.phytoNPP_FIESTY-forcing.nc

Colleen Petrik (Aug 13 2021 at 17:30):

Thanks, you're the best! I'm going to test if phyto production rather than phyto biomass is a better way to split the zoo into small and large. I'll let you know how it goes.

Matt Long (Aug 19 2021 at 21:07):

@Kristen Krumhardt, Can you help get @Zhuomin Chen pointed in the right direction working with the DPLE output? She is ramping up on an analysis looking at predictability of O2, pO2, and T. In particular, it would be helpful to point her to (and perhaps walk her thru) the drift correction/analysis codes you have.

cc @Stephen Yeager, @Elizabeth Maroon, @Samantha Siedlecki

Kristen Krumhardt (Aug 19 2021 at 21:25):

Sure, @Matt Long ! Hi @Zhuomin Chen ! I'd be delighted to help you out. I have a few notebooks that have been working nicely for me here: /glade/u/home/kristenk/fish-offline/notebooks .. sometimes it helps to preprocess data into a 2-D variable before doing the drift correction. I think it might be best to set up a meeting so I can walk you through these notebooks. I'll send you a PM to set up a time to meet.

Matt Long (Aug 19 2021 at 21:27):

@Stephen Yeager, @Elizabeth Maroon, please weigh in if you have important updates...or possibly this is an opportunity to expand your circle of potential developers.

Colleen Petrik (Sep 01 2021 at 22:10):

Kristen Krumhardt said:

... just to clarify, do you need the photoC_sp, photoC_diat, and photoC_diaz terms depth integrated from the FOSI? these are net primary production from each phytoplankton functional type. I ran my version of proc_cesm-dple-fields.ipynb (with an updated dask part) on those three variables...

Using all 3 phytoplankton types seemed to help with splitting the zooplankton into mesozooplankton, leading to better regional variability (oligotrophic vs. eutrophic areas) of the fish biomass. Could you please run your version of proc_cesm-dple-fields.ipynb to get the integrated diazC fields from the FOSI? I'm trying a few more parameter tests, but we may need to include this output with the full DPLE.

Colleen Petrik (Sep 01 2021 at 22:48):

@Kristen Krumhardt , didn't realize that the above wouldn't ping you.

Kristen Krumhardt (Sep 02 2021 at 13:19):

@Colleen Petrik Sure, I'll do the diazC FOSI processing today and let you know when it's done.

Also, no problem to process diazC for the DPLE, just let me know:)

Kristen Krumhardt (Sep 02 2021 at 18:05):

Hi @Colleen Petrik , the FOSI diazC file is here:
/glade/work/kristenk/fish-offline/g.e11_LENS.GECOIAF.T62_g16.009.diazC_FIESTY-forcing.nc

Colleen Petrik (Sep 02 2021 at 18:43):

Thanks, @Kristen Krumhardt ! And I finally was able to run one of your notebooks, so hopefully I won't have to ask for such minor requests in the future.

Colleen Petrik (Oct 20 2021 at 22:35):

Hi @Kristen Krumhardt. The biomass of all 3 phytoplankton was better than just diat and small. Could you please update and reprocess the DPLE notebook to create the FEISTY outputs with all 3 phytoC variables? Thanks!

Kristen Krumhardt (Oct 20 2021 at 22:45):

Hi @Colleen Petrik ! So you just need DPLE diazC integrated from 150m, right? no drift correction, right?

Colleen Petrik (Oct 21 2021 at 17:00):

Yes, I need the diazC 150m integration in regular biomass units in addition to zooC, zoo_loss, diatC, spC, etc., non-drift corrected.

Kristen Krumhardt (Oct 21 2021 at 18:43):

Ok, I'll let you know when I'm done processing this (probably tomorrow)

Kristen Krumhardt (Oct 22 2021 at 18:04):

Hi @Colleen Petrik , I just finished processing DPLE depth integrated diazC (no drift correction). All output is here: /glade/scratch/kristenk/fish-offline. please let me know if there's anything missing

Colleen Petrik (Dec 02 2021 at 00:04):

Thanks, @Kristen Krumhardt ! Looks good so far.

Colleen Petrik (Jan 19 2022 at 21:23):

@Kristen Krumhardt , after discussing order of operations again with Matt and Charlie, we decided to use the drift-corrected anomalies (added to the climatology to get fish relevant values) to force FEISTY. Would you mind processing these again? I need the 150m mean of temp, bottom temp, bottom POC, and the 150m integration of zooC, zoo_loss, diatC, spC, diazC. The diazC might not have been in your notebook the last time you did this.

Kristen Krumhardt (Jan 19 2022 at 21:27):

Hi Colleen! Sure I will start on this shortly. If I remember correctly, there are two steps to make these drift-corrected anomalies, so it may take a bit of time. I will also add diazC to my workflow. Will be in touch when I have an idea on when this will all be ready :) (hopefully within the next week or so)

Colleen Petrik (Jan 21 2022 at 20:30):

@Kristen Krumhardt , thank you!

Kristen Krumhardt (Jan 26 2022 at 14:21):

Hi @Colleen Petrik! Drift-corrected anomalies for forcing FEISTY are ready. They are here: /glade/scratch/kristenk/fish-offline/ .

Colleen Petrik (Mar 02 2022 at 23:10):

Hey @Kristen Krumhardt , maybe I took too long to work with these files, but it seems that TEMP_bottom and diazC_150m files are missing.

Also, does your drift correction notebook have a good example of how to match up the DPLE ensemble member with the equivalent time in the FOSI? Thanks!

Kristen Krumhardt (Mar 03 2022 at 00:17):

Hi @Colleen Petrik , are you looking here for the DPLE files?: /glade/scratch/kristenk/fish-offline

I think all the files are there... I actually haven't done any time alignment of the DPLE and FOSI in any of my notebooks but I think I might be able to find you an example from the SMYLE repo.. I'll take a look soon and let you know :)

Colleen Petrik (Mar 03 2022 at 00:19):

I was looking at your home folder :face_palm:
They are all in scratch.

Kristen Krumhardt (Mar 04 2022 at 16:33):

Hey @Colleen Petrik , glad you found the files :) I tried to find an example for lining up the FOSI with DPLE ensemble members in the SMYLE repo.. so I haven't found precisely that but there's an example of lining up ERA5 sea level pressure observational data with the DPLE ensemble mean in this notebook (it's not my notebook, but it's in the SMYLE repo so I have a copy of it in my directory): /glade/u/home/kristenk/SMYLE-analysis/notebooks/PaperFigsSMYLEvsDPLE_GlobMaps_SLP.ipynb

Look at cells 22 and 23. There's a function called leadtime_corr_byseas that uses the xarray align function. that notebook uses seasonal averages rather than monthly, though. Not sure if that's helpful...

Colleen Petrik (Mar 04 2022 at 18:35):

Thanks, @Kristen Krumhardt , I'll look into the function in this notebook. @Matt Long , do you have any other suggestions for aligning the DPLE ensemble members with the equivalent time in the FOSI?

Matt Long (Mar 04 2022 at 18:44):

I think it's mainly a question of careful indexing. unfortunately, I don't think I have a clear example to point you to.

Michael Levy (Mar 16 2022 at 17:23):

@Colleen Petrik Like we talked about yesterday, I'm going to do one more set of "matlab vs python" comparisons using FOSI_cesm.m; I used daily_interp_cesm_fosi_totmort_extend_LfracB_allphytos.m to generate 68 Data_cesm_fosi_v6_daily_NN.mat files (NN in 01 -> 68). I also found Data_grid_POP_gx1v6_noSeas.mat (I think I copied it out of your directory on cheyenne); the last thing I'm missing is the initialization file. The .m file is trying to run

load(['/Volumes/MIP/NC/CESM_MAPP/',simname '/Last_mo_spin_' init_sim '.mat']);

I changed all the /Volumes/MIP/NC paths to input_data in FOSI_cesm.m and sub_fname_cesm_fosi_exper.m, but the former is now giving me an error because it can't find the file Dc_Lam700_enc70-b200_m400-b175-k086_c20-b250_D075_A050_sMZ090_mMZ045_nmort1_BE08_CC80_RE00100/Last_mo_spin_v14_Dc_Lam700_enc70-b200_m400-b175-k086_c20-b250_D075_A050_sMZ090_mMZ045_nmort1_BE08_CC80_RE00100.mat. Do you have this on your laptop? Could you either copy it to cheyenne or let me know how to generate it?

Thanks!

Colleen Petrik (Mar 21 2022 at 22:37):

@Michael Levy , I totally meant to upload this on Tuesday before going out of town. I uploaded the initialization file to the input_data folder. I also added an example of the daily forcing files Data_cesm_fosi_daily_1.mat. I will take a look at the daily files you created.

Colleen Petrik (Mar 21 2022 at 22:44):

Where can I find the daily_interp_cesm_fosi_totmort_extend_LfracB_allphytos.m file?

Michael Levy (Mar 21 2022 at 22:59):

@Colleen Petrik /glade/work/mlevy/codes/fish-offline/MatFEISTY/cesm_mfiles/daily_interp_cesm_fosi_totmort_extend_LfracB_allphytos.m

Michael Levy (Mar 23 2022 at 19:58):

@Colleen Petrik I'm seeing differences between matlab's FOSI results and python's FOSI results, which I think might be due to different parameterizations. The differences between the two setups are as follows (< is testcase, > is FOSI):

62c62
<     param.CC = 0; % 80
---
>     param.CC = 80; % 80
91,94d85
<
<     param.D  = 0.75;   %Demersal feeding in pelagic reduction
<     param.A  = 0.5;    %Adult predation reduction %*****
<     param.MZ = 1.0;    %Preference on one mesozooplankton group
96,97c87,91
<     param.MF_phi_MZ = 0.45 * param.MZ;
<     param.MF_phi_LZ = 1.0;
---
>     param.D  = 0.75;  %Demersal feeding in pelagic reduction
>     param.A  = 0.5;   %Adult predation reduction %*****
>     param.MZ = 0.9;   %Preference on one mesozooplankton group
>
>     param.MF_phi_MZ = 0.5 * param.MZ;
100,101c94
<     param.MP_phi_MZ = 0.45 * param.MZ;
<     param.MP_phi_LZ = 1.0;
---
>     param.MP_phi_MZ = 0.5 * param.MZ;

I was able to change the carrying capacity, but I'm not sure what to make of the rest.

1. What are the values of param.MF_phi_LZ and param.MP_phi_LZ if they aren't specified in this file?
2. Is param.MZ = 1.0; used directly? Because it looks like param.MF_phi_MZ and param.MP_phi_MZ are unchanged (0.5*0.9 == 0.45*1.0), but maybe other values need to change from 1 -> 0.9?

Michael Levy (Mar 23 2022 at 20:19):

Actually, I think I figured it out -- param.MF_phi_LZ and param.MP_phi_LZ are used in sub_futbio.m but not sub_futbio_1meso.m so we don't need it, and param.MZ is the preference rate of Sf, Sp, and Sd for zoo. That might not be the cleanest definition, but it's changing preference from 1. to 0.9 below (I added the comment as a helpful reminder ):

food_web:
  # small forage
  - predator: Sf
    prey: Zoo
    preference: 1. # [params.MZ]

  # small piscivore
  - predator: Sp
    prey: Zoo
    preference: 1. # [params.MZ]

  # small demersal
  - predator: Sd
    prey: Zoo
    preference: 1. # [params.MZ]

Michael Levy (Mar 23 2022 at 20:41):

Differences between matlab and python results in a 10-day FOSI-forced run:

filtering out values < 1e-300, things look promising:

performance is an issue, though. Running the matlab code for a year takes about 90s, while the python is in the neighborhood of 20s / day (more than 2 hours / year). That's a factor of 80 slower, and I thought we had gotten closer to 50x but even that would be 75 minutes per year...

Colleen Petrik (Mar 23 2022 at 21:38):

You should be using sub_futbio_1meso.m not sub_futbio.m because we only have one zooplankton group from CESM.
Screen-Shot-2022-03-23-at-2.33.42-PM.png
The correct prey preferences are (ones we eventually want to use for global runs):
param.CC = 80
param.D = 0.75; %Demersal feeding in pelagic reduction
param.A = 0.5; %Adult predation reduction
param.MZ = 0.9; %Preference on one mesozooplankton group
param.MF_phi_MZ = 0.5 * param.MZ;
param.MP_phi_MZ = 0.5 * param.MZ;

Michael Levy (Mar 24 2022 at 01:50):

Okay, I ran for a full year and it wasn't quite as slow as I feared (but was still really slow): 93 minutes (so a little over 60x slower than the matlab code). Results looked okay, though:

Colleen Petrik (Apr 08 2022 at 00:17):

@Matt Long and @Michael Levy , did the path to the DPLE change? When I try accessing it I am told that the directory does not exist.

Matt Long (Apr 08 2022 at 01:52):

Are you trying to access /glade/campaign from the Cheyenne login node? That’s not possible; campaign is only accessible via Casper.

Last updated: May 16 2025 at 17:14 UTC