Hi all,
I haven't used Vapor in a few years but am gearing up to use it a lot with some new data. I installed the latest version of the Vapor software and tried running with my old shell script to generate the Vapor data files. But things have changed a bit since I last used the software and I am trying to figure out how to convert my old code to something that works. For instance some of the flags I am passing are deprecated and do not work (like -dimnames on ncdf2vdf). My old script is below. I looked at the man pages for vdfcreate and also the (new?) ncdfvdfcreate command. I actually did find the correct flags (I think) to pass to vdfcreate but I am stuck with ncdf2vdf complaining about my input netcdf files.
The way I do things, I create my netcdf files (that ncdf2vdf operates on) on the fly from my own conversion tool (below, hdf2ncvapor - which sticks the ELEVATION 3D field to deal with a stretched vertical mesh; this does not apply to my current data however as I am on an isotropic grid) that operates on the model output format that I have. It seems as if some of the new tools rely on existing datasets in the netcdf file that weren't necessarily required before (specifically time arrays?).
Do I need to use the new ncdfvdfcreate instead of the old vdfcreate? Do I have to create time arrays for each of my netcdf files? Can I loop over variables as I did before, or do I pass all of the variables to the ncdf2vdf command?
Thanks for any help....I looked at the online docs and couldn't find examples that worked.
Here is my old conversion script which worked say 5 years ago:
#!/bin/bash
base=dburst
dir=dburst.03924.cdir
t0=4000
t1=5000
dt=5
x0=0
y0=0
x1=449
y1=549
z0=0
z1=119
mkdir -p nc
#first make times file for vdfcreate
for time in $(seq $t0 $dt $t1); do echo $time ; done > times
vdfcreate -dimension 450x550x120 -extents 11.0:11.0:0.0025:20.0:22.0:2.8816 \
-usertime ./times -gridtype layered -level 2 \
-varnames uinterp:vinterp:winterp:hwin:thpert:xvort:yvort:zvort:hvortmag:ppert:qr:ELEVATION den1hr_2011.vdf
let counter=0
for time in $(seq -w $t0 $dt $t1); do
hdf2ncvapor $dir/${base}. nc/${base} $x0 $y0 $x1 $y1 $z0 $z1 $time \
uinterp vinterp winterp hwin thpert xvort yvort zvort hvortmag ppert qr
ncfilename=nc/${base}.0${time}.nc
for i in uinterp vinterp winterp hwin thpert xvort yvort zvort hvortmag ppert qr ELEVATION; do
ncdf2vdf -ts $counter -dimnames nx:ny:nz -varname $i den1hr_2011.vdf ${ncfilename}
done
# /bin/rm $ncfilename
let counter=${counter}+1
done
Hi Leigh,
Sorry we broke your code :-(. When we re-factored the ncdf conversion utilities the intent was to make them easier to use. However, I can see that we missed an important use-case: not having all of the NetCDF files you want to translate on-line at once. In your specific case you're creating temporary NetCDF files, and trying to convert them one at at time. The conversion utilities expect to have access to *all* of the NetCDF files at once (and will process them without the need to script a loop). The only workaround that I can think of with the current utilities is to modifiy your code to add a time dimension and time variable (the latter is optional), and then process everything at once. The downside (in addition to having to change your code) is the additional disk space required.
In the longer term we need to look at adding an option to ncdf2vdf to allow the specification of a time index, but this is not something we will be able to do until the next release in the fall.
Sorry I don't have an easier workaround for you. Let me know if any of this doesn't make sense.
Best,
john
ncdfvdfcreate -dims ${nx}x${ny}x${nz} -extents $gx0:$gy0:$gz0:$gx1:$gy1:$gz1 -timedims time -timevars time \ -level 2 nc/*.nc tmaint.vdfI stuck the time variable in all netcdf files as you can see. This creates a good vdf file. Then I loop over time and run where ncfilename is one of the netcdf files; I loop over them. In both cases I can exploit the embarrassingly parallel nature of creating netcdf files :) So I can do them in chunks of 10 or more at a time on a multicore machine, and it goes pretty fast. My next question to you: While the flow fields look correct (when I do streamtubes) I need to be sure things are in the correct units and I don't know what assumptions Vapor uses (or I've forgotten!). My extents are specified in km and my components of velocity are specified in m/s. I do not specify dx dy or dz; I am assuming Vapor is assuming an isotropic grid (which I do have) and just dividing by the total number of points in each direction to calculate its own dx/dy/dz? Also, my time is in seconds. My netcdf files are spaced in 2 second intervals. Since I need to trace trajectories around I need to be absolutely sure my units are correct. I am wondering whether I need to specify units of time/speed/grid spacing somewhere. Thanks, LeighLeigh,
Glad you have a workaround and hopefully it wasn't too onerous. A couple of tips that might be helpful:
1. You should not need to specify the dimension to ncdfvdfcreate. It wont' hurt anything, but the command should be able to figure it out from the NetCDF input files
2. Consider adding the "-vdc2" option to the ncdfvdfcreate command line. This will generate "VDC Type 2" data. It will slow down the translation somewhat, but the compression results are typically better. This may or may not be helpful to you given the resolution of your data, and how you're using vaporgui. If you're streaming through time steps the VDC 2 format can have a significant benefit.
Regarding the units: unfortunately vapor currently has no support for units whatsoever. To ensure your integrations are accurate you need to make sure all of your input data and the extents us consistent units. I.e. if your velocity field is in m/s than make sure your extents are specified in meters and your time (via the NetCDF time variable) is in seconds.
FYI we're working on a complete re-factoring of the VDC to make it CF (Climate Forecast) conventions compliant, with support for specifying units, coordinate variables, etc. We expect to have a prototype released at the end of this year.
Best,
john