Stream: dask

Topic: Casper data bandwidth tricks?

view this post on Zulip Brian Bonnlander (Sep 09 2021 at 15:31):

I'm trying to use Casper to load/save slices of data from Zarr stores. Essentially the workflow is entirely data movement with very little computation. Does anyone know of tricks to optimize data bandwidth on Casper for this kind of workflow?

view this post on Zulip Haiying Xu (Sep 09 2021 at 15:49):

You shall set zarr chunksize around 384MB or 768MB, and run your job at late night when the network has less traffics.

Last updated: Jan 30 2022 at 12:01 UTC