Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

OSDF vs S3 streaming benchmark to EC2 (CESM2-LENS, us-west-2)

Overview

Compare data-access bandwidth to an EC2 instance for the same CESM2-LENS zarr store, read three ways:

  • S3 — native s3://ncar-cesm2-lens/... via s3fs (anonymous).

  • OSDF (cached) — osdf:///aws-opendata/us-west-2/ncar-cesm2-lens/... via pelicanfs, served through the OSDF director + cache nodes.

  • OSDF-direct — the same OSDF path with direct_reads=True, bypassing the cache to stream straight from the origin.

The bucket is in us-west-2, so this must run on an EC2 instance in us-west-2 for a fair, same-region comparison. Comparing OSDF-direct with OSDF-cached isolates the cache’s effect; comparing OSDF-direct with S3 isolates the pelican-client + origin overhead.

import os, time
from contextlib import contextmanager
import numpy as np
import pandas as pd
import xarray as xr
import fsspec
from pelicanfs.core import OSDFFileSystem   # importing registers the osdf:// protocol
import matplotlib.pyplot as plt
import seaborn as sns
from dask.distributed import Client, LocalCluster
import logging, warnings
warnings.filterwarnings("ignore", category=ResourceWarning)
for name in ("aiobotocore", "aiohttp", "botocore", "s3fs"):
    logging.getLogger(name).setLevel(logging.CRITICAL)
# m5.4xlarge = 16 vCPU / 64 GB -> 4 workers x 4 threads
cluster = LocalCluster(n_workers=4, threads_per_worker=4)
client = Client(cluster)
client
Loading...
store    = "atm/daily/cesm2LE-historical-smbb-TREFHTMX.zarr"
variable = "TREFHTMX"
s3_url   = f"s3://ncar-cesm2-lens/{store}"
osdf_url = f"osdf:///aws-opendata/us-west-2/ncar-cesm2-lens/{store}"
print("S3  :", s3_url)
print("OSDF:", osdf_url)
S3  : s3://ncar-cesm2-lens/atm/daily/cesm2LE-historical-smbb-TREFHTMX.zarr
OSDF: osdf:///aws-opendata/us-west-2/ncar-cesm2-lens/atm/daily/cesm2LE-historical-smbb-TREFHTMX.zarr

ds_s3   = xr.open_zarr(fsspec.get_mapper(s3_url, anon=True), consolidated=True, zarr_format=2)[variable]
ds_osdf = xr.open_zarr(fsspec.get_mapper(osdf_url),consolidated=True, zarr_format=2)[variable]
ds_s3
Loading...

Build increasing-size subsets

The store is chunked [member=1, time=730, lat=192, lon=288] (float32) ≈ 161 MB per chunk, so the smallest meaningful transfer is one time-chunk. We sweep request size by reading n whole chunks (n = 1, 2, 4, 8, 16 → ≈ 161 MB … 2.6 GB).

To keep every first read genuinely uncached:

  • each size draws from a disjoint block of time-chunks (via offsets), so a larger read is never pre-warmed by a smaller one within a pass; and

  • each outer try uses a different member_id (see the run cell), so repeats across passes hit fresh objects.

# OSDF direct reads = bypass the cache node, stream straight from the origin
ds_osdf_direct = xr.open_zarr(fsspec.get_mapper(osdf_url, direct_reads=True),
                              consolidated=True, zarr_format=2)[variable]
# Fallback if your pelicanfs doesn't accept direct_reads via get_mapper:
#   from pelicanfs.core import PelicanMap
#   fs = OSDFFileSystem(direct_reads=True)
#   ds_osdf_direct = xr.open_zarr(PelicanMap(osdf_url.replace("osdf://",""), pelfs=fs),
#                                 consolidated=True, zarr_format=2)[variable]
ds_osdf_direct
Loading...
chunk_time = 730
n_chunks   = [1, 2, 4, 8, 16]                   # ~0.16, 0.32, 0.65, 1.3, 2.6 GB
offsets    = np.cumsum([0] + n_chunks[:-1])     # disjoint time-block per size: [0,1,3,7,15]

def subset(da, n, member, start_chunk):
    t0 = start_chunk * chunk_time
    return da.isel(member_id=member, time=slice(t0, t0 + n * chunk_time))
class DiagnosticTimer:
    def __init__(self):
        self.diagnostics = []
    @contextmanager
    def time(self, **kwargs):
        tic = time.time()
        yield
        toc = time.time()
        kwargs["runtime"] = toc - tic
        self.diagnostics.append(kwargs)
    def dataframe(self):
        return pd.DataFrame(self.diagnostics)
csv_file_path = "ec2_uswest2_osdf_vs_s3.csv"
N_TRIES = 3            # OUTER loop: independent passes, a new member_id each pass
N_CALLS = 3            # INNER loop: reads per subset (call 1 = first, 2..N = subsequent)
members = [3, 4, 5]    # len(members) == N_TRIES
n_workers = 4
def benchmark_da(da, protocol, member, try_id, diag):
    for n, start in zip(n_chunks, offsets):
        ds = subset(da, n, member, int(start))
        size_mb = ds.nbytes / 1e6
        for call in range(1, N_CALLS + 1):
            cluster.scale(0); cluster.scale(n_workers); client.wait_for_workers(n_workers)
            try:
                with diag.time(dataset_size=size_mb, protocol=protocol, call_number=call,
                               try_id=try_id, member=member):
                    ds.copy().load()
                status = "ok"
            except Exception as e:
                diag.diagnostics.append({"dataset_size": size_mb, "protocol": protocol,
                    "call_number": call, "try_id": try_id, "member": member,
                    "runtime": np.nan, "error": f"{type(e).__name__}: {str(e)[:80]}"})
                status = f"FAIL {type(e).__name__}"
            print(f"try {try_id}  {protocol:11s}  member {member}  {size_mb:8.1f} MB  call {call}/{N_CALLS}  [{status}]")
protocols = [("S3", ds_s3), ("OSDF", ds_osdf), ("OSDF-direct", ds_osdf_direct)]
diag = DiagnosticTimer()
for try_id, member in enumerate(members, start=1):
    for name, da in protocols:
        benchmark_da(da, name, member, try_id, diag)

df = diag.dataframe()
df["MBps"] = df["dataset_size"] / df["runtime"]
df.to_csv(csv_file_path, index=False)
df
try 1  S3           member 0     161.5 MB  call 1/3  [ok]
try 1  S3           member 0     161.5 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77d14ded4980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x77d14dcf25d0>, 17140.588240742), (<aiohttp.client_proto.ResponseHandler object at 0x77d14e0cde50>, 17140.681944892)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x77d14e11ba10>

try 1  S3           member 0     161.5 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x76e881984ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x76e881c825d0>, 17142.959919576), (<aiohttp.client_proto.ResponseHandler object at 0x76e8823a1e50>, 17143.02090031)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x76e8823e3b60>

try 1  S3           member 0     322.9 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f8fa6f84980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f8fa70633e0>, 17145.452747026), (<aiohttp.client_proto.ResponseHandler object at 0x7f8fa76da750>, 17145.780258364), (<aiohttp.client_proto.ResponseHandler object at 0x7f8fa6fcf580>, 17145.914059547), (<aiohttp.client_proto.ResponseHandler object at 0x7f8fa6fce2d0>, 17146.15611302)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f8fa6f65bd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f8fa6f846e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f8fa76d9cd0>, 17144.721652675)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f8fa701f770>

try 1  S3           member 0     322.9 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x74dd94ad4980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x74dd94d1b580>, 17148.872020394), (<aiohttp.client_proto.ResponseHandler object at 0x74dd95521fd0>, 17148.927544574), (<aiohttp.client_proto.ResponseHandler object at 0x74dd95522750>, 17149.192912683), (<aiohttp.client_proto.ResponseHandler object at 0x74dd94d1a890>, 17149.381201576)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x74dd94cb1090>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x74dd94ad46e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x74dd94d8f3e0>, 17148.034198168)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x74dd94d6b770>

try 1  S3           member 0     322.9 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x70ab24b7c980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x70ab4432a750>, 17152.017806993), (<aiohttp.client_proto.ResponseHandler object at 0x70ab4437e5b0>, 17152.215706143), (<aiohttp.client_proto.ResponseHandler object at 0x70ab4421b3e0>, 17152.54578656), (<aiohttp.client_proto.ResponseHandler object at 0x70ab4437f580>, 17152.591251481)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x70ab24b61bd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x70ab24b7c6e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x70ab44329cd0>, 17151.248492049)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x70ab441cf770>

try 1  S3           member 0     645.9 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cb145ee0c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cb145ee27b0>, 17156.419653916), (<aiohttp.client_proto.ResponseHandler object at 0x7cb1461b57b0>, 17156.52668004), (<aiohttp.client_proto.ResponseHandler object at 0x7cb144519590>, 17156.585055205), (<aiohttp.client_proto.ResponseHandler object at 0x7cb1445196d0>, 17156.621703093), (<aiohttp.client_proto.ResponseHandler object at 0x7cb14610a720>, 17156.659459438), (<aiohttp.client_proto.ResponseHandler object at 0x7cb14610b580>, 17156.878817423), (<aiohttp.client_proto.ResponseHandler object at 0x7cb144519450>, 17156.913521431), (<aiohttp.client_proto.ResponseHandler object at 0x7cb145ee2510>, 17156.954331804)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cb144518410>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cb145ee06e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cb145ee0980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cb145ee0ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cb145f14820>, 17154.604353197)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cb146157770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cb1462aabd0>, 17154.540577662)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cb145eba490>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cb1462aa8d0>, 17154.664914485)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cb145ebae90>

try 1  S3           member 0     645.9 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7614a7510c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7614a77e68d0>, 17161.141119311), (<aiohttp.client_proto.ResponseHandler object at 0x7614a783efc0>, 17161.271178854), (<aiohttp.client_proto.ResponseHandler object at 0x7614a75127b0>, 17161.379060808), (<aiohttp.client_proto.ResponseHandler object at 0x7614a7531590>, 17161.436384969), (<aiohttp.client_proto.ResponseHandler object at 0x7614a7531450>, 17161.478069782), (<aiohttp.client_proto.ResponseHandler object at 0x7614a75316d0>, 17161.9735532), (<aiohttp.client_proto.ResponseHandler object at 0x7614a7512510>, 17162.459900511), (<aiohttp.client_proto.ResponseHandler object at 0x7614a76e9e90>, 17162.695535421)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7614a7530410>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7614a75106e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7614a7510980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7614a7510ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7614a752c820>, 17159.708253524)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7614a788b770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7614a783dff0>, 17159.831808228)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7614a74d3110>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7614a77e6bd0>, 17159.771439609)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7614a74d2850>

try 1  S3           member 0     645.9 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f556e980c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f556ec662d0>, 17166.187860195), (<aiohttp.client_proto.ResponseHandler object at 0x7f556e9b0cd0>, 17166.460078062), (<aiohttp.client_proto.ResponseHandler object at 0x7f556eb69650>, 17166.579339204), (<aiohttp.client_proto.ResponseHandler object at 0x7f556e9b1950>, 17166.662897096), (<aiohttp.client_proto.ResponseHandler object at 0x7f556e982660>, 17166.808817665), (<aiohttp.client_proto.ResponseHandler object at 0x7f556e9823c0>, 17167.027351245), (<aiohttp.client_proto.ResponseHandler object at 0x7f556ecb76f0>, 17167.317194702), (<aiohttp.client_proto.ResponseHandler object at 0x7f556e9b1810>, 17167.616598546)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f556e9b0910>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f556e9806e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f556e980980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f556e980ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f556ecb6b70>, 17165.128461394)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f556eb0b770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f556e9ac820>, 17165.006696774)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f556e94b750>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f556ec668d0>, 17165.068800522)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f556e9b0050>

try 1  S3           member 0    1291.7 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75fb6c534ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x75fb6c4fa2b0>, 17171.412194106), (<aiohttp.client_proto.ResponseHandler object at 0x75fb6ceb5fd0>, 17171.716886084)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x75fb6c75bb60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75b8261dd160>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x75b826503d40>, 17171.846826634), (<aiohttp.client_proto.ResponseHandler object at 0x75b826541d30>, 17172.072069014), (<aiohttp.client_proto.ResponseHandler object at 0x75b82668ea00>, 17172.165272011), (<aiohttp.client_proto.ResponseHandler object at 0x75b82668efc0>, 17172.215029792), (<aiohttp.client_proto.ResponseHandler object at 0x75b8261de660>, 17172.445189658), (<aiohttp.client_proto.ResponseHandler object at 0x75b8261de900>, 17172.743381084)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x75b8261b7d90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75b8261dcd70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75b8261dd010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x75b8265a2750>, 17170.809525015)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x75b826643e00>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x75b8265a2450>, 17170.686376621)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x75b8261b6350>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7eb5b74d0c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7eb5b770ace0>, 17171.627103527), (<aiohttp.client_proto.ResponseHandler object at 0x7eb5b74d2510>, 17171.812266057), (<aiohttp.client_proto.ResponseHandler object at 0x7eb5b75081e0>, 17171.860471997), (<aiohttp.client_proto.ResponseHandler object at 0x7eb5b74f96d0>, 17171.861945692), (<aiohttp.client_proto.ResponseHandler object at 0x7eb5b77b17b0>, 17171.938466781), (<aiohttp.client_proto.ResponseHandler object at 0x7eb5b74f9950>, 17172.158077024), (<aiohttp.client_proto.ResponseHandler object at 0x7eb5b74f9810>, 17172.510485406), (<aiohttp.client_proto.ResponseHandler object at 0x7eb5b74d27b0>, 17172.809272538)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7eb5b74f8550>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7eb5b74d06e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7eb5b74d0980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7eb5b74d0ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7eb5b770aa00>, 17170.024304407)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7eb5b775b770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7eb5b78228d0>, 17169.9644747)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7eb5b7697610>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7eb5b78222d0>, 17169.84004083)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7eb5b7697c50>

try 1  S3           member 0    1291.7 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x79ff750b8c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x79ff75a668d0>, 17178.532457369), (<aiohttp.client_proto.ResponseHandler object at 0x79ff75a65e50>, 17178.597838986), (<aiohttp.client_proto.ResponseHandler object at 0x79ff752f6890>, 17178.686895851), (<aiohttp.client_proto.ResponseHandler object at 0x79ff752f7580>, 17178.9471622)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x79ff750a1590>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x79ff750b8980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x79ff753a70c0>, 17177.497484377)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x79ff752e7a10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a5a20d98ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a5a210d6150>, 17177.585789437), (<aiohttp.client_proto.ResponseHandler object at 0x7a5a211c3410>, 17177.671684071), (<aiohttp.client_proto.ResponseHandler object at 0x7a5a211c2e50>, 17177.724592228), (<aiohttp.client_proto.ResponseHandler object at 0x7a5a210d68d0>, 17177.985870016)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a5a20d81e50>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a5a20d98830>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a5a21023570>, 17176.888767672)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a5a20fcb8c0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9b4759fa10>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9b4c4868d0>, 17178.538085177), (<aiohttp.client_proto.ResponseHandler object at 0x7c9b475b2e50>, 17178.580996159), (<aiohttp.client_proto.ResponseHandler object at 0x7c9b4c485e50>, 17178.625221397), (<aiohttp.client_proto.ResponseHandler object at 0x7c9b475b3410>, 17178.878824596)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9b4724a0d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9b472606e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9b474970c0>, 17177.552729696)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9b4759fcb0>

try 1  S3           member 0    1291.7 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cd5c47a8c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cd5c4ab70c0>, 17182.74015017), (<aiohttp.client_proto.ResponseHandler object at 0x7cd5c4a91bd0>, 17183.040501969), (<aiohttp.client_proto.ResponseHandler object at 0x7cd5c4bdf580>, 17183.041172159), (<aiohttp.client_proto.ResponseHandler object at 0x7cd5c4bde890>, 17183.119687553), (<aiohttp.client_proto.ResponseHandler object at 0x7cd5c6e6f230>, 17183.344651282), (<aiohttp.client_proto.ResponseHandler object at 0x7cd5c53528d0>, 17183.585712922), (<aiohttp.client_proto.ResponseHandler object at 0x7cd5c6e82490>, 17184.025563837), (<aiohttp.client_proto.ResponseHandler object at 0x7cd5c6e6fb60>, 17184.176195053)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cd5c478da90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cd5c47a8980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cd5c5351e50>, 17177.532766848)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cd5c4bd7a10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a21ef5acec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a21ef5ae3c0>, 17184.568292783), (<aiohttp.client_proto.ResponseHandler object at 0x7a21ef8fa5d0>, 17184.677201459), (<aiohttp.client_proto.ResponseHandler object at 0x7a21ef802fc0>, 17184.720706042), (<aiohttp.client_proto.ResponseHandler object at 0x7a21ef5ae660>, 17184.981449301), (<aiohttp.client_proto.ResponseHandler object at 0x7a21ef8b5d30>, 17184.983635368), (<aiohttp.client_proto.ResponseHandler object at 0x7a21ef802720>, 17185.238323759)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a21ef5979d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a21ef5acad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a21ef5acd70>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a21ef8fabd0>, 17183.187262901)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a21ef7efb60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a21ef6041e0>, 17183.110354905)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a21ef597390>

try 1  S3           member 0    2583.4 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71fb1a8c8ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71fb1aa0f130>, 17191.003043315), (<aiohttp.client_proto.ResponseHandler object at 0x71fb18efdf90>, 17191.261325324), (<aiohttp.client_proto.ResponseHandler object at 0x71fb18efe0d0>, 17191.321302165), (<aiohttp.client_proto.ResponseHandler object at 0x71fb1a8caa50>, 17191.365724587), (<aiohttp.client_proto.ResponseHandler object at 0x71fb1aa0ea00>, 17191.598103832), (<aiohttp.client_proto.ResponseHandler object at 0x71fb1aab90d0>, 17191.645736491), (<aiohttp.client_proto.ResponseHandler object at 0x71fb18efe210>, 17191.683693759), (<aiohttp.client_proto.ResponseHandler object at 0x71fb1a8ca7b0>, 17192.194886766)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71fb1a8a2850>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71fb1a8c86e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71fb1aa03a10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71fb1a8c8d70>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71fb18f18050>, 17189.503024976)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71fb1aa03cb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71fb1ab0b050>, 17189.381827664)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71fb1a8a2d50>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71fb1ab0aa50>, 17189.301649353)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71fb18efc7d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b476bd10ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b476bf21d30>, 17190.834544576), (<aiohttp.client_proto.ResponseHandler object at 0x7b47780f6a00>, 17190.881067925), (<aiohttp.client_proto.ResponseHandler object at 0x7b476bd123c0>, 17190.883025983), (<aiohttp.client_proto.ResponseHandler object at 0x7b476bd12660>, 17190.967767141), (<aiohttp.client_proto.ResponseHandler object at 0x7b477820e5d0>, 17190.969552806), (<aiohttp.client_proto.ResponseHandler object at 0x7b47780f6fc0>, 17191.438746983)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b476bede850>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b476bd10ad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b476bd10d70>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b477820ebd0>, 17189.302178921)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b4778107b60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b476bf07700>, 17189.381754767)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b476bedefd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7998e152d160>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7998e19e6fc0>, 17191.477183905), (<aiohttp.client_proto.ResponseHandler object at 0x7998e155df90>, 17191.54304688), (<aiohttp.client_proto.ResponseHandler object at 0x7998e1889a70>, 17191.597734886), (<aiohttp.client_proto.ResponseHandler object at 0x7998e155de50>, 17191.839321163), (<aiohttp.client_proto.ResponseHandler object at 0x7998e152ea50>, 17191.886554176), (<aiohttp.client_proto.ResponseHandler object at 0x7998e155e0d0>, 17192.113838667), (<aiohttp.client_proto.ResponseHandler object at 0x7998e19e6720>, 17192.250555591), (<aiohttp.client_proto.ResponseHandler object at 0x7998e152ecf0>, 17192.29009859)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7998e14ef250>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7998e152cc20>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7998e152cec0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7998e152d010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7998e1ade750>, 17189.490654368)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7998e19b7cb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7998e1568500>, 17189.429905855)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7998e14eead0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7998e1adf050>, 17189.365463431)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7998e155c690>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x72c41c2ec980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x72c41c4aa2b0>, 17190.266246865), (<aiohttp.client_proto.ResponseHandler object at 0x72c41ceba8d0>, 17191.05841158), (<aiohttp.client_proto.ResponseHandler object at 0x72c41c72a5b0>, 17191.113586942)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x72c41c723a10>

try 1  S3           member 0    2583.4 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d3c7dbecad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d3c7dd16e50>, 17200.10549192), (<aiohttp.client_proto.ResponseHandler object at 0x7d3c7c21a490>, 17200.17112119), (<aiohttp.client_proto.ResponseHandler object at 0x7d3c7dbeecf0>, 17200.250645982), (<aiohttp.client_proto.ResponseHandler object at 0x7d3c7c21a5d0>, 17200.491457573), (<aiohttp.client_proto.ResponseHandler object at 0x7d3c7dd89d30>, 17200.542711036), (<aiohttp.client_proto.ResponseHandler object at 0x7d3c7c21a710>, 17200.543354425), (<aiohttp.client_proto.ResponseHandler object at 0x7d3c7dd16b70>, 17200.827474955), (<aiohttp.client_proto.ResponseHandler object at 0x7d3c7dbeef90>, 17200.870718587)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d3c7dbae0d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d3c7dbec590>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d3c7dbec830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d3c7dbec980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d3c7de2a5d0>, 17198.382568363)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d3c7dd8f620>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d3c7c2109b0>, 17198.258639229)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d3c7dbae850>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d3c7de2aed0>, 17198.442291627)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d3c7dbafd90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x710238c21400>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x710238c51bd0>, 17201.060512319), (<aiohttp.client_proto.ResponseHandler object at 0x7102390f57b0>, 17201.10130541), (<aiohttp.client_proto.ResponseHandler object at 0x71023902ace0>, 17201.172273557), (<aiohttp.client_proto.ResponseHandler object at 0x710238c22a50>, 17201.174616009), (<aiohttp.client_proto.ResponseHandler object at 0x710238c227b0>, 17201.448972354), (<aiohttp.client_proto.ResponseHandler object at 0x710238c51d10>, 17201.449335913), (<aiohttp.client_proto.ResponseHandler object at 0x71023902b410>, 17201.757066178), (<aiohttp.client_proto.ResponseHandler object at 0x710238c51a90>, 17202.13389075)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x710238c50050>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x710238c206e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x710238c20d70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x710238c20ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x710238c4c370>, 17198.963040626)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71023903fcb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71023915ea50>, 17199.097889359)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x710238deee90>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71023915e450>, 17199.036872438)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x710238defb10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7100753cfa10>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7100753deb70>, 17201.009639431), (<aiohttp.client_proto.ResponseHandler object at 0x710074fb8c20>, 17201.103151053), (<aiohttp.client_proto.ResponseHandler object at 0x71007516ee90>, 17201.348443182), (<aiohttp.client_proto.ResponseHandler object at 0x7100753dee50>, 17201.405879949), (<aiohttp.client_proto.ResponseHandler object at 0x710074fea210>, 17201.453522344), (<aiohttp.client_proto.ResponseHandler object at 0x710074fbaba0>, 17201.498900378), (<aiohttp.client_proto.ResponseHandler object at 0x710075289bd0>, 17201.71878099), (<aiohttp.client_proto.ResponseHandler object at 0x710074fe8550>, 17202.142099663)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71007516ec10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x710074fb86e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x710074fb8d70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x710074fb8ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x710074ff4500>, 17199.201830583)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7100753cfcb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7100754eed50>, 17198.991285621)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71007516f390>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7100754eea50>, 17199.138486871)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x710074fe87d0>

try 1  S3           member 0    2583.4 MB  call 3/3  [ok]

2026-09-21 19:52:41,838 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/comm/tcp.py", line 226, in read
    frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/worker.py", line 1165, in heartbeat
    response = await retry_operation(
               ^^^^^^^^^^^^^^^^^^^^^^
    ...<14 lines>...
    )
    ^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/utils_comm.py", line 420, in retry_operation
    return await retry(
           ^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/utils_comm.py", line 399, in retry
    return await coro()
           ^^^^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/core.py", line 1212, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/core.py", line 982, in send_recv
    response = await comm.read(deserializers=deserializers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/comm/tcp.py", line 237, in read
    convert_stream_closed_error(self, e)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:53134 remote=tcp://127.0.0.1:46127>: Stream is closed
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x744d2f2fcc20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x744d2f542440>, 17209.538490788), (<aiohttp.client_proto.ResponseHandler object at 0x744d2f775fd0>, 17209.661668224)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x744d2f633cb0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a4508abcec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a4508abee40>, 17208.214464985), (<aiohttp.client_proto.ResponseHandler object at 0x7a4508ee2fc0>, 17208.26234317), (<aiohttp.client_proto.ResponseHandler object at 0x7a4508abeba0>, 17208.300171242), (<aiohttp.client_proto.ResponseHandler object at 0x7a4508ee2890>, 17208.532348399), (<aiohttp.client_proto.ResponseHandler object at 0x7a4508d91a70>, 17208.932814958), (<aiohttp.client_proto.ResponseHandler object at 0x7a4508af5bd0>, 17209.161678644), (<aiohttp.client_proto.ResponseHandler object at 0x7a4508af5d10>, 17209.773357229), (<aiohttp.client_proto.ResponseHandler object at 0x7a4508a8efd0>, 17209.944116744)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a4508a8ee90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a4508abc980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a4508abcc20>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a4508abcd70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b5531e5bb60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b5531e73130>, 17210.279059519), (<aiohttp.client_proto.ResponseHandler object at 0x7b5531cd5d30>, 17210.36132335), (<aiohttp.client_proto.ResponseHandler object at 0x7b5531a7a0d0>, 17210.718423995), (<aiohttp.client_proto.ResponseHandler object at 0x7b5531a7a210>, 17210.762480783), (<aiohttp.client_proto.ResponseHandler object at 0x7b55325d2750>, 17210.797684175), (<aiohttp.client_proto.ResponseHandler object at 0x7b5531a79f90>, 17211.76025466), (<aiohttp.client_proto.ResponseHandler object at 0x7b5531a46ba0>, 17211.991591344), (<aiohttp.client_proto.ResponseHandler object at 0x7b5531a46900>, 17212.100206609)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b5531a78f50>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b5531a44830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b5531a44ec0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b5531a45010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b5531e72e50>, 17207.999987014)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b5531a44050>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b5531a07d40>, 17208.185105498)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b5531a1ee90>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b55325d2d50>, 17208.124743467)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b5531a787d0>

try 1  OSDF         member 0     161.5 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x739e89ae9010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x739e89aeacf0>, 17209.774525296), (<aiohttp.client_proto.ResponseHandler object at 0x739e89f93580>, 17209.962916686), (<aiohttp.client_proto.ResponseHandler object at 0x739e89aeaf90>, 17210.118348607), (<aiohttp.client_proto.ResponseHandler object at 0x739e89e3c890>, 17210.345257216), (<aiohttp.client_proto.ResponseHandler object at 0x739e89f92a00>, 17210.593530184), (<aiohttp.client_proto.ResponseHandler object at 0x739e89b1a5d0>, 17210.59407118), (<aiohttp.client_proto.ResponseHandler object at 0x739e89b1a710>, 17210.677642784), (<aiohttp.client_proto.ResponseHandler object at 0x739e89b1a490>, 17210.897121959)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x739e89a7fb10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x739e89ae8ad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x739e89ae8d70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x739e89ae8ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x739e8a6e6d50>, 17208.230729523)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x739e89f7bb60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x739e89b9c1e0>, 17208.021718885)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x739e89b187d0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x739e8a6e7050>, 17208.14948024)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x739e89a7f390>

try 1  OSDF         member 0     161.5 MB  call 2/3  [ok]
try 1  OSDF         member 0     161.5 MB  call 3/3  [ok]
try 1  OSDF         member 0     322.9 MB  call 1/3  [ok]
try 1  OSDF         member 0     322.9 MB  call 2/3  [ok]
try 1  OSDF         member 0     322.9 MB  call 3/3  [ok]
try 1  OSDF         member 0     645.9 MB  call 1/3  [ok]
try 1  OSDF         member 0     645.9 MB  call 2/3  [ok]
try 1  OSDF         member 0     645.9 MB  call 3/3  [ok]
try 1  OSDF         member 0    1291.7 MB  call 1/3  [ok]
try 1  OSDF         member 0    1291.7 MB  call 2/3  [ok]
try 1  OSDF         member 0    1291.7 MB  call 3/3  [ok]
try 1  OSDF         member 0    2583.4 MB  call 1/3  [ok]
try 1  OSDF         member 0    2583.4 MB  call 2/3  [ok]
try 1  OSDF         member 0    2583.4 MB  call 3/3  [ok]
try 1  OSDF-direct  member 0     161.5 MB  call 1/3  [ok]
try 1  OSDF-direct  member 0     161.5 MB  call 2/3  [ok]
try 1  OSDF-direct  member 0     161.5 MB  call 3/3  [ok]
try 1  OSDF-direct  member 0     322.9 MB  call 1/3  [ok]
try 1  OSDF-direct  member 0     322.9 MB  call 2/3  [ok]

2026-09-21 19:55:24,236 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/comm/tcp.py", line 226, in read
    frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/worker.py", line 1165, in heartbeat
    response = await retry_operation(
               ^^^^^^^^^^^^^^^^^^^^^^
    ...<14 lines>...
    )
    ^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/utils_comm.py", line 420, in retry_operation
    return await retry(
           ^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/utils_comm.py", line 399, in retry
    return await coro()
           ^^^^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/core.py", line 1212, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/core.py", line 982, in send_recv
    response = await comm.read(deserializers=deserializers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/comm/tcp.py", line 237, in read
    convert_stream_closed_error(self, e)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:37644 remote=tcp://127.0.0.1:46127>: Stream is closed

try 1  OSDF-direct  member 0     322.9 MB  call 3/3  [ok]
try 1  OSDF-direct  member 0     645.9 MB  call 1/3  [ok]
try 1  OSDF-direct  member 0     645.9 MB  call 2/3  [ok]
try 1  OSDF-direct  member 0     645.9 MB  call 3/3  [ok]
try 1  OSDF-direct  member 0    1291.7 MB  call 1/3  [ok]
try 1  OSDF-direct  member 0    1291.7 MB  call 2/3  [ok]
try 1  OSDF-direct  member 0    1291.7 MB  call 3/3  [ok]
try 1  OSDF-direct  member 0    2583.4 MB  call 1/3  [ok]
try 1  OSDF-direct  member 0    2583.4 MB  call 2/3  [ok]
try 1  OSDF-direct  member 0    2583.4 MB  call 3/3  [ok]
try 2  S3           member 1     161.5 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x74edfde41fd0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x74edfde0ec10>, 17480.273857102), (<aiohttp.client_proto.ResponseHandler object at 0x74edfde0f890>, 17480.380856569)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x74edfde41be0>

try 2  S3           member 1     161.5 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x757581804ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x757581a2a5d0>, 17482.663833573), (<aiohttp.client_proto.ResponseHandler object at 0x757581b15e50>, 17482.81837539)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x757581b57b60>

try 2  S3           member 1     161.5 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d755fd1cad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d755ff4a5d0>, 17485.113722042), (<aiohttp.client_proto.ResponseHandler object at 0x7d7564795fd0>, 17485.337821448)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d756413fb60>

try 2  S3           member 1     322.9 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9af72f4980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9af76be750>, 17487.928089299), (<aiohttp.client_proto.ResponseHandler object at 0x7c9af7512e50>, 17487.97647711), (<aiohttp.client_proto.ResponseHandler object at 0x7c9af75873e0>, 17488.03091207), (<aiohttp.client_proto.ResponseHandler object at 0x7c9af7513410>, 17488.261786698)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9af72d5bd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9af72f46e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9af76bdcd0>, 17487.095413357)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9af7567770>

try 2  S3           member 1     322.9 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x70d1b16e8830>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x70d1b190e2d0>, 17490.963678677), (<aiohttp.client_proto.ResponseHandler object at 0x70d1b1ab6750>, 17491.048305959), (<aiohttp.client_proto.ResponseHandler object at 0x70d1b197f3e0>, 17491.505667343), (<aiohttp.client_proto.ResponseHandler object at 0x70d1b190f580>, 17491.58196313)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x70d1b16c9f90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x70d1b16e8590>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x70d1b1ab5fd0>, 17490.1504884)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x70d1b1967620>

try 2  S3           member 1     322.9 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d5f40960980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d5f6010de50>, 17494.178892662), (<aiohttp.client_proto.ResponseHandler object at 0x7d5f60162e50>, 17494.307427513), (<aiohttp.client_proto.ResponseHandler object at 0x7d5f40b173e0>, 17494.415405126), (<aiohttp.client_proto.ResponseHandler object at 0x7d5f60163410>, 17494.65531379)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d5f40941bd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d5f409606e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d5f6010dcd0>, 17493.516240591)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d5f601b3770>

try 2  S3           member 1     645.9 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d23fef18980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d23feee22b0>, 17498.750725731), (<aiohttp.client_proto.ResponseHandler object at 0x7d2404115fd0>, 17499.167564201)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d23ff14ba10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7dc1abb08ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7dc1abcd62b0>, 17498.57368616), (<aiohttp.client_proto.ResponseHandler object at 0x7dc1b068dfd0>, 17498.909435117)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7dc1abe57b60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7ec5cbef06e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7ec5cbb0a760>, 17497.920433545), (<aiohttp.client_proto.ResponseHandler object at 0x7ec5d0815e50>, 17498.114223597)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7ec5cbf97770>

try 2  S3           member 1     645.9 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7e1866f44980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7e186729a5b0>, 17502.171411998), (<aiohttp.client_proto.ResponseHandler object at 0x7e1866f062b0>, 17502.306786937), (<aiohttp.client_proto.ResponseHandler object at 0x7e186c309fd0>, 17502.358853471), (<aiohttp.client_proto.ResponseHandler object at 0x7e1866f44830>, 17502.746655497), (<aiohttp.client_proto.ResponseHandler object at 0x7e1867161ff0>, 17502.747542917), (<aiohttp.client_proto.ResponseHandler object at 0x7e186ce58050>, 17502.933969101), (<aiohttp.client_proto.ResponseHandler object at 0x7e186729b2a0>, 17502.976311722), (<aiohttp.client_proto.ResponseHandler object at 0x7e186c30a8d0>, 17503.441453542)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7e1867293a10>

try 2  S3           member 1     645.9 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7864e3d1cc20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7864e3d496d0>, 17507.223974709), (<aiohttp.client_proto.ResponseHandler object at 0x7864e3d1e510>, 17507.428686847), (<aiohttp.client_proto.ResponseHandler object at 0x7864e3d49590>, 17507.506136465), (<aiohttp.client_proto.ResponseHandler object at 0x7864e3d49450>, 17507.509453931), (<aiohttp.client_proto.ResponseHandler object at 0x7864e3f11bd0>, 17507.786340865), (<aiohttp.client_proto.ResponseHandler object at 0x7864f0126fc0>, 17507.854495813), (<aiohttp.client_proto.ResponseHandler object at 0x7864f0126b70>, 17507.897297578), (<aiohttp.client_proto.ResponseHandler object at 0x7864e3d1e7b0>, 17508.652239488)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7864e3d48410>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7864e3d1c6e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7864e3d1c980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7864e3d1cad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7864e3d44820>, 17505.868420933)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7864f0173770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7864f023ebd0>, 17505.808474125)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7864e3eee850>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7864f023e8d0>, 17505.684641607)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7864e3eefb10>

try 2  S3           member 1    1291.7 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x798b7c1c4ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x798b7c1922b0>, 17513.445349261), (<aiohttp.client_proto.ResponseHandler object at 0x798b7cd45fd0>, 17513.997760811)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x798b7c5e7b60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a72496acc20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a72496d81e0>, 17512.218185353), (<aiohttp.client_proto.ResponseHandler object at 0x7a72496d1310>, 17512.713086386), (<aiohttp.client_proto.ResponseHandler object at 0x7a72496ae3c0>, 17512.779533438), (<aiohttp.client_proto.ResponseHandler object at 0x7a72496d1450>, 17513.01282417), (<aiohttp.client_proto.ResponseHandler object at 0x7a72496ae660>, 17513.18041346), (<aiohttp.client_proto.ResponseHandler object at 0x7a72499ef410>, 17513.408155661), (<aiohttp.client_proto.ResponseHandler object at 0x7a72496d0cd0>, 17513.845365458), (<aiohttp.client_proto.ResponseHandler object at 0x7a7249a8de90>, 17513.880987147)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a72496d0550>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a72496ac6e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a72496ac980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7a72496acad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a724999e5d0>, 17511.07998617)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a7249a3b770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a72499eee50>, 17511.019867014)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a7249877610>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7a724999e2d0>, 17510.959903839)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7a7249877c50>

try 2  S3           member 1    1291.7 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7861f9193b60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7861f9061d30>, 17518.257242702), (<aiohttp.client_proto.ResponseHandler object at 0x7861f90a65d0>, 17518.394873494), (<aiohttp.client_proto.ResponseHandler object at 0x7861f8fb36f0>, 17518.635236896), (<aiohttp.client_proto.ResponseHandler object at 0x7861f8fb2e50>, 17518.709634299), (<aiohttp.client_proto.ResponseHandler object at 0x7861fbc11450>, 17518.763647065), (<aiohttp.client_proto.ResponseHandler object at 0x7861f8d7a510>, 17519.090994645), (<aiohttp.client_proto.ResponseHandler object at 0x7861f8d7a270>, 17519.925527777), (<aiohttp.client_proto.ResponseHandler object at 0x7861fbb9bed0>, 17520.369922432)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7861f8d4fd90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7861f8d78830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7861f8d78ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7861f90a6bd0>, 17511.77963937)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7861f8d78050>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7861f9027bb0>, 17511.839909404)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7861f8d4e210>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d2797041010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d27973b1d30>, 17519.373368174), (<aiohttp.client_proto.ResponseHandler object at 0x7d27974f6e50>, 17519.461092485), (<aiohttp.client_proto.ResponseHandler object at 0x7d279740aa50>, 17519.529046752), (<aiohttp.client_proto.ResponseHandler object at 0x7d2797042510>, 17519.75814875), (<aiohttp.client_proto.ResponseHandler object at 0x7d27974f6890>, 17519.866336054), (<aiohttp.client_proto.ResponseHandler object at 0x7d27970427b0>, 17520.517190291)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d279702fc50>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d2797040c20>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7d2797040ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d279740a450>, 17518.005033089)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d27974cfcb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7d2797397d40>, 17518.13046573)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7d279702e850>

try 2  S3           member 1    1291.7 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7abd4eaf1010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7abd4ede1d30>, 17525.847987326), (<aiohttp.client_proto.ResponseHandler object at 0x7abd4f03a450>, 17525.903940957), (<aiohttp.client_proto.ResponseHandler object at 0x7abd4ef2a890>, 17526.212116338), (<aiohttp.client_proto.ResponseHandler object at 0x7abd4eaf2510>, 17526.261074676), (<aiohttp.client_proto.ResponseHandler object at 0x7abd4ef2ae50>, 17526.392841288), (<aiohttp.client_proto.ResponseHandler object at 0x7abd4eaf27b0>, 17526.654941874)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7abd4ecc6210>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7abd4eaf0c20>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7abd4eaf0ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7abd4ef9fd40>, 17524.439653113)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7abd4ef03cb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7abd4f03a750>, 17524.504265158)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7abd4ecc74d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7dc3ab948ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7dc3abd21fd0>, 17526.478184384), (<aiohttp.client_proto.ResponseHandler object at 0x7dc3ab9162b0>, 17526.557877324)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7dc3abdf7b60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x785d05e40c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x785d05e427b0>, 17525.390523519), (<aiohttp.client_proto.ResponseHandler object at 0x785d05e6d950>, 17525.532323404), (<aiohttp.client_proto.ResponseHandler object at 0x785d05e689b0>, 17525.590981766), (<aiohttp.client_proto.ResponseHandler object at 0x785d05e42510>, 17525.647819901), (<aiohttp.client_proto.ResponseHandler object at 0x785d0617ea00>, 17525.887643439), (<aiohttp.client_proto.ResponseHandler object at 0x785d0602da70>, 17525.941348905), (<aiohttp.client_proto.ResponseHandler object at 0x785d05e6d590>, 17526.265267368), (<aiohttp.client_proto.ResponseHandler object at 0x785d05e6d6d0>, 17526.596118904)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x785d05e6c410>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x785d05e406e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x785d05e40980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x785d05e40ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x785d067c62d0>, 17523.813581652)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x785d05fdb770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x785d0617eb70>, 17523.754174359)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x785d05e134d0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x785d067c65d0>, 17523.630623688)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x785d05e13b10>

try 2  S3           member 1    2583.4 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77fd0c265550>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x77fd0c266a50>, 17532.699049647), (<aiohttp.client_proto.ResponseHandler object at 0x77fd0c493580>, 17532.854603103), (<aiohttp.client_proto.ResponseHandler object at 0x77fd0c266cf0>, 17532.925072472), (<aiohttp.client_proto.ResponseHandler object at 0x77fd0c299f90>, 17532.975676692), (<aiohttp.client_proto.ResponseHandler object at 0x77fd0c299e50>, 17533.028906986), (<aiohttp.client_proto.ResponseHandler object at 0x77fd0c53dbd0>, 17533.277147608), (<aiohttp.client_proto.ResponseHandler object at 0x77fd0c492ce0>, 17533.316872502), (<aiohttp.client_proto.ResponseHandler object at 0x77fd0c29a0d0>, 17533.844479961)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x77fd0c298cd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77fd0c264980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77fd0c265400>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77fd0c667cb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x77fd0c54c500>, 17530.83204882)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x77fd0c264050>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x77fd0c5aa750>, 17530.896251405)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x77fd0d75f4d0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x77fd0c5aad50>, 17530.698120305)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x77fd0c2982d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fc8e16bcc20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7fc8e0ced450>, 17531.750744092), (<aiohttp.client_proto.ResponseHandler object at 0x7fc8e0ced590>, 17531.790596469), (<aiohttp.client_proto.ResponseHandler object at 0x7fc8e17fb580>, 17531.845476807), (<aiohttp.client_proto.ResponseHandler object at 0x7fc8e16be510>, 17531.884051165), (<aiohttp.client_proto.ResponseHandler object at 0x7fc8e0ced6d0>, 17532.133088138), (<aiohttp.client_proto.ResponseHandler object at 0x7fc8e16be7b0>, 17532.364688275), (<aiohttp.client_proto.ResponseHandler object at 0x7fc8e1899a70>, 17532.809788244), (<aiohttp.client_proto.ResponseHandler object at 0x7fc8e17fae50>, 17533.124948578)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7fc8e0cec410>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fc8e16bc6e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fc8e16bc980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fc8e16bcad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7fc8e0ce49b0>, 17530.016996149)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7fc8e1843770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7fc8e199ebd0>, 17529.957355754)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7fc8e16a2850>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7fc8e199e8d0>, 17529.832410535)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7fc8e16a3b10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73577abfd010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73577b036a00>, 17532.085840411), (<aiohttp.client_proto.ResponseHandler object at 0x73577abfe3c0>, 17532.313632385), (<aiohttp.client_proto.ResponseHandler object at 0x73577abfe660>, 17532.411496591), (<aiohttp.client_proto.ResponseHandler object at 0x73577aeedd30>, 17532.744299023), (<aiohttp.client_proto.ResponseHandler object at 0x73577b036fc0>, 17532.943469663), (<aiohttp.client_proto.ResponseHandler object at 0x73577adca0d0>, 17533.565031266), (<aiohttp.client_proto.ResponseHandler object at 0x73577adc9810>, 17533.614931708), (<aiohttp.client_proto.ResponseHandler object at 0x73577b1265d0>, 17533.870533456)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73577adcaad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73577abfc830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73577b01bb60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73577b126bd0>, 17530.690500443)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73577abfc050>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73577aef3700>, 17530.755090874)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73577adcb610>

try 2  S3           member 1    2583.4 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x792764f81940>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x79276524d7b0>, 17541.879452383), (<aiohttp.client_proto.ResponseHandler object at 0x792764f82900>, 17541.951047855), (<aiohttp.client_proto.ResponseHandler object at 0x7927651aa890>, 17541.991317429), (<aiohttp.client_proto.ResponseHandler object at 0x7927651aae50>, 17541.996414617), (<aiohttp.client_proto.ResponseHandler object at 0x792764f82ba0>, 17542.267971005), (<aiohttp.client_proto.ResponseHandler object at 0x792764fae210>, 17542.506219277), (<aiohttp.client_proto.ResponseHandler object at 0x792764fae0d0>, 17542.630705341), (<aiohttp.client_proto.ResponseHandler object at 0x792764fadf90>, 17542.945076453)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x792764fac550>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x792764f80830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x792765383b60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x792764f80ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x79276529ed50>, 17539.835844621)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x792764f80050>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7927645c0050>, 17539.900446392)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x792764f3f390>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x79276529e750>, 17539.961303291)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x792764fac690>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x72144a1d9010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x721448809f90>, 17541.966210057), (<aiohttp.client_proto.ResponseHandler object at 0x72144a316b70>, 17542.043894254), (<aiohttp.client_proto.ResponseHandler object at 0x72144880a0d0>, 17542.090795557), (<aiohttp.client_proto.ResponseHandler object at 0x72144a316e50>, 17542.142168822), (<aiohttp.client_proto.ResponseHandler object at 0x72144a3c9d30>, 17542.188313088), (<aiohttp.client_proto.ResponseHandler object at 0x72144a1daba0>, 17542.431137249), (<aiohttp.client_proto.ResponseHandler object at 0x721448809e50>, 17542.467489688), (<aiohttp.client_proto.ResponseHandler object at 0x72144a1da900>, 17542.982430181)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x72144a1ae710>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x72144a1d8ad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x72144a1d8d70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x72144a1d8ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x72144a417050>, 17539.926298497)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x72144a30fb60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7214488149b0>, 17539.865207508)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x72144a1af750>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x72144a416d50>, 17539.797323037)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x72144a1aee90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77a3a1681400>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x77a3a1682900>, 17539.885679789), (<aiohttp.client_proto.ResponseHandler object at 0x77a3a16ad310>, 17540.135248386), (<aiohttp.client_proto.ResponseHandler object at 0x77a3a1975e90>, 17540.370648289), (<aiohttp.client_proto.ResponseHandler object at 0x77a3a808b6f0>, 17540.547360926), (<aiohttp.client_proto.ResponseHandler object at 0x77a3a16ade50>, 17540.777883722), (<aiohttp.client_proto.ResponseHandler object at 0x77a3a16add10>, 17541.356588789), (<aiohttp.client_proto.ResponseHandler object at 0x77a3a1682660>, 17541.697391305), (<aiohttp.client_proto.ResponseHandler object at 0x77a3a81cabd0>, 17541.905476318)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x77a3a16ace10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77a3a16806e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77a3a1681160>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x77a3a16812b0>

try 2  S3           member 1    2583.4 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7521a27412b0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7521a2771f90>, 17550.862288371), (<aiohttp.client_proto.ResponseHandler object at 0x7521a27720d0>, 17551.019837733), (<aiohttp.client_proto.ResponseHandler object at 0x7521a2742900>, 17551.075117372), (<aiohttp.client_proto.ResponseHandler object at 0x7521a2a94e10>, 17551.080152462), (<aiohttp.client_proto.ResponseHandler object at 0x7521b03eb050>, 17551.188295576), (<aiohttp.client_proto.ResponseHandler object at 0x7521a2742ba0>, 17551.502359455), (<aiohttp.client_proto.ResponseHandler object at 0x7521a2771e50>, 17551.731247182), (<aiohttp.client_proto.ResponseHandler object at 0x7521a2beace0>, 17552.142899787)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7521a2770e10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7521a2740830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7521a2bcfb60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7521a2740ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7521a2beae50>, 17549.010980472)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7521a2740050>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7face6db5010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7521b03ea750>, 17548.886480716)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7521a2714cd0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7521a27749b0>, 17548.821329596)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7521a27174d0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7face6fe3410>, 17550.905557052), (<aiohttp.client_proto.ResponseHandler object at 0x7face7089bd0>, 17551.026915157), (<aiohttp.client_proto.ResponseHandler object at 0x7face6fe2ce0>, 17551.080509126), (<aiohttp.client_proto.ResponseHandler object at 0x7face6de2210>, 17551.315330252), (<aiohttp.client_proto.ResponseHandler object at 0x7face6de20d0>, 17551.357863382), (<aiohttp.client_proto.ResponseHandler object at 0x7face6db6900>, 17551.358058293), (<aiohttp.client_proto.ResponseHandler object at 0x7face6db6ba0>, 17551.437884623), (<aiohttp.client_proto.ResponseHandler object at 0x7face6de1f90>, 17551.933996846)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7face6d53610>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7face6db4830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7face71c3b60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7face6db4ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7face4bf81e0>, 17548.83168847)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7face6db4050>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7face70daa50>, 17548.897992219)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7face6d52e90>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7face70db050>, 17548.959825725)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7face6de07d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75f1990fcec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x75f19b70f0e0>, 17548.918615483), (<aiohttp.client_proto.ResponseHandler object at 0x75f1993d5650>, 17549.139363294), (<aiohttp.client_proto.ResponseHandler object at 0x75f19944a8d0>, 17549.181288774), (<aiohttp.client_proto.ResponseHandler object at 0x75f19cdb8190>, 17549.425885314), (<aiohttp.client_proto.ResponseHandler object at 0x75f19b70f230>, 17549.540332537), (<aiohttp.client_proto.ResponseHandler object at 0x75f1994170c0>, 17550.753464135), (<aiohttp.client_proto.ResponseHandler object at 0x75f19933ae50>, 17551.463411191), (<aiohttp.client_proto.ResponseHandler object at 0x75f19933b410>, 17551.517025117)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x75f1990f1950>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75f1990fcc20>

try 2  OSDF         member 1     161.5 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x74f28e543cb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x74f28e56f410>, 17549.903240258), (<aiohttp.client_proto.ResponseHandler object at 0x74f28e56ee50>, 17549.982415634), (<aiohttp.client_proto.ResponseHandler object at 0x74f2a41768d0>, 17550.545308541), (<aiohttp.client_proto.ResponseHandler object at 0x74f2a4175e50>, 17551.038623484), (<aiohttp.client_proto.ResponseHandler object at 0x74f28e115d30>, 17551.383605077)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x74f28e30df90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x74f28e318980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x74f28e14f0c0>, 17548.897783022)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x74f28e318050>

try 2  OSDF         member 1     161.5 MB  call 2/3  [ok]
try 2  OSDF         member 1     161.5 MB  call 3/3  [ok]
try 2  OSDF         member 1     322.9 MB  call 1/3  [ok]
try 2  OSDF         member 1     322.9 MB  call 2/3  [ok]
try 2  OSDF         member 1     322.9 MB  call 3/3  [ok]
try 2  OSDF         member 1     645.9 MB  call 1/3  [ok]
try 2  OSDF         member 1     645.9 MB  call 2/3  [ok]
try 2  OSDF         member 1     645.9 MB  call 3/3  [ok]
try 2  OSDF         member 1    1291.7 MB  call 1/3  [ok]
try 2  OSDF         member 1    1291.7 MB  call 2/3  [ok]
try 2  OSDF         member 1    1291.7 MB  call 3/3  [ok]
try 2  OSDF         member 1    2583.4 MB  call 1/3  [ok]
try 2  OSDF         member 1    2583.4 MB  call 2/3  [ok]
try 2  OSDF         member 1    2583.4 MB  call 3/3  [ok]
try 2  OSDF-direct  member 1     161.5 MB  call 1/3  [ok]
try 2  OSDF-direct  member 1     161.5 MB  call 2/3  [ok]
try 2  OSDF-direct  member 1     161.5 MB  call 3/3  [ok]
try 2  OSDF-direct  member 1     322.9 MB  call 1/3  [ok]
try 2  OSDF-direct  member 1     322.9 MB  call 2/3  [ok]
try 2  OSDF-direct  member 1     322.9 MB  call 3/3  [ok]
try 2  OSDF-direct  member 1     645.9 MB  call 1/3  [ok]
try 2  OSDF-direct  member 1     645.9 MB  call 2/3  [ok]
try 2  OSDF-direct  member 1     645.9 MB  call 3/3  [ok]
try 2  OSDF-direct  member 1    1291.7 MB  call 1/3  [ok]
try 2  OSDF-direct  member 1    1291.7 MB  call 2/3  [ok]
try 2  OSDF-direct  member 1    1291.7 MB  call 3/3  [ok]
try 2  OSDF-direct  member 1    2583.4 MB  call 1/3  [ok]
try 2  OSDF-direct  member 1    2583.4 MB  call 2/3  [ok]
try 2  OSDF-direct  member 1    2583.4 MB  call 3/3  [ok]
try 3  S3           member 2     161.5 MB  call 1/3  [ok]
try 3  S3           member 2     161.5 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x75574546c980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x755745eedfd0>, 17901.922709144), (<aiohttp.client_proto.ResponseHandler object at 0x7557457428f0>, 17902.021994054)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7557458a3a10>

try 3  S3           member 2     161.5 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x733a69874ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x733a69a965d0>, 17904.315814443), (<aiohttp.client_proto.ResponseHandler object at 0x733a6a1b5e50>, 17904.462113134)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x733a6a1f3b60>

try 3  S3           member 2     322.9 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x751920a28980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x751920bff3e0>, 17907.163308775), (<aiohttp.client_proto.ResponseHandler object at 0x751920d16750>, 17907.265639063), (<aiohttp.client_proto.ResponseHandler object at 0x751920d6b580>, 17907.343307432), (<aiohttp.client_proto.ResponseHandler object at 0x751920d6aa00>, 17907.569548298)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x751920a0dbd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x751920a286e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x751920d15cd0>, 17906.151551977)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x751920db7770>

try 3  S3           member 2     322.9 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x79ec7af30980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x79ec7b215fd0>, 17910.39891781), (<aiohttp.client_proto.ResponseHandler object at 0x79ec7b216750>, 17910.519823604), (<aiohttp.client_proto.ResponseHandler object at 0x79ec7b2722d0>, 17910.555132289), (<aiohttp.client_proto.ResponseHandler object at 0x79ec7b273580>, 17910.807028703)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x79ec7af120d0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x79ec7af306e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x79ec7b10b3e0>, 17909.452316029)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x79ec7b2bf770>

try 3  S3           member 2     322.9 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7e4cf6608980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7e4cf6956890>, 17913.679360745), (<aiohttp.client_proto.ResponseHandler object at 0x7e4cf69576f0>, 17913.729455677), (<aiohttp.client_proto.ResponseHandler object at 0x7e4cf69cb3e0>, 17913.767282268), (<aiohttp.client_proto.ResponseHandler object at 0x7e4cf6902750>, 17913.984930511)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7e4cf65edbd0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7e4cf66086e0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7e4cf6901cd0>, 17912.681564382)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7e4cf69a3770>

try 3  S3           member 2     645.9 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7890bdec8c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7890be286bd0>, 17917.570702555), (<aiohttp.client_proto.ResponseHandler object at 0x7890bdef5590>, 17917.743384896), (<aiohttp.client_proto.ResponseHandler object at 0x7890bdeca510>, 17917.785156852), (<aiohttp.client_proto.ResponseHandler object at 0x7890bdef5450>, 17918.004270969), (<aiohttp.client_proto.ResponseHandler object at 0x7890be18d7b0>, 17918.263470865), (<aiohttp.client_proto.ResponseHandler object at 0x7890bdef56d0>, 17918.305821176), (<aiohttp.client_proto.ResponseHandler object at 0x7890bdeca7b0>, 17918.570730284), (<aiohttp.client_proto.ResponseHandler object at 0x7890be2df410>, 17918.960870405)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7890bdef4410>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7890bdec86e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7890bdec8980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7890bdec8ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7890bdeec820>, 17915.867521268)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7890be137770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7890be2868d0>, 17915.99035832)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7890bde9a850>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7890be2deb70>, 17916.049881536)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7890bde9bb10>

try 3  S3           member 2     645.9 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cf979990ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cf979cd65d0>, 17922.592243721), (<aiohttp.client_proto.ResponseHandler object at 0x7cf9799c16d0>, 17922.834942704), (<aiohttp.client_proto.ResponseHandler object at 0x7cf9799c1810>, 17922.911508031), (<aiohttp.client_proto.ResponseHandler object at 0x7cf979c15e90>, 17923.04450349), (<aiohttp.client_proto.ResponseHandler object at 0x7cf9799c1950>, 17923.095166067), (<aiohttp.client_proto.ResponseHandler object at 0x7cf979d7f410>, 17923.375574417), (<aiohttp.client_proto.ResponseHandler object at 0x7cf979992510>, 17923.991595464), (<aiohttp.client_proto.ResponseHandler object at 0x7cf979992270>, 17924.252170957)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cf9799c0910>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cf979990590>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cf979990830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7cf979990980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cf9799bc820>, 17921.465642457)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cf979c1b620>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cf979d7ee50>, 17921.406141716)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cf97995ed50>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7cf979cd68d0>, 17921.346322144)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7cf9799c0050>

try 3  S3           member 2     645.9 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b37e0b30c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b37e0b32510>, 17928.443665458), (<aiohttp.client_proto.ResponseHandler object at 0x7b37e0b556d0>, 17928.526185437), (<aiohttp.client_proto.ResponseHandler object at 0x7b37e0e5ace0>, 17928.577724982), (<aiohttp.client_proto.ResponseHandler object at 0x7b37e0d097b0>, 17928.62636443), (<aiohttp.client_proto.ResponseHandler object at 0x7b37e0b327b0>, 17928.880665861), (<aiohttp.client_proto.ResponseHandler object at 0x7b37e0b55590>, 17928.918294313), (<aiohttp.client_proto.ResponseHandler object at 0x7b37e0b55450>, 17928.958752472), (<aiohttp.client_proto.ResponseHandler object at 0x7b37e0b50820>, 17928.998678499)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b37e0b54410>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b37e0b306e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b37e0b30980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b37e0b30ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b37e0f768d0>, 17926.662743982)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b37e0eb7770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b37e0f76bd0>, 17926.542280665)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b37e0af6e90>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b37e0e5a890>, 17926.478356823)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b37e0af6850>

try 3  S3           member 2    1291.7 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7515c4840ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7515c5392bd0>, 17934.21794982), (<aiohttp.client_proto.ResponseHandler object at 0x7515c4842660>, 17934.30992893), (<aiohttp.client_proto.ResponseHandler object at 0x7515c4c6e890>, 17934.353655746), (<aiohttp.client_proto.ResponseHandler object at 0x7515c48423c0>, 17934.396121776), (<aiohttp.client_proto.ResponseHandler object at 0x7515c4c6ee50>, 17934.434419209), (<aiohttp.client_proto.ResponseHandler object at 0x7515c4b25d30>, 17934.830683205)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7515c4a0b610>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7515c4840ad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7515c4840d70>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7515c4cdfbb0>, 17932.574574078)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7515c4c53b60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7515c53928d0>, 17932.701355827)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7515c4a08550>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f0948174c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f094853e5d0>, 17933.594594635), (<aiohttp.client_proto.ResponseHandler object at 0x7f0948176510>, 17933.807475962), (<aiohttp.client_proto.ResponseHandler object at 0x7f09481a1590>, 17934.046048503), (<aiohttp.client_proto.ResponseHandler object at 0x7f09481a1810>, 17934.11512579), (<aiohttp.client_proto.ResponseHandler object at 0x7f0948445bd0>, 17934.164499516), (<aiohttp.client_proto.ResponseHandler object at 0x7f09481a16d0>, 17934.202831732), (<aiohttp.client_proto.ResponseHandler object at 0x7f0948593130>, 17934.503836374), (<aiohttp.client_proto.ResponseHandler object at 0x7f09481767b0>, 17934.990827305)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f09481a0550>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f09481746e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f0948174980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f0948174ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f094d741310>, 17931.7535238)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f09483e3770>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f0948592ce0>, 17931.81763914)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f094814b610>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f094853e8d0>, 17931.877987849)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f094814bc50>

try 3  S3           member 2    1291.7 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f7adabf0ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7f7adabba2b0>, 17938.826776999), (<aiohttp.client_proto.ResponseHandler object at 0x7f7adabf0980>, 17938.944446964), (<aiohttp.client_proto.ResponseHandler object at 0x7f7adb7c68d0>, 17938.988358808), (<aiohttp.client_proto.ResponseHandler object at 0x7f7adb7c5fd0>, 17939.028469455), (<aiohttp.client_proto.ResponseHandler object at 0x7f7adb8f41a0>, 17939.071970714), (<aiohttp.client_proto.ResponseHandler object at 0x7f7adb0365b0>, 17939.286989881), (<aiohttp.client_proto.ResponseHandler object at 0x7f7adb0372a0>, 17939.54404049), (<aiohttp.client_proto.ResponseHandler object at 0x7f7adaef5ff0>, 17939.847913143)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7f7adb01bb60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x712cfcd38d70>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x712cfcd3a120>, 17940.275346098), (<aiohttp.client_proto.ResponseHandler object at 0x712cfd167e20>, 17940.374746849), (<aiohttp.client_proto.ResponseHandler object at 0x712cfd166fc0>, 17940.515743022), (<aiohttp.client_proto.ResponseHandler object at 0x712cfd25abd0>, 17940.614804747), (<aiohttp.client_proto.ResponseHandler object at 0x712cfd019d30>, 17940.834246204), (<aiohttp.client_proto.ResponseHandler object at 0x712cfcd3a3c0>, 17941.159995612)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x712cfceff890>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x712cfcd38980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x712cfcd38c20>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x712cfd25a5d0>, 17939.075065513)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x712cfd15ba10>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x712cfd027a20>, 17939.135339055)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x712cfcefc550>

try 3  S3           member 2    1291.7 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71c1b2318ad0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71c1b2861fd0>, 17944.405332559), (<aiohttp.client_proto.ResponseHandler object at 0x71c1b28628d0>, 17944.824037355), (<aiohttp.client_proto.ResponseHandler object at 0x71c1b24da2b0>, 17944.944789961), (<aiohttp.client_proto.ResponseHandler object at 0x71c1b23182f0>, 17945.286723055), (<aiohttp.client_proto.ResponseHandler object at 0x71c1b41a8050>, 17945.322990978), (<aiohttp.client_proto.ResponseHandler object at 0x71c1b274f580>, 17945.32354325), (<aiohttp.client_proto.ResponseHandler object at 0x71c1b274ea00>, 17945.579699347), (<aiohttp.client_proto.ResponseHandler object at 0x71c1b2609ff0>, 17946.639049626)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71c1b2737b60>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x774a4b064ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x774a4b066660>, 17946.491543756), (<aiohttp.client_proto.ResponseHandler object at 0x774a4b3cdd30>, 17946.615229044), (<aiohttp.client_proto.ResponseHandler object at 0x774a4b3dbd40>, 17946.65242017), (<aiohttp.client_proto.ResponseHandler object at 0x774a4b0663c0>, 17946.877981108), (<aiohttp.client_proto.ResponseHandler object at 0x774a4b31e2d0>, 17947.175235257), (<aiohttp.client_proto.ResponseHandler object at 0x774a4b31efc0>, 17947.216772649)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x774a4b04fb10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x774a4b064ad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x774a4b064d70>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x774a4b4128d0>, 17945.18514695)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x774a4b307b60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x774a4b4125d0>, 17945.120909429)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x774a4b04f4d0>

try 3  S3           member 2    2583.4 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x70550d01bb60>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x70550cc0a660>, 17952.648681396), (<aiohttp.client_proto.ResponseHandler object at 0x70550ce36a00>, 17952.694114404), (<aiohttp.client_proto.ResponseHandler object at 0x70550cf2ea50>, 17952.760784444), (<aiohttp.client_proto.ResponseHandler object at 0x70550cc0a3c0>, 17952.806528262), (<aiohttp.client_proto.ResponseHandler object at 0x70550cee5d30>, 17953.087879281), (<aiohttp.client_proto.ResponseHandler object at 0x70550ce36fc0>, 17953.463716861)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x70550cbd7c50>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x70550cc08830>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x70550cc08ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x70550cf2e5d0>, 17951.306643675)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x70550cc08050>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x70550ceede00>, 17951.242068747)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x70550cbd6c10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71f098bd9160>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71f098efea50>, 17952.990506736), (<aiohttp.client_proto.ResponseHandler object at 0x71f098bf9bd0>, 17953.31127815), (<aiohttp.client_proto.ResponseHandler object at 0x71f098bda900>, 17953.433033172), (<aiohttp.client_proto.ResponseHandler object at 0x71f098df7410>, 17953.483312821), (<aiohttp.client_proto.ResponseHandler object at 0x71f098bdaba0>, 17953.728727757), (<aiohttp.client_proto.ResponseHandler object at 0x71f098bf91d0>, 17954.064689197), (<aiohttp.client_proto.ResponseHandler object at 0x71f098bf9d10>, 17954.247398361), (<aiohttp.client_proto.ResponseHandler object at 0x71f098ea1d30>, 17954.471560018)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71f098bf8e10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71f098bd8c20>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71f098bd8ec0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x71f098bd9010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71f098214500>, 17951.348693952)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71f098fd7cb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71f098df6e50>, 17951.410189492)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71f098b97250>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x71f098eff050>, 17951.472566344)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x71f098bf8550>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x78866aff4980>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x78866afbe2b0>, 17952.106519219), (<aiohttp.client_proto.ResponseHandler object at 0x78866aff4830>, 17952.302021636), (<aiohttp.client_proto.ResponseHandler object at 0x7886701328d0>, 17953.145429701), (<aiohttp.client_proto.ResponseHandler object at 0x788670131fd0>, 17953.215806257)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x78866b147a10>

try 3  S3           member 2    2583.4 MB  call 2/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe97e494ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7fe97e761bd0>, 17960.248868715), (<aiohttp.client_proto.ResponseHandler object at 0x7fe97d2c1f90>, 17960.510665832), (<aiohttp.client_proto.ResponseHandler object at 0x7fe98dab9a30>, 17960.749452127), (<aiohttp.client_proto.ResponseHandler object at 0x7fe97e496660>, 17961.169165077), (<aiohttp.client_proto.ResponseHandler object at 0x7fe97e7aaa50>, 17961.233442207), (<aiohttp.client_proto.ResponseHandler object at 0x7fe97d2c1e50>, 17961.361066813), (<aiohttp.client_proto.ResponseHandler object at 0x7fe97e496900>, 17962.103912745), (<aiohttp.client_proto.ResponseHandler object at 0x7fe97d2c1d10>, 17962.345374685)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7fe97e467610>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe97e494980>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe97e494c20>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe97e494d70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7523c44f16a0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7523c4325e50>, 17961.932993713), (<aiohttp.client_proto.ResponseHandler object at 0x7523c44f2e40>, 17962.004601509), (<aiohttp.client_proto.ResponseHandler object at 0x7523c4716e50>, 17962.257534053), (<aiohttp.client_proto.ResponseHandler object at 0x7523c4716ce0>, 17962.386484548), (<aiohttp.client_proto.ResponseHandler object at 0x7523c44f2ba0>, 17962.387078134), (<aiohttp.client_proto.ResponseHandler object at 0x7523c4325d10>, 17962.640655878), (<aiohttp.client_proto.ResponseHandler object at 0x7523c47c5a70>, 17962.891124091), (<aiohttp.client_proto.ResponseHandler object at 0x7523c4325bd0>, 17965.120528316)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7523c4324910>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7523c44f0ad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7523c44f1550>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7523c48d3e00>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7523c4797d40>, 17960.373885136)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7523c44f01a0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7523c4822750>, 17960.434354876)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7523c44c6850>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73704e1e4ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7523c4822d50>, 17960.303559471)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7523c44c6fd0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73704e32ab70>, 17961.919791897), (<aiohttp.client_proto.ResponseHandler object at 0x73704d81c190>, 17962.060473215), (<aiohttp.client_proto.ResponseHandler object at 0x73704e32b580>, 17962.199541368), (<aiohttp.client_proto.ResponseHandler object at 0x73704d81c050>, 17962.44692644), (<aiohttp.client_proto.ResponseHandler object at 0x73704e1e6660>, 17962.523781061), (<aiohttp.client_proto.ResponseHandler object at 0x73704e1e6900>, 17962.564174805), (<aiohttp.client_proto.ResponseHandler object at 0x73704d81d310>, 17962.933148376), (<aiohttp.client_proto.ResponseHandler object at 0x73704e391bd0>, 17963.158449788)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73706f0aed50>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73704e1e46e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73704e323a10>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73704e1e4d70>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73704e19fd40>, 17960.366461175)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73704e323cb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73704e422d50>, 17960.427514955)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73704e1bead0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73704e422ed0>, 17960.487697067)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73704e1bf250>

try 3  S3           member 2    2583.4 MB  call 3/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x777e6c103a10>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x777e6ed11a90>, 17970.998593598), (<aiohttp.client_proto.ResponseHandler object at 0x777e6f09b110>, 17971.267245016), (<aiohttp.client_proto.ResponseHandler object at 0x777e6cb9cb90>, 17971.641214544), (<aiohttp.client_proto.ResponseHandler object at 0x777e6f09ad50>, 17971.684490718), (<aiohttp.client_proto.ResponseHandler object at 0x777e709abed0>, 17971.752116025), (<aiohttp.client_proto.ResponseHandler object at 0x777e709abd90>, 17972.439535639), (<aiohttp.client_proto.ResponseHandler object at 0x777e6cb9ca50>, 17972.822976402), (<aiohttp.client_proto.ResponseHandler object at 0x777e6ed13110>, 17972.954754742)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x777e6cb9ee90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x777e5fecc6e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x777e5feccd70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73ce8607ba10>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73ce8617abd0>, 17972.491466201), (<aiohttp.client_proto.ResponseHandler object at 0x73ce86087580>, 17972.783137532), (<aiohttp.client_proto.ResponseHandler object at 0x73ce85d6e0d0>, 17972.8213475), (<aiohttp.client_proto.ResponseHandler object at 0x73ce85d42a50>, 17973.081363396), (<aiohttp.client_proto.ResponseHandler object at 0x73ce860e10d0>, 17973.144995932), (<aiohttp.client_proto.ResponseHandler object at 0x73ce85d6df90>, 17973.379645429), (<aiohttp.client_proto.ResponseHandler object at 0x73ce85d427b0>, 17973.500706104), (<aiohttp.client_proto.ResponseHandler object at 0x73ce85d6e210>, 17974.100884191)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73ce85d12ad0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73ce85d406e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73ce85d40ec0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x73ce85d41010>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9fb4ad16a0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73ce85f47bb0>, 17971.115327953)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73ce8607bcb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4cf6a00>, 17972.603379382), (<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4ad2a50>, 17972.708155337), (<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4af9f90>, 17972.77787538), (<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4ad27b0>, 17972.778874734), (<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4af9e50>, 17973.04049617), (<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4afa0d0>, 17973.284898018), (<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4cf76f0>, 17973.757923375), (<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4d58cb0>, 17974.053172302)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9fb4a9d6d0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73ce8617aed0>, 17970.994316341)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73ce85d10910>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x73ce86086ce0>, 17971.054962696)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x73ce85d6c690>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9fb4ad06e0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9fb4ad0d70>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7c9fb4ad0ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9fb5656a50>, 17970.985556082)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9fb4eefcb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9fb4dafbb0>, 17971.047212358)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9fb4a9d810>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7c9fb5656ed0>, 17970.920314609)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7c9fb4a9ead0>

try 3  OSDF         member 2     161.5 MB  call 1/3  [ok]

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b31e6845010>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b31e6846510>, 17972.36390303), (<aiohttp.client_proto.ResponseHandler object at 0x7b31e6cfaa00>, 17972.495774217), (<aiohttp.client_proto.ResponseHandler object at 0x7b31e68467b0>, 17972.549914936), (<aiohttp.client_proto.ResponseHandler object at 0x7b31e6cfafc0>, 17972.817255996), (<aiohttp.client_proto.ResponseHandler object at 0x7b31e6c0a5d0>, 17973.128348667), (<aiohttp.client_proto.ResponseHandler object at 0x7b31e6b5dff0>, 17973.279644346)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b31e682bd90>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b31e6844c20>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7b31e6844ec0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b31e6be3890>, 17970.99703271)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b31e6ccbcb0>
Unclosed connector
connections: ['deque([(<aiohttp.client_proto.ResponseHandler object at 0x7b31e6c0a8d0>, 17970.932789398)])']
connector: <aiobotocore.httpsession._ProxySSLTCPConnector object at 0x7b31e682ae90>

try 3  OSDF         member 2     161.5 MB  call 2/3  [ok]
try 3  OSDF         member 2     161.5 MB  call 3/3  [ok]
try 3  OSDF         member 2     322.9 MB  call 1/3  [ok]
try 3  OSDF         member 2     322.9 MB  call 2/3  [ok]
try 3  OSDF         member 2     322.9 MB  call 3/3  [ok]
try 3  OSDF         member 2     645.9 MB  call 1/3  [ok]
try 3  OSDF         member 2     645.9 MB  call 2/3  [ok]
try 3  OSDF         member 2     645.9 MB  call 3/3  [ok]
try 3  OSDF         member 2    1291.7 MB  call 1/3  [ok]
try 3  OSDF         member 2    1291.7 MB  call 2/3  [ok]
try 3  OSDF         member 2    1291.7 MB  call 3/3  [ok]
try 3  OSDF         member 2    2583.4 MB  call 1/3  [ok]
try 3  OSDF         member 2    2583.4 MB  call 2/3  [ok]
try 3  OSDF         member 2    2583.4 MB  call 3/3  [ok]
try 3  OSDF-direct  member 2     161.5 MB  call 1/3  [ok]
try 3  OSDF-direct  member 2     161.5 MB  call 2/3  [ok]
try 3  OSDF-direct  member 2     161.5 MB  call 3/3  [ok]
try 3  OSDF-direct  member 2     322.9 MB  call 1/3  [ok]
try 3  OSDF-direct  member 2     322.9 MB  call 2/3  [ok]
try 3  OSDF-direct  member 2     322.9 MB  call 3/3  [ok]
try 3  OSDF-direct  member 2     645.9 MB  call 1/3  [ok]
try 3  OSDF-direct  member 2     645.9 MB  call 2/3  [ok]
try 3  OSDF-direct  member 2     645.9 MB  call 3/3  [ok]
try 3  OSDF-direct  member 2    1291.7 MB  call 1/3  [ok]
try 3  OSDF-direct  member 2    1291.7 MB  call 2/3  [ok]
try 3  OSDF-direct  member 2    1291.7 MB  call 3/3  [ok]

2026-09-21 20:09:18,512 - distributed.worker - ERROR - Compute Failed
Key:       ('open_dataset-TREFHTMX-getitem-c00ee5bc9771e7cad50726a39d9467e4', 6, 0, 0)
State:     executing
Task:  <Task ('open_dataset-TREFHTMX-getitem-c00ee5bc9771e7cad50726a39d9467e4', 6, 0, 0) _execute_subgraph(...)>
Exception: "RuntimeError('error during blosc decompression: -1')"
Traceback: '  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/dask/array/core.py", line 140, in getter\n    c = np.asarray(c)\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 691, in __array__\n    return np.asarray(to_numpy(self.get_duck_array()), dtype=dtype, copy=copy)\n                               ~~~~~~~~~~~~~~~~~~~^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 697, in get_duck_array\n    return self.array.get_duck_array()\n           ~~~~~~~~~~~~~~~~~~~~~~~~~^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 930, in get_duck_array\n    return self.array.get_duck_array()\n           ~~~~~~~~~~~~~~~~~~~~~~~~~^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 770, in get_duck_array\n    array = self.array[self.key]\n            ~~~~~~~~~~^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/backends/zarr.py", line 316, in __getitem__\n    return indexing.explicit_indexing_adapter(\n           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^\n        key, array.shape, indexing.IndexingSupport.VECTORIZED, method\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 1162, in explicit_indexing_adapter\n    result = raw_indexing_method(raw_key.tuple)\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/backends/zarr.py", line 279, in _getitem\n    return self._array[key]\n           ~~~~~~~~~~~^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 2639, in __getitem__\n    return self.get_orthogonal_selection(pure_selection, fields=fields)\n           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 3116, in get_orthogonal_selection\n    return sync(\n        self.async_array._get_selection(\n            indexer=indexer, out=out, fields=fields, prototype=prototype\n        )\n    )\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/sync.py", line 154, in sync\n    raise return_result\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/sync.py", line 114, in _runner\n    return await coro\n           ^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 1456, in _get_selection\n    return await _get_selection(\n           ^^^^^^^^^^^^^^^^^^^^^\n    ...<9 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 5480, in _get_selection\n    results = await codec_pipeline.read(\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^\n    ...<14 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 830, in read\n    batch_results = await concurrent_map(\n                    ^^^^^^^^^^^^^^^^^^^^^\n    ...<6 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 131, in concurrent_map\n    return await asyncio.gather(*concurrent_iter(items, func, limit))\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 121, in run\n    return await func(*item)\n           ^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 778, in read_batch\n    return await _async_read_fallback(self, list(batch_info), out, drop_axes)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 352, in _async_read_fallback\n    await pipeline.decode(\n    ...<6 lines>...\n    )\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 812, in decode\n    output.extend(await self.decode_batch(batch_info))\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 697, in decode_batch\n    chunk_array_batch = await ab_codec.decode(\n                        ^^^^^^^^^^^^^^^^^^^^^^\n        zip(chunk_bytes_batch, chunk_spec_batch, strict=False)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/abc/codec.py", line 204, in decode\n    return await _batching_helper(self._decode_single, chunks_and_specs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/abc/codec.py", line 515, in _batching_helper\n    return await concurrent_map(\n           ^^^^^^^^^^^^^^^^^^^^^\n    ...<3 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 131, in concurrent_map\n    return await asyncio.gather(*concurrent_iter(items, func, limit))\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 121, in run\n    return await func(*item)\n           ^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/abc/codec.py", line 528, in wrap\n    return await func(chunk, chunk_spec)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/codecs/_v2.py", line 104, in _decode_single\n    return await asyncio.to_thread(self._decode_sync, chunk_bytes, chunk_spec)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3.14/asyncio/threads.py", line 25, in to_thread\n    return await loop.run_in_executor(None, func_call)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3.14/concurrent/futures/thread.py", line 86, in run\n    result = ctx.run(self.task)\n  File "/usr/lib/python3.14/concurrent/futures/thread.py", line 73, in run\n    return fn(*args, **kwargs)\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/codecs/_v2.py", line 34, in _decode_sync\n    chunk = self.compressor.decode(cdata)\n  File "numcodecs/blosc.pyx", line 500, in numcodecs.blosc.Blosc.decode\n  File "numcodecs/blosc.pyx", line 399, in numcodecs.blosc.decompress\n'


try 3  OSDF-direct  member 2    2583.4 MB  call 1/3  [FAIL RuntimeError]
try 3  OSDF-direct  member 2    2583.4 MB  call 2/3  [ok]
try 3  OSDF-direct  member 2    2583.4 MB  call 3/3  [FAIL RuntimeError]

2026-09-21 20:09:41,448 - distributed.worker - ERROR - Compute Failed
Key:       ('open_dataset-TREFHTMX-getitem-c00ee5bc9771e7cad50726a39d9467e4', 6, 0, 0)
State:     executing
Task:  <Task ('open_dataset-TREFHTMX-getitem-c00ee5bc9771e7cad50726a39d9467e4', 6, 0, 0) _execute_subgraph(...)>
Exception: "RuntimeError('error during blosc decompression: -1')"
Traceback: '  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/dask/array/core.py", line 140, in getter\n    c = np.asarray(c)\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 691, in __array__\n    return np.asarray(to_numpy(self.get_duck_array()), dtype=dtype, copy=copy)\n                               ~~~~~~~~~~~~~~~~~~~^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 697, in get_duck_array\n    return self.array.get_duck_array()\n           ~~~~~~~~~~~~~~~~~~~~~~~~~^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 930, in get_duck_array\n    return self.array.get_duck_array()\n           ~~~~~~~~~~~~~~~~~~~~~~~~~^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 770, in get_duck_array\n    array = self.array[self.key]\n            ~~~~~~~~~~^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/backends/zarr.py", line 316, in __getitem__\n    return indexing.explicit_indexing_adapter(\n           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^\n        key, array.shape, indexing.IndexingSupport.VECTORIZED, method\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/core/indexing.py", line 1162, in explicit_indexing_adapter\n    result = raw_indexing_method(raw_key.tuple)\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/xarray/backends/zarr.py", line 279, in _getitem\n    return self._array[key]\n           ~~~~~~~~~~~^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 2639, in __getitem__\n    return self.get_orthogonal_selection(pure_selection, fields=fields)\n           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 3116, in get_orthogonal_selection\n    return sync(\n        self.async_array._get_selection(\n            indexer=indexer, out=out, fields=fields, prototype=prototype\n        )\n    )\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/sync.py", line 154, in sync\n    raise return_result\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/sync.py", line 114, in _runner\n    return await coro\n           ^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 1456, in _get_selection\n    return await _get_selection(\n           ^^^^^^^^^^^^^^^^^^^^^\n    ...<9 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/array.py", line 5480, in _get_selection\n    results = await codec_pipeline.read(\n              ^^^^^^^^^^^^^^^^^^^^^^^^^^\n    ...<14 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 830, in read\n    batch_results = await concurrent_map(\n                    ^^^^^^^^^^^^^^^^^^^^^\n    ...<6 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 131, in concurrent_map\n    return await asyncio.gather(*concurrent_iter(items, func, limit))\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 121, in run\n    return await func(*item)\n           ^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 778, in read_batch\n    return await _async_read_fallback(self, list(batch_info), out, drop_axes)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 352, in _async_read_fallback\n    await pipeline.decode(\n    ...<6 lines>...\n    )\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 812, in decode\n    output.extend(await self.decode_batch(batch_info))\n                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/codec_pipeline.py", line 697, in decode_batch\n    chunk_array_batch = await ab_codec.decode(\n                        ^^^^^^^^^^^^^^^^^^^^^^\n        zip(chunk_bytes_batch, chunk_spec_batch, strict=False)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/abc/codec.py", line 204, in decode\n    return await _batching_helper(self._decode_single, chunks_and_specs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/abc/codec.py", line 515, in _batching_helper\n    return await concurrent_map(\n           ^^^^^^^^^^^^^^^^^^^^^\n    ...<3 lines>...\n    )\n    ^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 131, in concurrent_map\n    return await asyncio.gather(*concurrent_iter(items, func, limit))\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/core/common.py", line 121, in run\n    return await func(*item)\n           ^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/abc/codec.py", line 528, in wrap\n    return await func(chunk, chunk_spec)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/codecs/_v2.py", line 104, in _decode_single\n    return await asyncio.to_thread(self._decode_sync, chunk_bytes, chunk_spec)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3.14/asyncio/threads.py", line 25, in to_thread\n    return await loop.run_in_executor(None, func_call)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/lib/python3.14/concurrent/futures/thread.py", line 86, in run\n    result = ctx.run(self.task)\n  File "/usr/lib/python3.14/concurrent/futures/thread.py", line 73, in run\n    return fn(*args, **kwargs)\n  File "/home/ubuntu/osdf-env/lib/python3.14/site-packages/zarr/codecs/_v2.py", line 34, in _decode_sync\n    chunk = self.compressor.decode(cdata)\n  File "numcodecs/blosc.pyx", line 500, in numcodecs.blosc.Blosc.decode\n  File "numcodecs/blosc.pyx", line 399, in numcodecs.blosc.decompress\n'
Loading...
colors = {"S3": "#1f77b4", "OSDF": "#ff7f0e", "OSDF-direct": "#2ca02c"}
marker = {"S3": "o", "OSDF": "x", "OSDF-direct": "s"}
fig, ax = plt.subplots(figsize=(10, 6))
for proto in ["S3", "OSDF", "OSDF-direct"]:
    sub = df[df.protocol == proto]
    first  = sub[sub.call_number == 1].groupby("dataset_size").MBps.mean()
    subseq = sub[sub.call_number  > 1].groupby("dataset_size").MBps.mean()
    ax.plot(first.index,  first.values,  color=colors[proto], marker=marker[proto], ms=7, label=f"{proto} — first")
    ax.plot(subseq.index, subseq.values, color=colors[proto], marker=marker[proto], ms=7, ls="--", label=f"{proto} — subsequent")
ax.set_xlabel("Data size (MB)"); ax.set_ylabel("Bandwidth (MB/s)"); ax.set_xscale("log")
ax.set_title(f"CESM2-LENS us-west-2 → EC2: S3 vs OSDF vs OSDF-direct\n"
             f"{N_TRIES} outer tries (member cycled) × {N_CALLS} inner calls — first vs subsequent")
ax.grid(True, which="both", alpha=0.3); ax.legend(ncol=3, fontsize=8)
plt.show()
<Figure size 1000x600 with 1 Axes>

Summary — bandwidth vs data size

Method: N_TRIES = 3 outer passes (a different member_id each pass, so first reads are genuinely uncached) × N_CALLS = 3 inner reads per subset (call 1 = first, calls 2–3 = subsequent), each size from a disjoint block of chunks. Median bandwidth (MB/s):

SizeS3 first / subsOSDF first / subsOSDF-direct first / subs
161 MB59 / 68110 / 5520 / 28
323 MB98 / 10067 / 5339 / 49
646 MB126 / 124153 / 13369 / 93
1292 MB207 / 20892 / 161116 / 139
2583 MB287 / 275135 / 190175 / 211

  • S3 scales smoothly and monotonically (≈ 59 → 287 MB/s) and is essentially flat first-vs-subsequent (no client cache), making it the fastest, most predictable path at ≳ 646 MB.

  • OSDF-direct also scales cleanly (≈ 20 → 175 MB/s) and is the most consistent OSDF path, but is slowest at small/mid sizes (pelican-client + origin overhead). Its modest first→subsequent lift is connection warm-up, not data caching.

  • OSDF-cached is erratic and non-monotonic — sometimes the fastest at a size (110 MB/s at 161 MB), sometimes among the slowest, with the first-vs-subsequent ordering flipping size to size. That is the signature of the OSDF director routing successive requests to different cache nodes rather than a clean warm-up.

  • None of the three nears the m5.4xlarge NIC ceiling (~10 Gbps ≈ 1250 MB/s; peak here ≈ 290 MB/s ≈ 2.3 Gbps): decode/parallelism, not the network, is the bottleneck. These are relative protocol comparisons, not peak throughput.

dfb = df.copy()
dfb["dataset_size"] = dfb["dataset_size"].astype(float).round(0).astype(int)
dfb["access"] = np.where(dfb["call_number"] == 1, "first", "subsequent")
dfb["series"] = dfb["protocol"] + " — " + dfb["access"]
order = sorted(dfb["dataset_size"].unique())
hue_order = ["S3 — first","S3 — subsequent","OSDF — first","OSDF — subsequent",
             "OSDF-direct — first","OSDF-direct — subsequent"]
palette = {"S3 — first":"#1f4e79","S3 — subsequent":"#7fb3e0",
           "OSDF — first":"#b3560a","OSDF — subsequent":"#f5b27a",
           "OSDF-direct — first":"#1b5e20","OSDF-direct — subsequent":"#8bc98f"}
plt.figure(figsize=(15, 6))
sns.boxplot(data=dfb, x="dataset_size", y="MBps", hue="series", order=order, hue_order=hue_order, palette=palette)
plt.xlabel("Data size (MB)"); plt.ylabel("Bandwidth (MB/s)")
plt.title(f"OSDF vs S3 to EC2 (us-west-2), CESM2-LENS TREFHTMX daily\n"
          f"{N_TRIES} outer tries (member cycled) × {N_CALLS} inner calls — first vs subsequent")
plt.legend(title="", ncol=3, fontsize=8); plt.show()
<Figure size 1500x600 with 1 Axes>

Summary — distributions (first vs subsequent)

Each box spans up to N_TRIES × N_CALLS reads (3 first + 6 subsequent per size per series).

  • Spread is the story. S3 is tight and reproducible (e.g. 284–289 MB/s across all first-access reads at 2583 MB). OSDF-cached is by far the widest — its subsequent reads span 10–187 MB/s at 323 MB and 81–239 MB/s at 1292 MB. That run-to-run swing, together with the non-monotonic medians above, is consistent with the director spreading requests across different cache nodes (varying proximity/state) rather than a repeatable cold→warm transition.

  • OSDF-direct is the tight OSDF path (e.g. 19.5–30 MB/s across all reads at 161 MB), confirming that most of OSDF-cached’s variance lives in the cache/routing layer, not the client or origin.

  • Reliability: 2 of 135 reads failed — both OSDF-direct at 2583 MB (RuntimeError: error during blosc decompression: -1), i.e. a truncated/corrupt chunk on the origin path for the largest reads.

  • Takeaway: for same-region, AWS-native data read from EC2, native S3 is fastest and most consistent. OSDF’s value here is not raw in-region throughput — it is federation, a uniform client across heterogeneous origins, and cache locality for the repeated/shared access patterns this single-client, single-region test does not reward.

Other computational Platforms
Equilibrium Climate Sensitivity from CMIP6
Other computational Platforms
Benchmark NCAR-origin access (DART)