Class BatchClient

Namespace

DatabentoDotNet.Historical

Assembly

DatabentoDotNet.Historical.dll

The batch.* endpoints: jobs that produce files on Databento's side, and the transfer of those files to this one.

public sealed class BatchClient

Inheritance

object

BatchClient

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Examples

// Submitting is the billable act; everything after it is free and repeatable.
BatchJob job = await client.Batch.SubmitJobAsync(new SubmitJobParams
{
    Dataset = "GLBX.MDP3",
    Symbols = Symbols.From("ESH4"),
    Schema = Schema.Trades,
    DateTimeRange = DateRange.OnDay(new LocalDate(2024, 1, 2)).ToDateTimeRange(),
    SplitDuration = SplitDuration.None,   // one file rather than one per day
});

// Record the id before polling. The job is paid for from here on, and this is how it is
// collected if this process dies.
Console.WriteLine(job.Id);

while (job.State is not (JobState.Done or JobState.Expired or JobState.Purged))
{
    await Task.Delay(5_000);
    job = await client.Batch.GetJobDetailsAsync(job.Id);
}

// Files land in {OutputDirectory}/{JobId}/, and a partial file from an interrupted run is
// resumed rather than restarted — a download costs nothing however many times it runs.
IReadOnlyList<string> files = await client.Batch.DownloadAsync(new DownloadParams
{
    JobId = job.Id,
    OutputDirectory = "/tmp/databento",
});

// A job delivers its metadata and a condition report alongside the data, so the .dbn.zst has to
// be picked out rather than assumed to be the only file.
await using var reader = await TimeseriesClient.OpenFileAsync(
    files.First(path => path.EndsWith(".dbn.zst", StringComparison.Ordinal)));

Remarks

Reached through Batch rather than constructed. Port of upstream's BatchClient (batch.rs:33-36), which holds a mutable borrow of the outer client; this holds a reference, there being no borrow checker to satisfy.

One method here costs money: SubmitJobAsync(SubmitJobParams, CancellationToken). Listing, inspecting and downloading are free, and downloading stays free however many times it is repeated — a job is billed once, when it is submitted, and its files remain fetchable until ExpirationTimestamp. Upstream marks exactly one of its methods with a cost warning and #39 confirmed the asymmetry against the live API. So a download may be retried, resumed after a crash, or run again from scratch without a second charge, which is what makes the resumption logic below worth having rather than merely tidy.

Batch versus timeseries.get_range. Both move market data and they suit different shapes of request: GetRangeAsync(GetRangeParams, CancellationToken) streams a range this process decodes as it arrives, while a batch job runs on Databento's side, may produce hundreds of files, survives the process that asked for it, and can be fetched days later. Ranges too large to wait on, or wanted as CSV or JSON, are batch jobs; anything a program consumes as it reads is a range.

Fields

MaximumRetries

How many times a file transfer is retried after a mid-stream failure before it gives up.

public const int MaximumRetries = 5

Field Value

int

Remarks

Upstream's MAX_RETRIES (batch.rs:252), and its counting rule with it: the counter measures consecutive failures, resetting whenever an attempt manages to write a byte before failing. Six failures in a row give up; six failures spread across an hour of otherwise healthy progress do not.

#39's own porting notes said the opposite of this, and reading upstream settled it. The note read: "upstream's retry counter is shared across the whole file rather than reset on progress, so a long download that fails six times over an hour gives up; decide whether to keep that and write down which." Upstream does reset it — if retries > 0 { retries = 0; info!("Resumed download"); } at batch.rs:308-311, on the first chunk to arrive after a retry — so there was no decision to make and nothing to depart from. The behaviour described in the note is the one neither library has.

Methods

DownloadAsync(DownloadParams, CancellationToken)

Downloads a job's files into {OutputDirectory}/{JobId}/, resuming any that are partly there already.

public Task<IReadOnlyList<string>> DownloadAsync(DownloadParams parameters, CancellationToken cancellationToken = default)

Parameters

parameters DownloadParams

Which job, where to put it, and how many files at once.

cancellationToken CancellationToken

Cancels the download.

Returns

Task<IReadOnlyList<string>>

The path of every file written or already present, in the order ListFilesAsync(string, CancellationToken) reported them — or the single path, when Filename named one.

Remarks

Port of upstream's download and download_file (batch.rs:195-322). Downloading costs nothing, so everything below is free to repeat.

Three cases decide what happens to a file already on disk, and they are upstream's (check_if_exists, batch.rs:325-368). A file shorter than Size resumes: the bytes already there are fed through the checksum and the request carries Range: bytes=N-. A file of exactly that size is left alone and no request is made at all. A file longer than it is an error — it is not this library's place to decide that somebody else's larger file is wrong and truncate it.

Two departures from upstream, both named on #39, and one defect found while porting.

1. A checksum mismatch throws. Upstream hashes the file and, on a mismatch, logs a warning and returns success (verify_hash, batch.rs:370-383) — so a corrupt download is reported by a log line the caller may not be listening to, and the path it returns points at bad data. This throws InvalidDataException naming the file. The partial file is left on disk rather than deleted: it is evidence, and deleting it would also delete a resumable transfer that failed for some reason other than corruption. A caller who wants a clean retry deletes it themselves.

2. Files transfer in parallel, bounded by MaximumConcurrency, where upstream's loop is sequential. Per-file behaviour is untouched; see that property.

3. The hasher is rebuilt on every attempt, and upstream's is not. Upstream creates one hasher outside its retry loop and calls check_if_exists inside it, which re-reads the whole partial file into that same hasher — so after any retry the bytes already on disk have been hashed twice and the final digest cannot match. The bug is invisible upstream because a mismatch there is only a warning; it would be fatal here, where a mismatch throws, and it would fire on exactly the resumed transfers this issue exists to make work. Each attempt below gets its own hasher, seeded once with whatever is on disk.

A file that was already complete before the call is trusted on its size alone, as it is upstream: no request is made for it and its checksum is not recomputed, which is what makes re-running a finished download free. A file this call completed is verified even when the transfer that completed it reported failure — see DownloadFileAsync(BatchFileDescription, string, CancellationToken), since a reset arriving after the final chunk is exactly that case.

A checksum this library cannot compute is skipped, not failed, which is upstream's behaviour and the right one — an unrecognised algorithm means Databento has added one, not that the data is bad. It is logged, because it silently downgrades the guarantee point 1 just strengthened. See Internal/HistoricalLog.cs.

Exceptions

ArgumentNullException

parameters is null.

ArgumentException

OutputDirectory holds a file where the job's directory belongs, or Filename names a file the job does not have.

InvalidDataException

A file's contents did not match its checksum.

IOException

A file on disk is larger than the API says it should be, or the transfer failed more than MaximumRetries times in a row.

DatabentoApiException

The API answered with a non-success status.

GetJobDetailsAsync(string, CancellationToken)

Fetches everything the API knows about one job.

public Task<BatchJob> GetJobDetailsAsync(string jobId, CancellationToken cancellationToken = default)

Parameters

jobId string

The job's Id.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<BatchJob>

The job.

Remarks

Port of upstream's get_job_details (batch.rs:164-172). This is how a submitted job is watched: poll it until State is Done, reading Progress on the way. It costs nothing to call.

Exceptions

ArgumentException

jobId is null or empty.

DatabentoApiException

The API answered with a non-success status — 404 with Case batch_job_not_found for an id this account does not have, whatever the id looks like. That differs from ListFilesAsync(string, CancellationToken), and #39 got it wrong before measuring it. See that method.

ListFilesAsync(string, CancellationToken)

Lists the files one job produced.

public Task<IReadOnlyList<BatchFileDescription>> ListFilesAsync(string jobId, CancellationToken cancellationToken = default)

Parameters

jobId string

The job's Id.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<IReadOnlyList<BatchFileDescription>>

One description per file.

Remarks

Port of upstream's list_files (batch.rs:179-187). Free to call, and the only source of a file's download URL — DownloadAsync(DownloadParams, CancellationToken) calls it rather than composing a URL from a filename, which is upstream's rule and #39's porting note both.

The list includes three files Databento packages with every job — manifest.json, metadata.json and condition.json — beside the data itself, which #39 confirmed against four separate jobs. A full download fetches all of them.

This endpoint checks the shape of a job id and GetJobDetailsAsync(string, CancellationToken) does not, which is a difference #39 assumed away and then measured. A malformed id — NOPE-123 — is a 400 here carrying the API's simple error body, while a well-formed id for a job that does not exist is a 404 carrying the business one, batch_job_not_found. GetJobDetailsAsync(string, CancellationToken) answers 404 to both.

The porting lesson is CLAUDE.md's, restated: probe the endpoint you are about to describe, not the one next to it. A single probe of NOPE-123 against this endpoint had already been written into GetJobDetailsAsync(string, CancellationToken)'s documentation as its behaviour, and it was wrong — the same shape of mistake as #45, and caught the same way, by a test that calls the real API.

Exceptions

ArgumentException

jobId is null or empty.

DatabentoApiException

The API answered with a non-success status — 400 for a malformed job id, 404 for a well-formed one this account does not have.

ListJobsAsync(ListJobsParams?, CancellationToken)

Lists previous jobs, returning the id, state and receipt time of each.

public Task<IReadOnlyList<BatchJobSummary>> ListJobsAsync(ListJobsParams? parameters = null, CancellationToken cancellationToken = default)

Parameters

parameters ListJobsParams

The state and submission-time filters, or null to filter nothing.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<IReadOnlyList<BatchJobSummary>>

One summary per job, newest last, as the API orders them.

Remarks

Port of upstream's list_jobs (batch.rs:107-118), which asks for the short form by sending short=true alongside the filters. Fetch the rest of a job with GetJobDetailsAsync(string, CancellationToken).

Prefer this over ListJobsFullAsync(ListJobsParams?, CancellationToken), which the API is retiring — see that method. The short form is the one that will keep working.

Exceptions

DatabentoApiException

The API answered with a non-success status.

ListJobsFullAsync(ListJobsParams?, CancellationToken)

Lists previous jobs, returning every field of each.

[Obsolete("The batch.list_jobs endpoint will stop returning full job details at a future date. Use ListJobsAsync and GetJobDetailsAsync instead. Deprecated upstream in databento-rs 0.60.0.")]
public Task<IReadOnlyList<BatchJob>> ListJobsFullAsync(ListJobsParams? parameters = null, CancellationToken cancellationToken = default)

Parameters

parameters ListJobsParams

The state and submission-time filters, or null to filter nothing.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<IReadOnlyList<BatchJob>>

One full job per entry, newest last, as the API orders them.

Remarks

Port of upstream's list_jobs_full (batch.rs:131-137), including its deprecation. Upstream deprecated it in 0.60.0 — the version this library ports — with the note that "the batch.list_jobs endpoint will stop returning full job details at a future date; use list_jobs() and get_job_details() instead". A doc comment alone would not carry that to a caller, so it is an attribute here and their compiler tells them.

It is ported at all because the endpoint still answers today — #39 confirmed both forms against the live API — and a library that silently dropped a working endpoint would send a caller who needs it back to raw HTTP.

Exceptions

DatabentoApiException

The API answered with a non-success status.

SubmitJobAsync(SubmitJobParams, CancellationToken)

Submits a batch job and returns the API's description of it.

public Task<BatchJob> SubmitJobAsync(SubmitJobParams parameters, CancellationToken cancellationToken = default)

Parameters

parameters SubmitJobParams

What to produce, over what range, in what encoding.

cancellationToken CancellationToken

Cancels the request.

Returns

Task<BatchJob>

The submitted job.

Remarks

Port of upstream's submit_job (batch.rs:67-93). This costs money, and it commits to the whole range at once: unlike a stream, a submitted job cannot be stopped part-way to limit what is billed. Price it first with GetCostAsync(MetadataQueryParams, CancellationToken), handing it ToQuery() so the quote covers the request actually being sent.

The returned job is answered immediately, long before it has run, so most of BatchJob's optional properties are null at this point. Watch it with GetJobDetailsAsync(string, CancellationToken) until State reaches Done, then fetch its files with DownloadAsync(DownloadParams, CancellationToken).

Exceptions

ArgumentNullException

parameters is null.

InvalidOperationException

parameters asks for a combination the API rejects — see ToFormParameters().

DatabentoApiException

The API answered with a non-success status.