Handle long runs, rate limits and timeouts

What a 202 means, which errors to retry, how long to wait, and what is never charged.

Long Default mode runs: 202

A Default mode decision can take longer than one HTTP response: a large one takes about a minute, and a model waking from idle adds a few minutes. When it is not done after about 50 seconds, POST /v1/decisions answers 202 with {id, status: "running", progress, poll_url, retry_after}. The decision goes on running; poll GET /v1/decisions/{id} every retry_after seconds until its status is ok (the decide answer is in response) or error (errorMessage says why; nothing is billed). It is billed when it completes, also if you stop polling. Send "async": true to get the 202 at once. Up to 3 Default mode decisions of an organization run at once; another one gets 429 until one is done.

Under a key with retention "Not stored", the answer of a decision that went on after a 202 is held for you to fetch once, and deleted on that fetch or after one hour at the latest.

Retry

Retry these failures; none of them is charged.

• 503 free_capacity_busy: a request on free decisions found the model busy; requests paid from credit go first. Wait the Retry-After seconds and retry. No free decision is used.
• 429 rate_limited: over 600 requests a minute for a key, or 1,200 for the organization, or 3 Default mode decisions already running. Wait the Retry-After seconds, or until a running decision is done.
• 502, 503, 504: the model did not answer, is starting after being idle, or a run was interrupted by a restart. Wait the Retry-After seconds if there are any, else a minute, and retry.

Do not retry 400, 401, 402 or 409: the request itself needs fixing (Errors).

const headers = { Authorization: `Bearer ${process.env.EEVO_API_KEY}`, 'Content-Type': 'application/json' };
const sleep = seconds => new Promise(resolve => setTimeout(resolve, seconds * 1000));

// A decision still running (202): poll it until it is done.
async function waitFor(running) {
  for (;;) {
    await sleep(running.retry_after);
    const decision = await (await fetch(running.poll_url, { headers })).json();
    if (decision.status === 'ok') return decision.response;
    if (decision.status === 'error') throw new Error(decision.errorMessage);
  }
}

async function decide(body) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const response = await fetch('https://eevo.io/api/v1/decisions', { method: 'POST', headers, body: JSON.stringify(body) });
    if (response.status === 202) return await waitFor(await response.json());
    if (response.ok) return await response.json();
    const { error } = await response.json();
    const retryable =
      ['free_capacity_busy', 'rate_limited'].includes(error.type) || response.status >= 502;
    if (!retryable) throw new Error(`${error.type}: ${error.message}`);
    await sleep(Number(response.headers.get('Retry-After') ?? 30));
  }
  throw new Error('Eevo did not answer after 5 attempts');
}