Ingest gateway

Your applications send errors, logs and metrics to CloseYourIt. There are two ways for that data to arrive: direct, which is what you get after the install, and through the gateway, which you turn on with one command.

The two paths

Direct (default)

your application  →  Caddy  →  app  →  worker  →  database

1. Your application sends a request to https://<your-domain>/api/….
2. Caddy, the web server, hands it to the app.
3. The app checks the token and answers 202.
4. The worker saves the data a moment later.

If the app is restarting or too busy at step 2, the request fails. The data is lost unless the sender tries again.

Through the gateway

your application  →  Caddy  →  gateway  →  app  →  worker  →  database
                                  ↓ app not answering
                                queue  →  app, later

1. Your application sends the same request to the same address.
2. Caddy hands incoming data to the gateway. Everything else under /api/ — reading data back, the Automator — still goes straight to the app.
3. The gateway forwards the request to the app and waits a few seconds.
4. If the app answers, even with an error such as 401 or 422, your application gets that answer. Nothing is queued.
5. If the app does not answer, is overloaded (429) or fails (5xx), the gateway stores the request, encrypted, in a queue (NATS JetStream) and answers 202 at once.
6. The gateway keeps trying to deliver what is in the queue until the app takes it.

Side by side

When you need it

• Many applications send data, or one sends a lot.
• You update often and do not want to lose data during restarts.

For a small team with a few projects you can do without it.

Move from direct to the gateway

Nothing changes in your applications: same address, same tokens. No data is moved, and you can go back at any time.

1. Check there is room: df -h /opt/closeyourit should show at least 6 GB free.
2. Turn it on:

   closeyourit enable ingest
   

   Copy
3. Check that gateway and nats are running next to app, worker, postgres and caddy:

   closeyourit status
   

   Copy
4. Send one request by hand (see the HTTP API) and check it shows up in CloseYourIt.

The switch takes a few seconds. A request that arrives in that moment may fail. Caddy moves to the gateway only once the gateway has stayed up: if it does not, the command says so, turns it off again and data keeps going straight to the app.

How to tell a request was queued

When the gateway keeps a request instead of delivering it at once, it answers 202 with the header X-CloseYourIt-Queued: true (the Sentry-compatible addresses answer 200, as those SDKs expect). That answer means "safe in the queue", not "saved in CloseYourIt yet".

What the gateway accepts

Only the addresses that receive data: events, metrics, logs, page views, replays, releases, cron check-ins, server samples and the two Sentry-compatible addresses. See the HTTP API.

Everything else under /api/ — reading data back, the Automator picking up work — keeps going straight to the app, gateway on or off.

When the queue is full, new requests are refused rather than dropping the ones already waiting.

A request that keeps failing is retried with growing pauses. After 20 attempts it is moved to the "could not be delivered" store and no longer retried.

Reading data or the Automator stopped working

Early installs sent all of /api/ to the gateway, which refuses what it does not know: reading data back answers 405 and the Automator gets 404. Run closeyourit update: it also brings the web server file of the new version.

The encryption key

Requests in the queue are encrypted with the key in /opt/closeyourit/ingest/envelope_key. The queue password is in /opt/closeyourit/ingest/nats_password. Both are generated at install time. If you replace the key while requests are waiting, those requests can no longer be read.

Move back to direct

closeyourit disable ingest

Requests go back to the direct path at once. The gateway then gets 30 seconds to deliver what is still waiting, and stops. Anything not delivered by then stays in the queue on disk and is delivered the next time you turn the gateway on.

Do it while the app is healthy, so the queue is already empty.

OpenTelemetry is configured separately

The optional OTLP receiver has separate listeners, authentication and request limits. Enabling this native ingest gateway does not expose or activate those listeners automatically.