Fireworks AI's 368-line llms.txt shows what thorough AI preparation looks like

# Fireworks AI Docs

## Docs

- [Exporting Billing Metrics](https://docs.fireworks.ai/accounts/exporting-billing-metrics.md): Export billing and usage metrics for all Fireworks services
- [Service Accounts](https://docs.fireworks.ai/accounts/service-accounts.md): How to manage and use service accounts in Fireworks
- [Custom SSO](https://docs.fireworks.ai/accounts/sso.md): Set up custom Single Sign-On (SSO) authentication for Fireworks AI
- [Managing users](https://docs.fireworks.ai/accounts/users.md): Add, delete, and manage roles for users in your Fireworks account
- [Streaming Transcription](https://docs.fireworks.ai/api-reference/audio-streaming-transcriptions.md)
- [Transcribe audio](https://docs.fireworks.ai/api-reference/audio-transcriptions.md)
- [Translate audio](https://docs.fireworks.ai/api-reference/audio-translations.md)
- [Cancel Reinforcement Fine-tuning Job](https://docs.fireworks.ai/api-reference/cancel-reinforcement-fine-tuning-job.md)
- [Create API Key](https://docs.fireworks.ai/api-reference/create-api-key.md)
- [Create Batch Inference Job](https://docs.fireworks.ai/api-reference/create-batch-inference-job.md)
- [Create Batch Request](https://docs.fireworks.ai/api-reference/create-batch-request.md)
- [Create Dataset](https://docs.fireworks.ai/api-reference/create-dataset.md)
- [Load LoRA](https://docs.fireworks.ai/api-reference/create-deployed-model.md)
- [Create Deployment](https://docs.fireworks.ai/api-reference/create-deployment.md)
- [null](https://docs.fireworks.ai/api-reference/create-dpo-job.md)
- [Create Evaluation Job](https://docs.fireworks.ai/api-reference/create-evaluation-job.md)
- [Create Evaluator](https://docs.fireworks.ai/api-reference/create-evaluator.md): Creates a custom evaluator for scoring model outputs. Evaluators use the
[Eval Protocol](https://evalprotocol.io) to define test cases, run model
inference, and score responses. They are used with evaluation jobs and
Reinforcement Fine-Tuning (RFT).

## Source Code Requirements

Your project should contain:
- `requirements.txt` - Python dependencies for your evaluator
- `test_*.py` - Pytest test file(s) with
  [`@evaluation_test`](https://evalprotocol.io/reference/evaluation-test)
  decorated functions
- Any additional code/modules your evaluator needs

## Workflow

**Recommended:** Use the [`ep upload`](https://evalprotocol.io/reference/cli#ep-upload)
CLI command to handle all these steps automatically.

If using the API directly:

1. Call this endpoint to create the evaluator resource
2. Package your source directory as a `.tar.gz` (respecting `.gitignore`)
3. Call [Get Evaluator Upload Endpoint](/api-reference/get-evaluator-upload-endpoint) to get a signed upload URL
4. `PUT` the tar.gz file to the signed URL
5. Call [Validate Evaluator Upload](/api-reference/validate-evaluator-upload) to trigger server-side validation
6. Poll [Get Evaluator](/api-reference/get-evaluator) until ready

Once active, reference the evaluator in [Create Evaluation Job](/api-reference/create-evaluation-job) or [Create Reinforcement Fine-tuning Job](/api-reference/create-reinforcement-fine-tuning-job).
- [Create Model](https://docs.fireworks.ai/api-reference/create-model.md)
- [Create Reinforcement Fine-tuning Job](https://docs.fireworks.ai/api-reference/create-reinforcement-fine-tuning-job.md)
- [Create Reinforcement Fine-tuning Step](https://docs.fireworks.ai/api-reference/create-reinforcement-fine-tuning-step.md)
- [null](https://docs.fireworks.ai/api-reference/create-secret.md)
- [Create Supervised Fine-tuning Job](https://docs.fireworks.ai/api-reference/create-supervised-fine-tuning-job.md)
- [Create User](https://docs.fireworks.ai/api-reference/create-user.md)
- [Create embeddings](https://docs.fireworks.ai/api-reference/creates-an-embedding-vector-representing-the-input-text.md)
- [Delete API Key](https://docs.fireworks.ai/api-reference/delete-api-key.md)
- [Delete Batch Inference Job](https://docs.fireworks.ai/api-reference/delete-batch-inference-job.md)
- [Delete Dataset](https://docs.fireworks.ai/api-reference/delete-dataset.md)
- [Unload LoRA](https://docs.fireworks.ai/api-reference/delete-deployed-model.md)
- [Delete Deployment](https://docs.fireworks.ai/api-reference/delete-deployment.md)
- [null](https://docs.fireworks.ai/api-reference/delete-dpo-job.md)
- [Delete Evaluation Job](https://docs.fireworks.ai/api-reference/delete-evaluation-job.md)
- [Delete Evaluator](https://docs.fireworks.ai/api-reference/delete-evaluator.md): Deletes an evaluator and its associated versions and build artifacts.
- [Delete Model](https://docs.fireworks.ai/api-reference/delete-model.md)
- [Delete Reinforcement Fine-tuning Job](https://docs.fireworks.ai/api-reference/delete-reinforcement-fine-tuning-job.md)
- [Delete Reinforcement Fine-tuning Step](https://docs.fireworks.ai/api-reference/delete-reinforcement-fine-tuning-step.md)
- [Delete Response](https://docs.fireworks.ai/api-reference/delete-response.md): Deletes a model response by its ID. Once deleted, the response data will be gone immediately and permanently.

The response cannot be recovered and any conversations that reference this response ID will no longer be able to access it.
- [null](https://docs.fireworks.ai/api-reference/delete-secret.md)
- [Delete Supervised Fine-tuning Job](https://docs.fireworks.ai/api-reference/delete-supervised-fine-tuning-job.md)
- [Execute one training step for keep-alive Reinforcement Fine-tuning Step](https://docs.fireworks.ai/api-reference/execute-reinforcement-fine-tuning-step.md)
- [Generate an image with FLUX.1 [schnell] FP8](https://docs.fireworks.ai/api-reference/generate-a-new-image-from-a-text-prompt.md)
- [Generate or edit an image with FLUX.1 Kontext](https://docs.fireworks.ai/api-reference/generate-or-edit-image-using-flux-kontext.md)
- [Get Account](https://docs.fireworks.ai/api-reference/get-account.md)
- [Get Batch Inference Job](https://docs.fireworks.ai/api-reference/get-batch-inference-job.md)
- [Check Batch Status](https://docs.fireworks.ai/api-reference/get-batch-status.md)
- [Get Dataset](https://docs.fireworks.ai/api-reference/get-dataset.md)
- [Get Dataset Download Endpoint](https://docs.fireworks.ai/api-reference/get-dataset-download-endpoint.md)
- [Get Dataset Upload Endpoint](https://docs.fireworks.ai/api-reference/get-dataset-upload-endpoint.md)
- [Get LoRA](https://docs.fireworks.ai/api-reference/get-deployed-model.md)
- [Get Deployment](https://docs.fireworks.ai/api-reference/get-deployment.md)
- [Get Deployment Shape](https://docs.fireworks.ai/api-reference/get-deployment-shape.md)
- [Get Deployment Shape Version](https://docs.fireworks.ai/api-reference/get-deployment-shape-version.md)
- [null](https://docs.fireworks.ai/api-reference/get-dpo-job.md)
- [null](https://docs.fireworks.ai/api-reference/get-dpo-job-metrics-file-endpoint.md)
- [Get Evaluation Job](https://docs.fireworks.ai/api-reference/get-evaluation-job.md)
- [Get Evaluation Job execution logs (stream log endpoint + tracing IDs).](https://docs.fireworks.ai/api-reference/get-evaluation-job-log-endpoint.md)
- [Get Evaluator](https://docs.fireworks.ai/api-reference/get-evaluator.md): Retrieves an evaluator by name. Use this to monitor build progress after
creation (**step 6** in the [Create Evaluator](/api-reference/create-evaluator) workflow).

Possible states:

- `BUILDING` - Environment is being prepared
- `ACTIVE` - Evaluator is ready to use
- `BUILD_FAILED` - Check build logs via [Get Evaluator Build Log Endpoint](/api-reference/get-evaluator-build-log-endpoint)
- [Get Evaluator Build Log Endpoint](https://docs.fireworks.ai/api-reference/get-evaluator-build-log-endpoint.md): Returns a signed URL to download the evaluator's build logs. Useful for
debugging `BUILD_FAILED` state.
- [Get Evaluator Source Code Endpoint](https://docs.fireworks.ai/api-reference/get-evaluator-source-code-endpoint.md): Returns a signed URL to download the evaluator's source code archive.