← Customize

Intel® AI for Enterprise RAG supports multiple object store backends for storing ingested documents. The choice of backend affects where files are stored and how they are accessed by the EDP (Enhanced Data Preparation) service.

This matters when your documents already live somewhere: pointing EDP at an existing bucket or array means you ingest from the system of record instead of copying corpora into the cluster, and it decides who is responsible for retention and backup of the source files.

Storage backends

Set edp_storage_type in env/<name>/config.erag.yaml:

The backend determines which credentials and endpoints are rendered into the EDP Helm values.

SeaweedFS (default)

edp_storage_type: "seaweedfs"

Deploys SeaweedFS in-cluster. No additional configuration required. Files are accessible at https://seaweedfs.<base_domain_name> (in subdomain routing mode) or https://<base_domain_name>/seaweedfs (in path routing mode).

Storage capacity: Default persistent volume size is 5GB. To increase:

1. Edit deployment/components/edp/values.yaml
2. Under the persistence section, set size: 100Gi (or your desired size)
3. Redeploy

Note: Vector store storage should be larger than file storage because it contains both extracted text and vector embeddings for that data.

External S3

edp_storage_type: "s3"
edp_s3_region: "us-east-1"
edp_s3_access_key_id: "<your-access-key>"
edp_s3_secret_access_key: "<your-secret-key>"
edp_s3_sqs_event_queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue"
edp_s3_bucket_name_regex_filter: ""  # optional: restrict which buckets EDP tracks

Requires SQS event notifications configured on the S3 bucket for automatic ingestion.

S3-compatible (without SQS)

edp_storage_type: "s3compatible"
edp_s3_compatible_region: "us-east-1"
edp_s3_compatible_access_key_id: "<your-access-key>"
edp_s3_compatible_secret_access_key: "<your-secret-key>"
edp_s3_compatible_internal_url: "https://s3.example.com"
edp_s3_compatible_external_url: "https://s3.example.com"
edp_s3_compatible_bucket_name_regex_filter: ""  # optional

S3-compatible storage without SQS event notifications requires scheduled synchronization to detect new files. See Scheduled synchronization.

NetApp ONTAP S3 integration

Intel AI for Enterprise RAG can use a NetApp ONTAP object-store server as the document store instead of the in-cluster SeaweedFS.

Note: Persistent volume storage is a separate decision made by Enterprise AI Solutions (storage_backend: netapp-trident in env/<name>/global_config.yaml). This section covers only the object store used by EDP.

Why a reverse proxy is needed

EDP hands the browser a presigned SigV4 URL, and the browser PUTs the file directly to it. The ONTAP object-store data LIF sits on the storage network where browsers cannot reach it. Publishing the endpoint through the platform Envoy Gateway means EDP needs two endpoints for the same bucket:

• Internal: https://<ontap_s3_data_lif>:<port> (for EDP pods)
• External: https://s3.<base_domain_name> (for browsers)

Getting these out of step is the most common way to break ingestion. The role derives both automatically.

The reverse proxy objects (Service/EndpointSlice or an Envoy Gateway Backend, an HTTPRoute, and a route-scoped SecurityPolicy) are created in the edp namespace by roles/app_edp/tasks/ontap_s3_proxy.yaml.

Configuration

Set the following in the Enterprise AI Solutions env/<name>/global_config.yaml:

ontap_s3_data_lif: "10.0.0.102"     # Object-store data LIF, NOT the NFS LIF
ontap_s3_port: "443"                # default: 443
ontap_s3_tls: true                  # default: true

Set the following in env/<name>/config.erag.yaml:

edp_s3_compatible_access_key_id: "<access-key>"
edp_s3_compatible_secret_access_key: "<secret-key>"

edp_storage_type is automatically derived as s3compatible when ontap_s3_data_lif is set on a netapp-trident cluster. You can override it explicitly, but the role will reject a value that contradicts an active ONTAP S3 proxy.

Deploy or redeploy:

./es_auto_installer.sh install erag --env <name>

The installer prints the derived values at install time so a deployment can be read back from the log.

What is derived

The following values are automatically computed by roles/app_edp/tasks/ontap_s3_resolve.yaml:

edp_rbac_enabled must stay false because RBAC mode signs object-store requests as the logged-in user through MinIO's WebIdentityProvider, which needs STS (Security Token Service). ONTAP S3 has no STS endpoint. false is the default, and the role asserts it rather than silently overriding.

Upstream TLS modes

edp_ontap_s3_tls_mode selects how the gateway talks to the array. All three modes terminate client TLS at the gateway with the platform certificate; the difference is the second hop:

To use ca mode:

1. Place the CA certificate in PEM format on the installer host

2. Set in env/<name>/config.erag.yaml:

   yaml Copy

   edp_ontap_s3_tls_mode: "ca"
   ontap_s3_ca_cert_file: "/path/to/ca.pem"
   edp_ontap_s3_sni_hostname: "svm.example.com"

3. Redeploy

Optional settings

Limitations

• Object keys containing //, /../, or %2F cannot be served through the gateway. SigV4 signs the path verbatim and Envoy normalizes it, so the array recomputes a different signature and answers SignatureDoesNotMatch. EDP refuses to presign such names, so the UI path is unaffected, but keys created by other tools are not reachable.
• routing_mode: path gets no dedicated listener because the S3 API then shares the apex hostname with the UI. Use subdomain mode.
• No ONTAP-side provisioning. The object-store server, the bucket, and the S3 user must exist, and the access keys are inputs. Nothing here creates them.

For the complete end-to-end flow (from configure through init erag to install erag), array-side prerequisites, and the ontap_* values themselves, see docs/NETAPP_ONTAP.md in Enterprise AI Solutions repository.

Scheduled synchronization

S3-compatible storage without SQS event notifications requires scheduled synchronization to detect new files. Enable it in env/<name>/config.erag.yaml:

edp_scheduled_sync_enabled: true
edp_scheduled_sync_period_seconds: "60"

The scheduled task polls buckets for changes at the specified interval and ingests any new or updated files.

Related docs