Holdouts (beta)

Available for customers using Experimentation

Holdouts are available to customers using Experimentation.

This feature is in beta

To use this feature, pass in a header including the LD-API-Version key with value set to beta. Use this header with each call. To learn more, read Beta resources.

Resources that are in beta are still undergoing testing and development. They may change without notice, including becoming backwards incompatible.

Holdouts let you exclude a percentage of your audience from your Experimentation program. This enables you to see the overall effect of your experiments on your customer base, and helps determine how effective the experiments you're running are.

Using the holdouts API, you can create, delete, and manage holdouts. To learn more, read Holdouts.

Get all holdouts

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

query Parameters
limit
integer <int64>

The number of holdouts to return in the response. Defaults to 20

offset
integer <int64>

Where to start in the list. Use this with pagination. For example, an offset of 10 skips the first ten items and then returns the next items in the list, up to the query limit.

Responses
200

All Holdouts response

400

Invalid request

401

Invalid access token

403

Forbidden

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts
Request samples
Response samples
application/json
{
  • "items": [
    ],
  • "_links": {
    }
}

Create holdout

Create a new holdout in the specified project.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

Request Body schema: application/json
required
name
string

A human-friendly name for the holdout

key
string

A key that identifies the holdout

description
string

Description of the holdout

randomizationunit
string

The chosen randomization unit for the holdout base experiment

attributes
Array of strings

The attributes that the holdout iteration's results can be sliced by

holdoutamount
string

Audience allocation for the holdout

primarymetrickey
string

The key of the primary metric for this holdout

Array of objects (MetricInput)

Details on the metrics for this experiment

prerequisiteflagkey
string

The key of the flag that the holdout is dependent on

Responses
201

Holdout response

400

Invalid request

post/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts
Request samples
application/json
{
  • "name": "holdout-one-name",
  • "key": "holdout-key",
  • "description": "My holdout-one description",
  • "randomizationunit": "user",
  • "attributes": [
    ],
  • "holdoutamount": "10",
  • "primarymetrickey": "metric-key-123abc",
  • "metrics": [
    ],
  • "prerequisiteflagkey": "flag-key-123abc"
}
Response samples
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "experiments": [
    ]
}

Get Holdout by Id

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

holdoutId
required
string <string>

The holdout experiment ID

Responses
200

Holdout response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts/id/{holdoutId}
Request samples
Response samples
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "experiments": [
    ]
}

Get holdout

Get details about a holdout.

Expanding the holdout response

LaunchDarkly supports the following fields for expanding the "Get holdout" response. By default, these fields are not included in the response.

To expand the response, append the expand query parameter and add a comma-separated list with any of the following fields:

  • draftIteration includes the iteration which has not been started yet, if any, for this holdout.
  • previousIterations includes all iterations prior to the current iteration, for this holdout. By default only the current iteration is included in the response.
  • rel-draftIteration includes the iteration which has not been started yet, if any, for the experiments related to this holdout.
  • rel-metrics includes metrics for experiments related to this holdout.
  • rel-previousIterations includes all iterations prior to the current iteration, for the experiments related to this holdout.
  • rel-secondaryMetrics includes secondary metrics for experiments related to this holdout.
  • rel-treatments includes all treatment and parameter details for experiments related to this holdout.
  • secondaryMetrics includes secondary metrics for this holdout. By default only the primary metric is included in the response.
  • treatments includes all treatment and parameter details for this holdout. By default treatment data is not included in the response.

For example, expand=draftIteration,rel-draftIteration includes the draftIteration and rel-draftIteration fields in the response. If fields that you request with the expand query parameter are empty, they are not included in the response.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

holdoutKey
required
string <string>

The holdout experiment key

query Parameters
expand
string <string>

A comma-separated list of properties that can reveal additional information in the response. Supported fields are explained above. Holdout experiment expansion fields have no prefix. Related experiment expansion fields have rel- as a prefix.

Responses
200

HoldoutDetail response with full experiments

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts/{holdoutKey}
Request samples
Response samples
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "isDirty": true,
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "relatedExperiments": [
    ]
}

Patch holdout

Updates an existing holdout, and returns the updated holdout. Updating holdouts uses the semantic patch format.

To make a semantic patch request, you must append domain-model=launchdarkly.semanticpatch to your Content-Type header. To learn more, read Updates using semantic patch.

Instructions

Semantic patch requests support the following kind instructions for updating holdouts.

Click to expand instructions for updating holdouts

endHoldout

Ends a holdout.

Parameters

None.

Here's an example:

{
  "comment": "Optional comment describing why the holdout is ending",
  "instructions": [{
    "kind": "endHoldout"
  }]
}

removeExperiment

Removes an experiment from a holdout.

Parameters
  • value: The key of the experiment to remove

Here's an example:

{
  "comment": "Optional comment describing the change",
  "instructions": [{
    "kind": "removeExperiment",
    "value": "experiment-key"
  }]
}

updateDescription

Updates the description of the holdout.

Parameters
  • value: The new description.

Here's an example:

{
  "comment": "Optional comment describing the update",
  "instructions": [{
    "kind": "updateDescription",
    "value": "Updated holdout description"
  }]
}

updateName

Updates the name of the holdout.

Parameters
  • value: The new name.

Here's an example:

{
  "comment": "Optional comment describing the update",
  "instructions": [{
    "kind": "updateName",
    "value": "Updated holdout name"
  }]
}
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

holdoutKey
required
string <string>

The holdout key

Request Body schema: application/json
required
comment
string

Optional comment describing the update

required
Array of objects (Instructions)
Responses
200

Holdout response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

patch/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts/{holdoutKey}
Request samples
application/json
{
  • "comment": "Optional comment describing the update",
  • "instructions": [
    ]
}
Response samples
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "experiments": [
    ]
}