Holdouts (beta)

Available for customers using Experimentation

Holdouts are available to customers using Experimentation.

This feature is in beta

To use this feature, pass in a header including the LD-API-Version key with value set to beta. Use this header with each call. To learn more, read Beta resources.

Resources that are in beta are still undergoing testing and development. They may change without notice, including becoming backwards incompatible.

Holdouts let you exclude a percentage of your audience from your Experimentation program. This enables you to see the overall effect of your experiments on your customer base, and helps determine how effective the experiments you're running are.

Using the holdouts API, you can create, delete, and manage holdouts. To learn more, read Holdouts.

Get all holdouts

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

query Parameters
limit
integer <int64>

The number of holdouts to return in the response. Defaults to 20

offset
integer <int64>

Where to start in the list. Use this with pagination. For example, an offset of 10 skips the first ten items and then returns the next items in the list, up to the query limit.

Responses
200

All Holdouts response

400

Invalid request

401

Invalid access token

403

Forbidden

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts
Request samples 
Response samples 
application/json
{
  • "items": [
    ],
  • "_links": {
    },
  • "total_count": 0
}
Create holdout
Create a new holdout in the specified project.


Request 
path Parameters
projectKey
required
string <string> 
The project key


 
environmentKey
required
string <string> 
The environment key


 
Request Body schema: application/json
required
name
string
A human-friendly name for the holdout


 
key
string
A key that identifies the holdout


 
description
string
Description of the holdout


 
randomizationunit
string
The chosen randomization unit for the holdout base experiment


 
attributes
Array of strings
The attributes that the holdout iteration's results can be sliced by


 
holdoutamount
string
Audience allocation for the holdout


 
primarymetrickey
string
The key of the primary metric for this holdout


 
Array of objects (MetricInput) 
Details on the metrics for this experiment


 
prerequisiteflagkey
string
The key of the flag that the holdout is dependent on


 
Responses 
201
Holdout response


400
Invalid request


post/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts
Request samples 
application/json
{
  • "name": "holdout-one-name",
  • "key": "holdout-key",
  • "description": "My holdout-one description",
  • "randomizationunit": "user",
  • "attributes": [
    ],
  • "holdoutamount": "10",
  • "primarymetrickey": "metric-key-123abc",
  • "metrics": [
    ],
  • "prerequisiteflagkey": "flag-key-123abc"
}
Response samples 
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "experiments": [
    ]
}
Get Holdout by Id
Request 
path Parameters
projectKey
required
string <string> 
The project key


 
environmentKey
required
string <string> 
The environment key


 
holdoutId
required
string <string> 
The holdout experiment ID


 
Responses 
200
Holdout response


400
Invalid request


401
Invalid access token


403
Forbidden


404
Invalid resource identifier


429
Rate limited


get/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts/id/{holdoutId}
Request samples 
Response samples 
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "experiments": [
    ]
}
Get holdout
Get details about a holdout.


Expanding the holdout response


LaunchDarkly supports the following fields for expanding the "Get holdout" response. By default, these fields are not included in the response.


To expand the response, append the expand query parameter and add a comma-separated list with any of the following fields:


    

  • draftIteration includes the iteration which has not been started yet, if any, for this holdout.
    • 

  • previousIterations includes all iterations prior to the current iteration, for this holdout. By default only the current iteration is included in the response.
    • 

  • rel-draftIteration includes the iteration which has not been started yet, if any, for the experiments related to this holdout.
    • 

  • rel-metrics includes metrics for experiments related to this holdout.
    • 

  • rel-previousIterations includes all iterations prior to the current iteration, for the experiments related to this holdout.
    • 

  • rel-secondaryMetrics includes secondary metrics for experiments related to this holdout.
    • 

  • rel-treatments includes all treatment and parameter details for experiments related to this holdout.
    • 

  • secondaryMetrics includes secondary metrics for this holdout. By default only the primary metric is included in the response.
    • 

  • treatments includes all treatment and parameter details for this holdout. By default treatment data is not included in the response.
    • 



For example, expand=draftIteration,rel-draftIteration includes the draftIteration and rel-draftIteration fields in the response. If fields that you request with the expand query parameter are empty, they are not included in the response.


Request 
path Parameters
projectKey
required
string <string> 
The project key


 
environmentKey
required
string <string> 
The environment key


 
holdoutKey
required
string <string> 
The holdout experiment key


 
query Parameters
expand
string <string> 
A comma-separated list of properties that can reveal additional information in the response. Supported fields are explained above. Holdout experiment expansion fields have no prefix. Related experiment expansion fields have rel- as a prefix.


 
Responses 
200
HoldoutDetail response with full experiments


400
Invalid request


401
Invalid access token


403
Forbidden


404
Invalid resource identifier


429
Rate limited


get/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts/{holdoutKey}
Request samples 
Response samples 
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "isDirty": true,
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "relatedExperiments": [
    ]
}
Patch holdout
Updates an existing holdout, and returns the updated holdout. Updating holdouts uses the semantic patch format.


To make a semantic patch request, you must append domain-model=launchdarkly.semanticpatch to your Content-Type header. To learn more, read Updates using semantic patch.


Instructions


Semantic patch requests support the following kind instructions for updating holdouts.



Click to expand instructions for updating holdouts


endHoldout


Ends a holdout.


Parameters


None.


Here's an example:


{
  "comment": "Optional comment describing why the holdout is ending",
  "instructions": [{
    "kind": "endHoldout"
  }]
}



removeExperiment


Removes an experiment from a holdout.


Parameters


    

  • value: The key of the experiment to remove
    • 



Here's an example:


{
  "comment": "Optional comment describing the change",
  "instructions": [{
    "kind": "removeExperiment",
    "value": "experiment-key"
  }]
}



updateDescription


Updates the description of the holdout.


Parameters


    

  • value: The new description.
    • 



Here's an example:


{
  "comment": "Optional comment describing the update",
  "instructions": [{
    "kind": "updateDescription",
    "value": "Updated holdout description"
  }]
}



updateName


Updates the name of the holdout.


Parameters


    

  • value: The new name.
    • 



Here's an example:


{
  "comment": "Optional comment describing the update",
  "instructions": [{
    "kind": "updateName",
    "value": "Updated holdout name"
  }]
}





Request 
path Parameters
projectKey
required
string <string> 
The project key


 
environmentKey
required
string <string> 
The environment key


 
holdoutKey
required
string <string> 
The holdout key


 
Request Body schema: application/json
required
comment
string
Optional comment describing the update


 
required
Array of objects (Instructions) 
 
Responses 
200
Holdout response


400
Invalid request


401
Invalid access token


403
Forbidden


404
Invalid resource identifier


429
Rate limited


patch/api/v2/projects/{projectKey}/environments/{environmentKey}/holdouts/{holdoutKey}
Request samples 
application/json
{
  • "comment": "Optional comment describing the update",
  • "instructions": [
    ]
}
Response samples 
application/json
{
  • "_id": "string",
  • "status": "created",
  • "description": "string",
  • "holdoutAmount": "string",
  • "createdAt": 0,
  • "updatedAt": 0,
  • "baseExperiment": {
    },
  • "experiments": [
    ]
}
➔ Next to Insights charts (beta)