Available for Pro and Enterprise plans
Experimentation is available to customers on a Pro or Enterprise plan. To learn more, read about our pricing. To add Experimentation to your plan, contact Sales.
This feature is in beta
To use this feature, pass in a header including the
LD-API-Version
key with value set tobeta
. Use this header with each call. To learn more, read Beta resources.Resources that are in beta are still undergoing testing and development. They may change without notice, including becoming backwards incompatible.
Experimentation lets you validate the impact of features you roll out to your app or infrastructure. You can measure things like page views, clicks, load time, infrastructure costs, and more. By connecting metrics you create to flags in your LaunchDarkly environment, you can measure the changes in your customers' behavior based on what flags they evaluate. You can run experiments on boolean, string, and number flags, but not JSON flags. To learn more, read About Experimentation.
You can manage experiments by using the dedicated experiment endpoints described below.
Several of the endpoints require a treatment ID or a flag rule ID. Treatment IDs are returned as part of the Get experiment results response. They are the treatmentId
of each element in the treatmentResults
array. Winning treatment IDs are also returned as part of the Get experiment response. They are the winningTreatmentId
in the currentIteration
, the winningTreatmentId
in the draftIteration
, and the winningTreatmentId
in each element of the previousIterations
array. In the flags object, the rule ID is the ID of the variation or rollout of the flag. Each flag variation ID is returned as part of the Get feature flag response. It is the _id
field in each element of the variations
array.
Customers on older contracts may have experiments created under an older data model. These experiments are listed in the LaunchDarkly user interface as "Legacy experiments." You can manage them by making PATCH requests to the flags API, and you can analyze them by making a GET request to the deprecated Get legacy experiment results endpoint. To learn more about legacy experiments, read Legacy experiments.
To manage legacy experiments, make the following PATCH requests to the flags API:
To add an experiment to a flag, add a new object to the /experiments/items/
property.
The object must contain:
PATCH /api/v2/flags/{projKey}/{flagKey}
[
{
"op": "add",
"path": "/experiments/items/0",
"value": {
"metricKey": "metric-key-123abc",
"environments": []
}
}
]
To remove an experiment, remove the corresponding item from the experiments or items property.
PATCH /api/v2/flags/{projKey}/{flagKey}
[
{
"op": "remove",
"path": "/experiments/items/0"
}
]
To start recording data for an experiment in an environment, add the environment key to the list of environments for that experiment.
PATCH /api/v2/flags/{projKey}/{flagKey}
[
{
"op": "add",
"path": "/experiments/items/0/environments/0",
"value": "production"
}
]
To stop collecting data for an experiment in a specific environment, remove the environment from the list that experiment uses.
PATCH /api/v2/flags/{projKey}/{flagKey}
[
{
"op": "remove",
"path": "/experiments/items/0/environments/0"
}
]
To update the baseline for all experiments on a flag across all environments in the project, replace the baselineIdx
property with the index of the new baseline variation.
PATCH /api/v2/flags/{projKey}/{flagKey}
[
{
"op": "replace",
"path": "/experiments/baselineIdx",
"value": 1
}
]
Get detailed experiment result data for legacy experiments.
Experiment results response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Rate limited
{- "_links": {
- "property1": {
- "href": "string",
- "type": "string"
}, - "property2": {
- "href": "string",
- "type": "string"
}
}, - "metadata": [
- {
- "key": null
}
], - "totals": [
- {
- "cumulativeValue": 0,
- "cumulativeCount": 0,
- "cumulativeImpressionCount": 0,
- "cumulativeConversionRate": 0,
- "cumulativeConfidenceInterval": {
- "upper": 0,
- "lower": 0
}, - "pValue": 0,
- "improvement": 0,
- "minSampleSize": 0
}
], - "series": [
- {
- "Time": 0,
- "VariationData": [
- {
- "value": 0,
- "count": 0,
- "cumulativeValue": 0,
- "cumulativeCount": 0,
- "conversionRate": 0,
- "cumulativeConversionRate": 0,
- "confidenceInterval": {
- "upper": 0,
- "lower": 0
}, - "cumulativeConfidenceInterval": {
- "upper": 0,
- "lower": 0
}
}
]
}
], - "stats": {
- "pValue": 0,
- "chi2": 0,
- "winningVariationIdx": 0,
- "minSampleSizeMet": true
}, - "granularity": "string",
- "metricSeen": {
- "ever": true,
- "timestamp": 1657129307
}
}
Reset all experiment results by deleting all existing data for an experiment.
Experiment results reset successfully
Invalid access token
Forbidden
Invalid resource identifier
Rate limited
{- "code": "unauthorized",
- "message": "Invalid access token"
}
Get details about all experiments in an environment.
LaunchDarkly supports the filter
query param for filtering, with the following fields:
flagKey
filters for only experiments that use the flag with the given key.metricKey
filters for only experiments that use the metric with the given key.status
filters for only experiments with an iteration with the given status. An iteration can have the status not_started
, running
or stopped
.For example, filter=flagKey:my-flag,status:running,metricKey:page-load-ms
filters for experiments for the given flag key and the given metric key which have a currently running iteration.
LaunchDarkly supports four fields for expanding the "Get experiments" response. By default, these fields are not included in the response.
To expand the response, append the expand
query parameter and add a comma-separated list with any of the following fields:
previousIterations
includes all iterations prior to the current iteration. By default only the current iteration is included in the response.draftIteration
includes the iteration which has not been started yet, if any.secondaryMetrics
includes secondary metrics. By default only the primary metric is included in the response.treatments
includes all treatment and parameter details. By default treatment data is not included in the response.For example, expand=draftIteration,treatments
includes the draftIteration
and treatments
fields in the response. If fields that you request with the expand
query parameter are empty, they are not included in the response.
Experiment collection response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Method not allowed
Rate limited
{- "items": [
- {
- "_id": "12ab3c45de678910fgh12345",
- "key": "experiment-key-123abc",
- "name": "Example experiment",
- "description": "An example experiment, used in testing",
- "_maintainerId": "12ab3c45de678910fgh12345",
- "_creationDate": 0,
- "archivedDate": 0,
- "_links": {
- "parent": {
- "href": "/api/v2/projects/my-project/environments/my-environment",
- "type": "application/json"
}, - "self": {
- "href": "/api/v2/projects/my-project/environments/my-environment/experiments/my-experiment",
- "type": "application/json"
}
}, - "currentIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": null,
- "flagKey": null
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "draftIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": null,
- "flagKey": null
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "previousIterations": [
- {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- null
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}
]
}
], - "total_count": 0,
- "_links": {
- "property1": {
- "href": "string",
- "type": "string"
}, - "property2": {
- "href": "string",
- "type": "string"
}
}
}
Create an experiment.
To run this experiment, you'll need to create an iteration and then update the experiment with the startIteration
instruction.
To learn more, read Creating experiments.
Experiment response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Rate limited
{- "name": "Example experiment",
- "description": "An example experiment, used in testing",
- "maintainerId": "12ab3c45de678910fgh12345",
- "key": "experiment-key-123abc",
- "iteration": {
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "canReshuffleTraffic": true,
- "metrics": [
- {
- "key": "metric-key-123abc",
- "primary": true
}
], - "treatments": [
- {
- "name": "Treatment 1",
- "baseline": true,
- "allocationPercent": "10",
- "parameters": [
- {
- "flagKey": "example-flag-for-experiment",
- "variationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05"
}
]
}
], - "flags": {
- "property1": {
- "ruleId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",
- "flagConfigVersion": 12
}, - "property2": {
- "ruleId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",
- "flagConfigVersion": 12
}
}, - "randomizationUnit": "user"
}
}
{- "_id": "12ab3c45de678910fgh12345",
- "key": "experiment-key-123abc",
- "name": "Example experiment",
- "description": "An example experiment, used in testing",
- "_maintainerId": "12ab3c45de678910fgh12345",
- "_creationDate": 0,
- "archivedDate": 0,
- "_links": {
- "parent": {
- "href": "/api/v2/projects/my-project/environments/my-environment",
- "type": "application/json"
}, - "self": {
- "href": "/api/v2/projects/my-project/environments/my-environment/experiments/my-experiment",
- "type": "application/json"
}
}, - "currentIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "draftIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "previousIterations": [
- {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}
]
}
Get details about an experiment.
LaunchDarkly supports four fields for expanding the "Get experiment" response. By default, these fields are not included in the response.
To expand the response, append the expand
query parameter and add a comma-separated list with any of the following fields:
previousIterations
includes all iterations prior to the current iteration. By default only the current iteration is included in the response.draftIteration
includes the iteration which has not been started yet, if any.secondaryMetrics
includes secondary metrics. By default only the primary metric is included in the response.treatments
includes all treatment and parameter details. By default treatment data is not included in the response.For example, expand=draftIteration,treatments
includes the draftIteration
and treatments
fields in the response. If fields that you request with the expand
query parameter are empty, they are not included in the response.
Experiment response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Method not allowed
Rate limited
{- "_id": "12ab3c45de678910fgh12345",
- "key": "experiment-key-123abc",
- "name": "Example experiment",
- "description": "An example experiment, used in testing",
- "_maintainerId": "12ab3c45de678910fgh12345",
- "_creationDate": 0,
- "archivedDate": 0,
- "_links": {
- "parent": {
- "href": "/api/v2/projects/my-project/environments/my-environment",
- "type": "application/json"
}, - "self": {
- "href": "/api/v2/projects/my-project/environments/my-environment/experiments/my-experiment",
- "type": "application/json"
}
}, - "currentIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "draftIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "previousIterations": [
- {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}
]
}
Update an experiment. Updating an experiment uses the semantic patch format.
To make a semantic patch request, you must append domain-model=launchdarkly.semanticpatch
to your Content-Type
header. To learn more, read Updates using semantic patch.
Semantic patch requests support the following kind
instructions for updating experiments.
Updates the experiment name.
value
: The new name.Updates the experiment description.
value
: The new description.Starts a new iteration for this experiment. You must create a new iteration before calling this instruction.
An iteration may not be started until it meets the following criteria:
randomizationUnit
is settreatments
has a non-zero allocationPercent
changeJustification
: The reason for starting a new iteration. Required when you call startIteration
on an already running experiment, otherwise optional.Stops the current iteration for this experiment.
winningTreatmentId
: The ID of the winning treatmentwinningReason
: The reason for the winnerArchives this experiment. Archived experiments are hidden by default in the LaunchDarkly user interface. You cannot start new iterations for archived experiments.
Restores an archived experiment. After restoring an experiment, you can start new iterations for it again.
Experiment response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Rate limited
{- "comment": "Example comment describing the update",
- "instructions": [
- {
- "kind": "updateName",
- "value": "Updated experiment name"
}
]
}
{- "_id": "12ab3c45de678910fgh12345",
- "key": "experiment-key-123abc",
- "name": "Example experiment",
- "description": "An example experiment, used in testing",
- "_maintainerId": "12ab3c45de678910fgh12345",
- "_creationDate": 0,
- "archivedDate": 0,
- "_links": {
- "parent": {
- "href": "/api/v2/projects/my-project/environments/my-environment",
- "type": "application/json"
}, - "self": {
- "href": "/api/v2/projects/my-project/environments/my-environment/experiments/my-experiment",
- "type": "application/json"
}
}, - "currentIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "draftIteration": {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}, - "previousIterations": [
- {
- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}
]
}
Create an experiment iteration.
Experiment iterations let you record experiments in individual blocks of time. Initially, iterations are created with a status of not_started
and appear in the draftIteration
field of an experiment. To start or stop an iteration, update the experiment with the startIteration
or stopIteration
instruction.
To learn more, read Starting experiment iterations.
Iteration response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Rate limited
{- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "canReshuffleTraffic": true,
- "metrics": [
- {
- "key": "metric-key-123abc",
- "primary": true
}
], - "treatments": [
- {
- "name": "Treatment 1",
- "baseline": true,
- "allocationPercent": "10",
- "parameters": [
- {
- "flagKey": "example-flag-for-experiment",
- "variationId": "e432f62b-55f6-49dd-a02f-eb24acf39d05"
}
]
}
], - "flags": {
- "property1": {
- "ruleId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",
- "flagConfigVersion": 12
}, - "property2": {
- "ruleId": "e432f62b-55f6-49dd-a02f-eb24acf39d05",
- "flagConfigVersion": 12
}
}, - "randomizationUnit": "user"
}
{- "_id": "12ab3c45de678910fgh12345",
- "hypothesis": "Example hypothesis, the new button placement will increase conversion",
- "status": "running",
- "createdAt": 0,
- "startedAt": 0,
- "endedAt": 0,
- "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
- "canReshuffleTraffic": true,
- "flags": {
- "property1": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}, - "property2": {
- "targetingRule": "fallthrough",
- "targetingRuleDescription": "Customers who live in Canada",
- "targetingRuleClauses": [
- null
], - "flagConfigVersion": 12,
- "_links": {
- "self": {
- "href": "/api/v2/flags/my-project/my-flag",
- "type": "application/json"
}
}
}
}, - "primaryMetric": {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}, - "randomizationUnit": "user",
- "treatments": [
- {
- "_id": "122c9f3e-da26-4321-ba68-e0fc02eced58",
- "name": "Treatment 1",
- "allocationPercent": "10",
- "baseline": true,
- "parameters": [
- {
- "variationId": "string",
- "flagKey": "string"
}
]
}
], - "secondaryMetrics": [
- {
- "key": "metric-key-123abc",
- "name": "Example metric",
- "kind": "custom",
- "_links": {
- "self": {
- "href": "/api/v2/metrics/my-project/my-metric",
- "type": "application/json"
}
}
}
]
}
Get results from an experiment for a particular metric.
Experiment results response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Rate limited
{- "_links": {
- "property1": {
- "href": "string",
- "type": "string"
}, - "property2": {
- "href": "string",
- "type": "string"
}
}, - "treatmentResults": [
- {
- "treatmentId": "92b8354e-360e-4d67-8f13-fa6a46ca8077",
- "treatmentName": "variation 25% off",
- "mean": 0.5432525951557093,
- "credibleInterval": {
- "upper": 0.6713222134386467,
- "lower": 0.4060771673663068
}, - "pBest": 0.6083,
- "relativeDifferences": [
- {
- "fromTreatmentId": "92b8354e-360e-4d67-8f13-fa6a46ca8077",
- "lower": -0.13708601934659803,
- "upper": 0.42655970355712425
}
], - "units": 76
}
], - "metricSeen": {
- "ever": true,
- "timestamp": 1657129307
}, - "probabilityOfMismatch": 0.9999999999999738
}
Get current experimentation settings for the given project
Experimentation settings response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Method not allowed
Rate limited
{- "_projectId": "12345abcde67890fghij",
- "_projectKey": "project-key-123abc",
- "randomizationUnits": [
- {
- "randomizationUnit": "user",
- "standardRandomizationUnit": "user",
- "default": true,
- "_hidden": true,
- "_displayName": "User"
}
], - "_creationDate": 0,
- "_links": {
- "property1": {
- "href": "string",
- "type": "string"
}, - "property2": {
- "href": "string",
- "type": "string"
}
}
}
Update experimentation settings for the given project
Experimentation settings response
Invalid request
Invalid access token
Forbidden
Invalid resource identifier
Method not allowed
Rate limited
{- "randomizationUnits": [
- {
- "randomizationUnit": "user",
- "default": true,
- "standardRandomizationUnit": "guest"
}
]
}
{- "_projectId": "12345abcde67890fghij",
- "_projectKey": "project-key-123abc",
- "randomizationUnits": [
- {
- "randomizationUnit": "user",
- "standardRandomizationUnit": "user",
- "default": true,
- "_hidden": true,
- "_displayName": "User"
}
], - "_creationDate": 0,
- "_links": {
- "property1": {
- "href": "string",
- "type": "string"
}, - "property2": {
- "href": "string",
- "type": "string"
}
}
}