Experiments (beta)

Experimentation is an add-on feature

Experimentation is available as an add-on for customers on an Enterprise plan. To learn more, read about our pricing. To upgrade your plan, contact Sales.

This feature is in beta

To use this feature, pass in a header including the LD-API-Version key with value set to beta. Use this header with each call. To learn more, read Beta resources.

An experiment is a running tally of user behaviors based on which variation of a feature flag they receive. In LaunchDarkly, an experiment requires two things: a metric that defines what user behaviors to track, and at least one flag associated with that metric. To learn more, read Experimentation.

Experiments are largely managed by making PATCH requests to the flags API. There are also dedicated API resources for getting and deleting experiment data.

Adding an experiment

To add an experiment to a flag, add a new object to the /experiments/items/ property.

The object must contain:

  1. The key of the metric you want to use for the experiment.
  2. (Optional) A list of keys for the environments where the experiment should record data.
PATCH /api/v2/flags/{projKey}/{flagKey}
[
    {
        "op": "add",
        "path": "/experiments/items/0",
        "value": {
            "metricKey": "example-metric",
            "environments": []
        }
    }
]

Removing an experiment

To remove an experiment, remove the corresponding item from the experiments or items property.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
    {
        "op": "remove",
        "path": "/experiments/items/0"
    }
]

Starting or resuming an experiment

To start recording data for an experiment in an environment, add the environment key to the list of environments for that experiment.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
    {
        "op": "add",
        "path": "/experiments/items/0/environments/0",
        "value": "production"
    }
]

Pausing an experiment

To pause collecting data for an experiment in a specific environment, remove the environment from the list that experiment uses.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
  {
    "op": "remove",
    "path": "/experiments/items/0/environments/0"
  }
]

Updating the baseline

To update the baseline for all experiments on a flag across all environments in the project, replace the baselineIdx property with the index of the new baseline variation.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
  {
    "op": "replace",
    "path": "/experiments/baselineIdx",
    "value": 1
  }
]

Get legacy experiment results (deprecated)

Get detailed experiment result data for legacy experiments.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

featureFlagKey
required
string <string>

The feature flag key

environmentKey
required
string <string>

The environment key

metricKey
required
string <string>

The metric key

query Parameters
from
integer <int64>

A timestamp denoting the start of the data collection period, expressed as a Unix epoch time in milliseconds.

to
integer <int64>

A timestamp denoting the end of the data collection period, expressed as a Unix epoch time in milliseconds.

Responses
200

Experiment results response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

get/api/v2/flags/{projectKey}/{featureFlagKey}/experiments/{environmentKey}/{metricKey}
Request samples
curl -i -X GET \
  'https://app.launchdarkly.com/api/v2/flags/{projectKey}/{featureFlagKey}/experiments/{environmentKey}/{metricKey}' \
  -H 'Authorization: YOUR_API_KEY_HERE'
Response samples
application/json
{
  • "_links": {
    },
  • "metadata": [
    ],
  • "totals": [
    ],
  • "series": [
    ],
  • "stats": {
    },
  • "granularity": "string",
  • "metricSeen": {
    }
}

Reset experiment results

Reset all experiment results by deleting all existing data for an experiment.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

featureFlagKey
required
string <string>

The feature flag key

environmentKey
required
string <string>

The environment key

metricKey
required
string <string>

The metric's key

Responses
204

Experiment results reset successfully

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

delete/api/v2/flags/{projectKey}/{featureFlagKey}/experiments/{environmentKey}/{metricKey}/results
Request samples
curl -i -X DELETE \
  'https://app.launchdarkly.com/api/v2/flags/{projectKey}/{featureFlagKey}/experiments/{environmentKey}/{metricKey}/results' \
  -H 'Authorization: YOUR_API_KEY_HERE'
Response samples
application/json
{
  • "code": "unauthorized",
  • "message": "invalid key"
}

Get experiments

Get details about all experiments in an environment.

Filtering experiments

LaunchDarkly supports the filter query param for filtering, with the following fields:

  • flagKey filters for only experiments which use the flag with the given key.
  • status filters for only experiments with an iteration with the given status. An iteration can have the status not_started, running or stopped.

For example, filter=flagKey:my-flag,status=running filters for experiments for the given flag key which have a currently running iteration.

Expanding the experiments response

LaunchDarkly supports four fields for expanding the "List experiments" response. By default, these fields are not included in the response.

To expand the response, append the expand query parameter and add a comma-separated list with any of the following fields:

  • previousIterations includes all iterations prior to the current iteration. By default only the current iteration will be included in the response.
  • draftIteration includes a draft of an iteration which has not been started yet, if any.
  • secondaryMetrics includes secondary metrics. By default only the primary metric is included in the response.
  • treatments includes all treatment and parameter details. By default treatment data will not be included in the response.

For example, expand=draftIteration,treatments includes the draftIteration and treatments fields in the response.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

query Parameters
limit
integer <int64>

The maximum number of experiments to return. Defaults to 20

offset
integer <int64>

Where to start in the list. Use this with pagination. For example, an offset of 10 skips the first ten items and then returns the next items in the list, up to the query limit.

filter
string <string>

A comma-separated list of filters. Each filter is of the form field:value. Supported fields are explained above.

expand
string <string>

A comma-separated list of properties that can reveal additional information in the response. Supported fields are explained above.

Responses
200

Experiment response JSON

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

405

Method not allowed

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments
Request samples
curl -i -X GET \
  'https://app.launchdarkly.com/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments' \
  -H 'Authorization: YOUR_API_KEY_HERE'
Response samples
application/json
{
  • "items": [
    ],
  • "total_count": 0,
  • "_links": {
    }
}

Create experiment

Create an experiment. To learn more, read Creating experiments.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

Request Body schema: application/json
name
required
string
description
string
maintainerId
required
string
key
required
string
required
object (IterationInput)
Responses
201

Experiment response JSON

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

post/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments
Request samples
application/json
{
  • "name": "string",
  • "description": "string",
  • "maintainerId": "string",
  • "key": "string",
  • "iteration": {
    }
}
Response samples
application/json
{
  • "_id": "string",
  • "key": "string",
  • "name": "string",
  • "description": "string",
  • "_maintainerId": "string",
  • "_creationDate": 0,
  • "_links": {
    },
  • "currentIteration": {
    },
  • "draftIteration": {
    },
  • "previousIterations": [
    ]
}

Get experiment

Get details about an experiment.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

Responses
200

Experiment response JSON

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

405

Method not allowed

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}
Request samples
curl -i -X GET \
  'https://app.launchdarkly.com/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}' \
  -H 'Authorization: YOUR_API_KEY_HERE'
Response samples
application/json
{
  • "_id": "string",
  • "key": "string",
  • "name": "string",
  • "description": "string",
  • "_maintainerId": "string",
  • "_creationDate": 0,
  • "_links": {
    },
  • "currentIteration": {
    },
  • "draftIteration": {
    },
  • "previousIterations": [
    ]
}

Patch experiment

Update an experiment. Updating an experiment uses the semantic patch format.

To make a semantic patch request, you must append domain-model=launchdarkly.semanticpatch to your Content-Type header. To learn more, read Updates using semantic patch.

Instructions

Semantic patch requests support the following kind instructions for updating experiments.

updateName

Updates the experiment name.

Parameters
  • value: The new name.

updateDescription

Updates the experiment description.

Parameters
  • value: The new description.

startIteration

Starts a new iteration for this experiment.

stopIteration

Stops the current iteration for this experiment.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

Request Body schema: application/json
Comment
string
Array of objects (Instructions)
Responses
200

Experiment response JSON

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

patch/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}
Request samples
application/json
{
  • "Comment": "string",
  • "Instructions": [
    ]
}
Response samples
application/json
{
  • "_id": "string",
  • "key": "string",
  • "name": "string",
  • "description": "string",
  • "_maintainerId": "string",
  • "_creationDate": 0,
  • "_links": {
    },
  • "currentIteration": {
    },
  • "draftIteration": {
    },
  • "previousIterations": [
    ]
}

Create iteration

Create an experiment iteration. Experiment iterations let you record experiments in discrete blocks of time. To learn more, read Starting experiment iterations.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

Request Body schema: application/json
hypothesis
required
string
canReshuffleTraffic
boolean
required
Array of objects (MetricsInput)
required
Array of objects (TreatmentsInput)
required
object (FlagsInput)
Responses
200

Iteration response JSON

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

post/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}/iterations
Request samples
application/json
{
  • "hypothesis": "string",
  • "canReshuffleTraffic": true,
  • "metrics": [
    ],
  • "treatments": [
    ],
  • "flags": {
    }
}
Response samples
application/json
{
  • "_id": "string",
  • "hypothesis": "string",
  • "status": "string",
  • "createdAt": 0,
  • "startedAt": 0,
  • "endedAt": 0,
  • "winningTreatmentId": "string",
  • "winningReason": "string",
  • "canReshuffleTraffic": true,
  • "flags": {
    },
  • "primaryMetric": {
    },
  • "treatments": [
    ],
  • "secondaryMetrics": [
    ]
}

Get experiment results

Get results from an experiment for a particular metric.

SecurityApiKey
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

metricKey
required
string <string>

The metric key

Responses
200

Experiment results response JSON

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}/metrics/{metricKey}/results
Request samples
curl -i -X GET \
  'https://app.launchdarkly.com/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}/metrics/{metricKey}/results' \
  -H 'Authorization: YOUR_API_KEY_HERE'
Response samples
application/json
{
  • "_links": {
    },
  • "treatmentResults": [
    ],
  • "metricSeen": {
    }
}
➔ Next to Feature flags