Experiments (beta)

Available for Pro and Enterprise plans

Experimentation is available to customers on a Pro or Enterprise plan. To learn more, read about our pricing. To add Experimentation to your plan, contact Sales.

This feature is in beta

To use this feature, pass in a header including the LD-API-Version key with value set to beta. Use this header with each call. To learn more, read Beta resources.

Resources that are in beta are still undergoing testing and development. They may change without notice, including becoming backwards incompatible.

Experimentation lets you validate the impact of features you roll out to your app or infrastructure. You can measure things like page views, clicks, load time, infrastructure costs, and more. By connecting metrics you create to flags in your LaunchDarkly environment, you can measure the changes in your customers' behavior based on what flags they evaluate. You can run experiments on boolean, string, and number flags, but not JSON flags. To learn more, read About Experimentation.

You can manage experiments by using the dedicated experiment endpoints described below.

Several of the endpoints require a treatment ID or a flag rule ID. Treatment IDs are returned as part of the Get experiment results response. They are the treatmentId of each element in the treatmentResults array. Winning treatment IDs are also returned as part of the Get experiment response. They are the winningTreatmentId in the currentIteration, the winningTreatmentId in the draftIteration, and the winningTreatmentId in each element of the previousIterations array. In the flags object, the rule ID is the ID of the variation or rollout of the flag. Each flag variation ID is returned as part of the Get feature flag response. It is the _id field in each element of the variations array.

Customers on older contracts may have experiments created under an older data model. These experiments are listed in the LaunchDarkly user interface as "Legacy experiments." You can manage them by making PATCH requests to the flags API, and you can analyze them by making a GET request to the deprecated Get legacy experiment results endpoint. To learn more about legacy experiments, read Legacy experiments.

Click to expand details on using the PATCH requests to the flags API to manage legacy experiments

Managing legacy experiments

To manage legacy experiments, make the following PATCH requests to the flags API:

Adding an experiment

To add an experiment to a flag, add a new object to the /experiments/items/ property.

The object must contain:

  1. The key of the metric you want to use for the experiment.
  2. (Optional) A list of keys for the environments where the experiment should record data.
PATCH /api/v2/flags/{projKey}/{flagKey}
[
    {
        "op": "add",
        "path": "/experiments/items/0",
        "value": {
            "metricKey": "metric-key-123abc",
            "environments": []
        }
    }
]

Removing an experiment

To remove an experiment, remove the corresponding item from the experiments or items property.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
    {
        "op": "remove",
        "path": "/experiments/items/0"
    }
]

Starting or resuming an experiment

To start recording data for an experiment in an environment, add the environment key to the list of environments for that experiment.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
    {
        "op": "add",
        "path": "/experiments/items/0/environments/0",
        "value": "production"
    }
]

Stopping an experiment

To stop collecting data for an experiment in a specific environment, remove the environment from the list that experiment uses.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
  {
    "op": "remove",
    "path": "/experiments/items/0/environments/0"
  }
]

Updating the baseline

To update the baseline for all experiments on a flag across all environments in the project, replace the baselineIdx property with the index of the new baseline variation.

PATCH /api/v2/flags/{projKey}/{flagKey}
[
  {
    "op": "replace",
    "path": "/experiments/baselineIdx",
    "value": 1
  }
]

Get legacy experiment results (deprecated)Deprecated

Get detailed experiment result data for legacy experiments.

Request
path Parameters
projectKey
required
string <string>

The project key

featureFlagKey
required
string <string>

The feature flag key

environmentKey
required
string <string>

The environment key

metricKey
required
string <string>

The metric key

query Parameters
from
integer <int64>

A timestamp denoting the start of the data collection period, expressed as a Unix epoch time in milliseconds.

to
integer <int64>

A timestamp denoting the end of the data collection period, expressed as a Unix epoch time in milliseconds.

Responses
200

Experiment results response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

get/api/v2/flags/{projectKey}/{featureFlagKey}/experiments/{environmentKey}/{metricKey}
Request samples
Response samples
application/json
{
  • "_links": {
    },
  • "metadata": [
    ],
  • "totals": [
    ],
  • "series": [
    ],
  • "stats": {
    },
  • "granularity": "string",
  • "metricSeen": {
    }
}

Get experiments

Get details about all experiments in an environment.

Filtering experiments

LaunchDarkly supports the filter query param for filtering, with the following fields:

  • flagKey filters for only experiments that use the flag with the given key.
  • metricKey filters for only experiments that use the metric with the given key.
  • status filters for only experiments with an iteration with the given status. An iteration can have the status not_started, running or stopped.

For example, filter=flagKey:my-flag,status:running,metricKey:page-load-ms filters for experiments for the given flag key and the given metric key which have a currently running iteration.

Expanding the experiments response

LaunchDarkly supports four fields for expanding the "Get experiments" response. By default, these fields are not included in the response.

To expand the response, append the expand query parameter and add a comma-separated list with any of the following fields:

  • previousIterations includes all iterations prior to the current iteration. By default only the current iteration is included in the response.
  • draftIteration includes the iteration which has not been started yet, if any.
  • secondaryMetrics includes secondary metrics. By default only the primary metric is included in the response.
  • treatments includes all treatment and parameter details. By default treatment data is not included in the response.

For example, expand=draftIteration,treatments includes the draftIteration and treatments fields in the response. If fields that you request with the expand query parameter are empty, they are not included in the response.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

query Parameters
limit
integer <int64>

The maximum number of experiments to return. Defaults to 20.

offset
integer <int64>

Where to start in the list. Use this with pagination. For example, an offset of 10 skips the first ten items and then returns the next items in the list, up to the query limit.

filter
string <string>

A comma-separated list of filters. Each filter is of the form field:value. Supported fields are explained above.

expand
string <string>

A comma-separated list of properties that can reveal additional information in the response. Supported fields are explained above.

lifecycleState
string <string>

A comma-separated list of experiment archived states. Supports archived, active, or both. Defaults to active experiments.

Responses
200

Experiment collection response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

405

Method not allowed

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments
Request samples
Response samples
application/json
{
  • "items": [
    ],
  • "total_count": 0,
  • "_links": {
    }
}

Create experiment

Create an experiment.

To run this experiment, you'll need to create an iteration and then update the experiment with the startIteration instruction.

To learn more, read Creating experiments.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

Request Body schema: application/json
required
name
required
string

The experiment name

description
string

The experiment description

maintainerId
string

The ID of the member who maintains this experiment

key
required
string

The experiment key

required
object (IterationInput)
Responses
201

Experiment response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

post/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments
Request samples
application/json
{
  • "name": "Example experiment",
  • "description": "An example experiment, used in testing",
  • "maintainerId": "12ab3c45de678910fgh12345",
  • "key": "experiment-key-123abc",
  • "iteration": {
    }
}
Response samples
application/json
{
  • "_id": "12ab3c45de678910fgh12345",
  • "key": "experiment-key-123abc",
  • "name": "Example experiment",
  • "description": "An example experiment, used in testing",
  • "_maintainerId": "12ab3c45de678910fgh12345",
  • "_creationDate": 0,
  • "environmentKey": "string",
  • "archivedDate": 0,
  • "_links": {
    },
  • "currentIteration": {
    },
  • "draftIteration": {
    },
  • "previousIterations": [
    ]
}

Get experiment

Get details about an experiment.

Expanding the experiment response

LaunchDarkly supports four fields for expanding the "Get experiment" response. By default, these fields are not included in the response.

To expand the response, append the expand query parameter and add a comma-separated list with any of the following fields:

  • previousIterations includes all iterations prior to the current iteration. By default only the current iteration is included in the response.
  • draftIteration includes the iteration which has not been started yet, if any.
  • secondaryMetrics includes secondary metrics. By default only the primary metric is included in the response.
  • treatments includes all treatment and parameter details. By default treatment data is not included in the response.

For example, expand=draftIteration,treatments includes the draftIteration and treatments fields in the response. If fields that you request with the expand query parameter are empty, they are not included in the response.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

query Parameters
expand
string <string>

A comma-separated list of properties that can reveal additional information in the response. Supported fields are explained above.

Responses
200

Experiment response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

405

Method not allowed

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}
Request samples
Response samples
application/json
{
  • "_id": "12ab3c45de678910fgh12345",
  • "key": "experiment-key-123abc",
  • "name": "Example experiment",
  • "description": "An example experiment, used in testing",
  • "_maintainerId": "12ab3c45de678910fgh12345",
  • "_creationDate": 0,
  • "environmentKey": "string",
  • "archivedDate": 0,
  • "_links": {
    },
  • "currentIteration": {
    },
  • "draftIteration": {
    },
  • "previousIterations": [
    ]
}

Patch experiment

Update an experiment. Updating an experiment uses the semantic patch format.

To make a semantic patch request, you must append domain-model=launchdarkly.semanticpatch to your Content-Type header. To learn more, read Updates using semantic patch.

Instructions

Semantic patch requests support the following kind instructions for updating experiments.

updateName

Updates the experiment name.

Parameters
  • value: The new name.

Here's an example:

{
  "instructions": [{
    "kind": "updateName",
    "value": "Example updated experiment name"
  }]
}

updateDescription

Updates the experiment description.

Parameters
  • value: The new description.

Here's an example:

{
  "instructions": [{
    "kind": "updateDescription",
    "value": "Example updated description"
  }]
}

startIteration

Starts a new iteration for this experiment. You must create a new iteration before calling this instruction.

An iteration may not be started until it meets the following criteria:

  • Its associated flag is toggled on and is not archived
  • Its randomizationUnit is set
  • At least one of its treatments has a non-zero allocationPercent
Parameters
  • changeJustification: The reason for starting a new iteration. Required when you call startIteration on an already running experiment, otherwise optional.

Here's an example:

{
  "instructions": [{
    "kind": "startIteration",
    "changeJustification": "It's time to start a new iteration"
  }]
}

stopIteration

Stops the current iteration for this experiment.

Parameters
  • winningTreatmentId: The ID of the winning treatment. Treatment IDs are returned as part of the Get experiment response. They are the _id of each element in the treatments array.
  • winningReason: The reason for the winner

Here's an example:

{
  "instructions": [{
    "kind": "stopIteration",
    "winningTreatmentId": "3a548ec2-72ac-4e59-8518-5c24f5609ccf",
    "winningReason": "Example reason to stop the iteration"
  }]
}

archiveExperiment

Archives this experiment. Archived experiments are hidden by default in the LaunchDarkly user interface. You cannot start new iterations for archived experiments.

Here's an example:

{
  "instructions": [{ "kind": "archiveExperiment" }]
}

restoreExperiment

Restores an archived experiment. After restoring an experiment, you can start new iterations for it again.

Here's an example:

{
  "instructions": [{ "kind": "restoreExperiment" }]
}
Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

Request Body schema: application/json
required
comment
string

Optional comment describing the update

required
Array of objects (Instructions)
Responses
200

Experiment response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

patch/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}
Request samples
application/json
{
  • "comment": "Example comment describing the update",
  • "instructions": [
    ]
}
Response samples
application/json
{
  • "_id": "12ab3c45de678910fgh12345",
  • "key": "experiment-key-123abc",
  • "name": "Example experiment",
  • "description": "An example experiment, used in testing",
  • "_maintainerId": "12ab3c45de678910fgh12345",
  • "_creationDate": 0,
  • "environmentKey": "string",
  • "archivedDate": 0,
  • "_links": {
    },
  • "currentIteration": {
    },
  • "draftIteration": {
    },
  • "previousIterations": [
    ]
}

Create iteration

Create an experiment iteration.

Experiment iterations let you record experiments in individual blocks of time. Initially, iterations are created with a status of not_started and appear in the draftIteration field of an experiment. To start or stop an iteration, update the experiment with the startIteration or stopIteration instruction.

To learn more, read Starting experiment iterations.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

Request Body schema: application/json
required
hypothesis
required
string

The expected outcome of this experiment

canReshuffleTraffic
boolean

Whether to allow the experiment to reassign traffic to different variations when you increase or decrease the traffic in your experiment audience (true) or keep all traffic assigned to its initial variation (false). Defaults to true.

required
Array of objects (MetricsInput)
primarySingleMetricKey
string

The key of the primary metric for this experiment. Either primarySingleMetricKey or primaryFunnelKey must be present.

primaryFunnelKey
string

The key of the primary funnel group for this experiment. Either primarySingleMetricKey or primaryFunnelKey must be present.

required
Array of objects (TreatmentsInput)
required
object (FlagsInput)
randomizationUnit
string

The unit of randomization for this iteration. Defaults to user.

Responses
200

Iteration response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

post/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}/iterations
Request samples
application/json
{
  • "hypothesis": "Example hypothesis, the new button placement will increase conversion",
  • "canReshuffleTraffic": true,
  • "metrics": [
    ],
  • "primarySingleMetricKey": "metric-key-123abc",
  • "primaryFunnelKey": "metric-group-key-123abc",
  • "treatments": [
    ],
  • "flags": {
    },
  • "randomizationUnit": "user"
}
Response samples
application/json
{
  • "_id": "12ab3c45de678910fgh12345",
  • "hypothesis": "The new button placement will increase conversion",
  • "status": "running",
  • "createdAt": 0,
  • "startedAt": 0,
  • "endedAt": 0,
  • "winningTreatmentId": "122c9f3e-da26-4321-ba68-e0fc02eced58",
  • "winningReason": "We ran this iteration for two weeks and the winning variation was clear",
  • "canReshuffleTraffic": true,
  • "flags": {
    },
  • "primaryMetric": {
    },
  • "primarySingleMetric": {
    },
  • "primaryFunnel": {
    },
  • "randomizationUnit": "user",
  • "treatments": [
    ],
  • "secondaryMetrics": [
    ],
  • "metrics": [
    ]
}

Get experiment results for metric group

Get results from an experiment for a particular metric group.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

metricGroupKey
required
string <string>

The metric group key

query Parameters
iterationId
string <string>

The iteration ID

Responses
200

Experiment results response for metric group

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}/metric-groups/{metricGroupKey}/results
Request samples
Response samples
application/json
{
  • "_links": {
    },
  • "metrics": [
    ]
}

Get experiment results

Get results from an experiment for a particular metric.

LaunchDarkly supports one field for expanding the "Get experiment results" response. By default, this field is not included in the response.

To expand the response, append the expand query parameter with the following field:

  • traffic includes the total count of units for each treatment.

For example, expand=traffic includes the traffic field for the project in the response.

Request
path Parameters
projectKey
required
string <string>

The project key

environmentKey
required
string <string>

The environment key

experimentKey
required
string <string>

The experiment key

metricKey
required
string <string>

The metric key

query Parameters
iterationId
string <string>

The iteration ID

expand
string <string>

A comma-separated list of fields to expand in the response. Supported fields are explained above.

Responses
200

Experiment results response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

429

Rate limited

get/api/v2/projects/{projectKey}/environments/{environmentKey}/experiments/{experimentKey}/metrics/{metricKey}/results
Request samples
Response samples
application/json
{
  • "_links": {
    },
  • "treatmentResults": [
    ],
  • "metricSeen": {
    },
  • "probabilityOfMismatch": 0.9999999999999738,
  • "results": [
    ]
}

Get experimentation settings

Get current experimentation settings for the given project

Request
path Parameters
projectKey
required
string <string>

The project key

Responses
200

Experimentation settings response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

405

Method not allowed

429

Rate limited

get/api/v2/projects/{projectKey}/experimentation-settings
Request samples
Response samples
application/json
{
  • "_projectId": "12345abcde67890fghij",
  • "_projectKey": "project-key-123abc",
  • "randomizationUnits": [
    ],
  • "_creationDate": 0,
  • "_links": {
    }
}

Update experimentation settings

Update experimentation settings for the given project

Request
path Parameters
projectKey
required
string <string>

The project key

Request Body schema: application/json
required
required
Array of objects (RandomizationUnitInput)

An array of randomization units allowed for this project.

Responses
200

Experimentation settings response

400

Invalid request

401

Invalid access token

403

Forbidden

404

Invalid resource identifier

405

Method not allowed

429

Rate limited

put/api/v2/projects/{projectKey}/experimentation-settings
Request samples
application/json
{
  • "randomizationUnits": [
    ]
}
Response samples
application/json
{
  • "_projectId": "12345abcde67890fghij",
  • "_projectKey": "project-key-123abc",
  • "randomizationUnits": [
    ],
  • "_creationDate": 0,
  • "_links": {
    }
}