logo
logo
search
EXPAND ALL
  • Home
Open Source Docs

New Relic Data Retention Plugin

This tutorial will demonstrate how to use the New Relic data retention plugin for Pixie.

Exporting Pixie data to New Relic allows for the following:

One of Pixie's unique features is that it collects, stores and queries all telemetry data locally in your cluster. Retention time depends on the level of traffic in your cluster, but will generally be on the order of hours. For many use cases, several hours of data is enough to debug issues with your Kubernetes applications. For long-term visibility, Pixie offers a Plugin System that can be used to export Pixie data to an external storage provider. The New Relic plugin makes it easy for you to export Pixie data to the New Relic platform.

Prerequisites

  1. You will need a Kubernetes cluster. If you don’t already have one, you can create a minikube cluster following the directions here.

  2. You will need to install Pixie on your cluster using our install guide.

  3. This tutorial will export HTTP metrics and spans. To follow along with the exact steps in this tutorial, you will want to install our demo microservices application.

  4. You will need a New Relic account. You can sign up for a free trial here. Note: you do not need to follow the instructions to install a New Relic agent to import OpenTelemetry Pixie data to New Relic.

Enable the New Relic plugin

Pixie uses eBPF to automatically capture telemetry data without the need for manual instrumentation. Immediately after installing Pixie on your cluster, Pixie will begin collecting a rich set of telemetry data, including service performance metrics and individual full-body HTTP requests.

Let’s export Pixie telemetry data to New Relic for long-term storage:

  1. Open Pixie’s Live UI.

  2. Navigate to the Plugins tab on the Admin page.

  3. Click the toggle to enable the New Relic plugin and expand the plugin row (by clicking the arrow next to the toggle).

You can enable the New Relic plugin from Pixie’s Admin page.
  1. This plugin requires a New Relic API key. Create a New Relic’s API key and copy this key into the “api-key” field.
  1. (Optional) If your New Relic account is based outside of the US, enter your region-specifc OTLP endpoint in the “Custom export URL” field. Leave this field empty to use the default US OTLP endpoint.

  2. Click the SAVE button.

Set up the New Relic plugin

Now that you’ve enabled the New Relic plugin, let’s choose which Pixie data to export:

  1. Click the database icon in the left sidebar to open the Long-term Data Export page.
To configure long-term data export, select the database icon in the left sidebar.
  1. Check that the scripts under Presets from New Relic are enabled.

The New Relic Pixie plugin provides several preset scripts to help you get started exporting Pixie data to New Relic.

These scripts are written in PxL (Pixie’s query language) and use Pixie’s OpenTelemetry methods to transform Pixie data into the OpenTelemetry format. The New Relic Pixie plugin periodically sends this OpenTelemetry data to the New Relic OpenTelemetry collector.

This tutorial will use the preset scripts. If you’d like to write custom scripts to export additional Pixie data (for example, Redis trace spans), you can do so under the Custom Scripts section.

The New Relic data retention plugin comes with several preset scripts. You can also add your own scripts to export custom Pixie data in the OpenTelemetry format.

Explore Pixie data in New Relic

Now that you’ve enabled and configured the New Relic plugin, you should be exporting Pixie data in the OpenTelemetry format.

The data will all be exported to New Relic’s long term storage. You can create custom dashboards out of Pixie data in New Relic once the plugin is set up. For ease of use, the preset scripts in the Pixie New Relic Plugin automatically power built-in views in New Relic.

Let’s look at your Pixie data in New Relic using the built-in views:

  1. Go to one.newrelic.com.

  2. In the left sidebar, click Services - OpenTelemetry. Note that this option won’t appear until OpenTelemetry data has been received by New Relic’s OpenTelemetry endpoint. If you don’t see this option, check back in a minute or so.

If you deployed the demo microservices app, you should see the px-sock-shop services listed:

New Relic OpenTelemetry services view.
  1. Click on the front-end service (or any service from your cluster).

The service Summary view shows the response time (latency), error rate and throughput for the HTTP/2 requests made to this service.

New Relic OpenTelemetry service Summary view.
  1. Click on the Service Map tab in the left sidebar.

The Service Map view helps visualize your application’s dependencies by showing a graph of all the entities that the front-end service communicates with.

New Relic OpenTelemetry Service Map view.
  1. Select the Transactions tab in the left sidebar.

The service Transactions view shows the different endpoints that the front-end service makes requests to. You can sort the endpoints by response time or throughput.

New Relic OpenTelemetry service Transactions view.
  1. Select the catalogue service from the service drop-down menu near the top of the page.
Change services with the drop-down menu at the top of the page.
  1. Select the Databases tab in the left sidebar.

The service Databases view shows the MySQL requests the catalogue service makes to the catalogue-db service.

New Relic OpenTelemetry service Databases view.

To learn more about how to view your OpenTelemetry data in New Relic, check out the docs.

Create a Dashboard for Pixie metrics

New Relic Dashboards allow you to create custom visualizations for your data.

Let’s install a pre-build dashboard for the data exported from Pixie:

  1. Navigate to the New Relic Dashboards page.

  2. Select the Browse pre-built dashboards button.

  3. Search for “Pixie” and select the Pixie option.

  4. Select the Install this quickstart button.

  5. On the Installation plan page, select “Skip this step” to skip installing the Kubernetes integration.

  6. Click “See your data” to finish the dashboard installation.

The Pixie Quickstart Dashboard allows you to quickly identify the slowest HTTP endpoints and services, monitor HTTP errors by service and target, and see throughput by service.

New Relic Pixie Quickstart dashboard.
  1. Click the HTTP Spans tab at the top of the dashboard to see a list of the most recent HTTP spans captured by Pixie.
See Pixie HTTP spans in the New Relic Pixie Quickstart dashboard.

To learn how to customize your New Relic dashboard, check out the docs.

Create an alert for Pixie metrics

Pixie does not offer alerting yet, but if you export Pixie data to New Relic, you can take advantage of New Relic alerts.

When you installed the Pixie Quickstart Guide, it came with some pre-built alerts. Let’s check them out:

  1. Navigate to the New Relic Alerts page.

  2. Select Alert conditions (Policies) from the left sidebar.

The New Relic Pixie Quickstart Guide comes with a pre-built Pixie alert policy.
  1. Click on the Pixie Quickstart alert policy.

A policy is a group of one or more alert conditions. This policy contains three alerts:

  • Service Errors (%): This alert monitors the HTTP/2 metrics exported by Pixie. An error is counted if http.status_code >= 400 AND http.status_code != 404. If the HTTP/2 error rate goes above 0 at least once in 5 minutes, an alert will be created.
  • Service Throughput (req/min): This alert monitors the HTTP/2 metrics exported by Pixie. If HTTP/2 throughput exceeds the upper or lower baseline for this metric (by 1 standard deviation) for at least 5 minutes, an alert will be created.
  • Service Response Time (ms): This alert monitors the HTTP/2 response time metric exported by Pixie. If HTTP/2 response time exceeds the upper or lower baseline for this metric (by 1 standard deviation) for at least 5 minutes, an alert will be created.
The Pixie Quickstart alert policy comes with 3 pre-built alerts.
  1. Select the On/Off toggles to enable one or more of the alerts.

To learn how to customize your New Relic alerts, check out the docs.

Next Steps

Congrats! You're now exporting Pixie data to New Relic for long-term storage, dashboarding, and alerting.

This tutorial used several of the preset plugin scripts to export Pixie data to New Relic, including the http_metrics, http_spans and mysql scripts. If you’d like to export custom Pixie data (see the full list of Pixie data sources), you can write your own custom export script.

Note that the New Relic plugin requires that you export Pixie data in the OpenTelemetry format. To learn how to write a PxL script to export Pixie data in the OpenTelemetry format, check out the “Write the PxL script” section of the OpenTelemetry export tutorial.

Troubleshooting

Having problems? Check out the Pixie Plugin Troubleshooting guide.

© 2018-21 New Relic, Inc. All Rights Reserved.
This site uses cookies to provide you with a better user experience. By using Pixie, you consent to our use of cookies.