# Observability

> For the complete documentation index, see [llms.txt](https://docs.temporal.io/llms.txt).
> Any documentation page is available as raw Markdown by appending `.md` to its URL.

> Monitor your Temporal Application state using Metrics, Tracing, Logging, and Visibility features. Emit metrics, configure tracing, customize logging, and use Search Attributes with the Temporal Go SDK for enhanced Workflow Execution insights.

This page covers the many ways to view the current state of your [Temporal Application](/temporal#temporal-application)—that is, ways to view which [Workflow Executions](/workflow-execution) are tracked by the [Temporal Platform](/temporal#temporal-platform) and the state of any specified Workflow Execution, either currently or at points of an execution.

This section covers features related to viewing the state of the application, including:

- [Metrics](#metrics)
- [Tracing](#tracing)
- [Logging](#logging)
- [Visibility](#visibility)

## How to emit metrics 

Each Temporal SDK is capable of emitting an optional set of metrics from either the Client or the Worker process.
For a complete list of metrics capable of being emitted, see the [SDK metrics reference](/references/sdk-metrics).

> **💡 Use the OpenTelemetry v2 integration:**
>
> To instrument Temporal applications with OpenTelemetry, use the
> [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2)
> ([Pre-release](/evaluate/development-production-features/release-stages#pre-release)).
> It propagates OpenTelemetry context across Temporal boundaries and
> is replay-safe when instrumenting Workflows.
>

- For a list of metrics, see the [SDK metrics reference](/references/sdk-metrics).
- For an end-to-end example that exposes metrics with the Go SDK, refer to the [samples-go](https://github.com/temporalio/samples-go/tree/main/metrics) repo.

To emit metrics from the Temporal Client in Go, create a [metrics handler](https://pkg.go.dev/go.temporal.io/sdk/internal/common/metrics#Handler) from the [Client Options](https://pkg.go.dev/go.temporal.io/sdk@v1.15.0/internal#ClientOptions) and specify a listener address to be used by Prometheus.

```go
client.Options{
    MetricsHandler: sdktally.NewMetricsHandler(newPrometheusScope(prometheus.Configuration{
      ListenAddress: "0.0.0.0:9090",
      TimerType:     "histogram",
    }
}
```

The Go SDK provides metrics handlers for [Tally](https://pkg.go.dev/go.temporal.io/sdk/contrib/tally) and [OpenTelemetry](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry). Tally offers [extensible custom metrics reporting](https://github.com/uber-go/tally#report-your-metrics), which is exposed through the [`WithCustomMetricsHandler`](/references/server-options#withcustommetricshandler) API.

For more information, see the [Go sample for metrics](https://github.com/temporalio/samples-go/tree/main/metrics).

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Call [`WithTags`](https://pkg.go.dev/go.temporal.io/sdk/client#MetricsHandler) on the metrics handler before you set it on the Client Options.
Every metric created from that handler carries the tags, from both the Client and the Worker.

```go
func main() {
  // Create the base OTel metrics handler
  metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{})

  // Add global/static tags to all emitted metrics
  globalTagsHandler := metricsHandler.WithTags(map[string]string{
    "team": "content-platform",
    "service": "checkout",
    "cost_center": "cc-1042",
    "environment": "production",
  })

  // Attach the tagged handler to client options
  clientOptions := client.Options{
    MetricsHandler: globalTagsHandler,
  }

  temporalClient, err := client.Dial(clientOptions)
}
```

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag           | Example            | Question it answers                                       |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team`        | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service`     | `checkout`         | Which application emits these metrics?                    |
| `cost_center` | `cc-1042`          | Which budget does this Worker fleet belong to?            |
| `environment` | `production`       | Is this production traffic, or staging or test?           |
| `region`      | `us-east-2`        | Where does the Worker fleet run?                          |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

### Configure OpenTelemetry counters as monotonic 

> **📝 Note:**
>
> `UseMonotonicCounters` is available in `go.temporal.io/sdk/contrib/opentelemetry` version 0.8.0 and later.
>

By default, the OpenTelemetry metrics handler represents counters as `Int64UpDownCounter` instruments to preserve compatibility with earlier releases.
To represent Temporal SDK counters as monotonic `Int64Counter` instruments, set `UseMonotonicCounters` to `true` when you create the handler:

```go
metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{
    Meter: otel.GetMeterProvider().Meter("temporal-sdk-go"),
    UseMonotonicCounters: true,
})

temporalClient, err := client.Dial(client.Options{
    MetricsHandler: metricsHandler,
})
```

Monotonic counters let exporters and metrics backends classify Temporal SDK counters correctly.
The [`MetricsCounter`](https://pkg.go.dev/go.temporal.io/sdk/client#MetricsCounter) contract defines counters as ever-increasing.
If you create custom counters through the same metrics handler, pass only non-negative values to [`client.MetricsCounter.Inc`](https://pkg.go.dev/go.temporal.io/sdk/internal/common/metrics#Counter).
Negative values can produce invalid or backend-dependent metric data when `UseMonotonicCounters` is enabled.

## Tracing 

Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and Child Workflows.

> **💡 Use the OpenTelemetry v2 integration:**
>
> To instrument Temporal applications with OpenTelemetry, use the
> [OpenTelemetry v2 integration](/develop/go/integrations/opentelemetry-v2)
> ([Pre-release](/evaluate/development-production-features/release-stages#pre-release)).
> It propagates OpenTelemetry context across Temporal boundaries and
> is replay-safe when instrumenting Workflows.
>

The Go SDK provides tracing interceptors for [OpenTelemetry](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentelemetry), [OpenTracing](https://pkg.go.dev/go.temporal.io/sdk/contrib/opentracing), and [Datadog](https://pkg.go.dev/go.temporal.io/sdk/contrib/datadog/tracing). 

First, create a tracing interceptor for Client instantiation.

```go
// OpenTelemetry
tracingInterceptor, err := opentelemetry.NewTracingInterceptor(opentelemetry.TracerOptions{})

// OpenTracing
tracingInterceptor, err := opentracing.NewInterceptor(opentracing.TracerOptions{})

// Datadog
tracingInterceptor, err := tracing.NewTracingInterceptor(tracing.TracerOptions{})
```

 and register it by passing it to [ClientOptions](https://pkg.go.dev/go.temporal.io/sdk/internal#ClientOptions):

```go
c, err := client.Dial(client.Options{
    Interceptors: []interceptor.ClientInterceptor{tracingInterceptor},
})
```

You can also register interceptors through a [Plugin](/develop/plugins-guide#interceptors) if you’re building a reusable library.

Each tracing interceptor uses its library's native propagation mechanism to serialize trace spans into Temporal headers. For example, OpenTelemetry uses its `TextMapPropagator` with the W3C TraceContext format. The SDK carries these headers across Workflow, Activity, and Child Workflow boundaries, so the tracing library can reconstruct the call graph.
For more information, see the documentation for [OpenTelemetry](https://opentelemetry.io/), [OpenTracing](https://opentracing.io), and [Datadog](https://docs.datadoghq.com/tracing/).

To build custom context propagation (for example, tenant IDs, auth tokens), see [Context Propagation](/develop/go/best-practices/context-propagation).

## Log from a Workflow 

Send logs and errors to a logging service, so that when things go wrong, you can see what happened.

Loggers create an audit trail and capture information about your Workflow's operation.
An appropriate logging level depends on your specific needs.
During development or troubleshooting, you might use debug or even trace.
In production, you might use info or warn to avoid excessive log volume.

You can find the log levels supported by `slog` in [their official documentation](https://pkg.go.dev/log/slog#Level). The Temporal SDK core normally uses `WARN` as its default logging level.

In Workflow Definitions you can use [`workflow.GetLogger(ctx)`](https://pkg.go.dev/go.temporal.io/sdk/workflow#GetLogger) to write logs.

```go
import (
  "context"
  "time"

  "go.temporal.io/sdk/activity"
  "go.temporal.io/sdk/workflow"
)

// Workflow is a standard workflow definition.
// Note that the Workflow and Activity don't need to care that
// their inputs/results are being compressed.
func Workflow(ctx workflow.Context, name string) (string, error) {
// ...

workflow.WithActivityOptions(ctx, ao)

// Getting the logger from the context.
  logger := workflow.GetLogger(ctx)
// Logging a message with the key value pair `name` and `name`
  logger.Info("Compressed Payloads workflow started", "name", name)

  info := map[string]string{
    "name": name,
  }

  logger.Info("Compressed Payloads workflow completed.", "result", result)

  return result, nil
}
```

### Provide a custom logger 

This field sets a custom Logger that is used for all logging actions of the instance of the Temporal Client.

The Go SDK supports custom loggers via `log.NewStructuredLogger()`, which wraps Go's standard [`slog.Logger`](https://pkg.go.dev/log/slog) (Go 1.21+).
Because most modern logging libraries (zap, zerolog, logrus, etc.) can back a `slog.Handler`, `slog` serves as the universal bridge to third-party loggers.

**Using slog directly:**

```go
import (
  "log/slog"
  "os"

  "go.temporal.io/sdk/client"
  "go.temporal.io/sdk/log"
)

func main() {
  // ...
  slogHandler := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})
  logger := log.NewStructuredLogger(slog.New(slogHandler))
  clientOptions := client.Options{
    Logger: logger,
  }
  temporalClient, err := client.Dial(clientOptions)
  // ...
}
```

**Bridging a third-party logger through slog (example with zap):**

```go
import (
  "log/slog"

  "go.uber.org/zap"
  "go.uber.org/zap/exp/zapslog"
  "go.temporal.io/sdk/client"
  "go.temporal.io/sdk/log"
)

func main() {
  // ...
  zapLogger, _ := zap.NewProduction()
  handler := zapslog.NewHandler(zapLogger.Core())
  logger := log.NewStructuredLogger(slog.New(handler))
  clientOptions := client.Options{
    Logger: logger,
  }
  temporalClient, err := client.Dial(clientOptions)
  // ...
}
```

As an alternative, you can implement the `log.Logger` interface directly.
The Temporal samples repo has a [zap adapter](https://github.com/temporalio/samples-go/blob/main/zapadapter/zap_adapter.go) that can be used as a reference.

## Visibility APIs 

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Temporal Service.

### Search Attributes 

The typical method of retrieving a Workflow Execution is by its Workflow Id.

However, sometimes you'll want to retrieve one or more Workflow Executions based on another property. For example, imagine you want to get all Workflow Executions of a certain type that have failed within a time range, so that you can start new ones with the same arguments.

You can do this with [Search Attributes](/search-attribute).

- [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions.
- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`).

The steps to using custom Search Attributes are:

- Create a new Search Attribute in your Temporal Service using `temporal operator search-attribute create` or the Cloud UI.
- Set the value of the Search Attribute for a Workflow Execution:
  - On the Client by including it as an option when starting the Execution.
  - In the Workflow by calling `UpsertSearchAttributes`.
- Read the value of the Search Attribute:
  - On the Client by calling `DescribeWorkflow`.
  - In the Workflow by looking at `WorkflowInfo`.
- Query Workflow Executions by the Search Attribute using a [List Filter](/list-filter):
  - [In the Temporal CLI](/cli/command-reference/workflow#list).
  - In code by calling `ListWorkflowExecutions`.

Here is how to query Workflow Executions:

The [ListWorkflow()](https://pkg.go.dev/go.temporal.io/sdk/client#Client.ListWorkflow) function retrieves a list of [Workflow Executions](/workflow-execution) that match the [Search Attributes](/search-attribute) of a given [List Filter](/list-filter).
The metadata returned from the [Visibility](/temporal-service/visibility) store can be used to get a Workflow Execution's history and details from the [Persistence](/temporal-service/persistence) store.

Use a List Filter to define a `request` to pass into `ListWorkflow()`.

```go
request := &workflowservice.ListWorkflowExecutionsRequest{ Query: "CloseTime = missing" }
```

This `request` value returns only open Workflows.
For more List Filter examples, see the [examples provided for List Filters in the Temporal Visibility guide.](/list-filter#list-filter-examples)

```go
resp, err := temporalClient.ListWorkflow(ctx.Background(), request)
if err != nil {
  return err
}

fmt.Println("First page of results:")
for _, exec := range resp.Executions {
  fmt.Printf("Workflow ID %v\n", exec.Execution.WorkflowId)
}
```

### Set custom Search Attributes 

After you've created custom Search Attributes in your Temporal Service (using the `temporal operator search-attribute create` command or the Cloud UI), you can set the values of the custom Search Attributes when starting a Workflow.

Provide key-value pairs in [`StartWorkflowOptions.SearchAttributes`](https://pkg.go.dev/go.temporal.io/sdk/internal#StartWorkflowOptions).

Search Attributes are represented as `map[string]interface{}`.
The values in the map must correspond to the [Search Attribute's value type](/search-attribute#supported-types):

- Bool = `bool`
- Datetime = `time.Time`
- Double = `float64`
- Int = `int64`
- Keyword = `string`
- Text = `string`

If you had custom Search Attributes `CustomerId` of type Keyword and `MiscData` of type Text, you would provide `string` values:

```go
func (c *Client) CallYourWorkflow(ctx context.Context, workflowID string, payload map[string]interface{}) error {
  // ...
  searchAttributes := map[string]interface{}{
    "CustomerId": payload["customer"],
    "MiscData": payload["miscData"]
  }
  options := client.StartWorkflowOptions{
    SearchAttributes:   searchAttributes
    // ...
  }
  we, err := c.Client.ExecuteWorkflow(ctx, options, app.YourWorkflow, payload)
  // ...
}
```

### Upsert Search Attributes 

You can upsert Search Attributes to add or update Search Attributes from within Workflow code.

In advanced cases, you may want to dynamically update these attributes as the Workflow progresses.
[UpsertSearchAttributes](https://pkg.go.dev/go.temporal.io/sdk/workflow#UpsertSearchAttributes) is used to add or update Search Attributes from within Workflow code.

`UpsertSearchAttributes` will merge attributes to the existing map in the Workflow.
Consider this example Workflow code:

```go
func YourWorkflow(ctx workflow.Context, input string) error {

  attr1 := map[string]interface{}{
    "CustomIntField": 1,
    "CustomBoolField": true,
  }
  workflow.UpsertSearchAttributes(ctx, attr1)

  attr2 := map[string]interface{}{
    "CustomIntField": 2,
    "CustomKeywordField": "seattle",
  }
  workflow.UpsertSearchAttributes(ctx, attr2)
}
```

After the second call to `UpsertSearchAttributes`, the map will contain:

```go
map[string]interface{}{
  "CustomIntField": 2, // last update wins
  "CustomBoolField": true,
  "CustomKeywordField": "seattle",
}
```

### Remove a Search Attribute from a Workflow 

To remove a Search Attribute that was previously set, set it to an empty array: `[]`.

**There is no support for removing a field.**

However, to achieve a similar effect, set the field to some placeholder value.
For example, you could set `CustomKeywordField` to `impossibleVal`.
Then searching `CustomKeywordField != 'impossibleVal'` will match Workflows with `CustomKeywordField` not equal to `impossibleVal`, which includes Workflows without the `CustomKeywordField` set.