{"id":388655,"date":"2024-06-29T08:05:01","date_gmt":"2024-06-29T08:05:01","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=388655"},"modified":"-0001-11-30T00:00:00","modified_gmt":"2024-06-29T08:05:01","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=388655","title":{"rendered":"Distributed Tracing for Microservice Architecture<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n

What is distributed tracing? Distributed tracing<\/a> is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.<\/p>\n

Let\u2019s have a look at a simple prototype. A user fetches information about a shipment from logistic<\/code>service. logistic<\/code>service does some computation and fetches the data from a database. logistic service doesn\u2019t know the actual status of the shipment, so it has to fetch the updated status from another service tracking<\/code> tracking<\/code>service also needs to fetch the data from a database and to do some computation.<\/p>\n

In the screenshot below, we see a whole life cycle of the request issued to logistics<\/code>service:<\/p>\n

<\/figcaption><\/figure>\n
    \n
  1. \n

    GET request hits endpoint \/shipment\/:id<\/code>in the service logistics<\/code><\/p>\n<\/li>\n

  2. \n

    The service does some internal processing that takes 10 ms (span logistics processing<\/code>.<\/p>\n<\/li>\n

  3. \n

    The service fetches data from the database for 30 ms (span SQL select<\/code>. <\/p>\n<\/li>\n

  4. \n

    logistics<\/code>fetches the status of the shipment from another service tracking by sending a GET request to endpoint \/tracking:\/:id<\/code>(span tracking<\/code>. We have detailed information about this part of the journey as well, i.e. how much time tracking<\/code>service used for the processing or how much time the service used for the database operation.<\/p>\n<\/li>\n<\/ol>\n

    From the screen below, we can see that the request was successful and returned 200. <\/p>\n

    <\/figcaption><\/figure>\n

    It is especially handy as we didn\u2019t log the response code or created the metadata explicitly: it comes for free from the tracing middleware. <\/p>\n

    Let\u2019s say we want to profile our logistics service. As we can see, we spend most of the time on the database operation. It would be helpful to see the exact query we issued:<\/p>\n

    <\/figcaption><\/figure>\n

    Now we can check whether the query was optimal and which tables it affected. The span contains the information about the caller, including the line number and file name, so it is easy to navigate in the source code.<\/p>\n

    Speaking in general, we can profile any part of the code be it a module or a single function. Such data helps to spot the bottlenecks and make informed decisions about the optimization. <\/p>\n

    Tracing also automatically creates a snapshot of our architecture (DAG):<\/p>\n

    <\/figcaption><\/figure>\n<\/p>\n

    Obviously, the picture above doesn\u2019t look that impressive. But what happens when we have not 2 services, but 30? Let\u2019s say a user logs in and we need to fetch the information from services such as users,<\/code> settings,<\/code> billing,<\/code>etc. In that case, this diagram would be very handy to understand the actual interactions between our services. The tracing would also reveal which of our services takes the most time or which one is the least reliable. <\/p>\n

    Tracing also helps to debug the problems in the microservice architecture. Let\u2019s say each of our microservices is up and behaves well on an individual level. However, we have a problem on a system-wide level. I.e. there is a particular combination of input and output data between the microservices that causes a problem. In that case, individual logs from each of the microservices won\u2019t reveal a problem. We need to tie all the logs from the affected microservices and see the whole picture. Tracing is the right instrument to solve such kind of problems. <\/p>\n

    Distributed Tracing APIs<\/h2>\n

    OpenTracing<\/h3>\n

    What is OpenTracing? It\u2019s a vendor-agnostic API to help developers easily instrument tracing into their\u00a0codebase. This means that if a developer wants to try out a different distributed tracing system, then instead of repeating the whole instrumentation process for the new distributed tracing system, the developer can simply change the configuration of the Tracer.<\/p>\n

    Most mental models for tracing descend from Google\u2019s Dapper paper. OpenTracing uses similar nouns and verbs (source<\/a>):<\/p>\n

    <\/figcaption><\/figure>\n
      \n
    1. \n

      Trace<\/strong>: The description of a transaction as it moves through a distributed system.<\/p>\n<\/li>\n

    2. \n

      Span<\/strong>: A named, timed operation representing a piece of the workflow. Spans accept key:value tags as well as fine-grained, timestamped, structured logs attached to the particular span instance.<\/p>\n<\/li>\n

    3. \n

      Span context<\/strong>: Trace information that accompanies the distributed transaction, including when it passes the service to service over the network or through a message bus. The span context contains the trace identifier, span identifier, and any other data that the tracing system needs to propagate to the downstream service.<\/p>\n<\/li>\n<\/ol>\n

      There are several tracers that support<\/a> OpenTracing API, including Jaeger, Instana, and Elastic APM. <\/p>\n

      OpenTelemetry<\/h3>\n

      OpenTelemetry<\/a> is a set of APIs, SDKs, tooling and integrations that are designed for the creation and management of\u00a0telemetry data<\/em>\u00a0such as traces, metrics, and logs. OpenTelemetry is a CNCF Sandbox member, formed through a merger of the OpenTracing and OpenCensus projects.<\/p>\n

      OpenTelemetry implementation for Go is in pre-GA phase. As of today (15.01.2021), the estimated<\/a> percentage of completion for the first release candidate is 72%.<\/p>\n

      <\/figcaption><\/figure>\n

      One of the three main components — logging have not yet been implemented<\/a>:<\/p>\n

      <\/figcaption><\/figure>\n

      As OpenTelemetry implementation in Go hasn\u2019t yet matured, this proposal uses OpenTracing standard. Both standards use the same terminology and concepts, so the migration to newer OpenTelemetry should be relatively straightforward. <\/p>\n

      Tracers<\/h2>\n

      Jaeger<\/h3>\n

      The Jaeger<\/a> tracing system is an open-source tracing system for microservices, and it supports the\u00a0OpenTracing<\/u><\/a>\u00a0standard. Jaeger was initially published as open source by Uber Technologies and has evolved since then. The system gives you distributing tracing, root cause analysis, service dependency analysis, and more.<\/p>\n

      New Relic APM<\/h3>\n

      New Relic APM (Application Performance Monitoring) provides monitoring of application\u2019s performance and has support for distributed tracing.<\/p>\n

      New Relic ATM uses the same terminology as OpenTracing with the addition of transaction. At New Relic, a\u00a0transaction<\/strong>\u00a0is defined as one logical unit of work in a software application. Specifically, it refers to the function calls and method calls that make up that unit of work.\u00a0For APM, it will often refer to a\u00a0web transaction<\/strong>, which represents an activity that happens from when the application receives a web request to when the response is sent.<\/p>\n

      New Relic APM has support for the instrumentation of Go applications.<\/a> The API can be found here<\/a>.<\/p>\n

      New Relic also provides open-source integrations that report telemetry data from telemetry tools to a New Relic account. In particular, it provides an exporter<\/a> for sending OpenTelemetry data to New Relic. <\/p>\n

      Grafana Tempo<\/h3>\n

      Grafana Tempo is a tracing backend that implements OpenTelemetry\/OpenTracing standards. Here<\/a> is example instrumentation with opentelementry<\/code>library.<\/p>\n

      Sentry<\/h3>\n

      Sentry is an application monitoring system with support for distributed tracing<\/a>. The API can be found here<\/a>.<\/p>\n

      Prototype<\/h2>\n

      Prototype microservices were written in Go and are using OpenTracing API. <\/p>\n

      Service logistics<\/h3>\n

      Service logistics<\/code>exposes a REST endpoint \/shipment\/:id.<\/code>We use echo<\/code>library that has excellent support for the tracing middleware. Note, though, that echo<\/code>is not a hard requirement — it is possible to write our own tracing middleware and achieve the same result.<\/p>\n

      We instantiate a logging middleware that automatically logs a response code and some metadata. <\/p>\n

      SERVICE = \"logistics\"  e := echo.New() os.Setenv(\"JAEGER_SERVICE_NAME\", SERVICE)  \/\/ Enable tracing middleware c := jaegertracing.New(e, nil) defer c.Close()<\/code><\/pre>\n
      Create a simple handler: <\/p>\n
      e.GET(\"\/shipment\/:id\", func(c echo.Context) error {    sp := jaegertracing.CreateChildSpan(c, \"get shipment request\")    defer sp.Finish()     id := c.Param(\"id\")     processing(c, id)    shipment := getShipmentFromDatabase(c, id)    shipment.Status = fetchStatus(c, id)    time.Sleep(time.Millisecond * 15)     return c.JSON(http.StatusOK, shipment) })<\/code><\/pre>\n
      We create the child span sp<\/code>from the context. The \u201cspan<\/strong>\u201d is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. Each component of the distributed system contributes a span — a named, timed operation representing a piece of the workflow.<\/p>\n
      After successful processing, we respond with status code 200 and return the shipment information as a JSON.<\/p>\n
      Function processing<\/code>simulates some internal processing:<\/p>\n
      func processing(c echo.Context, id string) {    sp := jaegertracing.CreateChildSpan(c, \"logistic processing\")    defer sp.Finish()     \/\/ simulate the workload    time.Sleep(time.Millisecond * 10)     sp.LogKV(\"successfully processed shipment\", id) }<\/code><\/pre>\n
      In this function, we added a structured logging<\/strong> record by calling sp.LogKV<\/code>.<\/p>\n
      Function getShipmentFromDatabase<\/code>simulates a database operation:<\/p>\n
      type shipment struct {     ID      string     Status  string     Carrier string }  func getShipmentFromDatabase(c echo.Context, id string) shipment {    sp := jaegertracing.CreateChildSpan(c, \"SQL select\")    defer sp.Finish()    sp.SetTag(\"sql.query\", fmt.Sprintf(\"SELECT * FROM shipment WHERE id = %s\", id))     time.Sleep(time.Millisecond * 30) \/\/ simulate a select operation    return shipment{ID: id, Status: \"dispatched\"} }<\/code><\/pre>\n
      In this function, we added a tag<\/strong> to the span by calling sp.SetTag<\/code> When should we use a tag and when should we use a logging record? Usually, a tag describes some characteristic that is true during the whole span. While a logging record describes an event that happened at a particular timestamp. <\/p>\n
      Now we have to fetch the status of the shipment from service tracking<\/code> How do we pass the tracking info across the boundaries between the services? Our GlobalTracer<\/code>singleton injects the span information into the headers of the request to tracking<\/code> service:<\/p>\n
      func fetchStatus(c echo.Context, id string) string {    sp := jaegertracing.CreateChildSpan(c, \"fetch status from tracking service\")    defer sp.Finish()     url := fmt.Sprintf(\"http:\/\/localhost:8080\/tracking\/%s\", id)    req, err := http.NewRequest(http.MethodGet, url, nil)    if err != nil {       panic(err)    }     opentracing.GlobalTracer().Inject(       sp.Context(),       opentracing.HTTPHeaders,       opentracing.HTTPHeadersCarrier(req.Header))     res, err := http.DefaultClient.Do(req)    if err != nil {       panic(err)    }    defer res.Body.Close()     status, err := ioutil.ReadAll(res.Body)    if err != nil {       panic(err)    }     return string(status) }<\/code><\/pre>\n
      Specifically, the tracer package added the following header to the request to tracking service.<\/p>\n
      Uber-Trace-Id: 679420a750136ab6:413a9aeeefd2abe8:679420a750136ab6:1<\/code><\/pre>\n
      With this information tracking<\/code>service is able to continue the trace.<\/p>\n
      Service tracking<\/h3>\n
      tracking<\/code>service is quite similar to logistic<\/code>service. Here it’s full listing:<\/p>\n
      package main  import (    \"fmt\"    \"io\/ioutil\"    \"net\/http\"    \"os\"    \"time\"     \"github.com\/labstack\/echo-contrib\/jaegertracing\"    \"github.com\/labstack\/echo\/v4\"    \"github.com\/opentracing\/opentracing-go\" )  const (    SERVICE = \"logistics\" )  func main() {    e := echo.New()    os.Setenv(\"JAEGER_SERVICE_NAME\", SERVICE)     \/\/ Enable tracing middleware    c := jaegertracing.New(e, nil)    defer c.Close()     e.GET(\"\/shipment\/:id\", func(c echo.Context) error {       sp := jaegertracing.CreateChildSpan(c, \"get shipment request\")       defer sp.Finish()        id := c.Param(\"id\")        processing(c, id)       shipment := getShipmentFromDatabase(c, id)       shipment.Status = fetchStatus(c, id)       time.Sleep(time.Millisecond * 15)        return c.JSON(http.StatusOK, shipment)    })    e.Logger.Fatal(e.Start(\":1323\")) }  type shipment struct {    ID      string    Status  string    Carrier string }  \/\/ simulate some processing func processing(c echo.Context, id string) {    sp := jaegertracing.CreateChildSpan(c, \"logistic processing\")    defer sp.Finish()     \/\/ simulate the workload    time.Sleep(time.Millisecond * 10)     sp.LogKV(\"successfully processed shipment\", id) }  \/\/ database dependency func getShipmentFromDatabase(c echo.Context, id string) shipment {    sp := jaegertracing.CreateChildSpan(c, \"SQL select\")    defer sp.Finish()    sp.SetTag(\"sql.query\", fmt.Sprintf(\"SELECT * FROM shipment WHERE id = %s\", id))     time.Sleep(time.Millisecond * 30) \/\/ simulate a select operation    return shipment{ID: id, Status: \"dispatched\", Carrier: \"UPS\"} }  \/\/ external dependency: tracking service func fetchStatus(c echo.Context, id string) string {    sp := jaegertracing.CreateChildSpan(c, \"fetch status from tracking service\")    defer sp.Finish()     url := fmt.Sprintf(\"http:\/\/localhost:8080\/tracking\/%s\", id)    req, err := http.NewRequest(http.MethodGet, url, nil)    if err != nil {       panic(err)    }     opentracing.GlobalTracer().Inject(       sp.Context(),       opentracing.HTTPHeaders,       opentracing.HTTPHeadersCarrier(req.Header))     res, err := http.DefaultClient.Do(req)    if err != nil {       panic(err)    }    defer res.Body.Close()     status, err := ioutil.ReadAll(res.Body)    if err != nil {       panic(err)    }     return string(status) } <\/code><\/pre>\n
      Jaeger<\/h3>\n
      We run an all-in-one Docker container for Jaeger:<\/p>\n
      jaegar:    docker run -d -p6831:6831\/udp -p16686:16686 jaegertracing\/all-in-one:latest<\/code><\/pre>\n
      Jaeger UI will be accessible on port 16686<\/p>\n
      Demo<\/h3>\n
      We issue a GET request to logistic<\/code>service and receive the response:<\/p>\n
      $ http localhost:1323\/shipment\/134324                                                      aynur@aynur-manjaro HTTP\/1.1 200 OK Content-Length: 61 Content-Type: application\/json; charset=UTF-8 Date: Fri, 15 Jan 2021 17:54:28 GMT  {     \"Carrier\": \"UPS\",     \"ID\": \"134324\",     \"Status\": \"dispatched_to_hub\" }  <\/code><\/pre>\n
      Jaeger UI displays all recorded traces for logistic service:<\/p>\n
      <\/figcaption><\/figure>\n
      We search a trace for the requested shipment by tag shipment_id (search panel on the right):<\/p>\n<\/p>\n
      <\/figcaption><\/figure>\n<\/p>\n
      The trace contains in-depth information about the request\u2019s life cycle:<\/p>\n<\/p>\n
      <\/figcaption><\/figure>\n<\/p>\n
      The information can also be represented as a graph:<\/p>\n
      <\/figcaption><\/figure>\n
      Conclusion<\/h2>\n
      Distributed tracing is a powerful method that used to profile and monitor applications, especially those built using a microservices architecture. As an individual squad doesn\u2019t have too many microservices, the introduction of the tracing would be most beneficial on a company-wide level or at least on a department level.<\/p>\n
      <\/h2>\n<\/p>\n<\/div>\n<\/div>\n<\/div>\n
      <\/div>\n

       \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438  https:\/\/habr.com\/ru\/articles\/539022\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
      <\/div>\n
      \n
      \n
      \n
      \n
      What is distributed tracing? Distributed tracing<\/a> is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.<\/p>\n
      Let\u2019s have a look at a simple prototype. A user fetches information about a shipment from logistic<\/code>service. logistic<\/code>service does some computation and fetches the data from a database. logistic service doesn\u2019t know the actual status of the shipment, so it has to fetch the updated status from another service tracking<\/code> tracking<\/code>service also needs to fetch the data from a database and to do some computation.<\/p>\n
      In the screenshot below, we see a whole life cycle of the request issued to logistics<\/code>service:<\/p>\n
      <\/figcaption><\/figure>\n
        \n
      1. \n
        GET request hits endpoint \/shipment\/:id<\/code>in the service logistics<\/code><\/p>\n<\/li>\n
      2. \n
        The service does some internal processing that takes 10 ms (span logistics processing<\/code>.<\/p>\n<\/li>\n
      3. \n
        The service fetches data from the database for 30 ms (span SQL select<\/code>. <\/p>\n<\/li>\n
      4. \n
        logistics<\/code>fetches the status of the shipment from another service tracking by sending a GET request to endpoint \/tracking:\/:id<\/code>(span tracking<\/code>. We have detailed information about this part of the journey as well, i.e. how much time tracking<\/code>service used for the processing or how much time the service used for the database operation.<\/p>\n<\/li>\n<\/ol>\n
        From the screen below, we can see that the request was successful and returned 200. <\/p>\n
        <\/figcaption><\/figure>\n
        It is especially handy as we didn\u2019t log the response code or created the metadata explicitly: it comes for free from the tracing middleware. <\/p>\n
        Let\u2019s say we want to profile our logistics service. As we can see, we spend most of the time on the database operation. It would be helpful to see the exact query we issued:<\/p>\n
        <\/figcaption><\/figure>\n
        Now we can check whether the query was optimal and which tables it affected. The span contains the information about the caller, including the line number and file name, so it is easy to navigate in the source code.<\/p>\n
        Speaking in general, we can profile any part of the code be it a module or a single function. Such data helps to spot the bottlenecks and make informed decisions about the optimization. <\/p>\n
        Tracing also automatically creates a snapshot of our architecture (DAG):<\/p>\n
        <\/figcaption><\/figure>\n<\/p>\n
        Obviously, the picture above doesn\u2019t look that impressive. But what happens when we have not 2 services, but 30? Let\u2019s say a user logs in and we need to fetch the information from services such as users,<\/code> settings,<\/code> billing,<\/code>etc. In that case, this diagram would be very handy to understand the actual interactions between our services. The tracing would also reveal which of our services takes the most time or which one is the least reliable. <\/p>\n
        Tracing also helps to debug the problems in the microservice architecture. Let\u2019s say each of our microservices is up and behaves well on an individual level. However, we have a problem on a system-wide level. I.e. there is a particular combination of input and output data between the microservices that causes a problem. In that case, individual logs from each of the microservices won\u2019t reveal a problem. We need to tie all the logs from the affected microservices and see the whole picture. Tracing is the right instrument to solve such kind of problems. <\/p>\n
        Distributed Tracing APIs<\/h2>\n
        OpenTracing<\/h3>\n
        What is OpenTracing? It\u2019s a vendor-agnostic API to help developers easily instrument tracing into their\u00a0codebase. This means that if a developer wants to try out a different distributed tracing system, then instead of repeating the whole instrumentation process for the new distributed tracing system, the developer can simply change the configuration of the Tracer.<\/p>\n
        Most mental models for tracing descend from Google\u2019s Dapper paper. OpenTracing uses similar nouns and verbs (source<\/a>):<\/p>\n
        <\/figcaption><\/figure>\n
          \n
        1. \n
          Trace<\/strong>: The description of a transaction as it moves through a distributed system.<\/p>\n<\/li>\n
        2. \n
          Span<\/strong>: A named, timed operation representing a piece of the workflow. Spans accept key:value tags as well as fine-grained, timestamped, structured logs attached to the particular span instance.<\/p>\n<\/li>\n
        3. \n
          Span context<\/strong>: Trace information that accompanies the distributed transaction, including when it passes the service to service over the network or through a message bus. The span context contains the trace identifier, span identifier, and any other data that the tracing system needs to propagate to the downstream service.<\/p>\n<\/li>\n<\/ol>\n
          There are several tracers that support<\/a> OpenTracing API, including Jaeger, Instana, and Elastic APM. <\/p>\n
          OpenTelemetry<\/h3>\n
          OpenTelemetry<\/a> is a set of APIs, SDKs, tooling and integrations that are designed for the creation and management of\u00a0telemetry data<\/em>\u00a0such as traces, metrics, and logs. OpenTelemetry is a CNCF Sandbox member, formed through a merger of the OpenTracing and OpenCensus projects.<\/p>\n
          OpenTelemetry implementation for Go is in pre-GA phase. As of today (15.01.2021), the estimated<\/a> percentage of completion for the first release candidate is 72%.<\/p>\n
          <\/figcaption><\/figure>\n
          One of the three main components — logging have not yet been implemented<\/a>:<\/p>\n
          <\/figcaption><\/figure>\n
          As OpenTelemetry implementation in Go hasn\u2019t yet matured, this proposal uses OpenTracing standard. Both standards use the same terminology and concepts, so the migration to newer OpenTelemetry should be relatively straightforward. <\/p>\n
          Tracers<\/h2>\n
          Jaeger<\/h3>\n
          The Jaeger<\/a> tracing system is an open-source tracing system for microservices, and it supports the\u00a0OpenTracing<\/u><\/a>\u00a0standard. Jaeger was initially published as open source by Uber Technologies and has evolved since then. The system gives you distributing tracing, root cause analysis, service dependency analysis, and more.<\/p>\n
          New Relic APM<\/h3>\n
          New Relic APM (Application Performance Monitoring)  provides monitoring of application\u2019s performance and has support for distributed tracing.<\/p>\n
          New Relic ATM uses the same terminology as OpenTracing with the addition of transaction. At New Relic, a\u00a0transaction<\/strong>\u00a0is defined as one logical unit of work in a software application. Specifically, it refers to the function calls and method calls that make up that unit of work.\u00a0For APM, it will often refer to a\u00a0web transaction<\/strong>, which represents an activity that happens from when the application receives a web request to when the response is sent.<\/p>\n
          New Relic APM has support for the instrumentation of Go applications.<\/a> The API can be found here<\/a>.<\/p>\n
          New Relic also provides open-source integrations that report telemetry data from telemetry tools to a New Relic account. In particular, it provides an exporter<\/a> for sending OpenTelemetry data to New Relic. <\/p>\n
          Grafana Tempo<\/h3>\n
          Grafana Tempo is a tracing backend that implements OpenTelemetry\/OpenTracing standards. Here<\/a> is example instrumentation with opentelementry<\/code>library.<\/p>\n
          Sentry<\/h3>\n
          Sentry is an application monitoring system with support for distributed tracing<\/a>. The API can be found here<\/a>.<\/p>\n
          Prototype<\/h2>\n
          Prototype microservices were written in Go and are using OpenTracing API. <\/p>\n
          Service logistics<\/h3>\n
          Service logistics<\/code>exposes a REST endpoint \/shipment\/:id.<\/code>We use echo<\/code>library that has excellent support for the tracing middleware. Note, though, that echo<\/code>is not a hard requirement — it is possible to write our own tracing middleware and achieve the same result.<\/p>\n
          We instantiate a logging middleware that automatically logs a response code and some metadata. <\/p>\n
          SERVICE = \"logistics\"  e := echo.New() os.Setenv(\"JAEGER_SERVICE_NAME\", SERVICE)  \/\/ Enable tracing middleware c := jaegertracing.New(e, nil) defer c.Close()<\/code><\/pre>\n
          Create a simple handler: <\/p>\n
          e.GET(\"\/shipment\/:id\", func(c echo.Context) error {    sp := jaegertracing.CreateChildSpan(c, \"get shipment request\")    defer sp.Finish()     id := c.Param(\"id\")     processing(c, id)    shipment := getShipmentFromDatabase(c, id)    shipment.Status = fetchStatus(c, id)    time.Sleep(time.Millisecond * 15)     return c.JSON(http.StatusOK, shipment) })<\/code><\/pre>\n
          We create the child span sp<\/code>from the context. The \u201cspan<\/strong>\u201d is the primary building block of a distributed trace, representing an individual unit of work done in a distributed system. Each component of the distributed system contributes a span — a named, timed operation representing a piece of the workflow.<\/p>\n
          After successful processing, we respond with status code 200 and return the shipment information as a JSON.<\/p>\n
          Function processing<\/code>simulates some internal processing:<\/p>\n
          func processing(c echo.Context, id string) {    sp := jaegertracing.CreateChildSpan(c, \"logistic processing\")    defer sp.Finish()     \/\/ simulate the workload    time.Sleep(time.Millisecond * 10)     sp.LogKV(\"successfully processed shipment\", id) }<\/code><\/pre>\n
          In this function, we added a structured logging<\/strong> record by calling sp.LogKV<\/code>.<\/p>\n
          Function getShipmentFromDatabase<\/code>simulates a database operation:<\/p>\n
          type shipment struct {     ID      string     Status  string     Carrier string }  func getShipmentFromDatabase(c echo.Context, id string) shipment {    sp := jaegertracing.CreateChildSpan(c, \"SQL select\")    defer sp.Finish()    sp.SetTag(\"sql.query\", fmt.Sprintf(\"SELECT * FROM shipment WHERE id = %s\", id))     time.Sleep(time.Millisecond * 30) \/\/ simulate a select operation    return shipment{ID: id, Status: \"dispatched\"} }<\/code><\/pre>\n
          In this function, we added a tag<\/strong> to the span by calling sp.SetTag<\/code> When should we use a tag and when should we use a logging record? Usually, a tag describes some characteristic that is true during the whole span. While a logging record describes an event that happened at a particular timestamp. <\/p>\n
          Now we have to fetch the status of the shipment from service tracking<\/code> How do we pass the tracking info across the boundaries between the services? Our GlobalTracer<\/code>singleton injects the span information into the headers of the request to tracking<\/code> service:<\/p>\n
          func fetchStatus(c echo.Context, id string) string {    sp := jaegertracing.CreateChildSpan(c, \"fetch status from tracking service\")    defer sp.Finish()     url := fmt.Sprintf(\"http:\/\/localhost:8080\/tracking\/%s\", id)    req, err := http.NewRequest(http.MethodGet, url, nil)    if err != nil {       panic(err)    }     opentracing.GlobalTracer().Inject(       sp.Context(),       opentracing.HTTPHeaders,       opentracing.HTTPHeadersCarrier(req.Header))     res, err := http.DefaultClient.Do(req)    if err != nil {       <\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-388655","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/388655","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=388655"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/388655\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=388655"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=388655"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=388655"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}