Black Lives Matter. Support the Equal Justice Initiative.

The Go Blog

The Go Collective on Stack Overflow

Steve Francia
23 June 2021

Since the earliest days of Go, Stack Overflow has been a significant part of the Go user experience. For the past five years, the Go user survey has consistently identified Stack Overflow as the #1 place users go to find answers to their Go questions.

We are excited to share the Go Collective, the very first open source project in Collectives™ on Stack Overflow, which came as a result of a partnership between Go & Stack Overflow.

Go Collective on Stack Overflow web page

Stack Overflow’s mission for the community is to help make the most out of Stack Overflow as a tool for knowledge reusability. For the millions of users in the Go community, the Go Collective will be an improved Stack Overflow experience. It will retain the great question and answer experience we’ve all come to appreciate from Stack Overflow. But it will also provide a home for gophers and an opportunity for Go community leaders to have a voice and help establish best practices.

The benefits of the Go Collective will be:

  • Find trusted answers faster: we understand developers often have to test multiple solutions to a problem before finding the right one. With the Go Collective, you’ll now see answers that are recommended by the Go team or provided by Recognized Members, meaning Go team members, Go Google Developer Experts, and Stack Overflow users who have been recognized by Stack Overflow as subject matter experts on Go ecosystem technologies. These users will have a badge identifying them as Recognized Members when they post or edit a question, provide an answer, or write an article.

  • Get product information all in one place: the Go Collectives page on Stack Overflow centralizes all of the questions and answers and content associated with Go and related tags. There will be links from the collective to critical Go websites, and Go websites will direct viewers to the Go Collective on Stack Overflow.

  • Explore new content formats: stay up-to-date on our technologies and find more in-depth product information with articles, long-form content like how-to guides and announcements that can be found on the Go Collective page.

  • Earn recognition: Collectives on Stack Overflow also gives us a way to recognize you for your contributions to the community. There’s a leaderboard you can access via the Go Collective (see the “Members” tab) that will help identify trusted contributors designated as Recognized Members. If you are interested in becoming a Recognized Member, please email stackoverflow@golang.org.

Today Google Cloud is launching their own collective alongside the Go Collective. We are excited to take this step together: Go is the best language for building cloud infrastructure and applications, and Google Cloud is the best place to run Go applications. And now Stack Overflow is the best place to find answers to your Go and Cloud questions.

To read more about this see Stack Overflow's blog post and Google Cloud's blog post.

Fuzzing is Beta Ready

Katie Hockman and Jay Conrod
3 June 2021

We are excited to announce that native fuzzing is ready for beta testing in its development branch, dev.fuzz!

Fuzzing is a type of automated testing which continuously manipulates inputs to a program to find issues such as panics or bugs. These semi-random data mutations can discover new code coverage that existing unit tests may miss, and uncover edge case bugs which would otherwise go unnoticed. Since fuzzing can reach these edge cases, fuzz testing is particularly valuable for finding security exploits and vulnerabilities.

See golang.org/s/draft-fuzzing-design for more details about this feature.

Getting started

To get started, you may run the following

$ go get golang.org/dl/gotip
$ gotip download dev.fuzz

This builds the Go toolchain from the dev.fuzz development branch, and won’t be needed once the code is merged to the master branch in the future. After running this, gotip can act as a drop-in replacement for the go command. You can now run commands like

$ gotip test -fuzz=FuzzFoo

There will be ongoing development and bug fixes in the dev.fuzz branch, so you should regularly run gotip download dev.fuzz to use the latest code.

For compatibility with released versions of Go, use the gofuzzbeta build tag when committing source files containing fuzz targets to your repository. This tag is enabled by default at build-time in the dev.fuzz branch. See the go command documentation about build tags if you have questions about how to use them.

// +build gofuzzbeta

Writing a fuzz target

A fuzz target must be in a *_test.go file as a function in the form FuzzXxx. This function must be passed a *testing.F argument, much like a *testing.T argument is passed to a TestXxx function.

Below is an example of a fuzz target that’s testing the behavior of the net/url package.

// +build gofuzzbeta

package fuzz

import (
    "net/url"
    "reflect"
    "testing"
)

func FuzzParseQuery(f *testing.F) {
    f.Add("x=1&y=2")
    f.Fuzz(func(t *testing.T, queryStr string) {
        query, err := url.ParseQuery(queryStr)
        if err != nil {
            t.Skip()
        }
        queryStr2 := query.Encode()
        query2, err := url.ParseQuery(queryStr2)
        if err != nil {
            t.Fatalf("ParseQuery failed to decode a valid encoded query %s: %v", queryStr2, err)
        }
        if !reflect.DeepEqual(query, query2) {
            t.Errorf("ParseQuery gave different query after being encoded\nbefore: %v\nafter: %v", query, query2)
        }
    })
}

You can read more about the fuzzing APIs with go doc

gotip doc testing
gotip doc testing.F
gotip doc testing.F.Add
gotip doc testing.F.Fuzz

Expectations

This is a beta release in a development branch, so you should expect some bugs and an incomplete feature set. Check the issue tracker for issues labelled “fuzz” to stay up-to-date on existing bugs and missing features.

Please be aware that fuzzing can consume a lot of memory and may impact your machine’s performance while it runs. go test -fuzz defaults to running fuzzing in $GOMAXPROCS processes in parallel. You may lower the number of processes used while fuzzing by explicitly setting the -parallel flag with go test. Read the documentation for the go test command by running gotip help testflag if you want more information.

Also be aware that the fuzzing engine writes values that expand test coverage to a fuzz cache directory within $GOCACHE/fuzz while it runs. There is currently no limit to the number of files or total bytes that may be written to the fuzz cache, so it may occupy a large amount of storage (ie. several GBs). You can clear the fuzz cache by running gotip clean -fuzzcache.

What’s next?

This feature will not be available in the upcoming Go release (1.17), but there are plans to land this in a future Go release. We hope that this working prototype will allow Go developers to start writing fuzz targets and provide helpful feedback about the design in preparation for a merge to master.

If you experience any problems or have an idea for a feature request, please file an issue.

For discussion and general feedback about the feature, you can also participate in the #fuzzing channel in Gophers Slack.

Happy fuzzing!

Go Developer Survey 2020 Results

Alice Merrick
9 March 2021

Thank you for the amazing response!

In 2020, we had another great turnout with 9,648 responses, about as many as 2019. Thank you for putting in the time to provide the community with these insights on your experiences using Go!

New modular survey design

You may notice some questions have smaller sample sizes ("n=") than others. That's because some questions were shown to everyone while others were only shown to a random subset of respondents.

Highlights

Who did we hear from?

Demographic questions help us distinguish which year-over-year differences may result from changes in who responded to the survey versus changes in sentiment or behavior. Because our demographics are similar to last year, we can be reasonably confident that other year-over-year changes aren't primarily due to demographic shifts.

For example, the distribution of organization sizes, developer experience, and industries remained about the same from 2019 to 2020.

Bar chart of organization size for 2019 to 2020 where the majority have fewer than 1000 employees Bar chart of years of professional experience for 2019 to 2020 with the majority having 3 to 10 years of experience Bar chart of organization industries for 2019 to 2020 with the majority in Technology

Almost half (48%) of respondents have been using Go for less than two years. In 2020, we had fewer responses from those using Go for less than a year.

Bar chart of years of experience using Go

Majorities said they use Go at work (76%) and outside of work (62%). The percentage of respondents using Go at work has been trending up each year.

Bar chart where Go is being used at work or outside of work

This year we introduced a new question on primary job responsibilities. We found that 70% of respondents’ primary responsibility is developing software and applications, but a significant minority (10%) are designing IT systems and architectures.

Primary job responsibilities

As in prior years, we found that most respondents are not frequent contributors to Go open-source projects, with 75% saying they do so "infrequently" or "never".

How often respondents contribute to open source projects written in Go from 2017 to 2020 where results remain about the same each year and only 7% contribute daily

Developer tools and practices

As in prior years, the vast majority of survey respondents reported working with Go on Linux (63%) and macOS (55%) systems. The proportion of respondents who primarily develop on Linux appears to be slightly trending down over time.

Primary operating system from 2017 to 2020

For the first time, editor preferences appear to have stabilized: VS Code remains the most preferred editor (41%), with GoLand a strong second (35%). Together these editors made up 76% of responses, and other preferences did not continue to decrease as they had in previous years.

Editor preferences from 2017 to 2020

This year we asked respondents to prioritize improvements to their editor by how much they would hypothetically spend if they had 100 “GopherCoins” (a fictional currency). Code completion received the highest average number of GopherCoins per respondent. Half of respondents gave the top 4 features (code completion, navigating code, editor performance and refactoring) 10 or more coins.

Bar char of average number of GopherCoins spent per respondent

A majority of respondents (63%) spend 10–30% of their time refactoring, suggesting that this is a common task and we want to investigate ways to improve it. It also explains why refactoring support was one of the most-funded editor improvements.

Bar chart of time spent refactoring

Last year we asked about specific developer techniques and found that almost 90% of respondents were using text logging for debugging, so this year we added a follow-up question to find out why. Results show that 43% use it because it allows them to use the same debugging strategy across different languages, and 42% prefer to use text logging over other debugging techniques. However, 27% don't know how to get started with Go's debugging tools and 24% have never tried using Go's debugging tools, so there's an opportunity to improve the debugger tooling in terms of discoverability, usability and documentation. Additionally, because a quarter of respondents have never tried using debugging tools, pain points may be underreported.

Sentiments towards Go

For the first time, this year we asked about overall satisfaction. 92% of respondents said they were very or somewhat satisfied using Go during the past year.

Bar chart of overall satisfaction on a 5 points scale from very dissatisfied to very satisfied

This is the 3rd year we've asked the "Would you recommend…" Net Promoter Score (NPS) question. This year our NPS result is a 61 (68% "promoters" minus 6% "detractors"), statistically unchanged from 2019 and 2018.

Stacked bar chart of promoters, passives, and detractors

Similar to previous years, 91% of respondents said they would prefer to use Go for their next new project. 89% said Go is working well for their team. This year we saw an increase in respondents who agreed that Go is critical to their company’s success from 59% in 2019 to 66% in 2020. Respondents working at organizations of 5,000 or more employees were less likely to agree (63%), while those at smaller organizations were more likely to agree (73%).

Bar chart of agreement with statements I would prefer to use Go for my next project, Go is working well for me team, 89%, and Go is critical to my company's success

Like last year, we asked respondents to rate specific areas of Go development according to satisfaction and importance. Satisfaction with using cloud services, debugging, and using modules (areas that last year were highlighted as opportunities for improvement) increased while most importance scores remained about the same. We also introduced a couple new topics: API and Web frameworks. We see that web frameworks satisfaction is lower than other areas (64%). It wasn't as critically important to most current users (only 28% of respondents said it was very or critically important), but it could be a missing critical feature for would-be Go developers.

Bar chart of satisfaction with aspects of Go from 2019 to 2020, showing highest satisfaction with build speed, reliability and using concurrency and lowest with web frameworks

81% of respondents said they felt very or extremely productive using Go. Respondents at larger organizations were more likely to feel extremely productive than those at smaller organizations.

Stacked bar chart of perceived productivity on 5 point scale from not all to extremely productive

We’ve heard anecdotally that it’s easy to become productive quickly with Go. We asked respondents who felt at least slightly productive how long it took them to become productive. 93% said it took less than one year, with the majority feeling productive within 3 months.

Bar chart of length of time before feeling productive

Although about the same as last year, the percentage of respondents who agreed with the statement "I feel welcome in the Go community" appears to be trending down over time, or at least not holding to the same upward trends as other areas.

We've also seen a significant year-over-year increase in the proportion of respondents who feel Go’s project leadership understands their needs (63%).

All of these results show a pattern of higher agreement correlated with increased Go experience, beginning at about two years. In other words, the longer a respondent has been using Go, the more likely they were to agree with each of these statements.

Bar chart showing agreement with statements I feel welcome in the Go community, I am confident in the Go leadership, I feel welcome to contribute, The Go project leadership understands my needs, and The process of contributing to the Go project is clear to me

We asked an open text question on what we could do to make the Go community more welcoming and the most common recommendations (21%) were related to different forms of or improvements/additions to learning resources and documentation.

Bar chart of recommendations for improving the welcomeness of the Go community

Working with Go

Building API/RPC services (74%) and CLIs (65%) remain the most common uses of Go. We don't see any significant changes from last year, when we introduced randomization into the ordering of options. (Prior to 2019, options towards the beginning of the list were disproportionately selected.) We also broke this out by organization size and found that respondents use Go similarly at large enterprises or smaller organizations, although large orgs are a little less likely to use Go for web services returning HTML.

Bar chart of Go use cases from 2019 to 2020 including API or RPC services, CLIs, frameworks, web services, automation, agents and daemons, data processing, GUIs, games and mobile apps

This year we now have a better understanding of which kinds of software respondents write in Go at home versus at work. Although web services returning HTML is the 4th most common use case, this is due to non-work related use. More respondents use Go for automation/scripts, agents and daemons, and data processing for work than web services returning HTML. A greater proportion of the least common uses (desktop/GUI apps, games, and mobile apps) are being written outside of work.

Stacked bar charts of proportion of use case is at work, outside of work, or both

Another new question asked how satisfied respondents were for each use case. CLIs had the highest satisfaction, with 85% of respondents saying they were very, moderately or slightly satisfied using Go for CLIs. Common uses for Go tended to have higher satisfaction scores, but satisfaction and popularity don’t perfectly correspond. For example, agents and daemons has 2nd highest proportion of satisfaction but it’s 6th in usage.

Bar chart of satisfaction with each use case

Additional follow-up questions explored different use cases, for example, which platforms respondents target with their CLIs. It's not surprising to see Linux (93%) and macOS (59%) highly represented, given the high developer use of Linux and macOS and high Linux cloud usage), but even Windows is targeted by almost a third of CLI developers.

Bar chart of platforms being targeted for CLIs

A closer look at Go for data processing showed that Kafka is the only broadly adopted engine, but a majority of respondents said they use Go with a custom data processing engine.

Bar chart of data processing engines used by those who use Go for data processing

We also asked about larger areas in which respondents work with Go. The most common area by far was web development (68%), but other common areas included databases (46%), DevOps (42%) network programming (41%) and systems programming (40%).

Bar chart of the kind of work where Go is being used

Similar to last year, we found that 76% of respondents evaluate the current Go release for production use, but this year we refined our time scale and found that 60% begin evaluating a new version before or within 2 months of release. This highlights the importance for platform-as-a-service providers to quickly support new stable releases of Go.

Bar chart of how soon respondents begin evaluating a new Go release

Modules

This year we found near-universal adoption for Go modules, and a significant increase in the proportion of respondents who only use modules for package management. 96% of respondents said they were using modules for package management, up from 89% last year. 87% of respondents said they were using only modules for package management, up from 71% last year. Meanwhile, the use of other package management tools has decreased.

Bar chart of methods used for Go package management

Satisfaction with modules also increased from last year. 77% of respondents said they were very, moderately or slightly satisfied with modules, compared to 68% in 2019.

Stacked bar chart of satisfaction with using modules on a 7 point scale from very dissatisfied to very satisfied

Official documentation

Most respondents said they struggle with official documentation. 62% of respondents struggle to find enough information to fully implement a feature of their application and over a third have struggled to get started with something they haven’t done before.

Bar chart of struggles using official Go documentation

The most problematic areas of official documentation were on using modules and CLI development, with 20% of respondents finding modules documentation slightly or not at all helpful, and 16% for documentation around CLI development.

Stacked bar charts on helpfulness of specific areas of documentation including using modules, CLI tool development, error handling, web service development, data access, concurrency and file input/output, rated on a 5 point scale from not at all to very helpful

Go in the clouds

Go was designed with modern distributed computing in mind, and we want to continue to improve the developer experience of building cloud services with Go.

  • The three largest global cloud providers (Amazon Web Services, Google Cloud Platform, and Microsoft Azure) continue to increase in usage among survey respondents, while most other providers are used by a smaller proportion of respondents each year. Azure in particular had a significant increase from 7% to 12%.
  • On-prem deployments to self-owned or company-owned servers continue to decrease as the most common deployment targets.
Bar chart of cloud providers used to deploy Go programs where AWS is the most common at 44%

Respondents deploying to AWS and Azure saw increases in deploying to a managed Kubernetes platform, now at 40% and 54%, respectively. Azure saw a significant drop in the proportion of users deploying Go programs to VMs and some growth in container usage from 18% to 25%. Meanwhile, GCP (which already had a high proportion of respondents reporting managed Kubernetes use) saw some growth in deploying to serverless Cloud Run from 10% to 17%.

Bar charts of proportion of services being used with each provider

Overall, a majority of respondents were satisfied with using Go on all three major cloud providers, and the figures are statistically unchanged from last year. Respondents reported similar satisfaction levels with Go development for AWS (82% satisfied) and GCP (80%). Azure received a lower satisfaction score (58% satisfied), and free-text responses often cited a need for improvements to Azure's Go SDK and Go support for Azure functions.

Stacked bar chart of satisfaction with using Go with AWS, GCP and Azure

Pain points

The top reasons respondents say they are unable to use Go more remain working on a project in another language (54%), working on a team that prefers to use another language (34%), and the lack of a critical feature in Go itself (26%).

This year we introduced a new option, “I already use Go everywhere I would like to,” so that respondents could opt out of making selections that don't prevent them from using Go. This significantly lowered the rate of selection of all other options, but did not change their relative ordering. We also introduced an option for “Go lacks critical frameworks”.

If we look at only the respondents who selected reasons for not using Go, we can get a better idea of year-over-year trends. Working on an existing project in another language and project/team/lead preference for another language are decreasing over time.

Bar charts of reasons preventing respondents from using Go more

Among the 26% of respondents who said Go lacks language features they need, 88% selected generics as a critical missing feature. Other critical missing features were better error handling (58%), null safety (44%), functional programming features(42%) and a stronger / expanded type system (41%).

To be clear, these numbers are from the subset of respondents who said they would be able to use Go more were it not missing one or more critical features they need, not the entire population of survey respondents. To put that in perspective, 18% of respondents are prevented from using Go because of a lack of generics.

Bar chart of missing critical features

The top challenge respondents reported when using Go was again Go's lack of generics (18%), while modules/package management and problems with learning curve/best practices/docs were both 13%.

Bar chart of biggest challenges respondents face when using Go

The Go community

This year we asked respondents for their top 5 resources for answering their Go-related questions. Last year we only asked for top 3, so the results aren't directly comparable, however, StackOverflow remains the most popular resource at 65%. Reading source code (57%) remains another popular resource while reliance on godoc.org (39%) has significantly decreased. The package discovery site pkg.go.dev is new to the list this year and was a top resource for 32% of respondents. Respondents who use pkg.go.dev are more likely to agree they are able to quickly find Go packages / libraries they need: 91% for pkg.go.dev users vs. 82% for everyone else.

Bar chart of top 5 resources respondents use to answer Go-related questions

Over the years, the proportion of respondents who do not attend any Go-related events has been trending up. Due to Covid-19, this year we modified our question around Go events, and found over a quarter of respondents have spent more time in online Go channels than in prior years, and 14% attended a virtual Go meetup, twice as many as last year. 64% of those who attended a virtual event said this was their first virtual event.

Bar chart of respondents participation in online channels and events

We found 12% of respondents identify with a traditionally underrepresented group (e.g., ethnicity, gender identity, et al.), the same as 2019, and 2% identify as women, fewer than in 2019 (3%). Respondents who identified with underrepresented groups showed higher rates of disagreement with the statement "I feel welcome in the Go community" (10% vs. 4%) than those who do not identify with an underrepresented group. These questions allow us to measure diversity in the community and highlight opportunities for outreach and growth.

Bar chart of underrepresented groups Bar chart of those who identify as women Bar chart of welcomeness of underrepresented groups

We added an additional question this year on assistive technology usage, and found that 8% of respondents are using some form of assistive technology. The most commonly used assistive tech was contrast or color settings (2%). This is a great reminder that we have users with accessibility needs and helps drive some of our design decisions on websites managed by the Go team.

Bar chart of assistive technology usage

The Go team values diversity and inclusion, not simply as the right thing to do, but because diverse voices can illuminate our blindspots and ultimately benefit all users. The way we ask about sensitive information, including gender and traditionally underrepresented groups, has changed according to data privacy regulations and we hope to make these questions, particularly around gender diversity, more inclusive in the future.

Conclusion

Thank you for joining us in reviewing the results of our 2020 developer survey! Understanding developers’ experiences and challenges helps us measure our progress and directs the future of Go. Thanks again to everyone who contributed to this survey—we couldn't have done it without you. We hope to see you next year!

Contexts and structs

Jean de Klerk, Matt T. Proud
24 February 2021

Introduction

In many Go APIs, especially modern ones, the first argument to functions and methods is often context.Context. Context provides a means of transmitting deadlines, caller cancellations, and other request-scoped values across API boundaries and between processes. It is often used when a library interacts — directly or transitively — with remote servers, such as databases, APIs, and the like.

The documentation for context states:

Contexts should not be stored inside a struct type, but instead passed to each function that needs it.

This article expands on that advice with reasons and examples describing why it's important to pass Context rather than store it in another type. It also highlights a rare case where storing Context in a struct type may make sense, and how to do so safely.

Prefer contexts passed as arguments

To understand the advice to not store context in structs, let's consider the preferred context-as-argument approach:

type Worker struct { /* … */ }

type Work struct { /* … */ }

func New() *Worker {
  return &Worker{}
}

func (w *Worker) Fetch(ctx context.Context) (*Work, error) {
  _ = ctx // A per-call ctx is used for cancellation, deadlines, and metadata.
}

func (w *Worker) Process(ctx context.Context, work *Work) error {
  _ = ctx // A per-call ctx is used for cancellation, deadlines, and metadata.
}

Here, the (*Worker).Fetch and (*Worker).Process methods both accept a context directly. With this pass-as-argument design, users can set per-call deadlines, cancellation, and metadata. And, it's clear how the context.Context passed to each method will be used: there's no expectation that a context.Context passed to one method will be used by any other method. This is because the context is scoped to as small an operation as it needs to be, which greatly increases the utility and clarity of context in this package.

Storing context in structs leads to confusion

Let's inspect again the Worker example above with the disfavored context-in-struct approach. The problem with it is that when you store the context in a struct, you obscure lifetime to the callers, or worse intermingle two scopes together in unpredictable ways:

type Worker struct {
  ctx context.Context
}

func New(ctx context.Context) *Worker {
  return &Worker{ctx: ctx}
}

func (w *Worker) Fetch() (*Work, error) {
  _ = w.ctx // A shared w.ctx is used for cancellation, deadlines, and metadata.
}

func (w *Worker) Process(work *Work) error {
  _ = w.ctx // A shared w.ctx is used for cancellation, deadlines, and metadata.
}

The (*Worker).Fetch and (*Worker).Process method both use a context stored in Worker. This prevents the callers of Fetch and Process (which may themselves have different contexts) from specifying a deadline, requesting cancellation, and attaching metadata on a per-call basis. For example: the user is unable to provide a deadline just for (*Worker).Fetch, or cancel just the (*Worker).Process call. The caller's lifetime is intermingled with a shared context, and the context is scoped to the lifetime where the Worker is created.

The API is also much more confusing to users compared to the pass-as-argument approach. Users might ask themselves:

  • Since New takes a context.Context, is the constructor doing work that needs cancelation or deadlines?
  • Does the context.Context passed in to New apply to work in (*Worker).Fetch and (*Worker).Process? Neither? One but not the other?

The API would need a good deal of documentation to explicitly tell the user exactly what the context.Context is used for. The user might also have to read code rather than being able to rely on the structure of the API conveys.

And, finally, it can be quite dangerous to design a production-grade server whose requests don't each have a context and thus can't adequately honor cancellation. Without the ability to set per-call deadlines, your process could backlog and exhaust its resources (like memory)!

Exception to the rule: preserving backwards compatibility

When Go 1.7 — which introduced context.Context — was released, a large number of APIs had to add context support in backwards compatible ways. For example, net/http's Client methods, like Get and Do, were excellent candidates for context. Each external request sent with these methods would benefit from having the deadline, cancellation, and metadata support that came with context.Context.

There are two approaches for adding support for context.Context in backwards compatible ways: including a context in a struct, as we'll see in a moment, and duplicating functions, with duplicates accepting context.Context and having Context as their function name suffix. The duplicate approach should be preferred over the context-in-struct, and is further discussed in Keeping your modules compatible. However, in some cases it's impractical: for example, if your API exposes a large number of functions, then duplicating them all might be infeasible.

The net/http package chose the context-in-struct approach, which provides a useful case study. Let's look at net/http's Do. Prior to the introduction of context.Context, Do was defined as follows:

func (c *Client) Do(req *Request) (*Response, error)

After Go 1.7, Do might have looked like the following, if not for the fact that it would break backwards compatibility:

func (c *Client) Do(ctx context.Context, req *Request) (*Response, error)

But, preserving the backwards compatibility and adhering to the Go 1 promise of compatibility is crucial for the standard library. So, instead, the maintainers chose to add a context.Context on the http.Request struct in order to allow support context.Context without breaking backwards compatibility:

type Request struct {
  ctx context.Context

  // ...
}

func NewRequestWithContext(ctx context.Context, method, url string, body io.Reader) (*Request, error) {
  // Simplified for brevity of this article.
  return &Request{
    ctx: ctx,
    // ...
  }
}

func (c *Client) Do(req *Request) (*Response, error)

When retrofitting your API to support context, it may make sense to add a context.Context to a struct, as above. However, remember to first consider duplicating your functions, which allows retrofitting context.Context in a backwards compatibility without sacrificing utility and comprehension. For example:

func (c *Client) Call() error {
  return c.CallContext(context.Background())
}

func (c *Client) CallContext(ctx context.Context) error {
  // ...
}

Conclusion

Context makes it easy to propagate important cross-library and cross-API information down a calling stack. But, it must be used consistently and clearly in order to remain comprehensible, easy to debug, and effective.

When passed as the first argument in a method rather than stored in a struct type, users can take full advantage of its extensibility in order to build a powerful tree of cancelation, deadline, and metadata information through the call stack. And, best of all, its scope is clearly understood when it's passed in as an argument, leading to clear comprehension and debuggability up and down the stack.

When designing an API with context, remember the advice: pass context.Context in as an argument; don't store it in structs.

New module changes in Go 1.16

Jay Conrod
18 February 2021

We hope you're enjoying Go 1.16! This release has a lot of new features, especially for modules. The release notes describe these changes briefly, but let's explore a few of them in depth.

Modules on by default

The go command now builds packages in module-aware mode by default, even when no go.mod is present. This is a big step toward using modules in all projects.

It's still possible to build packages in GOPATH mode by setting the GO111MODULE environment variable to off. You can also set GO111MODULE to auto to enable module-aware mode only when a go.mod file is present in the current directory or any parent directory. This was previously the default. Note that you can set GO111MODULE and other variables permanently with go env -w:

go env -w GO111MODULE=auto

We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate. If there is a problem preventing you from migrating, please consider filing an issue or an experience report.

No automatic changes to go.mod and go.sum

Previously, when the go command found a problem with go.mod or go.sum like a missing require directive or a missing sum, it would attempt to fix the problem automatically. We received a lot of feedback that this behavior was surprising, especially for commands like go list that don't normally have side effects. The automatic fixes weren't always desirable: if an imported package wasn't provided by any required module, the go command would add a new dependency, possibly triggering upgrades of common dependencies. Even a misspelled import path would result in a (failed) network lookup.

In Go 1.16, module-aware commands report an error after discovering a problem in go.mod or go.sum instead of attempting to fix the problem automatically. In most cases, the error message recommends a command to fix the problem.

$ go build
example.go:3:8: no required module provides package golang.org/x/net/html; to add it:
    go get golang.org/x/net/html
$ go get golang.org/x/net/html
$ go build

As before, the go command may use the vendor directory if it's present (see Vendoring for details). Commands like go get and go mod tidy still modify go.mod and go.sum, since their main purpose is to manage dependencies.

Installing an executable at a specific version

The go install command can now install an executable at a specific version by specifying an @version suffix.

go install golang.org/x/tools/gopls@v0.6.5

When using this syntax, go install installs the command from that exact module version, ignoring any go.mod files in the current directory and parent directories. (Without the @version suffix, go install continues to operate as it always has, building the program using the version requirements and replacements listed in the current module’s go.mod.)

We used to recommend go get -u program to install an executable, but this use caused too much confusion with the meaning of go get for adding or changing module version requirements in go.mod. And to avoid accidentally modifying go.mod, people started suggesting more complex commands like:

cd $HOME; GO111MODULE=on go get program@latest

Now we can all use go install program@latest instead. See go install for details.

In order to eliminate ambiguity about which versions are used, there are several restrictions on what directives may be present in the program's go.mod file when using this install syntax. In particular, replace and exclude directives are not allowed, at least for now. In the long term, once the new go install program@version is working well for enough use cases, we plan to make go get stop installing command binaries. See issue 43684 for details.

Module retraction

Have you ever accidentally published a module version before it was ready? Or have you discovered a problem right after a version was published that needed to be fixed quickly? Mistakes in published versions are difficult to correct. To keep module builds deterministic, a version cannot be modified after it is published. Even if you delete or change a version tag, proxy.golang.org and other proxies probably already have the original cached.

Module authors can now retract module versions using the retract directive in go.mod. A retracted version still exists and can be downloaded (so builds that depend on it won't break), but the go command won’t select it automatically when resolving versions like @latest. go get and go list -m -u will print warnings about existing uses.

For example, suppose the author of a popular library example.com/lib releases v1.0.5, then discovers a new security issue. They can add a directive to their go.mod file like the one below:

// Remote-triggered crash in package foo. See CVE-2021-01234.
retract v1.0.5

Next, the author can tag and push version v1.0.6, the new highest version. After this, users that already depend on v1.0.5 will be notified of the retraction when they check for updates or when they upgrade a dependent package. The notification message may include text from the comment above the retract directive.

$ go list -m -u all
example.com/lib v1.0.0 (retracted)
$ go get .
go: warning: example.com/lib@v1.0.5: retracted by module author:
    Remote-triggered crash in package foo. See CVE-2021-01234.
go: to switch to the latest unretracted version, run:
    go get example.com/lib@latest

For an interactive, browser-based guide, check out Retract Module Versions on play-with-go.dev. See the retract directive docs for syntax details.

Controlling version control tools with GOVCS

The go command can download module source code from a mirror like proxy.golang.org or directly from a version control repository using git, hg, svn, bzr, or fossil. Direct version control access is important, especially for private modules that aren't available on proxies, but it's also potentially a security problem: a bug in a version control tool may be exploited by a malicious server to run unintended code.

Go 1.16 introduces a new configuration variable, GOVCS, which lets the user specify which modules are allowed to use specific version control tools. GOVCS accepts a comma-separated list of pattern:vcslist rules. The pattern is a path.Match pattern matching one or more leading elements of a module path. The special patterns public and private match public and private modules (private is defined as modules matched by patterns in GOPRIVATE; public is everything else). The vcslist is a pipe-separated list of allowed version control commands or the keyword all or off.

For example:

GOVCS=github.com:git,evil.com:off,*:git|hg

With this setting, modules with paths on github.com can be downloaded using git; paths on evil.com cannot be downloaded using any version control command, and all other paths (* matches everything) can be downloaded using git or hg.

If GOVCS is not set, or if a module does not match any pattern, the go command uses this default: git and hg are allowed for public modules, and all tools are allowed for private modules. The rationale behind allowing only Git and Mercurial is that these two systems have had the most attention to issues of being run as clients of untrusted servers. In contrast, Bazaar, Fossil, and Subversion have primarily been used in trusted, authenticated environments and are not as well scrutinized as attack surfaces. That is, the default setting is:

GOVCS=public:git|hg,private:all

See Controlling version control tools with GOVCS for more details.

What's next?

We hope you find these features useful. We're already hard at work on the next set of module features for Go 1.17, particularly lazy module loading, which should make the module loading process faster and more stable. As always, if you run into new bugs, please let us know on the issue tracker. Happy coding!

See the index for more articles.