1// Copyright 2018 The Go Authors. All rights reserved.
2// Use of this source code is governed by a BSD-style
3// license that can be found in the LICENSE file.
6Package packages loads Go packages for inspection and analysis.
8Note: Though this package is ready for widespread use, we may make minor
9breaking changes if absolutely necessary. Any such change will be
10announced on golang-tools@ at least one week before it is committed. No
11more breaking changes will be made after December 1, 2018.
13The Load function takes as input a list of patterns and return a list of Package
14structs describing individual packages matched by those patterns.
15The LoadMode controls the amount of detail in the loaded packages.
17Load passes most patterns directly to the underlying build tool,
18but all patterns with the prefix "query=", where query is a
19non-empty string of letters from [a-z], are reserved and may be
20interpreted as query operators.
22Only two query operators are currently supported, "file" and "pattern".
24The query "file=path/to/file.go" matches the package or packages enclosing
25the Go source file path/to/file.go.  For example "file=~/go/src/fmt/print.go"
26might returns the packages "fmt" and "fmt [fmt.test]".
28The query "pattern=string" causes "string" to be passed directly to
29the underlying build tool. In most cases this is unnecessary,
30but an application can use Load("pattern=" + x) as an escaping mechanism
31to ensure that x is not interpreted as a query operator if it contains '='.
33A third query "name=identifier" will be added soon.
34It will match packages whose package declaration contains the specified identifier.
35For example, "name=rand" would match the packages "math/rand" and "crypto/rand",
36and "name=main" would match all executables.
38All other query operators are reserved for future use and currently
39cause Load to report an error.
41The Package struct provides basic information about the package, including
43  - ID, a unique identifier for the package in the returned set;
44  - GoFiles, the names of the package's Go source files;
45  - Imports, a map from source import strings to the Packages they name;
46  - Types, the type information for the package's exported symbols;
47  - Syntax, the parsed syntax trees for the package's source code; and
48  - TypeInfo, the result of a complete type-check of the package syntax trees.
50(See the documentation for type Package for the complete list of fields
51and more detailed descriptions.)
53For example,
55	Load(nil, "bytes", "unicode...")
57returns four Package structs describing the standard library packages
58bytes, unicode, unicode/utf16, and unicode/utf8. Note that one pattern
59can match multiple packages and that a package might be matched by
60multiple patterns: in general it is not possible to determine which
61packages correspond to which patterns.
63Note that the list returned by Load contains only the packages matched
64by the patterns. Their dependencies can be found by walking the import
65graph using the Imports fields.
67The Load function can be configured by passing a pointer to a Config as
68the first argument. A nil Config is equivalent to the zero Config, which
69causes Load to run in LoadFiles mode, collecting minimal information.
70See the documentation for type Config for details.
72As noted earlier, the Config.Mode controls the amount of detail
73reported about the loaded packages, with each mode returning all the data of the
74previous mode with some extra added. See the documentation for type LoadMode
75for details.
77Most tools should pass their command-line arguments (after any flags)
78uninterpreted to the loader, so that the loader can interpret them
79according to the conventions of the underlying build system.
80See the Example function for typical usage.
83package packages // import "golang.org/x/tools/go/packages"
87Motivation and design considerations
89The new package's design solves problems addressed by two existing
90packages: go/build, which locates and describes packages, and
91golang.org/x/tools/go/loader, which loads, parses and type-checks them.
92The go/build.Package structure encodes too much of the 'go build' way
93of organizing projects, leaving us in need of a data type that describes a
94package of Go source code independent of the underlying build system.
95We wanted something that works equally well with go build and vgo, and
96also other build systems such as Bazel and Blaze, making it possible to
97construct analysis tools that work in all these environments.
98Tools such as errcheck and staticcheck were essentially unavailable to
99the Go community at Google, and some of Google's internal tools for Go
100are unavailable externally.
101This new package provides a uniform way to obtain package metadata by
102querying each of these build systems, optionally supporting their
103preferred command-line notations for packages, so that tools integrate
104neatly with users' build environments. The Metadata query function
105executes an external query tool appropriate to the current workspace.
107Loading packages always returns the complete import graph "all the way down",
108even if all you want is information about a single package, because the query
109mechanisms of all the build systems we currently support ({go,vgo} list, and
110blaze/bazel aspect-based query) cannot provide detailed information
111about one package without visiting all its dependencies too, so there is
112no additional asymptotic cost to providing transitive information.
113(This property might not be true of a hypothetical 5th build system.)
115In calls to TypeCheck, all initial packages, and any package that
116transitively depends on one of them, must be loaded from source.
117Consider A->B->C->D->E: if A,C are initial, A,B,C must be loaded from
118source; D may be loaded from export data, and E may not be loaded at all
119(though it's possible that D's export data mentions it, so a
120types.Package may be created for it and exposed.)
122The old loader had a feature to suppress type-checking of function
123bodies on a per-package basis, primarily intended to reduce the work of
124obtaining type information for imported packages. Now that imports are
125satisfied by export data, the optimization no longer seems necessary.
127Despite some early attempts, the old loader did not exploit export data,
128instead always using the equivalent of WholeProgram mode. This was due
129to the complexity of mixing source and export data packages (now
130resolved by the upward traversal mentioned above), and because export data
131files were nearly always missing or stale. Now that 'go build' supports
132caching, all the underlying build systems can guarantee to produce
133export data in a reasonable (amortized) time.
135Test "main" packages synthesized by the build system are now reported as
136first-class packages, avoiding the need for clients (such as go/ssa) to
137reinvent this generation logic.
139One way in which go/packages is simpler than the old loader is in its
140treatment of in-package tests. In-package tests are packages that
141consist of all the files of the library under test, plus the test files.
142The old loader constructed in-package tests by a two-phase process of
143mutation called "augmentation": first it would construct and type check
144all the ordinary library packages and type-check the packages that
145depend on them; then it would add more (test) files to the package and
146type-check again. This two-phase approach had four major problems:
1471) in processing the tests, the loader modified the library package,
148   leaving no way for a client application to see both the test
149   package and the library package; one would mutate into the other.
1502) because test files can declare additional methods on types defined in
151   the library portion of the package, the dispatch of method calls in
152   the library portion was affected by the presence of the test files.
153   This should have been a clue that the packages were logically
154   different.
1553) this model of "augmentation" assumed at most one in-package test
156   per library package, which is true of projects using 'go build',
157   but not other build systems.
1584) because of the two-phase nature of test processing, all packages that
159   import the library package had to be processed before augmentation,
160   forcing a "one-shot" API and preventing the client from calling Load
161   in several times in sequence as is now possible in WholeProgram mode.
162   (TypeCheck mode has a similar one-shot restriction for a different reason.)
164Early drafts of this package supported "multi-shot" operation.
165Although it allowed clients to make a sequence of calls (or concurrent
166calls) to Load, building up the graph of Packages incrementally,
167it was of marginal value: it complicated the API
168(since it allowed some options to vary across calls but not others),
169it complicated the implementation,
170it cannot be made to work in Types mode, as explained above,
171and it was less efficient than making one combined call (when this is possible).
172Among the clients we have inspected, none made multiple calls to load
173but could not be easily and satisfactorily modified to make only a single call.
174However, applications changes may be required.
175For example, the ssadump command loads the user-specified packages
176and in addition the runtime package.  It is tempting to simply append
177"runtime" to the user-provided list, but that does not work if the user
178specified an ad-hoc package such as [a.go b.go].
179Instead, ssadump no longer requests the runtime package,
180but seeks it among the dependencies of the user-specified packages,
181and emits an error if it is not found.
183Overlays: the ParseFile hook in the API permits clients to vary the way
184in which ASTs are obtained from filenames; the default implementation is
185based on parser.ParseFile. This features enables editor-integrated tools
186that analyze the contents of modified but unsaved buffers: rather than
187read from the file system, a tool can read from an archive of modified
188buffers provided by the editor.
189This approach has its limits. Because package metadata is obtained by
190fork/execing an external query command for each build system, we can
191fake only the file contents seen by the parser, type-checker, and
192application, but not by the metadata query, so, for example:
193- additional imports in the fake file will not be described by the
194  metadata, so the type checker will fail to load imports that create
195  new dependencies.
196- in TypeCheck mode, because export data is produced by the query
197  command, it will not reflect the fake file contents.
198- this mechanism cannot add files to a package without first saving them.
200Questions & Tasks
203  They are not portable concepts, but could be made portable.
204  Our goal has been to allow users to express themselves using the conventions
205  of the underlying build system: if the build system honors GOARCH
206  during a build and during a metadata query, then so should
207  applications built atop that query mechanism.
208  Conversely, if the target architecture of the build is determined by
209  command-line flags, the application can pass the relevant
210  flags through to the build system using a command such as:
211    myapp -query_flag="--cpu=amd64" -query_flag="--os=darwin"
212  However, this approach is low-level, unwieldy, and non-portable.
213  GOOS and GOARCH seem important enough to warrant a dedicated option.
215- How should we handle partial failures such as a mixture of good and
216  malformed patterns, existing and non-existent packages, successful and
217  failed builds, import failures, import cycles, and so on, in a call to
218  Load?
220- Support bazel, blaze, and go1.10 list, not just go1.11 list.
222- Handle (and test) various partial success cases, e.g.
223  a mixture of good packages and:
224  invalid patterns
225  nonexistent packages
226  empty packages
227  packages with malformed package or import declarations
228  unreadable files
229  import cycles
230  other parse errors
231  type errors
232  Make sure we record errors at the correct place in the graph.
234- Missing packages among initial arguments are not reported.
235  Return bogus packages for them, like golist does.
237- "undeclared name" errors (for example) are reported out of source file
238  order. I suspect this is due to the breadth-first resolution now used
239  by go/types. Is that a bug? Discuss with gri.