gget

a cli. to get things. from git repos.

$ gget github.com/gohugoio/hugo 'hugo_extended_*_Linux-64bit.deb'
Found 1 file (13.9M) from github.com/gohugoio/hugo@v0.73.0 √ hugo_extended_0.73.0_Linux-64bit.deb done (sha256 OK)
  • Standalone CLI

    No git or local clones required.

  • Public & Private Servers

    Supporting both GitHub and GitLab.

  • Public & Private Repos

    API-based access for private resources.

  • Tags, Branches, and Commits

    Also semver-based constraint matching.

  • Archives, Assets, and Blobs

    Download any type of resource from repos.

  • Built-in File Operations

    Easy rename, binary, and stream options.

  • Checksum Verification

    Verify downloads by SHA (if published).

  • Open-Source Project

    Contribute and discuss at dpb587/gget.

Build with go

Compile into a local executable.

$ git clone https://github.com/dpb587/gget.git
$ cd gget/
$ git checkout v0.5.2
$ go build .

Getting Started

An introduction of the gget concepts.

Pass the Origin

The only required argument to gget is the download origin – it is a shortened URL specifying where to find the repository. By default, the latest version will be discovered and all release assets will be downloaded.

$ gget github.com/dpb587/gget
Found 3 files (33M) from github.com/dpb587/gget@v0.5.2 √ gget-0.5.2-darwin-amd64 done (sha256 OK) √ gget-0.5.2-linux-amd64 done (sha256 OK) √ gget-0.5.2-windows-amd64.exe done (sha256 OK)

If you want something other than latest, add an @REF to the end. For example, to always download from the v0.5.2 release tag you would use github.com/dpb587/gget@v0.5.2.

Name the Resources

To avoid downloading all resources, list only the names you want as additional arguments. Basic glob patterns like wildcards (*) can be used if you don't know the exact name.

$ gget github.com/dpb587/gget '*linux*' gget-0.5.2-darwin-amd64
Found 2 files (22.4M) from github.com/dpb587/gget@v0.5.2 √ gget-0.5.2-darwin-amd64 done (sha256 OK) √ gget-0.5.2-linux-amd64 done (sha256 OK)

When patterns may match more resources than desired, use the --exclude option to restrict further.

$ gget github.com/dpb587/gget --exclude='*-amd64'
Found 1 file (10.6M) from github.com/dpb587/gget@v0.5.2 √ gget-0.5.2-windows-amd64.exe done (sha256 OK)

If an argument does not end up matching anything, the command will error before starting any downloads. If missing resources are expected, the --ignore-missing option may be used to ignore that error.

$ gget github.com/dpb587/gget --ignore-missing '*-darwin-*' '*-macos-*'
Found 1 file (11.7M) from github.com/dpb587/gget@v0.5.2 √ gget-0.5.2-darwin-amd64 done (sha256 OK)

Remote Servers

These examples have been using public repositories from GitHub; but gget also supports GitLab and self-hosted installations of the two. If the service is not automatically detected, the --service option may be used to override the API used to either github or gitlab.

To access private repositories, be sure to configure your access token through the appropriate environment variable or a machine password in your ~/.netrc file. For GitHub, the $GITHUB_TOKEN environment variable is used, and for GitLab it is $GITLAB_TOKEN.

$ export GITLAB_TOKEN=a1b2c3d4...
$ gget --service=gitlab gitlab.acme.corp/my-team/product '*-linux-amd64'
Found 1 file from gitlab.acme.corp/my-team/product@v8.21.2 √ product-8.21.2-darwin-amd64 done

Blobs and Archives

In addition to user-uploaded release assets, gget also supports downloading individual source files from a repository. The --type option can be used to switch to blob-mode.

$ gget github.com/dpb587/gget --type=blob README.md
Found 1 file (6.8K) from github.com/dpb587/gget@v0.5.2 √ README.md done

Alternatively, to get an archive of the entire repository, use the archive type.

$ gget github.com/dpb587/gget --type=archive '*.zip'
Found 1 file from github.com/dpb587/gget@v0.5.2 √ gget-v0.5.2.zip done

For these source types, branches and commits may also be used as a reference.

$ gget github.com/dpb587/gget@d75361c --type=archive '*.zip'
Found 1 file from github.com/dpb587/gget@d75361c1316c6004cde13c04460c57552a839d57 √ gget-d75361c13.zip done

Troubleshooting

If you are unable to find the resources you expect, try verifying gget sees it with the --list option.

$ gget github.com/dpb587/gget --type=archive --list
Found 2 files from github.com/dpb587/gget@v0.5.2 gget-v0.5.2.tar.gz gget-v0.5.2.zip

If gget is erroring or not behaving as you expect, try enabling logging with the -v verbosity flag. If logs are unhelpful or you still have a concern, refer to them when searching and reporting a project issue.

$ gget github.com/dpb587/gget -vvv --no-progress '*linux*'

Advanced

More behaviors and tricks of gget.

Finding Versions

By default, releases tagged as "pre-release" are not used when finding the latest version. To find the latest pre-release, use the --ref-stability option and specify pre-release (or, if you don't care if it is pre-release or stable, use any).

$ gget github.com/dpb587/gget-test --ref-stability=pre-release '*linux*'
Found 1 file (11.7M) from github.com/dpb587/gget-test@v0.5.0-rc.1 √ gget-0.5.0-rc.1-linux-amd64 done (sha256 OK)

Sometimes, instead of a literal version, you may want to use a semver constraint to continue following the latest patches of a version. The --ref-version option allows you to specify a set of version constraints. The first semver-parseable tag which matches will be used.

$ gget github.com/dpb587/gget --ref-version=0.5.x '*linux*'
Found 1 file (11.7M) from github.com/dpb587/gget@v0.5.2 √ gget-0.5.2-linux-amd64 done (sha256 OK)

Manipulating Files

To save a resource under a different local name (or path), prefix it to the resource name with an =.

$ gget github.com/dpb587/gget /tmp/gget='*linux*'
Found 1 file (11.7M) from github.com/dpb587/gget@v0.5.2 √ gget-0.4.0-linux-amd64 done (sha256 OK)

To automatically mark a resource (such as a binary) as executable, use the --executable option.

$ gget github.com/dpb587/gget --executable '*linux*'
Found 1 file (11.7M) from github.com/dpb587/gget@v0.5.2 √ gget-0.4.0-linux-amd64 done (sha256 OK; executable)

When working with archive resources, you may want to use the --stdout option. This causes all downloaded contents to be streamed to STDOUT for redirects or chained commands.

$ gget github.com/dpb587/gget --type=archive '*.tar.gz' --stdout | tar -tzf- | tail -n2
Found 1 file from github.com/dpb587/gget@v0.5.2 √ gget-v0.5.2.tar.gz done dpb587-gget-75447d7/scripts/integration.test.sh dpb587-gget-75447d7/scripts/website.build.sh

Verifying Checksums

By default, gget attempts to discover and verify checksums (based on conventional file names or release notes). If a checksum is not found, no verification is attempted; but if one is found and it fails, the command will error. Use the --verify-checksum option to require a specific algorithm, minimum algorithm strength, or disable the behavior entirely.

$ gget --verify-checksum=sha512-min github.com/dpb587/gget
Found 1 file (11.7M) from github.com/dpb587/gget@v0.5.2 '*linux*' gget: error: preparing transfer of gget-0.4.0-linux-amd64: acceptable checksum required but not found: sha512

Script Integrations

The only output sent to STDOUT is from downloads configured to write there (or the --export option is used). All other information (status updates, progress, and log messages) will be written to STDERR. If you want to reduce runtime output or rely on your own messaging, use the --quiet option.

The --export option is helpful for capturing details about matched resources, the resolved origin, and some additional metadata. It supports outputs for JSON (example), JSONPath-like (docs), YAML, and a plain format.

$ gget github.com/dpb587/gget --export=jsonpath='{.origin.ref}' --no-download --quiet
v0.5.2
$ gget github.com/dpb587/gget --export=json --no-download --quiet \
  | jq '.metadata | from_entries'
{
"commit": "75447d782da3173dcd70d18936e9d91d8257e343",
"github-release-id": "28260723",
"github-release-published-at": "2020-07-06T14:56:17Z",
"tag": "v0.5.2",
}
$ gget github.com/dpb587/gget --export=plain --no-download --quiet \
  | awk '{ if ( $1 == "resource-checksum" ) { print $4 "  " $2 } }'
2b5f150526e626c96897f89ea9f3e84f05e02ff52bde02c29280d6f199a65992 gget-0.5.2-darwin-amd64
c8758e0f25eec1ef53c0f120a67e248c76c1b25d044fe361f992b0cb777d9407 gget-0.5.2-linux-amd64
330f04f773af63cc89af94552f717d94bcf082a04b954fa0523bd293ee8bf547 gget-0.5.2-windows-amd64.exe

The --version option can optionally be passed a constraint if you need to rely on a particular version. When the constraint is not met, the command will exit with an error.

if ! gget --version='>=0.5' >/dev/null 2>&1
then
  echo "oops. gget version 0.5 or greater must be installed first." >&2
  exit 1
fi

Reference

Showing gget --help from v0.5.2.

Usage:
  gget HOST/OWNER/REPOSITORY[@REF] [[LOCAL-PATH=]RESOURCE-GLOB...]

Runtime Options:
  -h, --help                              show documentation of this command
  -q, --quiet                             suppress runtime status messages
  -v, --verbose                           increase logging verbosity (multiple)
      --version=[CONSTRAINT]              show version of this command (with optional constraint to validate)

Repository Options:
      --ref-stability=STABILITY           acceptable stability level(s) for latest (values: stable, pre-release, any; default: stable)
      --ref-version=CONSTRAINT            version constraint(s) to require of latest (e.g. 4.x)
      --service=NAME                      specific git service to use (values: github, gitlab; default: auto-detect)

Resource Options:
      --exclude=RESOURCE-GLOB             exclude resource(s) from download (multiple)
      --ignore-missing=[RESOURCE-GLOB]    if a resource is not found, skip it rather than failing (multiple)
      --type=TYPE                         type of resource to get (values: asset, archive, blob; default: asset)
      --list                              list matching resources and stop before downloading

Download Options:
      --cd=DIR                            change to directory before writing files
      --executable=[RESOURCE-GLOB]        apply executable permissions to downloads (multiple)
      --export=FORMAT                     export details about the download profile (values: json, jsonpath=TEMPLATE, plain, yaml)
      --fail-fast                         fail and exit immediately if a download fails
      --no-download                       do not perform any downloads
      --no-progress                       do not show live-updating progress during downloads
      --parallel=NUM                      maximum number of parallel downloads (default: 3)
      --stdout                            write file contents to stdout rather than disk
      --verify-checksum=[METHOD]          strategy for verifying checksums (values: auto, required, none, {algo}, {algo}-min; default: auto)

Arguments:
  HOST/OWNER/REPOSITORY[@REF]:            repository reference
  [LOCAL-PATH=]RESOURCE-GLOB:             resource name(s) to download

Docker

Using gget to build containers.

A gget Docker image is available and easily integrates as a build stage for downloads. It includes tools for uncompressing archives, and the default working directory is /result for access in later stages.

FROM dpb587/gget as gget
RUN gget github.com/cloudfoundry/bosh-cli --ref-version=5.x \
      --executable bosh=bosh-cli-*-linux-amd64
RUN gget github.com/cloudfoundry/bosh-bootloader \
      --executable bbl=bbl-*_linux_x86-64
RUN gget github.com/pivotal-cf/om \
      --stdout om-linux-*.tar.gz | tar -xzf- om

FROM ubuntu
COPY --from=gget /result/* /usr/local/bin/
# ...everything else for your image...