Github News

• 

  Get up to speed with partial clone and shallow clone
  • sorangutan  

  1
  1
  Posts
  21
  Views

  

  As your Git repositories grow, it becomes harder and harder for new developers to clone and start working on them. Git is designed as a distributed version control system. This means that you can work on your machine without needing a connection to a central server that controls how you interact with the repository. This is only fully realizable if you have all reachable data in your local repository.

  What if there was a better way? Could you get started working in the repository without downloading every version of every file in the entire Git history? Git’s partial clone and shallow clone features are options that can help here, but they come with their own tradeoffs. Each option breaks at least one expectation from the normal distributed nature of Git, and you might not be willing to make those tradeoffs.

  If you are working with an extremely large monorepo, then these tradeoffs are more likely to be worthwhile or even necessary to interact with Git at that scale!

  Before digging in on this topic, be sure you are familiar with how Git stores your data, including commits, trees, and blob objects. I presented some of these ideas and other helpful tips at GitHub Universe in my talk, Optimize your monorepo experience.

  Quick Summary

  There are three ways to reduce clone sizes for repositories hosted by GitHub.

  • git clone --filter=blob:none <url> creates a blobless clone. These clones download all reachable commits and trees while fetching blobs on-demand. These clones are best for developers and build environments that span multiple builds.
  • git clone --filter=tree:0 <url> creates a treeless clone. These clones download all reachable commits while fetching trees and blobs on-demand. These clones are best for build environments where the repository will be deleted after a single build, but you still need access to commit history.
  • git clone --depth=1 <ulr> creates a shallow clone. These clones truncate the commit history to reduce the clone size. This creates some unexpected behavior issues, limiting which Git commands are possible. These clones also put undue stress on later fetches, so they are strongly discouraged for developer use. They are helpful for some build environments where the repository will be deleted after a single build.

  Full clones

  As we discuss the different clone types, we will use a common representation of Git objects:

  • Boxes are blobs. These represent file contents.
  • Triangles are trees. These represent directories.
  • Circles are commits. These are snapshots in time.

  We use arrows to represent a relationship between objects. Basically, if an OID B appears inside a commit or tree A, then the object A has an arrow to the object B. If we can follow a list of arrows from an object A to another object C, then we say C is reachable from A. The process of following these arrows is sometimes referred to as walking objects.

  We can now describe the data downloaded by a git clone command! The client asks the server for the latest commits, then the server provides those objects and every other reachable object. This includes every tree and blob in the entire commit history!

  

  In this diagram, time moves from left to right. The arrows between a commit and its parents therefore go from right to left. Each commit has a single root tree. The root tree at the HEAD commit is fully expanded underneath, while the rest of the trees have arrows pointing towards these objects.

  This diagram is purposefully simple, but if your repository is very large you will have many commits, trees, and blobs in your history. Likely, the historical data forms a majority of your data. Do you actually need all of it?

  These days, many developers always have a network connection available as they work, so asking the server for a little more data when necessary might be an acceptable trade-off.

  This is the critical design change presented by partial clone.

  Partial clone

  Git’s partial clone feature is enabled by specifying the --filter option in your git clone command. The full list of filter options exist in the git rev-list documentation, since you can use git rev-list --filter=<filter> --all to see which objects in your repository match the filter. There are several filters available, but the server can choose to deny your filter and revert to a full clone.

  On github.com and GitHub Enterprise Server 2.22+, there are two options available:

  1. Blobless clones: git clone --filter=blob:none <url>
  2. Treeless clones: git clone --filter=tree:0 <url>

  Let’s investigate each of these options.

  Blobless clones

  When using the --filter=blob:none option, the initial git clone will download all reachable commits and trees, and only download the blobs for commits when you do a git checkout. This includes the first checkout inside the git clone operation. The resulting object model is shown here:

  

  The important thing to notice is that we have a copy of every blob at HEAD but the blobs in the history are not present. If your repository has a deep history full of large blobs, then this option can significantly reduce your git clone times. The commit and tree data is still present, so any subsequent git checkout only needs to download the missing blobs. The Git client knows how to batch these requests to ask the server only for the missing blobs.

  Further, when running git fetch in a blobless clone, the server only sends the new commits and trees. The new blobs are downloaded only after a git checkout. Note that git pull runs git fetch and then git merge, so it will download the necessary blobs during the git merge command.

  When using a blobless clone, you will trigger a blob download whenever you need the contents of a file, but you will not need one if you only need the OID of a file. This means that git log can detect which commits changed a given path without needing to download extra data.

  This means that blobless clones can perform commands like git merge-base, git log, or even git log -- <path> with the same performance as a full clone.

  Commands like git diff or git blame <path> require the contents of the paths to compute diffs, so these will trigger blob downloads the first time they are run. However, the good news is that after that you will have those blobs in your repository and do not need to download them a second time. Most developers only need to run git blame on a small number of files, so this tradeoff of a slightly slower git blame command is worth the faster clone and fetch times.

  Blobless clones are the most widely-used partial clone option. I’ve been using them myself for months without issue.

  Treeless clones

  In some repositories, the tree data might be a significant portion of the history. Using --filter=tree:0, a treeless clone downloads all reachable commits, then downloads trees and blobs on demand. The resulting object model is shown here:

  

  Note that we have all of the data at HEAD, but otherwise only have commit data. This means that the initial clone can be much faster in a treeless clone than in a blobless or full clone. Further, we can run git fetch to download only the latest commits. However, working in a treeless clone is more difficult because downloading a missing tree when needed is more expensive.

  For example, a git checkout command changes the HEAD commit, usually to a commit where we do not have the root tree. The Git client then asks the server for that root tree by OID, but also for all reachable trees from that root tree. Currently, this request does not tell the server that the client already has some root trees, so the server might send many trees the client already has locally. After the trees are downloaded, the client can detect which blobs are missing and request those in a batch.

  It is possible to work in a treeless clone without triggering too many requests for extra data, but it is much more restrictive than a blobless clone.

  For example, history operations such as git merge-base or git log (without extra options) only use commit data. These will not trigger extra downloads.

  However, if you run a file history request such as git log -- <path>, then a treeless clone will start downloading root trees for almost every commit in the history!

  We strongly recommend that developers do not use treeless clones for their daily work. Treeless clones are really only helpful for automated builds when you want to quickly clone, compile a project, then throw away the repository. In environments like GitHub Actions using public runners, you want to minimize your clone time so you can spend your machine time actually building your software! Treeless clones might be an excellent option for those environments.

  Warning: While writing this article, we were putting treeless clones to the test beyond the typical limits. We noticed that repositories that contain submodules behave very poorly with treeless clones. Specifically, if you run git fetch in a treeless clone, then the logic in Git that looks for changed submodules will trigger a tree request for every new commit! This behavior can be avoided by running git config fetch.recurseSubmodules false in your treeless clones. We are working on a more robust fix in the Git client.

  Shallow clones

  Partial clones are relatively new to Git, but there is an older feature that does something very similar to a treeless clone: shallow clones. Shallow clones use the --depth=<N> parameter in git clone to truncate the commit history. Typically, --depth=1 signifies that we only care about the most recent commits. Shallow clones are best combined with the --single-branch --branch=<branch> options as well, to ensure we only download the data for the commit we plan to use immediately.

  The object model for a shallow clone is shown in this diagram:

  

  Here, the commit at HEAD exists, but its connection to its parents and the rest of the history is severed. The commits whose parents are removed are called shallow commits and together form the shallow boundary. The commit objects themselves have not changed, but there is some metadata in the client repository directing the Git client to ignore those parent connections. All trees and blobs are downloaded for any commit that exists on the client.

  Since the commit history is truncated, commands such as git merge-base or git log show different results than they would in a full clone! In general, you cannot count on them to work as expected. Recall that these commands work as expectedly in partial clones. Even in blobless clones, commands like git blame -- <path> will work correctly, if only a little slower than in full clones. Shallow clones don’t even make that a possibility!

  The other major difference is how git fetch behaves in a shallow clone. When fetching new commits, the server must provide every tree and blob that is “new” to these commits, relative to the shallow commits. This computation can be more expensive than a typical fetch, partly because a well-maintained server can make use of reachability bitmaps. Depending on how others are contributing to your remote repository, a git fetch operation in a shallow clone might end up downloading an almost-full commit history!

  Here are some descriptions of things that can go wrong with shallow clones that negate the supposed values. For these reasons we do not recommend shallow clones except for builds that delete the repository immediately afterwards. Fetching from shallow clones can cause more harm than good!

  Remember the “shallow boundary” mentioned earlier? The client sends that boundary to the server during a git fetch command, telling the server that it doesn’t have all of the reachable commits behind that boundary. The client then asks for the latest commits and everything reachable from those until hitting a shallow commit in the boundary. If another user starts a topic branch below that boundary and then the shallow client fetches that topic (or worse, the topic is merged into the default branch), then the server needs to walk the full history and serve the client what amounts to almost a full clone! Further, the server needs to calculate that data without the advantage of performance features like reachability bitmaps.

  

  Comparing Clone Options

  Let’s recall each of our clone options. Instead of looking them at a pure object level, let’s explore each category of object. The figures below group the data that is downloaded by each repository type. In addition to the data downloaded at clone, let’s consider the situation where some time passes and then the client runs git fetch and then git checkout to move to a new commit. For each of these options, how much data is downloaded.

  Full clones download all reachable objects. Typically, blobs are responsible for most of this data.

  

  In a partial clone, some data is not served immediately and is delayed until the client needs it. Blobless clones skip blobs except those needed at checkout time. Treeless clones skip all trees in the history in favor of downloading a full copy of the trees needed for each checkout.

  Blobless clone	Treeless clone

  git clone --depth=1, git fetch	git clone --depth=1, git fetch --depth=1

  What do the numbers say?

  Fellow GitHub engineer @solmazabbaspour designed and ran an experiment to compare these different clone options on a variety of open source repositories. She will post a blog post tomorrow giving full details and data for the experiment, but I’ll share the executive summary here. Here are some common themes we identified that could help you choose the right scenario for your own usage:

  There are many different types of clones beyond the default full clone. If you truly need to have a distributed workflow and want all of the data in your local repository, then you should continue using full clones. If you are a developer focused on a single repository and your repository is reasonably-sized, the best approach is to do a full clone.

  You might switch to a blobless partial clone if your repository is very large due to many large blobs, as that clone will help you get started more quickly. The trade-off is that some commands such as git checkout or git blame will require downloading new blob data when necessary.

  In general, calculating a shallow fetch is computationally more expensive compared to a full fetch. Always use a full fetch instead of a shallow fetch both in fully and shallow cloned repositories.

  In workflows such as CI builds when there is a need to do a single clone and delete the repository immediately, shallow clones are a good option. Shallow clones are the fastest way to get a copy of the working directory at the tip commit with the additional cost that fetching from these repositories is much more expensive, so we do not recommend shallow clones for developers. If you need the commit history for your build, then a treeless partial clone might work better for you than a full clone.

  In general, your mileage may vary. Now that you are armed with these different options and the object model behind them, you can go and play with these kinds of clones. You should also be aware of some pitfalls of these non-full clone options:

  • Shallow clones skip the commit history. This makes commands such as git log or git merge-base unavailable. Never fetch from a shallow clone!
  • Treeless clones contain commit history, but it is very expensive to download missing trees. Thus, git log (without a path) and git merge-base are available, but commands like git log -- <path> and git blame are extremely slow and not recommended in these clones.
  • Blobless clones contain all reachable commits and trees, so Git downloads blobs when it needs access to file contents. This means that commands like git log -- <path> are available but commands like git blame are a bit slower on their first run. However, this can be a great way to get started on a very large repository with a lot of old, large blobs.
  • Full clones work as expected. The only downside is the time required to download all of that data, plus the extra disk space for all those files.

  Be sure to upgrade to the latest Git version so you have all the latest performance improvements!

  
  
  https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/
• 

  Visualizing GitHub’s global community
  • sorangutan  

  1
  1
  Posts
  20
  Views

  

  This is the second post in a series about how we built our new homepage. In the first post, my teammate Tobias shared how we made the 3D globe come to life, with lots of nitty gritty details about Three.js, performance optimization, and delightful touches.

  But there’s another side to the story—the data! We hope you enjoy the read.

  Data goals

  When we kicked off the project, we knew that we didn’t want to make just another animated globe. We wanted the data to be interesting and engaging. We wanted it to be real, and most importantly, we wanted it to be live.

  Luckily, the data was there.

  The challenge then became designing a data service that addressed the following challenges:

  1. How do we query our massive volume of data?
  2. How do we show you the most interesting bits?
  3. How do we geocode user locations in a way that respects privacy?
  4. How do we expose the computed data back to the monolith?
  5. How do we not break GitHub?

  Let’s begin, shall we?

  Querying GitHub

  So, how hard could it be to show you some recent pull requests? It turns out it’s actually very simple:

  class GlobeController < ApplicationController
    def data
      pull_requests = PullRequest
        .where(open: true)
        .joins(:repositories)
        .where("repository.is_open_source = true")
        .last(10_000)
  
      render json: pull_requests
    end
  end

  Just kidding

  Because of the volume of data generated on GitHub every day, the size of our databases, as well as the importance of keeping GitHub fast and reliable, we knew we couldn’t query our production databases directly.

  Luckily, we have a data warehouse and a fantastic team that maintains it. Data from production is fetched, sanitized, and packaged nicely into the data warehouse on a regular schedule. The data can then be queried using Presto, a flavor of SQL meant for querying large sets of data.

  We also wanted the data to be as fresh as possible. So instead of querying snapshots of our MySQL tables that are only copied over once a day, we were able to query data coming from our Apache Kafka event stream that makes it into the data warehouse much more regularly.

  As an example, we have an event that is reported every time a pull request is merged. The event is defined in a format called protobuf, which stands for “protocol buffer.”

  Here’s what the protobuf for a merged pull request event might look like:

  message PullRequestMerge {
    github.v1.entities.User actor = 1;
    github.v1.entities.Repository repository = 2;
    github.v1.entities.User repository_owner = 3;
    github.v1.entities.PullRequest pull_request = 4;
    github.v1.entities.Issue issue = 5;
  }

  Each row corresponds to an “entity,” each of which is defined in its own protobuf file. Here’s a snippet from the definition of a pull request entity:

  message PullRequest {
    uint64 id = 1;
    string global_relay_id = 2;
    uint64 author_id = 3;
  
    enum PullRequestState {
      UNKNOWN = 0;
      OPEN = 1;
      CLOSED = 2;
      MERGED = 3;
    }
    PullRequestState pull_request_state = 4;
  
    google.protobuf.Timestamp created_at = 5;
    google.protobuf.Timestamp updated_at = 6;
  }
  

  Including an entity in an event will pass along all of the attributes defined for it. All of that data gets copied into our data warehouse for every pull request that is merged.

  This means that a Presto query for pull requests merged in the past day could look like:

  SELECT
    pull_request.created_at,
    pull_request.updated_at,
    pull_request.id,
    issue.number,
    repository.id
  FROM kafka.github.pull_request_merge
  WHERE
    day >= CAST((CURRENT_DATE - INTERVAL '1' DAY) AS VARCHAR)

  There are a few other queries we make to pull in all the data we need. But as you can see, this is pretty much standard SQL that pulls in merged pull requests from the last day in the event stream.

  Surfacing interesting data

  We wanted to make sure that whatever data we showed was interesting, engaging, and appropriate to be spotlighted on the GitHub homepage. If the data was good, visitors would be enticed to explore the vast ecosystem of open source being built on GitHub at that given moment. Maybe they’d even make a contribution!

  So how do we find good data?

  Luckily our data team came to the rescue yet again. A few years ago, the Data Science Team put together a model to rank the “health” of repositories based on 30-plus features weighted by importance. A healthy repository doesn’t necessarily mean having a lot of stars. It also takes into account how much current activity is happening and how easy it is to contribute to the project, to name a few.

  The end result is a numerical health score that we can query against in the data warehouse.

  SELECT repository_id
  FROM data_science.github.repository_health_scores
  WHERE 
    score > 0.75

  Combining this query with the above, we can now pull in merged pull requests from repositories with health scores above a certain threshold:

  WITH
  healthy_repositories AS (
    SELECT repository_id
    FROM data_science.github.repository_health_scores
    WHERE 
      score > 0.75
  )
  
  SELECT
    a.pull_request.created_at,
    a.pull_request.updated_at,
    a.pull_request.id,
    a.issue.number,
    a.repository.id
  FROM kafka.github.pull_request_merge a
  JOIN healthy_repositories b
  ON a.repository.id = b.repository_id
  WHERE
    day >= CAST((CURRENT_DATE - INTERVAL '1' DAY) AS VARCHAR)

  We do some other things to ensure the data is good, like filtering out accounts with spammy behavior. But repository health scores are definitely a key ingredient.

  Geocoding user-provided locations

  Your GitHub profile has an optional free text field for providing your location. Some people fill it out with their actual location (mine says “San Francisco”), while others use fake or funny locations (42 users have “Middle Earth” listed as theirs). Many others choose to not list a location. In fact, two-thirds of users don’t enter anything and that’s perfectly fine with us.

  For users that do enter something, we try to map the text to a real location. This is a little harder to do than using IP addresses as proxies for locations, but it was important to us to only include data that users felt comfortable making public in the first place.

  In order to map the free text locations to latitude and longitude pairs, we use Mapbox’s forward geocoding API and their Ruby SDK. Here’s an example of a forward geocoding of “New York City”:

  MAPBOX_OPTIONS = {
    limit: 1,
    types: %w(region place country),
    language: "en"
  }
  
  Mapbox::Geocoder.geocode_forward("New York City", MAPBOX_OPTIONS)
  
  => [{
    "type" => "FeatureCollection",
    "query" => ["new", "york", "city"],
    "features" => [{
      "id" => "place.15278078705964500",
      "type" => "Feature",
      "place_type" => ["place"],
      "relevance" => 1,
      "properties" => {
        "wikidata" => "Q60"
      },
      "text_en" => "New York City",
      "language_en" => "en",
      "place_name_en" => "New York City, New York, United States",
      "text" => "New York City",
      "language" => "en",
      "place_name" => "New York City, New York, United States",
      "bbox" => [-74.2590879797556, 40.477399, -73.7008392055224, 40.917576401307],
      "center" => [-73.9808, 40.7648],
      "geometry" => {
        "type" => "Point", "coordinates" => [-73.9808, 40.7648]
      },
      "context" => [{
        "id" => "region.17349986251855570",
        "wikidata" => "Q1384",
        "short_code" => "US-NY",
        "text_en" => "New York",
        "language_en" => "en",
        "text" => "New York",
        "language" => "en"
      }, {
        "id" => "country.19678805456372290",
        "wikidata" => "Q30",
        "short_code" => "us",
        "text_en" => "United States",
        "language_en" => "en",
        "text" => "United States",
        "language" => "en"
      }]
    }],
    "attribution" => "NOTICE: (c) 2020 Mapbox and its suppliers. All rights reserved. Use of this data is subject to the Mapbox Terms of Service (https://www.mapbox.com/about/maps/). This response and the information it contains may not be retained. POI(s) provided by Foursquare."
  }, {}]

  There is a lot of data there, but let’s focus on text, relevance, and center for now. Here are those fields for the “New York City”:

  result = Mapbox::Geocoder.geocode_forward("New York City", MAPBOX_OPTIONS)
  result[0]["features"][0].slice("text", "relevance", "center")
  
  => {"text"=>"New York City", "relevance"=>1, "center"=>[-73.9808, 40.7648]}
  

  If you use “NYC” query string, you get the exact same result:

  result = Mapbox::Geocoder.geocode_forward("NYC", MAPBOX_OPTIONS)
  result[0]["features"][0].slice("text", "relevance", "center")
  
  => {"text"=>"New York City", "relevance"=>1, "center"=>[-73.9808, 40.7648]}

  Notice that the text is still “New York City” in this second example? That is because Mapbox is normalizing the results. We use the normalized text on the globe so viewers get a consistent experience. This also takes care of capitalization and misspellings.

  The center field is an array containing the longitude and latitude of the location.

  And finally, the relevance score is an indicator of Mapox’s confidence in the results. A relevance score of 1 is the highest, but sometimes users enter locations that Mapbox is less sure about:

  result = Mapbox::Geocoder.geocode_forward("Middle Earth", MAPBOX_OPTIONS)
  result[0]["features"][0].slice(text", "relevance", "center")
  
  => {"text"=>"Earth City", "relevance"=>0.5, "center"=>[-90.4682, 38.7689]}

  We discard anything with a score of less than 1, just to get confidence that the location we show feels correct.

  Mapbox also provides a batch geocoding endpoint. This allows us to query multiple locations in one request:

  MAPBOX_ENDPOINT = "mapbox.places-permanent"
  
  query_string = "{San Francisco};{Berlin};{Dakar};{Tokyo};{Lima}"
  
  Mapbox::Geocoder.geocode_forward(query_string, MAPBOX_OPTIONS, MAPBOX_ENDPOINT)

  After we’ve geocoded and normalized all of the results, we create a JSON representation of the pull request and its locations so our globe JavaScript client knows how to parse it.

  Here’s a pull request we recently featured that was opened in San Francisco and merged in Tokyo:

  {
     "uml":"Tokyo",
     "gm":{
        "lat":35.68,
        "lon":139.77
     },
     "uol":"San Francisco",
     "gop":{
        "lat":37.7648,
        "lon":-122.463
     },
     "l":"JavaScript",
     "nwo":"mdn/browser-compat-data",
     "pr":7937,
     "ma":"2020-12-17 04:00:48.000",
     "oa":"2020-12-16 10:02:31.000"
  }

  We use short keys to shave off some bytes from the JSON we end up serving so the globe loads faster.

  Airflow, HDFS, and Munger

  We run our data warehouse queries and geocoding throughout the day to ensure that the data on the homepage is always fresh.

  For scheduling this work, we use another system from Apache called Airflow. Airflow lets you run scheduled jobs that contain a sequence of tasks. Airflow calls these workflows Direct Acyclical Graphs (or DAGs for short), which is a borrowed term from graph theory in computer science. Basically this means that you schedule one task at a time, execute the task, and when the task is done, then the next task is scheduled and eventually executed. Tasks can pass along information to each other.

  At a high level, our DAG executes the following tasks:

  1. Query the data warehouse.
  2. Geocode locations from the results.
  3. Write the results to a file.
  4. Expose the results to the GitHub Rails app.

  We covered the first two steps earlier. For writing the file, we use HDFS, which is a distributed file system that’s part of the Apache Hadoop project. The file is then uploaded to Munger, an internal service we use to expose results from the data science pipeline back to the GitHub Rails app that powers github.com.

  Here’s what this might look like in the Airflow UI:

  

  Each column in that screenshot represents a full DAG run of all of the tasks. The last column with the light green circle at the top indicates that the DAG is in the middle of a run. It’s completed the build_home_page_globe_table task (represented by a dark green box) and now has the next task write_to_hdfs scheduled (dark blue box).

  Our Airflow instance runs more than just this one DAG throughout the day, so we may stay in this state for some time before the scheduler is ready to pick up the write_to_hdfs task. Eventually the remaining tasks should run. If everything ends up running smoothly, we should see all green:

  

  Wrapping up

  Hope that gives you a glimpse into how we built this!

  Again, thank you to all the teams that made the GitHub homepage and globe possible. This project would not have been possible without years of investment in our data infrastructure and data science capabilities, so a special shout out to Kim, Jeff, Preston, Ike, Scott, Jamison, Rowan, and Omoju.

  More importantly, we could not have done it without you, the GitHub community, and your daily contributions and projects that truly bring the globe to life. Stay tuned—we have even more in store for this project coming soon.

  In the meantime, I hope to see you on the homepage soon.

  
  
  https://github.blog/2020-12-21-visualizing-githubs-global-community/
• 

  How we built the GitHub globe
  • sorangutan  

  1
  1
  Posts
  26
  Views

  

  GitHub is where the world builds software. More than 56 million developers around the world build and work together on GitHub. With our new homepage, we wanted to show how open source development transcends the borders we’re living in and to tell our product story through the lens of a developer’s journey.

  Now that it’s live, we would love to share how we built the homepage-directly from the voices of our designers and developers. In this five-part series, we’ll discuss:

  1. How our globe is built
  2. How we collect and use the data behind the globe
  3. How we designed the homepage and wrote the narrative
  4. How we made the page fast and performant
  5. How our illustrators work with designers and engineers

   

  At Satellite in 2019, our CEO Nat showed off a visualization of open source activity on GitHub over a 30-day span. The sheer volume and global reach was astonishing, and we knew we wanted to build on that story.

   

   The main goals we set out to achieve in the design and development of the globe were:

  • An interconnected community. We explored many different options, but ultimately landed on pull requests. It turned out to be a beautiful visualization of pull requests being opened in one part of the world and closed in another.
  • A showcase of real work happening now. We started by simply showing the pull requests’ arcs and spires, but quickly realized that we needed “proof of life.” The arcs could quite as easily just be design animations instead of real work. We iterated on ways to provide more detail and found most resonance with clear hover states that showed the pull request, repo, timestamp, language, and locations. Nat had the idea of making each line clickable, which really upleveled the experience and made it much more immersive. Read more here.
  • Attention to detail and performance. It was extremely important to us that the globe not only looked inspiring and beautiful, but that it performed well on all devices. We went through many, many iterations of refinement, and there’s still more work to be done.

  Rendering the globe with WebGL

  At the most fundamental level, the globe runs in a WebGL context powered by three.js. We feed it data of recent pull requests that have been created and merged around the world through a JSON file. The scene is made up of five layers: a halo, a globe, the Earth’s regions, blue spikes for open pull requests, and pink arcs for merged pull requests. We don’t use any textures: we point four lights at a sphere, use about 12,000 five-sided circles to render the Earth’s regions, and draw a halo with a simple custom shader on the backside of a sphere.

   

  To draw the Earth’s regions, we start by defining the desired density of circles (this will vary depending on the performance of your machine—more on that later), and loop through longitudes and latitudes in a nested for-loop. We start at the south pole and go upwards, calculate the circumference for each latitude, and distribute circles evenly along that line, wrapping around the sphere:

  for (let lat = -90; lat <= 90; lat += 180/rows) {
    const radius = Math.cos(Math.abs(lat) * DEG2RAD) * GLOBE_RADIUS;
    const circumference = radius * Math.PI * 2;
    const dotsForLat = circumference * dotDensity;
    for (let x = 0; x < dotsForLat; x++) {
      const long = -180 + x*360/dotsForLat;
      if (!this.visibilityForCoordinate(long, lat)) continue;
  
      // Setup and save circle matrix data
    }
  }
  

  To determine if a circle should be visible or not (is it water or land?) we load a small PNG containing a map of the world, parse its image data through canvas’s context.getImageData(), and map each circle to a pixel on the map through the visibilityForCoordinate(long, lat) method. If that pixel’s alpha is at least 90 (out of 255), we draw the circle; if not, we skip to the next one.

  After collecting all the data we need to visualize the Earth’s regions through these small circles, we create an instance of CircleBufferGeometry and use an InstancedMesh to render all the geometry.

  Making sure that you can see your own location

  As you enter the new GitHub homepage, we want to make sure that you can see your own location as the globe appears, which means that we need to figure where on Earth that you are. We wanted to achieve this effect without delaying the first render behind an IP look-up, so we set the globe’s starting angle to center over Greenwich, look at your device’s timezone offset, and convert that offset to a rotation around the globe’s own axis (in radians):

  const date = new Date();
  const timeZoneOffset = date.getTimezoneOffset() || 0;
  const timeZoneMaxOffset = 60*12;
  rotationOffset.y = ROTATION_OFFSET.y + Math.PI * (timeZoneOffset / timeZoneMaxOffset);

  It’s not an exact measurement of your location, but it’s quick, and does the job.

  Visualizing pull requests

  The main act of the globe is, of course, visualizing all of the pull requests that are being opened and merged around the world. The data engineering that makes this possible is a different topic in and of itself, and we’ll be sharing how we make that happen in an upcoming post. Here we want to give you an overview of how we’re visualizing all your pull requests.

   

  Let’s focus on pull requests being merged (the pink arcs), as they are a bit more interesting. Every merged pull request entry comes with two locations: where it was opened, and where it was merged. We map these locations to our globe, and draw a bezier curve between these two locations:

  const curve = new CubicBezierCurve3(startLocation, ctrl1, ctrl2, endLocation);

  We have three different orbits for these curves, and the longer the two points are apart, the further out we’ll pull out any specific arc into space. We then use instances of TubeBufferGeometry to generate geometry along these paths, so that we can use setDrawRange() to animate the lines as they appear and disappear.

  As each line animates in and reaches its merge location, we generate and animate in one solid circle that stays put while the line is present, and one ring that scales up and immediately fades out. The ease out easings for these animations are created by multiplying a speed (here 0.06) with the difference between the target (1) and the current value (animated.dot.scale.x), and adding that to the existing scale value. In other words, for every frame we step 6% closer to the target, and as we’re coming closer to that target, the animation will naturally slow down.

  // The solid circle
  const scale = animated.dot.scale.x + (1 - animated.dot.scale.x) * 0.06;
  animated.dot.scale.set(scale, scale, 1);
  
  // The landing effect that fades out
  const scaleUpFade = animated.dotFade.scale.x + (1 - animated.dotFade.scale.x) * 0.06;
  animated.dotFade.scale.set(scaleUpFade, scaleUpFade, 1);
  animated.dotFade.material.opacity = 1 - scaleUpFade;

  Creative constraints from performance optimizations

  The homepage and the globe needs to perform well on a variety of devices and platforms, which early on created some creative restrictions for us, and made us focus extensively on creating a well-optimized page. Although some modern computers and tablets could render the globe at 60 FPS with antialias turned on, that’s not the case for all devices, and we decided early on to leave antialias turned off and optimize for performance. This left us with a sharp and pixelated line running along the top left edge of the globe, as the globe’s highlighted edge met the darker color of the background:

  

  This encouraged us to explore a halo effect that could hide that pixelated edge. We created one by using a custom shader to draw a gradient on the backside of a sphere that’s slightly larger than the globe, placed it behind the globe, and tilted it slightly on its side to emphasize the effect in the top left corner:

  const halo = new Mesh(haloGeometry, haloMaterial);
  halo.scale.multiplyScalar(1.15);
  halo.rotateX(Math.PI*0.03);
  halo.rotateY(Math.PI*0.03);
  this.haloContainer.add(halo);
  

  

  This smoothed out the sharp edge, while being a much more performant operation than turning on antialias. Unfortunately, leaving antialias off also produced a fairly prominent moiré effect as all the circles making up the world came closer and closer to each other as they neared the edges of the globe. We reduced this effect and simulated the look of a thicker atmosphere by using a fragment shader for the circles where each circle’s alpha is a function of its distance from the camera, fading out every individual circle as it moves further away:

  if (gl_FragCoord.z > fadeThreshold) {
    gl_FragColor.a = 1.0 + (fadeThreshold - gl_FragCoord.z ) * alphaFallOff;
  }
  

  Improving perceived speed

  We don’t know how quickly (or slowly) the globe is going to load on a particular device, but we wanted to make sure that the header composition on the homepage is always balanced, and that you got the impression that the globe loads quickly even if there’s a slight delay before we can render the first frame.

  We created a bare version of the globe using only gradients in Figma and exported it as an SVG. Embedding this SVG in the HTML document adds little overhead, but makes sure that something is immediately visible as the page loads. As soon as we’re ready to render the first frame of the globe, we transition between the SVG and the canvas element by crossfading between and scaling up both elements using the Web Animations API. Using the Web Animations API enables us to not touch the DOM at all during the transition, ensuring that it’s as stutter-free as possible.

  const keyframesIn = [
        { opacity: 0, transform: 'scale(0.8)' },
        { opacity: 1, transform: 'scale(1)' }
      ];
  const keyframesOut = [
        { opacity: 1, transform: 'scale(0.8)' },
        { opacity: 0, transform: 'scale(1)' }
      ];
  const options = { fill: 'both', duration: 600, easing: 'ease' };
  
  this.renderer.domElement.animate(keyframesIn, options);
  const placeHolderAnim = placeholder.animate(keyframesOut, options);
  placeHolderAnim.addEventListener('finish', () => {
    placeholder.remove();
  });
  

  Graceful degradation with quality tiers

  We aim at maintaining 60 FPS while rendering an as beautiful globe as we can, but finding that balance is tricky—there are thousands of devices out there, all performing differently depending on the browser they’re running and their mood. We constantly monitor the achieved FPS, and if we fail to maintain 55.5 FPS over the last 50 frames we start to degrade the quality of the scene.

   

  There are four quality tiers, and for every degradation we reduce the amount of expensive calculations. This includes reducing the pixel density, how often we raycast (figure out what your cursor is hovering in the scene), and the amount of geometry that’s drawn on screen—which brings us back to the circles that make up the Earth’s regions. As we traverse down the quality tiers, we reduce the desired circle density and rebuild the Earth’s regions, here going from the original ~12 000 circles to ~8 000:

  // Reduce pixel density to 1.5 (down from 2.0)
  this.renderer.setPixelRatio(Math.min(AppProps.pixelRatio, 1.5));
  // Reduce the amount of PRs visualized at any given time
  this.indexIncrementSpeed = VISIBLE_INCREMENT_SPEED / 3 * 2;
  // Raycast less often (wait for 4 additional frames)
  this.raycastTrigger = RAYCAST_TRIGGER + 4;
  // Draw less geometry for the Earth’s regions
  this.worldDotDensity = WORLD_DOT_DENSITY * 0.65;
  // Remove the world
  this.resetWorldMap();
  // Generate world anew from new settings
  this.buildWorldGeometry();
  

  A small part of a wide-ranging effort

  These are some of the techniques that we use to render the globe, but the creation of the globe and the new homepage is part of a longer story, spanning multiple teams, disciplines, and departments, including design, brand, engineering, product, and communications. We’ll continue the deep-dive in this 5-part series, so come back soon or follow us on Twitter @GitHub for all the latest updates on this project and more.

  Next up: how we collect and use the data behind the globe.

  In the meantime, don’t miss out on the new GitHub globe wallpapers from the GitHub Illustration Team to enjoy the globe from your desktop or mobile device:

  • 4K UHD 16:9 Footer
  • 4K Ultrawide 21:9 Footer
  • 4K Superwide 32:9 Footer
  • 4K Superwide 32:9 Header
  • 4k UHD 16:9 Header
  • 4K Ultrawide 21:9 Header

  ---

  Love the new GitHub homepage or any of the work you see here? Join our team! 

  
  
  https://github.blog/2020-12-21-how-we-built-the-github-globe/
• 

  Commits are snapshots, not diffs
  • sorangutan  

  1
  1
  Posts
  22
  Views

  

  Git has a reputation for being confusing. Users stumble over terminology and phrasing that misguides their expectations. This is most apparent in commands that “rewrite history” such as git cherry-pick or git rebase. In my experience, the root cause of this confusion is an interpretation of commits as diffs that can be shuffled around. However, commits are snapshots, not diffs!

  I believe that Git becomes understandable if we peel back the curtain and look at how Git stores your repository data. After we investigate this model, we’ll explore how this new perspective helps us understand commands like git cherry-pick and git rebase.

  If you want to go really deep, you should read the Git Internals chapter of the Pro Git book.

  I’ll be using the git/git repository checked out at v2.29.2 as an example. Follow along with my command-line examples for extra practice.

  Object IDs are hashes

  The most important part to know about Git objects is that Git references each by its object ID (OID for short), providing a unique name for the object. We will use the git rev-parse <ref> command to discover these OIDs. Each object is essentially a plain-text file and we can examine its contents using the git cat-file -p <oid> command.

  You might also be used to seeing OIDs given as a shorter hex string. This string is given as something long enough that only one object in the repository has an OID that matches that abbreviation. If we request the type of an object using an abbreviated OID that is too short, then we will see the list of OIDs that match

  $ git cat-file -t e0c03
  error: short SHA1 e0c03 is ambiguous
  hint: The candidates are:
  hint: e0c03f27484 commit 2016-10-26 - contrib/buildsystems: ignore irrelevant files in Generators/
  hint: e0c03653e72 tree
  hint: e0c03c3eecc blob
  fatal: Not a valid object name e0c03

  What are these types: blob, tree, and commit? Let’s start at the bottom and work our way up.

  Blobs are file contents

  At the bottom of the object model, blobs contain file contents. To discover the OID for a file at your current revision, run git rev-parse HEAD:<path>. Then, use git cat-file -p <oid> to find its contents.

  $ git rev-parse HEAD:README.md
  eb8115e6b04814f0c37146bbe3dbc35f3e8992e0
  
  $ git cat-file -p eb8115e6b04814f0c37146bbe3dbc35f3e8992e0 | head -n 8
  [![Build status](https://github.com/git/git/workflows/CI/PR/badge.png)](https://github.com/git/git/actions?query=branch%3Amaster+event%3Apush)
  
  Git - fast, scalable, distributed revision control system
  =========================================================
  
  Git is a fast, scalable, distributed revision control system with an
  unusually rich command set that provides both high-level operations
  and full access to internals.

  If I edit the README.md file on my disk, then git status notices that the file has a recent modified time and hashes the contents. If the contents don’t match the current OID at HEAD:README.md, then git status reports the file as “modified on disk.” In this way, we can see if the file contents in the current working directory match the expected contents at HEAD.

  Trees are directory listings

  Note that blobs contain file contents, but not the file names! The names come from Git’s representation of directories: trees. A tree is an ordered list of path entries, paired with object types, file modes, and the OID for the object at that path. Subdirectories are also represented as trees, so trees can point to other trees!

  

  We will use diagrams to visualize how these objects are related. We use boxes for blobs and triangles for trees.

  $ git rev-parse HEAD^{tree}
  75130889f941eceb57c6ceb95c6f28dfc83b609c
  
  $ git cat-file -p 75130889f941eceb57c6ceb95c6f28dfc83b609c  | head -n 15
  100644 blob c2f5fe385af1bbc161f6c010bdcf0048ab6671ed    .cirrus.yml
  100644 blob c592dda681fecfaa6bf64fb3f539eafaf4123ed8    .clang-format
  100644 blob f9d819623d832113014dd5d5366e8ee44ac9666a    .editorconfig
  100644 blob b08a1416d86012134f823fe51443f498f4911909    .gitattributes
  040000 tree fbe854556a4ae3d5897e7b92a3eb8636bb08f031    .github
  100644 blob 6232d339247fae5fdaeffed77ae0bbe4176ab2de    .gitignore
  100644 blob cbeebdab7a5e2c6afec338c3534930f569c90f63    .gitmodules
  100644 blob bde7aba756ea74c3af562874ab5c81a829e43c83    .mailmap
  100644 blob 05f3e3f8d79117c1d32bf5e433d0fd49de93125c    .travis.yml
  100644 blob 5ba86d68459e61f87dae1332c7f2402860b4280c    .tsan-suppressions
  100644 blob fc4645d5c08bd005238fc72cfa709495d8722e6a    CODE_OF_CONDUCT.md
  100644 blob 536e55524db72bd2acf175208aef4f3dfc148d42    COPYING
  040000 tree a58410edddbdd133cca6b3322bebe4fb37be93fa    Documentation
  100755 blob ca6ccb49866c595c80718d167e40cfad1ee7f376    GIT-VERSION-GEN
  100644 blob 9ba33e6a141a3906eb707dd11d1af4b0f8191a55    INSTALL

  Trees provide names for each sub-item. Trees also include information such as Unix file permissions, object type (blob or tree), and OIDs for each entry. We cut the output to the top 15 entries, but we can use grep to discover that this tree has a README.md entry that points to our earlier blob OID.

  $ git cat-file -p 75130889f941eceb57c6ceb95c6f28dfc83b609c | grep README.md
  100644 blob eb8115e6b04814f0c37146bbe3dbc35f3e8992e0    README.md

  Trees can point to blobs and other trees using these path entries. Keep in mind that those relationships are paired with path names, but we will not always show those names in our diagrams.

  The tree itself doesn’t know where it exists within the repository, that is the role of the objects pointing to the tree. The tree referenced by <ref>^{tree} is a special tree: the root tree. This designation is based on a special link from your commits.

  Commits are snapshots

  A commit is a snapshot in time. Each commit contains a pointer to its root tree, representing the state of the working directory at that time. The commit has a list of parent commits corresponding to the previous snapshots. A commit with no parents is a root commit and a commit with multiple parents is a merge commit. Commits also contain metadata describing the snapshot such as author and committer (including name, email address, and date) and a commit message. The commit message is an opportunity for the commit author to describe the purpose of that commit with respect to the parents.

  

  For example, the commit at v2.29.2 in the Git repository describes that release, and is authored and committed by the Git maintainer.

  $ git rev-parse HEAD
  898f80736c75878acc02dc55672317fcc0e0a5a6
  
  /c/_git/git ((v2.29.2))
  $ git cat-file -p 898f80736c75878acc02dc55672317fcc0e0a5a6
  tree 75130889f941eceb57c6ceb95c6f28dfc83b609c
  parent a94bce62b99be35f2ee2b4c98f97c222e7dd9d82
  author Junio C Hamano <[email protected]> 1604006649 -0700
  committer Junio C Hamano <[email protected]> 1604006649 -0700
  
  Git 2.29.2
  
  Signed-off-by: Junio C Hamano <[email protected]>

  Looking a little farther in the history with git log, we can see a more descriptive commit message talking about the change between that commit and its parent.

  $ git cat-file -p 16b0bb99eac5ebd02a5dcabdff2cfc390e9d92ef
  tree d0e42501b1cf65395e91e22e74f75fc5caa0286e
  parent 56706dba33f5d4457395c651cf1cd033c6c03c7a
  author Jeff King <[email protected]> 1603436979 -0400
  committer Junio C Hamano <[email protected]> 1603466719 -0700
  
  am: fix broken email with --committer-date-is-author-date
  
  Commit e8cbe2118a (am: stop exporting GIT_COMMITTER_DATE, 2020-08-17)
  rewrote the code for setting the committer date to use fmt_ident(),
  rather than setting an environment variable and letting commit_tree()
  handle it. But it introduced two bugs:
  
  - we use the author email string instead of the committer email
  
  - when parsing the committer ident, we used the wrong variable to
  compute the length of the email, resulting in it always being a
  zero-length string
  
  This commit fixes both, which causes our test of this option via the
  rebase "apply" backend to now succeed.
  
  Signed-off-by: Jeff King <[email protected]> Signed-off-by: Junio C Hamano <[email protected]>

  In our diagrams, we will use circles to represent commits. Notice the alliteration? Let’s review:

  • Boxes are blobs. These represent file contents.
  • Triangles are trees. These represent directories.
  • Circles are commits. These are snapshots in time.

  Branches are pointers

  In Git, we move around the history and make changes without referring to OIDs most of the time. This is because branches provide pointers to the commits we care about. A branch with name main is actually a reference in Git called refs/heads/main. These files literally contain hex strings referencing the OID of a commit. As you work, these references change their contents to point to other commits.

  This means branches are significantly different from our previous Git objects. Commits, trees, and blobs are immutable, meaning you can’t change their contents. If you change the contents, then you get a different hash and thus a new OID referring to the new object! Branches are named by users to provide meaning, such as trunk or my-special-project. We use branches to track and share work.

  The special reference HEAD points to the current branch. When we add a commit to HEAD, it automatically updates that branch to the new commit.

  We can create a new branch and update our HEAD using git switch -c:

  $ git switch -c my-branch
  Switched to a new branch 'my-branch'
  $ cat .git/refs/heads/my-branch
  1ec19b7757a1acb11332f06e8e812b505490afc6
  $ cat .git/HEAD
  ref: refs/heads/my-branch

  Notice how creating my-branch created a file (.git/refs/heads/my-branch) containing the current commit OID and the .git/HEAD file was updated to point at this branch. Now, if we update HEAD by creating new commits, the branch my-branch will update to point to that new commit!

  The big picture

  Let’s put all of these new terms into one giant picture. Branches point to commits, commits point to other commits and their root trees, trees point to blobs and other trees, and blobs don’t point to anything. Here is a diagram containing all of our objects all at once:

  

  In this diagram, time moves from left to right. The arrows between a commit and its parents go from right to left. Each commit has a single root tree. HEAD points to the main branch here, and main points to the most-recent commit. The root tree at this commit is fully expanded underneath, while the rest of the trees have arrows pointing towards these objects. The reason for that is that the same objects are reachable from multiple root trees! Since these trees reference those objects by their OID (their content) these snapshots do not need multiple copies of the same data. In this way, Git’s object model forms a Merkle tree.

  When we view the object model in this way, we can see why commits are snapshots: they link directly to a full view of the expected working directory for that commit!

  Computing diffs

  Even though commits are snapshots, we frequently look at a commit in a history view or on GitHub as a diff. In fact, the commit message frequently refers to this diff. The diff is dynamically generated from the snapshot data by comparing the root trees of the commit and its parent. Git can compare any two snapshots in time, not just adjacent commits.

  To compare two commits, start by looking at their root trees, which are almost always different. Then, perform a depth-first-search on the subtrees by following pairs when paths for the current tree have different OIDs. In the example below, the root trees have different values for the docs, so we recurse into those two trees. Those trees have different values for M.md, so those two blobs are compared line-by-line and that diff is shown. Still within docs, N.md is the same, so that is skipped and we pop back to the root tree. The root tree then sees that the things directories have equal OIDs as well as the README.md entries.

  

  In the diagram above, we notice that the things tree is never visited, and so none of its reachable objects are visited. This way, the cost of computing a diff is relative to the number of paths with different content.

  Now we have the understanding that commits are snapshots and we can dynamically compute a diff between any two commits. Then why isn’t this common knowledge? Why do new users stumble over this idea that a commit is a diff?

  One of my favorite analogies is to think of commits as having a wave/partical duality where sometimes they are treated like snapshots and other times they are treated like diffs. The crux of the matter really goes into a different kind of data that’s not actually a Git object: patches.

  Wait, what’s a patch?

  A patch is a text document that describes how to alter an existing codebase. Patches are how extremely-distributed groups can share code without using Git commits directly. You can see these being shuffled around on the Git mailing list.

  A patch contains a description of the change and why it is valuable, followed by a diff. The idea is that someone could use that reasoning as a justification to apply that diff to their copy of the code.

  Git can convert a commit into a patch using git format-patch. A patch can then be applied to a Git repository using git apply. This was the dominant way to share code in the early days of open source, but most projects have moved to sharing Git commits directly through pull requests.

  The biggest issue with sharing patches is that the patch loses the parent information and the new commit has a parent equal to your existing HEAD. Moreover, you get a different commit even if you use the same parent as before due to the commit time, but also the committer changes! This is the fundamental reason why Git has both “author” and “committer” details in the commit object.

  The biggest problem with using patches is that it is hard to apply a patch when your working directory does not match the sender’s previous commit. Losing the commit history makes it difficult to resolve conflicts.

  This idea of “moving patches around” has transferred into several Git commands as “moving commits around.” Instead, what actually happens is that commit diffs are replayed, creating new commits.

  If commits aren’t diffs, then what does git cherry-pick do?

  The git cherry-pick <oid> command creates a new commit with an identical diff to <oid> whose parent is the current commit. Git is essentially following these steps:

  1. Compute the diff between the commit <oid> and its parent.
  2. Apply that diff to the current HEAD.
  3. Create a new commit whose root tree matches the new working directory and whose parent is the commit at HEAD.
  4. Move the ref at HEAD to that new commit.

  

  After Git creates the new commit, the output of git log -1 -p HEAD should match the output of git log -1 -p <oid>.

  It is important to recognize that we didn’t “move” the commit to be on top of our current HEAD, we created a new commit whose diff matches the old commit.

  If commits aren’t diffs, then what does git rebase do?

  The git rebase command presents itself as a way to move commits to have a new history. In its most basic form it is really just a series of git cherry-pick commands, replaying diffs on top of a different commit.

  The most important thing is that git rebase <target> will discover the list of commits that are reachable from HEAD but not reachable from <target>. You can show these yourself using git log --oneline <target>..HEAD.

  Then, the rebase command simply navigates to the <target> location and starts performing git cherry-pick commands on this commit range, starting from the oldest commits. At the end, we have a new set of commits with different OIDs but similar diffs to the original commit range.

  For example, consider a sequence of three commits in the current HEAD since branching off of a target branch. When running git rebase target, the common base P is computed to determine the commit list A, B, and C. These are then cherry-picked on top of target in order to construct new commits A', B', and C'.

  

  The commits A', B', and C' are brand new commits that share a lot of information with A, B, and C, but are distinct new objects. In fact, the old commits still exist in your repository until garbage collection runs.

  We can even inspect how these two commit ranges are different using the git range-diff command! I’ll use some example commits in the Git repository to rebase onto the v2.29.2 tag, then modify the tip commit slightly.

  $ git checkout -f 8e86cf65816
  $ git rebase v2.29.2
  $ echo extra line >>README.md
  $ git commit -a --amend -m "replaced commit message"
  $ git range-diff v2.29.2 8e86cf65816 HEAD
  1:  17e7dbbcbc = 1:  2aa8919906 sideband: avoid reporting incomplete sideband messages
  2:  8e86cf6581 ! 2:  e08fff1d8b sideband: report unhandled incomplete sideband messages as bugs
      @@ Metadata
       Author: Johannes Schindelin <[email protected]>
       
        ## Commit message ##
      -    sideband: report unhandled incomplete sideband messages as bugs
      +    replaced commit message
       
      -    It was pretty tricky to verify that incomplete sideband messages are
      -    handled correctly by the `recv_sideband()`/`demultiplex_sideband()`
      -    code: they have to be flushed out at the end of the loop in
      -    `recv_sideband()`, but the actual flushing is done by the
      -    `demultiplex_sideband()` function (which therefore has to know somehow
      -    that the loop will be done after it returns).
      -
      -    To catch future bugs where incomplete sideband messages might not be
      -    shown by mistake, let's catch that condition and report a bug.
      -
      -    Signed-off-by: Johannes Schindelin <[email protected]>
      -    Signed-off-by: Junio C Hamano <[email protected]>
      + ## README.md ##
      +@@ README.md: and the name as (depending on your mood):
      + [Documentation/giteveryday.txt]: Documentation/giteveryday.txt
      + [Documentation/gitcvs-migration.txt]: Documentation/gitcvs-migration.txt
      + [Documentation/SubmittingPatches]: Documentation/SubmittingPatches
      ++extra line
       
        ## pkt-line.c ##
       @@ pkt-line.c: int recv_sideband(const char *me, int in_stream, int out)

  Notice that the resulting range-diff claims that commits 17e7dbbcbc and 2aa8919906 are “equal”, which means they would generate the same patch. The second pair of commits are different, showing that the commit message changed and there is an edit to the README.md that was not in the original commit.

  If you are following along, you can also see how the commit history still exists for these two commit sets. The new commits have the v2.29.2 tag as the third commit in the history while the old commits have the (earlier) v2.29.0-rc0 tag as the third commit.

  $ git log --oneline -3 HEAD
  e08fff1d8b2 (HEAD) replaced commit message
  2aa89199065 sideband: avoid reporting incomplete sideband messages
  898f80736c7 (tag: v2.29.2) Git 2.29.2
  
  $ git log --oneline -3 8e86cf65816
  8e86cf65816 sideband: report unhandled incomplete sideband messages as bugs
  17e7dbbcbce sideband: avoid reporting incomplete sideband messages
  47ae905ffb9 (tag: v2.28.0) Git 2.28

  Since commits aren’t diffs, how does Git track renames?

  If you were looking carefully at the object model, you might have noticed that Git never tracks changes between commits in the stored object data. You might have wondered “how does Git know a rename happened?”

  Git doesn’t track renames. There is no data structure inside Git that stores a record that a rename happened between a commit and its parent. Instead, Git tries to detect renames during the dynamic diff calculation. There are two stages to this rename detection: exact renames and edit-renames.

  After first computing a diff, Git inspects the internal model of that diff to discover which paths were added or deleted. Naturally, a file that was moved from one location to another would appear as a deletion from the first location and an add in the second. Git attempts to match these adds and deletes to create a set of inferred renames.

  The first stage of this matching algorithm looks at the OIDs of the paths that were added and deleted and see if any are exact matches. Such exact matches are paired together.

  The second stage is the expensive part: how can we detect files that were renamed and edited? Git iterates through each added file and compares that file against each deleted file to compute a similarity score as a percentage of lines in common. By default, anything larger than 50% of lines in common counts as a potential edit-rename. The algorithm continues comparing these pairs until finding the maximum match.

  Did you notice a problem? This algorithm runs A * D diffs, where A is the number of adds and D is the number of deletes. This is quadratic! To avoid extra-long rename computations, Git will skip this portion of detecting edit-renames if A + D is larger than an internal limit. You can modify this limit using the diff.renameLimit config option. You can also avoid the algorithm altogether by disabling the diff.renames config option.

  I’ve used my awareness of the Git rename detection in my own projects. For example, I forked VFS for Git to create the Scalar project and wanted to re-use a lot of the code but also change the file structure significantly. I wanted to be able to follow the history of these files into the versions in the VFS for Git codebase, so I constructed my refactor in two steps:

  1. Rename all of the files without changing the blobs.
  2. Replace strings to modify the blobs without changing filenames.

  These two steps ensured that I can quickly use git log --follow -- <path> to see the history of a file across this rename.

  $ git log --oneline --follow -- Scalar/CommandLine/ScalarVerb.cs
  4183579d console: remove progress spinners from all commands
  5910f26c ScalarVerb: extract Git version check
  ...
  9f402b5a Re-insert some important instances of GVFS
  90e8c1bd [REPLACE] Replace old name in all files
  fb3a2a36 [RENAME] Rename all files
  cedeeaa3 Remove dead GVFSLock and GitStatusCache code
  a67ca851 Remove more dead hooks code
  ...

  I abbreviated the output, but these last two commits don’t actually have a path corresponding to Scalar/CommandLine/ScalarVerb.cs, but instead it is tracking the previous path GVSF/GVFS/CommandLine/GVFSVerb.cs because Git recognized the exact-content rename from the commit fb3a2a36 [RENAME] Rename all files.

  Won’t be fooled again!

  You now know that commits are snapshots, not diffs! This understanding will help you navigate your experience working with Git.

  Now you are armed with deep knowledge of the Git object model. You can use this knowledge to expand your skills in using Git commands or deciding on workflows for your team. In a future blog post, we will use this knowledge to learn about different Git clone options and how to reduce the data you need to get things done!

  
  
  https://github.blog/2020-12-17-commits-are-snapshots-not-diffs/
• 

  Reducing flaky builds by 18x
  • sorangutan  

  1
  1
  Posts
  14
  Views

  

  Part of the Building GitHub blog series.

  It’s four o’clock in the afternoon as you push the last tweak to your branch. Your teammate already reviewed and approved your pull request and now all that’s left is to wait for CI. But, fifteen minutes later, your commit goes red. Surprised and a bit annoyed because the last five commits were green, you take a closer look only to find a failing test unrelated to your changes. When you run the test locally, it passes.

  Half an hour later, after running the test again and again locally without a single failure and retrying a still-red CI, you’re no closer to deploying your code. As the clock ticks past five, you retry CI once more. But this time, something is different: it’s green!

  You deploy, merge your pull request, and, an hour later than expected, close your laptop for the day.

  This is the cost of a flaky test. They’re puzzling, frustrating, and waste your time. And try as we might to stop them, they’re about as common as a developer who brews pour-over coffee every morning (read: very).

  Bonus points if you can guess what caused the test to fail.

  How far we’ve come

  Earlier this year in our monolith, 1 in 11 commits had at least one red build caused by a flaky test, or about 9 percent of commits. If you were trying to deploy something with a handful of commits, there was a good chance you’d need to retry the build or spend time diagnosing a failure, even if your code was fine. This slowed us down.

  Six weeks ago, after introducing a system to manage flaky tests, the percentage of commits with flaky builds dropped to less than half a percent, or 1 in 200 commits.

  

  This is an 18x improvement and the lowest rate of flaky builds since we began tracking flaky tests in 2016.

  So, how does it work?

  

  Say you just merged a test that would fail once every 1,000 builds. It didn’t fail during development or review but a few hours later, the test fails on your teammate’s branch.

  When this failure occurs, the new system inspects it and finds it to be flaky. After ensuring the test passes when run against the same code, it keeps the build green. And it happens quickly: unless your teammate is watching closely, they won’t notice.

  But your flaky test is still out there, failing on other branches. Every time it does, the system keeps track of where it happened and how it failed, each time learning more about the failure. If it continues to affect developers, the system identifies it as high impact: it’s time for a human to investigate. Using test failure history and git blame, the system finds the commit most likely to have introduced the problem—your commit—and assigns an issue to you.

  

  Screenshot of GitHub’s internal CI tooling.

  From there, you can see information about the failure, including what may have caused it, where it failed, and who else might be responsible. After some poking around, you find the problem and merge a fix.

  The system noticed a problem, contained it, and delegated it the right person. In short, the only person bothered by a flaky test is the person who wrote it.

  # This ain't a blocker
  def test_fails_one_in_a_thousand
    assert rand(1000).zero?
  end

  How it works

  When we set out to build this new system, our intent wasn’t to fix every flaky test or to stop developers from introducing new flaky tests. Such goals, if not impossible, seemed impractical. Similar to telling a developer to never write another bug, it would be much more costly and only slightly less flaky. Rather, we set out to manage the inevitability of flaky tests.

  Focusing on what matters

  When inspecting the history of failures in our monolith, we found that about a quarter of our tests had failed flaky across three or more branches in the past two years. But as we filtered by occurence, we learned that the flakiness was not evenly distributed: most flaky test failed fewer than ten times and only 0.4 percent of flaky tests failed 100 times or more.

  

  This made one thing clear: not every flaky failure should be investigated.

  Instead, by focusing on this top 0.4 percent, we could make the most of our time. So which tests were in this group?

  Automating flake detection

  We define a “flaky” test result as a test that exhibits both a passing and a failing result with the same code. –John Micco, Google

  Since 2016, our CI has been able to detect whether a test failure is flaky using two complementary approaches:

  1. Same code, different results. Once a build finishes, CI checks for other builds run against the same code using the root git tree hash. If another build had different results—for example, a test failed on the first build but passed on the second—the test failure was marked as flaky. While this approach was accurate, it only worked if a build was retried.
  2. Retry tests that fail. When a test failed, it was retried again later within the same build. This could be used on every build at minimal cost. If the test passed when rerun, it was marked as flaky. However, certain types of flaky tests couldn’t be detected with this approach, such as a time-based flaky test. (If your test failed because it was a leap year, rerunning it two minutes later won’t help.)

  Unfortunately, these approaches were only able to identify 25 percent of the flaky failures, counting on developers to find the rest. Before delegating flaky test detection to CI, we needed an approach that was as good or better than a person.

  Making it better

  We decided to iterate on the test retry approach by rerunning the test three times, each in a scenario targeting a common cause of flakiness.

  1. Retry in the same process. This retry attempts to replicate the same conditions in which the test failed: same Ruby VM, same database, same host. If the test passes under the same conditions, it is likely caused by randomness in the code or a race condition.
  2. Retry in the same process, shifted into the future. This retry attempts to replicate the same conditions with one exception: time. If the test passes when run in the future, as simulated by test helpers, it is likely caused by an incorrect assumption about time (e.g., “there are 28 days in February”).
  3. Retry on a different host. This attempts to run the same code in a completely separate environment: different Ruby VM, different database, different host. If the test passes under these conditions but fails in the other two retries, it is likely caused by test order-dependence or some other shared state.

  Using this approach, we are able to automatically identify 90 percent of flaky failures.

  Further, because we kept a history of how a test would fail, we were also able to estimate the cause of flakiness: chance, time-based, or order-dependent. When it came time to fix a test, this gave the developer a headstart.

  Measuring impact

  Once we could accurately detect flaky failures, we needed a way to quantify impact to automate prioritization.

  To do this, we used information tracked with every test failure: build, branch, author, commit, and more. Using this information, a flaky test is given an impact score based on how many times it has failed as well as how many branches, developers, and deploys were affected by it. The higher the score, the more important the flaky test.

  Once the score exceeds a certain threshold, an issue is automatically opened and assigned to the people who most recently modified either the test files or associated code prior to the test becoming flaky. To help jog the memory of those assigned, a link to the commit that may have introduced the problem is added to the issue.

  Teams can also view flaky tests by impact, CODEOWNER, or suite, giving insight into the test suite and giving developers a TODO list for problem areas.

  Closing

  By reducing the number of flaky builds by 18x, the new system makes CI more trustworthy and a red builds more meaningful. If your pull request has a failure, it’s a sign you need to change something, not a sign you should hit Rebuild. When it comes time to deploy, you can be sure that your build won’t go red late in the day because a test doesn’t take into account daylight saving time.

  This keeps us moving, even when we make mistakes.

  
  
  https://github.blog/2020-12-16-reducing-flaky-builds-by-18x/
• 

  Encapsulating Ruby on Rails views
  • sorangutan  

  1
  1
  Posts
  37
  Views

  

  With the recent release of version 6.1, Ruby on Rails now supports the rendering of objects that respond to render_in, a change we introduced to the framework. It may be small (the two pull requests were less than a dozen lines), but this change has enabled us to develop a framework for building encapsulated views called ViewComponent.

  Why encapsulation matters

  Unlike models and controllers, Rails views are not encapsulated. All Rails views in an application exist in a single execution context, meaning they can share state. This makes them hard to reason about and difficult to test, as they cannot be easily isolated.

  The need for a new way of building views in our application emerged as the number of templates in the GitHub application grew into the thousands. We depended on a combination of presenters and partials with inline Ruby, tested by expensive integration tests that exercise the routing and controller layers in addition to the view layer.

  Inspired by our experience building component-based UI with React, we set off to build a framework to bring these ideas to server-rendered Rails views.

  Enter ViewComponent

  We created the ViewComponent framework for building reusable, testable & encapsulated view components in Ruby on Rails. A ViewComponent is the combination of a Ruby file and a template file. For example:

  test_component.rb

  class TestComponent < ViewComponent::Base
    def initialize(title:)
      @title = title
    end
  end
  

  test_component.html.erb

  <span title="<%= @title %>">
    <%= content %>
  </span>
  

  Which is rendered in a view:

  <%= render(TestComponent.new(title: "my title")) do %>
    Hello, World!
  <% end %>

  Returning:

  <span title="my title">Hello, World!</span>

  Unlike traditional Rails views, ViewComponent templates are executed within the context of the ViewComponent object, encapsulating their state and allowing them to be unit tested in isolation.

  For example, to test our TestComponent, we can write:

  require "view_component/test_case"
  
  class MyComponentTest < ViewComponent::TestCase
    def test_render_component
      render_inline(TestComponent.new(title: "my title")) { "Hello, World!" }
  
      assert_selector("span[title='my title']", text: "Hello, World!")
      # or, to just assert against the text:
      assert_text("Hello, World!")
    end
  end
  

  These kinds of unit tests enable us to test our view code directly, instead of via controller tests. They are also significantly faster: in the GitHub codebase, component unit tests take around 25 milliseconds each, compared to about six seconds for controller tests.

  In practice

  Over the past two years, we’ve made significant strides in using ViewComponent: we now have over 400 components used in over 1600 of our 4500+ templates. For example, every Counter and Blankslate on GitHub.com is rendered with a ViewComponent.

  We’re seeing several significant benefits from this architecture. Because ViewComponents can be unit tested against their rendered DOM, we’ve been able to reduce duplication of test coverage for shared templates, which we previously covered with controller tests. And since ViewComponent tests are so fast, we’re finding ourselves writing more of them, leading to higher confidence in our view code.

  We’ve also seen the positive impact of the consistency that component-driven view architecture can provide. When we implemented a ViewComponent for the status of pull requests, we discovered several locations where we had not updated our previous, copy-pasted implementation, to handle the then-recently-shipped draft pull request status. By standardizing on a source of truth for the UI pattern, we now have a single, consistent implementation.

  To see some of the ViewComponents we’re using in the GitHub application, check out the Primer ViewComponents library.

  Support for 3rd-party component frameworks such as ViewComponent is just one of many contributions GitHub engineers contributed to Rails 6.1. Other notable additions include support for horizontal sharding, strict loading, and template annotations.

  
  
  https://github.blog/2020-12-15-encapsulating-ruby-on-rails-views/
• 

  Applying DevSecOps to your software supply chain
  • sorangutan  

  1
  1
  Posts
  10
  Views

  

  This article was originally published on InfoWorld, and is republished here with permission. This is part of our blog series on DevSecOps and software security.

  Developers often want to do the ‘right’ thing when it comes to security, but they don’t always know what that is. In order to help developers continue to move quickly, while achieving better security outcomes, organizations are turning to DevSecOps. DevSecOps is the mindset shift of making all parties who are part of the application development lifecycle accountable for security of the application, by continuously integrating security across your development process. In practice, this means shifting security reviews and testing left – from auditing or enforcing at deployment time, to also checking security controls earlier at build or development time.

  For code your developers write, that means providing feedback on issues during the development process, so the developer doesn’t lose their flow. For dependencies your code pulls in as part of your software supply chain, what should you do?

  Let’s first define a dependency. A dependency is another binary that your software needs in order to run, specified as part of your application. Using a dependency allows you to leverage the power of open source, and to pull in code for functions that aren’t a core part of your application, or where you might not be an expert. They often define your software supply chain — GitHub’s 2019 State of the Octoverse Report showed that on average, each repository has more than 200 dependencies (disclosure: I work for GitHub). An upstream vulnerability in any one of these dependencies means you’re likely affected too. The reality of the software supply chain is that you are dependent on code you didn’t write, yet the dependencies still require work from you for ongoing upkeep. So where should you get started in implementing security controls?

  Create a unified CI/CD pipeline to shift security controls left

  Part of the goal of DevSecOps, and shifting left, is to provide not only feedback but also consistency and repeatability as part of the development environment. This isn’t unique to your supply chain, but applies to any security control.

  The earlier you can unify your CI/CD pipeline, the earlier you can implement controls, allowing your security controls to shift left. You don’t want to apply the same controls multiple times in different systems – it doesn’t scale, spreads your (already thin) security resources even thinner, allows inconsistencies to be introduced via drift or incompatibility in systems, and potentially worst of all, means you might miss something.

  The precursor to shifting left and applying DevSecOps isn’t a security control at all – it’s about improving developer tooling to provide a consistent way to write, build, test, and deploy code. Introducing a centralized system for any one of these can help you improve your security. Organizations will frequently tackle developer tools from the last step, and work backwards to the first – that is, adopting a consistent deployment strategy before adopting a consistent build strategy – with one exception, code. Even if you build locally, chances are, you’re checking your code in for posterity.

  You can start applying security controls to your code even without getting all the other steps unified. A developer-centric approach means your developers can stay in context and respond to issues as they code, not days later at deployment, or months later from a penetration test report. Building on a unified CI/CD pipeline, here are some tips on how your development team can apply DevSecOps to secure your software supply chain.

  Declare dependencies in code, so you discover them at development time

  First things first, in order to maintain your dependencies – for example, applying security patches – you need to know what your dependencies are. Seems straightforward, right?

  There are many ways to detect your dependencies, at different parts of your development process: by analyzing at the dependencies declared in code, for example, specified by a developer in a manifest file or lockfile; by tracking the dependencies pulled in as part of a build process; or by examining completed build artifacts, for example once they’re in your registry. Unfortunately, there is no perfect solution as all methods have their challenges, but you should pick the solution that best integrates with your existing development pipeline or use multiple solutions to give you insights into dependencies at each step in your development process.

  However, there are benefits to detecting dependencies in code, rather than later: you’re shifting that dependency management step left. This allows developers to immediately perform maintenance for dependencies – performing updates, applying security patches, or removing unnecessary dependencies – without waiting for feedback from a build or deployment step. And, if you don’t have a centralized or consistent build pipeline, and can’t apply a check later, detecting your dependencies in code means you can still infer this information. The main downside to detecting dependencies in code is that you might miss any artifacts pulled in later, for example, Gradle allows for dependencies to be resolved as part of a build, meaning build-time detection will contain more complete information.

  To accurately detect dependencies in code – and to more easily control what dependencies you use – you’ll want to explicitly specify them as part of your application’s manifest file or lockfile, rather than vendoring them into a repository (forking a copy of a dependency as part of your project, aka copy-pasting it). Vendoring makes sense if you have a good reason to fork the code, for example, to modify or limit functionality for your organization, or use this as a step to review dependencies (you know, actually tracking inputs from vendors). (Some ecosystems also favor using vendoring.) However, if you’re planning on using the upstream version, vendoring makes updating your dependencies harder. By specifying your dependencies explicitly, it’s easier for your development team to update: with a single line of code change in a manifest, rather than re-forking and copying a whole repository. In certain ecosystems, you can use a lockfile to ensure consistency, so you’re using the same version in your development environment as you are for your production build, and review changes like any other code changes.

  Provide a path paved with ‘golden’ packages for your development team

  You might already be familiar with the concept of ‘golden’ images, which are maintained and sanctioned by your organization, including the latest security patches. This is a common concept for containers, to provide developers with a base image on which they can build their containers, without having to worry about the underlying OS. The idea here is to only have to maintain one set of OSes, managed by a central team, that you know have been reviewed for security issues and validated in your environment. Well, why not do that for any other artifacts too?

  To supplement a unified CI/CD pipeline, you can provide a reference set of maintained artifacts and libraries. This is just a pre-emptive security control – rather than verifying that a package is up to date once it’s been built, give your developers what they need as an input to their build. For example, if multiple teams are using OpenSSL, you shouldn’t need every team to update it; if one team updates it (and there are sufficient tests in place!), then you should be able to change the default for all teams. This could be implemented by having a central internal package registry of your known good artifacts, that have already passed any security requirements, and have a clear owner responsible for updating if new versions are released.

  By providing a single set of packages, you’re ensuring all teams reference these. Keep in mind, the latest you can do this is in the build system, but this could also be done earlier in code, especially if you’re using a monorepo. An added benefit of sharing common artifacts and libraries is making it easier to tell if you’re ‘affected’ by a newly discovered vulnerability. If the corresponding artifact hasn’t been updated, you are! And then it’s just one change to address the issue, and for the update to flow downstream to all teams. Phew.

  Automate downstream build and deployment

  To make sure that developers’ hard work pays off, their changes actually need to make it to production! In creating a unified CI/CD pipeline, you cleared a path for changes that are made to code in a development environment to propagate downstream to testing and production environments. The next step is to simplify this with automation. In an ideal world, your development team only makes changes to a development environment, with any changes to that environment automatically pushed to testing, validated, and automatically rolled out (and back, if needed).

  Rather than applying DevOps and DevSecOps by requiring your development team to learn operations tools, you simplify those tools and feedback to what these teams need to know in order to make changes where they’re most familiar, in code. This should sound familiar – it’s what’s happening with trends like infrastructure as code, or GitOps – define things in code, and let your workflow tools handle making the actual change.

  If you can automate downstream build, testing, and deployment of your code – then your developers only need to focus on fixing code. Following DevSecOps principles, they don’t need to learn tooling to do validating testing, phased deployments, or whatever you might need in your environment. Crucially, for security, your development team doesn’t need to learn how to roll out a fix in order to apply a fix. Fixing a security issue in code and committing it is sufficient to ensure it (eventually) gets fixed in production. Instead, you can focus on quickly finding and fixing bugs in code.

  Creating a unified CI/CD pipeline allows you to shift security controls left, including for supply chain security. Then, to best apply DevSecOps principles to improve the security of your dependencies, you should ask your developers to declare your dependencies in code and in turn provide them with maintained ‘golden’ artifacts and automated downstream actions so they can focus on code. Since this requires changes to not just security controls, but your developers’ experience, just using security tooling isn’t sufficient to implement DevSecOps. In addition to enabling platform-native dependency management features, you’ll also want to take a closer look at your CI/CD pipeline and artifact management.

  Altogether, applying DevSecOps means you can have a better understanding of what’s in your supply chain. By using DevSecOps, it should be simpler to manage your dependencies, with a change to a manifest or lockfile easily updating a single artifact in use across multiple teams, and automation of your CI/CD pipeline ensuring that changes developers make quickly end up in production.

  
  
  https://github.blog/2020-12-03-applying-devsecops-to-your-software-supply-chain/
• 

  The evolving role of operations in DevOps
  • sorangutan  

  1
  1
  Posts
  14
  Views

  

  This is the third blog post in our series of DevOps fundamentals. For a quick intro on what DevOps is, check out part one; for a primer on automation in DevOps, visit part two.

  As businesses reorganize for DevOps, the responsibilities of teams throughout the software lifecycle inevitably shift. Operations teams that traditionally measure themselves on uptime and stability—often working in silos separate from business and development teams—become collaborators with new stakeholders throughout the software lifecycle. Development and operations teams begin to work closely together to build and continually improve their delivery and management processes. In this blog post, we’ll share more on what these evolving roles and responsibilities look like for IT teams today, and how operations help drive consistency and success across the entire organization.

  The Ops role in DevOps compared to traditional IT operations

  To better understand how DevOps changes the responsibilities of operations teams, it will help to recap the traditional, pre-DevOps role of operations. Let’s take a look at a typical organization’s software lifecycle: before DevOps, developers package an application with documentation, and then ship it to a QA team. The QA teams install and test the application, and then hand off to production operations teams. The operations teams are then responsible for deploying and managing the software with little-to-no direct interaction with the development teams.

  These dev-to-ops handoffs are typically one-way, often limited to a few scheduled times in an application’s release cycle. Once in production, the operations team is then responsible for managing the service’s stability and uptime, as well as the infrastructure that hosts the code. If there are bugs in the code, the virtual assembly line of dev-to-qa-to-prod is revisited with a patch, with each team waiting on the other for next steps. This model typically requires pre-existing infrastructure that needs to be maintained, and comes with significant overhead. While many businesses continue to remain competitive with this model, the faster, more collaborative way of bridging the gap between development and operations is finding wide adoption in the form of DevOps.

  Accelerating through public cloud adoption

  Over the past decade, the maturation of the public cloud has added complexity to the responsibilities of operations teams. The ability to rent stable, secure infrastructure by the minute and provide everything as a service to customers has enabled organizations to deploy rapidly and frequently, often several times per day. Smaller, faster delivery cycles give organizations the critical capability of improving their customer experience through rapid feedback and automated deployments. Cloud technologies have made development velocity a fundamental part of delivering a competitive customer experience.

  What the cloud, DevOps, and developer velocity mean for operations teams

  Cloud technologies have transformed how we deliver and operate software, impacting how we do DevOps  today. Developers now focus more on stability and uptime in addition to developer velocity, and operations teams now have a stake in developer velocity along with their traditional role of maintaining uptime. When it comes to the specific role of operations in DevOps, this often means:

  • Enabling self-service for developers. In order to support developer velocity—and minimize risks that stem from “shadow operations”, where developers seek their own solutions—operations teams work more closely with developers to provide on-demand access to secure, compliant tooling and environments.

  • Standardized tooling and processes across the business. The best way to enable a sustainable self-service model and empower teams to work more efficiently together is by standardizing on the tooling that is in use. Tools and processes that are shared across the business unit enable organizational unity and greater collaboration. In turn, this reduces the friction developers and operations teams experience when sharing responsibilities.

  • Bringing extensible automation to traditional operations tasks. As operations teams focus more on empowering other teams through self-service and collaboration, there is less time to handle other work. Traditional operations tasks like resolving incidents, updating systems, or scaling infrastructure still need to be addressed—only smarter. When development and operations unite under DevOps, operations teams turn to automation for more of the repeatable tasks and drive consistency across the organization. This also enables teams and business units to track and measure the results of their efforts.

  • Working and shipping like developers. As operations teams shift more towards greater automation, ‘X’ as code becomes the new normal. Like application source code, the code controlling operations systems needs to be stored, versioned, secured, and maintained. As a result, the development-operations relationship starts to feel more balanced on both sides: operations specialists become more like the developers and more familiar with their working models, and in some organizations, developers become more like operations, sharing in the responsibility of debugging problems with their own code in production.

  Closing the development-operations gap

  While it’s well understood that DevOps requires close collaboration between teams, we’re often asked “How are development and operations functions really coordinated in a DevOps model?” At GitHub, we’re fortunate to partner with thousands of businesses every year on improving their DevOps practices. Sometimes these organizations focus on the clearest target, asking developers and delivery teams to go to market faster while paying less attention to the post-deployment operations teams.

  However, we find the best results come through improving the practices of all the teams involved in the software lifecycle, together. Operations teams aren’t simply infrastructure and process owners for the organizations, but are also a critical part of the feedback loop for development. Try it out for yourself—a small pilot project that includes developers, release engineering, operations, and even InfoSec can give more teams the momentum they need. It can give them confidence to continue their work, establish best practices, and even train others within your organization along the way.

  For a closer look at IT operations in DevOps, tune in to next week’s GitHub Universe session: Continuous delivery with GitHub Actions. 

  
  
  https://github.blog/2020-12-03-the-evolving-role-of-operations-in-devops/
• 

  Improving the GHES release process: release candidates
  • sorangutan  

  1
  1
  Posts
  15
  Views

  

  In our ongoing “Building GitHub” series, we talk about some of the projects we’re working on to improve how efficiently we build GitHub, as well as increase GitHub’s availability, stability, and resilience. We know how important the stability of our platform is for developers and enterprises, and it continues to be a priority area of investment across GitHub. 

  In that spirit, we want to share a change in how we make new feature releases available to our GitHub Enterprise Server customers. This change will take effect with our next release, and we hope this increases our collaboration with our GHES customers and improves our release process.

  What are Release Candidates?

  Release candidates, or RCs, are builds that allow our GitHub Enterprise Server customers to try the latest release early. These RCs are a way for us to work with our customers on bugs and issues that will be used to improve the quality of every release. 

  Working in the open like this is the best way for us to collaborate with our customers to improve GitHub Enterprise Server and ensure that we are delivering a product that meets and (hopefully) exceeds expectations. 

  The Release Candidate Process

  What can I expect with this new process?

  Customers can start testing an RC as soon as it’s available, and release notes will accompany each RC. We expect each feature release will have one or more RC versions (eg. 2.22.0.RC1, 2.22.0.RC2), with each new version adding bug fixes for issues found in prior versions. The number of RCs will be driven by customer feedback, and we’ll decide based on quality and customer feedback when to publish and make generally available a final production release. 

  RCs can be upgraded from any version and can upgrade to any version. They can be deployed on staging or production environments. 

  Customers that test RCs can raise issues with GitHub Support. Each RC is supported while live, but is not included in long-term support. 

  What does this mean for other releases?

  • Production releases will continue to be numbered as they are today (2.20, 2.21, etc.)
  • Patch releases will not be released as RCs

  With this new RC process, testing and feedback from our customers will be critical. We’re confident this will help us improve GitHub Enterprise Server, together. We’ll have more to share about upcoming RCs at GitHub Universe next week. Make sure you tune in!

  
  
  https://github.blog/2020-12-03-improving-the-ghes-release-process-release-candidates/
• 

  GitHub Availability Report: November 2020
  • sorangutan  

  1
  1
  Posts
  9
  Views

  

  Introduction

  In November, we experienced two incidents resulting in significant impact and degraded state of availability for issues, pull requests, and GitHub Actions services.

  November 2 12:00 UTC (lasting 32 minutes)

  The SSL certificate for *.githubassets.com expired, impacting web requests for GitHub.com UI and services. There was an auto-generated issue indicating the certificate was within 30 days of expiration, but it was not addressed in time. Impact was reported, and the on-call engineer remediated it promptly.

  We are using this occurrence to evaluate our current processes, as well as our tooling and automation, within this area to prevent the likelihood of such instances in the future.

  November 27 16:04 UTC (lasting one hour and one minute)

  Our service monitors detected abnormal levels of replication lag within one of our MySQL clusters affecting GitHub Actions.

  Due to the recency of this incident, we are still investigating the contributing factors and will provide a more detailed update in next month’s report.

  In summary

  We place great importance in the reliability of our services along with the trust that our users place in us every day. We’ll continue to keep you updated on the progress we’re making to ensure this. To learn more about what we’re working on, visit the GitHub engineering blog.

  
  
  https://github.blog/2020-12-02-availability-report-november-2020/
• 

  Nbdev: A literate programming environment that democratizes software engineering best practices
  • sorangutan  

  1
  1
  Posts
  15
  Views

  

  At GitHub, we are deeply invested in democratizing software development. Part of this is accomplished by serving as the home of open source and providing tools to educators and students. We are also building features that lower the barrier to entry for software development, such as Codespaces. However, there is much work left to be done in order to make software development more approachable and to make it easier to employ best practices, such as continuous integration, distribution, and documentation of software.

  This is why we decided to assist fastai in their development of a new, literate programming environment for Python, called nbdev. A discussion of the motivations behind nbdev as well as a primer on the history of literate programming can be found in this blog post. For the uninitiated, literate programming, as described by Donald Knuth, it is:

  …a move away from writing computer programs in the manner and order imposed by the computer, and instead enables programmers to develop programs in the order demanded by the logic and flow of their thoughts.

  While a subset of ideas from literate programming have shown up in tools, such as Swift Playgrounds, Jupyter, and Mathematica, there has been a lack of tools that encompass the entire software development life cycle. nbdev builds on top of Jupyter notebooks to fill these gaps and provides the following features, many of which are integrated with GitHub:

  • Automated generation of docs from Jupyter notebooks hosted on GitHub Pages. These docs are searchable and automatically hyperlinked to appropriate documentation pages by introspecting keywords you surround in backticks. An example of this documentation is the official fastai docs.
  • Continuous integration (CI) comes setup for you with GitHub Actions, which will run unit tests automatically for you. Even if you are not familiar with GitHub Actions, this starts working right away without any manual intervention.
  • The nbdev environment, which consists of a web server for previewing a docs site, a Jupyter server for writing code, and a series of CLI tools are set up to work with GitHub Codespaces, which makes getting started even easier. A detailed discussion of how CodeSpaces integrates with nbdev is discussed in this blog post.

  As a teaser, this is a preview of this literate programming environment in Codespaces, which includes a notebook, a docs site and an IDE:

   

  In addition to this GitHub integration, nbdev also offers the following features:

  • A robust, two-way sync between notebooks and source code, which allow you to use your IDE for code navigation or quick edits if desired.
  • The ability to write tests directly in notebooks without having to learn special APIs. These tests get executed in parallel with a single CLI command and also with GitHub Actions.
  • Tools for merge/conflict resolution with notebooks in a human readable format.
  • Utilities to automate the publishing of pypi and conda packages.

  nbdev promotes software engineering best practices by allowing developers to write unit tests and documentation in the same context as source code, without having to learn special APIs or worry about web development. Similarly, GitHub Actions run unit tests automatically by default without requiring any prior experience with these tools. We believe removing friction from writing documentation and tests promotes higher quality software and makes software more inclusive.

  Aside from using nbdev to create Python software, you can extend nbdev to build new types of tools. For example, we recently used nbdev to build fastpages, an easy to use blogging platform that allows developers to create blog posts directly with Jupyter notebooks. fastpages uses GitHub Actions and GitHub Pages to automate the conversion of notebooks to blog posts and offers a variety of other features to Python developers that democratize the sharing of knowledge. We have also used nbdev and fastpages to create covid19-dashboards, which demonstrates how to create interactive dashboards that automatically update with Jupyter notebooks.

  We are excited about the potential of nbdev to make software engineering more inclusive, friendly, and robust.  We are also hopeful that tools like nbdev can inspire the next generation of literate programming tools. To learn more about nbdev, please see the following resources:

  • The official nbdev docs
  • A minimal example of using nbdev end-to-end
  • The blog post discussing the background and motivations behind nbdev, by its creator, Jeremy Howard
  • A presentation discussing nbdev from a keynote at this year’s JupyterCon

  Finally, If you are building any projects with nbdev or would like to have further discussions, please feel free to reach out on the nbdev forums or on GitHub.

  
  
  https://github.blog/2020-11-20-nbdev-a-literate-programming-environment-that-democratizes-software-engineering-best-practices/
• 

  Getting started with DevOps automation
  • sorangutan  

  1
  1
  Posts
  12
  Views

  

  {"@context":"https://schema.org","@type":"FAQPage","mainEntity":[{"@type":"Question","name":"Is DevOps all about automation?","acceptedAnswer":{"@type":"Answer","text":"Answered succinctly, no. But automation is an important means to accomplishing the work. Whether you’re new to DevOps or migrating from another set of automation solutions, adopting new tooling with a small with a trial project or process will give you the base set of learnings to help scale and standardize automation across your entire organization. With this automation in place you will lay the foundation for measuring effectiveness and further progress toward your goals."}}]}

  This is the second post in our series on DevOps fundamentals. For a guide to what DevOps is and answers to common DevOps myths check out part one.

  What role does automation play in DevOps?

  First things first—automation is one of the key principles for accelerating with DevOps. As noted in my last blog post, it enables consistency, reliability, and efficiency within the organization, making it easier for teams to discover and troubleshoot problems. 

  However, as we’ve worked with organizations, we’ve found not everyone knows where to get started, or which processes can and should be automated. In this post, we’ll discuss a few best practices and insights to get teams moving in the right direction.

  A few helpful guidelines

  The path to DevOps automation is continually evolving. Before we dive into best practices, there are a few common guidelines to keep in mind as you’re deciding what and how you automate. 

  • Choose open standards. Your contributors and team may change, but that doesn’t mean your tooling has to. By maintaining tooling that follows common, open standards, you can simplify onboarding and save time on specialized training. Community-driven standards for packaging, runtime, configuration, and even networking and storage—like those found in Kubernetes—also become even more important as DevOps and deployments move toward the cloud.
  • Use dynamic variables. Prioritizing reusable code will reduce the amount of rework and duplication you have, both now and in the future. Whether in scripts or specialized tools, securely using externally-defined variables is an easy way to apply your automation to different environments without needing to change the code itself.
  • Use flexible tooling you can take with you. It’s not always possible to find a tool that fits every situation, but using a DevOps tool that allows you to change technologies also helps reduce rework when companies change direction. By choosing a solution with a wide ecosystem of partner integrations that works with any cloud, you’ll be able to  define your unique set of best practices and reach your goals—without being restricted by your toolchain.

  DevOps automation best practices

  Now that our guidelines are in place, we can evaluate which sets of processes we need to automate. We’ve broken some best practices for DevOps automation into four categories to help you get started. 

  1. Continuous integration, continuous delivery, and continuous deployment

  We often think of the term “DevOps” as being synonymous with “CI/CD”. At GitHub we recognize that DevOps includes so much more, from enabling contributors to build and run code (or deploy configurations) to improving developer productivity. In turn, this shortens the time it takes to build and deliver applications, helping teams add value and learn faster. While CI/CD and DevOps aren’t precisely the same, CI/CD is still a core component of DevOps automation.

  • Continuous integration (CI) is a process that implements testing on every change, enabling users to see if their changes break anything in the environment. 
  • Continuous delivery (CD) is the practice of building software in a way that allows you to deploy any successful release candidate to production at any time.
  • Continuous deployment (CD) takes continuous delivery a step further. With continuous deployment, every successful change is automatically deployed to production. Since some industries and technologies can’t immediately release new changes to customers (think hardware and manufacturing), adopting continuous deployment depends on your organization and product.

  Together, continuous integration and continuous delivery (commonly referred to as CI/CD) create a collaborative process for people to work on projects through shared ownership. At the same time, teams can maintain quality control through automation and bring new features to users with continuous deployment. 

  2. Change management

  Change management is often a critical part of business processes. Like the automation guidelines, there are some common principles and tooling that development and operations teams can use to create consistency.  

  • Version control: The practice of using version control has a long history rooted in helping people revert changes and learn from past decisions. From RCS to SVN, CVS to Perforce, ClearCase to Git, version control is a staple for enabling teams to collaborate by providing a common workflow and code base for individuals to work with. 
  • Change control: Along with maintaining your code’s version history, having a system in place to coordinate and facilitate changes helps to maintain product direction, reduces the probability of harmful changes to your code, and encourages a collaborative process.
  • Configuration management: Configuration management makes it easier for everyone to manage complex deployments through templates and manage changes at scale with proper controls and approvals.

  3. ‘X’ as code

  By now, you also may have heard of “infrastructure as code,” “configuration as code,” “policy as code,” or some of the other “as code” models. These models provide a declarative framework for managing different aspects of your operating environments through high level abstractions. Stated another way, you provide variables to a tool and the output is consistently the same, allowing you to recreate your resources consistently. DevOps implements the “as code” principle with several goals, including: an auditable change trail for compliance, collaborative change process via version control, a consistent, testable and reliable way of deploying resources, and as a way to lower the learning curve for new team members. 

  • Infrastructure as code (IaC) provides a declarative model for creating immutable infrastructure using the same versioning and workflow that developers use for source code. As changes are introduced to your infrastructure requirements, new infrastructure is defined, tested, and deployed with new configurations through automated declarative pipelines.
  • Platform as code (PaC) provides a declarative model for services similar to how infrastructure as code provides a framework for recreating the same infrastructure—allowing you to rapidly deploy services to existing infrastructure with high-level abstractions.
  • Configuration as code (CaC) brings the next level of declarative pipelining by defining the configuration of your applications as versioned resources.
  • Policy as code brings versioning and the DevOps workflow to security and policy management. 

  4. Continuous monitoring

  Operational insights are an invaluable component of any production environment. In order to understand the behaviors of your software in production, you need to have information about how it operates. Continuous monitoring—the processes and technology that monitor performance and stability of applications and infrastructure throughout the software lifecycle—provides operations teams with data to help troubleshoot, and development teams the information needed to debug and patch. This also leads into an important aspect of security, where DevSecOps takes on these principles with a security focus. Choosing the right monitoring tools can be the difference between a slight service interruption and a major outage. When it comes to gaining operational insights, there are some important considerations: 

  • Logging gives you a continuous stream of data about your business’ critical components. Application logs, infrastructure logs, and audit logs all provide important data that helps teams learn and improve products.
  • Monitoring provides a level of intelligence and interpretation to the raw data provided in logs and metrics. With advanced tooling, monitoring can provide teams with correlated insights beyond what the raw data provides.
  • Alerting provides proactive notifications to respective teams to help them stay ahead of major issues. When effectively implemented, these alerts not only let you know when something has gone wrong, but can also provide teams with critical debugging information to help solve the problem quickly.
  • Tracing takes logging a step further, providing a deeper level of application performance and behavioral insights that can greatly impact the stability and scalability of applications in production environments.

  Putting DevOps automation into action

  At this point, we’ve talked much about automation in the DevOps space, so is DevOps all about automation? Put simply, no. Automation is an important means to accomplishing this work efficiently between teams. Whether you’re new to DevOps or migrating from another set of automation solutions, testing new tooling with a small project or process is a great place to start. It will lay the foundation for scaling and standardizing automation across your entire organization, including how to measure effectiveness and progression toward your goals. 

  Regardless of which toolset you choose to automate your DevOps workflow, evaluating your teams’ current workflows and the information you need to do your work will help guide you to your tool and platform selection, and set the stage for success. Here are a few more resources to help you along the way:

  • Periodic Table of DevOps Tools
  • DevOps tools glossary
  • Top DevOps tools list

  Want to see what DevOps automation looks like in practice? See how engineers at Wiley build faster and more securely with GitHub Actions.

  
  
  https://github.blog/2020-10-29-getting-started-with-devops-automation/
• 

  Making GitHub CI workflow 3x faster
  • sorangutan  

  1
  1
  Posts
  12
  Views

  

  Welcome to the first deep dive of the Building GitHub blog series, providing a look at how teams across the GitHub engineering organization identify and address opportunities to improve our internal development tooling and infrastructure.

  At GitHub, we use the Four Key Metrics of high performing software development to help frame our engineering fundamentals effort. As we measured Lead Time for Changes—the time it takes for code to be successfully running in production—we identified that developers waited an average of 45 minutes for a successful run of our continuous integration suite to complete before merging any change. This 45-minute lead time was repeated once more before deploying a merge branch. In a perfect scenario, a developer waited almost two hours after checking in code before the change went live on GitHub.com. This 45-minute CI now takes only 15 minutes to run! Here is a deep dive on how we made GitHub’s CI workflow 3x faster.

  Analyzing the problem

  At this moment the monumental Ruby monolith that powers millions of developers on GitHub.com, has over 7,000 test suites and over 5,000 test files. Every commit to a pull request triggers 25 CI jobs and requires 15 of those CI jobs to complete before merging a pull request. This meant that a developer at GitHub spent approximately 45 minutes and 600 cores of computing resources for every commit. That’s a lot of developer-hours and machine-hours that could be spent creating value for our customers.

  Analyzing the types of CI jobs, we identified four categories: unit testing, linting/performance, integration testing, builds/deployments. All jobs except two of the integration testing jobs took less than 13 minutes to run. The two integration testing jobs were the bottleneck in our Lead Time for Changes. As it is true for most DevOps cycles, several test suites were also flaky. Although this blog post isn’t going to share how we solved for the flakiness of our tests, spoiler alert, a future post in this series will explain that process. Apart from being flaky, the two integration testing jobs increased developer friction and reduced productivity at GitHub.

  Engineering decision

  GitHub Enterprise Server, the on-premise offering of GitHub used by our enterprise customers, ships a new patch release every two weeks and a major release every quarter. The two long running test suites were added to the CI workflow to ensure a pull request did not break the GitHub experience for our Enterprise Server customers. It was also clear that these 45-minute test suites did not provide additional value blocking GitHub.com deployments that happen continuously throughout the day. Driven by customer obsession and developer satisfaction, we developed the deferred compliance tool.

  Deferred compliance

  The deferred compliance tool integrated along with our CI workflow system aims to strike a critical balance between improving Lead Time for Change in deploying GitHub.com and creating accountability for the quality of Enterprise Server. The long running CI jobs are no longer required to pass before a pull request is merged but the deferred compliance tool is monitoring for any test failure.

  

  If a CI job fails, a GitHub issue with a deferred compliance label is created and the pull request author and code segment’s code owners are tagged. A warning message is sent on Slack to the developer and a 72-hour timer is kicked off. The developer now has 72 hours to fix the build, push a change or revert the pull request. A successful run of the CI job automatically closes the compliance issue and the 72-hour timer is turned off. If the CI job remains broken for more than 72 hours, all deployments to GitHub.com are halted, barring any exceptional situations, until the integration tests for Enterprise Server are fixed. This creates accountability and ownership for all our developers to build features that work flawlessly on GitHub.com and Enterprise Server. The 72-hour timer is customizable but our analysis showed that with a global team of developers, 72 hours reduced the possibility that a change merged by a developer in San Francisco on a Friday afternoon did not unintentionally block deployments for a developer in Sydney on Monday morning. Deferred compliance can be used for any long running CI run that does not need to block deployments while creating a call for action for CI run failures.

  Key Takeaways

  • Internal engineering tooling is a powerful resource to support developers and at the same time provide guardrails for product consistency.
  • Focusing on a key metric allows us to identify bottlenecks and develop simple and creative solutions.
  • Comprehending historical context for past decisions and being customer obsessed provides us an opportunity to build a more thoughtful engineering design.

  Overall, this project is a testimony that a simple solution can significantly improve developer productivity and that can have long-term positive implications to an engineering organization. And of course, since numbers matter, we made our CI 3x faster.

  
  
  https://github.blog/2020-10-29-making-github-ci-workflow-3x-faster/
• 

  Building GitHub: introduction
  • sorangutan  

  1
  1
  Posts
  13
  Views

  

  Here at GitHub, we pride ourselves on providing a first-class developer experience to you, our customers. We’re developers, too, and we love that the features that we build for GitHub.com make your day easier — and make ours easier, too. We also know that the more we invest in the infrastructure and tooling that powers GitHub, the faster we can deliver those features, and we’ll have a more delightful experience to boot.

  In addition to investing in the infrastructure, we also want to shine a light on all the hard work we do behind the scenes to make GitHub better, specifically focusing on our internal development tooling and infrastructure. And, today, we’re excited to introduce the Building GitHub blog series, providing deep-dives on how teams across the engineering organization have been banding together to identify and address opportunities that would provide us an even smoother internal development experience, up our technical excellence, and improve system reliability in the process. From running the latest and greatest Ruby version, to dramatically decreasing our application boot time, to smoother and more reliable progressive deploys, these efforts paid off greatly and decreased our cycle times.

  To help frame our efforts for potential investments, we revisited the Four Key Metrics of high performing software delivery, as our very own Dr. Nicole Forsgren found in her research and outlined by DevOps Research and Assessment. These include:

  • Deploy Frequency. How frequently is the team deploying?
  • Lead Time for Changes. How long does it take to get code successfully running in production?
  • Time to Restore Service. How long does it take to recover from an incident?
  • Change Fail Rate. What percentage of changes to production result in degraded service?

  Ideally, any investment we make in our development tooling would move the needle in at least one of these areas. We’ve had teams across the organization join together to tackle these, sometimes diving into areas of our internal systems that they weren’t previously familiar with. This approach provides the opportunity to explore new solutions, and collaborate cross-team and cross-discipline. The excitement of engineers involved in each of these efforts is palpable — not only are we thrilled when we notice a dramatic shift in boot time or introduce new tooling that makes monitoring and debugging even easier, but teams enjoy working more closely with engineers in other parts of the org.

  Continue reading along with us in the Building GitHub blog series, where we’ll share specific goals and lessons, the impact of our work, and how we did it. To continue this journey, we’ll start the series with a deep dive on faster CI and we hope to share more soon.

  
  
  https://github.blog/2020-10-29-building-github-introduction/
• 

  How to get your organization started with containerized deployments
  • sorangutan  

  1
  1
  Posts
  26
  Views

  

  This is our second post on cloud deployment with containers. Looking for more? Join our upcoming GitHub Actions webcast with Sarah, Solutions Engineer Benedict Oleforo, and Senior Product Manager Kayla Ngan on October 22.

  In the past few years, businesses have moved towards cloud-native operating models to help streamline operations and move away from costly infrastructure. When running applications in dynamic environments with Docker, Kubernetes, and other tooling, a container becomes the tool of choice as a consistent, atomic unit of packaging, deployment, and application management. This sounds straightforward: build a new application, package it into containers, and scale elastically across the infrastructure of your choice. Then you can automatically update with new images as needed and focus more on solving problems for your end users and customers.

  However, organizations don’t work in vacuums. They’re part of a larger ecosystem of customers, partners, and open source communities, with unique cultures, existing processes, applications, and tooling investments in place. This adds new challenges and complexity for adopting cloud native tools such as containers, Kubernetes, and other container schedulers.

  Challenges for adopting container-based strategies in organizations

  At GitHub, we’re fortunate to work with many customers on their container and DevOps strategy. When it comes to adopting containers, there are a few consistent challenges we see across organizations.

  • Containerizing and maintaining applications: Most organizations have existing applications and need to make the decision about whether to keep them as-is, or to place them in containers for an easier transition to the cloud. Even then, teams need to determine whether a single container for the application is appropriate (in a lift-and-shift motion to the cloud), or if more extensive work is needed to break it down into multiple services, delivered as a set of containers.
  • Efficiently configuring and managing permissions: Adopting containers often translates to better collaboration for everyone in your organization. DevOps is now more than just core developers and IT operators. It includes release and infosec engineers, data scientists, QA, project managers, and other roles. But collaborating across multiple teams introduces new needs for configuring and managing permissions for code, along with the automation to support it.
  • Standardizing best practices across the organization: Containers help teams scale and integrate quickly, but may also require updating your CI/CD practices to match. You have to validate they work well for existing applications, while incorporating the correct user and package permissions and policies.. The best practices you set have to be flexible for others too. Individual teams—who are transitioning to new ways of working—need to be able to optimize for their own goals.

  Connecting teams and cloud-native tools with GitHub

  Despite the few challenges of adopting containers and leveraging Kubernetes, more and more organizations continue to use them. Stepping over those hurdles allows enterprises to automate and streamline their operations, here with a few examples of how enterprises make it work successfully with support from package managers and CI/CD tools. At GitHub, we’ve introduced container support in GitHub Packages, CI/CD through GitHub Actions, and partnered within the ecosystem to simplify cloud-native workflows. Finding the right container tools should mean less work, not more—easily integrating alongside other tools, projects, and processes your organization already uses.

  See container best practices in action

  

  Want to simplify container deployments in your organization? Join me, Solutions Engineer Benedict Oleforo, and Senior Product Manager Kayla Ngan on October 22 to learn more about successfully adopting containers. We’ll walk through how to use them in the real world and demo best practices for deploying an application to Azure with GitHub Container Registry.

  When
  October 22, 2020
  11:00 am PT / 2:00 pm ET

  Sign up for the webcast.

  
  
  https://github.blog/2020-10-15-how-to-get-your-organization-started-with-containerized-deployments/
• 

  Testing cloud apps with GitHub Actions and cloud-native open source tools
  • sorangutan  

  1
  1
  Posts
  15
  Views

  

  What makes a project successful? For developers building cloud-native applications, successful projects thrive on transparent, consistent, and rigorous collaboration. That collaboration is one of the reasons that many open source projects, like Docker containers and Kubernetes, grow to become standards for how we build, deliver, and operate software. Our Open Source Guides and Introduction to innersourcing are great first steps to setting up and encouraging these best practices in your own projects.

  However, a common challenge that application developers face is manually testing against inconsistent environments. Accurately testing Kubernetes applications can differ from one developer’s environment to another, and implementing a rigorous and consistent environment for end-to-end testing isn’t easy. It can also be very time consuming to spin up and down Kubernetes clusters. The inconsistencies between environments and the time required to spin up new Kubernetes clusters can negatively impact the speed and quality of cloud-native applications.

  Building a transparent CI process

  On GitHub, integration and testing becomes a little easier by combining GitHub Actions with open source tools. You can treat Actions as the native continuous integration and continuous delivery (CI/CD) tool for your project, and customize your Actions workflow to include automation and validation as next steps.

  Since Actions can be triggered based on nearly any GitHub event, it’s also possible to build in accountability for updating tests and fixing bugs. For example, when a developer creates a pull request, Actions status checks can automatically block the merge if the test fails.

  Here are a few more examples:

  

  Branch protection rules in the repository help enforce certain workflows, such as requiring more than one pull request review or requiring certain status checks to pass before allowing a pull request to merge.

  

  GitHub Actions are natively configured to act as status checks when they’re set up to trigger `on: [pull_request]`.

  Continuous integration (CI) is extremely valuable as it allows you to run tests before each pull request is merged into production code. In turn, this will reduce the number of bugs that are pushed into production and increases confidence that newly introduced changes will not break existing functionality.

  But transparency remains key: Requiring CI status checks on protected branches provides a clearly-defined, transparent way to let code reviewers know if the commits meet the conditions set for the repository—right in the pull request view.

  Using community-powered workflows

  Now that we’ve thought through the simple CI policies, automated workflows are next. Think of an Actions workflow as a set of “plug and play” open sourced, automated steps contributed by the community. You can use them as they are, or customize and make them your own. Once you’ve found the right one, open sourced Actions can be plugged into your workflow with the`- uses: repo/action-name` field.

  

  You might ask, “So how do I find available Actions that suit my needs?”

  The GitHub Marketplace!

  

  As you’re building automation and CI pipelines, take advantage of Marketplace to find pre-built Actions provided by the community. Examples of pre-built Actions span from a Docker publish and the kubectl CLI installation to container scans and cloud deployments. When it comes to cloud-native Actions, the list keeps growing as container-based development continues to expand.

  Testing with kind

  Testing is a critical part of any CI/CD pipeline, but running tests in Kubernetes can absorb the extra time that automation saves. Enter kind. kind stands for “Kubernetes in Docker.” It’s an open source project from the Kubernetes special interest group (SIGs) community, and a tool for running local Kubernetes clusters using Docker container “nodes.” Creating a kind cluster is a simple way to run Kubernetes cluster and application testing—without having to spin up a complete Kubernetes environment.

  

  As the number of Kubernetes users pushing critical applications to production grows, so does the need for a repeatable, reliable, and rigorous testing process. This can be accomplished by combining the creation of a homogenous Kubernetes testing environment with kind, the community-powered Marketplace, and the native and transparent Actions CI process.

  Bringing it all together with kind and Actions

  { "@context": "http://schema.org/", "@type": "VideoObject", "thumbnailurl": "https://github.blog/wp-content/uploads/2020/10/DevOps-Header.png?w=2415", "name": "GitHub Demo Days - Using GitHub Actions for testing cloud native applications", "description": "A common challenge that cloud native application developers face is manually testing against inconsistent environments. GitHub Actions can be triggered based on nearly any GitHub event making it possible to build in accountability for updating tests and fixing bugs.", "uploadDate": "2020-10-08T08:00:00+00:00", "embedUrl": "https://www.youtube.com/embed/-sprH1_jWAY", "publication": { "@type": "BroadcastEvent", "isLiveBroadcast": true, "startDate": "2020-10-16T3:00:00+00:00", "endDate": "2020-10-16T4:00:00+00:00" } }

  Come see kind and Actions at work during our next GitHub Demo Day live stream on October 16, 2020 at 11am PT. I’ll walk you through how to easily set up automated and consistent tests per pull request, including how to use kind with Actions to automatically run end-to-end tests across a common Kubernetes environment.

  
  
  https://github.blog/2020-10-09-devops-cloud-testing/
• 

  GitHub Availability Report: August 2020
  • sorangutan  

  1
  1
  Posts
  19
  Views

  

  Introduction

  In August, we experienced no incidents resulting in service downtime. This month’s GitHub Availability Report will dive into updates to the GitHub Status Page and provide follow-up details on how we’ve addressed the incident mentioned in July’s report.

  Status page refresh

  At GitHub, we are always striving to be more transparent and clear with our users. We know our customers trust us to communicate the current availability of our services and we are now making that more clear for everyone to see. Previously, our status page shared a 90-day history of GitHub’s availability by service, but this history can distract users from what’s happening actively, which during an incident is the most important piece of information. Starting today, the status page will display our current availability and inform users of any degradation in service with real-time updates from the team. The history view will continue to be available and can be found under the “incident history” link. As a reminder, if you want to receive updates on any status changes, you can subscribe to get notifications whenever GitHub creates, updates, or resolves an incident.

  Check out the new GitHub Status Page.

  

  Follow up to July 13 08:18 UTC

  Since the incident mentioned in July’s GitHub Availability Report, we’ve worked on a number of improvements to both our deployment tooling and to the way we configure our Kubernetes deployments, with the goal of improving the reliability posture of our systems.

  First, we audited all the Kubernetes deployments used in production to remove all usages of the ImagePullPolicy of Always configuration.

  Our philosophy when dealing with Kubernetes configuration is to make it easy for internal users to ship their code to production while continuing to follow best practices. For this reason, we implemented a change that automatically replaces the ImagePullPolicy of Always setting in all our Kubernetes-deployed applications, while still allowing experienced users with particular needs to opt out of this automation.

  Second, we implemented a mechanism equivalent to the one of Kubernetes mutating admission controllers that we use to inject the latest version of sidecar containers, identified by the SHA256 digest of the image.

  These changes allowed us to remove the strong coupling between Kubernetes Pods and the availability of our Docker registry in case of container restarts. We have more improvements in the pipeline that will help further increase the resilience of our Kubernetes deployments and we plan to share more information about those in the future.

  In Summary

  We’re excited to be able to share these updates with you and look forward to future updates as we continue our efforts in making GitHub more resilient every day.

  
  
  https://github.blog/2020-09-02-github-availability-report-august-2020/
• 

  Upgrading GitHub to Ruby 2.7
  • sorangutan  

  1
  1
  Posts
  10
  Views

  

  After many months of work, we deployed GitHub to production using Ruby 2.7 in July. For those who aren’t familiar with GitHub’s stack, we’ve been running on Ruby since the beginning. Many years ago, we ran GitHub on a fork of Ruby (and Rails!) and while that hasn’t been the case for some time, that experience taught us how important it is to keep up with new releases.

  Ruby 2.7 is a unique upgrade because the Ruby Core team has deprecated how keyword arguments behave. With this release, future versions of Ruby will no longer accept passing an options hash when a method expects keyword arguments. At GitHub, we’re committed to running deprecation-free on both Ruby and Rails to prevent falling behind on future upgrades. It’s important to identify major changes early so we can evolve the application when necessary.

  In order to run Ruby 2.7 deprecation-free, we had to fix over 11k warnings. Fixing that many warnings, some of which were coming from external libraries, takes a lot of coordination and teamwork. In order to be successful we needed a solid strategy for sharing the work.

  Strategy

  Just like we did with our Rails upgrade, we set up our application to be dual-bootable in both Ruby 2.6 and Ruby 2.7 by using an environment variable. This made it easy for us to make backwards compatible changes, merge those to the main branch, and avoid maintaining a long running branch for our upgrade. It also made it easier for other engineering teams who needed to make changes to get their system running with the new Ruby version. Due to how large our application is (over 400k lines!) and how many changes go in daily (100’s of PRs!), this drastically simplifies our upgrade process.

  Once we had the build running, we weren’t quite yet ready to ask other teams to help fix warnings. Since Ruby warnings are simply strings in the test output we needed to capture the deprecations and turn them into lists for each team to fix.

  To accomplish this we monkey patched the Warning module in Ruby. Here’s a simplified version of our monkey patch:

  module Warning
    def self.warn(warning)
      root = ENV["RAILS_ROOT"].to_s + "/"
      warning = warning.gsub(root, "")
  
      line = caller_locations.find do |location|
        location.path.end_with?("_test.rb")
      end
  
      origin = line&.path&.gsub(root, "")
  
      WarningsCollector.instance << [warning.chomp, origin]
  
      STDERR.print(message)
    end
  end

  The patch stores the deprecation warning and the test path that caused the warning in a WarningCollector object which writes the warnings to a file and then processes them:

  class WarningsCollector < ParallelCollector
    def process
      filename = "warnings.txt"
      path = File.join(dir, filename)
  
      File.open(path, "a") do |f|
        @data.each do |message, origin|
          f.puts [message, origin].join("*^.^*") # ascii art so we can split on it later.
        end
      end
  
      script = File.absolute_path("../../../script/process-ruby-warnings", __FILE__)
      system(script, dir)
    end
  end
  

  The WarningCollector#process method stores all the warnings in a file called warnings.txt. We then parse warnings using CODEOWNERS and turn them into files that correspond to each owning team.

  Once we had all the warnings processed, we opened issues for those teams with easy-to-follow directions for booting the application in the new Ruby version. Our warning reports included the file emitting the warning, the warning itself, and the test suites that triggered the warnings. They looked like this:

  - [x] `app/jobs/delete_job.rb`
    - **warnings**
      - Line 16: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
    - **test suites that trigger these warnings**
      - test/jobs/delete_job.rb
  

  This process helped us avoid duplicating work across teams and made it simple to determine ownership and status of each warning.

  We tracked warning counts in the Ruby 2.7 CI build to ensure that new code wasn’t introducing new warnings. After a few months, coordinating with 40 teams, 30+ gem upgrades, and 11k warnings our CI build was 100% warning-free. Gems that were unmaintained were replaced with maintained gems. Once we had fixed the warnings, we altered our monkey patch to raise errors in Ruby 2.7 which ensured that all new code going into the GitHub codebase was warning-free.

  Benefits of Upgrading to 2.7

  You may be reading this and wondering why it’s worth doing all this work and investing the engineering resources and time in the Ruby upgrade. If you’ve been writing Ruby for a while you’re likely aware of the difficulty with this particular upgrade. It’s been the topic of conversation in the Ruby community since before the release in December. Regardless of how hard this upgrade was, we saw an impressive improvement in performance. The Ruby Core team is well on their way to fulfilling the promise of Ruby 3.0 being 3x faster.

  First, we saw a drop in the amount of time it takes the application to boot in production mode. In production (this is when the entire application is eager loaded) we saw our boot time drop from an average of ~90 seconds to ~70 seconds. That’s a 20-second drop. Here’s a graph:

  

  This faster boot time means faster deploys which means you get our features, bug fixes, and performance improvements faster as well!

  In addition to an improvement in boot time, we saw a decrease in object allocations which went from ~780k allocations to ~668k allocations. Object allocations affect available memory so it’s important to lower these numbers whenever possible.

  

  Aside from the performance benefits of upgrading, ensuring you stay on the most recent version of your languages and frameworks helps keep your application healthy. Through this process we found a lot of unowned code that was no longer used in the application and deleted it. We also took this opportunity to remove or replace unmaintained gems in our application.

  For gems that were maintained we gave back to the community by sending patches for any gems that were emitting warnings in our application including Rails, rails-controller-testing, capybara, factory_bot, view_component, posix-spawn, github-ds, ruby-kafka, and many others. GitHub believes strongly in supporting the open source community and upgrades are one of many ways that we do so directly.

  Deployment

  There are risks to deploying any major upgrade, but at GitHub we’ve designed processes that reduce this risk drastically.

  For Ruby and Rails upgrades, we run in dual-builds until we’re sure all the tests are passing and code is stable. In addition, we have all teams that work on the core product click test their area of the codebase in a staging environment to ensure there are no obvious issues with the new version.

  Rolling out the upgrade is a big deal, so we do it carefully by increasing the percentage of traffic running on the new version and verifying each deployment is error-free in Sentry and regression-free in Datadog. For this deploy, we rolled out to 2% of traffic and quickly saw a new frozen string exception. Due to our process we were able to rollback quickly and less than 10 users saw an error in one endpoint.

  Once we had a fix for the frozen string exception, we restarted the rollout process and again deployed to 2% of traffic. We let this one sit for 15 minutes before going to the next percentage: 30% of Kubernetes partitions. Again we waited about 15 minutes and after verifying there was no regression we deployed to another 30% to total 60% of Kubernetes partitions.

  Finally, we deployed to 30% of our non-Kubernetes deployment partitions. These deploys take longer because they need to compile Ruby. It’s a bit nerve-wracking waiting 15 minutes for Ruby to compile, but everything went smoothly. From there we did a full-production deploy and merged the upgrade after 30 minutes. Overall the entire deploy took about 2 hours.

  At GitHub, we’ve invested in building out processes for deploying Ruby and Rails upgrades so that we can be confident they are the lowest possible risk. We had no downtime while deploying the Ruby upgrade and our customer impact was almost zero.

  Was it worth it?

  For any companies that are wondering if this upgrade is worth it the answer is: 100%. Even without the performance improvements, falling behind on Ruby upgrades has drastic negative effects on the stability of your codebase. Upgrading Ruby supports your application health, improves performance, fixes language and framework bugs, and guides the future of the language!

  At GitHub, not only do we believe in the open source community, we believe that a strong foundation is the first step to a stable, resilient, and functioning application. Running on the most recent version Ruby helps us do just that. We’re looking forward to Ruby 2.8 and beyond. Happy upgrading!

  
  
  https://github.blog/2020-08-25-upgrading-github-to-ruby-2-7/
• 

  Introducing the Rally + GitHub integration
  • sorangutan  

  1
  1
  Posts
  9
  Views

  

  GitHub’s Professional Services Engineering team has decided to open source another project: Rally + GitHub. You may have seen our most recent open source project, Super Linter. Well, the team has done it again, this time to help users ensure that Rally stays up to date with the latest development in GitHub! 

  Rally + GitHub

  This project integrates GitHub Enterprise Server (and cloud, if you host it yourself) with Broadcom’s Rally project management.

  Every time a pull request is created or updated, Rally + GitHub will check for the existence of a Rally User Story or Defect in the title, body, or commit messages, and then validate that they both exist and are in the correct state within Rally.

  

  Why was it created?

  GitHub Enterprise Server had a legacy Services integration with Rally. The deprecation of legacy Services for GitHub was announced in 2018, and the release of GitHub Enterprise Server 2.20 officially removed this functionality. As a result, many GitHub Enterprise users will be left without the ability to integrate the two platforms when upgrading to recent releases of GitHub Enterprise Server.

  While Broadcom created a new integration for github.com, this functionality does not extend to GitHub Enterprise Server environments.

  Get Started

  We encourage you to check out this project and set it up with your existing Rally instance. A good place to start getting set up is the Get Started guide in the project’s README.md

  We invite you to join us in developing this project! Come engage with us by opening up an issue even just to share your experience with the project.

  

  
  
  https://github.blog/2020-08-18-introducing-the-rally-github-integration/
• 

  Why Write ADRs
  • sorangutan  

  1
  1
  Posts
  9
  Views

  

  Architecture decision records, also known as ADRs, are a great way to document how and why a decision was reached within a codebase. We’ve started to adopt them within the mobile team here at GitHub, documenting decisions that affect the iOS codebase and Android codebase, as well as decisions that affect both mobile clients.

  ADRs are not the most common within open source codebases, but have gained more popularity since ~2017 within long-lived, “evolutionary” codebases like those in more enterprise-y settings.

  So why write an ADR? Why spend time documenting something when a decision has already been made?

  They’re not for you, they’re for future you

  ADRs are not meant to be a self-discovery, reflection process on what you decided. ADRs will help you 6-12 months from now recall what your mindset was when you decided upon that architecture.

  ADRs capture the decision at the time it’s being made. ADRs are the culmination of all those minutes and hours you spent in meetings, on Zoom, in Slack, or jamming through various proof-of-concepts in Xcode. All of the context that’s in your head has a chance to get out into words so that when you’re revisiting the architecture down the road, you can put that context right back into your head.

  The real bonus comes when someone git blames you some months from now and asks you how the GitHubAPIClient module works. Better than setting up a 30-minute pairing call to walk them through the code, you can now link to the ADR you wrote to explain some more about the decisions made while building the GitHubAPIClient module.

  They’re not for you, they’re for your peers

  ADRs force you to write more than a one-liner “this ships the feature for #3128”. They are a longer form of prose to help your teammates understand why the feature is built the way it is, and not built some other way (see: “Alternatives Considered” and “Pros/Cons” within ADRs themselves).

  Something that may be simple for you might be complicated for your teammates. Taking the time to write down what your thought process was as you made decisions gives your teammates the chance to get inside your head. Writing ADRs allows for “decision socialization”, where your team comes to a decision that the team is responsible for maintaining, rather than decisions made in isolation.

  By expanding upon what you write in your pull request titles and descriptions (and you are still writing quality pull request descriptions, right?), you give your teammates more information about how a patch or diff works in a larger system.

  Better yet, by writing an ADR ahead of putting your pull requests up, you’ll get better pull request reviews from the team reviewing it. No longer do you need to explain how line 387 in APIClient+Caching.swift will affect data fetching and caching architecture, because your teammates already understand how you’re changing the system from the ADR you wrote about “Adding cache support to E-Tag entities”.

  They’re not for you, they’re for your future peers

  ADRs are not meant you to show off how smart you are or for folks to fawn over the architecture you built. ADRs are for helping onboard new teammates as they work to understand the codebase and how it has evolved over time.

  As teams scale and grow, the number of lines of communication between teammates increases. A team of three individuals only has three lines of communication (A <> B, A <> C, B <> C). A team of four has six (A <> B, A <> C, A <> D, B <> C, B <> D, C <> D). Wanna do the math for a team of five or six? How about 14 engineers, two designers, two PMs, and three EMs?

  Writing down decisions made helps communicate to your current teammates, but also those who join your team as it scales and grows. By informing your team how and why a decision was made in an asynchronous fashion, you no longer need to “hop on a Zoom call” to onboard each new teammate on a per-architectural-decision basis.

  In the best-case scenario, you’ll have your teammates writing new ADRs for you, superseding the decisions past you made, so that you can learn from your teammates in the future.

  Write more ADRs

  I hope this has convinced you to document the decisions that you’re making as we build software that (hopefully) millions of folks use! As our team grows larger and our codebases grow more entangled and intertwined, architecture decision records are a great way to help future us, our current teammates, and future teammates.

   Thanks for coming to my TED talk.

  More Info

  • joelparkerhenderson/architecture_decision_record: ADR examples and documentation
  • David Ayers from #LeadDevNewYork: “Communicating and documenting architectural decisions”
  • Understand Legacy Code: “Earn future maintainers esteem by writing simple ADRs”
  • IEEE Software: “Architecture Decisions: Demystifying Architecture”

  
  
  https://github.blog/2020-08-13-why-write-adrs/